Patient (2013) 6:143159

DOI 10.1007/s40271-013-0022-3

REVIEW ARTICLE

Not Just Little Adults: Qualitative Methods to Support

the Development of Pediatric Patient-Reported Outcomes
Rob Arbuckle Linda Abetz-Webb

Published online: 3 August 2013

 Springer International Publishing Switzerland 2013

Abstract The US FDA and the European Medicines

Agency (EMA) have issued incentives and laws mandating
clinical research in pediatrics. While guidances for the development and validation of patient-reported outcomes (PROs) or
health-related quality of life (HRQL) measures have been
issued by these agencies, little attention has focused on pediatric PRO development methods. With reference to the literature, this article provides an overview of specific
considerations that should be made with regard to the development of pediatric PRO measures, with a focus on performing
qualitative research to ensure content validity. Throughout the
questionnaire development process it is critical to use developmentally appropriate language and techniques to ensure
outcomes have content validity, and will be reliable and valid
within narrow age bands (02, 35, 68, 911, 1214, 1517
years). For qualitative research, sample sizes within those age
bands must be adequate to demonstrate saturation while taking
into account childrens rapid growth and development. Interview methods, interview guides, and length of interview must
all take developmental stage into account. Drawings, play-doh,
or props can be used to engage the child. Care needs to be taken
during cognitive debriefing, where repeated questioning can
lead a child to change their answers, due to thinking their
answer is incorrect. For the PROs themselves, the greatest
challenge is in measuring outcomes in children aged 58 years.
In this age range, while self-report is generally more valid,
parent reports of observable behaviors are generally more
reliable. As such, team completion or a parent-administered
child report is often the best option for children aged 58 years.
For infants and very young children (aged 04 years), patient
R. Arbuckle (&)
L. Abetz-Webb
Adelphi Values Ltd, Grimshaw Lane, Bollington,
Cheshire SK10 5JB, UK
e-mail: rob.arbuckle@adelphivalues.com

rating of observable behaviors is necessary, and, for adolescents and children aged 9 years and older, self-reported outcomes are generally valid and reliable. In conclusion, the
development of PRO measures for use in children requires
careful tailoring of qualitative methods, and performing
research within narrow age bands. The best reporter should be
carefully considered dependent on the childs age, developmental ability, and the concept being measured, and team
completion should be considered alongside self-completion
and observer measures.
Key Points for Decision Makers
Developing PROs for use in children is more complex than
developing in adultsthe developmental stage of the child
must be considered throughout.
During PRO development it is important that qualitative
methods are tailored to the age groupfor younger children
(aged 69 years) the use of drawings, props and toys can
especially help to elicit information
For the instruments themselves, when obtaining ratings for
infants and younger children (aged \5 years), an observer
report, usually completed by the parent, is necessary
For children aged 69 years, either a parent observer report, a
parent-administered but child-completed PRO or an entirely
child-completed PRO can be appropriate, depending on the
concept
Older children/adolescents can answer themselves but the
content and wording of questions might still be different from
that used in adult measures

1 Background
1.1 Pediatric Legislation (Carrot and Stick)
Historically, relatively few clinical studies were conducted
to evaluate the efficacy of pharmaceutical treatments in

144

pediatric populations. The reasons included ethical concerns about testing experimental treatments with children
and an expectation of limited return on investment. In
addition, the challenges posed by the continuous growth
and development of children makes dosing and staging of
disease difficult, and increases the sample sizes needed to
demonstrate safety and efficacy in all age groups. Developmental changes and their impact on childrens physical
and cognitive capabilities are additional challenges for selfreport endpoints in trials [1, 2].
The result is that many treatments are used off-label in
pediatric populationswithout marketing authorization or
licensing and without evidence of safety and efficacy. The
lack of pediatric clinical trials has resulted in misdosing of
children and withholding therapies from children because
there is insufficient evidence to make a valid risk/benefit
evaluation [3].
In response, between 2002 and 2007 both the US FDA
and European Medicines Agency (EMA) have issued legislation to stimulate pediatric research [49].
These legislations require by law that sponsors initiate a
pediatric development program for any drug which could
benefit children (characterized as the stick). If the sponsor
carries out that pediatric program to the satisfaction of the
respective regulators, the sponsor is then rewarded with a
6-month patent extension for marketing exclusivity in the
EU and US markets (the carrot). The extension is further
extended to 2 years for orphan drugs.
This legislation has generally been deemed a success,
with rapid increase in the number of treatments for which
pediatric studies are being conducted, or are at least planned [1012].

Fig. 1 Overview of the

pediatric patient-reported
outcome development process

R. Arbuckle, L. Abetz-Webb

1.2 Patient-Reported Outcomes (PRO)/Health-Related

Quality of Life (HRQL) Guidances
In adult indications, the use of patient-reported outcomes
(PROs) to support labelling claims has been the focus of a
great deal of interest from both regulators and the research
community in recent years [1316]. In particular, the FDA
PRO guidance [14] and, to a lesser extent, the EMA HRQL
reflection paper [13] have been influential in establishing
and driving standards for the development, validation and
use of PROs in clinical trials and other research studies. In
particular, the FDA has placed far greater importance on
input of patients to ensure the content validity of PROs
used to support label claims [17]. Figure 1 illustrates the
iterative process that the FDA recommends for the rigorous
development of a PRO instrument that will be used to
support product label claims.
However, the development, validation, and use of PROs
in pediatric studies has received less attention. One paragraph in the FDA PRO guidance makes reference to the
specific considerations that must be taken into account for
pediatric PROs [14]. This paragraph highlights the need for
development within narrow age groupings and discourages
proxy-reported outcome measures, stressing that observer reports that focus on events or behaviors that can be
observed should be used for patients who cannot respond
themselves. The EMA HRQL reflection paper makes no
specific mention of pediatric HRQL outcomes [13]. However, there is repeated acknowledgement in the literature
that there are many complicating issues and specific considerations that must be taken account of in pediatric PRO
development [10, 1820].

Not Just Little Adults

Matza et al. [18] and, more recently, Solans et al. [21]

reviewed the state of the art in terms of pediatric HRQL
measurement, while Rothman and Kleinman [19] have
provided a summary of the expectations of regulators. In
addition, survey development theorists [2224] provide
valuable guidance regarding cognitive interviewing in
pediatrics, and Bevans et al. [1] recently published a
valuable paper focussed on the conceptualization and
understanding of child health, with guidance regarding
testing psychometric properties, and some suggestions
regarding the capabilities of different ages of children to
complete self-report measures. However, since the FDA
and EMA guidance documents have been released (in 2009
and 2005, respectively), there have been relatively few
publications in the literature that provide detailed guidance
regarding qualitative research methods for performing
concept elicitation and cognitive testing to support the
development of pediatric PRO measures.
Thus, in a rapidly evolving PRO development context,
there is a need for debate regarding specific considerations
for pediatrics. How should qualitative PRO development
methods be tailored for pediatric populations? Are there
content validity considerations that are specific to the
development of pediatric PROs? For the instruments
themselves, at what age/developmental stage are children
too young to provide reliable reports? For these children,
how can we evaluate important concepts that cannot be
observed by caregivers or clinicians?
The objective of this article is to provide an overview of
specific methods and considerations for performing qualitative research to support the development of pediatric PRO
instruments. The methods described are particularly relevant
for pediatric PROs that will form clinical trial endpoints, but
can be relevant for PROs that will be used for other purposes.
With reference to examples from the literature and our own
research, the aim is to outline the state of the art in terms of
qualitative methods for pediatric PRO development, and to
stimulate debate and research in this important area.

2 Pediatric Developmental Milestones and Age Ranges

for Pediatric PRO Development and PRO
Completion
The FDA PRO guidance for industry acknowledges that, for
children and adolescents, instrument development and
validation testing within fairly narrow age groupings is
important to account for developmental differences and to
determine the lower age limit at which children can understand the questions and provide reliable and valid responses
that can be compared across age categories [14]. However,
to the knowledge of the present authors, the FDA has not
made any public statements regarding what the fairly

145

narrow age groupings should be, or what the lower age

limits should be.
It is generally acknowledged that the exact cut-offs for
age ranges will vary according to the disease population,
the context of use, and the concept targeted for measurement. It is of relevance here to consider the literature
regarding pediatric developmental milestones, in order to
determine what children of different ages are likely to be
capable of [1, 25]. However, it is important to bear in mind
that all boundaries are fuzzy and in fact there can be wide
variation among different children of the same age in terms
of their motor, cognitive, and linguistic capabilities [26
31]. Abilities will vary depending on genetic differences,
learning, experience, and socio-environmental factors [31].
A task that might be easy for one 6-year-old child to
complete may be very challenging for a less cognitively
advanced 8-year-old.
It should be noted that there are two separate issues to
consider here: the age at which a child can provide reliable
information in an interview (as part of PRO development)
and the age at which a child can reliably respond to a
questionnaire which is an outcome measure in a clinical
trial or any other large, quantitative study.
2.1 Youngest Age at Which a Child can Report or be
Interviewed
For an understanding of child developmental milestones,
the Piagetian theory and information processing theory
provide competing theoretical perspectives. Piaget suggested that a childs cognitive development progresses
across the following four stages: sensori-motor development, pre-operational thought (broadly age 26 years),
concrete operational thought (411 years), and formal
thought (1217 years). In contrast, the information processing theory posits that cognition evolves as children
develop improvements in attention, memory, and problemsolving skills separately [32]. While no formal development stages are outlined, the information processing theory
proposes that improvements in cognition develop during
middle childhood where the child is able to learn new
memory strategies (e.g. rehearsal), pay attention to relevant
information, and widen their knowledge basis through their
experiences [33]. Also of relevance for younger children is
the development of a theory of mindthe ability to
attribute mental states to oneself and otherswhich is
considered to be central to the development of a childs
reasoning, ability to relate to others, and development of
representation and language [34, 35]. Theorists suggest that
theory of mind and representational abilities initially
develop around the age of 35 years but that the ability to
introspect about ones own thoughts does not seem to fully
develop until ages 68 years.

146

So, at what age can children self-report or be interviewed? Matza et al. [18] suggest that children can begin
reporting on concrete domains between the ages of 4 and 6
years, whereas elsewhere it has been suggested that below
the age of 7 years children do not have sufficient cognitive
skills to be effectively and systematically questioned [24].
Borgers et al. [31] argue that the stages of cognitive
development are generally more overlapping than Piaget
suggested, but still argue that they provide a useful
framework, supporting the age of 6 years as a cut-off for
self-reporting [26, 27]. At the age of 56 years children can
talk about people and events, make drawings, and re-enact
events in playall skills important for participation in
concept elicitation interviews. However, before the age of
6 years children do not necessarily take into account what
the listener can and cannot know, and may not understand
the difference between the past, present, and future. Thus,
the child may be able to understand and report on how they
feel right now, but have difficulty reporting on their
symptoms or quality of life over the previous week or
month.
During middle childhood, children begin to think logically and are able to apply rules among objects and events
in the real world. Changes in perceptual and cognitive
abilities between the ages of 5 and 7 years include learning
to tell right from left and being able to draw the biggest
and smallest squares [36, 37]. Before age 7 years most
children are not concerned if they have contradicted
themselves; after this age they avoid contradiction. From
an information processing perspective, children aged 57
years develop the ability to screen out irrelevant information, learn new memory strategies, and have a better
understanding of their memory ability (meta-memory).
Such skills are relevant to concept elicitation and cognitive
interviews in which children are asked about their ability to
complete a PRO and report on their recall ability. Children
at this age are better able to recall specific concrete events
in time (e.g. the last time they threw up, were in hospital),
compared with recalling a more vague timeframe that
might require them to average over several days (e.g. the
last week).
Abstractions and hypothetical situations are still not
understood well. This can pose significant challenges for
cognitive debriefing of questionnaires, which often relies
on hypothetical questioning. It should also be noted that
children aged 47 years are highly suggestible [31] and
will often give interviewers responses that they want to
hear, and so care must be taken to ask questions in a neutral
manner. Children of this age can be very literal and may
have difficulty with questions that use logical operators
such as or or negations [24]. Double negatives should
also be avoided. These findings provide support for the cutoff for a child being able to be interviewed on their own

R. Arbuckle, L. Abetz-Webb

falling somewhere between ages 6 and 8 years. At 5 years

of age it is unlikely that linguistically and cognitively the
child will able to self-report on their own, but by 8 years of
age the majority of children will have moved to Piagets
concrete operational stage and will be able to self-report on
simple concepts asked in questions that are simply and
clearly worded.
Finally, adolescents are best able to think like adults and
respond to probes in an interview and self-report on
questionnaires. Adolescents are far better able to recall
over longer periods (e.g. 7 days) and to rate more abstract
concepts. Adolescents are able to understand questions in
an interview that ask about hypothetical events, e.g. Tell
me about how you would answer this question if you had a
really bad asthma attack? In an interviewing context,
adolescents are much better at understanding the purpose
of the interview and what information is required. Compared with children, adolescents are generally more
focused on the task and tend not to talk about irrelevant
topics. Yes or no answers are less frequent and, instead,
adolescents are able to offer helpful suggestions and
insights.
Nevertheless, the issues that are most important to
adolescents may be quite different from adults. Adolescents are developing their ego identities and peer perceptions become of paramount importance. Health problems
that impact on their body image or ability to socialize with
peers can be of far greater importance than other issues.
This egocentric approach will play a predominant role in
their answers to questions and the way in which a questionnaire should be worded. Likewise, adolescents may
become insulted if they perceive that they are given a
childrens questionnaire with faces as response continuums or the like.
In summary, a broad age range of between 6 and 8 years
has strong support as the youngest age at which a child can
be meaningfully interviewed, and asked to self-report on a
PRO. Some researchers have shown that children as young
as 4 or 5 years of age may be capable of providing some
information on concrete aspects of their health status [10,
18, 38]. It has also been reported that children as young as
4 years can report pain [39]; however, children this young
will likely have difficulty with more complex questions.
As noted above, the developmental stages that each
child goes through must be taken into account when
designing both interview guides for qualitative research
and PRO questions. Table 1 provides a summary of
developmental milestones, appropriate qualitative methods
for questionnaire development, and who are appropriate
reporters for different age groups. Where health conditions
impact on the cognitive or physical development of the
child, (e.g. Downs Syndrome, autism), these chronological
milestones may not apply. Details regarding questionnaire

Not Just Little Adults

development methods and the optimal reporters are provided in the remainder of the paper.

3 Methods for the Development of Pediatric PRO

Instruments
3.1 Reviewing the Literature and Pediatric Concept
Elicitation Interviewing
In adults or children, the first steps in developing or
selecting a PRO must include identifying the concept(s) targeted for measurement. The next step is to obtain
concept-relevant information to support the development
of a conceptual framework, item and response options to
measure each concept adequately [14]. This can be done
through review of the literature, and/or by performing
qualitative research with appropriate reporters (in pediatrics this can include children, parents/caregivers, and
occasionally other reporters such as teachers or clinical
staff). Methods for reviewing the literature are much the
same in pediatrics as in adults. However, there is the added
challenge that the qualitative literature (which is preferable
for identifying concepts) in pediatrics is often sparse, and
rarely breaks findings down into the narrow age bands
required for thorough instrument development. One option
is to consider the grey literature and online blogs and
chatrooms as additional sources of information. Otherwise,
the limited pediatric literature available places further
importance on conducting rigorous, open-ended qualitative
research. When conducting such research, the childs age
and developmental stage must be taken into account and
methods tailored appropriately. Below we outline key
considerations.
3.1.1 Qualitative Research: Interviews vs. Focus Groups
In adults, open-ended qualitative research typically
involves conducting either interviews or focus groups (or
some combination of the two methods). Consideration of
appropriate methods for qualitative research with children/
adolescents should be based on their competencies and
interests [40], and not on the assumption that what is
appropriate for adults is also appropriate for children.
Focus groups with children have been successfully used to
support pediatric PRO development [41, 42]. It has been
suggested that focus groups have the advantage that they
remove the emphasis on the adultchild relationship, perhaps reducing the likelihood that participants will respond
in the way they think the adult would desire [43]. There is
less pressure on a child to respond to every question,
reducing the chance that children will respond to questions
they do not fully understand [44]. It has also been

147

suggested that focus groups have the advantage of

acknowledging that the participants are the experts; however, the same could be equally said of one-to-one interviews if they are framed appropriately [45]. Focus groups
have also been suggested to generate greater depth due to
group members sparking ideas off one another, although
there has been found to be little empirical evidence in
support of this idea [46].
However, there is reason for caution in conducting focus
groups in children and adolescents. Intimidation within the
group setting may inhibit interaction [43, 44], and this is
even more likely where children of mixed ages are included within a single group. Furthermore, children in focus
groups are more likely to adopt previously raised themes
than suggest their own [43, 44]. Even where children are of
the same age, conformity to peer pressure can exert considerable influence on the comments individual children/
adolescents make [24]. Even adults are susceptible to social
desirability bias in group situations [47, 48], and it is likely
that this is further amplified in children and adolescents
[24, 49]. This is of most concern among young adolescents
when peer pressure is perhaps most influential [43, 50].
However, Donaldson [51] has provided evidence that
younger children are also extremely suggestible, and may
give answers that they think adults are looking for if
questions are not posed in a very neutral way (a view
corroborated by survey development theorists) [24, 31].
While an interviewer can try to control such bias in a oneon-one interview situation, it may be more difficult in a
focus group. In terms of practicalities, for children younger
than 12 years of age, it can be difficult to keep the attention
of all children in a focus group situation, even if focus
groups are kept relatively small in size (i.e. not more than
four participants). For all of these reasons, we suggest oneon-one interviews in preference to focus groups in pediatric
populations.
If, because of practical and/or financial limitations,
focus groups must used, then limiting the size of the groups
is recommended. Heary and Hennessy [43] suggest the
optimum size is smaller than for adults, and propose
between four and six participants. Participants in each
group should be broadly similar in age, for the reasons
suggested above, and also due to developmental and cognitive differences among children of different ages [43,
50]. Conducting groups with participants all of the same
gender should also be considered, particularly for sensitive
topics or health conditions, such as dermatological, urological, or bowel conditions [43, 50, 52].
3.1.2 Who to Interview?
There is debate in the pediatric PRO literature regarding
who is the most appropriate respondent, and who will

148

R. Arbuckle, L. Abetz-Webb

Table 1 Summary of developmental milestones, appropriate methods for questionnaire development, and who are appropriate reporters for
different age groups [22, 24, 31, 104]
Stage/
age
range
(years)

Developmental milestones

Qualitative questionnaire development

methods

Best reporter in outcome measure

02

Sensori-motor stage:

Parent/caregiver interviews or focus

groups onlycannot interview child
directly

Child cannot self-report, must rely on a

parent/caregiver-completed outcome
measure focussed on observable
behaviors

Usually child interviews are not

recommended and only parent/
caregiver interviews or focus groups
are conducted

For most concepts, child is unable to

self-report, must rely on parent/
caregiver-completed outcome measure
focussed on observable behaviors

Can consider interviewing parent and

child together, but can only ask the
child about very simple concepts, and
about what they are experiencing at
that moment as accurate recall for
children of this age is limited

Parent can be asked to report on what

their child has told them (e.g. Did
your child tell you he/she had a
stomach ache today?)

Recommend interviewing both the child

and a parent/caregiver (in separate
interviews)

Recommend including both a childcompleted patient-reported outcome

(PRO) measure and a parent/caregivercompleted measure. Which is
considered primary will depend on
the concepts being measured and the
findings of the qualitative development
research (including both open-ended
interviews and cognitive interviews)

Differentiates self from objects and recognizes

self as agent of action. Towards the end,
achieves object permanence, realising that
things continue to exist even when no longer
present
35

Preoperational stage:
Acquisition of language is primary cognitive
development task. Able to formulate mental
representations of objects that are not there
and to understand that symbols that stand for
something, such as a red light meaning stop.
Able to report how they feel right now, but
any recall likely to be challenging. Prone to
precausal reasoning: unable to distinguish
between psychological and physical causes.
Likewise, thinking is based on concrete
mental images and memory is limited

611

Concrete operational stage where logical

thinking becomes possible. These children
think more like adults, although they do tend
to think in terms of black and white rather
than grey areas in between. Capacity to store
and manipulate information still limited. At
this stage, the child can think back and recall
over short timeframes. However, he/she is
limited to thinking back to events that are
concrete. Abstractions and hypothetical
situations are not understood well

For child interviews, must keep

questions simple and direct, focussed
on the here and now, and tangible,
salient concepts as much as possible.
Interviewer must be careful not to lead
the child. Parent should not be present
Use of toys and drawing activities
recommended to keep the childs
attention

For simple concepts (e.g. presence of

pain), can perhaps ask the child to
report in an interview-administered
format

Can be valuable to ask the parent to

remind the child to complete the PRO
and regarding salient events relevant to
the childs symptoms, but the child
should determine responses to each
question. Alternatively, child measure
can be interviewer administered
Child-completed measure must use very
simple, direct language, recall period
should not be more than 24 h for most
concepts and instantaneous/right now
recall periods are preferable

1214

1517

Formal operational:
Able to think logically and respond to
questionnaires. Can start to understand
hypothetical questions. Capacity to
temporarily store and process information
considerably superior to that of younger
children. Can think like adults but the issues
that are most important may be quite
different to adults. Developing ego identity
and peer perceptions are of paramount
importance. Thus, symptoms that inhibit
participation in social activities or are
embarrassing are likely to be perceived as the
most bothersome. This egocentric
perspective will play a predominant role in
answers to questions

Recommend interviewing both the

adolescent and a parent/caregiver (in
separate interviews). Parent interviews
less important and could be dropped
depending on the concept

Adolescent-completed PRO may be

sufficient, parent/caregiver-completed
measure focussed on observations
could add insight depending on the
condition and the concept of interest

Adolescent interviews or focus groups

essential. Interviews recommended in
preference to focus groups for most
concepts

Adolescent-completed PRO likely to be

sufficient, parent/caregiver-completed
measure focussed on observations
could add insight depending on the
condition and the concept of interest,
but parents are less likely to be as
knowledgable about their childs
condition/experiences

Parent/caregiver interviews/focus groups

optionaltheir importance depends on
the concept

Not Just Little Adults

provide the most reliable and valid information to support

the generation of a PRO conceptual framework and items,
response scales and recall periods [18, 53]. As described
above, while there is some evidence children aged 45
years can report on specific, concrete aspects of their
health, they cannot answer more complicated questions,
and will often become very shy and uncommunicative. We
recommend therefore that only interviews with parent/
caregivers and clinicians are performed for this age group
and that age 6 years is the youngest age that should be
interviewed in a pediatric sample, to be confident of being
able to achieve saturation.
We recommend that both a parent/caregiver and the
child should be interviewed separately for children aged
611 years. In addition, for development of disease-specific
symptom measures, it is recommended that clinicians are
also interviewed. Similarly, when studying more severe
neurological conditions, a carer, psychologist, or neurologist who has worked closely with children with the targeted
condition can provide valuable information. Also for some
symptoms or domains of functioning or HRQL, teachers
can provide a valuable additional source of information
(e.g. in attention-deficit hyperactivity disorder (ADHD),
the childs behavior and ability to concentrate in class
might be very different from their behavior at home, and
this is a key consideration).
Inclusion of many different categories of interview
participant increases the resources, costs, and time
required. However, for pediatric studies each different
respondent provides a different piece of the puzzle, giving information that is complementary and rarely entirely
redundant. For example, if interviewing about asthma
symptoms, the child him/herself can best report on how
irritating their cough feels and how much their asthma
symptoms limit their participation in sports. However, their
parent might be better able to recall and report on how
much the childs asthma symptoms disturb their sleep.
It might be tempting to think that mom knows best and
that parents provide more objective information than the
child; however, some information can only be known by
the children themselves, e.g. their experience of pain and
emotions. Parental views can be influenced by their own
health experiences and expectations, and the child can
learn to hide their true emotions from parents and carers
[54, 55]. Furthermore, for children of school age (and also
younger children where the child is in daycare) the parent
will often spend only a few hours a day with the child, and
so may not know the frequency or severity of symptoms, or
the impact on functioning or HRQL domains [54].
There is increasing consensus that parents should be
asked to only report on observable signs, symptoms and
behavior [56] (defined by Donald Patrick in a breakout
session at the 2009 Drug Information Association

149

Outcomes Special Interest Group as anything the caregiver can see, hear, smell, feel or, strictly speaking, taste).
Arguably, this can include things the child tells the parent,
although they should be recorded separately from signs,
symptoms, and behaviors the parent observes directly [54].
Sometimes there may be differences in the information
provided by different interview participants that can be
difficult to resolve. Other evidence from the interviews
should be used to judge who might be considered the more
reliable reporter, but in some cases it may be necessary to
flag both reports as questionable. One approach that can be
useful when parents and children are interviewed together
is to bring them together at the end of the interview. Then it
can be possible to highlight discrepancies for discussion,
taking care to avoid judgment or the suggestion that either
reporter is more trusted.
3.1.3 Presence of Parent/Caregiver when Interviewing
An issue for consideration when designing a pediatric
interview study is whether parent/caregivers should be
allowed to be present during the interview. Generally
parents should not be present as they may coach or influence the child. Even if asked to keep silent, their presence
might influence the child to give responses that he/she
knows the parent would prefer. Ethics review boards may
insist that the parent be given the option of being in the
room during the interview of children aged younger than
11 years, and some parents may insist upon it too. One
solution is to use a research facility that has a one-way
mirror viewing facility. This allows the parent to view the
interview from the next room (without being able to hear
the discussion) and be reassured that their child is content.
As the child is unable to hear or see their parent they are
typically less inhibited by their presence. However, for
ethical reasons, the child should be made aware that the
parent is behind the mirror watching.
When interviewing both the parent and child in the same
family it is recommended to have a team of two interviewers so that both can be interviewed simultaneously.
This approach also allows for the possibility that the child
and parent can be brought together for a short section of
interviewing at the end.
3.1.4 Saturating and Analysing Concept Elicitation
and Cognitive Debriefing Interviews by Age
The idea of conceptual saturation is generally used to
determine sample sizes in qualitative interviews. Saturation
is commonly defined as the point at which no new
information emerges [57, 58]. In pediatric qualitative
research, it is important to demonstrate that saturation has
been achieved within narrow, pre-specified age ranges,

150

R. Arbuckle, L. Abetz-Webb

given the rapid developmental changes taking place. If

there is any doubt that saturation has been achieved, then
further interviews/groups should be conducted. The specific age ranges that are appropriate will depend on the
complexity of the concept(s) and condition being studied.
As an approximate guide we recommend demonstrating
saturation within each of the following age ranges, based
on the developmental milestones outlined in Table 1:

02 years (interview parent/caregivers only)

35 years (interview parent/caregivers only)
68 years (interview children and parent/caregivers)
911 years (interview children and parent/caregivers)
1214 years (interview adolescents and parent/
caregivers)
1517 years (interview children and possibly parent/
caregivers [parent/caregivers not essential in this age
band])

Researchers often combine the 6- to 8-year and 9- to

11-year age bands into a single group. However, given the
considerable cognitive and linguistic changes that occur
during this period we suggest that it is critical that saturation and psychometric validity are both evaluated within
each of these more narrow age ranges. It is during this
period that the majority of children develop causal reasoning and start to learn to read. The 12- to 14-year and 15to 17-year age bands are also frequently combined. The
differences between these age bands are less profound than
between the 6- to 8-year and 9- to 11-year age bands, and
combining the adolescent age bands can at times be
appropriate; this decision needs to be made based upon the
disease and concept(s) under study.
3.1.5 Interview Guides
When developing a pediatric concept elicitation discussion
guide, a key consideration is to ensure the guide is written
in child-friendly, age-appropriate language. This is particularly important for young children (those aged 68 years),
for whom questions should focus on the concrete and
abstract enquiries should be avoided. For example, in disease areas as diverse as asthma and restless legs syndrome
(RLS), the word severe can be difficult for even young
adolescents aged 1214 years to understand, thus making
the concept of symptom severity more challenging to ask
about [20]. Even when using simple words, care must be
taken to ensure the child understands what is intended.
Words considered naughty for children should be avoided. For example, snot can be considered rude by some
children. Getting input from parents on appropriate terminology that the child uses prior to the start of the interview
is essential.

In addition it should be considered that there are some

concepts relevant to adults, and perhaps adolescents, that
are inappropriate for children (e.g. work, sexual functioning) and vice versa.
Using role-play to test a guide is particularly useful for
pediatric interviewing where interviewers are likely to run
into difficulty or to find they are getting monosyllabic or
yes/no responses. Warm-up exercises can also be helpful to
put the child at ease and to ensure they understand what is
required and the purpose of the questioning, although such
exercises should be kept short. Similarly, making it clear
during the consenting process what the reason for and
value of the interview is can help the child understand and
therefore engage with the process.
The interviewer should learn the guide well, allowing
them to improvise and adapt if they run into difficulty or
the childs attention waivers. As much as possible the
children themselves should be encouraged to initiate discussions on topics so that the words used are ones that they
are familiar with, and the interviewer should then use those
terms throughout. There are choices to be made regarding
the level of detail provided in the interview guide. It could
be argued that for a truly open-ended or grounded theory
approach an entirely blank canvas should be used. However, such an approach runs the risk that topics of interest
may not arise, or that the interviewer might forget to ask
issues of importance. In contrast, a guide that is too
detailed can be followed too prescriptively. Generally, a
balance can be struck in having a guide that reminds the
interviewer of the main topics to be covered without being
too inflexible.
Children easily become bored in interviews (particularly
children with ADHD or mental health conditions), so
having options that allow the interviewer to vary the focus
of the interview and utilize interactive techniques is
important. The interviewer should be careful to talk to the
child on their level, using appropriate language and
avoiding the use of any adult or clinical terminology. As in
all qualitative interviewing, open-ended questions and
taking time to engage with the respondent are of even more
importance with children and can help interviewers to
avoid some of the issues raised above.
3.1.6 Length of the Interview
The appropriate length of time for an interview with a child
is generally shorter than for adults [43]. The exact length of
time a childs attention can be held will vary according to
the task but, typically, even adolescents struggle to concentrate for more than 1 h, and no more than 30 min
without a break is recommended for children aged 611
years. For focus groups, a slightly longer duration of

Not Just Little Adults

approximately 45 min is feasible. Taking breaks is recommended, as is allowing children to occasionally talk
about subjects that are off-topic if it helps them to remain
engaged with the interviewer.
3.1.7 Use of Drawings and Props
One method that can be of value for helping children to talk
about their health is the use of drawings [20]. Asking a child
to draw what their health condition feels like not only gives
the child a more fun task, but it can also help to initiate
discussion. This often results in powerful, unprompted
qualitative quotesseveral examples of drawings with
accompanying quotes are provided in Fig. 2 [59]. The
drawings can also be subject to content analysis.
The use of props and toys can also be helpful, although
too many can be distracting. For example, play-doh has
proved useful when interviewing children about constipation, asking them to model their last bowel movement [60,
61]. Such aids can break the monotony of the interview and

Fig. 2 Examples of drawings depicting restless legs sensations and

associated descriptions (reproduced from Picchietti et al. [59], with
permission). a A 7-year-old boy: They (my legs) like feel weird, and
they want to kick. b An 8-year-old boy: Its like my legs are wiggly
and like (inaudible). c A 9-year-old boy: Its going to be hard to tell
that this is like a bruise. Im trying to stretch my legs out, my thighs
out right here, so Im going like this. Im like trying to stretch my legs
out but at the same time its sort of hurting. d An 11-year-old girl:
Well this picture shows like, see like its ant bites thats kind of
showing you that its really hurting me like in those areas.

151

engage the child. The downside of using drawings, props, or

toys is that they can be distracting, can waste time or lead
the interview off-topic, and comments can be more open to
interpretation. As such, these methods should be used
judiciously, and are of most value when children are having
difficulty focussing and are already distracted.
3.2 Generation or Modification of Items for Pediatric
PROs
When developing or modifying items that will be used in
pediatric populations the key is to ensure that the resulting
items are as simple and clear as possible, and to ensure they
will be understood by the target population [1]. This is
generally the aim for the development of adult PROs too
(in adult PRO development a common rule of thumb is to
aim for a 6th grade reading level, to ensure the PRO is
comprehensible to adults of all abilities), but with children
it is clearly of greater importance. Woolley et al. [22]
provides an example where an item worded I am happy
with myself was misunderstood and interpreted to be
simply asking about mood, rather than self-esteem,
whereas simpler, more concrete items such as I am smart
or I am proud of myself were better understood. Children (especially those aged 710 years) can misinterpret or
have difficulty with vague terms because they tend to
interpret words literally [24].
Common wording pitfalls include asking about more
than one concept within a single item; items that ask an
individual to compare their current symptoms/functioning/
HRQL with their symptoms/functioning/HRQL at some
earlier time (the beginning of the study, or prior to having
the illness); use of double negatives; wording that is vague
and unclear; and response options that are vague or do not
fit with the question. In addition, using a large font and
laying out questions in a clear manner are additional
practical considerations that can aid comprehension considerably for children but which are often neglected. In
addition, for electronic PROs (ePROs), having the device
present items to the child through audio as well as visually
is increasingly an option. While there is limited literature
on the use of this method in PROs, it is a method likely to
further aid comprehension, and so improve reliability.
If a symptom measure is being developed it is strongly
recommended that at least one expert physician is included
in the item generation process, to ensure that the items
generated are clinically relevant. Consulting the literature
is also recommended at this juncture, and input from parents and teachers (in addition to incorporation of parent
interviews during concept elicitation) can also add insight.
Teacher involvement is particularly recommended for
pediatric measures that include behavioral or psychiatric
domains.

152

For a child self-report, modifying an adult questionnaire

is not recommended, even if guided by qualitative research.
The optimum item wording and ideal response options are
generally very different for children compared with adults.
Moreover, not all concepts relevant to adults are relevant or
appropriate for children and vice versa (e.g. questions
about work, sexual health, play). Instead, it is recommended that items are generated based on qualitative data
findings, and then afterwards the concepts and items created can be compared with the equivalent adult questionnaire to evaluate conceptual equivalence.
Parent/caregiver questionnaires (for infants and younger
children) should focus on asking about observable behaviors, as outlined in the FDA PRO guidance. For parentcompleted observer measures, there may be times when a
parent is only aware of a symptom or impact being experienced because of what their child tells them, not due to
direct observation. Questions should make it clear whether
a parent is to base their response on direct observation or
their child telling them about the event. (Did your child
tell you his/her head hurt). Another option is to use a
single question and provide check boxes for the parent to
indicate if a rating is based on what he/she observed or
what the child told him/her. For item generation based on
both child and parent data, the child comments should be
given primacy over parent/caregiver quotes.
3.2.1 Response Options
Particular consideration should be given to developing
appropriate response options for pediatric PROs. As mentioned above, even children as old as 14 years can have
difficulty understanding what is meant by questions that
ask about severity or use severity response options (e.g.
mild, moderate, severe) [20]. Wording that is better
understood by children and adolescents is to ask how bad
a symptom is.
Response options of not like me, a little like me and
a lot like me have also been found to be misunderstood by
young children [22]. There is evidence that children prefer
adjectival response scales over a visual analog scale
(VAS), and find them easier to complete [62], perhaps
because the discrete number of choices is better suited to
the concrete and dichotomous thinking exhibited by young
children [63].
There is some evidence that children younger than 8
years of age cannot use Likert-type response scales [64,
65], and it has been suggested that fewer response options
are better for children younger than 8 years of age [18].
Using only dichotomous yes/no response options is certainly preferable in terms of the comprehension and
understanding, but limits the responsiveness of the measure
being developed. However, in a recent study, when the

R. Arbuckle, L. Abetz-Webb

response options were limited to No/Yes, a little/Yes, a

lot it was found that several children, aged 717 years,
requested an in between option [20]. It may be that with
the increasing use of computers and electronic games by
children, younger children are becoming increasingly literate at using such response scales.
A decision has to be made regarding whether the
response scale should ask about severity/intensity, frequency, bothersomeness, or use an agreement scale. Frequency scales and severity scales that avoid the use of
high-level words are widely used and can be understood by
children, but the specific wording should be considered
carefully. Bothersomeness is a more complex concept and
so is best avoided, as are agreement scales. Generally, the
choice between severity and frequency should be made
based on which concept is more relevant to measure.
Graphics used to help make response scales more
meaningful to children include happy and sad faces [66,
67], circles of increasing sizes [18], and boxes that vary in
how full they are. See Fig. 3 for examples of each of these.
Such graphics make response scales more concrete for
children [24]. Scales made up of faces that vary in terms of
happiness/unhappiness are commonly used, particularly
when rating pain, as in the Wong-Baker pain scale [68].
However, the extreme negative end of the scale typically
includes a very unhappy face with tears. This particular
response option has been critiqued on the basis that some
children, particularly boys, may not choose it if the
symptom/impact has not made them cry (or they do not
wish to admit they cried). In a study comparing three types
of graphic response scales (circles, faces, and thermometer), the graded circles scale was found to have the highest
reliability [69]. In addition, ability items showed higher
reliability when the circles scale was used, but responses to
social items had higher reliability when the faces scale was
used, suggesting it is helpful to fit the graphic to the concept [69]. Any visual aid may lead children to focus more
on the visuals than the words, and so should be considered
carefully. Moreover, some visual aids are harder to add into
ePROs than others, although the increasing availability of
larger screens and more sophisticated devices makes this
less and less of a problem.
3.2.2 Recall Period
There is evidence that children older than 8 years of age
can use a 4-week recall period [64, 70]. However, findings
from a recent study that included cognitive debriefing with
children have suggested that recalling accurately over a
period of more than 24 h can be challenging for children
younger than 11 years of age [20]. Survey-design theorists
recommend avoiding retrospective recall where possible,
and otherwise linking recall to concrete, salient,

Not Just Little Adults

153

including frequent reminders of the targeted recall period is

strongly recommended.
3.3 Conducting Cognitive Debriefing Interviews
with Children

Fig. 3 Examples of response scales

memorable experiences [24]. For example, for a constipation measure it might be advised to ask children to recall
their last bowel movement (or poop in child-friendly
language), rather than their bowel movements in the last
24 h. However, this is not always practical depending on
the symptom/concept being measured.
There is evidence that younger children develop
scripts about what usually happens for regularly occurring events/activities [71, 72]. Such scripts involve the
child generalizing their memories for events or activities
that occur repeatedly and routinely, and could limit the
accuracy of recall for specific events. The child might not
recall over the last 24 hours/7 days/4 weeks as they are
being asked to, but instead might choose an answer based
on their script or general experience. During cognitive
debriefing, children, adolescents, and adults often indicate
that they are recalling over a longer time period than is
specified in a questionnaire. However, it is often unclear
whether this is due to the complicated hypothetical questioning that is necessary in a cognitive debriefing interview
(where the patient is usually only administered the PRO
once, and is asked to discuss it in detail).
To guard against the risk that recall periods might be
ignored, it is recommended to bold the recall period and
include it in every question, rather than just at the top of the
page or on an introductory screen on an ePRO. Moreover,
detailed training prior to PRO completion is essential to
increase the likelihood that the children understand and use
the recall period. Including graphical illustrations of the
recall period is also recommended. At a minimum,

Following item generation, the next step in development of

a PRO is cognitive debriefing in the target population. Such
cognitive debriefing involves administering the PRO to a
small number (typically between 12 and 20) of patients in
the target population and then interviewing them in detail
to ensure all instructions, questions, response options, and
the recall period (i.e. all aspects of the instrument) are
relevant and well understood [14, 17]. This testing of the
content validity of the PRO is considered an essential step
in the development of a PRO by the FDA [14], although it
has been pointed out that it is still not a method that is
consistently applied for all self-report instruments [22].
When performing pediatric cognitive debriefing, the process is much the same as in adults; however, as with the
steps described in the previous sections, researchers must
adjust their expectations.
Cognitive debriefing can be repetitive and young children in particular will likely lose focus after debriefing as
few as 1015 items. Also of relevance here is the literature on processing speed. De Leeuw suggests that even a
child aged 12 years takes approximately 1.5 times longer
than an adult to process information; thus, children must
be given more time to consider their response during
debriefing [24]. Likewise, the level of detail that young
children can provide in relation to each item is more
limited. Children may rely on response sets or provide
inaccurate answers if their attention span is exceeded [18].
In adult cognitive debriefing, a single item can at times be
discussed for 510 minutes (with questions about the
relevance of the item content, the subjects understanding
of the item, discussion of why the subject chose a particular response, and what would need to change for them
to choose a different response). With children aged 811
years and adolescents, the participant will generally only
be able to answer two or three questions about each item
before becoming bored. For children aged 68 years, even
less feedback is possible, and these children often only
give yes/no answers and are generally unable to explain
further, making it difficult to be completely confident that
they have understood correctly.
As in concept elicitation, taking regular breaks during
the cognitive debriefing is recommended, as is varying
what is being asked of the child to keep their interest. For
questionnaires longer than approximately 30 items it is
recommended that researchers only debrief half of the
questionnaire with any one child, or interview the same
child on two occasions.

154

R. Arbuckle, L. Abetz-Webb

Another issue is whether the cognitive demands associated with completing the cognitive debriefing interview
are too great for younger children. It has been suggested
that asking children to put questions in their own words is
too cognitively demanding for younger children [22].
Asking instead what does the question mean?, what is it
asking? has been suggested as being better understood
[22]. The researcher should be ready to try two or three
ways of getting at the childs ability to comprehend, as the
first approach used may not be successful.
A key issue that deserves greater attention with children
and adolescents compared with adults is their ability to
recall accurately. Unsurprisingly, there is evidence that
children aged 611 years recall symptoms and HRQL
impact more accurately if shorter recall periods are used
(e.g. 24 h) rather than longer recall periods (e.g. 7 days,
4 weeks) [7375]. From our own work, we found that
while children as young as 9 years of age and their parents
had confidence in their ability to recall over the previous
24 h, they were not confident in their ability to recall over a
week, and certainly not over 4 weeks. Thus, it is very
important when conducting and analysing cognitive
debriefing interviews with children to pay a great deal of
attention to the time period children seem to be recalling
over and whether they are having difficulty recalling. Sleep
problems are one area where children generally report
having a great deal of difficulty recalling accurately [20].

problems, but they struggle with providing verbal justification of their reasoning. The implications for interviewing
children about symptoms or functioning is that while
children as young as 3 or 4 years of age may be able to
report on their current health state, they may not be able to
explain their reasoning in an interview
There is evidence that when young children are asked
about events of personal significance to them (such as
whether or not they have been given an injection), they
usually provide correct answers [81]. However, young
children have also been shown to have trouble locating
events in time, or giving specific examples of recurrent
eventstwo issues of particular concern if asking young
children to report on frequent symptoms [82, 83]. In one
experiment, pre-schoolers were asked about what happens
when they eat lunch. The researcher found that if she asked
for information the children did not have, they would
provide answers from other similar scripted events [82].
All of these findings highlight that the researcher must
be very careful when conducting cognitive debriefing
interviews with children aged 611 years. For cognitive
debriefing, it is important that interviewers make it as clear
as possible that it is okay (and moreover helpful!) if the
child admits when he/she cannot understand. One useful
instruction in this context is to say at the beginning we
need your help to make these questions better by making
them easier for younger kids [84].

3.3.1 Questionnaires Violating Conversation Norms Might

Confuse Young Children

4 PRO Completion

There is evidence that young children first learn basic

conversation conventions, and only later learn how to deal
with exceptions where these conventions are violated [76].
If asked the same question more than once, young children
may change their answer because they assume that something was wrong with the first answer [77]. Therefore,
repeated questioning should be used sparingly with young
children, and interviewers should take care to ask questions
as neutrally as possible. In a similar vein, hypothetical
questioning can be particularly difficult for young children.
For daily diary instruments, one method that can help
researchers to avoid hypothetical questioning is to have the
child complete the diary for several days before the
debriefing interview. Then debriefing questions can focus
on their real experience with the diary and avoid hypothetical how would you answer if questions.
The work of Piaget [78] suggested that children aged
35 years have difficulty understanding cause and effect.
However, subsequent research demonstrated that if the task
is made very simple, young childrens reasoning far
exceeds the level Piaget thought possible [51, 79, 80]. In
fact, the children are able to reason logically on simple

4.1 Parent vs. Child PRO Completion

So who should rate the symptoms, functioning, HRQL, and
other concepts typically measured using PROs (e.g. satisfaction, adherence)? For children\3 years of age it is clear
that the children themselves cannot provide even an
interviewer-administered PRO rating, and either a parent/
caregiver- or clinician-completed measure is essential.
Ratings have been obtained on occasion from children as
young as 35 years. Obtaining a child rating for this age
group should only be considered for the most concrete and
the simplest of concepts, and even then, an interviewer- or
parent/caregiver-administered PRO is necessary as children
this age would not be able to complete the instrument
themselves
For the 6- to 11-year range is it recommended that either
complementary child and parent ratings are obtained or that
the child report is parent- or interview-administered, so that
the parent can help ensure the child can read and understand the question as intended. This is particularly important for the 6- to 8-year age range (given that this is the age
at which most children move from Piagets pre-causal

Not Just Little Adults

reasoning to causal reasoning), but is also recommended

for children aged 911 years. Where parent administration
of a child-completed PRO is employed (in any age group)
it is very important that the parent is given specific and
clear instructions on how he/she can and cannot help the
child. This should include clear guidance not to influence
the childs response. Rather, the main function of the
parent should be to help the child read and understand the
question. In the 6- to 8-year age range, generally neither the
child, the parent, nor the clinician can be considered the
gold standard but rather each reporter provides different
complementary information, which provides a different
piece of the puzzle.
As a result, including complementary parent and child
measures creates particular challenges when it comes to
designing clinical trials in which one (or a composite
including ratings from more than one reporter) endpoint
must be chosen as a primary endpoint and considerations
must be made regarding multiplicity. Decisions regarding
the hierarchy of endpoints should be made taking into
account qualitative research findings from clinician, parent,
and child interviews and, if available, the evidence of the
psychometric validity of the different measures/versions.
Ideally these decisions should also be reached following
discussions with regulators.
Which rating (child, parent/caregiver, parent-administered child measure, clinician, etc.) is considered primary
will vary depending on the concept being measured
(whether rating is of observable behaviors, observable
functional issues or non-observable symptoms, emotions,
or cognitions), the condition being studied, and, most
importantly, the age and ability of the child. As Matza et al.
[18] point out, for the 6- to 8-year age group, while the
child rating has more validity, the parent/caregiver rating
might have more reliability. As a result of these difficulties,
a parent-administered child report can prove the best
option.
In studies that include children across the range of ages
from 317 years, it is possible there will be the situation
that a parent report is used for children aged 35 years
(with no child rating at all for the 3- to 5-year age group),
and a child report is the primary endpoint for the children
aged 916 years, with a parent-administered version for
ages 68 years. However, inclusion of so many versions
has implications for pooling of data and sample sizes.
Data can only be pooled for scores created for which
there is the same number of items in each age group, and
the items can be considered conceptually equivalent. Often
with different age versions and separate child and parent
versions of PRO measures this is not the case, and so
pooling of data is not possible. Different age versions can
be considered analogous to different language versions of a
PRO [85]the items need not be worded identically, but it

155

is important to show conceptual equivalence both from a

qualitative and quantitative perspective. If data cannot be
pooled it has the implication that sample sizes within each
age group must be sufficient to allow statistical differences
to be demonstrated within that age group.
4.2 Age Ranges Used in Existing Pediatric PROs
A summary of the different age-specific versions of some
of the most widely used pediatric HRQL instruments has
been provided by Matza et al. [18]. The majority of
instruments suggest a minimum age of around 6 years for
self-report, with a few allowing reporting for simple
questions to be used in children as young as 4 years of age
[10, 18, 38]. Moreover, most have separate versions for
adolescents and younger or middle-aged children (i.e. those
aged approximately 611 years). Among generic HRQL
measures, the Child Health Questionnaire has a parent
report version for children aged 417 years, and a child
report version for ages 1017 years only [86]. The Child
Health and Illness Profile (CHIP) [87], which later led to
the development of the Healthy Pathways measure, [1] has
child versions for ages 611 years and 1217 years [88],
and a parent version for ages 611 years only [89]. There
are two widely used generic European HRQL measures
the KINDL [90] and the KIDSCREEN [91]. The KINDL
[90] is unusual in having a simplified, interviewer-administered version for children as young as 47 years of age, as
well as versions for 811 years and 1217 years. The
KIDSCREEN includes a version for self-report by children
and adolescents aged 818 years, as well as a parent version. Lastly, the Pediatric Quality of Life Inventory
(PedsQL) [38]the most widely used generic measure
has child self-report versions for ages 57, 812, and 1317
years, and a version for young adults aged 1825 years.
4.3 Differences Between Child Self-Report and Parent
Report
Findings vary regarding the level of agreement between
child and parent ratings of health. Some researchers have
found high levels of agreement between parent and child
ratings [18, 92, 93], while others have much lower levels of
agreement [9496]. One recent study found very low levels
of agreement between 5- and 8-year-olds and parents, with
intraclass correlation coefficients of just 0.020.23 [96]. In
a further recent study in asthma and epilepsy presented at
the International Society of Quality of Life Research [97] it
was reported that the low level of agreement between
parent and child decreased with the severity of the condition. Parents in this study rated their childs HRQL as
worse than the child did (and also worse than the clinician
did) and the level of correlation was lower for children of

156

R. Arbuckle, L. Abetz-Webb

worse severity. This pattern has also been found in studies

of other pediatric chronic conditions such as chronic pain
[98, 99]. However, in studies comparing the responses of
healthy children and their parents, the opposite has been
found, with parents/caregivers rating their childs health as
lower than the healthy child/adolescent [92, 96, 100, 101].
Finally, two studies provide evidence that the level of
agreement varies with age, with closer agreement for older
children/adolescents [102, 103].

ePROs provide additional options to aid comprehension for

young children. Despite improvements in our understanding, too often pediatric studies still rely on instruments that
are inappropriate for the population, poorly developed, or
inadequately measure the concept of interest. Improvements in the art and the science of pediatric PRO research
are needed to help ensure that instruments used in pediatric
studies are truly fit for purpose, thus aiding in the accumulation of robust evidence regarding the health of
children.

5 Ethical Considerations in Relation to Pediatric PRO

Research

Acknowledgments We would like to acknowledge the support

provided by Kate Bolton in preparing one draft of the manuscript, and
the support of Nicola Moss in helping with formatting and quality
checking. In addition, we are grateful for the helpful comments of two
anonymous reviewers.

Broadly, pediatric PRO development research should be

conducted within the ethical guidelines that exist for
medical research. Clearly, ethical issues are of even greater
importance for a vulnerable population such as children.
All research should be conducted in line with the Declaration of Helsinki, and should meet with national and
regional guidelines for research in the country in which it is
conducted. Written informed consent should be obtained
from a parent/guardian for all child participants in interview or questionnaire studies, and assent should be
obtained from the children themselves [43]. Of note, while
it is important to first obtain consent from the parent/
guardian to ask the child to participate, obtaining the
childs assent to participate is also critical. To assent, the
child/adolescent needs to have the aims of the research and
what will be expected of them explained in simple,
developmentally appropriate language. This should be
through verbal explanation, but also through an assent form
that uses an appropriate level of language. It should also be
made very clear to children that they are able to withdraw
at any time, for any reason, with no adverse consequences
for them or their medical care.

6 Conclusions
The aim of this article was to provide an overview of the
state of the art for qualitative pediatric PRO development
research. It is most critical that PRO development methods
used in adults are carefully tailored to the age and developmental abilities of the children under study. Conceptual
saturation and comprehension should be evaluated within
narrow age ranges. A particular challenge is the issue of
how to evaluate children who are too young to report
themselves, but who experience symptoms that cannot be
adequately measured by any means other than asking the
child. Both parent reports that rely on observations and
parent-administered child self-reports are two options that
to some degree overcome these challenges. Moreover,

Disclosure of interests Both Rob Arbuckle and Linda Abetz-Webb

are employees of Adelphi Values, a health outcomes consultancy that
specialises in working with healthcare companies on the development, validation, and use of PRO instruments. As such, both authors
have been contracted to perform research for numerous pharmaceutical companies. Neither author owns stocks in any pharmaceutical
company, nor have they been a direct employee of a pharmaceutical
company.
Author contributions Both authors contributed to the conception
and writing of all parts of the manuscript and both authors read and
approved the final version.

References
1. Bevans KB, Riley AW, Moon J, Forrest CB. Conceptual and
methodological advances in child-reported outcomes measurement. Expert Rev Pharmacoecon Outcomes Res. 2010;10(4):
38596.
2. Forrest CB, Simpson L, Clancy C. Child health services
research: challenges and opportunities. JAMA. 1997;277(22):
178793.
3. Maldonado S. United States perspective. In: Mulberg A, Silber
S, van den Anker J, editors. Pediatric drug development: concepts and applications. New Jersey: Wiley; 2009. p. 1336.
4. The Best Pharmaceuticals for Children Act of 2002. Public Law
107-109, 115 Stat.1408-1424. 2002. http://www.fda.gov/
downloads/Drugs/DevelopmentApprovalProcess/Development
Resources/UCM049874.pdf.
5. The Pedatric Research Equity Act of 2003. Public Law 108-155,
117 Stat.1936-1943. 2003. http://www.fda.gov/downloads/Drugs/
DevelopmentApprovalProcess/DevelopmentResources/UCM077
853.pdf.
6. The Pediatric Research Equity Act of 2007. Public Law 110-85,
121 STAT.823 (Food and Drug Administration Amendments
Act (FDAAA) of 2007Title IV). 2007. http://www.fda.gov/
downloads/Drugs/DevelopmentApprovalProcess/Development
Resources/UCM049870.pdf.
7. The Best Pharmaceuticals for Children Act of 2007. Public Law
110-85, 121 STAT.823 (Food and Drug Administration
Amendments Act (FDAA) of 2007Title V). 22007. http://
www.fda.gov/downloads/Drugs/DevelopmentApprovalProcess/
DevelopmentResources/UCM049870.pdf.

Not Just Little Adults

8. Regulation (EC) no 1901/2006 of the european parliament and
of the council of 12 December 2006 on medicinal products for
paediatric use and amending Regulation (EEC) No 1768/92,
Directive2001/20/EC, Directive 2001/83/EC and Regulation
(EC) No 726/2004. Official Journal of the European Union.
2006. http://ec.europa.eu/health/files/eudralex/vol-1/reg_2006_
1901/reg_2006_1901_en.pdf.
9. Regulation (EC) no 1902/2006 of the european parliament and
of the council of 20 December 2006 amending Regulation
1901/2006 on medicinal products for paediatric use. Official
Journal of European Union. 2006. http://ec.europa.eu/health/
files/eudralex/vol-1/reg_2006_1902/reg_2006_1902_en.pdf.
10. Landgraf J, Abetz L. Measuring health outcomes in pediatric
populations: issue in psychometrics and application. In: Spilker
B, editor. Quality of life and pharmacoeconomics in clinical
trials. Philadelphia: Lippincott-Raven Publishers; 1996.
p. 793802.
11. Pasquali SK, Burstein DS, Benjamin DK, Smith PB, Li JS.
Globalization of pediatric research: analysis of clinical trials
completed for pediatric exclusivity. Pediatrics. 2010;126(3):
e68792.
12. Pharmaceutical Research and Manufacturers of Americas
(PhRMA). PhRMA Statement on Tremendous Success of BPCA
and PREA. 2011. http://www.phrma.org/media/releases/phrmastatement-tremendous-success-bpca-prea.
13. European Medicines Agency. Reflection Paper on the Regulatory Guidance for the Use of Health-Related Quality of Life
(HRQL) Measures in the Evaluation of Medicinal Products.
2005. http://www.ema.europa.eu/docs/en_GB/document_library/
Scientific_guideline/2009/09/WC500003637.pdf.
14. US Department of Health and Human Services, Food and Drug
Administration, Center for Drug Evaluation and Research
(CDER), Center for Biologics Evaluation and Research (CBER),
Center for Devices and Radiological Health (CDRH). Guidance
for IndustryPatient-Reported Outcome Measures: Use in
Medical Product Development to support Labeling Claims. 2009.
http://www.fda.gov/downloads/Drugs/GuidanceCompliance
RegulatoryInformation/Guidances/UCM193282.pdf.
15. Leidy NK, Vernon M. Perspectives on patient-reported outcomes: content validity and qualitative research in a changing
clinical trial environment. Pharmacoeconomics. 2008;26(5):
36370.
16. Patrick DL, Burke LB, Powers JH, Scott JA, Rock EP, Dawisha
S, et al. Patient-reported outcomes to support medical product
labeling claims: FDA perspective. Value Health. 2007;10(Suppl
2):S12537.
17. Lasch KE, Marquis P, Vigneux M, Abetz L, Arnould B, Bayliss
M, et al. PRO development: rigorous qualitative research as the
crucial foundation. Qual Life Res. 2010;19(8):108796.
18. Matza L, Swensen A, Flood E, Secnik K, Kline Leidy N.
Assessment of health-related quality of life in children: a review
of conceptual, methodological, and regulatory issues. Value
Health. 2004;7(Number 1):7990.
19. Rothman M, Kleinman L. Patient-reported outcomes in pediatric
drug development. In: Mulberg A, Silber S, van den Anker J,
editors. Pediatric drug development: concepts and applications.
New Jersey: Wiley; 2009. p. 51324.
20. Arbuckle R, Abetz L, Durmer J, Ivenenko A, Owens J, Croenlin
J, et al. Development of the Pediatric Restless Legs Syndrome
Severity Scale (R-RLS-SS): a patient reported outcome measure
of pediatric RLS symptoms and impact. Sleep Med. 2010;
11(9):897906.
21. Solans M, Pane S, Estrada MD, Serra-Sutton V, Berra S,
Herdman M, et al. Health-related quality of life measurement in
children and adolescents: a systematic review of generic and
disease-specific instruments. Value Health. 2008;11(4):74264.

157
22. Woolley ME, Bowen GL, Bowen NK. Cognitive pretesting and
the developmental validity of child self-report instruments:
theory and applications. Res Soc Work Pract. 2004;14(3):
191200.
23. Fuchs M. The reliability of childrens survey responses: the
impact of cognitive functioning on respondent behavior. In:
Proceedings of Statistics Canada Symposium 2008: Data Collection: Challenges, Achievements and New Directions; 2009.
24. de Leeuw ED. Improving data quality when surveying children
and adolescents: cognitive and social development and its role in
questionnaire construction and pretesting. Report prepared for the
Annual Meeting of the Academy of Finland: Research Programs
Public Health Challenges and Health and Welfare of Children and
Young People, May 10-12. Naantali, Finland; 2011.
25. Ravens-Sieberer U, Erhart M, Wille N, Wetzel R, Nickel J,
Bullinger M. Generic health-related quality-of-life assessment in
children and adolescents: methodological considerations. Pharmacoeconomics. 2006;24(12):1199220.
26. Inhelder B, Piaget J. The growth of logical thinking from
childhood to adolescence. London: Routledge; 1958.
27. Inhelder B, Piaget J. The early growth of logic in the child.
London: Routledge; 1964.
28. Vygotsky LS. Mind in society: the development of higher psychological processes. Cambridge: Harvard University Press; 1978.
29. Bronfenbrenner U. The ecology of human development:
experiments by nature and design. Cambridge: Harvard University Press; 1981.
30. Kail RE. Children and their development. 5th ed. Englewood
Cliffs: Prentice Hall; 2006.
31. Borgers N, de Leeuw E, Hox J. Children as respondents in
survey research: cognitive development and response quality.
Bulletin de Methodologie Sociologique. 2000;66:6075.
32. Siegler RS, Richards DD. The development of intelligence. In:
Sternberg RJ, editor. Handbook of human intelligence. Cambridge: Cambridge University Press; 1982. p. 897971.
33. Scarr S. Understanding development. New York: Harcourt
Publications; 1986.
34. Sodian B, Kristen S. Theory of mind. Towards a theory of
thinking. Berlin: Springer; 2010. p. 189201.
35. Perner J. Understanding the representational mind. Cambridge:
MIT Press; 1991.
36. White SH. Evidence for a hierarchical arrangement of learning
processes. In: Lewis PL, editor. Advances in child development
and behavior, vol 2. JAI; 1965. p. 187220.
37. White SH. The childs entry into the age of reason. In: Sameroff AJ, Haith MM, editors. The five to seven year shift: the
age of reason and responsibility. Chicago: University of Chicago
Press; 1996. p. 1730.
38. Manificat S, Dazord A. Childrens Quality of Life Assessment:
preliminary results obtained with the AUQUEI Questionnaire.
Qual Life Newsletter 1998;23.
39. Ross DM, Ross SA. Childhood pain: the school-aged childs
viewpoint. Pain. 1984;20(2):17991.
40. Punch S. Research with children: the same or different from
research with adults? Childhood. 2002;9(3):32141.
41. Walsh TR, Irwin DE, Meier A, Varni JW, DeWalt DA. The use
of focus groups in the development of the PROMIS pediatrics
item bank. Qual Life Res. 2008;17(5):72535.
42. Ravens-Sieberer U, Gosch A, Rajmil L, Erhart M, Bruil J, Duer
W, et al. The KIDSCREEN-52 Quality of Life measure for
children and adolescents: development and first results from a
European survey. Expert Rev Pharmacoecon Outcomes Res.
2005;5(3):35364.
43. Heary CM, Hennessy E. The use of focus group interviews in
pediatric health care research. J Pediatr Psychol. 2002;27(1):
4757.

158
44. Lewis A. Group child interviews as a research tool. Br Edu Res
J. 1992;18(4):41321.
45. Levine IS, Zimmerman JD. Using qualitative data to inform
public policy: evaluating choose to de-fuse. Am J Orthopsychiatry. 1996;66(3):36377.
46. McColl E. Developing Questionnaires. Assessing Quality of
Life in Clinical Trials. New York: Oxford University Press;
2013. p. 923.
47. Tajfel H, Turner JC. An integrative theory of intergroup conflict.
In: Worchel S, Austin W, editors. The Social Psychology of
Intergroup Relations. Belmont: Brooks/Cole; 1979. p. 3348.
48. Tajfel H, Turner JC. The social identity theory of intergroup
behavior Psychology of Intergroup Relations. Chicago: NelsonHall; 1986. p. 724.
49. van Hattum MJC, de Leeuw ED. A disk by mail survey of pupils
in primary schools; data quality and logistics. J Off Stat.
1999;15:41330.
50. Hoppe MJ, Wells EA, Wilsdon A, Gillmore MR, Morrison DM.
Childrens knowledge and beliefs about AIDS: qualitative data
from focus group interviews. Health Educ Q. 1994;21(1):
11726.
51. Donaldson MC. Children;s minds. London: Fontana Press; 1978.
52. Vaughn SS, Sinagub J. Focus group interviews in education and
psychology. London: Sage; 1996.
53. Eiser C, Morse R. Can parents rate their childs health-related
quality of life? Results of a systematic review. Qual Life Res.
2001;10(4):34757.
54. Petrou S. Methodological issues raised by preference-based
approaches to measuring the health status of children. Health
Econ. 2003;12(8):697702.
55. Pal DK. Quality of life assessment in children: a review of
conceptual and methodological issues in multidimensional
health status measures. J Epidemiol Community Health.
1996;50(4):3916.
56. US FDA, CDER, CBER, CDRH. Guidance for industry: patientreported outcome measures: use in medical product development to support labelling claims. FDA 2009. http://www.fda.
gov/downloads/Drugs/GuidanceComplianceRegulatoryInformat
ion/Guidances/UCM193282.pdf.
57. Glaser B, Strauss A. The discovery of grounded theory: strategies for qualitative research. Chicago: Aldine; 1967.
58. Bowen G. Grounded theory and sensitizing concepts. Int J Qual
Methods. 2006;5(No 3):19.
59. Picchietti DL, Arbuckle RA, Abetz L, Durmer JS, Ivanenko A,
Owens JA, et al. Pediatric restless legs syndrome: analysis of
symptom descriptions and drawings. J Child Neurol. 2011;
26(11):136576.
60. Abetz LN, Arbuckle RA, Carson RT, Lewis BE, Hunter LJ,
Shiff SJ, et al. Development of a pediatric chronic constipation
symptom measure: results of qualitative interviews with children
and their parents. 2010. In: 17th Annual Conference of the
International Society for Quality of Life Research October
2730 2010.
61. Carson R, Arbuckle R, Abetz L, Lewis B, Hunter L, Johnston J,
et al. The symptoms of irritable bowel syndrome with constipation (IBS-C): results of qualitative interviews with children
and their parents. 27-10-2010. In: ISOQOL 17th Annual Conference, London, UK, 2730 October 2010.
62. van Laerhoven H, van der Zaag-Loonen H, Derkx BHF. A
comparison of Likert scale and visual analogue scales as
response options in childrens questionnaires. Acta Paediatr.
2004;93(6):8305.
63. Shields BJ, Palermo TM, Powers JD, Grewe SD, Smith GA.
Predictors of a childs ability to use a visual analogue scale.
Child Care Health Dev. 2003;29(4):28190.

R. Arbuckle, L. Abetz-Webb
64. Rebok G, Riley A, Forrest C, Starfield B, Green B, Robertson J,
et al. Elementary school-aged childrens reports of their health:
a cognitive interviewing study. Qual Life Res.
2001;10(1):5970.
65. Juniper EF. Quality of life in adults and children with asthma
and rhinitis. Allergy. 1997;52(10):9717.
66. Wong D, Wilson D, Whaley L. Whaley and wongs nursing care
of infants and children. St Louis: Mosby; 1995.
67. Wong DL, Baker CM. Pain in children: comparison of assessment scales. Pediatr Nurs. 1988;14(1):917.
68. Hockenberry MJ, Wilson D, Winkelstein ML. Wongs essential
of pediatric nursing. 7th ed. St. Louis: Mosby; 2005.
69. Cremeens J, Eiser C, Blades M. Brief report: assessing the
impact of rating scale type, types of items, and age on the
measurement of school-age childrens self-reported quality of
life. J Pediatr Psychol. 2007;32(2):1328.
70. Willis GB. Cognitive interviewing and questionnaire design: a
training manual. Cognitive methods staff working paper series,
no. 7. Office of Research and Methodology, National Center for
Health Statistics. 1994. Hyattsville (MD).
71. Farrar MJ, Goodman GS. Developmental changes in event
memory. Child Dev. 1992;63(1):17387.
72. Brainerd C, Ornstein PA. Childrens memory for witnessed
events: the developmental backdrop. In: Doris J, editor. The
suggestibility of childrens recollections: implications for eyewitness testimony. Washington, DC: American Psychological
Association; 1991. p. 1020.
73. Feikin DR, Audi A, Olack B, Bigogo GM, Polyak C, Burke H,
et al. Evaluation of the optimal recall period for disease symptoms in home-based morbidity surveillance in rural and urban
Kenya. Int J Epidemiol. 2010;39(2):4508.
74. Gill LJ, Shand PAX, Fuggle P, Dugan B, Davies SC. Pain
assessment for children with sickle cell disease: Improved
validity of diary keeping versus interview ratings. Br J Health
Psychol. 1997;2(2):13140.
75. van der Plas RN, Benninga MA, Redekop WK, Taminiau JA,
Buller HA. How accurate is the recall of bowel habits in children
with defaecation disorders? Eur J Pediatr. 1997;156(3):17881.
76. Scarr S, Weinberg R, Levine A. The intellectual revolution of
middle childhood. Underst Dev. 1986. p. 343-81.
77. Siegal M. Knowing children: experiments in conversation and
cognition. Hove: Lawrence Erlbaum Associates; 1991.
78. Piaget J. The childs conception of physical causality. Boston:
Adams & Co; 1960.
79. Bullock M, Gelman R. Preschool childrens assumptions about
cause and effect: temporal ordering. Child Dev. 1979;50:8996.
80. Bullock M. Preschool childrens understanding of causal connections. Br J Dev Psychol. 1984;2(2):13948.
81. Goodman G, Rudy L, Bottoms BL, Aman C. Childrens concerns and memory: issues of ecological validity in the study of
childrens eyewitness testimony. In: Fivush R, Hudson JA,
editors. Knowing and remembering in young children. London:
Cambridge University Press; 1990. p. 24984.
82. Nelson K. How children represent knowledge of their world in
and out of language. Childrens thinking: What develops?. New
Jersey: Lawrence Erlbaum Associates Inc; 1978. p. 22573.
83. Nelson K. Event knowledge: structure and function in development. London: Psychology Press; 1986.
84. Willis G. Cognitive interviewinga tool for improving questionnaire design. New York: Sage Publications; 2005.
85. Herdman M, Fox-Rushby J, Badia X. A model of equivalence in
the cultural adaptation of HRQoL instruments: the universalist
approach. Qual Life Res. 1998;7(4):32335.
86. Landgraf JM, Abetz LN, DeNardo BA, Tucker LB. Clinical
validity of the Child Health Questionnaire-Parent Form in

Not Just Little Adults

87.

88.

89.

90.

91.

92.

93.

94.

95.

children with rheumatoid arthritis. In: Poster presented at the

1995 National Scientific Meeting of the American College of
Rheumatology, San Francisco, CA, October 21, 1995. 21-101995.
Starfield B, Bergner M, Ensminger M, Riley A, Ryan S, Green
B, et al. Adolescent health status measurement: development of
the Child Health and Illness Profile. Pediatrics. 1993;91(2):
4305.
Riley AW, Forrest CB, Rebok GW, Starfield B, Green BF,
Robertson JA, et al. The Child Report Form of the CHIP-Child
Edition: reliability and validity. Med Care. 2004;42(3):22131.
Riley AW, Forrest CB, Starfield B, Rebok GW, Robertson JA,
Green BF. The Parent Report Form of the CHIP-Child Edition:
reliability and validity. Med Care. 2004;42(3):21020.
Ravens-Sieberer U. The revised KINDL-R: Final results on
reliability, validity and responsiveness of a modular HRQOL
instrument for children and adolescents. 8. Jahrestagung der
International Society for Quality of Life Research (ISOQOL).
Qual Life Res. 2001;3:199.
Ravens-Sieberer U, Gosch A, Abel T, Auquier P, Bellach BM,
Bruil J, et al. Quality of life in children and adolescents: a
European public health perspective. Soz Praventivmed.
2001;46(5):294302.
Theunissen NC, Vogel T, Koopman HM, Verrips GH, Zwinderman KA, Verloove-Vanhorick SP, et al. The proxy problem:
child report versus parent report in health related quality of life
research. Qual Life Res. 1998;7:38797.
Verrips GHW, Vogels AGC, Ouden AL, Paneth N, VerlooveVanhorick SP. Measuring health-related quality of life in adolescents: agreement between raters and between methods of
administration. Child Care Health Dev. 2000;26(6):45769.
Ennett ST, DeVellis BM, Anne Earp J, Kredich D, Warren RW,
Wilhelm CL. Disease experience and psychosocial adjustment
in children with juvenile rheumatoid arthritis: childrens versus
mothers reports. J Pediatr Psychol. 1991;16(5):55768.
Vogel T, Verrips GH, Verloove-Vanhorick SP, Fekkes M,
Kamphuis RP, Koopman HM, et al. Measuring health related
quality of life in children: the development of the TACQOL
parent form. Qual Life Res. 1998;7:45765.

159
96. Cremeens J, Eiser C, Blades M. Factors influencing agreement
between child self-report and parent proxy-reports on the
Pediatric Quality of Life Inventory TM 4.0 (PedsQLTM) generic core scales. Health Qual Life Outcomes. 2006;4(1):58.
97. Carona CP, Morais TF, Leitao S, Marques D, Silva N, Nazare B,
et al. Assessing health-related quality of life in children and
adolescents with asthma and epilepsy: results from the pilot
validation study of the Portuguese versions of DISABKIDS-37.
In: 16th Annual Conference of the International Society for
Quality of Life Research (ISOQOL) [New Orleans, Louisiana,
USA]. 31-10-2009.
98. Vetter TR, Bridgewater CL, McGwin G. An observational study
of patient versus parental perceptions of health-related quality of
life in children and adolescents with a chronic pain condition:
who should the clinician believe? Health Qual Life Outcomes.
2012;10(1):85.
99. Youssef NN, Murphy TG, Langseder AL, Rosh JR. Quality of
life for children with functional abdominal pain: a comparison
study of patients and parents perceptions. Pediatrics. 2006;
117(1):549.
100. Johnson SB, Wang C. Why do adolescents say they are less
healthy than their parents think they are? The importance of
mental health varies by social class in a nationally representative
sample. Pediatrics. 2008;121(2):e30713.
101. Waters E, Stewart-Brown S, Fitzpatrick R. Agreement between
adolescent self-report and parent reports of health and wellbeing: results of an epidemiological study. Child Care Health
Dev. 2003;29(6):5019.
102. Annett RD, Bender BG, DuHamel TR, Lapidus J. Factors
influencing parent reports on quality of life for children with
asthma. J Asthma. 2003;40(5):57787.
103. Ronen G-M, Streiner D-L, Rosenbaum P. Canadian pediatric
EN. Health-related quality of life in children with epilepsy:
development and validation of self-report and parent proxy
measures. Epilepsia. 2003;44(4):598612.
104. Atherton JS. Learning and teaching; Piagets developmental theory.
2010. http://www.learningandteaching.info/learning/piaget.htm.

Reproduced with permission of the copyright owner. Further reproduction prohibited without
permission.