Assessment http://asm.sagepub.

com/

Multimethod, Multi-informant Agreement, and Positive Predictive Value in the Identification of Child Anxiety Disorders Using the SCAS and ADIS-C
Amy M. Brown-Jacobsen, Dustin P. Wallace and Stephen P. H. Whiteside Assessment 2011 18: 382 originally published online 19 July 2010 DOI: 10.1177/1073191110375792 The online version of this article can be found at: http://asm.sagepub.com/content/18/3/382

Published by:
http://www.sagepublications.com

Additional services and information for Assessment can be found at: Email Alerts: http://asm.sagepub.com/cgi/alerts Subscriptions: http://asm.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations: http://asm.sagepub.com/content/18/3/382.refs.html

>> Version of Record - Aug 8, 2011 OnlineFirst Version of Record - Jul 19, 2010 What is This?

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

Multimethod, Multi-informant Agreement, and Positive Predictive Value in the Identification of Child Anxiety Disorders Using the SCAS and ADIS-C
Amy M. Brown-Jacobsen1, Dustin P. Wallace2, and Stephen P. H. Whiteside2

Assessment 18(3) 382392 The Author(s) 2011 Reprints and permission: http://www. sagepub.com/journalsPermissions.nav DOI: 10.1177/1073191110375792 http://asm.sagepub.com

Abstract The current study sought to provide practical information for the clinical use of child and parent reports of child anxiety symptoms by investigating agreement between parent, child, and clinician as well as the predictive value of this information. Examining 88 anxious children and their parents, the study compared agreement by correlating parent and child responses on the Spence Childrens Anxiety Scale (SCAS); comparing parent and child responses to clinician data operationalized through continuous and dichotomous variables from the Anxiety Disorder Interview Schedule for Children; and examining the relative clinical utility of parent and child reports on the SCAS in terms of positive and negative predictive value. Results indicated that parent and child agreement on the SCAS was moderate to high for most anxiety disorder symptoms and that both were generally consistent with clinician impressions. Moreover, both child and parent provided unique information to the diagnostic process. Keywords multi-informant agreement, child anxiety disorders, positive predictive value, diagnosis, self-report measures In assessing and diagnosing childhood psychiatric disorders, researchers and clinicians alike highlight the importance of gathering data using a multimodal, multi-informant approach (Comer & Kendall, 2004; Grills & Ollendick, 2003; Jensen et al., 1999). This approach is believed to increase the probability of an accurate diagnosis and in turn assist clinicians in identifying an appropriate treatment plan. In particular, obtaining information from different individuals (e.g., parents, children, and teachers) and by means of a variety of assessment techniques (e.g., self-report questionnaires, structured interviews, behavioral observations) is believed to lead to an enhanced understanding of the child and a more valid diagnostic impression. Although the multimodal, multi-informant approach appears optimal from a theoretical standpoint, its practicality is complicated by two factors: informant disagreement and a lack of information regarding the utility of questionnaire data in predicting diagnosis in a particular individual. particularly on structured diagnostic interviews (Grills & Ollendick, 2003; Jensen et al., 1999). Grills and Ollendick (2003) examined concordance of parent, child, and clinician agreement on the Anxiety Disorders Interview Schedule Child and Parent versions (ADIS-C/P) and found poor levels of agreement among parents and children across all diagnostic categories. Other studies suggest that informant agreement for separation anxiety disorder (SAD), generalized anxiety disorder (GAD), and social phobia (SoP) may range between no better than chance and extremely poor (Foley et al., 2004, 2005; Safford, Kendall, FlannerySchroeder, Webb, & Sommer, 2005). With disagreements being common place, it is difficult to determine which source of information should be considered primary when diagnosing a child with an anxiety disorder. Parent report is typically considered more accurate
1 2

University of Kansas Medical Center, Kansas City Mayo Clinic, Rochester, MN

Informant Disagreement
Informant agreement (e.g., child, mother, father) regarding child anxiety symptoms has been found to be meager,

Corresponding Author: Stephen P. H. Whiteside, PhD, Mayo Clinic, Mayo W-11, 200 First St. SW, Rochester, MN 55905 Email: whiteside.stephen@mayo.edu

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

Brown-Jacobsen et al. for externalizing behaviors (Achenbach, 1999), whereas children may provide more accurate descriptions of internally experienced anxiety symptoms as parents are restricted to reporting on observable signs of the childs anxiety. This view is supported by research finding that children are able to provide reliable information about internalizing symptoms as early as first grade (Ialongo, Edelsohn, WerthamerLarsson, Crockett, & Kellam, 1993). In contrast, parent report is favored by observations that parentchild agreement increases with child age (presumably as the reliability of child report improves) and that parent report tends to be more consistent with clinician impressions (Edelbrock, Costello, Dulcan, Conover, & Kala, 1986; Grills & Ollendick, 2003). One limitation of these previous studies is that they have focused on comparing diagnostic decisions based on child versus parent interviews (Foley et al., 2004, 2005; Safford et al., 2005); with some studies also including a consensus diagnosis (Grills & Ollendick, 2003; Rapee, Barrett, Dadds, & Evans, 1994). By dichotomizing each variable (i.e., presence or absence of the diagnosis per informant), these studies increase the likelihood of finding disagreements. Specifically, a diagnosis is basically a single-item test, which intrinsically has low reliability. In contrast, measuring anxiety at the symptom level not only has the potential of increasing the reliability by including multiple test items but also reduces the risk of interrater disagreement resulting from two quantitatively similar ratings falling on opposite sides of an arbitrary division. Thus, measuring child and parent reports of symptoms on a continuum could lead to higher levels of agreement and provide a more realistic context for deciding how to combine information from different informers. In fact, in the only study to date which assessed parentchild agreement on a diagnostic interview at the level of child anxiety symptoms, agreement was indeed stronger at the symptom level than at the diagnostic level (Comer & Kendall, 2004). To provide a clearer understanding of informant agreement, further studies are needed to assess and report levels of consistency on a continuous variable, rather than solely based on cutoffs for diagnosis. Moreover, relative agreement with the clinician also deserves further study and could benefit from this same methodology to provide additional insight into the utility of information from each informant. In short, by not dichotomizing continuous data, both clinicians and researchers would better understand the actual agreement between informants as well as informant agreement with the diagnostician.

383 would be expected, when continuous data is collected from anxiety questionnaires, agreement between parents and children tends to be in the medium to high range (e.g., Nauta et al., 2004). However, even if the responses provided by children and parents on report forms tend to agree, it is not clear how this information should be used by the clinician when forming a diagnosis. More specifically, few scales have been examined comprehensively enough to inform the clinician as to what an elevated score means beyond the degree to which the subject is endorsing more or fewer symptoms than average (Silverman & Ollendick, 2005). For example, little is known about the degree of impairment, likelihood of diagnosis, or the daily experience of children who score two standard deviations above the mean on a scale of SoP versus children who score closer to the mean. Unfortunately, most validity studies do not provide clinicians with the information needed to use scales in clinical practice. Specifically, most studies report mean differences between groups of children with and without the diagnosis of interest and potentially the scales sensitivity and specificity for identifying children with that diagnosis (e.g., Nauta et al., 2004; Spence, 1998; Wood, Piacentini, Bergman, McCracken, & Barrios, 2002). Such information is important and suggests that the scale measures the intended construct. In addition, such information allows a clinician to use the scale as a screening device to identify children who may be at risk for having a disorder and could benefit from additional assessment to establish a diagnosis. However, mean differences and sensitivity thresholds provide insufficient information for a clinician attempting to incorporate report forms into the diagnostic process. Although sensitivity provides information on the percentage of children with a diagnosis that will obtain an elevated score, it provides no information about what it means to have an elevated score. To answer this question, the clinician needs to know the percentage of children scoring in the clinical range that have a given diagnosis, that is, the positive predictive value (PPV) of the test. Even a test with strong sensitivity and specificity can have quite low PPV depending on prevalence of the disorder in the population of interest (see Smith, Cerhan, & Ivnik, 2003, for further information and examples). For instance, if a cutoff score is set to maximize sensitivity (e.g., capture 90% of patients), it may also capture many nonpatients and the PPV could be low (e.g., 25%). In this example, the clinician needs to know that although the questionnaire is meaningful, most children (75%) with a positive score do not have the given diagnosis. Thus, to optimize the utility of child- and parent-report measures in the diagnostic process, clinicians need more information than is typically reported. It should be noted that PPVs are not without limitations, as these values are

Diagnostic Utility of Questionnaire Data

Parent- and child-report forms are a convenient method for gathering information regarding anxiety symptoms. As

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

384 affected by the base rate of disorder in the population under study, which certainly varies across settings. Nevertheless, of recent studies investigating the relation between child anxiety rating scales and structured interviews, only one provided any information regarding the meaning of scores above a certain cutoff (Bodden, Bogels, & Muris, 2009). Even then, the authors presented the rate of false positives for the total score but did not explore similar metrics for specific disorders or subscales.

Assessment 18(3) 88.3% of the families. The majority of patients (83%) received a diagnosis on the structured interview of GAD, obsessive compulsive disorder (OCD), SoP, SAD, or specific phobia (SP). Approximately, 48% had more than one diagnosis.

Materials
ADIS-C for DSM-IV. The ADIS-C is a semistructured diagnostic interview of childhood anxiety disorders as well as mood and externalizing disorders (Albano & Silverman, 1996). The ADIS-C has good interviewerobserver reliability (kappa = .75) and testretest reliability (.75). As the ADIS-C was administered as a part of routine clinical practice, the child version was administered to the patient and parent together. Only the anxiety portion of the ADIS-C was included in the present study. The information gathered from the ADIS-C is considered an operationalization of the clinicians impressions in both a dichotomous (diagnosis vs. no diagnosis) and continuous (number of symptoms, described below) manner. SCAS. Anxiety symptoms were examined with the SCAS, a 45-item Likert-type, self-report questionnaire designed to measure anxiety symptoms in children and adolescents (Spence, 1998). The SCAS yields six basic scales: Panic Attacks and Agoraphobia, Separation Anxiety, Physical Injury Fears, Social Phobia, Obsessive-Compulsive, and Generalized Anxiety. Spence (1998) reports internal reliability coefficients ranging from .60 to .82 for the subscales and .92 for the total scale. Six-month testretest reliability ranges from .45 to .57 for the subscales and is .60 for the total scale. Validity data supporting the SCAS has been reported by Spence and others (Muris, Merckelbach, Ollendick, King, & Bogie, 2002; Muris, Schmidt, & Merckelbach, 2000; Spence, 1998). The childrens parents completed the SCAS-P, a parent-report version that also has acceptable psychometric properties (Nauta et al., 2004). The internal consistency coefficients in the current sample were as follows (child/parent): total score (.87/.90), panic (.75/.76), separation (.66/.71), physical injury fears (.47/.62), social (.70/.78), OCD (.76/.87), GAD (.66/.65).

Goals of Current Research

The current investigation sought to address the two obstacles summarized above to implementing the multimethod, multi-informant paradigm in a clinical practice assessing children with anxiety. First, the agreement between parent, child, and clinician impressions of child anxiety symptoms was evaluated by operationalizing the variables in a continuous fashion. In contrast to previous studies which have found low agreement for dichotomized data (diagnosis vs. no diagnosis), it was predicted that the use of continuous data would result in acceptable levels of agreement between the three parties. Second, the utility of questionnaire data in the diagnostic process was examined through logistical regressions and by calculating the PPV and negative predictive value (NPV) of the Spence Childrens Anxiety Scale (SCAS) and its parent version (SCAS-P) for predicting clinician diagnosis operationalized with the ADIS-C. Consistent with the use of other self-report measures for prediction of childhood anxiety symptoms (e.g., Berge, Veerkamp, Hoostraten, & Prins, 2002), it was hypothesized that the PPV of child and parent reports would be low to moderate.

Method Participants
The participants were 88 predominately White children living at home (49 boys, 55.7%) ranging in age from 7 to 18 years (M = 12.32 years, SD = 3.3) and their parent. Data were collected through a retrospective chart review with approval from the institutional review board. Children were referred from a wide variety of sources including self-referral, primary care physicians, and other mental health professionals for evaluation in a Child & Adolescent Anxiety Diagnostic Clinic in a large Midwestern medical center. The assessment consisted of a structured diagnostic interview conducted by a doctoral-level psychologist and questionnaires completed by the child and one parent. Most of the childrens parents were married (87.5%) and had at least a bachelors degree or equivalent (66.7%). Annual income was greater than US$40,000 in

Procedure
Children and their parents completed the SCAS and SCAS-P prior to evaluation as part of an intake packet. The ADIS-C was administered by a psychologist who made a diagnostic decision based on the results prior to reviewing the questionnaires. Thus, the clinicians impressions operationalized by the ADIS-C were independent from the questionnaires completed by the parent and child (Clinically, the results of the questionnaires, ADIS-C, and psychiatric evaluation were combined in a multidisciplinary case

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

Brown-Jacobsen et al. conference to finalize the diagnostic impressions and treatment recommendations). In addition to the dichotomous designation of diagnosis (present or absent), a continuous variable from the ADIS-C was identified for each diagnosis: total number of symptoms of SAD (SADsx), number of social situations avoided (SoPavd), number of phobic objects avoided (SPavd), number of worries that are hard to stop (GADhtp), and number of obsessions and compulsions that are resisted (OCDres). To use the SCAS and SCAS-P diagnostically, dichotomous variables were abstracted from the continuous data provided by these questionnaires. Cutoff scores were set at 1.5 standard deviations above the mean from a community sample of children from the local area (Whiteside & Brown, 2008). A cutoff of 1.5 standard deviations was used based on the common use of the T score equivalent (65) as the beginning of the at-risk range on measures of child emotional and behavioral functioning (e.g., Achenbach & Rescorla, 2001; Briere, 1996; Conners, Sitarenios, Parker, & Epstein, 1998). Consistent with the literature (e.g., Nauta et al., 2004), cut scores were calculated separately for each of four groups: boys 12 and older, girls 12 and older, boys 11 and below, and girls 11 and below. Thus, a dichotomous and a continuous source of information were available for each informant (child and parent) and for the clinician.

385 25 with SP (28.4%), 24 with SoP (27.3%), and 17 with SAD (19.3%). Seventeen percent had none of the above diagnoses, 35.2% of patients had one of these diagnoses, 25% had two of these diagnoses, and 22.7% had three or more. Prior to analyzing the main hypotheses of the study, analyses were conducted to determine whether child gender or age significantly related to anxiety symptoms. Pearson correlations revealed that age was negatively correlated with parent and child reports of separation anxiety symptoms (SCAS: .24, SCAS-P: .33, ps < .05) and parent report of physical injury symptoms (SCAS-P: .25, p < .05), and positively correlated with child report and clinician impression of social anxiety symptoms (SCAS: .28, SoPavd: .21, ps < .05). A series of independent t tests indicated that patients with SoP were older than those without (diagnosis vs. no diagnosis: 13.46 vs. 11.89; t(86) = 2.04, p < .05) and that patients with SP were younger than those without (diagnosis vs. no diagnosis: 10.64 vs. 12.98; t(86) = 3.19, p < .05). Gender differences were apparent for some diagnoses. Independent samples t tests indicated greater mean scores for girls than boys on continuous variables associated with SoP (SCAS, SCAS-P, SoPavd), GAD (SCAS, SCAS-P, GADhtp), and SAD (SADsx), all ts > 2.17, ps < .05. Finally, chi-square analyses indicated that diagnoses of SoP, 2 = 9.40, p < .05, and GAD, 2 = 3.87, p < .05, were more prevalent in girls than boys. All other comparisons for age and gender were not statistically significant.

Analysis
To investigate agreement between child and parent report of symptoms, correlations between the SCAS and the SCAS-P subscales were examined. In addition, standardized difference scores were computed and compared with demographic variables. Agreement with the clinicians impressions was then examined through correlations between the SCAS and SCAS-P subscales and the continuous measure of symptoms for the corresponding disorder from the ADIS-C (i.e., SCAS and SCAS-P separation anxiety scales with SADsx). Semipartial correlations were computed to determine whether child and parent provided unique information relevant to diagnosis. Next, the relative ability of parent and child reports to predict diagnostic status was examined through a series of logistical regressions conducted for each diagnosis (OCD, GAD, SAD, SoP, and SP) separately. Finally, the clinical utility of parent and child reports was examined through computation of sensitivity, specificity, PPV, and NPV for each diagnosis.

ParentChild Agreement
To examine the agreement between parent and child reports of symptoms, Pearson correlations were calculated between the SCAS and SCAS-P (Table 1). To determine if the correlations between corresponding subscales (e.g., SCAS SAD with SCAS-P SAD) were greater than correlations with other subscales, significance tests for dependent correlations were conducted with the next greatest correlations (e.g., SCAS SAD and SCAS-P GAD). The total scores were not included in these analyses. These analyses indicated that the correlations between corresponding scales were significantly greater than the next highest correlation for all scales except GAD; all ts(82) > 2.11, ps < .05 for SAD, SP, SoP, and OCD. Table 2 presents the means and standard deviations of the parent- and child-report SCAS scales as well as four methods for evaluating the differences between informants. As can be seen from an inspection of the t values (degrees of freedom 81 to 83), children had a tendency to report more symptoms on the total score, SP and GAD subscales, whereas parents reported more SAD symptoms. To estimate the clinical magnitude of the differences scores, effect sizes were determined in terms of Cohens d by subtracting the child mean score from the parent mean score and

Results Preliminary Analyses

The frequency of diagnoses from the ADIS-C were as follows: 39 patients with OCD (44.3%), 35 with GAD (39.8%),

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

386
Table 1. Bivariate Correlations Between Child and Parent Report of Symptoms Child report Parent report SAD SP SoP OCD GAD Total SAD .74* .44* .28* .09 .46* .51*
a

Assessment 18(3) These analyses indicated that all subscales of the SCAS and SCAS-P correlated most strongly with the corresponding diagnostic symptoms, with the exception of GAD-P. To determine if the correlations between scales and corresponding diagnostic symptoms (e.g., SCAS SAD with ADIS-C SADsx) were greater than correlations with other diagnostic symptoms, significance tests for dependent correlations were conducted with the next greatest correlations (e.g., SCAS SAD and ADIS-C SPavd). These analyses indicated that the correlation between corresponding scales were significantly greater than the next largest correlation for the following subscales: SCAS-P SAD, t(84) = 2.88, p < .05, SCAS SAD, t(85) = 3.65, p < .05, SCAS-P OCD, t(83) = 5.31, p < .05, and SCAS OCD, t(81) = 3.00, p < .05. Next, significance tests for dependent correlations were conducted for each diagnostic category to determine if the strength of the relation with ADIS-C diagnostic symptoms differed between the SCAS and SCAS-P. No differences were found to be significant. To determine if the SCAS and SCAS-P explained unique variance in the ADIS-C symptom ratings, semipartial correlations were computed for the corresponding scales (Table 3). The tolerance and variance inflation factors were in the acceptable range (all tolerance > .40 and all variance inflation factors < 2.5). Each of the semipartial correlations was significant with the exception of child report of specific phobias. Overall, these results indicated that the SCAS and SCAS-P are related but not redundant and that each source provides unique information related to a continuous measure of clinician diagnostic impression.

SP .30* .64*a .08 .11 .10 .24*

SoP .17 .15 .60*a .10 .28* .37*

OCD .22 .12 .26* .60*a .32* .46*

GAD .30* .19 .35* .18 .41* .42*

Total .45* .34* .48* .26* .45* .56*

Note: Ns = 82 to 84. SAD = separation anxiety disorder, SP = specific phobia, SoP = social phobia, OCD = obsessive compulsive disorder, GAD = generalized anxiety disorder. a. Correlations between corresponding child and parent subscales, if they are significantly greater than the next largest correlation. *p < .05.

dividing this difference by the pooled standard deviation (Cohen, 1977). Cohen suggested that effect size magnitudes of 0.20, 0.50, and 0.80 correspond to small, medium, and large effects, respectively. All effect sizes fell in the small range. In addition to raw difference scores which ranged from .3 to 1.11 for the subscales, the SCAS scores were converted to z scores and a difference score was calculated by subtracting the child report from the parent report. (De Los Reyes & Kazdin, 2004). To examine the magnitude of discrepancy between standardized scores, the mean of the absolute value of the difference between standard scores was calculated because the mean difference between nonabsolute values would be zero. These values, presented in Table 2, indicate that the difference between parent and child reports (regardless of who reported more symptoms) was generally small, less than one standard deviation (i.e., a value of 1). To determine if age or clinician-rated symptoms were related to the absolute level of disagreement, regardless of direction, correlational analyses were conducted. The amount of discrepancy in OCD symptoms was found to be related to age, r = .23, p < .05, but no other significant relations were found between age or symptoms from the diagnostic interview with absolute or nonabsolute discrepancy scores, all rs < .20, ps > .10. Finally, a series of t tests indicated that parents and girls (m = 1.02) disagreed more so than parents and boys (m = .72) regarding GAD symptoms, t(82) = 2.07, p < .05.

Discriminant Validity
To evaluate the relative ability of parent and child reports to discriminate between children with and without each diagnosis, a series of logistical regressions was conducted. For each diagnosis, sex, age, and parent report were entered as predictors of the diagnoses from the ADIS. The analysis was then repeated with sex, age, and child report entered as predictors of the diagnoses from the ADIS. Thus, a significant unstandardized B value indicates that the parent (or child report in the second analysis) significantly predicted the diagnostic status. Next, the other respondent report (i.e., parent) was entered as a second step in the second regression. In this case, a significant B coefficient in Step 2 indicates that the variable (child or parent report) explains variance in the diagnosis above and beyond the variance shared with the other predictor variable. In the first series of logistical regression analyses, the presence of a diagnosis of SAD was entered as the outcome variable with age, sex, and the parent-report SCAS-SAD entered as predictors. The overall model was significant (2 = 22.00; p < .05). When child-report SCAS-SAD was

Agreement With Clinician

To investigate the agreement of parent and child reports of symptoms with the clinicians diagnostic impression, correlations between the SCAS/SCAS-P and ADIS-C continuous symptom scales were computed and are presented in Table 3.

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

Brown-Jacobsen et al.
Table 2. Means, Standard Deviations, Raw and Absolute Value Standardized Differences Scores for SCAS Scales SCAS Parent M (SD) SAD SP SoP OCD GAD Total 6.54 (4.2) 3.11 (2.7) 6.76 (4.2) 6.82 (5.7) 6.94 (3.2) 33.71 (17.2) Child M (SD) 5.52 (3.7) 3.72 (2.5) 7.13 (3.9) 7.76 (4.7) 8.06 (3.5) 37.63 (16.4) t value 3.17* 2.52* .93 1.82 2.83* 2.27* ES .26 .23 .09 .20 .33 .23 Raw M (SD) 1.01 (2.9) .62 (2.2) .36 (3.7) .94 (4.8) 1.11 (3.6) 3.93 (15.7) Difference scores

387

|Standardized| M (SD) .58 (.4) .66 (.53) .71 (.6) .67 (.6) .85 (.7) .75 (.6)

Note: SCAS = Spence Childrens Anxiety Scale; ES = effect size; |Standardized| = absolute value of difference between parent and child score; SAD = separation anxiety disorder; SP = specific phobia; SoP = social phobia; OCD = obsessive compulsive disorder; GAD = generalized anxiety disorder. *p < .05.

Table 3. Correlations Between SCAS/SCAS-P and Interviewer Impressions From ADIS-C

SCAS report SAD-P SAD-C SP-P SP-C SoP-P SoP-C OCD-P OCD-C GAD-P GAD-C ADIS-C symptoms SAD .65* /.24* .68*a/.27* .34* .21 .34* .27* .00 .12 .47* .43*
a

Table 4. Unstandardized B and Standard Errors for Predicting Diagnosis From SCAS and SCAS-P Individual predictors Dx (N) SAD (15/67) SP (24/59) SoP (21/62) OCD (37/47) GAD (32/52) SCAS-P .34 (.10)* .10 (.10) .24 (.08)* .33 (.07)* .11 (.08) SCAS-C .31 (.10)* .17 (.11) .26 (.09)* .27 (.07)* .12 (.08) Combined predictors (Step 2) SCAS-P .27 (.13)* .00 (.13) .17 (.09) .28 (.08)* .08 (.08) SCAS-C .12 (.13) .17 (.14) .15 (.10) .11 (.08) .09 (.08)

SP .36* .32* .54*/.37* .41*/.08 .16 .07 .15 .08 .31* .11

SoP .22* .10 .20 .10 .53*/.25* .58*/.33* .05 .08 .33* .37*

OCD .00 .03 .13 .15 .11 .04 .67*a/.40* .58*a/.21* .17 .16

GAD .25* .21 .15 .05 .44* .42* .09 .22 .39*/.22* .46*/.34*

Note: Ns = 80 to 87. SCAS = Spence Childrens Anxiety Scale; ADIS = Anxiety Disorders Interview Schedule; SAD = Separation Anxiety Disorder; SP = Specific Phobia; SoP = Social Phobia; OCD = Obsessive Compulsive Disorder; GAD = Generalized Anxiety Disorder. P and C after the instruments refer to parent and child versions, respectively. The values reported are zero-order correlations. Semipartial correlations controlling for other reporter (i.e., parent or child) are presented after the / for each SCAS/SCAS-P scale and its corresponding ADIS-C diagnosis. a. Zero-order correlations between corresponding SCAS/SCAS-P and ADIS-C scales, if they are significantly greater than the next largest correlation. *p < .05.

Dx (N) = Diagnosis (number of patients with/without diagnosis); SCAS = Spence Childrens Anxiety Scale; SAD = Separation Anxiety Disorder; SP = Specific Phobia; SoP = Social Phobia; OCD = Obsessive Compulsive Disorder; GAD = Generalized Anxiety Disorder. P and C after the instruments refer to parent and child versions, respectively. Individual predictors = unstandardized B value (standard error) when the scale is entered with age and sex. Combined predictors (Step 2) = unstandardized B value (standard error) when both scales are entered together along with age and sex. *p < .05.

entered alone with age and sex as predictors, the overall model was again significant (2 = 17.66; p < .05). Finally, when age, sex, parent and child reports were entered simultaneously, the model remained significant ( 2 = 22.82; p < .05) with parent report, but not child report, remaining significant. All unstandardized B values are presented in Table 4. When the same procedure was followed for SP, all three models were again significant: parent report (2 = 10.13; p < .05), child report (2 = 11.63; p < .05), and both variables combined (2 = 11.63; p < .05). The same pattern held for SoP (parent report: 2 = 24.14; p < .05, child report: 2 = 22.64; p < .05, combined: 2 = 26.45; p < .05), OCD (parent report: 2 = 42.14; p < .05, child report: 2 = 25.31; p < .05, combined: 2 = 44.30; p < .05), and GAD (parent

report: 2 = 8.56; p < .05, child report: 2 = 8.87; p < .05, combined: 2 = 9.78; p < .05). When each analysis was conducted with parent and child reports entered simulateously, only parent report of SAD and OCD remained significant, whereas child report was not sigificant for any of the scales.

Clinical Utility
The predictive validity of the SCAS and SCAS-P were examined through calculations of sensitivity (i.e., ability to identify an individual with a given disorder), specificity (i.e., ability to identify individuals without the diagnosis), PPV (i.e., percentage of individuals identified as having the

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

388

Assessment 18(3)

Table 5. Clinical Utility of Parent and Child Report for Predicting Clinician Diagnoses Based on 1.5 Standard Deviation Cut Score From Local Norms Sensitivity SAD-P SAD-C SP-P SP-C SoP-P SoP-C OCD-P OCD-C GAD-P GAD-C Averages Total Parent Child 1.00 0.73 0.32 0.33 0.68 0.64 0.95 0.78 0.76 0.73 0.69 0.74 0.64 Specificity 0.32 0.56 0.82 0.84 0.70 0.62 0.54 0.65 0.36 0.42 0.58 0.55 0.62 PPV 0.26 0.26 0.42 0.44 0.44 0.37 0.63 0.63 0.42 0.44 0.43 0.43 0.43 NPV 1.00 0.91 0.75 0.76 0.86 0.83 0.93 0.79 0.70 0.71 0.82 0.85 0.80 Correcta 0.45 0.59 0.67 0.69 0.69 0.62 0.72 0.71 0.51 0.54 0.62 0.61 0.63 Base rateb 0.18 0.19 0.28 0.29 0.26 0.26 0.44 0.45 0.39 0.38 0.31 0.31 0.31 Kappa 0.17* 0.15* 0.15 0.18 0.32* 0.21* 0.47* 0.42* 0.10 0.14 0.14 0.13 0.16

Note: PPV = positive predictive value; NPV = negative predictive value; SAD = Separation Anxiety Disorder; SP = Specific Phobia; SoP = Social Phobia; OCD = Obsessive Compulsive Disorder; GAD = Generalized Anxiety Disorder. P and C after the instruments refer to parent and child versions, respectively. a. Correct = informant and clinician agree. b. Base rate = rate of patients receiving the diagnosis based on clinician impression. *p < .05.

diagnosis that are true positives), and NPV (i.e., percentage of individuals identified as being diagnosis free that actually are true negatives). These four values are presented in Table 5 and based on previous literature (e.g., Achenbach & Rescorla, 2001; Briere, 1996; Conners et al., 1998; Nauta et al., 2004) were computed using cut scores set at 1.5 standard deviations above the community mean (Whiteside & Brown, 2008) based on age and sex. Although the performance of the subscales varied widely, on average both parent and child reports identified approximately 70% of cases and agreed with the clinicians decision approximately 60% of the time (combining both true positives and true negatives). In addition, given a positive finding on either the parent or child report, there was a 43% chance the patient had the given diagnosis which exceeded the base rate of 31%. Also, the NPVs were generally fairly robust, indicating that patients who did not score above the cutoff were unlikely to meet diagnostic criteria based on clinical assessment. However, as can be seen in Table 4, the PPVs were substantially lower than the sensitivity values. Finally, the kappa coefficients were calculated to determine if the agreement between diagnoses based on the cut score and from the clinician were significantly related. Despite this studys findings indicating strong correlations between parent, child, and clinician report of symptoms using continuous data, Kappa coefficients calculated with dichotomous data were generally low to moderate. This suggests that dichotomizing diagnostic data may contribute to low informant agreement reported in previous studies.

Informant Disagreement
Further analyses were conducted to examine the effects of disagreement between parent and child reports on clinician diagnosis. When both parent and child reports were suggestive of a diagnosis, the clinician made the diagnosis 48.1% of the time compared with 15.4% of the time if neither informant reported symptoms at a level greater than 1.5 standard deviations above the community mean. When only one reporter endorsed symptoms, the clinician assigned a diagnosis just under 30% of the time, regardless of whether the child (26.8%) or parent (28.6%) endorsed the symptoms. The difference in frequency of diagnosis based on the number of reporters endorsing symptoms was significant, 2(3) = 38.67, p < .05.

Discussion
The current investigation sought to address two obstacles to implementing the multimethod, multi-informant paradigm in clinical practice: informant disagreement and a lack of information regarding the utility of questionnaire data in predicting diagnosis in a given individual. One primary conclusion of the study is that, as hypothesized, child, parent, and clinician were in general agreement regarding the childs symptoms when these symptoms were measured in a continuous manner. Moreover, both parent and child contributed unique information to the diagnostic process, supporting claims that a multi-informant approach may

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

Brown-Jacobsen et al. increase the probability of accurate diagnosis and lead to an enhanced understanding of the child. Finally, this study demonstrated that information gathered from parent- and child-report forms can be diagnostically useful to the clinician. However, results highlight the differences between sensitivity/specificity and PPV/NPV. Consistent with the initial hypothesis, there was considerable agreement between parents and children in their report of child anxiety symptoms when measured in a continuous manner. In addition, although parent and child reports differed significantly for some symptoms, the magnitude of these discrepancies was small, as measured by effect sizes or standardized scores. This finding is in stark contrast to previous studies concluding that informant agreement is no better than chance and extremely poor (Foley et al., 2005; Grills & Ollendick, 2003; Safford et al., 2005). The present results likely show better agreement as a direct result of the use of continuous variables for parent and child input. Indeed, when agreement on dichotomous variables (which restrict variance and thus limit the ability to identify meaningful relations) was measured in the present study, results were much poorer. Thus, the greater level of agreement is consistent with previous studies that have used continuous data obtained from structured interviews (Comer & Kendall, 2004) and questionnaires (Nauta et al., 2004; Spence, 1998). These finding demonstrate the shortcomings of dichotomous diagnostic measures and support the use of dimensional assessment or the incorporation of severity ratings into diagnostic systems. The second goal of the study was to examine the agreement of parent and child reports with the clinicians diagnostic impression. Overall, results demonstrated high agreement between parent/child and clinician, with the strongest correlations found between reports of corresponding diagnostic symptoms. This finding is consistent with previous studies that have found that anxiety diagnoses as obtained with the ADIS-C/P were best predicted by their corresponding subscales on the Screen for Child Anxiety Related Emotional Disorders (SCARED) parent and child versions (e.g., Bodden et al., 2009). However, the present study is believed to be the first investigation to compare the degree to which parent and child agree with a clinicians impression on a continuous scale. The present results suggest that clinician ratings tend to agree with both parent and child ratings relatively evenly. Moreover, this is the first study to demonstrate that the SCAS and SCAS-P are significantly related to the ADIS-C. Despite being in general agreement, both child and parent appear to provide unique information regarding child anxiety symptoms. Specifically, the semipartial correlations indicate that after controlling for the variance in clinician impressions explained by one reporter (e.g., parent), the other reporter (i.e., child) continues to explain additional

389 variance in the clinicians impressions. As with parent child agreement, the ability to detect meaningful relationships appears to be stronger with continuous variables as only parent report of SAD and OCD symptoms contributed unique variance in the prediction of dichotomous diagnoses. Moreover, the results of dichotomizing parent and child reports into a positive or negative indicator of a diagnosis suggest that the clinician should consider both sources of information. That is, when the parent and child disagreed, the clinician arrived at a diagnosis 25% to 30% of the time regardless of which reporter led to a positive result. As would be expected, this rate of diagnosis falls between the rates of diagnosis when neither (15%) or both (50%) reporters endorsed symptoms at a level suggestive of a diagnosis. Finally, there were some differences in the type of information provided by the informants. Specifically, parents tended to endorse more SAD symptoms than children did, possibly because they are by definition involved in these symptoms and may be particularly distressed by them. However, children tended to endorse more GAD symptoms, suggesting that they may be sensitive to some anxiety symptoms of which parents are less aware. Parents may also be less aware of, or less concerned about, physical injury fears which they also tended to report less of. Taken together, these results suggest that it is important to gather information from both the parent and child. A third and important aspect of the current investigation was to identify the clinical relevance and relative contribution of parent and child questionnaire data to the diagnostic process. Overall, the data suggests that parent and child ratings do make a useful contribution by identifying patients with clinically significant symptoms versus those who are unlikely to warrant a diagnosis. On average, the scales could identify approximately 70% of positive cases (sensitivity). Of identified cases, 40% were true positives (PPV), which is an increase over the base rate of approximately 30%. Probably the most important finding is that even with a positive test result, the majority of individuals (60%) were not deemed to meet the criteria for the diagnosis. Thus, the cutoffs chosen for the current investigation, which are consistent with those used for common emotional and behavioral assessment tools, appear to maximize sensitivity but do not provide particularly strong PPV. Conversely, the scales were very effective for ruling out a diagnosis; of patients who were not identified as having a diagnosis based on questionnaire data, 82% were true negatives (NPV). This information is very important to the clinician who is attempting to increase the empirical rigor of his or her practice through the use of objective questionnaires and is not conveyed through commonly reported mean differences or sensitivity rates. Beyond the average utility across all of the subscales, there were important differences in the level of performance

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

390 among the subscales. For instance, parent report was a very accurate measure of OCD as it identified 95% of cases, had over 60% true positives, and resulted in a significant interrater agreement with the clinician. At the other extreme, the SP scales performed poorly, identifying barely 30% of cases and not agreeing with the clinician to a statistically significant degree. Thus, the data suggest that different guidelines for interpretation need to be developed for each of the SCAS and SCAS-P subscales. Differences between the utility of the subscales likely results from a number of factors including the quality of the scales and the nature of the underlying anxiety construct. For instance, both the GAD and SP Scales performed more poorly than the other scales. The GAD Scale is, by nature, quite general and includes symptoms that are related to multiple anxiety disorders. For example, not only did the GAD Scale have the lowest parentchild agreement based on correlations and discrepancy scores but also the largest correlation with parent-reported GAD was child-reported SAD rather than GAD. Difficulties measuring GAD are consistent with past literature which has reported difficulty distinguishing children with GAD from children with other anxiety disorders based on child and parent report (Wood et al., 2002). The SP Scale, in contrast, may have suffered from overly specific items that covered too wide a range of idiosyncratic fears. This hypothesis is supported by the fact that the internal consistency of the SP Scale (.47/.62) was lower than that of the other scales. This may explain why the correlations between the ADIS- and SCAS- C/P were lower and the utility of scale was poor. Moreover, the SCAS subscale is named physical injury fears and thus may not cover the range of phobias reported on the ADIS. In contrast, other scales may have performed better because of the characteristics of the disorders they assess. For instance, OCD may be particularly easy to identify given its stark contrast to typical functioning and obvious behavioral abnormalities. However, GAD may be more difficult to identify because worry is more of an internal process and is an exaggerated form of everyday experience, that is, all children worry to some extent. One important limitation of the current study is the idiosyncratic nature of clinical utility estimates. The rates of sensitivity, specificity, PPV, and NPV are based on the specific cutoff scores, base rates, and normative information. Although changes to these parameters would significantly alter the results, this study demonstrates how information can be collected to better inform the use of these scales. Few would argue that sensitivity and specificity should not be reported because they differ depending on the characteristics of the sample; the present study makes the case for the additional utility of information related to PPV and NPV, which are more directly applicable to clinical practice. In a

Assessment 18(3) related note, although the PPV of many of the scales was marginal, their performance was fairly impressive given that the present use was to differentiate patients with and without a diagnosis in a clinical sample composed entirely of children with anxiety symptoms. A further limitation of the study is the relatively affluent nature of the current sample, which may limit generalization of these results. Future research, such as developing cut scores based on large and diverse samples that can be used clinically in a variety of settings, is needed to address these limitations. In particular, studies in clinical and community samples and investigations in clinical settings that are not anxiety-specific clinics would be of benefit as different base rates of anxiety disorders will significantly alter PPV estimates. In addition, further studies regarding the factors that lead to discordance between reporters and how clinicians integrate information from children and parents, as well as other sources, is needed. Ultimately, the goal of assessment is to inform treatment decisions. Future research could examine the implications of informant discrepancies for treatment planning and outcome. For instance, although discrepancy scores were not related to severity based on information from the structured interview, they could predict shorter treatment because they reflect less severe symptoms, longer treatment because they complicate treatment planning, or be unrelated. Such information may be present in existing data from treatment trials, although analyzing discrepancies based on dichotomous diagnostic categories may inflate the occurrence of disagreement and confound analysis. Despite these limitations, the current results can provide some concrete recommendations for clinicians. To begin with, children and their parents generally provide consistent information and the likelihood of a disorder being present increases when symptoms are reported by both individuals. However, it is important to gather information from both parent and child because they provide unique perspectives and disorders may be present even if only one of them endorses symptoms. In cases where a parent and child disagree, the clinician should keep in mind that it is less likely that the child meets criteria for a diagnosis than if they both agreed. When attempting to combine discrepant reports, the clinician should consider information from both parent and child, although he or she may wish to give more weight to parental reports of SAD or OCD (particularly for younger children). In contrast, children tend to report more symptoms of GAD, and thus the clinician may wish to rely more on the child report, although information regarding this diagnosis tended to be less reliable in general. Finally, clinicians are encouraged to use the SCAS to aid in the diagnostic process. A cutoff score set at 1.5 standard deviations above a nonclinical mean can be determined based on the data in this article; additionally, t scores developed for the child-report version and norms from a variety

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

Brown-Jacobsen et al. of countries are available (scaswebsite.com). Finally, a positive elevation merely increases the likelihood of a diagnosis, which often remains below 50%. Thus, the SCAS should be used as an adjunct to a thorough clinical interview and may be more helpful in ruling out diagnoses. Of course, all of the above recommendations need to be replicated by future investigations. Overall, the current study supports the recommendation that information be gathered from both children and parents when evaluating childhood anxiety disorders. In addition, the results reduce concerns regarding discordant information being provided by parents and children and conversely suggest that, overall, these reporters are consistent, agree with the clinician, and that each reporter provides information relevant to diagnosis of an anxiety disorder. Moreover, although rating scales appear to be useful, they need to be used carefully with knowledge of the meaning of results, that is, PPV and NPV. In conclusion, questionnaire data provide information that is useful and relevant to clinical practice. Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the authorship and/or publication of this article.

391
Comer, J. S., & Kendall, P. C. (2004). A symptom-level examination of parentchild agreement in the diagnosis of anxious youths. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 878-886. Conners, C., Sitarenios, G., Parker, J. D., & Epstein, J. N. (1998). The revised Conners Parent Rating Scale (CPRS-R): Factor structure, reliability, and criterion validity. Journal of Abnormal Child Psychology, 26, 257-268. De Los Reyes, A., & Kazdin, A. E. (2004). Measuring informant discrepancies in clinical child research. Psychological Assessment, 16, 330-334. Edelbrock, C., Costello, A. J., Dulcan, M. K., Conover, N. C., & Kala, R. (1986). Parentchild agreement on child psychiatric symptoms assessed via structured interview. Journal of Child Psychology & Psychiatry & Allied Disciplines, 27, 181-190. Foley, D. L., Rutter, M., Angold, A., Pickles, A., Maes, H. M., Silberg, J. L., et al. (2005). Making sense of informant disagreement for overanxious disorder. Journal of Anxiety Disorders, 19, 193-210. Foley, D. L., Rutter, M., Pickles, A., Angold, A., Maes, H., Silberg, J., et al. (2004). Informant disagreement for separation anxiety disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 452-460. Grills, A. E., & Ollendick, T. H. (2003). Multiple informant agreement and the anxiety disorders interview schedule for parents and children. Journal of the American Academy of Child & Adolescent Psychiatry, 42(1), 30-40. Ialongo, N., Edelsohn, G., Werthamer-Larsson, L., Crockett, L., & Kellam, S. (1993). Are self-reported depressive symptoms in first-grade children developmentally transient phenomena? A further look. Development and Psychopathology, 5, 433-457. Jensen, P. S., Rubio-Stipec, M., Canino, G., Bird, H. R., Dulcan, M. K., Schwab-Stone, M. E., et al. (1999). Parent and child contributions to diagnosis of mental disorder: Are both informants always necessary? Journal of the American Academy of Child & Adolescent Psychiatry, 38, 1569-1579. Muris, P., Merckelbach, H., Ollendick, T., King, N., & Bogie, N. (2002). Three traditional and three new childhood anxiety questionnaires: Their reliability and validity in a normal adolescent sample. Behaviour Research & Therapy, 40, 753-772. Muris, P., Schmidt, H., & Merckelbach, H. (2000). Correlations among two self-report questionnaires for measuring DSMdefined anxiety disorder symptoms in children: The screen for child anxiety related emotional disorders and the Spence Childrens Anxiety Scale. Personality & Individual Differences, 28, 333-346. Nauta, M. H., Scholing, A., Rapee, R. M., Abbott, M., Spence, S. H., & Waters, A. (2004). A parent-report measure of childrens anxiety: Psychometric properties and comparison with childreport in a clinic and normal sample. Behaviour Research & Therapy, 42, 813-839. Rapee, R. M., Barrett, P. M., Dadds, M. R., & Evans, L. (1994). Reliability of the DSM-III-R childhood anxiety disorders using

Funding
The authors received no financial support for the research and/or authorship of this article.

References
Achenbach, T. M. (1999). Developmental taxonomy. In W. K. Silverman & T. Ollendick (Eds.), Developmental issues in the clinical treatment of children (pp. 31-43). Needham Heights, MA: Allyn and Bacon. Achenbach, T. M., & Rescorla, L. A. (2001). Manual for the ASEBA School-Age Forms & Profiles. Burlington: University of Vermont, Research Center for Children, Youth, & Families. Albano, A. M., & Silverman, W. K. (1996). Anxiety Disorders Interview Schedule for DSM-IV Child Version: Clinical manual. Albany, NY: Graywind. Berge, M. T., Veerkamp, J. S. J., Hoostraten, J., & Prins, P. J. M. (2002). The Dental Subscale of the Childrens Fear Survey Schedule: Predictive value and clinical usefulness. Journal of Psychopathology and Behavioral Assessment, 24(2), 115-118. Bodden, D. H. M., Bogels, S. M., & Muris, P. (2009). The diagnostic utility of the Screen for Child Anxiety Related Emotional Disorders-71 (SCARED-71). Behaviour Research and Therapy, 47, 418-425. Briere, J. (1996). Trauma Symptom Checklist for Children: Professional manual. Odessa, FL: Psychological Assessment Resources. Cohen, J. (1977). Statistical power analysis for the behavioral sciences. New York: Academic Press.

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014

392
structured interview: Interrater and parentchild agreement. Journal of the American Academy of Child & Adolescent Psychiatry, 33, 984-992. Safford, S. M., Kendall, P. C., Flannery-Schroeder, E., Webb, A., & Sommer, H. (2005). A longitudinal look at parentchild diagnostic agreement in youth treated for anxiety disorders. Journal of Clinical Child and Adolescent Psychology, 34, 747-757. Silverman, W. K., & Ollendick, T. H. (2005). Evidence-based assessment of anxiety and its disorders in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34, 380-411. Smith, G. E., Cerhan, J. H., & Ivnik, R. J. (2003). Diagnostic Validity. In D. S. Tulsky, D. H. Saklofske, G. J. Chelune, R. K. Heaton, R. J. Ivnik, R. Bornstein, A. Prifitera, &

Assessment 18(3)
M. F. Ledbetter (Eds.), Clinical Interpretation of the WAIS-III and WMS-III (pp. 273-303). San Diego, CA: Academic Press. Spence, S. H. (1998). A measure of anxiety symptoms among children. Behaviour Research & Therapy, 36, 545-566. Whiteside, S. P., & Brown, A. (2008). Exploring the utility of the Spence Childrens Anxiety Scales Parent- and Child-Report Forms in a North American sample. Journal of Anxiety Disorders, 22, 1440-1446. Wood, J. J., Piacentini, J. C., Bergman, R. L., McCracken, J., & Barrios, V. (2002). Concurrent validity of the anxiety disorders section of the Anxiety Disorders Interview Schedule for DSM-IV: Child and Parent Versions. Journal of Clinical Child and Adolescent Psychology, 31, 335-342.

Downloaded from asm.sagepub.com at Universiti Teknologi Malaysia on January 12, 2014