Comparing Four Methods of Integrating Parent and Teacher Symptom Ratings of Attention-deficit/hyperactivity Disorder (ADHD)

Transcription

1 DOI /s Comparing Four Methods of Integrating Parent and Teacher Symptom Ratings of Attention-deficit/hyperactivity Disorder (ADHD) Shirag K. Shemmassian & Steve S. Lee # Springer Science+Business Media, LLC 2011 Abstract Although parents and teachers are valid informants in the assessment of childhood attention-deficit/ hyperactivity disorder (ADHD), there is relatively little systematic research on how these ratings should be optimally combined. We compared four methods of ADHD assessment to determine how well they identified impaired children: (1) parent only, (2) teacher only, (3) parent or teacher ( or rule ), and (4) parent and teacher ( and rule ). We obtained parent and teacher ratings of ADHD from the Disruptive Behavior Disorder Rating Scale on to 10-year-old children (69% male; 47% Caucasian) with (n= 121) and without (n=111) ADHD. We used receiver operating characteristic curves (ROC) and seemingly unrelated regression analyses (SUR) to evaluate how accurately each method identified categorically- and dimensionally-defined measures of functional impairment. Parent ratings of ADHD optimally identified globally impaired children based on categorical and dimensional measures. However, teacher ratings of ADHD most accurately identified children who were negatively regarded by peers using categorical, but not dimensional, measures. No ADHD assessment method effectively identified children with academic difficulties. Although multiple informants are valuable in the assessment of ADHD, no single method was consistently superior in identifying impaired children across domains. We consider alternative assessment strategies in ADHD as well as other potential factors that may contribute to modest agreement among informants. S. K. Shemmassian : S. S. Lee (*) Department of Psychology, UCLA, 1285 Franz Hall, Box , Los Angeles, CA , USA stevelee@psych.ucla.edu Keywords ADHD. Multiple informants. Impairment. Children Attention-deficit/hyperactivity disorder (ADHD) is characterized by developmentally extreme and impairing levels of inattention and/or hyperactivity and constitutes the most common childhood mental disorder with a worldwide prevalence of 5.3% (Polanczyk et al. 2007). ADHD prospectively predicts negative adolescent and adult outcomes: children with ADHD consistently exhibit more social, academic, and occupational impairment, in addition to elevated comorbidity with mood and externalizing disorders than controls (Biederman et al. 2010; Lee et al. 2008). Moreover, a recent meta-analysis of 27 longitudinal studies reported that childhood ADHD prospectively predicted adolescent and adult substance abuse/dependence across all relevant types of substances (Lee et al. 2011). Thus, childhood ADHD disrupts important domains of academic, social, behavioral, and emotional development. Given its clinical significance, there is a need for reliable and valid diagnostic procedures for ADHD given that they will facilitate early identification, timely delivery of interventions, and potentially reduce the risk of negative outcomes. Presently, there are several evidence-based assessment strategies for ADHD. Structured diagnostic interviews (e.g., Diagnostic Interview Schedule for Children, 4 th edition, DISC-IV; Shaffer et al. 2000) address all relevant diagnostic criteria (e.g., age of onset, persistence) and are considered the gold standard based on their superior psychometric properties. In a study of 247 children, the DISC 2.3 demonstrated moderate to high inter-rater agreement between diagnoses derived from lay interviewers and clinicians (retest interval was 1 15 days for 83% of the sample; κ=.60; Schwab-Stone et al. 1996).

2 They also found that agreement between diagnoses from the DISC and clinicians using symptom ratings from a clinical interview with the parent was acceptable (κ=.72). Furthermore, Shaffer et al. (2000) reported acceptable testretest reliability for ADHD diagnoses based on the DISC with the parent (κ=.79; mean retest interval=6.6 days). Semi-structured interviews (e.g., Kiddie Schedule for Affective Disorders and Schizophrenia for School-Age Children- Present and Lifetime Version, KSADS-PL; Kaufman et al. 2000) also probe diagnostic criteria and provide the interviewer with flexibility in querying key domains. Although the KSADS-PL is used extensively in clinical research with high inter-rater and test-retest reliability for internalizing and externalizing disorders, ADHD diagnoses from the KSADS-PL may be slightly less reliable (κ=.63 for current versus.55 for lifetime diagnosis) than the DISC (Kaufman et al. 1997). Finally, in addition to children and adolescents, structured and semi-structured diagnostic interviews have good empirical support in the assessment of ADHD in young children (i.e., preschool) (Lee and Humphreys, in press). Rating scales are also often used in the assessment of ADHD because they are efficient and can be used with multiple informants. The Conners Rating Scales (CRS; Conners 1997) and the Achenbach System of Evidence Based Assessment (e.g., Child Behavior Checklist, CBCL; Achenbach and Rescorla 2001) can be administered conveniently and provide developmentally-sensitive and empirically-derived indicators of psychopathology based on large normative samples, a key benefit given that symptoms of ADHD are common. Second, although they lack norms, the Disruptive Behavior Disorder Rating Scale (DBD; Pelham et al. 1992) and the Swanson, Nolan, and Pelham, IV (SNAP-IV) (Swanson et al. 2001) use language keyed to DSM-IV criteria for ADHD, oppositional defiant disorder (ODD), and conduct disorder (CD), thus providing a more direct assessment of DSM-IV phenomenology than the CBCL or CRS. Parent DBD data from 116 children yielded coefficient alphas of.82 and.85 for the inattentive and hyperactive dimensions, respectively, with slightly higher estimates for teachers (Massetti et al. 2005). Test-retest and inter-rater reliability estimates for the DBD in the sample were excellent and DBD ratings were sensitive to diagnoses derived from the DISC. The SNAP-IV also has strong psychometric properties. In a large sample of children in kindergarten through 5th grade, parent SNAP-IV ratings yielded alphas of.90 and.79 for inattention and hyperactivity symptoms according to parents and.96 and.92, respectively, for the same domains according to teachers (Bussing et al. 2008). We emphasize, however, that most ADHD rating scales prioritize the assessment of symptoms rather diagnostic criteria (e.g., age of onset). Thus, formal diagnosis of ADHD should not be based only on rating scales. The recommendation that the assessment of ADHD be based on multi-informant ratings (Pelham et al. 2005) is based on evidence that parents and teachers each provide incremental validity in predictions of negative outcome (Johnston and Murray 2003). However, parent and teacher ratings of ADHD are typically only modestly related. In their heuristic meta-analysis, Achenbach et al. (1987) found low to moderate agreement among informants on most major dimensions of child psychopathology, including ADHD. Kolko and Kazdin (1993) reported that parent and teacher ratings from the CBCL and TRF correlated at.29 and.51 for internalizing and externalizing problems, respectively. Informant discrepancies have important clinical implications because they may lead to different conclusions, including the assessment of the client s need for treatment (De Los Reyes and Kazdin 2005). Perhaps equally important is the possibility that discrepancies among informants exist due to the context in which the information was collected. Therefore, the situational validity of data is crucial to assessing child psychopathology and evaluating treatment planning (De Los Reyes and Kazdin 2005). Despite the value of multi-informant data to the assessment of ADHD, relatively little work has addressed how these data should be integrated. That is, we need new knowledge that evaluates which informant or diagnostic algorithm best identifies impaired children (Achenbach 2006). Mental health professionals perceived teachers as better informants of hyperactivity and inattention than mothers; children were not seen as useful informants (Loeber et al. 1990). Moreover, although combining multi-informant data using the or rule (i.e., symptom is present if endorsed by the parent or teacher; Piacentini et al. 1992) to diagnose ADHD may be valid (Lahey et al. 1998), this approach increases the number of true positives (impaired children correctly diagnosed with ADHD), but also the number of false positives (unimpaired children incorrectly diagnosed with ADHD), which may contribute to the diagnostic instability of ADHD. In one study, classification rates of ADHD (and its subtypes) varied dramatically by informant, instrument, and diagnostic algorithm, ranging from 52 cases (42%) to 103 cases (84%) (Valo and Tannock 2010). There may also be an optimal number, or even specific symptoms, that optimally assess ADHD. Among 5- to 12-year-old children, different ADHD symptoms endorsed by parents (i.e., avoids tasks that require sustained mental effort and has difficulty sustaining attention in tasks or play activities ) versus teachers ( loses things necessary for tasks or activities and has difficulty organizing tasks and activities ) showed the best positive predictive power (Power et al. 2001). Therefore, aggregating symptoms by predictive accuracy and informant may be more effective than the DSM-IV

3 approach, which treats items and informants equivalently (Power et al. 2001). Overall, there is a need to empirically evaluate how parent and teacher ratings of ADHD, individually and collectively, relate to important measures of functional impairment. In light of the critical role that ADHD assessment plays to delivering services and monitoring treatment outcome, as well as evidence on the substantial variability in determining ADHD caseness using common methods (Valo and Tannock 2010), we evaluated to 10-year-old children with and without ADHD (ascertained according to a structured diagnostic interview with the parent). We obtained separate parent and teacher ratings of ADHD and evaluated four common ADHD algorithms: (A) parent only; (B) teacher only; (C) parent or teacher, and (D) parent and teacher. Given that clinical significance is contingent on functional impairment (Pelham et al. 2005), we utilized a diverse range of outcome measures across informants (i.e., parent, teacher and interviewer) and domains (i.e., global functioning, academic achievement, and social preference). Method Participants Participants consisted of 232 (69% male) ethnically diverse 5- to 10-year-old children (M=7.4, SD=1.1; 98% of the sample was 6 9 years) with (n=121) and without (n=111) ADHD (Table 1). 47% of participants self-identified as White/Caucasian, 9% as Black/African American, 11% as Latino/Hispanic, 4% as Asian, 24% as Biracial, and 5% Other. Participants were recruited through presentations to self-help groups and advertisements mailed to local elementary schools, pediatric offices, and clinical service providers. Exclusionary criteria included a Full Scale IQ< 70, as well as a previous diagnosis of a pervasive developmental, seizure, or neurological disorder that prevented full participation in the study. Both parent and child were required to be fluent in English and children were required to live with at least one biological parent no less than half time. ADHD diagnostic status was ascertained by the DISC-IV with the parent (Shaffer et al. 2000), which probed all relevant diagnostic criteria (e.g., age of onset). Control children were recruited, screened, and assessed identically to probands. To improve the external validity of probands, comorbid disorders were permitted. In addition, children who met criteria for any disorder other than ADHD (e.g., anxiety) were conservatively placed into the control group, thus increasing the similarity between ADHD probands and controls. Procedures Study eligibility was assessed through an initial telephone screening. For eligible families, parents were mailed rating scales and families were invited to our research lab for inperson assessments of child behavior and family functioning. Each child s primary teacher was asked to complete Table 1 Demographic and clinical characteristics of diagnostic groups Variable ADHD (n=121) Controls (n=111) t/χ 2 M (SD) M (SD) Age 7.29 (1.17) 7.54 (1.12) 1.58 Number of Boys (%) 88 (73%) 71 (64%) 1.43 Number of non-white children (%) 67 (55%) 57 (51%).13 Number of DISC ADHD Symptoms (3.14) 3.11 (2.87) 23.38* Number of DISC ODD Symptoms 3.27 (2.41) 0.96 (1.72) 8.34* Number of DISC CD Symptoms 0.74 (1.03) 0.18 (0.47) 5.25* Interviewer CGAS (13.26) (14.85) 12.74* WIAT-II Word Reading (18.49) (15.89) 2.00* WIAT-II Math Reasoning (17.91) (17.50) 2.57* Parent DBD ADHD Symptoms (4.55) 2.36 (3.13) 15.72* Parent Dishion Negative SP Score 1.39 (1.27).55 (1.00) 5.43* Teacher DBD ADHD Symptoms 7.87 (5.04) 5.46 (5.42) 3.51* Teacher Dishion Negative SP Score 2.44 (2.22) 1.54 (2.08) 2.63* Note. DISC-IV = Diagnostic Interview Schedule for Children, 4th edition; ODD = Oppositional Defiant Disorder; CD = Conduct Disorder; CGAS = Children s Global Assessment Scale; WIAT-II = Wechsler Individual Achievement Test, 2nd Edition; DBD = Disruptive Behavior Disorder Rating Scale; Dishion = Dishion Social Preference Scale; SP = Social Preference *p<.05.

4 parallel rating scales of child behavior. Approximately 85% of children were assessed without psychotropic medication. If a child normally received medication, their parents and teachers were asked to provide ratings based on the child s unmedicated behavior. Similar procedures have been used in other ADHD studies (Lee et al. 2008). Parents consented and children assented to all relevant study procedures, which were approved by the IRB. Measures Diagnostic Interview Schedule for Children - Fourth Edition (DISC-IV; Shaffer et al. 2000) This highly structured interview with the parent probes full diagnostic criteria for ADHD including symptom thresholds, duration, age of onset, and impairment. Test retest reliability for ADHD from the DISC was between.51 and.64 in the DSM-IV Field Trials (Lahey et al. 1994). Diagnostic designations from the DISC have also shown predictive validity in other studies of ADHD (Owens et al. 2009; Lee et al. 2008). Disruptive Behavior Disorder Rating Scale (DBD; Pelham et al. 1992) Parent and teacher rated each behavior as not at all, just a little, pretty much, or very much. Consistent with previous studies, symptoms rated as pretty much or very much were considered present (Lahey et al. 1998; Pelham et al. 1992). Parents and teachers completed identical versions of this rating scale, which has excellent psychometric properties (Pelham et al. 1992). In our sample, the parent and teacher DBD rating scales had Cronbach s alphas of.96 and.94 for inattentive and hyperactive symptoms, respectively. Children s Global Assessment Scale (CGAS; Setterberg et al. 1992). The CGAS was completed by interviewers following completion of the DISC. Informants chose a number between 1 and 100 that best described the child s lowest level of functioning during the past 6 months (higher number indicated better functioning). Descriptions for score ranges were provided on the scale. The CGAS is reported as having strong test retest reliability (raters mean reliability coefficient=.87) (Bird et al. 1987; Shaffer et al. 1983). The Wechsler Individual Achievement Test - Second Edition (WIAT-II; Wechsler 2002) The Word Reading subtest assesses phonemic awareness and reading fluency. Math Reasoning assesses computational knowledge and the calculation of word problems. These subtests were sensitive to achievement in learning disabled children (Geary et al. 2007) and demonstrated excellent test retest reliability (Wechsler 2002). Dishion Social Preference Scale (Dishion 1990) This is a three-item (five-point metric) parent- and teacher-completed measure of peer acceptance, rejection, and being ignored. Social preference was predicted from initial sociometric ratings and was significantly correlated with antisocial behavior, depression, and deviant peer association (rs=.60,.30, and.51, respectively) (Dishion 1990). The rejection rating was subtracted from the acceptance rating and then reverse scored to estimate negative preference. This method has been sensitive to group differences in other studies of ADHD (Lee and Hinshaw 2006; Lahey et al. 2004). Data Analytic Strategies A significant challenge in identifying optimal informants or potential combinations of multi-informant data for ADHD is the absence of universally-accepted criteria. Many constructs in psychopathology, including clinical recovery, resilience, and treatment outcome, are based on face validity or arbitrary thresholds (Andreasen et al. 2005), including definitions of normalized functioning in studies of ADHD (Biederman et al. 1995). With the exception of standardized academic achievement, the absence of normative data for functional impairment measures necessitated that we designate meaningful thresholds. Using a standard in treatment outcome research (Jacobson and Truax 1991) and validated in recent prospective studies of children with ADHD (Owens et al. 2009; Lee et al. 2008), we defined the midpoint of ADHD and control group means as the threshold for impairment. For example, if the mean CGAS score was 70 for probands and 90 for controls, a score of 80 or less was defined as impaired. Thresholds were estimated separately for parents and teachers (Table 1). Given the availability of normative data, academic impairment was defined as having a score of 85 or less (i.e., 1 SD below the mean; 16th percentile) on the Word Reading or Math Reasoning subtest. However, given the centrality of dimensional perspectives in psychopathology (Helzer et al. 2008), continuous outcome measures were also analyzed. Further, because sex and ODD symptoms are reliably associated with ADHD (Newcorn et al. 2001), we controlled for age, sex, and the number of ODD symptoms from the DISC in all analyses involving continuous measures of impairment. Age was not significantly correlated with any diagnostic algorithm and was therefore not controlled. Using parent and teacher data from the DBD Rating Scale, children with at least 6 symptoms of inattention or hyperactivity were considered as having met symptom criteria for ADHD according to four different methods: (A) parent report only (i.e., children who exceeded 6 or more symptoms of inattention or hyperactivity according to the parent); (B) teacher report only (i.e., children who met

5 symptom criteria for ADHD according to the teacher); (C) parent and teacher ratings combined using the or rule (Piacentini et al. 1992); and (D) combined parent and teacher rating using the and rule (i.e., symptom was present if endorsed by the parent and teacher) (Table 2). Available parent data ranged from 95% (Dishion) to 99% for all other measures and 69% (Dishion) to 83% (DBD) for teachers. However, our person-centered analyses, which contrasted children who did versus did not meet symptom criteria for ADHD, made complete data a priority. Therefore, we imputed missing data using ten iterations of Markov Chain Monte Carlo in SAS PROC MI (Yuan 2002) for the parent and teacher DBD, Dishion, as well as the parent DISC and CGAS. To approximate the actual covariance structure, analyses were conducted using the mean of the ten iterations, which were highly consistent (see Table 1). We compared four methods of evaluating ADHD using Receiver Operating Characteristic (ROC) curves, consisting of graphical representations of true positive versus false positive rates across our ten imputations. By comparing the true (sensitivity) and false positive (1 - specificity) rates, ROC analyses compared the areas between two curves (i.e., each method s discriminative ability; Hanley and McNeil 1982) to evaluate whether one method of assessing ADHD was superior to another in identifying impaired children. We then averaged the significance level across all analyses. When there was a significant omnibus result in a given domain, we compared each of the four diagnostic algorithms. Bonferri corrections were performed for each functional domain (α=.05/6=.008) to control for multiple comparisons. We emphasize that the ROC analyses were extremely consistent across the 10 iterations of imputation. Continuous outcome data were analyzed for each domain using seemingly unrelated regressions (SUR; Zellner 1962) which account for correlated error terms in a set of linear equations to determine whether diagnostic algorithms were significantly associated with outcome variables in each domain. Post-hoc chi-squared tests compared the Table 2 Number of children who met DBD-rated symptom criteria for ADHD according to each diagnostic method Statistic Parent Report Only Teacher Report Only Or Rule And Rule Mean 94 (41%) 96 (41%) 147 (63%) 33 (14%) (%) Range Note. Means and ranges based on ten imputations. Or Rule : symptom considered present if either the parent or teacher endorsed it. And Rule : symptom considered present if both the parent and teacher endorsed it. significant diagnostic algorithm with largest coefficient with the others. Finally, we also calculated the number of true positives, true negatives, false positives, and false negatives for each diagnostic method using our categorical thresholds for functional impairment. We then calculated each method s sensitivity and specificity, in addition to their false positive and false negative rates (Table 3). Results As shown in Table 2, the four methods of using parent and teacher reports yielded the following number of children who met symptom criteria for ADHD based on the DBD rating scale: parent only=94; teacher only=96; or rule = 147; and rule =33. As expected, whereas combining parent and teacher data using the or rule yielded the most number of ADHD diagnoses across imputations (64%), the and rule yielded the fewest number of diagnoses (14%). Interestingly, parent-only or teacher-only rated symptoms provided a nearly identical number of youth with ADHD (41% each); however, only 51 of these 96 children in question met symptom criteria for ADHD according to the parent-only and teacher-only methods. Results of the ROC analyses are presented below (see Table 4) along with an example ROC output (see Fig. 1). Global Functioning (CGAS) The omnibus test for the CGAS was significant (χ 2 =25.90, p<.0001). Post-hoc comparisons among the four methods suggested that parent ratings of ADHD optimally identified globally impaired children and were superior to teacher only ratings (B=.14, p<.005) and the and rule (B=.14, p <.0001). However, there was no difference between ratings based on parents only versus ratings that were based on the or rule (B=.04, p=.23). The or rule similarly outperformed ratings from teachers only (B=.10, p<.005) and the and rule (B=.10, p<.01). SUR results were consistent with ROC analyses where parent only ratings of ADHD (B =.01, p<.005) and the or rule (B=.006, p<.05) were significantly associated with the parent-rated CGAS (i.e., global functioning) whereas teacher ratings and the and rule were not (B=.003, p=.33 and B=.006, p=.15, respectively). Parent ratings of ADHD optimally identified globally impaired children and were superior to teacher-only ratings (χ 2 =4.53, p=.03), but not the or rule (χ 2 =1.54, p=.21) or and rule (χ 2 =1.56, p=.21). Academic Achievement The omnibus test for academic achievement was not significant (χ 2 =9.18, p=.28), suggesting that the four

6 Table 3 Accuracy of classification of impaired children of four methods of diagnosing ADHD a FP Rate FN Rate Sensitivity Specificity FP Rate FN Rate Sensitivity Specificity Diagnostic Method CGAS Parent Dishion Parent Teacher Or Rule And Rule Diagnostic Method WIAT-II Teacher Dishion Parent Teacher Or Rule' And Rule Note. Parent = Parent DBD only; Teacher = Teacher DBD only; Or Rule = Combined parent and teacher data using the or rule ; And Rule = Combined parent and teacher data using the and rule ; FP = False Positive; FN = False Negative; CGAS = Children s Global Assessment Scale; WIAT-II = Wechsler Individual Achievement Test 2nd Edition; Dishion = Dishion Social Preference Scale a Data are presented as means across the ten imputations. algorithms were comparable in identifying academically impaired youth. Similarly, using SUR, none of the four ADHD diagnostic algorithms were significantly associated with the WIAT-II Word Reading (parent-only: B=.00, p=.79; teacher only: B=.00, p=.71; or rule : B=.00, p=.52; and rule : B=.00, p=.77) or Math Reasoning (parent-only: B=.00, p=.73; teacher only: B=.00, p=.56; or rule : B=.00, p=.26; and rule : B=.00, p=.47). 7.54, p=.08), the omnibus test for teacher-rated negative social preference was significant (χ 2 =12.85, p=.02). Specifically, teacher ratings of ADHD significantly outperformed the and rule (B=.10, p<.01) for teacher rated Negative Social Preference Although the omnibus test for parent-rated negative social preference was only marginally significant (χ 2 = Table 4 Receiver Operating Characteristic Comparisons of ADHD Evaluation Methods Comparison CGAS Dishion-Teacher B B Parent vs. Teacher 0.140* Parent vs. Or Rule Parent vs. And Rule 0.137* Teacher vs. Or Rule 0.103* Teacher vs. And Rule * Or Rule vs. And Rule 0.100* Note. Parent = Parent DBD only; Teacher = Teacher DBD only; Or Rule = Combined parent and teacher DBD data using or rule ; And Rule = Combined parent and teacher DBD data using and rule ; CGAS = Children s Global Assessment Scale; Dishion = Dishion Social Preference Scale. The omnibus tests for academic achievement and parent-rated negative social preference were not statistically significant. *p<.05 Fig. 1 Receiver Operating Characteristic (ROC) Ouput for CGAS Imputations. Note. Parent = Parent DBD only; Teacher = Teacher DBD only; Or_Rule = Combined parent and teacher data using the or rule ; And_Rule = Combined parent and teacher data using the and rule. The and rule is the reference point for this set of ROC curves

7 negative social preference. However, there was no difference between teacher only ratings and parent only ratings (B=.12, p=.02), or teacher only ratings and the or rule (B=.04, p=.23). SUR analyses yielded no significant associations between ADHD diagnostic algorithms and parent- (parent-only: B=.00, p=.36; teacher only: B=.00, p=.81; or rule : B=.00, p=.66; and rule : B=.00, p=.78) or teacher-rated social preference (parent-only: B=.00, p=.91; teacher only: B=.00, p=.54; or rule : B=.00, p=.86; and rule : B=.00, p=.78). Discussion Given the importance of accurate diagnostic procedures to identifying children with ADHD, we empirically evaluated four common methods to assess ADHD symptoms: (1) parent report only; (2) teacher report only; (3) parent or teacher ( or rule ), and (4) parent and teacher ( and rule ). Using a large and ethnically diverse sample of 5- to 10-year-old children with (n=121) and without (n= 111) ADHD, we utilized receiver operating characteristics (ROC) and seemingly unrelated regressions (SUR) to assess how well these four methods related to multiinformant measures of functional impairment (i.e., global impairment, social preference, academic achievement). For categorical measures, the parent only and the or rule were each superior to the and rule and teacher only for identifying global impairment whereas the parent only and or rule were comparable. SUR analyses of dimensional measures identified parent only as the optimal method for evaluating global impairment. For teacher-rated negative social preference, the teacher only approach was comparable to parent only ratings and the or rule, but outperformed the and rule. Overall, the results were consistent with previous results of discrepant adult ratings of child behavior. Namely, perhaps because parents interact with and observe their children across a wider range of settings and contexts, they may be better positioned to assess overall functioning (e.g., social, academic, mood, behavioral) than teachers. This formulation is consistent with recent evidence suggesting that parent ratings of ADHD may reflect the fact that parents receive more information from teachers than what teachers obtain from parents (e.g., information about the child s behavior at home) (Murray et al. 2007). Moreover, although parents are reliable observers of their children s peer interactions (Billman and McDevitt 1980), they may not have many opportunities to observe actual peer interactions. On the other hand, given that teachers routinely observe children s peer interactions, they may be more attuned to important patterns in peer relationships, which include initiating positive and negative social interactions (Gresham 1982). For instance, although not specifically related to social preference, a multi-informant study of antisocial behavior (ASB) in boys reported that teachers were more likely to focus on a relatively narrow range of negative behavior (e.g., physical aggression) compared to parents. In addition, teacher ratings were superior to parent ratings in predicting future delinquency and arrest (Bank et al. 1993). Moreover, a recent observational study of disruptive behavior in 327 preschoolers provided the first laboratory-based evidence of contextual influences on informant discrepancies (De Los Reyes et al. 2009). Preschool children were observed in three settings using the Disruptive Behavior Diagnostic Observation Schedule: Parent, Examiner Engaged (the examiner is actively engaged with the child), and Examiner Busy (the examiner is busy with other work and not engaged with the child) (Wakschlag et al. 2008). Results suggested that disruptive behaviors specific to parent and examiner contexts were uniquely associated with parent- and teacher-identified disruptive behavior, respectively (De Los Reyes et al. 2009). Taken together, informant discrepancies may be influenced not only by child characteristics, but also the context in which behaviors are observed (De Los Reyes 2011). In addition, our data suggest there may be specific patterns of association between ADHD and functional outcome based on informant. Therefore, clinicians may find utility in probing different informants when evaluating a child s functioning in a particular domain (e.g., obtaining teacher ratings to evaluate social functioning). An ongoing challenge in clinical assessment involves evaluating functional impairment, despite the absence of universally-accepted criterion. For example, two separate prospective studies of children with ADHD proposed criteria for positive adjustment, defined as surpassing thresholds in at least four of five aprioridomains (i.e., psychopathology, academic achievement, peer relationships, social skills) (Owens et al. 2009; Lee et al. 2008). We analyzed the utility of parent and teacher ratings of ADHD with respect to identifying functional impairment defined using methods inspired by standards in treatment outcome research (Jacobson and Truax 1991). However, future studies must establish methods for determining clinically significant outcomes (e.g., impairment, recovery/resilience). Specifically, developmentally-informed standards, using normative data and more dimensional measures (Helzer et al. 2008) are necessary to standardize definitions and procedures. Barkley (2011) provided a normed measure of functional impairment in adult psychopathology and a similar method for children is needed. As cogently argued by Andreasen et al. (2005) in the context of consensus-defined standards for schizophrenia assessment, we also contend that a standard

8 method for ADHD assessment must be able to efficiently predict comorbidity, socio-emotional functioning, and academic achievement. To our knowledge, this is the first study to empirically evaluate the association of four common ADHD assessment methods with differentiated measures of functional impairment. Although multi-informant data are considered the gold standard (Renk 2005), less guidance is available on how those data should be optimally combined (e.g., improving accuracy). However, despite the appeal of a single diagnostic algorithm for ADHD, its assessment is likely to betray the fact that multiple algorithms may be necessary to accurately capture the array of impairments associated with early ADHD (i.e., multifinality). Future research must prioritize issues beyond informants per se. For example, all ADHD symptoms are equally weighted with respect to diagnosis both within and across informants. However, this approach may misrepresent different psychometric properties of individual ADHD items, including positive and negative predictive power, as well as their association with different outcomes (Power et al. 2001). Only recently have empirical tests assessed which items provide the greatest discrimination between ADHD probands and controls. For example, item response theory (IRT) evaluates how informative and discriminating items are in reflecting individual differences in a latent trait. IRT may therefore improve the discriminatory power of ADHD symptoms by identifying which items are the most or least informative, including across informants. Gomez (2008) utilized IRT to evaluate parent and teacher ratings from the DSM-IV ADHD Rating Scale among 1,475 children. Results suggested that certain inattention symptoms (e.g., often fails to give close attention to details or makes careless mistakes in schoolwork, work, or other activities ) optimally discriminated the trait for parents and teachers whereas different hyperactivity symptoms discriminated the trait for parents (e.g., often runs about or climbs excessively in situations in which it is inappropriate ) versus teachers (e.g., often blurts out answers before questions have been completed ). Thus, beyond the validity of informants for ADHD per se, future research must consider that the informativeness of ADHD symptoms may vary by informant. We note several important limitations of this study. First, the associations reported were based on cross-sectional data. Clinical assessment of ADHD will benefit from methods that demonstrate superior predictive validity, thus necessitating prospective studies of outcomes. Second, although we utilized state-of-the-art methods to impute missing data, the imputed data may not be representative of actual clinical data in our sample and in the population overall. Third, although our determination of functional impairment thresholds for the CGAS and parent/teacher Dishion was based on the midpoint of the ADHD and control group s means in the ROC analyses, they did not incorporate the standard deviation (SD). Nevertheless, there was no significant difference in the SD between the ADHD and control groups for the CGAS (F 2,230 =1.08, p=.34), parent (F 2,230 =1.11, p=.33), or teacher Dishion (F 2,230 = 1.06, p=.35). This limitation may also have contributed to the divergent results between ROC and SUR analyses, particularly for teacher-rated social preference. Given the case control design of our study, dichotomizing impairment criteria may have accentuated the outcome measure differences between the ADHD and control group. However, dichotomization of impairment criteria in our study reflects the emphasis on clinical significance in clinical practice (Kazdin 1999) and therefore enhances the findings clinical utility. Fourth, ROC analyses do not permit statistical control of potential confounds, including comorbidity (e.g., ODD) which increment predictions of negative outcomes beyond ADHD (Hinshaw et al. 1993). However, we included dimensional outcomes which did permit statistical control of sex and ODD. Fifth, although the DBD rating scale contains identical language to DSM- IV, other rating scales (e.g., CRS) may yield different results. Sixth, parent ratings were used extensively in this study, both with the DISC that was used to ascertain ADHD (and served as the basis of our calculation of thresholds from ADHD and controls) and in ratings of social impairment, thus potentially inflating associations because of shared method variance (Campbell and Fiske 1959). Finally, it is important to note that several ROC coefficients, although significant, were relatively modest. We recognize that several potential factors that were unaccounted for in the ROC analyses, most notably comorbid DBD but other demographic and environmental factors as well, are known to affect children s functioning (e.g., Froehlich et al. 2011). Nevertheless, the current study is an important first step in detailing the impact of gathering ADHD diagnostic information from multiple informants, as well as different ways of combining their symptom ratings. The current study evaluated the validity of four different methods to assess ADHD based on measures of functional impairment. Whereas parent ratings of ADHD optimally identified children who exhibited global impairment using ROC and SUR, teacher ratings optimally predicted peer difficulties using ROC. We conclude that although multiinformant approaches to assessing ADHD are valuable, no single method of combining the ratings was definitively superior in identifying functionally impaired children across multiple domains. Future research must continue to use clinically significant criteria to evaluate which informants for ADHD are optimal, how data should be integrated, and how individual items should be weighted.

9 Acknowledgements We also thank the families for participation in the study and the research staff for their help with data collection and management. This work was supported by NIH grant 1R03AA to Steve S. Lee. This work was also supported by The Paul and Daisy Soros Fellowship for New Americans to the first author. References Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), Achenbach, T. M., & Rescorla, L. A. (2001). Manual for the ASEBA school-age forms & profiles. Burlington: University of Vermont, Research Center for Children, Youth, and Families. Achenbach, T. M. (2006). As others see us. Current Directions in Psychological Science, 15(2), Andreasen, N. C., Carpenter, W. T., Kane, J. M., Lasser, R. A., Marder, S. R., & Weinberger, D. R. (2005). Remission in schizophrenia: proposed criteria and rationale for consensus. The American Journal of Psychiatry, 162(3), Bank, L., Duncan, T., Patterson, G. R., & Reid, J. (1993). Parent and teacher ratings in the assessment and prediction of antisocial and delinquent behaviors. Journal of Personality, 61(4), Barkley, R. A. (2011). Barkley functional impairment scale (BFIS). New York: Guilford. Biederman, J., Milberger, S., Faraone, S. V., Kiely, K., Guite, J., Mick, E., et al. (1995). Impact of adversity on functioning and comorbidity in children with attention-deficit hyperactivity disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 34(11), Biederman, J., Petty, C. R., Evans, M., Small, J., & Faraone, S. V. (2010). How persistent is ADHD? A controlled 10-year followup study of boys with ADHD. Psychiatry Research, 177(3), Billman, J., & McDevitt, S. C. (1980). Convergence of parent and observer ratings of temperament with observations of peer interaction in nursery school. Child Development, 51(2), Bird, H. R., Canino, G., Rubio-Stipec, M., & Ribera, J. C. (1987). Further measures of the psychometric properties of the Children s Global Assessment Scale. Archives of General Psychiatry, 44(9), Bussing, R., Fernandez, M., Harwood, M., Hou, W., Garvan, C. W., Eyberg, S. M., et al. (2008). Parent and teacher SNAP-IV ratings of attention deficit hyperactivity disorder symptoms: Psychometric properties and normative ratings from a school district sample. Assessment, 15(3), Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, Conners, C. M. (1997). Conners rating scales revised. New York: Multi Health Systems. De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131(4), De Los Reyes, A., Henry, D. B., Tolan, P. H., & Wakschlag, L. S. (2009). Linking informant discrepancies to observed variations in young children s disruptive behavior. Journal of Abnormal Child Psychology, 37(5), De Los Reyes, A. (2011). Introduction to the special section: More than measurement error: Discovering meaning behind informant discrepancies in clinical assessments of children and adolescents. Journal of Clinical Child and Adolescent Psychology, 40(1), 1 9. Dishion, T. J. (1990). The family ecology of boys peer relations in middle childhood. Child Development, 61(3), Froehlich, T. E., Anixt, J. S., Loe, I. M., Chirdkiatgumchai, V., Kuan, L., & Gilman, R. C. (2011). Update on environmental risk factors for attention-deficit/hyperactivity disorder. Current Psychiatry Reports, 13(5), Geary, D. C., Hoard, M. K., Byrd-Craven, J., Nugent, L., & Numtee, C. (2007). Cognitive mechanisms underlying achievement deficits in children with mathematical learning disability. Child Development, 78(4), Gomez, R. (2008). Item response theory analyses of the parent and teacher ratings of the DSM-IV ADHD rating scale. Journal of Abnormal Child Psychology, 36(6), Gresham, F. M. (1982). Social interactions as predictors of children s likability and friendship patterns: A multiple regression analysis. Journal of Behavioral Assessment, 4(1), Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), Helzer, J. E., Kraemer, H. C., Krueger, R. F., Wittchen, H.-U., Sirovatka, P. J., & Regier, D. A. (Eds.). (2008). Dimensional approaches in diagnostic classification: refining the research agenda for DSM-V. Washington, DC: American Psychiatric Association. Hinshaw, S. P., Lahey, B. B., & Hart, E. L. (1993). Issues of taxonomy and comorbidity in the development of conduct disorder. Development and Psychopathology, 5, Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), Johnston, C., & Murray, C. (2003). Incremental validity in the psychological assessment of children and adolescents. Psychological Assessment, 15(4), Kaufman, J., Birmaher, B., Brent, D., Rao, U., Flynn, C., Moreci, P., et al. (1997). Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime Version (K- SADS-PL): Initial reliability and validity data. Journal of the American Academy of Child and Adolescent Psychiatry, 36(7), Kaufman, J., Birmaher, B., Brent, D. A., Ryan, N. D., & Rao, U. (2000). K-SADS-PL. Journal of the American Academy of Child and Adolescent Psychiatry, 39(10), Kazdin, A. E. (1999). The meanings and measurement of clinical significance. Journal of Consulting and Clinical Psychology, 67 (3), Kolko, D. J., & Kazdin, A. E. (1993). Emotional/behavioral problems in clinic and nonclinic children: Correspondence among child, parent and teacher reports. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 34(6), Lahey, B. B., Applegate, B., McBurnett, K., Biederman, J., Greenhill, L., Hynd, G. W., et al. (1994). DSM-IV field trials for attention deficit hyperactivity disorder in children and adolescents. The American Journal of Psychiatry, 151(11), Lahey, B. B., Pelham, W. E., Stein, M. A., Loney, J., Trapani, C., Nugent, K., et al. (1998). Validity of DSM-IV attention-deficit/ hyperactivity disorder for younger children. Journal of the American Academy of Child and Adolescent Psychiatry, 37(7), Lahey, B. B., Pelham, W. E., Loney, J., Kipp, H., Ehrhardt, A., Lee, S. S., et al. (2004). Three-year predictive validity of DSM-IV attention deficit hyperactivity disorder in children diagnosed at 4 6 years of age. The American Journal of Psychiatry, 161(11),

10 Lee, S. S., & Hinshaw, S. P. (2006). Predictors of adolescent functioning in girls with attention deficit hyperactivity disorder (ADHD): The role of childhood ADHD, conduct problems, and peer status. Journal of Clinical Child and Adolescent Psychology, 35(3), Lee, S. S., Lahey, B. B., Owens, E. B., & Hinshaw, S. P. (2008). Few preschool boys and girls with ADHD are well-adjusted during adolescence. Journal of Abnormal Child Psychology, 36(3), Lee, S. S., Humphreys, K. L., Flory, K., Liu, R., & Glass, K. (2011). Prospective association of childhood attention-deficit/hyperactivity disorder (ADHD) and substance use and abuse/dependence: A meta-analytic review. Clinical Psychology Review, 31(3), Lee, S. S., & Humphreys, K. L. (in press). Assessment of ADHD in young children. In B. Hoza & S. W. Evans (Eds.), Treating Attention Deficit Hyperactivity Disorder. Kingston, NJ: Civic Research Institute. Loeber, R., Green, S. M., & Lahey, B. B. (1990). Mental health professionals perception of the utility of children, mothers, and teachers as informants on childhood psychopathology. Journal of Clinical Child Psychology, 19(2), 136. Massetti, G., Pelham, W. E., & Gnagy, E. M. (2005). Situational variability of ADHD, ODD, and CD: Psychometric properties of the DBD interview and rating scale for an ADHD sample. Presented at the International Society for Research in Child and Adolescent Psychopathology. New York, NY Murray, D. W., Kollins, S. H., Hardy, K. K., Abikoff, H. B., Swanson, J. M., Cunningham, C., et al. (2007). Parent versus teacher ratings of attention-deficit/hyperactivity disorder symptoms in the Preschoolers with Attention-Deficit/Hyperactivity Disorder Treatment Study (PATS). Journal of Child and Adolescent Psychopharmacology, 17(5), Newcorn, J. H., Halperin, J. M., Jensen, P. S., Abikoff, H. B., Arnold, L. E., Cantwell, D. P., et al. (2001). Symptom profiles in children with ADHD: effects of comorbidity and gender. JournaloftheAmerican Academy of Child and Adolescent Psychiatry, 40(2), Owens, E. B., Hinshaw, S. P., Lee, S. S., & Lahey, B. B. (2009). Few girls with childhood attention-deficit/hyperactivity disorder show positive adjustment during adolescence. Journal of Clinical Child and Adolescent Psychology, 38, Pelham, W. E., Gnagy, E. M., Greenslade, K. E., & Milich, R. (1992). Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 31(2), Pelham, W. E., Fabiano, G. A., & Massetti, G. M. (2005). Evidencebased assessment of attention deficit hyperactivity disorder in children and adolescents. Journal of Clinical Child and Adolescent Psychology, 34(3), Piacentini, J. C., Cohen, P., & Cohen, J. (1992). Combining discrepant diagnostic information from multiple sources: Are complex algorithms better than simple ones? Journal of Abnormal Child Psychology, 20(1), Polanczyk, G., de Lima, M. S., Horta, B. L., Biederman, J., & Rohde, L. A. (2007). The worldwide prevalence of ADHD: a systematic review and metaregression analysis. The American Journal of Psychiatry, 164(6), Power, T. J., Costigan, T. E., Leff, S. S., Eiraldi, R. B., & Landau, S. (2001). Assessing ADHD across settings: Contributions of behavioral assessment to categorical decision making. Journal of Clinical Child Psychology, 30(3), Renk, K. (2005). Cross-informant ratings of the behavior of children and adolescents: The gold standard. Journal of Child and Family Studies, 14(4), Schwab-Stone, M. E., Shaffer, D., Dulcan, M. K., Jensen, P. S., Fisher, P., Bird, H. R., et al. (1996). Criterion validity of the NIMH Diagnostic Interview Schedule for Children Version 2.3 (DISC- 2.3). Journal of the American Academy of Child and Adolescent Psychiatry, 35(7), Setterberg, S., Bird, H., & Gould, M. (1992). Parent and interviewer version of the Children s Global Assessment Scale. New York: Columbia University. Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, H., et al. (1983). A Children s Global Assessment Scale (CGAS). Archives of General Psychiatry, 40(11), Shaffer, D., Fisher, P., Lucas, C. P., Dulcan, M. K., & Schwab-Stone, M. E. (2000). NIMH Diagnostic Interview Schedule for Children Version IV (NIMH DISC-IV): Description, differences from previous versions, and reliability of some common diagnoses. Journal of the American Academy of Child and Adolescent Psychiatry, 39(1), Swanson, J. M., Kraemer, H. C., Hinshaw, S. P., Arnold, L. E., Conners, C. K., Abikoff, H. B., et al. (2001). Clinical relevance of the primary findings of the MTA: Success rates based on severity of ADHD and ODD symptoms at the end of treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 40(2), Valo, S., & Tannock, R. (2010). Diagnostic instability of DSM-IV ADHD subtypes: Effects of informant source, instrumentation, and methods for combining symptom reports. Journal of Clinical Child and Adolescent Psychology, 39(6), Wakschlag, L. S., Hill, C., Carter, A. S., Danis, B., Egger, H. L., Keenan, K., et al. (2008). Observational assessment of preschool disruptive behavior, part I: reliability of the Disruptive Behavior Diagnostic Observation Schedule (DB-DOS). Journal of the American Academy of Child and Adolescent Psychiatry, 47(6), Wechsler, D. (2002). Wechsler individual achievement test (2nd ed.). San Antonio: Psychological Corporation. Yuan, Y. C. (2002). Multiple imputation for missing data: Concepts and new development. SUGI27 Conference Proceedings. Zellner, A. (1962). An Efficient Method of Estimating Seemingly Unrelated Regressions and Tests for Aggregation Bias. Journal of the American Statistical Association, 57(298),