Development and Reliability of a Pictorial Mental Disorders Screen for Young Adolescents

Transcription

1 Original Research Development and Reliability of a Pictorial Mental Disorders Screen for Young Adolescents Nicole Smolla, PhD 1, Jean-Pierre Valla, MD, MSc 2, Lise Bergeron, PhD 3, Claude Berthiaume, MSc 4, Ma rie St-Georges, MPs 4 Objective: To report psychometric data from preliminary studies of the Adolescent Dominic (AD), a pictorial screen for the most frequent Axis I youth mental s. Methods: We created 113 picture items based on DSM-III-R diagnostic criteria and assessed them for comprehension (sample 1, n = 114; sample 2, n = 40) and reliability (sample 3, n = 128) in a group of adolescents aged 12 to 16 years living in the community. We used the kappa statistic to estimate test retest reliability of symptoms, criteria and diagnoses, and intraclass correlation coefficients (ICCs) for symptom and criterion scores. We assessed internal consistency of symptom scores with the alpha coefficient. Results: For symptoms, 54.4% of kappas were higher than 0.60, while only 2% were poor. ICCs for symptom scores yielded higher values (0.81 to 0.89) than for criterion scores (0.51 to 0.86). Internal consistency of symptom scores ranged from 0.52 to Kappas for diagnoses ranged from 0.52 to Conclusions: Symptom reliability compared favourably with data from other assessment interviews of youth mental s. Following these positive results, a computerized DSM-IV version of the AD has focused on the assessment of symptoms and is currently being tested for reliability and criterion validity. (Can J Psychiatry 2004;49: ) Information on funding and support and author affiliations appears at the end of the article. Clin i cal Im pli ca tions The Adolescent Dominic (AD) is a DSM-based standardized screen with demonstrated reliability among adolescents as young as age 12 years, living in the community. The instrument can serve as a preliminary step in clinical interviewing and a complement to usual clinical practice. The instrument could encourage the expression of adolescents own concerns and thereby help clinicians identify priorities for intervention. Lim i ta tions The low prevalence of mental s in community samples is the main limitation of studies of this type. The AD does not assess all DSM mental s. Cut-off scores with clinical samples have yet to be established, as has standardization with a large sample to provide normative data. Results may not be generalizable to adolescents with physical or cognitive impairments or learning disabilities. Psychometric properties are population-specific and should be interpreted considering the characteristics of the population from which the samples were drawn. Key Words: ad o les cents, pic to rial as sess ment, men tal dis or ders, re li abil ity, com mu nity sam ples 828 W Can J Psy chi a try, Vol 49, No 12, December 2004

2 Development and Reliability of a Pictorial Mental Disorders Screen for Young Adolescents It is widely agreed that young sters should be as sessed di - rectly re gard ing their men tal health, be cause other in for - mants re ports can not re place self-reports (1). Adults tend to pay at ten tion to externalizing be hav iour prob lems, whereas chil dren and ad o les cents are better at iden ti fy ing their in ter - nal iz ing dis or ders or be hav iours that they may man i fest with - out their par ents knowl edge (1 14). To this end, highly so phis ti cated, com pre hen sive, DSM-based struc tured di ag - nos tic in ter views for youth have been de vel oped over the past 2 de cades. How ever, their very level of so phis ti ca tion makes them dif fi cult to use in real-world con di tions, and it is un likely that ser vice pro vid ers in pri mary care will ever use them ex - ten sively. Since stan dard ized as sess ment is the foun da tion of sci ence-based in ter ven tion, this lim ited ap pli ca bil ity of in - stru ments has be come a ma jor con cern (15). Another imped i ment to the adop tion of these instru ments by front-line ser vice pro vid ers is that psychometric stud ies have mostly been con ducted in clin i cal set tings (16). Reli abil ity and valid ity coef fi cients are highly spe cific to a given pop u la - tion. Con se quently, a mea sure that is reli able when used in a het er o ge neous sam ple (for exam ple, a clin i cal sam ple) may be much less reli able in a more homo ge neous one (for exam ple, a com mu nity sam ple) (17). Psychometric stud ies in com mu nity sam ples have shown that reli abil ity of youth responses to highly struc tured inter views is dif fi cult to achieve (Table 1). For instance, the Diag nos tic Inter view for Chil dren and Adolescents-Revised (DICA-R) and var i ous ver sions of the Diag nos tic Inter view Sched ule for Chil dren-2 (DISC-2) pro - vide sub stan tial reli abil ity for con duct dis or der but less reli - abil ity for other behav iour dis or ders and only mod er ate reli abil ity for depres sive dis or ders; they are prob lem atic for anx i ety dis or ders (16,18 21). In addi tion, because all ado les - cents in these stud ies were grouped together for data anal y ses and results were not reported accord ing to age level or any age group ings that would have revealed age dif fer ences in reli - abil ity, reported reli abil ity coef fi cients may be over es ti mated for youn ger par tic i pants (that is, those aged 9 to 14 years) and under es ti mated for older ones (that is, those aged 15 to 18 years). Age dif fer ences in the reli abil ity of child inter views have not been thor oughly explored (18,22 23). A reli abil ity study of the DISC symp tom scores found that results yielded by highly Table 1 Test-retest reliability studies of DICA-R and DISC-2 diagnoses: community child or adolescent informants Diagnoses Boyle and oth ers (18) 12 to 16 years (n = 137) DICA-R Jensen and oth ers (16) 9 to 17 years (n = 278) DISC-2.1 Ribera and oth ers (19) 9 to 17 years (n = 124) DISC-2.1 Schwab-Stone and oth ers (20) 9 to 18 years (n = 247) DISC-2.3 Breton and oth ers (21) 12 to 14 years (n = 145) DISC-2.25 κ κ κ κ (SE) κ (SE) Disruptive 0.38 not avail able 0.52 not avail able 0.37 (0.17) Attention-deficit hyperactivity Oppostional defiant (0.06) Fewer than 5 pos i tive cases at test (0.05) Fewer than 5 pos i tive cases at test Conduct (0.06) 0.49 (0.22) Depressive s (0.06) 0.55 (0.16) Major depressive 0.45 not avail able (0.06) not avail able Dysthymia (0.06) not avail able Anxiety s not available (0.06) 0.49 (0.11) Separation anxiety Overanxious and generalized anxiety 0.00 not avail able (0.06) 0.59 (0.19) 0.54/ not avail able 0.03/ (0.05)/ 0.53 (0.19) Simple phobia not avail able not avail able (0.06) 0.55 (0.12) κ (SE) = kappa value (standard error) DICA-R = Diagnostic Interview for Children and Adolescents-Revised DISC-2 = Diagnostic Interview Schedule for Children-2 = no data collected Can J Psy chi a try, Vol 49, No 12, December 2004 W 829

3 The Ca na dian Jour nal of Psy chia try Orig i nal Re search Figure 1 Do you find it difficult to wait for your turn, like Dominic? (boys version) struc tured inter views with clin i cally referred chil dren under age 10 years should be inter preted cau tiously (22). If a cri te - rion of 0.70 is used for test retest reli abil ity, coef fi cients are mod er ate for chil dren aged 10 to 13 years, espe cially in regard to depres sion (0.53) and anx i ety dis or ders (0.54). Test retest intraclass cor re la tion coef fi cients (ICCs) aver aged 0.60 for chil dren aged 10 to 13 years and 0.71 for ado les cents aged 14 to 18 years. The age issue is par tic u larly rel e vant, since ade - quate reli abil ity is a min i mal standard for an assess ment method and should usu ally be tested prior to eval u at ing validity (23). Sev eral fac tors may cause unre li abil ity (17). Among these is infor ma tion vari ance, and research ers have repeat edly tried to improve the infor ma tion-gath er ing phase of diag no sis. This vari ance reflects phe nom ena such as bad phras ing of ques - tions and record ing of responses and respon dents mis un der - stand ings, lapses of con cen tra tion, and inten tional resis tance. For instance, it was found that very few chil dren aged 9 to 11 years under stood DISC ques tions involv ing the time at which symp toms occurred (24). Apart from the issue of time con - cepts, only 16% of chil dren aged 9 years under stood ques tions assess ing depres sive diag no ses; this only increased to 31% of those aged 11 years. The ories of cog ni tive devel op ment may explain some of the unre li able data obtained when young sters are given struc tured inter views (25 26). The devel op ment of higher-level think ing skills in ado les cence depends highly on cul tural, social, and indi vid ual fac tors. Cog ni tive skills typ i cal of the con crete oper a tional stage may extend beyond the ages of 10 to 12 years (27). For many young ado les cents, mis - un der stand ing of abstract con cepts could be less ened by the use of more con crete rep re sen ta tions. To assess symp toms of men tal dis or ders in ado les cents aged 12 to 16 years, and keep - ing in mind the lim i ta tions of exist ing instru ments, we devel - oped a pic ture-based screen that would pro vide con crete rep re sen ta tions (28) of abstract DSM con structs. Infor ma tionpro cess ing the o ries sug gest that com bin ing visual and audi - tory stim uli allows for better con cep tual under stand ing (26,29 37), so we inte grated these sen sory modal i ties. Such inte gra tion has already been suc cess ful with school-age chil - dren (38 41). This paper describes the devel op men tal phase of the Ado les cent Dominic (AD). Methods The devel op men tal phase of the AD involved 3 stages. Stage I con sisted of the cre ation of pic tures cor re spond ing to DSM-III-R diag nos tic cri te ria (42). Stage II ver i fied whether par tic i pants ade quately under stood the con tent con veyed by the pic tures; if they did not, we edited and redrafted the pic - tures. In Stage III, we eval u ated test retest reli abil ity of the pic to rial inter view. We obtained institutional review board approved paren tal autho ri za tion and assent forms for every par tic i pant. Stage I: Creation of the Pictures Var i ous char ac ters (for exam ple, Dominic and his or her par - ents, teacher, and peers) were cre ated and shown to a small group of French-speak ing ado les cents drawn from the com - mu nity ( n = 17). Their com ments helped us to select or mod ify these char ac ters. We drafted 180 pic tures based on com pe - tency sit u a tions and on DSM-III-R diagnostic cri te ria for atten tion-def i cit hyper ac tiv ity dis or der (ADHD), oppositional defi ant dis or der (ODD), con duct dis or der (CD), major depres sive dis or der (MDD), over anx ious dis or der (OAD), sep a ra tion anx i ety dis or der (SAD), sim ple pho bia (SPH), and sub stance use (Fig ures 1 to 4). We excluded 3 SAD cri te ria (A3, A4, and A9) that apply more to youn ger chil dren. The main char ac ter, fam ily mem bers, and peers were sex-spe cific, result ing in a boy and a girl ver sion. Seven sit u a tions were slightly adapted for sex. We retained a sub set of 113 pic tures that corresponded closely to DSM-III-R symp tom descrip - tions (See the Dis cus sion sec tion below for the use of DSM-III-R rather than DSM-IV). Stage II: Comprehension Checks We tested these 113 pic tures for com pre hen sion in a sam ple of ado les cents aged 12 to 16 years (n = 114), bal anced for age and sex. We drew this sam ple from 13 French public high schools located in var i ous socio eco nomic regions of the Mon - treal urban area. Six schools were located in lower-mid dleclass areas, 4 in under priv i leged areas, and 3 in mid dle-class areas. All schools fol lowed the reg u lar aca demic cur ric u lum; we did not solicit young sters enrolled in spe cial classes (for exam ple, recent immi grants, those with learn ing dis abil i ties, and those with phys i cal or cog ni tive impair ment). We ran domly divided pic tures illus trat ing sex-specific ver - sions into 4 book lets, result ing in 8 dif fer ent book lets. We 830 W Can J Psy chi a try, Vol 49, No 12, December 2004

4 Development and Reliability of a Pictorial Mental Disorders Screen for Young Adolescents Figure 2 Are you very scared of bugs, like Dominic? (girls version) Figure 3 Do you feel good at school, like Dominic? (boys version) boost pic ture com pre hen sion. The inter viewer tran scribed all respon dents answers ver ba tim. For every respon dent, 2 judges (a child psy chi a trist and a child psy chol o gist) inde pend ently assessed the tran scribed responses and decided whether the 113 pic tures were under - stood, more or less under stood, not under stood, or miss ing. If one judge failed to assess a given pic ture and respon dent, the sec ond judge s qual i fi ca tion applied. A pic - ture was scored 1.0 when both judges con sid ered it under - stood by a given respon dent, 0.5 when only one judge con sid ered it under stood, and 0 for all other sit u a tions. We cal cu lated a com pre hen sion rate per pic ture ( CRP) by aver ag - ing all scores (for exam ple, 1.0, 0.5, and 0) attrib uted to a pic - ture. A CRP of 0.60 was selected as the cri te rion for qual i fy ing a pic ture as under stood (CRP 0.60). Pic tures with a CRP < 0.60 were edited or redrafted. We per formed a sec ond com - pre hen sion check on the edited and redrafted pic tures, fol low - ing the same pro ce dures with another sam ple (n = 40). We recruited at least 4 boys and 4 girls for each age, except for boys aged 15 years (n = 3). Again, pic tures not under stood (CRP < 0.60) were edited or redrafted. Stage III: Reliability Study (Test Retest Stability and Internal Consistency) Par tic i pants. Another sam ple of ado les cents aged 12 to 16 years (n = 128), bal anced for age but not for sex (70 girls and 58 boys), was drawn accord ing to the same recruit ment pro ce - dures and selec tion cri te ria. Two lay inter view ers con ducted the test and retest inter views in a coun ter bal anced design. Retest took place within an inter val of 7 to 13 days (mean inter val 9 days). The average dura tion of each test was approx i mately half an hour. Descrip tion of the Instru ment and Pro ce dures. One hun dred and two pic tures with stood the com pre hen sion checks and were exam ined for test retest reli abil ity. These 102 pic tures were orga nized as fol lows: 94 pic tures assessed 101 symp - toms of DSM-III-R criteria (ADHD, 16 pic tures; ODD, 13 pic tures; CD, 15 pic tures; MDD, 20 pic tures; OAD, 14 pic - tures; SAD, 8 pic tures; SPH, 10 pic tures; and sub stance use, 5 pic tures). The remain ing 8 pic tures described com pe tency sit - u a tions and nor mal behav iour. pre sented each sex-specific book let to a subsample of 14 to 17 ado les cents. All subsamples were bal anced for age. At this stage, the inter viewer asked every par tic i pant the ques tion, Could you tell me what is going on in this pic ture? with out sup ply ing any ver bal cue, not even a ver bal query that would In sev eral instances, 2, 3, or 4 pic tures illus trated symp toms per tain ing to a sin gle diag nos tic cri te rion. For exam ple, 3 pic - tures rep re sented depressed or irri ta ble mood in MDD. Conversely, 7 pic tures assessed symp toms per tain ing to more than 1 men tal dis or der. For exam ple, the pic ture for ODD loss of tem per was also used for MDD irri ta ble mood. Symp toms and com pe tency sit u a tions were ran domly mixed to avoid a halo effect. Ado les cents were not asked to elab o rate on the pic tures, as in pro jec tive tests, but to say whether they acted, thought, or felt Can J Psy chi a try, Vol 49, No 12, December 2004 W 831

5 The Ca na dian Jour nal of Psy chia try Orig i nal Re search Figure 4 Do you worry a lot about not having friends, like Dominic? (girls version) cri te rion A3 (weight gain or loss), cooccurrence with depressed mood or loss of inter est or plea sure was checked. Only a pos i tive response to the pre ced ing ques tion gen er ated fur ther subquestions. For SPH (10 pic tures), a pos i tive response to the symp tom query trig gered a ques tion about symp tom occur rence dur ing the past 6 months. A fur ther pos i tive response then gen er ated a group of 5 subquestions assess ing per sis tence of fear, invari - abil ity and imme di acy of the anx i ety response, stim u lus avoid ance, inter fer ence with usual social activ i ties, and rec - og ni tion that fear was exces sive. Con se quently, assess ment of SPH yielded 50 cri te ria for anal y sis. Finally, we did not assess all DSM-III-R cri te ria for sub stance use (5 pic tures): for alco - hol and tobacco, we assessed life time prev a lence and cur rent use; for drug con sump tion, we assessed life time prev a lence only. like the main char ac ter ( Dominic ). More spe cif i cally, a sim - ple ver bal ques tion refer ring to the symp tom or symptom query accom pa nied every pic ture. We lim ited symp tom que - ries to a sin gle con cept and used eas ily under stood words. Sen tence length rarely exceeded 12 words. Symp tom que ries were admin is tered in the same struc tured for mat to all par tic i - pants. The inter viewer read the ques tion to the ado les cent while she or he was look ing at the pic ture. A pos i tive response to the symp tom query (for exam ple, Do you have night - mares, like Dominic? ) trig gered a subquestion assess ing symp tom fre quency (based on an event hav ing occurred 6 months prior to the inter view) dur ing the past 6 months. For exam ple, the subquestion might be Since [the event which occurred 6 months ago], have you had fre quent night mares, like Dominic? Responses to symp tom que ries and subquestions were coded 0 (no) or 1 (yes). Assess ment of sever ity was restricted to this con tin gent ques tion on symptom fre - quency for ADHD, ODD, CD, OAD, and SAD. For MDD (20 pic tures), a pos i tive answer to the subquestion assess ing symp tom fre quency gen er ated addi tional ques tions per tain ing to DSM-III-R diag nos tic cri te ria. For 6 pictures assess ing cri te ria A1 (depressed mood) and A2 (loss of inter - est or plea sure), symp tom dura tion (that is, for 2 weeks or more) and daily occur rence were checked. For 12 pictures assess ing cri te ria A4 to A9, dura tion, daily occur rence, and cooccurrence with depressed mood or loss of inter est or plea - sure were checked. With the remain ing 2 pic tures designed for Statistical Analyses For MDD and SPH, neg a tive responses to symp tom que ries or to subquestions on symp tom occur rence resulted in miss ing data for other sever ity subquestions. To keep the num ber of obser va tions con stant, such miss ing data were recoded as implied neg a tive responses (43), that is, absence of such symp toms. Symp toms (101 dichot o mous vari ables) were defined by 0 or 1 responses to symptom que ries. Symp tom scores (8 con tin u ous vari ables) were obtained by sum ming the 0 or 1 responses for every symp tom query accord ing to diag nos tic group ings (ADHD, ODD, CD, MDD, OAD, SAD, SPH, and sub stance use). Criteria (108 dichot o mous vari - ables) were com puted from 0 or 1 responses to subquestions. Because we used more than one pic ture to assess a few symp - toms, we com bined responses, using the or rule. We obtained cri te rion scores (7 con tin u ous vari ables) by sum - ming the 0 or 1 cod ing for each cri te rion accord ing to diag nos - tic group ings. We com puted approx i ma tions of diag no ses (7 dichot o mous vari ables) accord ing to DSM-III-R cut-off points and algo rithms. We used the kappa sta tis tic (44,45) to assess tem po ral sta bil ity of dichot o mous vari ables. How ever, obtain ing accept able reli abil ity in com mu nity sam ples is chal leng ing because of the rel a tively low prev a lence of dis or ders (17). Because accu racy of kappa is sen si tive to very low or very high prev a lence, we required at least 5 pos i tive and 5 neg a tive responses at test for its cal cu la tion. Also, no cal cu la tion was per formed in the absence of positive or neg a tive cases at retest. We used Fleiss s cri te ria ( κ < 0.40, poor reli abil ity; 0.40 # κ < 0.60, fair reli abil ity; 0.60# κ < 0.75, good reli abil ity; κ $ 0.75, excel lent reli abil ity; 45) to des ig nate the strength of asso ci a tion. We used the ICC for the reli abil ity of symp tom cri te rion scores over time (46). Pre lim i nary anal y ses indi cated no sig nif i cant sex dif fer ences in reli abil ity, so we based reli abil ity anal y ses 832 W Can J Psy chi a try, Vol 49, No 12, December 2004

6 Development and Reliability of a Pictorial Mental Disorders Screen for Young Adolescents Table 2 Distribution of kappa values calculated on the Adolescent Dominic symptom queries (n = 128). Diagnoses Num ber of symp tom que ries < 5 pos i tive cases at test n κ < 0.40 n 0.40 κ < 0.60 n 0.60 κ < 0.70 n κ 0.70 n Attention-deficit hyperactivity Oppositional defiant 16 (100) 0 (0) 0 (0) 6 (37.5) 5 (31.3) 5 (31.3) 13 (100) 0 (0) 2 (15.3) 4 (30.7) 3 (23.1) 4 (30.7) Conduct 15 (100) 7 (46.6) 0 (0) 2 (13.3) 3 (20) 3 (20) Major depressive Separation anxiety Overanxious 20 (100) 2 (10) 0 (0) 6 (30) 8 (40) 4 (20) 8 (100) 0 (0) 0 (0) 6 (75) 1 (12.5) 1 (12.5) 14 (100) 0 (0) 0 (0) 5 (35.7) 8 (57.1) 1 (7.1) Simple phobia 10 (100) 3 (30) 0 (0) 1 (10) 1 (10) 5 (50) Substance use 5 (100) 1 (20) 0 (0) 1 (20) 0 (0) 3 (60) Total 101 (100) 13 (12.9) 2 (2) 31 (30.7) 29 (28.7) 26 (25.7) κ = kappa val ues; n = num ber of symp tom queries Fleiss s cri te ria: κ < 0.40, poor re li abil ity; 0.40 κ < 0.60, fair re li abil ity; 0.60 κ <.75, good re li abil ity; κ 0.75, ex cel lent re li abil ity. on the total sam ple. We eval u ated inter nal con sis tency of symp tom scores by using Cronbach alpha coef fi cients (47) on responses to the first assess ment (48). Results Comprehension Checks The first com pre hen sion check revealed high mean CRP scores for the set of 113 pic tures for both girls (mean CRP 0.84) and boys (mean CRP 0.86). Thir teen girls pictures (11.3%) and 11 boys pic tures (9.7%) were not under stood accord ing to the selected cri te rion. Fol low ing these results, we edited or redrafted 33 pic tures in the girls ver sion and 26 pic tures in the boys ver sion. Most changes involved mak ing the main char ac ter ( Dominic ) more con spic u ous, ren der ing emo tional expres sions more obvi ous, and chang ing the sex of peers in the pic tures. Irrel e vant visual ele ments were min i - mized. We added 7 new pic tures in the girls ver sion and 5 in the boys ver sion. These 40 girls pic tures and 31 boys pic - tures were submitted to the sec ond com pre hen sion check, which revealed high mean CRP scores for both girls and boys pic tures (mean CRPs 0.85 and 0.88, respectively). Mean CRPs were high for newly added pic tures (7 girls pic - tures, 0.88; 5 boys pic tures, 0.93). Only 5 girls pic tures and 1 boys pic ture were not under stood. Fol low ing these results, we elim i nated 7 pic tures, edited 1, and added 2. Reliability of Symptoms and Symptom Scores Ninety-four pic tures assess ing 101 symp toms of DSM-III-R cri te ria and 8 pic tures describ ing nor mal behav iour were checked for reli abil ity. Because there were fewer than 5 pos i - tive responses at test, we did not cal cu late kappa val ues for 13 out of 101 symp toms (12.9%). Included were a few pic tures with out score vari ance (2 for SPH and 3 for CD). Of the remain ing 88 symp tom que ries, only 2 (2%) were poor (κ < 0.40); 31 yielded kappa val ues between 0.40 and 0.59; 29 yielded kappa val ues between 0.60 and 0.69; and 26 yielded kappa val ues equal to or greater than Table 2 reports the dis tri bu tion of kappa val ues accord ing to diag no sis. Accord - ing to Fleiss s criteria (45), most kappa val ues (55/101, or 54.4%) were good to excel lent (κ $ 0.60). As for reli abil ity of symp tom scores (Table 3), ICCs ranged from 0.81 (OAD) to 0.89 (sub stance use) and were all sig nif i - cant at the P < 0.05 level. Cronbach alpha coef fi cients ranged from 0.52 (sub stance use) to 0.83 (ODD). A low num ber of items and low prev a lence neg a tively affected the alpha coef fi - cients for sub stance use, CD, and SPH. Also, sub stance use was less clearly a one-dimen sional scale. A com par i son of symp tom score ICCs by age revealed no sig - nif i cant dif fer ences (sub jects aged 12 to 14 years, mean ICC 0.84, range 0.72 to 0.96; sub jects aged 15 to 16 years, mean ICC 0.85, range 0.76 to 0.93). With regard to the 7 pic tures adapted for sex, kap pas were com pa ra ble for both sexes. Reliability of Criteria, Criterion Scores, and Diagnosis Approximations Owing to low prev a lence, kap pas could not be cal cu lated for 12 of 58 cri te ria (20.7%). How ever, of the remain ing 46 cri te - ria, 19 yielded kappa values between 0.40 and 0.59, 17 Can J Psy chi a try, Vol 49, No 12, December 2004 W 833

7 The Ca na dian Jour nal of Psy chia try Orig i nal Re search Table 3 Test retest reliability of the Adolescent Dominic (n = 128) Num ber of cases Symp tom scores Cri te rion scores Diagnoses +/+ +/ -/+ -/- n α ICC 95%CI ICC 95%CI κ (SE) 95%CI ADHD a a 0.86 a 0.81 to 0.90 ODD to 0.91 CD to 0.88 MDD to 0.90 SAD to 0.87 OAD to 0.86 SPH to 0.89 Substance use to b 0.68 to to to to to to to a (0.12) 0.69 (0.10) c d 0.52 (0.12) 0.62 (0.07) 0.54 (0.14) 0.53 to to 0.88) 0.28 to to to /+ = positive on test/pos i tive on re test; +/ = positive on test/neg a tive on re test; /+ = negative on test/pos i tive on re test; / = negative on test/neg a tive on re test ADHD = at ten tion-deficit hy per ac tiv ity dis or der; CD = con duct dis or der; MDD = ma jor de pres sive dis or der; SAD = sep a ra tion anx i ety dis or der; SPH = sim ple pho bia; OAD = over anx ious dis or der; ODD = oppositional de fi ant dis or der α = al pha co ef fi cients com puted on symp tom scores at test; ICC = intraclass cor re la tion co ef fi cient; κ (SE) = kappa value (stan dard er ror) a n = 112, ow ing to a mis take by an in ter viewer; b n = 77, ow ing to a mis take by an in ter viewer; c fewer than 5 pos i tive cases at test; d no pos i tive cases at re test = no data collected yielded kappa val ues between 0.60 and 0.69; and 3 yielded kappa val ues equal to or greater than Accord ing to Fleiss s cri te ria, 20 kappa val ues (34.5%) were good to excel - lent (κ $ 0.60), while only 7 (12%) were poor (κ < 0.40). For sim ple pho bia, 26 kap pas out of 50 (52%) could not be cal cu lated. Of the remain ing 24, 8 kappa val ues were between 0.40 and 0.59; 6 were between 0.60 and 0.69; and 7 were equal to or greater than Only 3 were less than Thus, accord ing to Fleiss s cri te ria, 13 kap pas (26%) were good to excel lent ( κ $ 0.60), while only 3 (6%) were poor (κ < 0.40). As for cri te rion scores (Table 3), ICCs ranged from 0.51 for MDD to 0.86 for ODD. Kappa val ues for diag no sis approx i - ma tions (Table 3) ranged from 0.52 for SAD to 0.76 for ADHD; they could not be cal cu lated for CDs or MDDs. Finally, Table 4 com pares reli abil ity results between the 20 MDD symp tom que ries, its 73 sever ity subquestions, and its 9 diag nos tic cri te ria. While only 10% of symptom queries (2/20) elic ited fewer than 5 pos i tive responses at test, nearly one-third of the sever ity subquestions (22/73) and diag nos tic cri te ria (3/9) did not yield enough pos i tive responses to allow an esti ma tion of kappa. Sta bil ity was at least mod er ate (κ $ 0.50) for 90% of symp tom que ries, but it was at least mod er ate for only 37% of sever ity subquestions and 22% of diag nos tic cri te ria. Discussion To our knowl edge, this is the first reli abil ity report on a DSM-based pic to rial inter view for the assess ment of men tal dis or ders in ado les cents in the com mu nity. The high reli abil - ity observed, par tic u larly with symp tom score ICCs of 0.81 to 0.89, sug gests that ado les cents may ben e fit from a pic to rial approach com bined with direct and sim ple symp tom que ries. Anal y ses accord ing to age show that ICCs obtained from youn ger ado les cents (aged 12 to 14 years) are as high as those obtained from older ado les cents (aged 15 to 16 years). Pictures dis play ing con crete exam ples of abstract con cepts seem to sup port the pro duc tion of sta ble responses to symp - tom queries, even from the youn gest respon dents in the sam ple. Comparison With Other DSM-Based Instruments Regard ing dis or ders for which we obtained suf fi cient cases, kappa val ues for AD diag no ses com pare favour ably with results from the DICA-R (18), and var i ous ver sions of the DISC-2 (16,19 21) (Table 1), except for SAD. How ever, reli - abil ity may appear mis lead ingly low when applied to cat e gor - i cal diag no ses (49). A change in a sin gle response can bring the case to above or below the diag nos tic thresh old. The use of ICCs to assess reli abil ity of symp tom and cri te rion scores incor po rates the whole dis tri bu tion of responses and there fore 834 W Can J Psy chi a try, Vol 49, No 12, December 2004

8 Development and Reliability of a Pictorial Mental Disorders Screen for Young Adolescents Table 4 Distribution of kappa (κ) values calculated on the Adolescent Dominic major depressive (MDD) symptom queries, severity subquestions, and diagnostic criteria (n = 128) MDD Num ber < 5 pos i tive cases at test κ < κ < κ< κ < κ < κ Symptom queries 20 (100) 2 (10) 0 (0) 0 (0) 6 (30) 8 (40) 2 (10) 2 (10) Severity 73 (100) 22 (30.1) 7 (9.6) 17 (23.3) 17 (23.3) 5 (6.8) 4 (5.5) 1 (1.4) subquestions Diagnostic criteria 9 (100) 3 (33.3) 3 (33.3) 1 (11) 0 (0) 2 (22) 0 (0) 0 (0) Fleiss s cri te ria: κ < 0.40, poor re li abil ity; 0.40 κ< 0.60, fair re li abil ity; 0.60 κ 0.75, good re li abil ity; κ 0.75, ex cel lent re li abil ity typ i cally yields higher reli abil ity esti mates than do the kappa sta tis tics for indi vid ual items. The reli abil ity of AD symp tom scores (that is, ICCs of 0.81 to 0.89) com pares favourably with DISC-2 symp tom scores admin is tered to young sters in the com mu nity. In a study using the DISC-2.3 with infor mants aged 9 to 18 years, symp - tom-score ICCs ranged from 0.11 (panic dis or der) to 0.83 (CD), with aver age ICCs of 0.40 for anx i ety dis or ders, 0.52 for depres sive dis or ders, and 0.70 for dis rup tive behav iour dis or ders (49,50). A study of the DISC-2.25 with infor mants aged 12 to 14 years yielded ICCs of 0.71 to 0.84 (21). The AD shares sev eral impor tant fea tures with the Dominic-R Ques tion naire for school-aged chil dren (38). Both are pic to - rial, DSM-based struc tured inter views that assess a com pa ra - ble set of men tal dis or ders. Reli abil ity of AD symp tom scores com pares favour ably with the Dominic Ques tion naire (ICCs of 0.59 to 0.74) (38), the Dominic-R (0.71 to 0.81) (39), and the Afri can-american ver sion of the Dominic-R (ICCs > 0.75) (41). In contrast with the Dominic ver sion for youn ger chil dren (38), the AD com prises a lan guage-based com po nent for assess ing symp tom sever ity. Results show that, while reli abil - ity at the symp tom level is good, it is much less so for diag nos - tic cri te ria and diag no ses. Exten sive ver bal que ries fol low ing ini tial pos i tive responses to the pic tures mean that assess ment of the diag nos tic cri te ria may require skills and inter ac tions sim i lar to those required in purely ver bal inter views. Symp - tom fre quency, dura tion, and cooccurrence all variables nec es sary for diagnostic assess ment are less ame na ble to graphic rep re sen ta tion. Elab o rate, sen tence-based con tent inves ti gat ing time-related and other sever ity cri te ria may thus reduce reli abil ity to levels achieved by highly struc tured instru ments that rely on ver bal ques tion ing; ulti mately, it may gain lit tle for diag nos tic assess ment. At this stage, how ever, we could not eas ily dis en tan gle the rel a tive impact of low prev a lence and ver bal ques tion ing on reli abil ity loss. The low prev a lence of men tal dis or ders in com mu nity ado les - cent sam ples is a lim i ta tion in this type of study a prob lem that has con fronted many researchers (20,23,49). None the - less, we decided not to use a clin i cally enriched sam ple for the devel op men tal phase of the AD, because such sam pling strat - e gies increase sam ple het er o ge ne ity and result in higher kap - pas than would oth er wise occur (16). Finally, because an instru ment s reli abil ity is usu ally the upper limit of its valid ity and because no gold stan dard exists for assess ing cri te rion valid ity, com par ing the AD with other mea sures of men tal dis or ders for val i da tion pur poses would be ques tion able at this early stage. To cap i tal ize on the sat is fac tory reli abil ity results obtained with the AD espe cially for inter nal ized dis or ders it would be meth od olog i cally sound to focus on what this pic to rial instru ment does best; that is, it allows young sters to reli ably assess symp toms them selves. The AD would then become a DSM-based, stan dard ized, user friendly screen for front-line ser vice pro vid ers. These char ac ter is tics could be con sid ered a fair trade-off for the instru ment s lim ited capac ity to yield reli able diag no ses. For the AD, clin i cal cut-off points sim i lar to the symp tom-loading approach adopted by such rat ing scales as the Youth Self-Report (51) will have to be deter - mined in cri te rion and discriminant valid ity stud ies. How ever, unlike many psychopathology rat ing scales (52), its basis in DSM diag nos tic cri te ria assures con struct valid ity. The DSM-IV and the AD Dur ing these pre lim i nary stud ies of the AD (from 1993 to 1995), the fourth edi tion of the DSM was in prep a ra tion (53). A com par i son of the DSM-III-R and the DSM-IV at the symp - tom level shows that the DSM-IV intro duced changes mainly for ADHD, but less so for dis or ders such as ODD, CD, MDD, gen er al ized anx i ety dis or der (for merly OAD) and SAD. The nec es sary updat ing of the AD accord ing to DSM-IV cri te ria has been under taken, and for tu nately, almost all pic tures have been reused. More impor tant, the instru ment has been rede - signed for the screen ing of symp toms. A com put er ized DSM-IV French ver sion of the AD is cur rently under go ing field tests with clin i cal and com mu nity sam ples for reli abil ity Can J Psy chi a try, Vol 49, No 12, December 2004 W 835

9 The Ca na dian Jour nal of Psy chia try Orig i nal Re search and cri te rion valid ity, and an Eng lish ver sion will be val i dated as the next step. Funding and Sup port This study was sup ported by the Fonds de la Re cher che en Santé du Qué bec through a grant ( ) awarded to Dr Valla and Dr Bergeron. Ac knowl edge ment Pat rick Bolland pro vided help ful trans la tion as sis tance. References 1. Achenbach TM, McConaughy SH, Howell CT. Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychol Bull 1987;101: Graham P, Rutter M. The reliability and validity of the psychiatric assessment of the child. II. Interview with the parent. Br J Psychiatry 1968;114: Herjanic B, Herjanic M, Brown F, Wheatt F. Are children reliable reporters? J Abnorm Child Psychol 1975;3: Cytryn L, McKnew DH, Bunney, WE. Diagnosis of depression in children: a reassessment. Am J Psychiatry 1980;137: Herjanic B, Reich W. Development of a structured psychiatric interview for children: agreement between child and parent on individual symptoms. J Abnorm Child Psychol 1982;10: Orvaschel H, Puig-Antich J, Chambers W, Tabrizi MA, Johnson R. Retrospective assessment of prepubertal major depression with the Kiddie-Sads-E. J Am Acad Child Adolesc Psychiatry 1982;21: Moretti MM, Fine S, Haley G, Marriage K. Childhood and adolescent depression: child-report versus parent-report information. J Am Acad Child Adolesc Psychiatry 1985;24: Edelbrock C, Costello AJ, Dulcan MK, Calabro-Conover N, Kalas R. Parent-child agreement on child psychiatric symptoms assessed via structured interview. J Child Psychol Psychiatry 1986;27: Weissman M, Merikangas KR, Gammon GD. Assessing psychiatric s in children. Discrepancies between mothers and children s reports. Arch Gen Psychiatry 1987;44: Costello EJ, Costello AJ, Edelbrock C, Burns BJ, Dulcan MK, Brent D, and others. Psychiatric s in pediatric primary care. Prevalence and risk factors. Arch Gen Psychiatry 1988;45: Bird HR, Gould, MS, Staghezza B. Aggregating data from multiple informants in child psychiatry epidemiological research. J Am Acad Child Adolesc Psychiatry 1992;31: Kazdin AE. Informant variability in the assessment of childhood depression. In: Reynolds WM, Johnston HF, editors. Handbook of depression in children and adolescents. New York: Plenum Press; p Seiffge-Krenke I, Kollmar F. Discrepancies between mothers and fathers perceptions of sons and daughters problem behaviour: a longitudinal analysis of parent-adolescent agreement on internalising and externalising problem behaviour. J Child Psychol Psychiatry 1998;39: Youngstrom E, Loeber R, Stouthamer-Loeber M. Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. J Consult Clin Psychol 2000;68: Hoagwood K, Olin S. The NIMH blueprint for change report: research priorities in child and adolescent mental health. J Am Acad Child Adolesc Psychiatry 2000;41: Jensen P, Roper M, Fisher P, Piacentini J, Canino G, Richters J, and others. Test retest reliability of the Diagnostic Interview Schedule for Children (DISC 2.1). Arch Gen Psychiatry 1995;52: Shrout PE, Spitzer RL, Fleiss JL. Quantification of agreement in psychiatric diagnosis revisited. Arch Gen Psychiatry 1987;44: Boyle MH, Offord DR, Racine Y, Sanford M, Szatmari P, Fleming JE, Price-Munn N. Evaluation of the Diagnostic Interview for Children and Adolescents for use in general population samples. J Abnorm Child Psychol 1993;21: Ribera JC, Canino G, Rubio-Stipec M, Bravo M, Bauermeister JJ, Alegria M, and others. The Diagnostic Interview Schedule for Children (DISC-2.1) in Spanish: reliability in a hispanic population. J Child Psychol Psychiatry 1996;37: Schwab-Stone ME, Shaffer D, Dulcan MK, Jensen PS, Fisher P, Bird HR, and others. Criterion validity of the NIMH Diagnostic Interview Schedule for Children Version 2.3 (DISC-2.3). J Am Acad Child Adolesc Psychiatry 1996;35: Breton JJ, Bergeron L, Valla JP, Berthiaume C, St-Georges M. Diagnostic Interview Schedule for Children (DISC-2.25) in Quebec: reliability findings in light of the MECA study. J Am Acad Child Adolesc Psychiatry 1998;37: Edelbrock C, Costello AJ, Dulcan MK, Kalas R, Calabro-Conover NC. Age differences in the reliability of the psychiatric interview of the child. Child Dev 1985;56: Schwab-Stone M, Fallon T, Briggs M, Crowther B. Reliability of diagnostic reporting for children aged 6 11 years: a test retest study of the Diagnostic Interview Schedule for Children-Revised. Am J of Psychiatry 1994;151: Breton JJ, Bergeron L, Valla, JP, Lépine S, Houde L, Gaudet N. Do children aged 9 through 11 years understand the DISC Version 2.25 questions? J Am Acad Child Adolesc Psychiatry 1995;34: Yates T. Theories of Cognitive Development. In: Lewis M, editor. Child and adolescent psychiatry. Baltimore (MD): Williams and Wilkins; p Anthony BJ. Cognitive Development in Adolescence. In: Flaherty LT, Sarles RM, editors. Handbook of child and adolescent psychiatry. Vol 3, Adolescence: development and syndromes. New York: Wiley & Sons; p Butcher HJ. Human Intelligence. Its nature and assessment. New York: Harper & Row; Harter S, Pike R. The Pictorial Scale of Perceived Competence and Social Acceptance for Young Children. Child Dev 1984;55: Gaddès WH. Can educational psychology be neurologized? Can J Behav Sci 1969;1: Luria AR. The working brain. New York: Allen Lane, Penguin; Shiffrin RM, Schneider W. Controlled and automatic human information processing. II. Perceptual learning, automatic attending and a general theory. Psychol Rev 1977;84: Frostig M, Maslow P. Neuropsychological contributions to education. J Learn Disabil 1979;12: Bereiter C. Aspects of an educational learning theory. Review of Educational Research 1990;60: Goldberg E. Contemporary neuropsychology and the legacy of Luria. Mahwah (NJ): Lawrence Erlbaum Associates; Iran-Nejada A. Active and dynamic self-regulation of learning processes. Review of Educational Research 1990;60: Caramazza A. Is cognitive neuropsychology possible? J Cogn Neurosci 1992;4: Kosslyn SM, Intriligator JM. Is cognitive neuropsychology plausible? The perils of sitting on a one-legged stool. J Cogn Neurosci 1992;4: Valla JP, Bergeron L, Bérubé H, Gaudet N, St-Georges M. A structured pictorial questionnaire to assess DSM-III-R-based diagnoses in children (6 11 years): development, validity and reliability. J Abnorm Child Psychol 1994;22: Valla JP, Bergeron L, Bidaut-Russell M, St-Georges M, Gaudet N. Reliability of the Dominic-R: a young child mental health questionnaire combining visual and auditory stimuli. J Child Psychol Psychiatry 1997;38: Valla JP, Bergeron L, Smolla N. The Dominic-R: a pictorial interview for 6- to 11-year-old children. J Am Acad Child Adolesc Psychiatry 2000;39: Bidaut-Russell M, Valla JP, Thomas JM, Bergeron L, Lawson E. Reliability of the Terry: a mental health cartoon-like screener for African-American children. Child Psychiatry Hum Dev 1998;28: American Psychiatric Association. Diagnostic and statistical manual of mental s: DSM-III-R, revised. 3rd ed. Washington (DC): American Psychiatric Association; Lucas PC, Fisher P, Piacentini J, Zhang H, Jensen PS, Shaffer D, and others. Features of interview questions associated with attenuation of symptoms reports. J Abnorm Child Psychol 1999;27: Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960;20: Fleiss JL. Statistical methods for rates and proportions. New York: John Wiley & Sons; Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1997;86: Cronbach L. Essentials of psychological testing. 3rd ed. New York: Harper; Anastasi A. Psychological testing. 6th ed. New York: McMillan Publishing Company; Shaffer D, Fisher P, Lucas CP, Dulcan MK, Schwab-Stone ME. NIMH Diagnostic Interview Schedule for Children Version IV (NIMH DISC-IV): description, differences from previous versions, and reliability of some common diagnoses. J Am Acad Child Adolesc Psychiatry 2000;39: Shaffer D, Fisher P, Dulcan MK, Davies M, Piacentini J, Schwab-Stone ME, and others. The NIMH Diagnostic Interview Schedule for Children Version 2.3 (DISC-2.3): description, acceptability, prevalence rates, and performance in the MECA study. J Am Acad Child Adolesc Psychiatry 1996;35: Achenbach TM, Edelbrock C. Manual for the Youth Self-Report and Profile. Burlington (VT): University of Vermont Department of Psychiatry; Myers K, Winters NC. Ten-year review of rating scales. II. Scales for internalizing s. J Am Acad Child Adolesc Psychiatry 2002;41: American Psychiatric Association. Diagnostic and statistical manual of mental s. 4th ed. Washington (DC): American Psychiatric Association; W Can J Psy chi a try, Vol 49, No 12, December 2004

10 Development and Reliability of a Pictorial Mental Disorders Screen for Young Adolescents Manuscript received May 2003, revised, and accepted January Researcher, Research Unit, Rivière-des-Prairies Hospital; Associate Professor, Department of Psychology, Université du Québec à Montréal, Montreal, Quebec. 2 Researcher, Research Unit, Rivière-des-Prairies Hospital; Clinical Professor, Department of Psychiatry, Université de Montréal, Montreal, Quebec. 3 Researcher, Research Unit, Rivière-des-Prairies Hospital; Researcher, Department of Psychiatry, Université de Montréal, Montreal, Quebec. 4 Staff members, Research Unit, Rivière-des-Prairies Hospital, Montreal, Quebec. Address for correspondence: Dr N Smolla, Research Unit, Rivière-des- Prairies Hospital, 7070 Perras Boulevard, Montreal, QC, H1E 1A4 nicole.smolla.hrdp@ssss.gouv.qc.ca. Résumé : Élaboration et fiabilité d un instrument pictural de dépistage des troubles mentaux chez les jeunes adolescents Objectif : Faire rapport des données psychométriques tirées d études préliminaires de l Adolescent Dominic (AD), un instrument pictural de dépistage des troubles mentaux de l axe I les plus fréquents chez les jeunes. Méthodes : Nous avons créé 113 items picturaux d après les critères diagnostiques du DSM-III-R et nous en avons évalué la compréhension (échantillon 1, n = 114; échantillon 2, n = 40) et la fiabilité (échantillon 3, n = 128) dans un groupe d adolescents âgés de 12 à 16 ans vivant dans la communauté. Nous avons utilisé les statistiques kappa pour estimer la fiabilité test-retest des symptômes, des critères et des diagnostics ainsi que les coefficients de corrélation intraclasse (CCI) pour les scores de symptôme et de critère. Nous avons évalué la cohésion interne des scores de symptôme avec le coefficient alpha. Résultats : Pour les symptômes, 54,4 % des kappas étaient plus élevés que 0,60, et seulement 2 % étaient faibles. Les CCI pour les scores de symptôme donnaient des valeurs supérieures (0,81 à 0,89) aux scores de critères (0,51 à 0,86). La cohésion interne des scores de symptôme allait de 0,52 à 0,83. Les kappas des diagnostics allaient de 0,52 à 0,76. Conclusions : La fiabilité des symptômes se comparait favorablement avec les données d autres entrevues d évaluation des troubles mentaux d adolescents. Par suite de ces résultats positifs, une version DSM-IV informatisée de l AD a porté sur l évaluation des symptômes et fait présentement l objet de tests de fiabilité et de validité des critères. Can J Psy chi a try, Vol 49, No 12, December 2004 W 837