Explicit diagnostic criteria (e.g., based on the Diagnostic and Statistical Manual of Mental Disorders

Transcription

1 JOURNAL OF CHILD AND ADOLESCENT PSYCHOPHARMACOLOGY Volume 8, Number 4, 1998 Mary Ann Lieben, Inc. Pp Study I: Development and Criterion Validity of the Children's Interview for Psychiatric Syndromes (ChlPS) MARUO TEARE, Ph.D.,1 MARY A. FRISTAD, Ph.D., A.B.P.P.,2 ELIZABETH B. WELLER, M.D.,3 RONALD A. WELLER, M.D.,4 and PAUL SALMON Ph.D.5 ABSTRACT Objective: To test the psychometric properties of the Children's Interview for Psychiatric Syndromes (ChlPS), a second-generation structured diagnostic interview designed to improve on the methodology provided by first-generation structured interviews, which have increased diagnostic reliability over unstructured clinical interviews. Method: Forty-two children hospitalized on a children's inpatient unit were administered the ChlPS and the Diagnostic Interview for Children and Adolescents (DICA). The level of agreement regarding syndrome presence or absence of 15 diagnoses according to the two instruments was assessed using a standard kappa coefficient or a rare kappa coefficient and percentage agreement. Results: Agreement between the two instruments was significant (p <.5) for 13 of 14 diagnoses for which either kappa coefficient could be calculated. Percentage agreement was 98% and 1% for the remaining two diagnoses. ChlPS and DICA results also were compared with a psychiatrist's diagnoses. Sensitivity was 8% for ChlPS and 61% for DICA. Specificity was 78% for ChlPS and 87% for DICA. Conclusion: ChlPS is proposed as a valid measure of child psychopathology that offers advantages over existing interviews. INTRODUCTION Explicit diagnostic criteria (e.g., based on the Diagnostic and Statistical Manual of Mental Disorders (DSM) have facilitated the development of structured diagnostic interviews that systematically assess symptoms of specific syndromes (Chambers et al. 1985). Such interviews offer the advantages of increased reliability and precision (Edelbrock and Costello 1984). To date, both semi-structured interviews and highly structured interviews have been developed (Robins 1985). Structured interviews employ specifically worded questions, use explicit criteria to assess responses, and include instructions on how to conduct the interview and record results. Questions are asked verbatim, and well-defined scoring algorithms 'Private Practice, Shawnee Mission, Kansas. departments of Psychiatry and Psychology, Ohio State University, Columbus, Ohio. 3Children's Hospital of Philadelphia, Pennsylvania. 4University of Pennsylvania, Philadelphia, Pennsylvania. department of Psychology, University of Louisville, Kentucky. 25

2 TEARE ET AL. are provided. Thus, the effect of clinical judgment is reduced and responses are more objective, thereby facilitating their use for research purposes (Edelbrock and Costello 1984). Semi-structured interviews differ from structured interviews in several ways. Semi-structured interviews typically provide question stems but generally do not require these stems to be asked verbatim. Thus, they rely more on the interviewer's clinical judgment. Semi-structured interviews may more closely approximate a "real-life" clinical interview but require more sophisticated interviewers than do structured interviews (Edelbrock and Costello 1984). Interviews for children Several structured and semi-structured interviews have been developed for use with children. These include the Diagnostic Interview for Children and Adolescents (DICA; Herjanic et al. 1975), the Kiddie- Schedule for Affective Disorders and Schizophrenia (K-SADS; Puig-Antich and Chambers 1978), the Diagnostic Interview Schedule for Children (DISC; Costello et al. 1982), the Interview Schedule for Children (ISC; Kovacs 1974), and the Children's Assessment Schedule (CAS; Hodges et al. 1982). Each has been developed and used in research settings, but none has achieved wide usage in purely clinical settings. Problems associated with one or more of these instruments include excessive length (particularly when administered to young children), awkward administration procedures, use of age-inappropriate language, lack of comprehensibility (Breton et al. 1995), diagnostic criteria inconsistent with the DSM, and low to moderate agreement with clinically based diagnoses (Orvaschel 1988; Edelbrock and Costello 1984). Development for ChlPS The Children's Interview for Psychiatric Syndromes (ChlPS; Weller et al. 1985) was developed to address these problems. Development occurred in four major stages. First, existing structured diagnostic interviews for children were reviewed. Second, questions were written using simple, age appropriate vocabulary; short, uncomplicated sentence structure; and clear, comprehensible phrasing that was detailed enough to assess symptomatology. For example, average question length for ChlPS is 12 words per question, whereas the DICA has an average of 18 words per question. Third, the interview was designed using a branching format and a sequencing of questions that would help to enhance rapport, maximize honest responses, and be as efficient as possible. Finally, psychometric studies were performed. ChlPS may offer four pragmatic advantages over existing interviews for children. First, questions used in ChlPS are short in length and use simple language. This is advantageous because children tend to understand shorter questions better than longer ones (Harris and Liebert 1987). Second, ChlPS strives for a clear operational representation of DSM criteria. Third, ChlPS is easy to score and interpret syndromes that are endorsed are clearly evident on the scoring form. Fourth, record keeping is condensed, in that either a one-page summary sheet or a six-page scoring sheet needs to be maintained, rather than an entire interview booklet for each child interviewed. Description of ChlPS ChlPS was designed to be a concise, comprehensive interview to assess major psychiatric syndromes in children 6 to 12 years of age using DSM-III criteria (American Psychiatric Association 198). ChlPS was designed to be given by trained lay interviewers. The DSM-III ChlPS has 17 sections. Fifteen assess psychiatric syndromes, and two screen for child abuse and psychosocial Stressors. Order of administration for the 15 sections that assess psychiatric syndromes generally reflects the estimated prevalence of each syndrome in the general population (Yule 1981). The interview starts with the most common syndromes (e.g., attention deficit disorder, oppositional disorder) and finishes with the less common syndromes (e.g., schizophrenia/psychosis). Within each section, cardinal questions are asked and a multiple "skip" procedure is used. This means that if a child answers "no" to a certain number of questions, the rest of the questions in that section can be skipped. Also, if a question is asked in several different ways, the alternate ways can 26

3 DEVELOPMENT AND CRITERION VALIDITY OF ChIPS be skipped if the child answers "yes" to the first presentation. Cardinal questions are always asked. If the child responds "yes" to a cardinal question, the interviewer continues with that syndrome; if the child responds "no," the interviewer has the option of skipping to the cardinal question for the next syndrome. This allows for quick screening of all 15 syndromes, with more detailed questioning when cardinal questions are endorsed. Remaining questions are sequenced to assess the most common symptoms first and the least common symptoms last. Scoring instructions are clearly indicated both on the interview and on the recording form, so the interviewer can skip the remaining questions in a section if the child is clearly not meeting criteria for a particular syndrome. Sections on child abuse and psychosocial Stressors are included because these are considered to be important in the assessment of a child (Achenbach 1982; Hetherington and Martin 1979). These sections do not follow a branching format; instead, every question is asked. They are placed at the end of the interview because they request specific information concerning potentially stressful and sensitive issues. It is anticipated that by the end of the interview rapport will be well established and such sensitive questions will more likely be answered accurately and honestly. Finally, the response booklet includes a one-page face sheet on which demographic data, chief complaint, behavioral observations, and other pertinent information are recorded. The remaining five pages contain the scoring forms. These are organized such that it can be determined easily for each diagnosis which symptoms are endorsed, whether duration criteria are met, and whether diagnostic criteria are met. The response booklet is separate from the interview, so the entire interview booklet does not need to be retained for each patient. Purpose of this study The purpose of this study was to assess the ChlPS' criterion validity. First, ChlPS results were compared with those obtained from the DICA (Herjanic et al. 1975). The DICA was chosen for comparison because it is a structured interview, uses a branching format, assesses the same syndromes as ChlPS, and has been widely used (Fristad et al. 1997). Second, results from ChlPS and DICA were compared with clinically determined diagnoses. Using this methodology, the following could be determined: (1) the extent of agreement between the two interviews and (2) sensitivity and specificity of ChlPS compared with DICA. METHOD Participants Participants were 42 children hospitalized on a psychiatric unit for children 6 to 12 years of age (mean age ± SD 8.6 ± 1.9). Most (74%) = were boys. The racial/ethnic distribution was white 79%, African- American 17%, African-American/Hispanic 2%, and African-American/Asian 2%. This distribution was similar to that of all patients admitted to this unit. All had an IQ above 7, as determined by school records. Informed consent was obtained from the parent and assent from the child prior to participation in this study. Interviewers Five female interviewers conducted the interviews. All had previous experience in administration of the DICA, and all underwent a standardized training program to learn how to administer ChlPS, which required familiarization with the instrument, observation of mock interviews, and practice using mock interviews prior to administration of the instrument to study participants. Most (88%) of the 84 interviews were conducted by two primary interviewers. The other three interviewers were always paired with one of the two primary interviewers in the Latin Square design described below. Instruments Basic demographic data were gathered on all participants. The two interviews being compared were the ChlPS (Weller et al. 1989) and the DICA (Herjanic et al. 1975). The DICA is a commonly used structured interview with acceptable reliability and validity (Herjanic and Campbell 1977; Herjanic et al. 1975). 27

4 TEARE ET AL. Procedure Each child was administered the ChlPS and DICA using a counterbalanced, Latin Square design (i.e., each instrument was administered first half of the time, each interviewer administered equal numbers of the two instruments, and each interviewer interviewed half their participants first). Most children were administered both interviews within a 48-hour period. All children were administered both interviews within a 1- week period. This methodology controlled for potential sensitization effects as the child proceeded through the interview process and for possible examiner bias in administration. All interviews were administered without knowledge of results from the other interview. Each child also received clinical discharge diagnoses by a board-certified child psychiatrist also unaware of ChlPS or DICA results. Only those clinical diagnoses assessed by both ChlPS and DICA were included in these analyses (e.g., although the psychiatrist may have made a diagnosis of Tourette's disorder, it would not have been included in these computations because ChlPS and DICA do not assess Tourette's disorder). Fifty-nine diagnoses were made by the psychiatrist for the 42 participants (modal number of diagnoses per child = 1 ). Data analysis Data regarding the presence or absence of 15 disorders were analyzed separately using a standard kappa coefficient or a low base rate kappa statistic (kx; Verducci et al. 1988) and percentage agreement. These three statistics were used because there is no single best statistical procedure to assess agreement for low base rate conditions (Verducci et al. 1988). Although the kappa coefficient is the standard measure of inter-interview reliability, it is dependent on base rates, making it an unreliable measure when the prevalence of a disorder is extremely low (Spitznagel and Heizer 1985). Those disorders for which there was less than 25% endorsement on both the ChlPS and the DICA were analyzed using kx. However, kx also has statistical limitations (e.g., depending on the nature of the distribution, a 1% agreement can produce a kx of 1., which does not reach statistical significance). Thus, percentage agreement also was used as an index of the overall concordance (i.e., a summation of present/present and absent/absent agreements) between the two sources. Comparisons were made between ChlPS and DICA and between ChlPS and the discharge diagnoses. In addition, diagnostic disagreement was examined on a case-by-case basis. When the two interviews did not agree on syndrome endorsement, the psychiatrist's discharge diagnosis was compared individually with ChlPS and DICA results to determine which interview agreed more often with the psychiatrist's discharge diagnosis. RESULTS ChlPS-DICA comparisons When the 15 syndromes were compared, ChlPS generated more diagnoses per child than did DICA (4.1 versus 2.7 mean number of diagnoses respectively). Syndromes endorsed for five or more children (i.e., for more than 1% of the sample) on ChlPS but not DICA included oppositional disorder, conduct disorderaggressive, conduct disorder-nonaggressive, depression, and overanxious disorder. The only syndrome endorsed for five or more children on DICA but not ChlPS was attention deficit disorder. Kappa coefficients (standard or rare), percentage agreement, and number of present/present and absent/absent agreements between ChlPS and DICA for these syndromes appear in Table 1. Neither kappa coefficient could be assessed for one syndrome (anorexia) due to 1% agreement on its absence in this sample. Most (13 of 14) kappa coefficients that could be calculated were statistically significant. Percentage agreement was 98% for the one syndrome (bulimia) with an insignificant kappa coefficient. Clinical diagnosis and structured interview comparisons The psychiatrist assigned 59 discharge diagnoses from the list of 15 disorders assessed on ChlPS (averaging 1.4 diagnoses per child), out of a possible 63 diagnoses (42 children X 15 disorders). From this, sensitivity (i.e., the percentage of true-positive cases identified by the test) and specificity (i.e., the percentage of true-negative cases identified by the test) for each instrument was calculated. Sensitivity was 28

5 - - DEVELOPMENT AND CRITERION VALIDITY OF ChIPS Table 1. Agreement of Syndrome Endorsement on ChIPS and DICA for 42 Child Psychiatry Inpatients Syndrome Kappa* P< Percent agreement yes/yesb no/no Attention deficit disorder.387a.1 Oppositional disorder.364a.5 Conduct disorder, aggressive.314a.5 Conduct disorder, nonaggressive.39a.6 Phobic disorder.432a.6 Separation anxiety disorder.525a.1 Overanxious disorder.455a.3 Obsessive-compulsive disorder.61a.1 Anorexiad Bulimia.494a.16 Depression.474a.1 Mania.482a.4 Schizophrenia.731a.6 Enuresis.731a.1 Encopresis.475a.2 69% 74% 62% 62% 76% 76% 83% 93% 1% 98% 64% 93% 95% 88% 9% "Regular kappas used except where rare kappa (Verducci et al. 1988) is indicated with footnote. bthe number of cases in which the syndrome was scored as present by both interviews. The number of cases in which the syndrome was scored as absent by both interviews. dcoefficient could not be computed due to perfect agreement on absence of disorder. 8% for ChIPS and 61% for DICA. Specificity was 78% for ChIPS and 87% for DICA. In 72% of cases, all three sources agreed (Table 2). For 24% of the cases, disagreements between the three sources occurred, with one or both structured interviews reporting a diagnosis not made by the psychiatrist. These disagreements were most common for syndromes typically considered secondary (e.g., phobia), that may have resolved during hospitalization (e.g., encopresis), or were not diagnosed by the psychiatrist due to hierarchical decision making (e.g., oppositional disorder was not diagnosed in a child diagnosed with conduct disorder). A combination of both interviews (8%), ChIPS but not DICA (12%) and DICA but not ChIPS (4%) produced diagnoses not made by the psychiatrist. In only 4% of cases did the clinician make a diagnosis not made by one or both interviews. These included 1.6% not made by either interview, 2% made by ChIPS but not DICA, and.3% made by DICA Concordance Between Clinician, ChIPS, and DICA on Table Disorders for 42 Chdld Psychiatry Inpatients Clinician diagnosis Negative Positive DICA negative ChIPS ChIPS + DICA positive ChlPs ChIPS + 42 (67%) 75 (12%) 26 (4%) 5 (8%) 571 (91%) 1 (1.6%) 13 (2%) 2 (.3%) 34 (5%) 59 (8.9%) 43 (68.6%) 88 (14%) 28 (4.3%) 84 (13%) 63 (1%) "Results in 63 potential diagnoses (42 patients X 15 disorders) 29

6 TEARE ET AL. but not ChIPS. In these cases, diagnoses were made based on all information available to the psychiatrist, which included information not always available when relying solely on the child's structured interview responses (i.e., parent interview, observation of the child, school records). There was no particular pattern of endorsements for the diagnoses made by the psychiatrist alone. DISCUSSION This study compared results obtained from a newly developed structured interview, ChIPS, to a commonly used structured interview with proven reliability, DICA. Additionally, ChIPS' results were compared with discharge diagnoses assigned by a psychiatrist after reviewing many sources of information in addition to a semistructured interview with the child. ChIPS fared well in each comparison. ChIPS and DICA demonstrated significant concordance for the 13 of 14 syndromes for which kappa coefficients could be calculated. There was 1% agreement on absence for the remaining syndrome. This suggests that ChIPS identifies syndromes with a similar degree of accuracy as does an existing structured interview with proven reliability and validity. Moreover, when results obtained on ChIPS and DICA were compared with the psychiatrist's discharge diagnoses, ChIPS' sensitivity was superior to DICA's sensitivity (8% versus 61%) although the DICA's specificity was higher than that of ChIPS. All three sources were in agreement for 72% of possible diagnoses. However, in 22% of cases, ChIPS results did not agree with the clinical diagnosis. At least five possible reasons might account for this discrepancy. First, low base rates for some syndromes may have resulted in an inadequate development and evaluation of some interview questions. Second, the psychiatrist, using less explicitly structured interviewing techniques, may have elicited symptoms or made observations that led to clues about the presence of these syndromes. Third, the psychiatrist had access to pertinent data from sources other than the child when formulating the diagnosis. Fourth, because the discharge diagnosis was used for comparative purposes, the psychiatrist may not have recorded a diagnosis that resolved during hospitalization (e.g., encopresis) or that was not a focus of treatment (e.g., phobia). Finally, the psychiatrist might be wrong. In this study, only one psychiatrist provided discharge diagnoses; thus, no interrater reliability could be determined. It should be noted that these possible reasons for discrepancies are not unique to this study. The issues that contribute to reduced concordance have been widely recognized (e.g., Robins 1985; Young et al., 1987). Syndromes were endorsed on ChIPS more frequently than on DICA. However, given ChIPS' purpose as a diagnostic screening instrument, overascertainment is more desirable than underascertainment. If an endorsement is made, it can be investigated further. If a syndrome is not endorsed on a screening instrument, it may take longer for it to eventually come to the clinician's attention. The main purpose of this study was to test criterion validity on the ChIPS. Acceptable criterion validity was supported by high levels of agreement between ChIPS and DICA. Furthermore, when compared with clinical diagnoses, sensitivity was higher, although specificity was lower, for ChIPS compared with DICA. It appears that ChIPS is a promising new structured diagnostic interview, albeit with limited usage to date. Further studies will describe (1) revisions to meet DSM-III-R (American Psychiatric Association 1987) and DSM-IV (American Psychiatric Association 1994) criteria and (2) development of a parent version of the interview. REFERENCES Achenbach TM: Developmental Psychology: Concepts, Strategies, Methods. New York, The Free Press, 1982 American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, 3rd ed. Washington, DC, Author, 198 American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, 3rd ed., revised. Washington, DC, Author,

7 DEVELOPMENT AND CRITERION VALIDITY OF ChIPS American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, 4th ed. Washington, DC, Author, 1994 Breton JJ, Bergeron L, Valla JP, Lepine S, Houde L, Gaudet N: Do children aged 9 through 11 years understand the DISC version 2.25 questions? J Am Acad Child Adolescent Psychiatry 34: , 1995 Chambers WJ, Puig-Antich J, Hirsch M, Paez P, Ambrosini PJ, Tabrizi MA, Davies M: The assessment of affective disorders in children and adolescents by semi-structured interview. Arch Gen Psychiatry 42:696-72, 1985 Costello AJ, Edelbrock CS, Kalas R, Kessler MD, Klaric S: The NIMH Diagnostic Interview Schedule for Children (DISC). Pittsburgh, Pergamon, 1982 Edelbrock C, Costello AJ: Structured psychiatric interviews for children and adolescents. In: Handbook of Psychological Assessment. Edited by Goldstein G, Hersen M. New York, Pergamon, 1984, pp Fristad MA, Emery B, Beck SJ: Use and abuse of the Children's Depression Inventory. J Consult Clin Psychol 65:699-72, 1997 Harris JR, Liebert RM: The Child: Development from Birth to Adolescence, 2nd ed. Englewood Cliffs, NJ, Prentice- Hall, 1987, p. 37 Herjanic B, Campbell W: Differentiating psychiatrically disturbed children on the basis of a structured interview. J Abnormal Child Psychol 3:41^18, 1977 Herjanic B, Herjanic M, Brown F, Wheatt T: Are children reliable reporters? J Abnormal Child Psychol 3:41^-8, 1975 Hetherington EM, Martin B: Family factors and psychopathology in children. In: H.C. Psychopathological Disorders of Childhood, 3rd ed. Edited by Quay HC, Werry JS. New York, John Wiley & Sons, 1979, pp Hodges K, Kline J, Stern L, Cytryn L, McKnew D: The development of a Child Assessment Interview for research and clinical use. J Abnormal Psychol 1: , 1982 Kovacs M: The Interview Schedule for Children (ISC). Psychopharmacol Bull 21: , 1974 Orvaschel H: Structured and semistructured psychiatric interviews for children. In: Handbook of Clinical Assessment of Children and Adolescents, Vol. 1. Edited by Kestenbaum CJ, Williams DT. New York, New York University Press, 1988, pp Puig-Antich J, Chambers W: The Schedule of Affective Disorders and Schizophrenia for School-Aged Children. New York, New York State Psychiatric Institute, 1978 Spitznagel EL, Heizer HE: A proposed solution to the base rate problem in the kappa statistic. Arch Gen Psychiatry 42: , 1985 Robins LN: Epidemiology: Reflections on testing the validity of psychiatric interviews. Arch Gen Psychiatry 42: , 1985 Verducci JS, Mack ME, DeGroot MH: Estimating multiple rater agreement for a rare diagnosis. J Multivariate Analysis 27: , 1988 Weiler EB, Weiler RA, Teare M, Fristad MA: Children's Interview for Psychiatric Syndromes (ChlPS), Unpublished document, Division of Child & Adolescent Psychiatry, University of Kansas, Kansas City, Kansas, 1985 Young JG, O'Brien JD, Gutterman EM, Cohen P: Research on the clinical interview. J Am Acad Child Adolescent Psychiatry 26:613-62, 1987 Yule W: The epidemiology of child psychopathology. In: Advances in Clinical Child Psychology, Vol. 4. Edited by Kazdin AE, Lahey BB. New York, Plenum, 1981 Address reprint requests to: Mary A. Fristad, Ph.D., A.B.P.P. Departments of Psychiatry and Psychology Ohio State University 167 Upham Drive, Suite 46 Columbus, OH fristad. l@osu.edu 211