The Pennsylvania State University. The Graduate School. College of Education VALIDITY AND DIAGNOSTIC ACCURACY OF SCORES FROM THE AUTISM

Transcription

1 The Pennsylvania State University The Graduate School College of Education VALIDITY AND DIAGNOSTIC ACCURACY OF SCORES FROM THE AUTISM DIAGNOSTIC OBSERVATION SCHEDULE-GENERIC A Dissertation in School Psychology by Melissa A. Reid 2012 Melissa A. Reid Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy August 2012

2 ii The dissertation of Melissa A. Reid was reviewed and approved* by the following: James C. DiPerna Associate Professor of Education Professor in Charge of the Program of School Psychology Dissertation Adviser Richard Hazler Professor of Education Robert Steven Professor of Education Beverly J. Vandiver Associate Professor of Education *Signatures are on file in the Graduate School

3 iii Abstract The purpose of this study was to examine the internal structure, relationships with other variables, and diagnostic accuracy of scores on the Autism Diagnostic Observation Schedule Generic (ADOS-G; Lord et al., 1999) for the purpose of diagnostic decision-making. Participants were 462 children enrolled in a public school district in the southern U.S. who were referred for a school-based psychoeducational evaluation. Four hypotheses were tested with mixed results. The first prediction was that items included in the Original Scoring Algorithm (OSA) would reflect a uni-dimensional construct, and items included in the Revised Scoring Algorithm (RSA) would reflect two constructs across modules. Exploratory factor analysis confirmed the onefactor structure of the OSA across modules. However, a two-factor structure was not retained for the Module 2 or Module 3 RSA. Second, it was predicted that total scores on the ADOS-G, across modules and scoring algorithms, would demonstrate moderate to strong relations with scores from other measures of autistic behavior, and weak relations with measures of emotional functioning. Weak relationships were consistently measured between participants scores on the ADOS-G across modules and algorithms and other measures of autistic and emotional functioning. Third, it was predicted that scores obtained from application of the RSA would result in greater diagnostic accuracy than those obtained from the OSA. Receiver Operating Curve (ROC) analysis was conducted to determine the sensitivity and specificity of ADOS-G scores. Consistent with hypotheses, the RSA typically resulted in greater diagnostic accuracy, and a better balance between sensitivity and specificity than did the OSA. Finally, the fourth hypothesis, which predicted that the diagnostic accuracy of the ADOS-G would be lower with an independent criterion relative to an interdependent criterion, was not consistently supported. In general, results of the current study confirm the structural validity and overall diagnostic

4 iv accuracy of the ADOS-G, but also highlight some of the limitations of the instrument. Despite its limitations, it was concluded that the strengths of the ADOS-G provide support for its continued use in school-based psychoeducational evaluations for the diagnosis of students with Autism Spectrum Disorders.

5 v Table of Contents List of Tables viii List of Appendices...x Acknowledgements xii Chapter 1. Introduction and Literature Review...1 Definition of Autism Spectrum Disorders..2 Common Characteristics of Autism Spectrum Disorders...3 Assessment and Diagnosis of Autism Spectrum Disorders...5 Autism Diagnostic Observation Schedule..8 Development and Evolution of the ADOS...9 Autism Diagnostic Observation Schedule-Generic...10 Rationale for Present Study..36 Purpose and Hypotheses...37 Chapter 2. Method.40 Participants...40 Measures...44 Autism Diagnostic Observation Schedule-Generic.44 Gilliam Autism Rating Scale, Second Edition 45 Behavior Assessment System for Children, Second Edition..47 Procedure..50 Chapter 3. Results..53 Preliminary Analyses and Testing of Assumptions..53 ADOS-G Item Analysis...53 Total, Scale, and Subscale Score Analysis..60

6 vi Hypothesis 1: Factor Structure of the Original and Revised Scoring Algorithms 60 Module 1-Original Scoring Algorithm 63 Module 1-Revised Scoring Algorithm 66 Module 2-Original Scoring Algorithm 70 Module 2-Revised Scoring Algorithm.71 Module 3-Original Scoring Algorithm 77 Module 3-Revised Scoring Algorithm 79 Hypothesis 2: Relationships between Scores on the ADOS-G and Other Measures...82 Module Module Module Hypothesis 3: Comparisons of Diagnostic Accuracy Indicators Across Scoring Algorithms..88 Original and Revised Scoring Algorithm Comparisons..89 Updated Scoring Algorithms and Optimal Cut-Score Comparisons...92 Hypothesis 4: Diagnostic Accuracy of Independent Clinical Diagnoses.95 Chapter 4. Discussion 99 Structural Validity Evidence Module Module Module Convergent and Discriminant Validity Evidence Evidence of Diagnostic Accuracy Module 1 105

7 vii Module Module Independent Clinical Diagnoses.111 Summary of Evidence by Module and Scoring Algorithm 112 Module Module Module Clinical Implications Limitations..120 Future Research 123 Conclusions..124 References 126 Footnotes..136

8 viii List of Tables Table 1. Sensitivity and Specificity of Original and Revised Scoring Algorithms by Research Study.23 Table 2. Demographic Characteristics of Total Sample (N = 462) and Independent Clinical Diagnosis Subsample (N = 100) 41 Table 3. Item Means, Standard Deviations, Skew and Kurtosis Values on Module 1 from the ADOS-G (N = 82)..54 Table 4. Item Means, Standard Deviations, Skew and Kurtosis Values on Module 2 from the ADOS-G (N =118).56 Table 5. Item Means, Standard Deviations, Skew and Kurtosis Values on Module 3 from the ADOS-G (N = 262)...58 Table 6. Participant Means, Standard Deviations, Score Range, Skew, and Kurtosis Values on the ADOS-G, GARS-2, and Selected Subscales from the BASC Table 7. Structure Coefficients and Communalities for the ADOS-G Module 1 (Original Scoring Algorithm) Items (N = 82).65 Table 8. Pattern Coefficients, Structure Coefficients, and Communalities for the ADOS-G Module 1 (Revised Scoring Algorithm) Items (N = 66) 67 Table 9. Structure Coefficients and Communalities for the ADOS-G Module 1 (Revised Scoring Algorithm) Items (N = 66).69 Table 10. Structure Coefficients and Communalities for the ADOS-G Module 2 (Original Scoring Algorithm) Items (N = 118).72 Table 11. Pattern Coefficients, Structure Coefficients, and Communalities for the ADOS-G Module 2 (Revised Scoring Algorithm) Items (N = 73)...74 Table 12. Structure Coefficients and Communalities for the ADOS-G Module 2 (Revised Scoring Algorithm) Items (N = 73)...76 Table 13. Structure Coefficients and Communalities for the ADOS-G Module 3 (Original Scoring Algorithm) Items (N = 262).78 Table 14. Structure Coefficients and Communalities for the ADOS-G Module 3 (Revised Scoring Algorithm) Items (N = 261).81

9 ix Table 15. Pearson Correlations between Participants Total Scores on the ADOS-G Original and Revised Scoring Algorithms for Module 3 and Parent and Teacher Ratings on the GARS Table 16. Pearson Correlations between Participants; Total Scores on the ADOS-G Original, Revised, and Updated Scoring Algorithms and Parent and Teacher Ratings on the BASC Table 17. AUC Values, Sensitivities, Specificities, Positive Predictive Values, Negative Predictive Values, and Hit Rates of ADOS-G Scores on the Original and Revised Scoring Algorithm...90 Table 18. AUC Values and Optimal Cut-Scores for the ADOS-G Updated and Retained Scoring Algorithms.93 Table 19. Sensitivities, Specificities, Positive Predictive Values, Negative Predictive Values, and Hit Rates of ADOS-G Scores on the Updated and Retained Scoring Algorithms 94 Table 20. Sensitivities, Specificities, Positive Predictive Values, Negative Predictive Values, and Hit Rates of ADOS-G Scores on the Original and Revised Scoring Algorithm Compared to Clinical Diagnoses Made With and Without the ADOS-G (N = 100)..96 Table 21. Sensitivities, Specificities, Positive Predictive Values, Negative Predictive Values, and Hit Rates of ADOS-G Scores from the Updated and Retained Original Scoring Algorithms Compared to Clinical Diagnoses Made With and Without the ADOS-G (N = 100) 98

10 x List of Appendices Appendix A: DSM-IV-TR Diagnostic Criteria for Autism Spectrum Disorders 137 Appendix B: 142 Table B1: Activities on the Autism Diagnostic Observation Schedule and their Purpose by Module (Lord, Rutter, DiLavore, & Risi, 1999).142 Table B2: Items Rated on the Autism Diagnostic Observation Schedule by Subdomain and Module (Lord, Rutter, DiLavore, & Risi, 1999).149 Table B3: Items Included in the Revised Scoring Algorithm on the Autism Diagnostic Observation Schedule-Generic by Developmental Cell.153 Appendix C.156 Table C1: Correlation Matrix of Items Included in the ADOS-G Module 1, Original Scoring Algorithm (N = 82).156 Table C2: Correlation Matrix of Items Included in the ADOS-G Module 1, Revised Scoring Algorithm (N = 66).157 Table C3: Correlation Matrix of Items Included in the ADOS-G Module 2, Original Scoring Algorithm (N = 118) Table C4: Correlation Matrix of Items Included in the ADOS-G Module 2, Revised Scoring Algorithm (N = 73).159 Table C5: Correlation Matrix of Items Included in the ADOS-G Module 3, Original Scoring Algorithm (N = 261) Table C6: Correlation Matrix of Items Included in the ADOS-G Module 3, Revised Scoring Algorithm (N = 262) Appendix D.162 Table D1: Corrected Item-Total Correlations and Cronbach s Alpha if Item Deleted Values for ADOG-G Original Scoring Algorithm One-Factor Solutions..162 Table D2: Corrected Item-Total Correlations and Cronbach s Alpha if Item Deleted Values for ADOG-G Revised Scoring Algorithm One-Factor Solutions Table D3: Corrected Item-Total Correlations and Cronbach s Alpha if Item Deleted Values for ADOG-G Revised Scoring Algorithm Two-Factor Solutions..166 Appendix E..168

11 xi Table E1. Structure Coefficients and Communalities for the ADOS-G Module 1 (Original Scoring Algorithm) Items with Deletion of Item A-5 (N = 82)..168 Table E2: Structure Coefficients and Communalities for the ADOS-G Module 2 (Revised Scoring Algorithm) Items with Deletion of Item D-2 (N = 73)..169 Table E3: Structure Coefficients and Communalities for the ADOS-G Module 3 (Original Scoring Algorithm) Items with Deletion of Item A-4 (N = 262) 170 Table E4: Structure Coefficients and Communalities for the ADOS-G Module 3 (Revised Scoring Algorithm) Items with Deletion of Items D-1 and D-2 (N = 261) 171 Appendix F..172 Table F1: Cut Scores Used for ADOS-G Classification Determinations by Module and Scoring Algorithm Table F2: Sensitivity and Specificity Values of Scores on the Original Scoring Algorithm from the Current Sample and Lord et al. s (1999) Original Sample Table F3: Sensitivity and Specificity Values of Scores on the Revised Scoring Algorithm from the Current Sample and Previous Studies. 174 Appendix G. Curriculum Vitae...175

12 xii Acknowledgements There are many people who have assisted me throughout the process of completing my graduate education and my dissertation that deserve thanks for their efforts. First, I want to thank Dr. James DiPerna, my adviser and dissertation chair, for all of his guidance, encouragement, and faith over the last eight years. I sincerely thank you Jim for not giving up on me, even when I had given up on myself. I truly appreciate all you have done and know that I would not be writing acknowledgements to a completed dissertation without you. I would also like to thank the other members of my doctoral committee, Drs. Richard Hazler, Robert Stevens, and Beverly Vandiver, for their feedback over the years and contributions to my dissertation. To my wonderful graduate school cohort, especially Miranda Freberg, Anne McGinnis, and Erin Meyer, I never would have survived graduate school without you ladies! Thank you for your collaboration and friendship over the years. Thank you to the administrative staff of the Lewisville Independent School District, Department of Special Education for allowing me to use district data to complete my dissertation. I d also like to thank my colleagues in Psychological Services who assisted me with data collection and evaluation review. Special thanks to Robin Chaney, Jennifer Key, Jill Littleton, Jessica Martin, Amorette Miller, Linda Pedersen, Shannon Spence, and Kimberly Ward for providing me with endless support and friendship while I was attempting to kill Earl. Thank you, Linda, for asking me about my dissertation progress each week in supervision, despite the inevitable outcome, and for always holding me accountable for working on it. Jennifer, thank you for reminding me that I would have never forgiven myself if I didn t finish what I started. You both played a special role in helping me get to the place that I am at today.

13 xiii To my other family and friends who have provided me with love and support throughout this long journey, your contributions have been greatly appreciated. My greatest thanks are to my mother, Patricia Reid, to whom this work is dedicated. I owe all that I am and all that I have achieved to you, and I wish that you were here to share in my greatest accomplishment. I hope you are looking down on me with pride.

14 1 Chapter 1. Introduction and Literature Review The Autism Diagnostic Observation Schedule-Generic (ADOS-G; Lord, Rutter, DiLavore & Risi, 1999) is one of the most widely utilized diagnostic instruments in the direct assessment of the social, communicative, and sensorimotor symptoms of Autism Spectrum Disorders (ASD) in both clinical and educational settings. Despite its popularity and widespread use, little independent research regarding the psychometric properties of ADOS-G scores has been conducted to date. Thus, the purpose of this study was to examine the internal structure, relationships with other variables, and diagnostic accuracy of ADOS-G scores for the purpose of diagnostic decision making. The following literature review begins with a brief overview of Autism Spectrum Disorders (ASDs) and information on current assessment and diagnostic practices used in the diagnosis of ASDs. The next section synthesizes existing research regarding the psychometric properties of ADOS-G scores. This chapter then concludes with the rationale, purpose, and primary hypotheses for the study. Relevant studies were identified by searching PsychINFO and PsychARTICLES databases with ADOS as the primary search term. This search yielded 191 studies that included the ADOS as a key study descriptor. The search was narrowed by selecting only studies that were published in a peer-reviewed journal, resulting in 166 possible articles for inclusion in the synthesis. Abstracts were reviewed to identify research studies that examined reliability and/or validity evidence (e.g., stability of measurement across examiners and/or time and internal consistency of assessment items; evidence of test structure and diagnostic accuracy) for ADOS scores as a study objective. If study outcomes were not clearly identified within the abstract, full text was reviewed for clarification. Based on the abstract review, the vast majority

15 2 of the research studies featured the ADOS-G as a diagnostic measure of ASDs rather than examining the instrument or its technical adequacy as a study outcome. As a result, only 17 studies were identified that met the criteria for inclusion in the synthesis. Definition of Autism Spectrum Disorders Autism is a general term often used to describe a group of disorders formally called Pervasive Developmental Disorders (PDDs) and commonly referred to as Autism Spectrum Disorders (ASD). ASDs can be defined as cognitive and neuro-behavioral disorders, including, but not limited to, three core-defining features: impairments in socialization, impairments in verbal and nonverbal communication, and restricted and repetitive patterns of behaviors (Filipek et al., 1999, p. 439). In a recent report published by the Center for Disease Control and Prevention (CDC; 2009), it was noted that ASDs affect approximately 1 in 110 children in the United States. Symptoms of ASDs, which often include deficits in the use and understanding of verbal and nonverbal communication, literal and repetitive patterns of thought, and sensory processing deficits (Autism, n.d.), are typically present from birth or very early in development. However, diagnosis often does not take place prior to the age of 2 years (Lord et al., 2006). First reported by Kanner in 1943 as a syndrome of autistic disturbances, ASDs were initially identified in case histories of children between the ages of 2 and 8 years that shared unique and previously unreported patterns of behavior, including social remoteness, obsessiveness, stereotypy, and echolalia (Filipek et al., 1999, p. 442). Although included in the first and second editions of the Diagnostic and Statistical Manual for Mental Disorders (DSM; American Psychiatric Association, 1952; 1968), ASDs were characterized as psychotic reactions in children, manifesting primarily autism and were classified as schizophrenic reaction or schizophrenia, childhood type (American Psychiatric Association, 1968, p. 28).

16 3 However, following the publication of the Diagnostic and Statistical Manual of Mental Disorders, Third Edition (DSM-III; American Psychiatric Association, 1980), ASDs were reclassified and reconceptualized. The term Pervasive Developmental Disorder (PDD) was first introduced in the DSM-III, as was the differentiation between ASD and childhood schizophrenia and other forms of psychoses (Filipek et al.). The terms Autistic Disorder and Pervasive Developmental Disorder Not Otherwise Specified (PDD-NOS) were introduced in the Diagnostic and Statistical Manual of Mental Disorders, Third Edition-Revision (DSM-III-R; American Psychiatric Association, 1987). According to the current Diagnostic and Statistical Manual for Mental Disorders, Fourth Edition-Text Revision (DSM-IV-TR; American Psychiatric Association, 2000), there are five distinct ASDs or PDDs: Autistic Disorder, Asperger s Disorder, Rett s Disorder, Childhood Disintegrative Disorder, and Pervasive Developmental Disorder Not Otherwise. The diagnostic criteria for each disorder, as listed within the DSM-IV-TR, are listed in Appendix A. Common Characteristics of Autism Spectrum Disorders As is evident from their definition and diagnostic criteria, ASDs affect essential human behaviors such as social interaction, communication, imagination, and establishing relationships, which typically result in life-long effects on learning, interpersonal interactions, independence, and level of participation in the community (Autism, n.d.). According to the National Research Council (2001), the level of impairment experienced by an individual with an ASD varies according to their age of onset and the severity of their symptoms, as well as the absence or presence of co-morbid psychiatric disorders. Across and within individuals, the manifestation of an Autism Spectrum Disorder can vary over time: there is no single behavior that is always typical or present in individuals with ASDs.

17 4 There are, however, several common behavioral characteristics that often are observed in individuals with Autism Spectrum Disorders. First, speech and language difficulties, as well as deficits in the use and understanding of nonverbal communication, are typically observed in individuals on the spectrum. Although the severity of communication impairment varies across the Autism Spectrum Disorders, all individuals with ASDs exhibit some of the following behaviors: deficits in verbal language, such as failing to speak, repeating words or phrases heard, and/or talking repetitively about one topic; atypical pitch, tone, prosody, and/or volume of speech; failure to use spoken and body language to communicate; does not appear to be listening, even when spoken to directly; and does not use nonverbal communication methods, such as gesturing or pointing. In addition to expressive language deficits, individuals on the spectrum also often experience difficulties with receptive language, or language comprehension (National Research Council, 2001). Cognitive and perceptual impairments also are often observed in individuals with Autism Spectrum Disorders. Specifically, individuals on the spectrum often exhibit a here-and-now way of thinking, which is typically very literal and repetitive in nature. They often demonstrate a lack of curiosity about their environment and surroundings, and, at times, fail to attend to important stimuli, focusing on irrelevant stimuli instead. An obsessive desire for sameness and repetition may also be observed (National Research Council, 2001). Deficits in reciprocal social interactions are the hallmark characteristics of all ASDs and a variety of social deficits are typically observed in individuals on the spectrum. Common social atypicalities include: resistance to being touched or held, failure to respond to name, an inability to relate to peers and adults in an ordinary way (e.g., ignores or avoids people), failure to

18 5 appropriately modulate eye contact, lack of use of social smiling, and a general lack of understanding of how other people think, feel, or view the world. In addition to communication, cognitive and perceptual, and reciprocal social impairments, individuals with ASDs also typically exhibit some degree of sensory processing deficits and engagement in stereotyped behaviors. For example, those on the Spectrum may exhibit extreme fear reactions to loud noises, strangers, new situations, changes, or surprises; may be under- or over-responsive to physical pain; and may demonstrate distinct food and clothing preferences. Further, individuals with spectrum disorders may rock or spin objects as a form of self-stimulatory behavior, may require compulsive adherence to specific routines, may become preoccupied with one or a few objects, and may tantrum or exhibit other aggressive behaviors when upset (National Research Council, 2001). Assessment and Diagnosis of Autism Spectrum Disorders Although there are clearly defined diagnostic criteria, difficulties exist in the diagnosis of ASDs. Despite being neurological in nature, the neuro-physiological markers of ASDs have not yet been clearly identified or documented. As a result, physicians, psychologists, and other professionals charged with diagnosing ASDs are required to rely on a child s observable patterns of behavioral functioning in order to make a diagnosis (Lord & Risi, 1998). Reliance on observable symptoms, however, can be challenging for several reasons. First, the symptoms of autism /ASDs can differ dramatically across individuals and within individuals across time (Lord, 2010; Tsai, 1992). Significant symptom overlap between Autistic Disorder and the various ASDs can make differential diagnosis between disorders quite difficult, especially in younger and older individuals (Lord & Risi, 1998; Lord & Volkmar, 2002). Further, symptom overlap between ASD s and other physiological and psychological conditions, such as mental

19 6 retardation, other developmental disabilities, expressive and receptive language disorders, Attention-Deficit/Hyperactivity Disorder (ADHD), and childhood-onset schizophrenia also complicates differential diagnosis (American Psychiatric Association, 2000; Ghaziuddin, 2005; Reaven, Hepburn, & Ross, 2008). Due to the complexity of diagnosis, a multi-disciplinary approach to the diagnostic assessment of Autism Spectrum Disorders is recommended (Filipek et al., 1999). Filipek et al. recommended that each diagnostic evaluation should include a number of components, including a comprehensive interview with parents and other caregivers in which a complete birth, medical, family, and developmental history is obtained; direct observations of and interactions with the child being assessed; assessment of the child s adaptive and general behavioral functioning, and direct assessment of the child s speech/language/communication skills, cognitive functioning, sensorimotor functioning, and academic functioning. Use of measures that are designed specifically for the screening and diagnosis of ASDs are also strongly recommended (Filipek et al., Risi et al., 2006). Several specific screening and diagnostic measures for ASDs are widely used by researchers and clinicians in the process of completing a multidisciplinary autism evaluation. Two of the most commonly used rating scales at this time are the Childhood Autism Rating Scale (CARS; Schopler, Reichler, & Rochen Renner, 1988) and the Gilliam Autism Rating Scale (GARS; Gilliam, 1995). Although authors for both assessments have indicated that their scores possess adequate reliability and validity for screening (CARS) and diagnostic (GARS) decisions (Gilliam; Schopler et al.), independent research has raised some questions regarding the usefulness and diagnostic accuracy of these assessments. Specifically, Lord and Risi (1998) noted that the CARS does not effectively differentiate individuals with communication deficits

20 7 and cognitive and behavioral difficulties related to autism from examinees with expressive language delays, cognitive impairments, and general behavioral difficulties that are not due to a pervasive developmental disorder. In their investigation of the discriminative ability and diagnostic utility of the GARS, Mazefsky and Oswald (2006) determined that the measure does not accurately discriminate children with autism from those with non-developmental disabilities. A 2008 study conducted by Sikora, Hartley, McCoy, Gerrard-Morris, and Dill confirmed the instrument s failure to consistently discriminate examinees on the autism spectrum from those that are not. More concerning, however, was Mazefsky and Oswald s conclusion that the GARS systematically underestimates the probability that examinees are on the autism spectrum. A previous study of the GARS conducted by South et al. (2002) presented similar concerns with the diagnostic accuracy of the instrument. In 2006, a second edition of the GARS was published by the test author (GARS-2; Gilliam). In an attempt to address the systematic concerns of the GARS raised by independent researchers, the GARS-2 was created with a new normative sample of participants (Montgomery, Newton, & Smith, 2008). Substantial revisions were made to the instrument, including the elimination of one of the four subscales found within the measure and the introduction of an interview component to allow for the evaluation of the child s development during early childhood (Gilliam, 2006). Independent research on the technical adequacy of the GARS-2 has yet to be completed. Another popular autism diagnostic measure is the Autism Diagnostic Interview-Revised (ADI-R; Lord, Rutter, & LeCouteur, 1994). The ADI-R is a standardized comprehensive interview that can be completed with parents/primary caregivers and, consistent with the

21 8 recommendations of Filipek et al. (1999), requests information about the child s birth, health, and developmental history. It is designed for use with caregivers of children under evaluation who demonstrate a developmental level of at least 2 years, 0 months of age. Validation studies completed by the authors indicate that scores from the ADI-R reliably and validly diagnose autism in children and adolescents (Rutter, LeCouteur, & Lord, 2003). Independent research has also confirmed its technical adequacy (Cicchetti, Lord, Koenig, Klin, & Volkmar, 2008; Noterdaeme, Mildenberger, Sitter, & Amorosa, 2002; Papanikolaou et al., 2009). However, concerns regarding the ADI-R have also been documented. Ventola et al. (2006) noted that the typical length of time required for appropriate administration of the ADI-R (i.e., 90 to 150 minutes; Rutter, LeCouteur, & Lord) is prohibitive and may make it impractical for use in school-based evaluations. In addition, unlike other diagnostic assessment currently in use, the ADI-R does not differentiate between Autistic Disorder and other ASDs (LeCouteur, Haden, Hammal, & McConachie, 2008). Autism Diagnostic Observation Schedule Perhaps the most widely used diagnostic assessment of autism, also considered the current gold standard in autism assessment (Kline-Tasman, Risi, & Lord, 2007), is the Autism Diagnostic Observation Schedule-Generic (ADOS-G; Lord, Rutter, DiLavore & Risi, 1999). Designed for use with individuals who are thought to have an ASD, the ADOS-G is a standardized assessment of communication, social interaction, play/imagination, and stereotyped behaviors and interests. The original ADOS was designed to provide researchers and clinicians with a standardized tool that could be used to record a child or adolescent s social and communicative behavior throughout the course of a comprehensive evaluation for an Autism Spectrum Disorder. Since the time of its initial release, the ADOS has evolved in order to be

22 9 used with a broader range of examinees, both in terms of age and expressive language level, and in a variety of settings (DiLavore, Lord, & Rutter, 1995), and in order to provide more consistent differential diagnosis between children and adolescents on the autism spectrum and those with other developmental disabilities that are not on the spectrum (Lord et al., 1999). Published in 1999, the most current version of the ADOS is the Autism Diagnostic Observation Schedule- Generic (ADOS-G; Lord et al., 1999). Development and evolution of the ADOS. First published in 1989, the Autism Diagnostic Observation Schedule (ADOS; Lord et al.) was intended to be used in the differential diagnosis of ASDs from other disorders, such as mental retardation, and typical childhood development. It also was designed as a research tool to directly study the social behaviors and communication patterns found in individuals with ASDs. At the time of its initial release, the ADOS was unique from other scales in two primary ways (Lord et al., 1989). First, unlike other diagnostic measures of autism available at that time, the ADOS was designed to focus examiners observations on clients social and communicative functioning to identify the presence or absence of behaviors that are specific to autism. In addition, the ADOS also provided examiners with specific administration directions to guide their own behavior in conjunction with the behavior of their examinees (Lord et al.). Despite its advances, the original ADOS was limited because it could only be utilized with examinees between the ages of 5 and 12 whose expressive language skills were, at a minimum, developmentally consistent with those of a 3-year-old child (Lord et al., 2000). However, individuals with autism frequently exhibit delays and deficits in all areas of language acquisition, including receptive, expressive, and pragmatic (social) language. Further, the majority of children are under 5 years of age when first referred for an autism assessment (Lord

23 10 et al.). Administration time of the ADOS was also lengthy due to its large number of items, and completion of the assessment was often problematic for examiners, especially with younger and more impaired children (Lord et al., 2000). In an attempt to address these limitations, DiLavore, Lord, and Rutter (1995) developed the Pre-Linguistic Autism Diagnostic Observation Scale (PL-ADOS), which was a downward extension of the ADOS for use with verbal children between the ages of 2 and 4 and with examinees of any age who do not exhibit spontaneous expressive language. Thus, the combination of the PL-ADOS and ADOS increased the overall utility of the instrument system by broadening the range of individuals with whom the ADOS could be used. Limitations remained with the ADOS and PL-ADOS, however. Most notably, research indicated that the PL-ADOS was not able to accurately differentiate between Autism Spectrum Disorders and non-spectrum developmental delays in children of preschool age (Lord et al., 2000). In addition, the ADOS did not include normative data for individuals above the age of 12, and its items and activities were not developmentally appropriate for adolescents and adults. In response to these needs, an updated measure (ADOS-G, Lord et al., 1999) was published in 1999 and is still in use today. Autism Diagnostic Observation Schedule-Generic. The ADOS-G was superior to its predecessors in several significant ways. As a replacement for both the ADOS and the PL- ADOS, the instrument was designed for use with individuals across the lifespan. Instead of consisting of a standard pool of items that is to be administered to all examinees (as was found in the original ADOS and PL-ADOS), the ADOS-G is composed of a set of modules including assessment activities that are appropriate for use with the individuals for whom the module was designed. Modules were designed with consideration of both the chronological age and verbal

24 11 fluency of the examinee in order to minimize the potential bias of expressive language ability on performance, as was observed in previous iterations of the instrument (Lord et al., 2000). In addition, across the modules, scoring determinations are based on deviations from the expectations of abilities given the examinee s expressive language level in order to better differentiate the social and communication difficulties that are related to language ability versus other developmental concerns (Lord et al.). Unlike the standardization samples used for normative comparisons of performance on the ADOS and the PL-ADOS, which only included individuals with Autistic Disorder, the standardization sample for the ADOS-G included individuals with Autistic Disorder, Asperger s Disorder, and Pervasive Developmental Disorder Not Otherwise Specified, allowing for the comparison of a participant s performance to those with a range of PDDs (Lord et al.). The ADOS-G consists of four modules. Only one module is administered to an examinee during a comprehensive evaluation. Each module includes items from four subscales: Communication, Reciprocal Social Interaction, Play/Creativity/Imagination, and Stereotyped Behaviors and Restricted Interests. However each ADOS-G module is unique in its item composition. Module 1 was designed for non-verbal examinees or for those that do not consistently use spontaneous phrase speech (Lord et al., 2000). It is composed of 10 activities (see Table B1 for a list of assessment activities by module), which result in ratings on 29 dimensions of functioning (see Table B2 for a list of rated dimensions by subscale for each of the four modules). Module 2 was designed for use with verbally fluent (i.e., individuals who produce a range of flexible sentence types, provide language beyond the immediate context, and describe logical connections within a sentence ) young children or older children who exhibit some spontaneous phrase speech, but who are not verbally fluent (Lord, Rutter, DiLavore, &

25 12 Risi, 1999, p. 5). It is composed of 14 activities which are rated on 28 dimensions of functioning. Older children and younger adolescents with regular use of fluent, spontaneous phrase speech are administered Module 3, which is comprised of 13 activities and results in ratings on 28 dimensions of functioning. Module 4, designed for use with older adolescents and adults with fluent expressive language abilities, is composed of 10 required and 5 optional activities that lead to ratings on 31 dimensions of functioning. Unlike the other modules, the required activities in Module 4 are not play-based and, instead, are comprised of a series of interview questions (Lord et al.). According to Lord et al., the activities of Modules 1 and 2 are designed to allow for a flexible, active assessment administration, whereas the administration of Modules 3 and 4 is more structured. Technical development of the Original Scoring Algorithm of ADOS-G. According to Lord et al. (1999), items included in the Original Scoring Algorithm for each module were selected from a larger pool of items included in the original version of the ADOS (Lord et al., 1989) and the PL-ADOS (DiLavore et al., 1995) that assessed aspects of the DSM-IV/ICD10 diagnostic criteria for Autism Spectrum Disorders. From the initial pool, items were examined for suitability. In addition, those that did not demonstrate adequate interrater reliability (i.e., r >.80) and/or consistently result in scoring differences between participants with ASDs and those without were discarded as potential scoring algorithm items. The remaining item pools were submitted to exploratory factor analysis to further eliminate items that were outliers or that demonstrated strong correlations to mental or chronological age (Lord et al.). Finally, ROC curve analyses were conducted on the retained items to determine appropriate cut-scores for non- Autism ASD and Autism classifications. Some items that contributed to the possible assessment

26 13 of improvement over time (p. 113) or that assessed behaviors of particular clinical importance were retained on the instrument but not included in the final scoring algorithm. Reliability evidence for scores from the ADOS-G Original Scoring Algorithm. Based on the information provided by test authors in the administration manual (Lord et al., 1999), the ADOS-G Original Scoring Algorithm consistently and accurately measures the symptoms and characteristics of Autistic Disorder and non-autism Autism Spectrum Disorders, and differentiates those with spectrum disorders from those without, and those with Autistic Disorder from those with non-autism ASDs. Reliability analyses were conducted on individual items, domain scores, and classification determinations. Item inter-rater reliabilities (i.e., kappa coefficients) ranged from.55 to 1.0 for Module 1 (mean percent agreement = 91.5%),.48 to.93 for Module 2 (mean percent agreement = 89%),.46 to 1.0 for Module 3 (mean percent agreement = 88.2%), and.41 to.93 for Module 4 (mean percent agreement = 88.25%). Interrater reliability coefficients for the Social Interaction domain ranged from.88 to.97 across modules, from.74 to.90 for the Communication domain across modules, and from.84 to.98 across modules for the Communication + Social Interaction Total used for diagnostic classification determinations. Inter-rater agreement in diagnostic classifications for Autistic Disorder versus non-spectrum disorders was 90% for Module 4, 91% for Module 2, and 100% for Modules 1 and 3. Although inter-rater agreement in diagnostic classifications for non-autism Autism Spectrum Disorders versus non-spectrum disorders was slightly lower (k =.84 to.93) than observed for Autistic Disorder, it was still measured to be within an acceptable range. Testretest reliability coefficients were also reported for the Social Interaction (r =.78) and Communication (r =.73) domain scores, and for the Communication + Social Interaction Total score (r =.82) across modules, and interpreted by authors as evidence of excellent stability of

27 14 measurement (Lord et al., p. 116). In addition, the internal consistency of items within each domain was assessed (α =.86 to.91 for the Social Interaction domain; α =.74 to.84 for the Communication domain; α =.47 to.65 for the Stereotyped Behaviors and Restricted Interests) and determined by authors to indicate good agreement (Lord et al.). Validity evidence for scores from the ADOS-G Original Scoring Algorithm. Validity analyses on scores from the ADOS-G Original Scoring Algorithm have been investigated by test authors and independent researchers. Structural validity. For each ADOS-G module, an exploratory factor analysis was run to investigate the structural validity of the items included within the Original Scoring Algorithm. Authors (Lord et al., 1999), reports indicated that, for each module, one major factor emerged, onto which almost all items in the Social Interaction and Communication domains loaded highly (p.116). However, pattern coefficients and other information regarding factorability were not provided. Other independent analyses of the structural validity of the ADOS-G Original Scoring Algorithm have not been conducted to date. Evidence of diagnostic accuracy. Diagnostic accuracy also was investigated by authors. For each participant, the diagnostic classification based on his or her Communication and Social Interaction Total Score on the ADOS-G Original Scoring Algorithm was compared to his or her clinical diagnosis. Sensitivity and specificity were calculated for each module using Receiver Operating Characteristic (ROC) curves. Across modules, sensitivity values ranged from.93 to 1.0 and specificity values ranged from.93 to 1.0 when differentiating Autistic Disorder from a nonspectrum disorder; sensitivity from.90 to.97 and specificity from.87 to.94 when differentiating all Autism Spectrum Disorders (including Autistic Disorder) from a nonspectrum

28 15 disorder; and sensitivity from.80 to.94 and specificity from.88 to.94 when differentiating a non-autism Autism Spectrum Disorder from a nonspectrum disorder. Mazefsky and Oswald (2006) also examined the diagnostic utility and discriminative ability of the ADOS-G Original Scoring Algorithm with a clinical sample of 75 children (ranging in age from 2 to 8 years) with and without ASDs. Results of the study indicated a 77 percent agreement between participants diagnostic classifications obtained from the ADOS-G and their clinical diagnoses provided by a multidisciplinary diagnostic team consisting of a child psychiatrist, clinical psychologist, education specialist, speech/language pathologist, and occupational therapist. In addition, Ventola et al. (2006) examined the usefulness of the ADOS-G Original Scoring Algorithm in diagnosing ASDs in toddlers and young children. Based on their results, the authors reported that the ADOS-G demonstrates high levels of sensitivity and positive predictive value when used with toddlers and young children under 3 years of age. In addition, Ventola et al. indicated that they observed high levels of agreement between the diagnostic classification determinations of the ADOS-G, the classification determinations of the CARS, and diagnostic determinations made using the evaluators clinical judgments. The research of Papanikolaou et al. (2009) provides further evidence of the diagnostic accuracy of the ADOS-G Original Scoring Algorithm. Papanikolaou et al. compared the diagnostic classification determination of the ADOS-G with the clinical diagnosis of 77 children ranging in age from 2 to 22 years. According to Papanikolaou et al., results of these comparisons indicated that participants diagnostic classifications on the ADOS-G demonstrated satisfactory to excellent agreement with participants clinical diagnoses (k = ). The specificity, sensitivity, and positive predictive value of the ADOS-G s diagnostic classifications were also

29 16 calculated and examined. Although the specificity ( ) and sensitivity ( ) values were measured to be slightly lower than those reported by Lord et. al. (2000), they were still deemed to be within acceptable ranges by the authors (Papanikolaou et al). Additional investigations into the diagnostic accuracy of scores from the ADOS-G Original Scoring Algorithm provide evidence to support their use in the accurate differentiation of individuals with ASDs from those with receptive language disorders (Noterdaeme, Mildenberger, Sitter, & Amorosa, 2002) and other mental health disorders, such as mood and behavior disorders (Sikora, Hartley, McCoy, Gerrard-Morris, & Dill, 2008). However, according to Reaven, Hepburn, and Ross (2008), scores derived from the ADOS-G Original Scoring Algorithm are unable to accurately differentiate between children with an ASD and those with active psychosis. Research has also been conducted to investigate the agreement between a participant s ADOS-G Original Scoring Algorithm diagnostic classification and his or her diagnostic classification on the ADI-R. Le Couteur, Haden, Hammel, and McConachie (2008) examined the percent agreement between the diagnostic classifications of scores on the two instruments in a sample of 101 preschoolers. Results of this study indicated that the ADOS-G and ADI-R scoring algorithms yielded consistent diagnostic classifications 76 percent of the time (k =.52). Tomanik, Pearson, Loveland, Lane, and Shaw (2007) also examined the percent agreement between classification determinations of the ADOS-G Original Scoring Algorithm and the ADI- R in a sample of 129 children and adolescents. Similar to the results reported by Le Couteur et al., Tomanik et al. s results indicated agreement between the ADOS-G and ADI-R 75 percent of the time.

30 17 In summary, although the research conducted to date has adequately demonstrated the diagnostic accuracy of the Original Scoring Algorithm and has documented acceptable levels of classification agreement between the scoring algorithms on the ADOS-G and the ADI-R, other forms of reliability and validity evidence are lacking at this time. Specifically, the literature review did not yield any studies focused on the internal structure of the measurement tool. Limitations of the ADOS-G Original Scoring Algorithm. Although research indicates that scores from the ADOS-G Original Scoring Algorithm demonstrate adequate technical properties for use and, in general, accurately categorizes examinees performance (Gotham, Risi, Pickles & Lord, 2007), several criticisms of the ADOS-G Original Scoring Algorithm have been reported in the literature. The ADOS-G authors also have identified several limitations of the instrument over the last decade. Bishop and Norbury (2002) reported that the ADOS-G Original Scoring Algorithm often over-classifies individuals with specific language impairments. In a 2004 study conducted by de Bildt et al., the ADOS-G Original Scoring Algorithm demonstrated lower levels of sensitivity and specificity when used to discriminate individuals with mild mental retardation from those with an Autism Spectrum Disorder. Gotham et al. also acknowledged limitations of the ADOS-G Original Scoring Algorithm related to an examinee s cognitive ability. Specifically, they noted that the instrument currently does not take developmental cognitive ability into account when selecting a module for administration or when scoring an examinee s performance, which may result in inaccurate diagnostic classifications for those with lower mental functioning than expected base d on their chronological age. In addition, Gotham et al. reported that the Original Scoring Algorithm, which utilizes different items across modules, makes comparisons of performance across modules difficult.

31 18 Of additional concern to Gotham et al. (2007) was the Original Scoring Algorithm s lack of consideration regarding an examinee s engagement in restricted, repetitive behaviors (RRB). Although items to assess RRB are included on the ADOS-G, they were intentionally excluded from the Original Scoring Algorithm due to the authors concern over the short period of time available to observe these behaviors throughout the ADOS-G administration. However, in a review of the stability of ASD diagnoses over time, Lord et al. (2006) reported that the inclusion of RRB in diagnostic determinations, even when only observed in a limited context, independently contribute to diagnostic stability. Revised scoring algorithm for the ADOS-G. In response to the current limitations of the ADOS-G, Gotham et al. (2007) conducted a study to review and make changes to the Original Scoring Algorithm in order to (a) improve the overall diagnostic accuracy of the instrument, (b) address the identified concerns regarding the impact of cognitive ability, expressive language level, and chronological age on an examinee s performance, (c) include RRB in diagnostic determinations; and (d) increase consistency of the conceptual items included in the scoring algorithm across modules to allow for easier comparison of performance across modules. Data from 1,630 cases (i.e., complete ADOS-G administrations) were used in the study s analyses. Data were obtained from 1,139 different participants. An unidentified number of participants completed more than one ADOS-G administration, and the data from each of the administrations were included in the analyses as a separate case. Participants ranged in age from 14 to 192 months at the time of ADOS-G administration, and completed the assessment as a part of a diagnostic evaluation at a mid-western autism/communication disorders clinic or as a research study participant recruited at several sites across the U.S. Fifty-six percent of participants had a clinical diagnosis of Autistic Disorder, 27 percent were diagnosed with a

32 19 milder Autism Spectrum Disorder, and 17 percent had a diagnosis of a non-asd developmental delay. Only data from Module 1, 2, and 3 administrations were included in the analyses due to the authors beliefs that older adolescents and adults on the Autism Spectrum exhibit distinct behavior patterns and, as such, require separate examination (Gotham et al., 2007). Technical development of the ADOS-G Revised Scoring Algorithm. When generating the new diagnostic algorithms, researchers took several steps. First, they looked at the correlations between total scores on the ADOS-G and chronological age, verbal ability, and mental age of participants and then divided the sample by chronological age and language ability to create cells that minimized the correlations between total scores and demographic variables. Once the new cells were generated, the authors examined individual items within each of the modules and selected those that best differentiated between clinical diagnoses for inclusion in the new scoring algorithm. Selected items were also subjected to exploratory multi-factor item response analysis to investigate factor structure and to organize the items into domains for each of the three modules. The new models were then examined using confirmatory factor analysis (CFA), and logistic regression was used to determine the predictive value of scores from each of the identified factors to diagnostic determination. Finally, Receiver Operating Characteristic (ROC) curve analysis was conducted to determine the sensitivity (i.e., accurate positive classifications, or the percentage of participants with a clinical disorder that are accurately diagnosed as having the disorder) and specificity (i.e., accurate negative classifications, or the percentage of participants without a clinical disorder that are accurately diagnosed as not having the disorder) of the original and the newly revised scoring algorithms within each of the generated cells (Gotham et al., 2007).