Evaluating Educational Services for Students with Autism Spectrum Disorders Erik Mayville, Ph.D., BCBA-D The Institute for Educational Planning, LLC Connecticut Center for Child Development, Inc. & mayville@iepinc.org Why evaluate? (1) Needs relevant to autism Numerous fundamental functioning domains affected Speaking and listening Social interaction Adaptive skills Self-care Employment Requires programming precision Why evaluate? (2) Adult outcomes Relatively poor Howlin et al 2013 Social outcomes in 60 adults (mean 44 years) Average nonverbal IQ 17%: good or or very good outcome. outcome. 60%: poor or or very poor outcomes. outcomes. Strongest predictor of adult outcome: Reciprocal Social Interaction domain score on the ADI. 1
Parent initiated Why evaluate (3) Questions about program appropriateness School initiated Conflict resolution Component of a regular peer review process Triennial evaluation Why evaluate? (4) Is student making appropriate progress? Variety of outcomes, range of programming needs. EIBI, primary and secondary programs. Access to evidence-based programming to influence? Why evaluate? (5) Individuals with Disabilities Education Act Sec. 300.15 - Evaluation Evaluation means procedures used in accordance with Sec. 300.304 through 300.311 to determine whether a child has a disability and the nature and extent of the special education and related services that the child needs. 2
Why evaluate? (6) Individuals with Disabilities Education Act Sec. 300.52 Independent Educational Evaluation A A parent has the right to an independent educational evaluation at public expense if the parent disagrees with an evaluation obtained by the public agency Interventions for students with ASD (1) Variety of approaches exist Often presented as if they have equal merit Importance of evidence base IDEA: Foundation of knowledge (Section 651, p. 101) Research, demonstration, & practice Interventions for students with ASD (2) IDEA (continued) (6) An effective educational system now and in the future must (G) Disseminate to teachers and other personnel serving children with disabilities research-based knowledge about successful teaching practices and models and provide technical assistance to local educational agencies and schools on how to improve results for children with disabilities (p. 102) 3
ABA & Special Education (1) Applied Behavior Analysis Among the myriad interventions that claim some degree of effectiveness with individuals with autism, interventions based upon the principles of applied behavior analysis (ABA) have the most comprehensive and sophisticated research base by far (Gerhardt, 2009) ABA & Special Education (2) National Standards Project (2009) 775 studies National Professional Development Center on ASD (2010; 2014) ABA & Special Education (3) ABA-based autism programs Application of teaching and treatment procedures consistent with Applied Behavior Analysis. Is not a a stagnate single continuum of prescribed methods (Anderson & Romanczyk, 1999, p. 167). No one approach is appropriate for every child Any intervention strategy could possibly be accepted by behavior analysts (described precisely, reproduced, demonstrated effective). ABA-based programs may vary widely in procedures. 4
Applied Behavior Analysis Applied - social significance Behavioral - measurable behavior Analytic - functional relations Technological - precise description of procedures Conceptually systematic - description of principles Effective - behavior must improve Generality - change is stable and broad ABA & Special Education (1) ABA & Problem solving (1) ABA as a problem solving model Reasons for absence of progress (e.g., consistent low rates of correct responding, variable performance, problem behavior) Solutions proposed across implementation domains for different types of progress difficulties 5
ABA & Problem Solving (2) No Skill Acquisition Inconsistent Progress Problem Behavior Skills assessment; consider prerequisites Conduct preference assessment Consider modifying reinforcement procedures Simplify or modify materials Change instructional format Evaluate and individualize prompting procedures Individualize error correction procedure Assess and address implementation variability Individualize teaching procedures Standardized Testing (1) Common assessment component One aspect of determining a child s functioning level Primary outcome measure in EIBI literature Limited as change-sensitive measures for older learners Standardized Testing (6) Clinical symptomatology Autism Autism Diagnostic Observation Schedule; Childhood Autism Rating Scale, Second Edition; Gilliam Scales; Autism Behavior Checklist Anxiety, Mood disorders Child Behavior Checklist, Behavioral Assessment System for Children, Nisonger Child Behavior Rating Form ADHD Conners Rating Scale, NCBRF Aberrant Behavior Aberrant Behavior Checklist, Behavior Problems Inventory 6
Standardized Testing (6) Social Functioning Fundamental component of ASD May be primary domain of interest in evaluation Careful assessment is rare ASD-specific social impairment ASD-Specific Specific Social Behavior Rating Scales ADOS-2 Reporting of Events Shared enjoyment in interaction. Conversation Quality of social overtures. Descriptive, conventional, instrumental, informational gestures. Quality of social response. Unusual eye contact. Amount of reciprocal social communication. Facial expressions directed to Overall quality of rapport examiner. ASD-Specific Specific Social Behavior Rating Scales (2) Social Responsiveness Scale, Second Edition Social Communication Social Cognition Is awkward in turn-taking interactions Takes things too literally and doesn t with peers. get the real meaning of the conversation. Avoids eye contact or has unusual eye contact. Has difficulty making friends, even when trying his or her best. Has trouble keeping up with the flow of a normal conversation. Is able to understand the meaning of other people s tone of voice and facial expressions. Concentrates too much on parts of things rather than seeing the whole picture. Recognizes when something is unfair. 7
Functional Assessment In IDEA (1997, 2004) Applies to students whose behavior is a manifestation of their disability Individualized evaluation of a child to assist in determining whether the child is, or continues to be, a child with a disability. The FBA process is frequently used to determine the nature and extent of the special education and related services that the child needs, including the need for a BIP. (http://idea.ed.gov/explore/view/p/,root,dynamic,qacorner,7) Program Review Tools Autism Program Quality Indicators (2001). New York Autism Networkhttp://www.p12.nysed.gov/specialed/autism/apqi.htm ASD Program Development and Evaluation Protocol (Magyar, 2006) Magyar, C. I. (2011). Developing and Evaluating Educational Programs for Students t with Autism. New York: Springer. Review Process Evaluating learning in context of evidence-based educational services Review of IEP and related documentation Observe student in educational environment, collect data Evaluate, report outcomes in context of current programming efforts. 8
Document Review Review IEP Review supporting documentation Evaluations & progress notes Curriculum development tools Program descriptions Behavior reduction plans, incident reports, HRC recommendations, or other related records Data collection procedures & graphs Motivational systems Data Collection (1) Measurement of progress through appropriate measurement dimension: Frequency Rate Accuracy Duration Latency Intensity Data Collection (2) Data should be: Recorded immediately and accurately Taken across all relevant environments Taken often enough and long enough to be representative of current performance Graphed regularly and displayed in an understandable, standard format Represent a dynamic learning system 9
Data Collection (3) Data should be: Periodically verified via inter-observer agreement by a trained observer Interpreted regularly by a qualified professional Analyzed regularly to make programming changes and decisions What should be taught? (1) Needs determined by assessments Standardized, curriculum-based Domains of functioning commonly relevant for students with ASD: Language and communication Adaptive Living and Independence Pre-academic and academic Play Skills Social skills Safety Skills Daily living and problem solving Healthy Lifestyle (e.g., diet & exercise, taking medication, routine health and dental) Observation Considerations (1) Review observation parameters: Physical space Classroom organization Time spent on what type of objective in what type of format Educational Options Survey Sundberg, M. L., & Partington, J. W. (1998). Teaching language to children with autism or other developmental disabilities.. Danville, CA.: Behavior Analysts, Inc. 10
Observation Considerations (2) Are evidence-based learning procedures observable? Is the learning process observable? Is the range of learning parameters accounted for? Acquisition, maintenance, fluency, generalization Observation Considerations (3) Is the observation sample representative of the variety and quality of typical day-to to-day activities? Range of staff? Instructional format (1:1 vs. 3:1, etc.) Phase of the learning process (acquisition, fluency, maintenance) Actual implementation of well-written written programs? Discrete trial considerations (1) Discrete Trial Instruction: the experimenter controls the occurrence of the discriminative stimulus; a stimulus is presented and a response occurs, which occasionally is followed by a reinforcer. The reinforcer in turn is followed by an interval of time under experimental control before the next discriminative stimulus is presented (Johnson & Layng, 1996, p. 283) 11
Discrete trial considerations (2) Organization of trials Rationale for procedural arrangements Probe trials Teaching trials Presentation of trials Clear, single presentation Appropriate language level Clear opportunity to respond Discrete trial considerations (3) Reinforcer delivery Identified reinforcers? Immediacy Delivery schedule Error correction strategy Corrective No Timing of correction within teaching protocol Discrete trial considerations (4) Prompt and prompt fading protocol Mastery criteria Discrimination training protocol Example: Isolation to Expansion format Data collection procedure Stimulus control factors Expansion of field size Rotation of comparison stimuli Systematic variation of sample, comparison stimuli Observing response 12
Incidental teaching considerations (1) Beginning instruction only following the individual s initiation (e.g., reaching for an out-of of-reach toy) Conducting instruction in traditionally non- instructional environments (e.g., play areas, cafeterias) Teaching mand relations in early instructional phases (i.e., requesting items and activities) Conducting generally fewer repetitions than DTI per teaching session Does it occur? Incidental teaching considerations (2) Pre-arranged environment? Prompting and prompt fading protocol Data collection If primary instructional format: Sufficient learning opportunities? Sufficient breadth in targets? Small Group Instruction Considerations (1) Quantitative: Attending behavior Eye contact with speaker, body orientation Interval recording with peer comparisons Opportunities to respond, responses Reinforcement procedures, frequency, type Number of IEP objectives addressed Number of interactions with peers (responses, initiations, exchanges) 13
Small Group Instruction Considerations (2) Qualitative characteristics: Connection between individual and small group activities Strategy in influencing group responding Careful language modifications Language rate, content Carry-over from DTI format Taubman, M., Brierly, S., Wishner, J., Baker, D., McEachin, J., & Leaf, R. B. (2001). The effectiveness of a group discrete-trial trial instructional approach for preschoolers with developmental disabilities. Research in Developmental Disabilities, 22, 205-219. 219. Inclusion considerations (1) Primary purpose Social, play skill acquisition Group learning Conditions necessary for learning evident? Reinforcement, shaping, prompting, prompt fading, etc. Skill acquisition evident? Data collection Inclusion considerations (1) Student skill level Match demands of environment? Observational learner? Sustains appropriate play for 5-10 minutes Asks and answers simple questions Stereotypic and/or disruptive behavior under stimulus control Follows one-step instructions Reciprocates greetings Responds to social reinforcers Transitions to and from activities independently Responds to delayed contingencies 14
Data Collection Targets: Inclusion setting (1) Quantitative Attending to speaker Teacher (Interval data) Child (Frequency data) Stereotypy (Frequency or interval) Opportunities to respond Data Collection Targets: Inclusion setting (2) Quantitative (continued) Responses to nonspecific language Degree of independence in managing transitions Use of self-management tools Social interactions (initiations, responses, exchanges) Hand raising (opportunities/responses) Data Collection Targets: Inclusion setting (3) Qualitative Responses to questions what does he/she say? Type of language responds to Content of exchanges between peers Subject-peer Peer-peer Degree of assistance required to complete work Transformation of purpose 15
Social programming considerations (1) Acquisition procedures Group instruction common, typically not enough Degree to which evidence-based procedures are utilized Reinforcement? Training for peers? Prompt fading? Generalization procedures Are there any? Social programming considerations (2) High Functioning Autism Spectrum Disorder Needs may fall mostly within this domain Social programming as fundamental part of a student s program. Important questions Appropriate staffing resources? Appropriate programming intensity? Running a program once or twice weekly Is there a consistent conceptualization of learning across activities? 16
Conclusion Evaluations of educational services should go beyond descriptions of standardized test data and surface descriptions of behavior, and should be detailed enough to inform specific educational and treatment directions. Thank you! 17