Understanding CELF-5 Reliability & Validity to Improve Diagnostic Decisions

Size: px

Start display at page:

Download "Understanding CELF-5 Reliability & Validity to Improve Diagnostic Decisions"

Dortha Foster
6 years ago
Views:

Understanding CELF-5 Reliability & Validity to Improve Diagnostic Decisions Senior Educational Consultant Pearson Disclosures Dr. Scheller is an employee of Pearson, publisher of the CELF-5.

1 Understanding CELF-5 Reliability & Validity to Improve Diagnostic Decisions Senior Educational Consultant Pearson Disclosures Dr. Scheller is an employee of Pearson, publisher of the CELF-5. No other language assessments will be presented in this presentation. 2 Copyright Pearson Education and its Affiliates. All rights reserved 1 Copyright Pearson, Inc., or its affiliates. All rights reserved.

2 Agenda 1. Introduction 1. Disclosure 2. Agenda 3. Learning Objectives 2. Research Overview 1. Standardization 2. Reliability 3. Validity 3. Summary/Questions 3 Copyright Pearson Education and its Affiliates. All rights reserved Learning Outcomes 1. Name at least one study conducted to evaluate the reliability of the CELF Describe two procedures used to evaluate test score and index score differences to determine if the differences are significant. 3. Describe the average difference of CELF-4 and CELF- 5 scores in a study conducted with typically developing students. 4. Identify the sensitivity/specificity on a chart using the cut score used in the clinicians place of employment. 5. Name the optimal cut score on CELF-5 that provides the best balance between the sensitivity and specificity measures. 4 Copyright Pearson Education and its Affiliates. All rights reserved 2 Copyright Pearson, Inc., or its affiliates. All rights reserved.

3 Research Overview Research Overview Multiple research phases Over 4000 students tested in standardization and related reliability and validity studies Students tested from March through December 2012 Over 450 SLPs across the U.S. participated in standardization testing 6 Copyright Pearson Education and its Affiliates. All rights reserved 3 Copyright Pearson, Inc., or its affiliates. All rights reserved.

4 Multiple Bias Studies Multiple phases of objective and subjective reviews of administration directions, cues, test items, and test formats Assessment/bias experts examined test items for potential bias related to Socioeconomic status Race/Ethnicity Gender Culture Region Clinicians in the field provided feedback about students responses and engagement in test tasks Statistical analysis of bias verified or refuted subjective bias concerns 7 Copyright Pearson Education and its Affiliates. All rights reserved A Diverse STDZ Sample: Race/Ethnicity Race/Ethnicity N % Asian % Black % Hispanic % Other % White % Total Sample % 8 Copyright Pearson Education and its Affiliates. All rights reserved 4 Copyright Pearson, Inc., or its affiliates. All rights reserved.

5 A Diverse STDZ Sample: Parent Education Level Parent Education Level N % Less than 11 years % H.S. Diploma or GED % 1-3 Years College or Technical School 4 or more Years of College % % Total Sample % 9 Copyright Pearson Education and its Affiliates. All rights reserved A Diverse STDZ Sample: Region Region of the US N % Midwest % Northeast % South % West % Total Sample % 10 Copyright Pearson Education and its Affiliates. All rights reserved 5 Copyright Pearson, Inc., or its affiliates. All rights reserved.

6 Other Demographic Variables Gender N % Female % Male % Total Sample % 11 Copyright Pearson Education and its Affiliates. All rights reserved Raw Score Means and SDs 12 Copyright Pearson Education and its Affiliates. All rights reserved 6 Copyright Pearson, Inc., or its affiliates. All rights reserved.

Raw Score Means and SDs (cont.) 13 Copyright 2013. Pearson Education and its Affiliates.

8 Reliability 101 How confident are you in the accuracy of a test score? Reliability = accuracy, consistency and stability of test scores across situations True Score = Observed Score + Error Errors are systemic and random 15 Copyright Pearson Education and its Affiliates. All rights reserved Internal Consistency & Test-retest Internal consistency: estimates how consistently the items of the test measure one construct (homogeneity) Split-half method (Spearman-Brown correction): correlation between the total scores of two half-tests Test-retest stability: correlation between test and retest scores. Time interval between the test and retest is as short as possible. 16 Copyright Pearson Education and its Affiliates. All rights reserved 8 Copyright Pearson, Inc., or its affiliates. All rights reserved.

9 Test Reliability Coefficients CELF-5 Test Sentence Comprehension.87 Good Linguistic Concepts.91 Excellent Word Structure.89 Good Word Classes.90 Excellent Following Directions.91 Excellent Formulated Sentences.86 Good Recalling Sentences.94 Excellent Understanding Spoken Paragraphs.85 Good 17 Copyright Pearson Education and its Affiliates. All rights reserved Average Reliability Coefficients (across target ages) Word Definitions.89 Good Sentence Assembly.93 Excellent Semantic Relationships.89 Good Pragmatics Profile.98 Excellent Reading Comprehension.87 Good Structured Writing.75 Acceptable Index Score Reliability Coefficients CELF-5 Index Scores Core Language Score Receptive Language Index Expressive Language Index Language Content Index Language Structure Index Language Memory Index Average Reliability Coefficients (across target ages).96 Excellent.95 Excellent.95 Excellent.95 Excellent.96 Excellent.95 Excellent 18 Copyright Pearson Education and its Affiliates. All rights reserved 9 Copyright Pearson, Inc., or its affiliates. All rights reserved.

10 Clinical Group Test Sentence Comprehension Linguistic Concepts Language Disorder (n=166) Reliabilities for Clinical Groups Learning Disability (Reading & Writing) (n=69) Autism Spectrum Disorder (n=66) Average r xx.94 Excellent Excellent.95 Excellent.96 Excellent Excellent.97 Excellent Word Structure.93 Excellent Excellent.94 Excellent Word Classes.96 Excellent.92 Excellent.97 Excellent.95 Excellent Following Directions.96 Excellent.90 Excellent.98 Excellent.96 Excellent Formulated Sentences.97 Excellent.89 Good.96 Excellent.95 Excellent Recalling Sentences.98 Excellent.92 Excellent.97 Excellent.96 Excellent 19 Copyright Pearson Education and its Affiliates. All rights reserved Clinical Group Test Understanding Spoken Paragraphs Word Definitions Sentence Assembly Semantic Relationships Pragmatics Profile Reading Comprehension Language Disorder (n=166) Reliabilities for Clinical Groups (cont.) Learning Disability (Reading & Writing) (n=69) Autism Spectrum Disorder (n=66) Average r xx.81 Good.75 Acceptable.91 Excellent.84 Good.87 Good.91 Excellent.95 Excellent.92 Excellent.92 Excellent.94 Excellent.97 Excellent.95 Excellent.88 Good.89 Good.96 Excellent.92 Excellent.99 Excellent.99 Excellent.99 Excellent.99 Excellent.93 Excellent.86 Good.93 Excellent.91 Excellent 20 Copyright Pearson Education and its Affiliates. All rights reserved 10 Copyright Pearson, Inc., or its affiliates. All rights reserved.

11 Standard Error of Measurement (SEM) & Confidence Interval 21 Copyright Pearson Education and its Affiliates. All rights reserved Standard Error of Measurement (SEM) & Confidence Interval (cont.) 22 Copyright Pearson Education and its Affiliates. All rights reserved 11 Copyright Pearson, Inc., or its affiliates. All rights reserved.

12 Test-retest Stability Test Retest Stability (n= 137) Test-retest interval: 7-46 days Mean: 19 days 23 Copyright Pearson Education and its Affiliates. All rights reserved Inter-scorer Agreement Inter-scorer agreement Reliability Word Structure.99 Formulated Sentences.95 Word Definitions.91 Structured Writing Copyright Pearson Education and its Affiliates. All rights reserved 12 Copyright Pearson, Inc., or its affiliates. All rights reserved.

13 Score Differences Interpretation of performance: 1. Examine if difference is statistically significant Reflection of Standard Error 2. Examine if difference is clinically significant (rare) 25 Copyright Pearson Education and its Affiliates. All rights reserved Validity 13 Copyright Pearson, Inc., or its affiliates. All rights reserved.

14 Validity 101 How well can your test results predict the presence of a disorder (or, predict one s skill)? Validity is demonstrated through evidence supporting a test s interpretations and uses. Validity Evidence: the degree to which specific data, research, or theory supports that: 1. A test measures the concepts it s supposed to measure 2. The test is applicable to its intended population 27 Copyright Pearson Education and its Affiliates. All rights reserved Evidence Based on Test Content Validity evidence related to test content Content Relevance: when the content areas being measured are accepted as relating to the proposed construct Content Coverage: when the content areas measured by the test are accepted to be an adequate sampling of these areas (Also developmentally appropriate) Validity evidence includes: literature review; users feedback; and expert review CELF-5 designed to measure 28 Copyright Pearson Education and its Affiliates. All rights reserved 14 Copyright Pearson, Inc., or its affiliates. All rights reserved.

15 Intercorrelation Studies 29 Copyright Pearson Education and its Affiliates. All rights reserved Relationship with Other Variables (CELF-4) 30 Copyright Pearson Education and its Affiliates. All rights reserved 15 Copyright Pearson, Inc., or its affiliates. All rights reserved.

16 CELF-5 CELF-4 (cont.) 31 Copyright Pearson Education and its Affiliates. All rights reserved Language Disorder Study 32 Copyright Pearson Education and its Affiliates. All rights reserved 16 Copyright Pearson, Inc., or its affiliates. All rights reserved.

17 LD Study (cont.) 33 Copyright Pearson Education and its Affiliates. All rights reserved Sensitivity and Specificity 101 Sensitivity: The probability that someone who has the condition will test positive for it. Specificity: The probability that someone who does not have the condition will test negative. Errors False Positive: A student who is falsely identified as having a condition or disorder. False Negative: A student with a condition or disorder who is not correctly identified by a test (the most serious error). 34 Copyright Pearson Education and its Affiliates. All rights reserved 17 Copyright Pearson, Inc., or its affiliates. All rights reserved.

18 CELF-5 Diagnostic Accuracy 35 Copyright Pearson Education and its Affiliates. All rights reserved Hypothetical using 1.5 SD (77) cutoff with CELF-5: 10,000 Students, 10% Prevalence Language Disorder (.85) No Language Disorder (.99) Total Positive Test Results Negative Test Results 150 8,910 9,060 Total 1,000 9,000 10, Copyright Pearson Education and its Affiliates. All rights reserved 18 Copyright Pearson, Inc., or its affiliates. All rights reserved.

19 Another Example: October is Breast Cancer Awareness Month!! (These numbers are approximates and based a 12% prevalence rate) Breast Cancer No Breast Cancer Total Positive Mammogram Results Negative Mammogram Results Total ( 17%) 120 ( 12%) 62 ( 7%) , Copyright Pearson Education and its Affiliates. All rights reserved Learning Disorder (Reading and/or Writing) 38 Copyright Pearson Education and its Affiliates. All rights reserved 19 Copyright Pearson, Inc., or its affiliates. All rights reserved.

20 Autism Spectrum Disorder 39 Copyright Pearson Education and its Affiliates. All rights reserved Summary/Questions? Reliability is reflection of error Validity helps us determine how we apply test interpretations and predict Choosing cutoff has implications for including or not including people in groups. 40 Copyright Pearson Education and its Affiliates. All rights reserved 20 Copyright Pearson, Inc., or its affiliates. All rights reserved.

com 1-866-335-8418 (Canada) PearsonAssess.ca Facebook.

21 CELF5Family.PearsonClinical.com PEARSON Customer Service Twitter.com/SpeechnLanguage (USA) PearsonClinical.com (Canada) PearsonAssess.ca Facebook.com/SpeechandLanguage pinterest.com/speechandlang/the-new-celf-5/ 21 Copyright Pearson, Inc., or its affiliates. All rights reserved.

Developmental Assessment of Young Children Second Edition (DAYC-2) Summary Report

Developmental Assessment of Young Children Second Edition (DAYC-2) Summary Report Section 1. Identifying Information Name: Marcos Sanders Gender: M Date of Testing: 05-10-2011 Date of Birth: 09-15-2009