Demonstrating Client Improvement to Yourself and Others

Transcription

1 Demonstrating Client Improvement to Yourself and Others Making Sure Client Numbers Reflect Client Reality (Part 3 of 3) Greg Vinson, Ph.D. Senior Researcher and Evaluation Manager Center for Victims of Torture - NCB May 2009

2 Series Overview 3-Part Sequence 1. Setting Up A System (December 2008) What, When, and How to Collect and Store Data 2. Using and Understanding - Analysis (Feb 2009) How to analyze, Excel Tools, How To Interpret 3. Making Sure Numbers Reflect Reality (May 2009)

3 Different From Other Webinars Webinars are overviews and part of a larger system of resources Online Resource Center Additional Written Resources Additional Tools (e.g., Excel) PTS-Talk List-Serv Updated Resource Center Establish stronger network for evaluationrelated activities

4 Last Two Times (Dec. & Feb) Setting Up Outcome Evaluation System (Dec 2008) What, When, and How to Collect Using Outcome Evaluation System Analysis (statistics), Interpretation Available at NCB Resource Center in Archive PowerPoints Excel Tools Resource Lists

5 Today s Session 1. Describe how poor client measurement can make it look like your clients are not getting better and make your program look like it does not work 2. Use provided excel tool and/or other resources to check your own client data for common quality-related issues 3. Identify and troubleshoot common issues for your client data

6 Where we left off Client Improvement Data Pre and Post-test design Hopkins Depression Items (continuous) Categorical Indicators (e.g., Asylum seeker or not) Scored the Measures, Examined Change Analysis and Interpretation Hopkins: looked at means, t-tests, plotted charts (Excel Tool) Legal Status: looked at percentages, chisquare, made charts (Excel Tool)

7 The Reality of Client Measures and Scores/Numbers Are they what we say they are? Hopkins Depression Score Really Represent Depression? Enmeshed with Cultural Issues Language Issues Administration Issues Item Issues

8 Poorer Match to Reality, More Error Error False Conclusions Practically Make Measures Less Useful Clinically Make Your Program Look Like it is Underperforming

9 Avoid Error When Possible A Measure s Scores are Reliable and Valid Valid Reflects the thing we say they are (and that thing exists) Reliable Reflects that thing consistently Off-the-Mark: Unreliable and Invalid On the Mark: Reliable and Valid Off the Mark: But Reliable Note: Reliability is necessary but not sufficient for validity claims. You can t hit the mark if your arrows are all over the place. Reliability limits validity

10 Scale Example Like a Weight Scale, Person A actually weighs 150 lbs. Which Scale Is Better? 5 repeated measurements on Scale A: 149, 150, 151, 152, 148 or 5 repeated measurements on Scale B: 142, 158, 130, 170, 152 Which one better reflects reality? Which one will be more useful?

11 Practically, reliability gets the focus Percent of Improvers Correctly Identified (as a function of Measurement Reliability) Percent Indicated as Improving by 20% Same Clients Same Actual Improvement (all improve, a lot) Different Apparent Improvement Misidentify up to 35% of improving Clients as not Improving Will be worse for actual client data 50 Perfect Reliability Reliability Estimate Program Objective Line Clients Very Poor Reliability

12 As observed in a number of programs Typically Measurement Score Reliabilities for: Depression, Anxiety, and PTSD with torture survivors Percent Indicated as Improving by 20% Observed Reliabilities of Social Scores with Torture Survivors (scored per instructions) Percent of Improvers Correctly Identified (as a function of Measurement Reliability) Program Objective Line Reliability Estimate Clients

13 Similar for individuals scores As reliability goes down, increased range of scores possible (more error in the scores) Precision of Measurement (68% CI) Score Measure Reliability

14 Take Away In Sum, If reliability is poor, validity is necessarily poor Not Hitting The Mark Arrows all over the place Reliability and Validity often confused Reliability can be estimated Validity claims are judgments from available data and the greater literature (not a single study/effort)

15 Typically Reliability of scores is an issue 1. When used with new (untested) populations 2. When a measure isn t developed in accordance with professional standards 3. Very likely an issue with torture survivors and social outcomes measures Single Items Typically Unreliable

16 Bottom Lines 1. These issues adversely affect your clients scores, evaluation activities, and analyses irrespective of your awareness of them. 2. Measures and scoring protocols developed without technical knowledge of these issues will tend to perform poorly irrespective of face validity. 3. Single items measuring simple constructs can have reliabilities up to.70. For us, single items will tend to be much less reliable than this.

17 What to Do? Examine Reliabilities for your clients scores Especially for social outcomes Any measures you are worried about Troubleshoot Problems Revise system

18 Reliability, Estimating Estimate Reliability for your own data Internal Consistency (Cronbach s Alpha) Very Common Assumptions Multiple Items Measuring the Same Thing Continuous (e.g., 1-5) or dichotomous (0-1) Test-Retest, Interrater Reliability 0Validity/Reliability.htm

19 Excel Tool (by Del Siegle)

20 Paste In Client Responses by Each Item Measuring the Same Thing Intakes Only!!! All items must have responses Careful of missing codes (e.g., 99) Follow Format Given in Previous Webinars Top Row calculates Cronbach s Alpha

21 CVT Depression-Related Items (n = 98) Cronbach s Alpha =

22 What s a Good Reliability Estimate? Reliability Estimate Description 1.0 Perfect.90 Excellent.80 Good.70 Acceptable.60 Substandard.50 Poor Below.50 Very Poor

23 Now what? Best Guess Anxiety, Depression -.70s to.80s PTSD -.60s to.70s (possibly lower for subscales) Social Outcomes -.50s to.60s and lower Good/Acceptable Reliability Estimates (.70+) 1. Breathe Easier 2. Scores are consistent 3. Valid scores must be reliable Reliable scores are not necessarily valid Validation claims follow from repeated efforts and understanding over time

24 Under.70, Troubleshooting (I) 1.Data Errors Complete Data Entry Errors Out-of-Bound Values (e.g., 99, 8 on 5 point scale) Solution: Check Data for Errors, Correct/Implement Procedures for Data Handling, Dec webinar

25 Troubleshooting (II) 2. Invalid Construct (thing being measured) Construct doesn t exist for population Items don t really go together, aren t measuring same thing Solutions: Different Measure, Revised Scoring Protocol, Professional Consult (PTS-Talk, other), Research Literature with same/similar population, Mental Measurement Yearbook (Dec webinar)

26 Troubleshooting (III) 3. Clients don t understand measure/items Measure doesn t make sense to (some) clients Language Translation Idioms Solutions: Translate Measure, Minor Revision/Explanation, Replace Measure, Talk to Clients, Professional Consult (PTS-Talk, other), October 2007 Webinar: Measurement Cross-Cultural Context (NCB Archive)

27 Troubleshooting (IV) 4. Measures Administered Poorly or Haphazardly Communicated (includes non-verbals) that this is unimportant or just paperwork Only good clients get measures Many scheduled administrations not administered Solutions: Accountability for administrations, Capability to track completed/missed administrations, Review professional training, new training, incorporate measures into practice

28 General Tips Consult with others PTS-Talk, NCB, etc. American Evaluation Association list-serv(s) EVALTALK Literature See Previous Webinar (Measurement in Cross-Cultural Contexts, Oct. 2007) Step-by-step instructions for more in-depth analysis

29 Consult Resources (Self-Educate) Measurement and Testing (technical) Introduction to Classical and Modern Test Theory (2006). Crocker and Algina Test Theory: A Unified Treatment (1999) McDonald Measurement and Culture Research Design in Counseling (2007). Heppner, Wampold, and Kivlighan Sue, S. & Chang, J. (2003). The state of psychological assessment in Asia. Psychological Assessment, 15(3), Cross-Cultural Measurement Webinar (Oct. 2007) Measurement/Assessment and Practice Finn, S.E. & Tonsanger, M.E. (1997). Information-gathering and therapeutic models of assessment: complementary paradigms. Psychological Assessment, 9(4), Hilsenroth, M.J., Peters, E.J., & Ackerman, S.J. (2004). The development of therapeutic alliance during psychological assessment: patient and therapist perspectives across treatment. Journal of Personality Assessment, 83(3), Finn, S.E. & Tonsanger, M.E. (1992). Therapeutic effects of providing MMPI-2 test feedback to college students awaiting therapy. Psychological Assessment, 4(3),

30 When Screening/Selecting Possible Measures 1. Report Reliability Estimates 2. Validity Claims are based on: Multiple Studies/Efforts (or at least honest) Not Reliability Estimates Does not depend on Face Validity Some caution if validated or culturally-validated language is used 3. Report Development Sample Characteristics (e.g., men, women, sample size, race, ethnicity, location, etc.) 4. Report Procedure Used to Develop Measure 5. Has a Manual

31 In Sum, 1. Poor client measurement can make it look like a) your clients are not getting better and b) make your program look like it does not work 2. The provided excel tool can be used to check your own client data for reliability issues 3. Provided issues, troubleshoot solutions and provide resources

32 Demonstrating Client Improvement to Yourself and Others Series Summary 1. Setting Up A System (December 2008) What, When, and How to Collect and Store Data. Excel Tools 2. Using and Understanding - Analysis (Feb 2009) How to Analyze and Interpret Excel Tools 3. Making Sure Numbers Reflect Reality (May 2009) All these webinar, presentations, excel tools, and resources are available at the NCB Resource Center

33 Thank You Any Questions Contact Information: Greg Vinson, Ph.D To access the archives of the NCB Webinars: Go to the following website: For the Username type: NCB For the Password type: NCB Click on Resource Center From the left-hand navigation, click on Webinar Archive NCB Access, located under the Webinars, Audio, and Video heading. Note: You may have to scroll down to see this link.