Demonstrating Client Improvement to Yourself and Others

Transcription

1 Demonstrating Client Improvement to Yourself and Others Understanding and Using your Outcome Evaluation System (Part 2 of 3) Greg Vinson, Ph.D. Senior Researcher and Evaluation Manager Center for Victims of Torture - NCB February 2009

2 Series Overview 3-Part Sequence 1. Setting Up A System (December 2008) What, When, and How to Collect and Store Data 2. Using and Understanding - Analysis (Today) 3. Audience Directed Topics/Remaining Topics (TBD/April 2009) Currently, a bit more open

3 Different From Other Webinars Webinars are overviews and part of a larger system of resources Online Resource Center Additional Written Resources Additional Tools (e.g., Excel) PTS-Talk List-Serv Updated Resource Center and Client Wiki Establish stronger network for evaluationrelated activities

4 Last Time (December) Identifying Objectives Identifying Outcomes And Measures/Indicators Gathering and Storing Data Tracking Administrations Format for Data Storage Using Excel for Data Entry Available at NCB Resource Center

5 Today s Session 1. Understand analyses related to documenting change for both continuous (e.g., depression symptoms) and categorical (e.g., legal status) client outcomes. 2. Use the provided excel tools to calculate related statistics and create charts and graphs depicting change 3. Sensibly interpret and communicate the results to others

6 Where we left off Dataset With Many Variables Pre and Post-test design Hopkins Items (continuous) Categorical Indicators (e.g., Asylum seeker or not) Scoring the Hopkins Could examine individual items (bad) Or score them (according to measure protocol) =AVERAGE(H2:W2) Highlight column below this cell, hit Cntr+D (fill) Average better than sum, takes better account for missing data.

7 Dataset

8 Two Columns of Continuous Scores Continuous (range of values) Time 1 Score for Hopkins Time 2 Score for Hopkins Fewer than Time 1 Is there a difference between the two times? Do clients get better? Often examined through average change

9

10 Time 1 and Time 2 Data Mean Comparisons Same Mean Difference, But Different Variability Around that Mean Differences in variability makes some mean differences more appreciably different than others

11 Account for the Means and Variability Paired t-test, Effect Size (Cohen s d) Both do the same thing Examine Mean Differences, Accounting for Variability Effect Size (Cohen s d) Magnitude of Difference In Standard Deviation Units t-test Difference is significantly different from 0 (i.e., if there is a difference) If t=0, the two distributions overlap completely If t is larger, get probability for distributions not overlapping The test Resource Effect size:

12 Tool (t-test and effect size) Excel Tool Just need two columns of excel data (T1 and T2) Paste Into Excel Spreadsheet Spreadsheet Calculates paired/dependent t- test and effect size Also makes some charts Pay more attention to the effect size (Cohen s d)

13 T-test/Effect Size Tool

14 Instructions and Sample Chart

15 Using This Tool Calculates Paired t-tests and Effect Sizes For Repeated Measures Only! Follows from Data Entry Files Has Instructions (second page) Produces A Chart (final page) Checks Assumptions Screens Out Any Missing Cases

16 Interpreting Guidelines Go with effect sizes (Cohen, 1988).2 small.5 medium.8 large Longitudinal will tend to be a bit inflated T-test (paired/dependent) Only if concerned with statistical test Likelihood that you observed change due to chance e.g., p <.05, p<.10, etc. Additional Resource

17 Communicating Traditional t(df) = X.X, p <.05. df = degrees of freedom N (number of clients) 1 d =.XX. APA Style Guide The 36 study participants had a mean age of 27.4 (SD = 12.6) and were significantly older than the university norm of 21.2 years, t(35) = 2.95, p = The 25 participants had an average difference from pre-test to post-test anxiety scores of -4.8 (SD = 5.5), indicating the anxiety treatment resulted in a highly significant decrease in anxiety levels, t(24) = -4.36, p =

18 Reporting, Practically Useful Report, in text, main findings Table the Rest Intake and 3-month Means, Paired Samples T-Test, and Effect Sizes (N 40) Mean at Intake Mean at 3-months t df Sig. (2- tailed) Effect Size Pair 1 Depression T1 - Depression 3-Month Pair 2 Anxiety T1 - Anxiety 3- Month Pair 3 Pair 4 Pair 5 Pair 6 Pair 7 PTSD Total T1 - PTSD Total 3-Month Somatic Total - Total Somatic T QOL Basic - T3 QOL Basic QOL Social - T3 QOL Social QOL Psych - T3 Qol Psych

19 Another Type of Data, Categorical Changes in Categorical Designation For example, asylum seeker to asylee Percentages 80% to 60% Works Pretty Well for Describing Your Sample More familiar metric

20 But, let s suppose Flipping A Coin and Measuring Heads and Tails (100 times) Expect 50% Heads, 50% Tails (50:50) Probably wouldn t get this exactly (e.g., 48:52) How far off (25:75?, 40:60?) before we start thinking something is wrong with the coin? What if 80% of clients becomes 75% of clients?

21 For Us How big of a percentage change where probably not due to chance? For us, percentages at two time points for the same clients Less important than using effect size for continuous data Percentages do a good job depicting your sample

22 Chi-Square (χ 2 ) Where we Left Off Two Columns of Categorical (Yes, No; 0,1) Variables (e.g., Asylum Seeker or Not) Paste into Tool Tool Calculates χ 2 (McNemar) Calculates Percentages Checks Some Assumptions Makes Charts

23

24

25

26 Interpreting and Communication Percentages are Familiar Report Percentages in Text Accompanied by Bar Graphs/Pie Charts For statistical significance, use Chi-Square (McNemar) statistic The sample included 32 clients. Approximately 60% met clinical criteria for depression at intake. Approximately 48% met criteria for depression at the three-month follow-up. The frequencies at the two time points were not significantly different, χ 2 (1, N = 32) =.56, p =.45.

27 Limitations Dichotomous Data (coded 1/0) only If start with text, use Find/Replace Function of Excel Pie/Bar Charts Easy to Make in Excel with Multiple Categories (Use Wizard) Beware of Data Entry Errors See Last Webinar (Dec. 2008) Statistics for Multiple Categories (use stats program) Categories generally less sensitive for detecting change than continuously scored measures

28 One Final Tool Correlation Coefficient (Pearson r) Association Between Two Variables (from same time point!) Tool calculates r, associated p-value, and makes a scatter plot = R-Square = 0.13

29 What if it doesn t work out? 1. Measurement Issues Measurement, Assessment, and Data in a Cross- Cultural Context (Oct. 2007) NCB Resource Center 2. Administration Issues 3. Methodological Issues e.g., attrition people who are getting better are leaving 4. Sample Size Issues 5. Program may not be doing what you think it is doing 6. Change (or lack thereof) may be due to factors outside of your control Evaluation Text (Ch. 10)

30 Other Resources Evaluation Text (Dec.) Rossi, P.H., Lipsey, M.W. & Freeman, H.E. (2003). Evaluation: A systematic approach. Web Resources (free) American Evaluation Association Analysis Text Pallant, J. (2007). SPSS Survival Manual: A Step by Step Guide to Data Analysis Using SPSS for Windows. Introductory Stats Textbook (Reference Only) Library, Cheap, Used Howell (Stats for Behavioral Sciences) grad level Pagano (Understanding Stats in Behavioral Sciences) undergrad level

31 Analysis Considerations Consider analysis packages other than excel More flexible More powerful SPSS, R-Commander (part of R) Can import excel data files Measurement, Assessment, and Data in a Cross-Cultural Context (Oct. 2007) NCB Resource Center Nonetheless, the provided tools will do what was shown for free!

32 In Summary, for Part 2 of 3 1. Followed from previous webinar in series 2. Covered some statistical analyses Just another type of summary 3. Provided Tools to Calculate these Stats Paired t-test, effect sizes Chi-square 4. Questions about these issues are very acceptable on PTS-Talk list-serv!

33 Recap Series So Far Part One (Dec. 2008) Set Up System Collect, Track, and Enter Data Part Two (Feb. 2009) Score Data Analyze, Interpret, and Communicate

34 Next Time (Part 3 of 3) One of Two Tracks Broader Scope of Evaluation Organizational Buy In Staff Buy In Using Data (Not Just for Funders) Other Topics (Let me Know) Lessons Learned in the Field Discussion Format Measurement Issues

35 Thank You Feel free to contact me if you have any questions.