Designing evaluations that work:

Transcription

1 Designing evaluations that work: Deciding where you want to go and how to get there December 11, 2012

2 Housekeeping Notes This webinar will be recorded, and will be available on the Centre s website as an educational resource The slides have been sent to participants Log in with a phone whenever possible for optimal audio quality We have staff online to assist with any technical difficulties There will be a short evaluation survey for all webinar attendees at the completion of the webinar.

3 Housekeeping Notes Addressing Questions: This webinar will be followed by a question and answer period, however questions are encouraged throughout the presentation Questions can be submitted electronically or verbally. Specifics around this process will be clarified at the end of the webinar.

4 Today s presenters Melissa Jennings, MA Manager, Evaluation Support Services Ontario Centre of Excellence for Child and Youth Mental Health Kyle Ferguson, B.Sc. Research Assistant, Evaluation and Research Ontario Centre of Excellence for Child and Youth Mental Health Fred Schmidt, Ph.D., C. Psych. Psychologist Children s Centre Thunder Bay

5 About the Centre

6 6 What is the Centre? Vision Optimal mental health and well-being for children and youth. Mission We bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth, and their families and caregivers.

7 7 What is the Centre? The Centre s strategic plan focuses on 3 key priorities: Learning Collaboration Leadership

8 The Learning Journey

9 Goals of this webinar 1. To describe three common evaluation design options 2. To explore various related concepts within each of these options 3. To help you to identify the design option that is best suited to help answer your team s evaluation questions

10 Why are we talking about evaluation design? We want to: know if our program is making a difference know what we need to change if it isn t making a difference We can do this by comparing two groups - one that gets an intervention and one that doesn t comparing conditions before and after an intervention

11 Some basic definitions Experimental/treatment group Control or comparison group Sampling Variables Validity Generalizability

12 Experimental/treatment group A group of people who participate in the treatment/program being evaluated

13 Control or comparison group A group of people who do not participate in the program being evaluated, but who are similar in most other ways (for experiments, this is a control group; for quasi-experiments, this is a comparison group)

14 Sampling Some possible sampling strategies: Random only useful with large potential participant populations (e.g. 300+) to draw from Purposive (targeted) ensures certain type of people are included in the sample Convenience smaller potential participant populations allows you to gather a range of experiences (but it is problematic to generalize from this type of sample)

15 Variables The specific concepts (or constructs) we are interested in for our evaluation Is either independent (the program/intervention) or dependent (the observed behaviours/changes) A confounding variable is a variable that is not controlled for in the program design which may be the cause of the changes we observe

16 Internal Validity Internal: the certainty with which results of an experiment/intervention can be attributed to the independent variable rather than to a confounding variable

17 Generalizability Generalizability = External validity External validity: The extent to which the results can be applied to other times, places, studies, etc. Note: Typically, as internal validity increases, external validity decreases.

18 Challenges to validity History effect Maturation effect Repeated testing effect Selection bias Attrition Regression to the mean effect Instrumentation effect

19 Three common evaluation designs

20 Comparing evaluation designs Type of evaluation design Random assignment? Control/comparison group Repeated measures Experimental Yes Yes Yes Quasi-experimental No Yes Yes Single case No No Not necessarily

21 Experimental design Randomized controlled trial (RCT) The gold standard Compares the results of two groups Experimental/treatment group Control group Participants are randomly assigned to each group Strengths: high internal validity; clear conclusions can be drawn re: cause and effect Limits: RCTs are not always feasible in clinical settings

22

23 Quasi-experimental design Compares two groups that are similar but not necessarily equivalent Treatment group Comparison group Does not require random assignment Strengths: more reasonable in clinical settings Limits: threats to internal validity; cannot establish cause and effect

24 With a comparison group(s)

25 Without comparison group

26 Single case designs Occurs when there is no comparison group and no random assignment of participants Involves looking at an individual or a group before and after an intervention to see if changes have occurred Often the best solution for small scale evaluations Strengths: Can offer a deep description of the relationship between an intervention and its effects over time Limits: Many threats to validity, does not establish cause-effect

27 Single subject design

28 Multiple baseline design

29 Questions to consider when designing your evaluation

30 An experimental design? Do we have an appropriate control group? Are we able to randomly assign participants to different conditions? Do we have appropriate tools to measure variables of interest? Do we have the time required to administer pre and post measures so that we can gain meaningful information?

31 A quasi experiment? Do we have an appropriate comparison group? Do we have appropriate tools to measure variables of interest? Do we have the time required to administer pre and post measures so that we can gain meaningful information?

32 A single case design? What kind of information are we interested in gaining? Do we have enough participants in a specified time frame to show significant results? Is it enough to know information about one point in time (i.e., post intervention)? Do we want our findings be generalizable to other similar settings? What will we do with the information we find?

33 Samples of Program Evaluation from the Field Fred Schmidt Children s Centre Thunder Bay

34 Example # 1 Brief Services Program Evaluation Identified need to evaluate Brief Services Program Intake & Brief Service programs formed an evaluation team Received support through the Centre of Excellence with a Planning and Doing Grant (Thanks!) Families were asked to participate in evaluation over a 1-year time period 53 families agreed: Follow-up results available for 37 families who completed pre and post measures

35 Example # 1 Brief Services Program Evaluation Were less severe cases assigned to Brief Services? Did clients in Brief Services improve? Collected measures related to multiple outcomes: Child (BCFPI) Adult (Depression, Anxiety Stress Scale) Caregiver Strain Use of Practice Elements by session

36 Were Less Severe Clients Assigned to Brief Services? Logistic Regression Results (N = 164) Variable B SE Wald statistic p Gender Age Urgency Level Internalizing Externalizing Managing Mood Conduct Global Family Functioning

37 Did Clients in Brief Services Improve? Outcome Measure Results (n = 37) Pre-Tx Post-Tx Measure Mean SD Mean SD Effect Size BCFPI-3 Total * Externalizing * Internalizing * DASS-21 Depression ** Stress ** Anxiety *** Total Score *** Caregiver Strain *

38 Use of Practice Elements in Sessions One and Two: Externalizing Issues

39 Use of Practice Elements in Sessions One and Two: Internalizing Issues

40 Example # 2 Evaluation of Triple P Services Data collection between October 2007 and June 2011 Collected measures at multiple time points: Before, during, and after treatment Obtained ethical approval Gave honorarium to parents who completed post-treatment measures

41 Triple P Outcome Measures Child Functioning Strengths and Difficulties Questionnaire (SDQ): Emotional, Conduct Hyperactivity Peer Problems Total Difficulties Parenting Skills Parenting Scale (PS): Laxness Over-reactivity Hostility

42 Triple P Outcome Measures Parenting Skills Parenting Sense of Competence Scale (PSOC): Efficacy Satisfaction Total Parental Well-Being Depression, Anxiety, and Stress Scale (DASS) Depression Anxiety Stress

43 Triple P Sample Characteristics Description of Triple P Participants 199 families completed pre-treatment measures 126 families completed post-treatment measures: 116 mothers and 50 fathers (63% response rate) M age = 6.83, SD = 2.93 Child gender: 69% males 54.6% of families had past/current CAS involvement

44 Triple P Selection Bias Analyses Selection Bias Analyses No difference between 126 families who completed post treatment measures and 73 who did not on: SDQ, DASS, PS, PSOC scales Child age and gender All parent/family demographics All child demographics Involvement with CAS

45 Triple P: Child Outcome Results Effect size Effect Size SDQ Subscale Mother (n=109) Father (n=49) Emotional.18*.27* Conduct.43***.18 Hyperactivity.30***.10 Peer problems.19**.19 Total difficulties.39***.18*

46 Clinical Severity Ratings Before and After Treatment (SDQ) Before Treatment (%) After Treatment (%) SDQ Scale Normal Borderline Clinical Normal Borderline Clinical Emotional Conduct Hyperactivity Peer Total Difficulties

47 Triple P: Parent Outcome Results Effect size Effect Size PS Subscale Mother (n=116) Fathers (n=50) Laxness.73***.39** Over reactivity.79***.45** Hostility.31***.41**

48 Triple P: Parent Self-Efficacy Results Parenting Skills Figure 5. PSOC results for parents completing the Triple P Program

49 Triple P: Maternal Well-being Maternal Well-Being Figure 6. DASS results for mothers who completed the Triple P Program Note: *p <.05, **p <.01, ***p <.001

50 Triple P: Client Satisfaction Client Satisfaction Figure 7. Program satisfaction for mothers and fathers who completed the Triple P Program

51 Triple P: Key Findings Significant and meaningful benefits for parents who participate in the level 4 Triple P program Changes in children s behavior fall within a small to moderate effect size range Mothers reported large effect size changes in parenting skills and parental self-efficacy and satisfaction; fathers reported small to moderate effects size changes Mothers identified small co-parenting improvements, while fathers did not identify improvements in co-parenting

52 Triple P: Key Findings Mothers reported important improvements in depression, anxiety, and stress; fathers reported improved stress, but no change in depression or anxiety Little improvement was noted in quality of couple relationship Both mothers and fathers reported a high level of satisfaction with the Triple P service

53 Program Evaluation Challenges Lack of a waitlist control group control for passage of time and regression to the mean Need for benchmarking values which replace a control group Difficulty finding appropriate outcome measures to use Failure to obtain post-treatment outcome results for an important minority of families Triple p = 37% Brief Services = 30% Limited knowledge about why a significant number of families do not complete Tx Long-term follow-up data, after discharge, not collected on families Results largely based on parent self-report

54 Contact Information Dr. Fred Schmidt Psychologist Children s Centre Thunder Bay 283 Lisgar St. Thunder Bay, Ontario P7B 6G6 fschmidt@childrenscentre.ca

55 Questions or comments?

56 Questions or comments? To submit questions electronically, use the Question box located in your control panel To submit questions verbally, use the Raised Hand icon also located in the control panel

57 Upcoming webinars To infinity and beyond: Sustaining evaluation Part 1 Melissa Jennings Kelly Lazure, Lynwood-Charlton January 8 th at 1 pm Selecting indicators and measures Ilana Smyth Julie Gamboz, Adventure Place January 22 nd at 1 pm

58 Register Archived webinars:

59 Finally Please don t forget to complete the survey at the end of this webinar. Your feedback is very important to us, so we thank you for taking the time to share your thoughts with us!

60 For more information For more information on this topic, please contact: Melissa Jennings, M.A. Manager, Evaluation Support Service Ontario Centre of Excellence for Child and Youth Mental Health Kyle Ferguson, B.Sc. Research Assistant, Evaluation Support Service Ontario Centre of Excellence for Child and Youth Mental Health