How are Interventions Being Evaluated? How Can Evaluation Be Improved? C Hendricks Brown Hendricks.brown@northwestern.edu
Outline Mental Health and Violence 1 Evaluate What: a. Could a program work: b. Does a program work: c. Making a program work: 2. Science-Driven Programs a. Prevent Violence and Promote Mental Health b. Prevent Suicide by Referral & Treatment for MH c. Trauma Treatment + Violence Prevention 3 Cheap(er) Evaluation Designs a. Administrative Records b. Rollout Randomized Designs c. Minimal Evaluation for an Evidence-Based Program 4 Community Driven Intervention: Partnerships for Evaluation a. Community Board b. Faith Community in Chicago c. Israeli Trauma Center & US Prevention System
1. Designs to Evaluate Different Program Evaluation Questions Could program work under optimal conditions? Does program work under realistic conditions? For whom does it help or harm? How does it work? Making the program work by implementing effectively. Can we improve program and/or its delivery? Quality Improvement Communities of Color Concerned about Evaluation Costs, especially for NGOs often prohibitive
Figure 1. Stages of Research and Phases of Dissemination and Implementation 1. Designs to Evaluate Three Stages of Evaluation IOM 2009 Dissemination and Implementation Studies Adoption / Preparation Implementation Sustainment Making a Program Work Could a Program Work? Does a Program Work? Efficacy Studies Effectiveness Studies Exploration Preintervention
2. Randomized Designs for Science-Based There are not that many Broad Street Pump Handles Left to Remove John Snow s Map of London 1849 proposed 1854 500 deaths to ~ 0 Removal of Pump Led to Immediate Reduction in Colera Deaths FMHI 5
Numerous Science-Based Interventions: Randomized Evaluations of Efficacy/Effectiveness IOM Preventing MEB Disorders 2009
Across the Life Course IOM 2009
Mental Health and Violence Program Strategies Prevention 1. Early intervention to prevent both violence and mental health problems / drug disorders in youth Good Behavior Game Kellam et al., 2008 2. Prevent ongoing risk of suicide, especially among those with mental disorders Treatment Sources of Strength Wyman et al., 2010 1. Trauma and PTSD from violence exposure NATAL multilevel Trauma focused system
Using Randomized Designs to Evaluate Do Programs Work Intervention Nurse Family Partnership (Olds) Good Behavior Game (Kellam) QPR/Sources of Strength (Wyman) Outcome Child Maltreatment NOT IPV Violence, arrests Suicide attempts EtOH/Drug Dx, ASPD Suicide behavior, MH referral What was Randomized Pregnant women 1 st Grade Classroom Schools and Time Outcome Measures Adolescent follow up interviews Adult Follow up interviews School district records, web surveys Triple P (Prinz) Child abuse County Administrative records Communities that Care (Hawkins) Violence, drugs County Youth, community surveys Evaluation Cost $$$$ $$$$ $$ $$
Evaluation with Low Baserates in Violence Magnitude of the Trials Depends Heavily on Rate of Outcome Figure 1. Person-Years Required to Achieve 80% Power by Intervention Effect and Population Suicide Rate 5M 5M 2M Adolescents 2M 1M 1M 500K Rural Youth 500K Total P erso n Ye ars 100K 50K A Few Depr Sx 100K 50K 10K 5K 2K Rate Per 100,000 10 20 100 1000 5000 Hospitalized Attempters 10K 5K 2K 1K 1K 20% 30% 40% 50% IOM Intervention Effect 10 Brown et al 2007
Message Very large trials needed to evaluate effectiveness on low baserate outcomes, homicide, suicide, AIDS Alternative Strategy Two Stage Effectiveness Evaluation Strategy A) Evaluate using a more frequent intermediary suicide attempts rather than suicide B) Evaluate more distal outcomes by combining data across trials and synthesizing findings Brown et al., 2007, 2013, Perrino et al., 2013
3 Cheap(er) Evaluation Designs For Group-Based Interventions: Randomize groups to new intervention/usual care Evaluate on existing, population-based administrative records Child maltreatment: Triple-P (Prinz et al., 2008) Suicide or Homicide: National Death Index+ (NDI+) National Violent Death Reporting System (NVDRS)
Randomized Designs Can Be Accepted by Communities and Sometimes Easy and Inexpensive Randomizing groups, rather than individuals, often lower community concerns Roll-Out Designs for new interventions delivered to a group (e.g., School) Randomly assign schools to when they receive a violence intervention Group randomized -- Brown et al., 2008, 2009 Time randomized -- Brown et al., 2006, 2009 Dynamic wait-list design Stepped wedge design
Roll-Out Evaluation Design Randomize when schools receive suicide gatekeeper training preventive intervention: QPR in 32 schools In each quarter of a year, add a randomly selected small number of schools to be trained Get all school trainings on master calendar
Wait-Listed versus Roll-Out (Dynamic Wait-Listed Design) Year Time Block Wait-Listed Design Dynamic Wait-Listed Design Time Intervention Wait-Listed Intervention Wait-Listed 1 1 16 16 4 28 2 8 24 3 12 20 4 16 16 2 5 32 0 20 12 6 24 8 7 28 4 8 32 0
Rollout Design: Community Advantages 1. What Needs to be done in a roll-out design Roll-out implies that community already decided all groups will get this. Only need to add: Permission to do an evaluation and hold design in place Randomly order the training of groups Collect # outcome events at each interval of time, all groups 2. Everyone gets intervention Just as quickly as without random assignment 3. Ordering is Fair, especially for serious outcomes Early trained schools get intervention immediately Later trained schools may get a better intervention 4. More complete and efficient training IOM 16
Roll-Out Designs from Research Standpoint Improvements in statistical power over wait-listed designs Less sensitive to external factors Can be used for effectiveness (Brown et al., 2006) and for implementation (Chamberlain et al. 2008)
4. Community/NGO Delivered Programs We can t afford or manage a traditional, large Randomized Trial for effectiveness Randomized trial not always the right thing to do. West et al., 2007 We can t afford any evaluation of anything Two Reasons for Evaluation 1. How many have heard:
The Path to Heaven is Paved by Good Intentions
The Path to Hell is Paved by Good Intentions
Second Reason for Evaluation Accountability: Self-Evaluation = Monitoring and Feedback : Some systems work (others don t!) Monitoring Institutions Partnerships around evaluation/implementation Brown et al., 2012
How do you know that you are doing good, or better? Minimum to Evaluate If you use an Evidence-Based Program - You should still require evaluation of: participant engagement program fidelity
How do you know that you are doing good, or better? Minimum Needed to Evaluate If you use an Evidence-Based Program / Principles- Still require: participant engagement Attend, Satisfaction program fidelity Ratings SAMHSA s National Registry of Evidence-Based Programs and Practices (NREPP) http://www.nrepp.samhsa.gov/
Evaluation Guided by How a Program Should Work Community Context Intervention Agency Intervention Agent Fidelity Participation Target Distal Outcome Proximal Outcome
Doing Better: Quality Improvement Strategies Statistical Control Charts Problematic for Low Rates
Number of Youth Suicide Deaths from 1988 to 2002 in County 6 5 4 3 2 deaths 1 0 1988 1990 1992 1994 1996 1998 2000 2002 years 26
Doing Better: Quality Improvement Strategies Statistical control Monitor One of the Key Hypothesized Change Factors Gatekeeper Training: Attitudes and Self-Reported Behaviors
Attitudes Changed through QPR Training Wyman et al., 2008 Improvements from Training and Time Effect Size Knowledge of Warning Signs and QPR behaviors Attitudes about Suicide Prevention Self-Evaluation of Suicide Prevention Knowledge Knowledge of Clinical Resources Efficacy to Perform Gatekeeper Role Reluctance to engage with suicidal students Null Low Med High 0.46 0.89 1.06 0.99 1.22 0.29 28
Control Chart Self Efficacy for Gatekeeper Role Efficacy 5 7 9 11 5 10 15 20 Time Benneyan et al., 2003
Vigilance in Mental Health Care for Suicide Risk Among Adolescents in Mental Health Inpatient Units
Ex: 15 year follow-up of Suicidal Ideation and Behavior in Hospitalized Adolescents (Goldston, personal communication) Multiple Attempt s Attempt Plan Ideatio n None rating 1 2 3 4 5 Class 1 = 16% Class 2 = 19% Class 3 = 29% Class 4 = 36% Triangle = Mean Growth Curve 1 1 1 1 4 1 1 2 4 1 1 2 4 1 2 1 1 2 4 12 2 1 1 4 1 1 2 2 2 2 2 2 2 2 2 2 4 3 3 3 4 3 3 3 3 4 3 4 4 3 34 34 3 4 34 34 43 3 15 20 25 30 age Northwestern University 31
Evaluate a Large-Scale, Long-Term Health Service Strategy where Vigilance Depends on Risk Repeated phone/email monitoring of those in higher risk categories to date Low or negligible monitoring of those in lower risk categories Evaluate if this works compared to Standard Strategy 1 T1 T2 T3 T4 T5 National Death Index High Risk X X X X x Proportion 1 S1 Low Risk X x Proportion 2 S2 Strategy 2 All x x x Proportion S2
Making a Program Work Research: Implementation Science Generalizability Practice: Quality Improvement Local Evaluation NIDA funded Center for Prevention Implementation Methodology (Ce-PIM) Landsverk et al., 2012, Brown et al., 2014, Cheung & Duan 2013
Making a Program Work: 1. RE-AIM Perspective for a Program to Succeed Reach % community who receive program Effectiveness does program have benefit Adoption - bring into host organizations/ service delivery systems Implementation with Fidelity Maintenance (Sustainability) Succeeding means all have to be high. Glasgow et al. 2001
Violence Focus in Bronzeville Community of Chicago Violence is the Community s Identified Priority Trauma for victims and their families Treatment as Prevention Reduce Retribution Delivery of trauma treatment through faith based organizations -- NATAL Prevention targeting youth and families in the community is a complementary strategy. -- Communities that Care -- Hawkins et al., 2014
Partnerships around Violence in Chicago Bright Star Church Pastor Chis Harris Communities that Care (CTC) Process to have communities decide what Preventive Interventions would be appropriate NATAL War/Terror Related Trauma Prevention and Treatment Technical Support Northwestern Hospital, U of Chicago
NATAL s Trauma Focused Intervention NATAL Services for war and terror-related Trauma, for Israeli society Serve as a training center Serve as a Trauma Research and Knowledge Center. Is an apolitical, non profit organization Helps all Israeli citizens regardless of age, gender, religious affiliation, ethnicity and socio-economic status Over 150 therapists working all across Israel 200 volunteers in routine and emergencies NATAL has touched the lives of over 160,000 people to date 3 7
Why Evaluate in Bronzeville? Quality Improvement Communities Deserve Programs that Work and Don t Do Harm Sustainability Exportability and Scaling Up
Summary There are diverse programs addressing mental health and violence Early Prevention Programs (Good Behavior Game) Communities that Care Trauma treatment as prevention There are low burden evaluation approaches that communities could support Roll-out randomized design Administrative Records for Outcomes Minimum Needed to Evaluate fidelity and participation
NATAL References Berger, R., & Gelkopf, M. (2011). An intervention for reducing secondary traumatization and improving professional self-efficacy in well baby clinic nurses following war and terror: A random control group trial. International Journal of Nursing Studies, 48, 601-610. Berger, R., Gelkopf, M., & Heineberg, Y. (2012). A teacher-delivered intervention for adolescents exposed to ongoing and intense traumatic war-related stress: A quasi-randomized controlled study. Journal of Adolescent Health, 51, 453-461. Berger, R., Gelkopf, M., & Heineberg, Y., & Zimbardo, P. Developing resiliency and promoting tolerance toward the other among Jewish Israeli elementary school students facing ongoing rocket shelling (submitted). Bleich, A., Gelkopf, M., Berger, R., & Solomon, Z. (2008). The psychological toll of the Intifada: Symptoms of distress and coping in Israeli soldiers. International Medical Association Journal, 10, 873-879. Bleich, A., Gelkopf, M., Melamed, Y., & Solomon, Z. (2005). Emotional impact of exposure to terrorism among young-old and old-old Israeli citizens. American Journal of Geriatric Psychiatry, 13(8), 705-712. Bleich, A., Gelkopf, M., Melamed, Y., & Solomon, Z. (2006). Mental health and resiliency following 44 months of terrorism: A survey of an Israeli national representative sample. BMC Medicine, 4(21), 1-11. Bleich, A., Gelkopf, M., & Solomon, Z. (2003). Exposure to terrorism, stress-related mental health symptoms, and coping behaviors among a nationally representative sample in Israel. Journal of American Medical Association, 290(5), 612-620. Gelkopf, M., & Berger, R. (2009). A school-based, teacher-mediated prevention program (ERASE-stress) for reducing terror-related traumatic reactions in Israeli youth: a quasi-randomized controlled trial. Journal of Child Psychology and Psychiatry, 50(8), 962-971. Gelkopf, M., Berger, R., Bleich, A., & Cohen-Silver, R.( 2012). Protective factors and predictors of vulnerability to chronic stress: A comparative study of 4 communities after 7 years of continuous rocket fire. Social Science & Medicine, 74, 757-766. Gelkopf, M., Berger, R., & Roe, D. Soldiers perpetrating or witnessing acts of humiliation: A quantitative study. (submitted). Gelkopf, M., Haimov, S., & Lapid, L. A community long-term hotline therapeutic intervention model for coping with the threat and trauma of war and terror. (submitted). Gelkopf, M., Solomon, Z., Berger, R., & Bleich, A. (2008). The mental health impact of terrorism in Israel: A repeat cross-sectional study of Arabs and Jews. Acta Psychiatr Scand, 1-12. Haimov S. (2005). Telephone interventions after terror attacks. In: E. Somer, A. Bleich, (Eds). Mental health in the shadow of terror: the Israeli experience. pp.: 123-129. Tel Aviv: Ramot. (In Hebrew) *All files are available in pdf format upon request.
Other References Benneyan J, Lloyd R, Plsek P (2003). Statistical quality control as tool for research and healthcare improvement. Qual Safety Health Care, 12(6), 458-64. Brown, CH, Ten Have TR, Jo B, Dagne G, Wyman PA, Muthén BO, Gibbons RD. Adaptive Designs in Public Health. Annual Review Public Health, 30: 17.1-17.25, 2009. Brown CH, Sloboda Z, Faggiano F, Teasdale B, Keller F, Burkhart G, Vigna-Taglianti F, Howe G, Masyn K, Wang W, Muthén B, Stephens P, Grey S, Perrino T, and the Prevention Science and Methodology Group. Methods for Synthesizing Findings on Moderation Effects Across Multiple Randomized Trials. Prevention Science, 14(2): 144-156, 2013. Brown CH, Kellam SG, Kaupert S, Muthén BO, Wang W, Muthén L, Chamberlain P, PoVey C, Cady R, Valente T, Ogihara M, Prado G, Pantin H, Szapocznik J, Czaja S, McManus J. Partnerships for the Design, Conduct, and Analysis of Effectiveness, and Implementation Research: Experiences of the Prevention Science and Methodology Group. Administration and Policy in Mental Health, 39: 301-316, 2012. Brown CH, Wyman PA, Brinales JM, and Gibbons RD. The role of randomized trials in testing interventions for the prevention of youth suicide. International Review of Psychiatry.19(6): 617-631, 2007. Brown CH, Mason WA, Brown EC (2014). Translating the Intervention Approach into an Appropriate Research Design -- The Next Generation Designs for Effectiveness and Implementation Research. In Z Sloboda and H Petras (Eds.), Advances in Prevention Science: Defining Prevention Science, Springer Publishing.
Chamberlain P, Brown CH, Saldana L, Reid J, Wang W, Marsenich L, Cosna T, Padgett C. Engaging and Recruiting Counties in an Experiment on Implementing Evidence Based Practice in California. Administration and Policy in Mental Health and Mental Health Services Research, 35(4): 250-260, 2008. Chamberlain, P, Brown, CH, Saldana, L. Observational Measure of Implementation Progress: The Stages of Implementation Completion (SIC), Implementation Science, 6(116), 1-8, 2011. Cheung K and Duan N (2013). Design of Implementation Studies for Quality Improvement Programs: An Effectiveness/Cost Effectiveness Framework. AJPH epub. Glasgow RE, McKay HG, Piette JD, Reynolds KD (2001). The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Ed and Counseling: 44: 119-127. Hawkins JD, Oesterle S, Brown EC, Abbott RD, Catalano RF (2014). Youth problem behaviors 8 years after implementing the Communities that Care Prevention System: A Community-Randomized Trial. JAMA Pediatr, 168(2) 122-129. IOM 2009 Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities. Committee on Prevention of Mental Disorders and Substance Abuse Among Children, Youth, and Young Adults: Research Advances and Promising Interventions. Mary Ellen O Connell, Thomas Boat, and Kenneth E. Warner, Editors. Board on Children, Youth, and Families, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press, 2009.
Kellam SG, Brown CH, Poduska J, et al. Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug and Alcohol Dependence, S95: S5- S28, 2008. Landsverk J, Brown CH, Chamberlain P, Palinkas L, Horwitz SM, Ogihara M. Design and Analysis in Dissemination and Implementation Research. (2012). In R Brownson, G Colditz and E Proctor (Eds.), Dissemination and Implementation Research in Health: Translating Science to Practice, Oxford University Press Olds DL; Eckenrode J; Henderson CR et al. Long-term Effects of Home Visitation on Maternal Life Course and Child Abuse and Neglect: Fifteen-year Follow-up of a Randomized Trial JAMA. 1997;278(8):637-643. Olds DL; Henderson CR.; Cole R et al. Long-term Effects of Nurse Home Visitation on Children's Criminal and Antisocial Behavior: Fifteen-Year Follow-up of a Randomized Controlled Trial JAMA. 1998;280:1238-1244.
Perrino T, Howe G, Sperling A, Beardslee W, Sander I, Shern D, Pantin H, Kaupert S, Cano N, Crudin G, Bandiera F, Brown CH. Advancing Science through Collaborative Data Sharing and Synthesis. Perspectives on Psychological Science, 8(4): 433-444, 2013. Petras H, Kellam SG, Brown CH, Muthén B, Ialongo N, Poduska J. Developmental Epidemiological Courses Leading to Antisocial Personality Disorder and Violent and Criminal Behavior: Effects by Young Adulthood of a Universal Preventive Intervention in First- and Second-Grade Classrooms. Drug and Alcohol Dependence, S95: S45-S59, 2008. Prinz RJ, Sanders MR, Shapiro CJ, Whitaker DJ, Lutzker JR(2009). Population-based prevention of child maltreatment: The U.S. Triple P population trial. Prevention Science, 10, 1-12. Wilcox HC, Kellam SG, Brown CH, Poduska J Ialongo NS, Wang W, and Anthony JC. The Impact of two Universal Randomized First and Second Grade Classroom-Based Interventions on Young Adult Suicide-Related Behaviors. Drug and Alcohol Dependence: S95: S60-S73, 2008. Wyman PA, Brown CH, Inman J, Cross W, Schmeelk-Cone K, Guo J, Peña J. Randomized Trial of a Gatekeeper Training Program for Suicide Prevention: Impact on School Staff after One Year. J Consulting and Clinical Psycholology, 76(1), 104-115, 2008.
Where Do You Find Evidence-Based Interventions? Blueprints Center for Study and Prevention of Violence no Self-Directed Violence http://www.colorado.edu/cspv/blueprints/ SAMHSA s National Registry of Evidence Based Prevention Programs (NREPP) http://www.nrepp.samhsa.gov/ Violence Prevention Evidence Base- WHO Regions http://www.preventviolence.info/evidence_base.aspx Suicide Prevention Resource Center http://www.sprc.org/bpr National Child Traumatic Stress Network http://nctsn.org/