Methods for Evaluating Effectiveness of Implementation Strategies Using a Randomized Implementation Trial, C Hendricks Brown Director, Prevention Science and Methodology Group Director, Social Systems Informatics University of Miami Miller School of Medicine chbrown@med.miami.edu 1
Co-Authors Acknowledgements Patti Chamberlain, OSLC John Landsverk, Rady Children s Hospital Lisa Saldano, Center for Research to Practice Larry Palinkas, USC Mitsu Ogihara, U Miami Prevention Science and Methodology Group 2
Acknowledgements NIMH R01 MH076158 Chamberlain PI, Community Development Teams to Scale-Up MTFC in California NIMH P30MH074678-03 Landsverk, PI, Advanced Center to Improve Pediatric Mental Health Care NIDA R01 DA019984 Poduska, PI, A Follow-up of Classroom Services to Prevent Drug Use NIDA R21DA024370 Poduska, PI, Scaling-up Prevention Services for Early Drug Abuse Risk in School Systems NIMH/NIDA R01-MH040859, Brown PI, Methods for MH/DA Prevention and Early Intervention, Prevention Science and Methodology Group (PSMG) 3
Outline of Concepts and Methods 1. Methods Problems and New Directions for (Prevention) Implementation Science Measurement to Monitor Implementation Stage Models to Represent and Analyze Implementation Testing to Evaluate Implementation Strategies 2. Roll-Out Implementation Trials: CAL-40 Experience Testing: Randomized Implementation Trial Measure: Stages of Implementation Completion (SIC) Model: Directed Action (DATI) 3. Conclusions 4
Implementation Research Involves the use of strategies to adopt and integrate evidence-based health interventions and change practice patterns within specific settings Chambers 2008 Implementation Science Producing general, replicable knowledge regarding the success (or failure) of implementation strategies 5
1. Methods Problems and New Directions for (Prevention) EXPERIMENTAL Implementation Science Individual Interventions in Medicine 1940 s Clinical Trial Individual Level... Adaptive Intervention Group Level Interventions in Prevention 1970s Place-Based Multilevel and... Longitudinal Growth Modeling Trial System Level Interventions for Implementation 2000s???? Interactions within and across systems 6
If Randomized Trials Are to Play an Important Part in Implementation Correspondences with Existing Trials Research Questions Testable Hypotheses Efficacy, Effectiveness, Implementation? Trial Conduct and Protocols Population Sample Selection Random Assignment Evaluation Extensions of Existing Trials Real Differences with Existing Trials 7
Changing Research Questions Effectiveness vs Implementation System to Support Adoption System to Support Adoption Intervention Intervention 8
Intervention Research Phases, Including Implementation Domain of Implementation Science Sustaining Sustainability Systemwide Going Fidelity to Scale Sustainability Adoption Effectiveness Efficacy O Connell, Boat & Warner, 2009 NAS Landsverk J, Brown CH, Chamberlain P, et al. Aarons et al., in press 9
Phases of Implementation Research and Related Hypotheses Adoption increase quantity and speed of adopting interventions in communities Does Strategy A increase adoption of B? Implementation Fidelity improve the quality of program delivery Does Strategy A increase fidelity, fit or successful adaptation over B? Sustainability maintain quality and expand the quantity of program delivery Does Strategy A improve quality of delivery of Intervention Agents over time compared to B? 10
Implementation Method: Testing There is a role for randomized implementation trials, especially at this early stage of the science 11
Why Trials? There are very few Broad Street Pump Handles Left to Remove John Snow s Map of London 1849 proposed 1854 500 deaths to ~ 0 Removal of Pump Led to Immediate Reduction in Cbolera Deaths 12
Dealing with Sample Size 40 Randomized Counties still need a large effect Roll-Out Trials eventually all communities get the active intervention Cumulative Trials -- Implementation strategy evolves over time; combined analyses reflecting these changes Brown et al., 2008 ARPH, Brown et al., 2006 Clinical Trials 13
Randomized Trials Are Well Represented in Implementation Prevention : PROSPER, Communities that Care 338 Papers Child welfare/mental health implementation 9 of 338 studies had a comparison group 8 of 9 used a randomized trial Quality Improvement in Health Care Cochrane Collaboration Effective Practice and Organization of Care Review Group (EPOC) Reviews 57% exclusively Randomized Trials 1 Landsverk, Brown, Chamberlain et al. (accepted for publication) Landsverk, Brown, Rolls Reutz et al (2011) 14
2. Methods Illustrations from the CAL- 40 Implementation Trial Test Implementation of Multidimensional Treatment Foster Care in 40 Counties in California Objective : Test the effectiveness of the Community Development Team (CDT), a theory driven model to promote the adoption, implementation, and sustainability for delivering the evidence-based Multidimensional Treatment Foster Care (MTFC) intervention in California counties that are not already using MTFC. Method: Randomize counties into 6 equivalent clusters, 3 of which receive CDT, other 3 receive standard implementation. Measures: Time it takes to adopt, recruit staff, train, and place youth in MTFC homes, receive certification 15
Outcome(s) of the CAL-40 Randomized Implementation Trial How fast do counties in the two conditions adopt, implement with fidelity, and sustain the intervention? Survival Analysis Primary Outcome: Time county takes to achieve certification of sustainability. Secondary Outcome: Timing in Attaining Stages of Implementation Completion (SIC) 16
Changing Research Questions Effectiveness vs Implementation System to Support Adoption System to Support Adoption Intervention Intervention 17
Effectiveness vs Existing Implementation Supports for County, Agency, Group Home Two-Arm Trials MTFC Implementation Supports for County, Agency, Clinicians, Parent Implementation Standard Implementation Supports for County CDT Implementation Supports for County Control Condition MTFC Intervention MTFC Intervention MTFC Intervention Youth Youth Youth Youth 18
Randomly Assign 40 Counties Inclusion/Exclusion Criteria All Counties Included Except Early Adopters Counties with too few Placements One where Court Case Occurring (LA) Counties balanced on Fixed Factors (SES, Size, EPSDT), then Randomized to TIME 3 Cohorts and CONDITION of Implementation Strategy (Community Development Team, CDT, and Independent) Account for County level Clustering in CDT condition 19
COHORT 1 COHORT 2 COHORT 3 Initial CAL-40 Design 40 CA Counties CDT Stnd 26 Wait LIsted Wait Listed 13 Wait LIsted 20
Intervention Assignment Randomly Assigned to Condition and Timing (Cohort) Roll-Out Trial, Dynamic Wait-Listed Design Brown et al., 2006 Clinical Trials REMARK NEARLY IMPOSSIBLE TO HAVE A TRUE CONTROL 21
Issues in the CA-MTFC Design 1. Acceptance of the Design was Complete 2. Some Counties Were Not Ready to take Part Moved Up Counties from Next Cohort, but remained in same implementation condition 22
Consort Diagram Assessed for eligibility, n=58. Excluded n=18. - Already had implemented MTFC, n=9 - Fewer than 6 youth per year, n=8 -Los Angels County, n=1 Formed 6 equivalent clusters by matching background, n=40. Randomized the clusters to 3 cohorts and 2 conditions, n=40. Cohort 1, n=12. Cohort 2, n= 14. Cohort 3, n= 14. CDT, n=6 IND, n=6 CDT, n=7 IND, n=7 CDT, n=7 IND, n=7 - Accepted, n=4 Declined, n=2 2 counties moved up. - Accepted, n=4 -Declined, n=2 - Filled by cohort 2 counties, n=0 - Received, n=4 - Accepted, n=7 - Moved to cohort 1, n=2 - Expected to be filled by cohort 3 counties, n=2 - Expected to receive, n=7 - Invited to go early, n=3 (2 counties accepted invitation) - Accepted, n=7 - Moved to cohort 1, n=0 - Expected to receive, n=7 - Invited to go early, n=3 (no counties accepted invitation) Accepted, n=4 Declined- n=1 - Pending, n=2 - Expect to be moved to cohort 2, n=2 - Expect to be filled by declined cohort 1 counties, n=2 - Expect to receive, n=7 - Accepted, n=6 -Declined n=0 - Pending, n=1 - Expect to be filled by declined cohort 1 counties, n=2 - Expect to receive, n=9 23
EBP Advice Network by Implementation Stage & Agency Implementation Stage: Agency: Blue = Implementing (1) Square = Child Welfare (1) Red = Considering (2) Up triangle = Mental Health (2) Green = Declined to participate (3) Diamond = Probation (3) Gray = Not Applicable (8) Plus = Not Applicable (4) White = missing data (0) Circle = missing data (0) Combination of Network Survey and Interview: (Source: Palinkas 2011) 176 172 158 159 17 59 30 121 36 70 64 165 164 23 174 119 90 118 91 120 43 75 74 24 117 116 60 122 67 115 111 163 7 5 87 167 49 66 154 8 138 157 65 140 156 137 155 135 144 147 146 145 56 142 13 51 55 46 33 113 18 98 171 81 82 139 26 143 110 109 58 133 132 89 9 112 11 72 31 35 6 114 99 94 100 44 16 92 39 32 25 93 148 127 73 22 47 150 69 48 57 149 136 20 34 1 173 52 141 40 79 61 53 2 101 131 130 95 97 41 29 85 12 96 84 151 152 105 3 78 62 77 42 15 134 28 38 128 4 71 19 63 129 37 54 68 50 80 125 153 21 169 27 170 168 86 106 126 83 166 107 14 123 175 108 124 45 10 162 76 102 103 104 88 160 161 24
Exogenous Factors Running Randomized Trials During a Recession Solution: Expand trial by Adding 13 counties in a second state (Ohio), using equivalent inclusion/exclusion criteria and random assignment 25
Types of Protocol Deviations Deviations Example Response Consequence No Need to Impose Protocol Rules Counties talk to other counties assigned to different strategies Protocols were Followed When County Was not Ready to Start in their Assigned Cohort Randomly and Blindly Selected County in SAME Condition in Next Cohort So Far Little Imbalance, little indication of differential replacement bias Interactions of Counties in Different Conditions Allowed and measured Potential reduction in statistical power Protocols were Not Followed No Protocols Available Stagnancy of County Governments Budgets Extend trial to another state with same inclusion/exclusio n criteria Statistical Power 26
Measurement: SIC. Ability to Measure Implementation Milestones SIC Stages of Implementation Completion - Whether they are achieved - How long it takes to achieve - With what quality are they achieved - In what quantity are they achieved 27
Measurement: Stages of Implementation Completion (SIC) Stage by Agent (s) 8 Stages Involvement 1. Engagement System Leader 1. Considering feasibility System Leader, Agency 1. Planning/readiness System Leader, Agency 1. Staff hired and trained Agency, Practitioners 1. Fidelity monitoring process in place Practitioners, Child/Family 1. Services and consultation begin Practitioners, Child/Family 1. Fidelity, competence, & adherence Practitioners, Child/Family 1. Sustainability (certification) System, Agency, Practitioner Chamberlain et al., 2010 28
Contemplation/Pre-Adoption (Wang, Saldana et al., Implementation Science, 2010) Factors affecting time decision to consider 29
Sub-Stages Stage 3: Readiness Planning Date of cost/funding plan review (start) Date of staff sequence, timeline, hire plan review Date of Foster Parent recruitment review Date of referral criteria review Date of communication plan review Date of second in-person meeting held Date written implementation plan complete (completion) 30
Fraction Complete 0.0 0.2 0.4 0.6 0.8 1.0 Counties' Completion of Substages 1 2 3 4 5 6 7 8 Stage 31
Three Interrelated Characteristics in Monitoring Implementation 32
First Sign in a Window Cheap Fast Accurate 33
Next Sign Cheap Fast Pick any Two Accurate 34
Cost, Speed, Accuracy are Interrelated 35
Thermodynamics Ideal Gas Law (Clausius Clapeyron) P Pressure V Volume T Temperature PV = r T T = PV / r 36
Three Interrelated Characteristics in Monitoring Implementation Speed time to milestone attainment Quality how well is a program implemented Quantity how much is implemented 37
Functional Relationship like PV = rt T Time Days F -- Fidelity or Quality 20% or 80% Q -- Quantity 1 st facilitator, vs 50% T = fctn ( F, Q ) + error Time T Quantity Q Fidelity F 38
Differential Goals of Implementation Strategies CAL-40 True Goal Good Behavior Game Implementation CDC-DEBI 39
Modeling: Directed Action Represent the Implementation Strategy through intentional steps Logic Model Agent-Based Modeling 40
Directed Action. DATI Diagram Director (D) Action (A) Target (T) Intent (I) 41
Computational/Simulation Modeling Ability to Describe What the Implementation Strategy is DATI Director, Action, Target, Intention 42
Multilevel, Systemic Logic Model for Implementation CDT County Plan for Implementation MTFC Support Training Institute Standard Clinical Support Staff: Select and Train Foster Parents Foster Parent: Parenting Practices Youth: Less Runaways 43
2. Measurement of the Implementation Process A. Directed Action: Cognitive Science -> Computer Sci. Who does what to whom for what intended purpose? DATI: Director, Action, Target, Intention Director (D) Action (A) Target (T) Intention (I) 44
Director (D) Action (A) Target (T) Intention (I) MTFC Intervention Director: Clinical Support Team (Intervention Agent) Action: Deliver MTFC Support Protocol Target: Foster Parent Intention: Improve Parental Monitoring Parent Actions Director Foster Parent Action: Improve Parental Monitoring Target: Youth Intention: Decrease Runaways 45
Director (D) Action (A) Target (T) Intention (I) Agency Services Director: Agency Action: Provide MTFC Services Target: Clinical Support Team Intention: Deliver MTFC Support Protocol MTFC Intervention Director: Clinical Support Team Action: Deliver MTFC Support Protocol Target: Foster Parent Intention: Improve Parental Monitoring 46
Director (D) Action (A) Target (T) Intention (I) MTFC Support Training Director: MTFC Training Organization Action: Train Staff in Implementing MTFC Support Target: Clinical Support Team Intention: Deliver MTFC Support Protocol Agency Services Director: Clinical Support Team Action: Deliver CDT Implementation Target: Foster Parent Intention: Improve Parental Monitoring 47
Director (D) Action (A) Target (T) Intention (I) CDT Program Director: California Institute of Mental Health Broker Action: Community Development Team Target: County Social Services Intention: Arrange Training of Agency County Implementation Plan Director: County Social Services Action: Arrange Training of Agency Target: Agency Intention: Train Staff to provide MTFC Support 48
Conclusions Randomized Implementation Trials (RITs) are likely to be used, especially in this first phase of research Randomization to time of implementation, and also type of implementation strategy, since communities are not likely to tolerate non-active control. Major Outcomes of RITS are Time, Quality, Quantity; potentially generic measures like SIC could be used. There is substantially less control in RITs, so modifications in the design may need to happen. Modeling of implementation strategies will need objects like DATIs to represent and simulate implementation effects. 49
Related Papers Aarons GA, Hurlburt M, Horwitz SM. Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors. Administration and Policy in Mental Health and Mental Health Services Research. In press. Brown, C. H., P. A. Wyman, et al. (2007). "The role of randomized trials in testing interventions for the prevention of youth suicide." International Review of Psychiatry 19(6): 617-631. Brown, C. H., P. A. Wyman, et al. (2006). "Dynamic wait-listed designs for randomized trials: new designs for prevention of youth suicide." Clinical Trials 3(3): 259-271. Brown, Kellam, Muthen, Wang, Kaupert, Ogihara, Valente, McManus, Pantin, Szapocznik (in preparation). Partnerships for Effectiveness and Implementation Research: Experiences of the Prevention Science and Methodology Group Chamberlain P, Saldana L, Brown CH, Leve LD. Implementation of MTFC in California: A Randomized Trial of an Evidence-Based Practice. In M Roberts-DeGennaro & SJ Fogel (Eds.) Empirically Supported Interventions for Community and Organizational Change. Lyceum Books, Inc, In Press. Chamberlain, P., Saldana, L., Brown, H., & Leve, L. (2010). Implementation of MTFC in California: A Randomized Trial of an Evidence-Based Practice. In M. Roberts-DeGennaro, & S. J. Fogel (Eds.), Using Evidence to Inform Practice for Community and Organizational Change (pp.218 234). Chicago: Lyceum. Chamberlain, P, Marsenich L, Sosna, T, et al. (accepted for publication). Three collaborative models for scaling up evidence-based programs. Landsverk J, Brown C, Rolls Reutz J, Palinkas L, Horwitz S. Design Elements in Implementation Research: A Structured Review of Child Welfare and Child Mental Health Studies. Administration and Policy in Mental Health and Mental Health Services Research. 2011:1-10. 50
References Ideker T, Sharan R. Protein networks in disease. Genome Res 2008, 18, 644-652. Landsverk, Brown, Chamberlain, Palinkas, Horwitz (in preparation). Design and Analysis in Dissemination and Implementation Research Luke DA, Harris JK, Shelton S, et al., 2010. Systems analysis of collaboration in 5 national tobacco control networks. AJPH 100: 1290-1297. O Connell, Boat & Warner, 2009 Preventing mental, emotional, and behavioral disorders among young people: progress and possibilities NAS Palinkas and Soydan (2010). Translation and Implementation of Evidence Based Practice in Social Work: A Strategy for Research Sonsa, T., & Marsenich, L. (2006) Community Development Team Model: Supporting the model adherent implementation of programs and practices. Sacramento, CA: California Institute for Mental Health Publication. program: A baseline survey within a randomized controlled trial. Implementation Science. Valente (2010), Social Networks and Health: Models, Methods and Applications Wang, Saldana, Brown, Chamberlain (2010). Factors that influenced county system leaders to implement an evidencebased program: a baseline survey within a randomized controlled trial. Implementation Science. 51