What is a complex (social / public health) intervention?

Transcription

1 Randomised trials of complex public health interventions: challenges, prospects, and example: NPRI trial of the Strengthening Families Programme Laurence Moore MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

2 What is a complex (social / public health) intervention? Complex interventions involve multiple, synergistic components [Bonell et al, 2012] Complex interventions interact with context There is therefore complexity both within the intervention model AND in its dependence on context, receipt, implementation; the wider system This presents challenges for the conduct of RCTs and the value of RCT evidence MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

3 RE-AIM framework Adoption Efficacy Effectiveness Building Programs and Policies with a Large Public Health Impact Implementation Reach Maintenance MRC/CSO Social and Public Health Sciences Unit, University of Glasgow. 3

4 RECOMMENDED PURPOSE OF TRANSLATION/EFFECTIVENESS RESEARCH To determine the characteristics of interventions / policies / programmes that can: Reach large numbers of people, especially those who can most benefit Be widely adopted by different settings Be consistently implemented by staff members with moderate levels of training and expertise Produce replicable and long-lasting effects (and minimal negative impacts) at reasonable cost Glasgow RE et al Am J Public Health. 2003;93: MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

5 RE-AIM Effective interventions that are not adopted will have no impact Frequent failure of evidence based interventions which may not be transferable and cannot be implemented without commitment, engagement, resources highly unlikely that interventions that are successful in efficacy studies will do well in effectiveness studies or real world application A lot of information on efficacy and very little on effectiveness Call for a moratorium on efficacy trials (Kessler and Glasgow 2011) MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

6 Standard translational model T1, T2, bench to bedside. Efficacy, Feasibility, Pilot, Effectiveness Far from optimal for complex interventions Production line for effective interventions that generally don t work! Provides a strong basis for critique of value of RCTs of complex interventions MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

7 We need complex interventions that: Are resilient to contextual variation and therefore more transferable Flexible Standardised function, flexible form and / or Have a clear program theory and thus a clear understanding of contextual dependencies, (target group, resources, system requirements etc) and system impacts MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

8 Not just what works? but Why? Theory of the problem Causal processes Intervention / logic model Key components and their impact Program theory How will the intervention bring about change, interact with context MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

9 Realist critiques & alternatives to RCTs [e.g. Pawson & Tilley 1997; Pawson, 2013] MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

10 Realist Evaluation Pawson & Tilley Purpose of evaluation as informing the development of policy and practice Experiments can identify the mean effect but this is rarely, if ever, evenly produced Mechanism + Context = Outcome what works, for whom, and in what circumstances? and even better, to also help us to understand why? Need to develop and continually update program theory MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

11 What would a Realist(ic) RCT look like? 1. Examine the effects of intervention components separately and in combination 2. Examine pathways via which change occurs 3. Examine how effects vary by sub-groups and with context more systematically 4. Draw on complementary quantitative and qualitative data to answer different RQs. 5. Build and test mid-level, program theories about how interventions work in context MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

12 New MRC Process Evaluation Guidance is focused on what works for whom in what context & why? Context Moo Contextual factors which affect (and may be affected by) implementation, intervention mechanisms and outcomes. Causal mechanisms present within the context which act to sustain the status quo, or lead to change. Description of intervention and its causal assumptions Logic model Theory of action/ change Implementation How delivery is achieved (training, resources etc..)? What is delivered? Fidelity Dose Adaptations Reach Mechanisms of impact Participant responses Intervention mediators Unanticipated pathways / consequence Outcomes Moore et al 2015 MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

13 Explanatory - Pragmatic Explanatory Efficacy Mechanisms Standardised delivery & dose Highly specified inclusion and exclusion criteria Per protocol analysis Pragmatic Effectiveness Outcomes Black box Some variation in adherence Less restrictive inclusion/exclusion Intention to treat analysis MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

14 Pragmatic - Realistic Pragmatic Effectiveness Outcomes Black box Some variation in adherence Less restrictive inclusion/exclusion Intention to treat analysis MRC/CSO Social and Public Health Sciences Unit, University of Glasgow. Realistic Effectiveness Context, Mechanisms, Outcomes Program theory Heterogeneity Mediation and moderation analyses Mixed methods Hypothesis testing and generation

15 Realistic RCT examples: ASSIST peer-led smoking intervention 59 schools randomised, found to be effective & now rolled out across UK [Campbell et al., 2008] Fruit tuck shop trial 43 schools randomised to run fruit-only shops. Significant interaction between shops and school policy [Moore et al., 2008] Primary School Free Breakfasts Initiative 111 schools randomised and found to be effective in most deprived areas [Murphy et al., 2010] Strengthening Families Programme 748 families randomised and results due in 2014 to inform Welsh Government policy [Segrott et al., 2014] MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

16 Conclusions Complex interventions depend on and influence external factors Traditional translational model is not optimised for complex interventions; external factors are considered too late Program theories, which include external factors, rather than closed logic / intervention models, are required Exploratory (Feasibility and Pilot) studies need to assess and optimise evaluation methods AND program theory before Phase III trials are commissioned Realistic trials, which not only provide an unbiased average effect estimate but also test and refine program theory, are a potential important extension of existing pragmatic trial methods MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

17 Laurence Moore J. Segrott, D. Gillespie, I.Humphreys, J. Holliday, S. Murphy, Z. Roberts, J. Scourfield, D. Foxcroft, H. Rothwell, M.Lau, C. Hurlow, C. Phillips, H. Reed, K. Hood NPRI Randomised controlled trial of the Strengthening Families Programme UK in Wales UK: Results DECIPHer & South East Wales Trials Unit, Cardiff University, Swansea University, Oxford Brookes University, University of Glasgow

18 SFP10-14 intervention Universal substance misuse prevention intervention Delivered to groups of families with children aged Focuses on parenting, family functioning and young people s peer-resistance skills Weekly sessions of 2.5 hours for seven weeks, child & family sessions Evidence of effectiveness from US trials Validated effective program Blueprints for Healthy Youth Development provides a registry of evidence-based positive youth development programs Communities that Care

19 SFP10-14 intervention Contested evidence base Does it work? It works! but why? And for whom and under what circumstances? Limited evidence of successful transferability No formal logic model Flawed implementation - Sweden Variation in context and targeting Variable TAU Universal / high risk SFP10-14 adapted for use in UK Initially targeted at high risk families in UK implementation (Barnsley)

20 SFP10-14: group composition strategy Adapted UK programme delivered in Cardiff Identified implementation risks from comprising groups of families with high levels of challenge Developed mixed families (70/30) approach (Proportionate Universalism) Aims to form groups of (5-12 families) comprised of Families from the General Population with no challenges in a group setting (70%) Families who may experience/present challenges within a group setting (30%) Examples of challenges: young person not attending school; ADHD; low literacy skills; learning difficulties 70/30 mix aims to maximise fidelity, promote positive group dynamics/behaviour change, and maximise retention Terminology used this morning: General Population (GP) / Families with Challenges (FWC), 70/30

21 Trial Design and Implementation

22 Trial Design Pragmatic RCT with families as the unit of randomisation Embedded process and economic evaluations Comparing normal care with normal care + Strengthening Families Programme SFP10-14 delivered by agencies such as charities and local government Research fieldworkers embedded in programme delivery teams during recruitment Knowledge exchange structures built into trial from outset: strong support from, and partnership with Welsh Government

23 Trial objectives Primary Objective To ascertain the impact of the SFP10-14UK on alcohol misuse in adolescents. Secondary objective: Ascertain the impact of SFP10-14Uk on drug misuse, and smoking behaviour in adolescents school performance alcohol initiation, drink-related problems, and other alcohol-related behaviours Tertiary objectives What impact does SFP10-14UK have on mental health and well being? What impact does the SFP10-14UK have on protective factors for alcohol and tobacco use/misuse located in the family, such as family functioning, parenting and young people s peer resistance skills? What are the costs associated with the SFP10-14UK and to what extent can it be regarded as an efficient use of public funds? How can SFP10-14UK best be implemented and is there important variation in delivery and receipt?

24 Trial Outcomes Primary outcomes: number of occasions that young people report having drunk alcohol and been drunk during the last 30 days, dichotomised as never and 1-2 times or more Secondary outcomes: use of cannabis, weekly smoking, age of alcohol use initiation, frequency of drinking, frequency of different types of alcoholic drinks, drink related problems, and GCSE performance Tertiary outcomes: age of initiation of use of drugs and tobacco; family functioning, parenting, and peer pressure resistance skills in young people; wellbeing and stress; and depression.

25 Data collection from families Questionnaires with parents and young people 0 months Observation of programme sessions Telephone interviews with parents 9 months 15 months Telephone interviews with parents 24 months Questionnaires with parents & young people

26 Recruitment and Retention

27 Recruitment Family recruitment completed in June families were recruited in total: 361 families (50.5%) allocated to intervention 354 families (49.5%) allocated to control The 715 families were made up of: 919 adults (459 control, 460 SFP) 931 young people (454 control, 477 SFP)

28 Findings: Recruitment and retention 715 families randomised 931 Young People (YP), 919 Parents/Carers (PC) 354 families randomised to Control 454 YP, 459 PC 361 families randomised to SFP 477 YP, 460 PC 240 PCs completed 9 month telephone follow-up (52%) 288 PCs completed 9 month telephone follow-up (63%) 166 PCs completed 15 month telephone follow-up (36%) 220 PCs completed 15 month telephone follow-up (48%) 354 YPs completed 24 month faceto-face follow-up (78%) 316 PCs completed 24 month faceto-face follow-up (69%) 403 YPs completed 24 month faceto-face follow-up (85%) 377 PCs completed 24 month faceto-face follow-up (82%) Project SFP Cymru Main Trial Findings: UK SBM Annual Scientific Meeting, Nottingham 2014

29 24 month follow up Follow up completed July month follow up (adults): 693 adults completed 24 month interview (316 Control, 377 SFP) The final completion rate of 75.4% (68.8% Control, 82.0% SFP) 24 month follow up (young people): 757 young people completed 24 month interview (354 Control, 403 SFP) The final completion rate of 81.3% (78.0% Control, 84.5% SFP)

30 Findings: Baseline Characteristics No differences of major note between trial arms Demographic Young person (YP) Parent/carer (PC) Median age (IQR) 12 (10 to 13) 37 (32 to 43) % Female % White British Substance use in YPs % Ever tried a cigarette 27 Usually smoke > 6 cigarettes a week 5 Had a proper alcoholic drink 31 Been a little bit drunk 17 Been very drunk 7 Tried drugs 5 Substance use in PCs % Smoker 52 Never drink alcohol 19 High risk from problematic drinking (AUDIT-C) Used drugs at least once in lifetime Project SFP Cymru Main Trial Findings: UK SBM Annual Scientific Meeting, Nottingham 2014

31 Economic and Process Evaluations

32 Programme cost, Attendance C per family, variable across areas 361 families were randomised to SFP 119 (33%) attended all 7 weeks 218 (60%) received the intervention, defined as attending at least 5 sessions without missing more than 1 session in a row 74 (20%) did not attend any sessions Overall, 287 attended at least 1 session

33 Process Evaluation Described fidelity, dose delivered, dose received, reach and context Used mixed methods approach to explain variation across trial sites and key influences on implementation, and interpret outcomes Fidelity (content coverage ): facilitators reported 95.8% of activities as mostly/fully covered; for sample of sessions also observed by researchers their scores agreed with facilitators 83% of the time Group size: 84% (n=47) of programmes enrolled 5-12 families Group composition: 37% (n=21) programmes achieved intended group composition (70% families from General Population, 30% families with challenge in group setting), but remaining groups also achieved mix of families Staffing: good adherence to staffing levels and delivery of Weeks 1-7 by same staff

34 Analysis

35 Outcome Results

36 Summary Challenging trial conducted with high follow-up rates over two years and minimal risk of bias MAJOR EFFORT Pragmatic trial, good fidelity of programme delivery, although groups sometimes small and variable composition No significant differences between groups in primary outcomes CONSISTENT WITH RECENT EUROPEAN STUDIES SFP group had better parenting/family outcomes but higher anxiety/discomfort Consistent pattern of benefit for families with challenges but poorer outcomes for general population

37 Summary Realistic trial: Mediation analysis of tertiary parenting variables on outcomes Analysis of variations in programme delivery and relationship with outcomes Investigate candidate explanations for the differential effects by FWC Recommendations for logic model and future implementation Update programme theory and understanding of for whom and under what circumstances the programme s effectiveness may be maximised

38 Acknowledgements Project SFP Cymru is funded by the National Prevention Research Initiative ( Funding partners: Alzheimer s Research Trust; Alzheimer s Society; Biotechnology and Biological Sciences Research Council; British Heart Foundation; Cancer Research UK; Chief Scientist Office, Scottish Government Health Directorate; Department of Health; Diabetes UK; Economic and Social Research Council; Engineering and Physical Sciences Research Council; Health & Social Care Research & Development Office for Northern Ireland; Medical Research Council; The Stroke Association; Welsh Government; and World Cancer Research Fund Additional funding for programme delivery was provided by the Welsh Government. Cardiff Drug and Alcohol team provide financial support for recruitment through schools. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.

39 Acknowledgements: Rona Campbell, Simon Murphy, Liz Waters, Chris Roberts, Jeremy Segrott, James White, Kerry Hood, Sharon Simpson, Chris Bonell and Adam Fletcher Bonell, C., Fletcher, A. et al. (2012) Realist randomised controlled trials: a new approach to evaluating complex public-health interventions. Social Science & Medicine 75(12): Bonell, C., Fletcher, A. et al. (2013) Methods don t make assumptions, researchers do: A response to Marchal et al. Social Science & Medicine 94: Campbell, M., et al., (2000) Framework for design and analysis of complex interventions to improve health. BMJ 321(7262): Campbell, R, et al., (2008) An informal school-based peer-led intervention for smoking prevention in adolescence (ASSIST): a cluster randomised trial. Lancet 371: Craig, P., et al. (2008) Developing and evaluating complex interventions: new guidance. London: MRC. Glasgow, R., et al., (2003) Why don t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93: Kessler and Glasgow (2011) A proposal to speed translation of healthcare research into practice. Am J Prev Med 40(6): Marchal, B., et al., (2013) Realist RCTs of complex interventions - an oxymoron. Social Science & Medicine 94: Moore, G., et al., (2015) Process evaluation of complex interventions: MRC guidance BMJ 350;h1258 Moore, L, Tapper, K. (2008) The impact of school fruit tuck shops and school food policies on children's fruit consumption: a cluster randomised trial of schools in deprived areas. J Epidemiol Community Health 62: Murphy, S., et al. (2010) Free healthy breakfasts in primary schools: a cluster randomised controlled trial of a policy intervention in Wales, UK. Public Health Nutrition14(2):219. Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage. Pawson, R. (2013) The Science of Evaluation. London: Sage. Segrott, J., et al., (2014) Preventing substance misuse: study protocol for a RCT of SFP UK. BMC Public Health 14:49. MRC/CSO Social and Public Health Sciences Unit, University of Glasgow.