Behaviour Change Intervention Design and Evaluation: Process Evaluation

Transcription

1 Behaviour Change Intervention Design and Evaluation: Process Evaluation Charles Abraham 28 th May, 2014

2 Are We Creating Evidence-Based Interventions Relevant to Government Policy UK House of Lords Inquiry in Behaviour Change Government response, October Chaired by Baroness Julia Neuberger. Many of the goals to which governments aspire such as bringing down levels of crime, reducing unemployment, increasing savings and meeting targets for carbon emissions can be achieved only if people change their behaviour. 32 recommendations and government responses

3 Lack of Population-Level Trial Data on Long Term Effectiveness of Behaviour Change Interventions 8.2 There is a lack of applied research at a population level to support specific interventions to change the behaviour of large groups of people (including a lack of evidence on cost-effectiveness and long-term impact). This is a barrier to the formulation of evidence-based policies to change behaviour.

4 Importance of Evaluation Effective evaluation requires that: evaluation should be considered at the beginning of the policy design process. External evaluation expertise should be sought, where necessary, from the policy s inception...

5 Psychology Applied to Health (PAtH) Group University of Exeter Medical School Charles Abraham Sarah Dean Colin Greaves Mark Tarrant Mat White

6 What can Applied Psychology Contribute? What and whose behaviours need to change? What processes initiate and maintain the target behaviour pattern? What change techniques can be used to modify these regulatory processes? Can these change techniques be reliably applied in practice? Interpersonal Processes Psychological Process Environmental Influences

7 Intervention Mapping: A BCI Design and Evaluation framework Needs Assessment Proximal programme objective matrices Bartholomew, L.K., Parcel, G.S., Kok, G. & Gottlieb, N.H., & Fernández (2011).Planning Health Promotion Programmes An Intervention mapping Approach. Jossey-Bass, San Francisco. Theory-based methods and practical strategies Programme Adoption and implementation plan Evaluation plan Lloyd et al., 2011 Evidence Theory and Context Using intervention mapping to develop a school-based intervention to prevent obesity in children. Int J of Bev Nut and Phys Act 8: 73.

8 Intervention Mapping Needs assessment and elicitation research Define measurable change objectives Identification of relevant regulatory mechanisms Selection of change techniques capable to altering identified mechanisms Identification of feasible, attractive and sustainable delivery formats, Co-creation of interventions with those who will deliver and receive them Planning marketing, adoption and implementation Planning evaluation before materials are created!

9 The Ecological Model SUPRANATIONAL SOCIETY COMMUNITY ORGANISATION INTERPERSONAL INDIVIDUAL Identifying Intervention Level/s

10 Theory-Based Intervention Design? Psychological theories are usually very limited in the regulatory mechanisms they specify. So interventions based on single theories... may omit regulatory processes important to real-world health-relevant, behaviour patterns. Intervention design must be mechanism-based but not necessarily based on particular theories.

11 Identifying Mechanisms of Change Information, Motivation, Behaviour Skills Model: An Initial Grouping of Behavioural Determinants IMB: Fisher & Fisher (1992, Psych Bull) Information Behavioural Skills Behaviour Motivation

12 Information: Key Conclusions To influence action, information must be easily understood and remembered relevant to current goals, readily available in the moment of decision or action.

13 Information, Motivation, Behaviour Skills Model Information Behavioural Skills Behaviour Motivation

14 Modifiable Determinants of Motivation Fishbein et al (2001 in Baum et al. Handbook of Health Psychology ) More likely to intend if you... believe advantages (e.g., benefits) outweigh the disadvantages (e.g., costs) of behaviour attitude, anticipate a positive emotional reaction to the behaviour, see more social (normative) pressure, perceive behaviour to be consistent with self-image, and believe you are capable of performing the behaviour in a range of circumstances i.e., high self-efficacy.

15 Motivation: Key Conclusions Understand modifiable determinants of motivation for a specific behaviour pattern for a specific target populations...target these in interventions.

16 Information, Motivation, Behaviour Skills Model Information Behavioural Skills Behaviour Motivation

17 Helping Motivated, Overweight and Obese Woman to Lose Weight Research Question Does action and if-then planning help motivated people lose weight? Sample Randomised Controlled Trial of 45 women attending Weight Watchers classes. Intervention Single session add-on planning session (including if-then planning) Outcome. Weight loss two months later. Luszczynska, A., Sobczyk, A, & Abraham, C. (2007) Planning to lose weight: RCT of an implementation intention prompt to enhance weight reduction among overweight and obese women. Health Psychology, 26,

18 Luszczynska, Sobczyk & Abraham, 2007 Results 2 Months Later Control Intervention 2.1kg Lost 4.2 Kg Lost Clinically significant reduction of weight (5%): 54.2% of Intervention participants 8.3% of Control participants

19 A Useful Checklist? IMB + Automaticity Development Information (I know about X) Motivation (I want to do X) Behavioural Skills (I am able to do X) May include undermining automaticity in conflicting behavioural patterns. Automaticity-Enhancing Skills (I do X without thinking) Resulting in maintenance.

20 Writing Health Communication: An Evidence-Based Guide Abraham C & Kools M (Eds. 2012) London, SAGE Publications Ltd.

21 Campbell (2000) MRC Framework Long-term implementation Definitive RCT Theory Development Review relevant theory and evidence Modelling Identify intervention components and how they relate to health outcomes Exploratory trial Describe components of replicable intervention and feasible trial protocol Compare fully defined theory-based intervention to appropriate alternative, using theorybased, reproducible and adequately controlled protocol Determine whether intervention and results can be replicated in uncontrolled settings over the long term Continuum of increasing evidence

22 Developed Using MRC Guidance on Development and Evaluation of Complex Interventions Feasibility and Piloting Testing procedures Estimating recruitment and retention Determining sample size Development Identifying the evidence base Identifying or developing theory Modelling process and outcomes Evaluation Assessing effectiveness Understanding change process Assessing cost effectiveness Implementation Dissemination Surveillance and monitoring Long term follow up Key elements of the development and evaluation process (Craig et al., 2008)

23 EVALUATING AN INTERVENTION: KEY QUESTIONS 1. Does it work? Efficacy vs. effectiveness. Observed differences and validity power crucial. 2. How well does it work? Effect size and clinical relevance. 3. How does it work? Mechanisms and techniques mediators. 4. For whom does it work and in what settings? e.g., group and setting moderators.

24 1. DOES IT WORK? Adequate study design Adequate statistical power Process evaluation to check implementation Report all outcomes multivariate analyses Describe intervention in detail see CONSORT guidelines.

25 STUDY DESIGN Evaluate by: Internal validity Is the study able to answer the question it posed? External validity To what extent do the results apply beyond the sample studied?

26 INTERNAL VALIDITY THREATENED BY: Adequate Measures validated, reliable History and selection effects randomly allocated or matched control group Differential attrition Intention to treat analysis Researcher effects blinding

27 EXTERNAL VALIDITY GLASGOW ET AL, 2002 In general, recent studies in the healthcare area have done a much better job of reporting data related to internal validity as compared to external validity. By making explicit participation and implementation results at the organizational (as well as individual) level, the impact on public health becomes clearer.

28 External Validity Maximises Sustainability and Impact Reach Participation rate, representativeness and exclusion rate Effectiveness Intended and unintended (inc. adverse) consequences Adoption Extent to which intervention adopted by implementers Implementation Whether components delivered as intended; time and cost Maintenance By both organisations and participants Glasgow, R. E., Bull, S. S., Gillette, C., Klesges, L. M., Dzewaltowski, D. M. (2002) Behavior Change Intervention Research in Healthcare Settings: A Review of Recent Reports with Emphasis on External Validity. American Journal of Preventive Medicine, 23,

29 2. HOW WELL DOES IT WORK? Arbitrary nature of p values vs. effect sizes Clinical vs statistical significance

30 Understanding Effectiveness Measurement Cohen s d d = difference between intervention and control means/ overall variation 1.0= changed by a standard deviation 0.5 = changed by half a standard deviation 0.25 = changed by a quarter of a standard deviation 0 = no change d = M1 M2/ Pooled SD

31 3. HOW DOES IT WORK? Requires description of intervention in sufficient detail to allow replication To determine effective ingredients/ components To determine necessary settings/ populations Process measures and mediation analyses Experimental testing of components

32 QUESTIONS TO BE ANSWERED BY A PROCESS EVALUATION Was the intervention delivered as per protocol/ design across intervention groups? Did the antecedent change processes identified change (regardless of the efficacy outcome)? Did the target behaviour change? How does the trial outcome reflect on the logic model?

33 Working definition of process evaluation Context Contextual factors which shape theories of change Contextual factors which shape (moderators), and are shaped by, implementation, intervention mechanisms and outcomes. Causal mechanisms present within the context which act to sustain the status quo, or potentiate effects Logic Model / theory of change Delivery mechanisms Structures and processes by which more or less successful implementation is achieved Implementation Fidelity Dose Adaptations Reach Mechanisms of impact Participant responses Mediators Unanticipated pathways and consequences Outcomes Short, medium and long term outcomes Moore et al (forthcoming). Process evaluation of complex interventions : UK Medical Research Council (MRC) guidance. See:

34 4. FOR WHOM DOES IT WORK AND IN WHAT SETTINGS? Clear description of samples and settings Moderation analyses in individual studies and meta anlayses.

35 Davidson et al (2003) Unspecified Active Controls e.g. Usual Care efforts should be to specify the therapeutic elements that constituted usual care so that the reader can compare the intensity of usual care with the treatment intervention (p.165) Trial 1 d = 0 (ineffective) UC BCI 1 Trial 2 UC BCI 2 d = (effective)

36 Davidson et al (2003) Unspecified Active Controls e.g. Usual Care efforts should be to specify the therapeutic elements that constituted usual care so that the reader can compare the intensity of usual care with the treatment intervention (p.165) Trial 1 d = 0 (ineffective) Good UC BCI 1 Trial 2 Poor UC BCI 2 d = (effective) BUT.. BCI 2 BCI 1 d = (effective) de Bruin, M., Viechtbauer, W., Schaalma, H. P., Kok, H., Abraham, C., & Hospers, H., J. (2010). Standard care impact on effects of Highly Active Antiretroviral Therapy Adherence Interventions: A Meta-Analysis of randomized controlled trials. Annals of Internal Medicine, 170,

37 Mapping Proposed Change Processes and Corresponding Measures: UEMS HeLP Trial

38 Evaluating Intervention Evaluation Reports and Proposals What health need led to the intervention development? What research question(s) was the evaluation designed to answer? In terms of the MRC framework, what stage of intervention was evaluated? What behaviours / behaviour patterns enacted by whom were targeted? What environmental determinants and regulatory mechanisms were identified? Did the process evaluation measure whether these hypothesized antecedents changed? What change techniques were incorporated into the intervention? Were these techniques used once or repeatedly? What delivery format was used? Who delivered the intervention and what competencies/training did they require? Does the study evaluation have good internal and external validity? How good are the process, behavioural and health outcome measures employed? Did the study identify moderators of effectiveness? To what extent did reported intervention design and evaluation follow an IM approach? How well did the evaluation answer the research questions posed? How likely is the intervention to be adopted and achieve wide reach? Could the intervention be replicated and tested in another context?

39 Questions? Contact