Seeing the Forest & Trees: novel analyses of complex interventions to guide health system decision making

Transcription

1 Seeing the Forest & Trees: novel analyses of complex interventions to guide health system decision making Noah Ivers MD CCFP PhD Family Physician, Women s College Hospital Scientist, Women s College Research Institute Assistant Professor, University of Toronto

2 Acknowledgements Co-PI: Jeremy Grimshaw RC/PhD(c): Kristin Danko Research team: Justin Presseau Issa Dahabreh Tom Trikalinos Braden Manns Marcello Tonelli David Moher Tim Ramsay Kaveh Shojania Sharon Strauss Knowledge users: John Lavis Alberta: Alun Edwards, Peter Sargious Ontario: Michael Hillmer, Alison Paprica 3

3 Meta-analysis: What does the mean mean?! 4

4 Complexity of QI interventions Context Mechanism Outcomes how do we predict how well a given strategy will work? 5

5 Outline Case study: Lancet SR of QI RCTs for diabetes care: challenges with current state of the art Proposed solutions for enhancing utility for decision makers: more data, better intervention classification, better analyses Early findings: implications and next steps 6

6 Systematic review of diabetes QI strategies. 7

7 Inclusion criteria types of interventions 1. Audit and feedback 2. Case management 3. Team changes (provider role changes) 4. Electronic patient registry 5. Clinician education 6. Clinician reminders 7. Facilitated relay of information to clinicians 8. Continuous quality improvement 9. Financial incentives 10.Patient education* 11.Promotion of self-management* 12.Patient reminder systems* (* Only included if part of a multifaceted intervention including professional targeted interventions) 8

8 Meta-analysis 9

9 Conclusions QI interventions decreased population HbA1c by 0.37%, larger effects with poorer baseline control All categories of QI interventions appeared effective, difficult to disentangle optimal combination of interventions 10

10 Challenges and potential solutions Hierarchical multivariate metaregression 1) Multiple intervention components 2) Impact of contextual factors Hierarchical multivariate metaregression 4) Poor reporting 3) Unclear mechanisms of action Author survey & update Behaviour change technique taxonomy 11

11 Growing Literature, Stagnant Science?

12 Proposed solution #1: Hierarchical multivariate meta-regression 13

13 Methodological trade-offs Lancet Review used simplifications to manage multi-arm trials, combinations of QI strategies, and active control arms Choose two arms max from a trial Modeled all trials with a given QI vs all without Excluded when QI strategies in both arms Our analyses did NOT: Use all available data Directly model the effects of each QI strategy 14

14 Standard model for MA (for continuous outcomes) One row per study: Two arms / study (choose most vs least intensive in multi arm trials) Control arm effects are removed by differencing: Trial of a+b+c vs c: considered as a test of a+b Unexplained variability of the treatment effect is accounted for (between-study variance component) 15

15 Hierarchical multivariate MR of RCTs Mean of each arm is modeled Heterogeneity of mean in each arm is accounted for with random effects (all intervention (and control) components considered) Randomization precludes correlation between arms Study characteristics modeled at second level Correlation between arms (across studies) is also modeled 16

16 Univariate meta-analysis of differences Does not allow modeling of arm-level information in an intuitive way Does not permit the recovery of inter-trial information Multivariate meta-analysis of arm-specific means Allows more flexible modeling of arm-level covariates Allows the recovery of inter-trial information (may lead to bias) Multivariate decomposes the mean difference into: - within-study means (one for each arm) - between-studies level This decomposition allows for very flexible modeling with all the QI components in the arm-level Typically lead to identical inferences about treatment effects Senn, Biom J 2010; van Houwelingen and Senn, Stat Med 1999; van Houwelingen, Stat Med

17 Bayesian model fitting Used the 114 studies that reported baseline and final A1c values standard errors of the final measurements Models were fit using JAGS 100,000 iterations for burn-in and 100,000 iterations to obtain the posterior distribution of parameters of interest. Gelman-Rubin diagnostic and trace plots In all cases the upper limit of the 95% credibility interval for the Gelman-Rubin statistic was < 1.01 (typically < 1.001) for all parameters monitored. 18

18 Overall mean effect same Intervention Standard meta-analytical approach Bivariate meta-analysis of arm specific means Self management (-0.71, -0.31) [1] (-0.38, -0.03) [3] Team changes (-0.71, -0.42) [2] (-0.49, -0.17) [1] Case management (-0.65, -0.36) [3] (-0.28, 0.06) [7] Patient education (-0.61, -0.34) [4] (-0.29, 0.10) [8] Facilitated relay (-0.60, -0.33) [5] (-0.33, 0.07) [6] Patient registry (-0.61, -0.24) [6] (-0.51, -0.08) [2] Patient reminders (-0.65, -0.12) [7] (-0.26, 0.17) [9] Audit and feedback (-0.44, -0.08) [8] (-0.41, 0.11) [=4] Clinician education (-0.35, 0.03) [9] (-0.40, 0.08) [=4] Clinician reminders (-0.31, -0.02) [10] (-0.17, 0.31) [10] 19

19 Ranking the most effective QI components 20

20 Rankings stratified by baseline A1c RED = BASELINE A1c > 8; GREEN = BASELINE A1c < 8 21

21 Assessment of covariates/ interactions/effect modifiers 22

22 Predicting effects for new studies of complex interventions Prediction for the mean difference in a perfect (no imbalance) randomized trial of infinite sample size Assume the trial uses no intervention in the controls, and combines case management + team change + facilitated relay 23

23 Hierarchical multivariate meta-regressions for systematic reviews: Implications Meta-analysis recast as a (complex) regression model Benefits All data used QI components isolated Can readily model effect modifiers Better predictions Limitations Assumption of additivity GIGO 24

24 Proposed solution #2: Behaviour change technique (BCT) taxonomy 25

25 Intervention content Content descriptions in published reports vary: Similar content described with different labels o e.g., audit vs. monitoring Same label used to describe different content o e.g., education may involve information provision, goal setting, information about others approval, etc. Issue for fidelity, replication, synthesis 26

26 Cochrane EPOC QI taxonomy Advantages Pragmatic categorization system, common terms Good face validity and acceptance in the literature Relatively few categories given complexity of interventions Disadvantages Inconsistently used terminology Conflates intervention content with mode of delivery No linkage with theory 27

27 The BCT Taxonomy 28

28 Research Questions Can we code BCTs within reports of QI interventions? Can BCTs be used to code higher-level (system) interventions? Does a BCT approach add value? Are there classes of BCTs not being used that may show promise for future interventions? Are particular BCTs/groups of BCTs associated with improvement in diabetes care and outcomes? 29

29 30

30 Methods 23 papers (stratified random sample) coded independently by 3 health psychologists Coding was compared; discrepancies resolved Assumptions when coding: To code behaviour change, needed to identify target recipient (whose behaviour was changing) Coded all interventions, including system-level, as though the recipient of the intervention was the healthcare professional (and patient where applicable) Where possible, coded to the particular behaviour targeted for change Provider: e.g., prescribing, advice, exams/measurements, referral Patient: attendance to clinic, adherence to medication, self-monitor, health behaviours 31

31 32

32 33

33 BCTs used in DM2 QI Trials BCTs targeting providers Only 21 of possible 93 used (22.5%) Most frequent: Adding objects to the environment (n=14) Prompts/cues (n=11) Instructions on how to perform the behaviour (n=10) BCTs targeting patients Only 18 of of possible 93 used (19.4%) Most frequent: Prompts/cues (n=12) Instruction on how to perform the behaviour (n=10) Information about health consequences (n=8) 34

34 BCTs: Focus on Clinician Education 43 trials: only 14/93 BCTs identified Usually targeted clinical knowledge, not attitudes or skills as determinants of behaviour change BCTs identified Instruction on how to perform the behavior 4.1 Credible source 9.1 Information about health consequences 5.1 Adding objects to the environment 12.5 Goal setting (outcome) 1.3 Goal setting (behaviour) 1.1 Prompts/cues 7.1 Social support (practical) 3.2 Problem solving 1.2 Feedback on outcomes of behavior 2.7 Social support (unspecified) 3.1 Salience of consequences 5.2 Information on social & environmental consequences 5.3 Behavioral practice/rehearsal Frequency of reported use in study arms containing CE 35

35 New Taxonomy: Implications Benefits Many BCTs not used: opportunities for future interventions Provides specificity of intervention content beyond broad QI labels Limitations Doesn t address dose Poor reporting makes classification difficult Ability to explain heterogeneity and facilitate replication both unproven 36

36 Proposed solution #3: Author survey & update 37

37 Enriching the data To tackle the issues of complexity and poor reporting, we will enrich the current existing review by: Extracting additional intervention and contextual variables, guided by theory and KU input Performing tailored author survey of missing data 38

38 Framework for exploring effect modifiers plus KU input 39

39 Author survey Tailored web-based survey to trial authors Personalized questionnaire (e.g. include details from our SR database specific to each trial) Questions refined by through pilots with familiar authors Predominately closed-ended to facilitate analysis; open text boxes to allow elaboration Request for unpublished intervention materials 40

40 41

41 42

42 43

43 44

44 45

45 Author survey: Implications Potential benefit Improved description of extant literature Contribution toward reporting standards Limitations Uncertain response rate Uncertain analytical implications 46

46 Reminder: our starting point Lancet finding: wide range of interventions that are effective (on average) Goals: Explore methods to develop more informative SRs of complex interventions Predict what combinations of interventions for diabetes will be most effective under which circumstances 47

47 Improved guidance for future DM QI initiatives 1) Updated review (nearly 300 RCTs up to end of 2014!) 2) Enhance data regarding context and interventions (author survey plus application of novel taxonomies) 3) Use advanced modeling techniques using better data (hierarchical multivariate meta-regression) 4) Deliberative dialogue with stakeholders (to plan new interventions) 48

48 Potential Methodological Contributions New QI programs should be tailored based upon: Diagnostic assessment of barriers Our approach will provide guidance on effect modifiers Understanding of mechanism of action Our approach will offer new ways to identify active ingredients Empirical evidence about effects of interventions Our approach will offer evidence-informed predictions Available resources Future directions include cost-effectiveness analyses 49

49 50

50 Questions? Ask now Or ask 51