The Second David Sackett Symposium. Peter Tugwell. (Non-)randomized studies in Systematic Reviews including SRs on equity.

Transcription

1 The Second David Sackett Symposium Peter Tugwell (Non-)randomized studies in Systematic Reviews including SRs on equity.

2 Campbell and Cochrane Collaborations are continualy struggling with how to provide the SR evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works

3 Campbell and Cochrane Collaborations are continualy struggling with how to provide the SR evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works

4 Campbell and Cochrane Collaborations are continualy struggling with how to provide the SR evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit?

5 Campbell and Cochrane Collaborations are continualy struggling with how to provide the SR evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views

6 Campbell and Cochrane Collaborations are continualy struggling with how to provide the SR evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views Synthesis of the likely diversity of effectiveness-equity This has produced some major Creative tensions The Lightening Rod is Study Design Is Cochrane only for RCT Randomistas Let me take you through some of the debate with the other disciplines

7 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No Lack of awareness that Non-Randomised Studies have arrived on the Cochrane Library!

8 Perhaps the most commonly cited Cochrane NRS Review. 2000

9 Pool fencing for preventing drowning of children Thompson DC, Rivara FP CDSR 2000 Objectives To determine if pool fencing prevents drowning in children (under 14 years of age). Selection criteria Studies evaluating pool fencing in a defined population and measuring the risk of drowning or near-drowning in fenced and unfenced pools. Main results Three case-control studies met the selection criteria. Pool fencing significantly reduces the risk of drowning. OR for the risk of drowning or near drowning is 0.27 (95% CI 0.16 to 0.47). Isolation fencing (enclosing pool only) is superior to perimeter fencing (enclosing property and pool); OR for the risk of drowning in a pool with isolation fencing compared to a pool with three-sided fencing is 0.17 (95% CI 0.07 to 0.44). Authors' conclusions Pool fences should have a dynamic and secure gate and should isolate the pool from the house. Legislation should require isolation fencing with secure, selflatching gates for all pools, public, semi-public and private. Legislation should require fencing of both newly constructed and existing pools and include enforcement provisions, in order to be effective.

10 Sunday 22 September 2013 Jos Verbeek 202 Cochrane reviews containing Non-Randomised Studies.. quasi-rcts (45%), CCT (55%), Controlled-before after studies (60%), interrupted time-series (52%), cohort-studies (37%), case-control studies (26%) Risk of bias assessment of NRS varies. Used a variety of checklists often self-constructed tools. Robust ROB urgently needed

11 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 2 Iain Chalmers points out that SRs are Observatiional Systematic Searching for studies relevant to the question is the hallmark of Cochrane IC uses them in his EBHC talks! Uses SIDS Back to Sleep example in his talks to emphasize the importance of Systematic Searching [with reproducible search] rather than one design..

12

13 Extract from publicity prepared for the UK Reduce the Risk Campaign in the early 1990s The risk of cot death is reduced if babies are NOT put on the tummy to sleep. Place your baby on the back to sleep...healthy babies placed on their backs are not more likely to choke.

14 Design. ITS: Interrupted Time Series

15 Reasons/events why the interest in expanding the scope of methods relevant to Health Policy and Systems Research in Cochrane, Campbell and others No 3 Cochrane EPOC (Effective Practice and Organisation of Care) Review Group Accept CCT, ITS, CBA.

16 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Relationship-Building :Cochrane/Campbell Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.;

17 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Increasing Deliberative Dialogue :Cochrane Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.; 2009 Freiburg Cochrane Colloquium final plenary.

18 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Increasing Deliberative Dialogue :Cochrane Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.; 2009 Freiburg Cochrane Colloquium final plenary Ottawa Non Randomised Studies Meeting

19 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Increasing Deliberative Dialogue :Cochrane Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.; 2009 Freiburg Cochrane Colloquium final plenary Ottawa Non Randomised Studies Meeting 2012 Bellagio PRISMA Equity meeting

20 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Increasing Deliberative Dialogue :Cochrane Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.; 2009 Freiburg Cochrane Colloquium final plenary Ottawa Non Randomised Studies Meeting 2012 Bellagio PRISMA Equity meeting 2012 Montebello Complex Interventions Meeting

21 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Increasing Deliberative Dialogue :Cochrane Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.; 2009 Freiburg Cochrane Colloquium final plenary Ottawa Non Randomised Studies Meeting 2012 Bellagio PRISMA Equity meeting 2012 Complex Interventions Meeting Montebello 2013 March Oxford Cochrane NRS Risk of Bias

22 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Increasing Deliberative Dialogue :Cochrane Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.; 2009 Freiburg Cochrane Colloquium final plenary Ottawa Non Randomised Studies Meeting 2012 Bellagio PRISMA Equity meeting 2012 Complex Interventions Meeting Montebello 2013 March Oxford Cochrane NRS Risk of Bias 2013 Sept Quebec City Cochrane NRS Risk of Bias

23 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Increasing Deliberative Dialogue :Cochrane Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.; 2009 Freiburg Cochrane Colloquium final plenary Ottawa Non Randomised Studies Meeting 2012 Bellagio PRISMA Equity meeting 2012 Complex Interventions Meeting Montebello 2013 March Oxford Cochrane NRS Risk of Bias 2013 Sept Quebec City Cochrane NRS Risk of Bias 2013 November Harvard Quasi Expt Design Mtg

24 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 5 Public Health Review topics with no RCTs eg : Helmets vs no Helmets for motorcycles.

25 Helmets vs no Helmets for motorcycles Helmets are shown to reduce motorcyclist head injury and death Liu BC, Ivers R, Norton R, Boufous S, Blows S, Lo SK January 21, 2009 Motorcyclists are at high risk in traffic crashes, particularly for head injury. Main results: Sixty-one observational studies : Cohort, Case control and Cross Sectional. Despite methodological differences there was a remarkable consistency in results, particularly for death and head injury outcomes. Conclusion: helmets reduce the risk of head injury by around 69% and death by around 42%. The review supports the view that helmet use should be actively encouraged worldwide for rider safety -

26

27 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 6. Stop ignoring Adverse Effects: Cochrane Handbook,GRADE and Decision Aid Movement: Present Balance of Benefit and Harm Holger has covered GRADE The Cochrane Patient : Patient Decision Aid Movement is now linking with Cochrane

28 Require both Benefits and Harms Should be Balanced The Cochrane Patient 28

29 Cochrane Review Etanercept Decision Aid What are the benefits? BMJ 2011

30 Cochrane Review Etanercept Decision Aid : Side Effects and Harms BMJ 2011;343:d4027

31 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 7 New C1 and C2 Methods and Review Groups focussing on mixed methods Campbell and Cochrane Equity Methods Group.

32 Equity Mtgs

33 The Campbell and Cochrane Equity Methods Group

34 Campbell & Cochrane Equity Methods Group We aim to develop methods to improve the relevance and quality of systematic reviews for policy-makers, practitioners, patients, and the public by developing explicit methods for considering equity effects. The Equity Group also develops criteria for when these methods should be applied.

35 What is health inequity? Difference in Health Outcomes Unavoidable Potentially avoidable Acceptable Unacceptable and unjust Evans and Kawachi

36 Commission on Social Determinants of Health

37 Erin.Ueffing:

38 7 crucial components for equityrelevant systematic reviews 1) developing a logic model; 2) defining disadvantage and for whom interventions are intended; 3) deciding on the appropriate study design(s); 4) identifying outcomes of interest; 5) process evaluation and understanding context; 6) analyzing and presenting data; and 7) judging the applicability of results.

39 7 crucial components for equityrelevant systematic reviews 1) developing a logic model; 2) defining disadvantage and for whom interventions are intended; 3) deciding on the appropriate study design(s); Study Design should be fit for purpose Many Upstream topics will use NRS

40 7 crucial components for equityrelevant systematic reviews 1) developing a logic model; 2) defining disadvantage and for whom interventions are intended; 3) deciding on the appropriate study design(s); 4) identifying outcomes of interest; 5) process evaluation and understanding context; 6) analyzing and presenting data; and 7) judging the applicability of results Cochrane Handbook Chapter on Assessing Equity

41 41 Equity Extension to PRISMA Reporting Guidelines

42 42 Equity-focused systematic reviews Bellagio Face to face consensus meeting February 8-10, 2012: 20 participants from 12 countries

43 PRISMA-E 2012

44 7 crucial components for equityrelevant systematic reviews 1) developing a logic model; 2) defining disadvantage and for whom interventions are intended; 3) deciding on the appropriate study design(s); 4) identifying outcomes of interest; 5) process evaluation and understanding context; 6) analyzing and presenting data; and 7) judging the applicability of results. Study Design should be fit for purpose Many Upstream topics will use NRS

45 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others NO 7 Contd 7. New C1 and C2 Methods and Review Groups focussing on mixed methods Campbell and Cochrane Equity Methods Group Campbell Collaboration International Development Group.

46

47

48 2012 While randomised control trials (RCTs) are probably the most famous example of causal studies, a range of other quasi-experimental techniques will usually also allow valid causal inference when well implemented [Shadish].

49 Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No Oxford UK Cochrane Meeting Don t be too purist Professor Sally Davies Relevance: prioritise the question, not the method David Tovey EIC Cochrane Library

50 Responses to requests for NRS Methods

51 Responses 1. Cochrane Handbook being revised with a sense of urgency to addressing the increased demand from inside and out for these needs. Non Randomised Studies Methods Meeting Ottawa June 2010.

52 Ottawa Cochrane and Campbell Non Randomised Studies Meeting 2010

53 Ottawa 2010 workshop Objectives When including non-rcts, is it appropriate to use systematic review methods used for RCTs? Discuss how standard methods for reviewing evidence need to be adapted when including non-rcts Formulate methodological guidance for review authors. Participants from diverse perspectives: 6 Cochrane & Campbell methods groups, Cochrane editor-in-chief, Cochrane & Campbell coordinating editors, representatives from stakeholder organisations and funders (AHRQ, UK NICE, ICES Ontario, 3ie). 53 Four core sessions: study design and bias, confounding and metaanalysis, selective reporting, applicability/directness

54 Ottawa Cochrane and Campbell Non Randomised Studies Meeting Can we be more explicit in guiding authors on what might be gained or lost under different circumstances? i.e when selection bias introduced in NRS might be particularly egregious? or situations in which RCT generalizability is likely to be attenuated by strict exclusion criteria? Really, the bottom line is that for many of the potential public health review topics there are simply no RCTs, so: when do experimental and observational studies produce comparable estimates? how to deal with bias & confounding? Providing guidance to review authors may require not only a statistical perspective, but also epidemiologic as well as social science perspectives. transparency in our assumptions approach to conveying uncertainty

55 NRS Recommendations 1 NRS may be reasonably free of bias a sizeable body of evidence the only feasible evidence the only available evidence...so NRS has an important role in Cochrane reviews The Collaboration should be more openly explicitly that NRS may be included in Cochrane reviews Review teams should routinely consider whether NRS should be included All eligibility criteria for types of studies need to be justified, whether the include randomized trials only, NRS only or both Considerations may be different for intended effects and adverse effects

56 NRS Recommendations 2 Situations in which NRS might be considered include: interventions cannot be randomized important outcomes were not studied in randomized trials (e.g. rare, longterm, or not considered when trials were done) gaining understanding of bias or consistency among NRS (particularly NRS with different potential biases) Some common study design labels should be avoided (e.g. retrospective ) and others should be used only when clearly explained (e.g. cohort ) Certain study design labels are useful to us when they are clearly defined (e.g. interrupted time series ) Considerations of eligibility into a review must be based on study features that ensure bias is adequately reduced and not on naive use of study labels

57 Papers from NRS Meeting 1. Setting the scene: 2. Study design and bias: 3. Confounding and meta-analysis 4. Selective reporting: 5. Applicability 6. Summing-up and way forward:

58

59 Draft Algorithm for including NRS in SRs

60 New Cochrane Tool : Risk of Bias for NRS

61 Risk of bias link to Forest Plot

62 Extending the Cochrane Risk of Bias tool to assess risk of bias in non-randomized studies Edited by Jonathan Sterne, Julian Higgins and Barney Reeves Version 1.0.0: 13 September 2013 Contributors (Listed alphabetically within category) Core group: Julian Higgins, Barney Reeves, Jelena Savović, Jonathan Sterne, Lucy Turner. Bias due to confounding: Nancy Berkman, Miguel Hernan, Pasqualina Santaguida, Jelena Savović, Beverley Shea, Jonathan Sterne, Meera Viswanathan. Bias in selection of participants into the study: Nancy Berkman, Miguel Hernan, Pasqualina Santaguida, Jelena Savović, Beverley Shea, Jonathan Sterne, Meera Viswanathan. Bias due to departures from intended interventions: David Henry, Julian Higgins, Peter Juni, Lakho Sandhu, Pasqualina Santaguida, Jonathan Sterne, Peter Tugwell. Bias due to missing data: James Carpenter, Julian Higgins, Terri Piggott, Hannah Rothstein, Ian Shrier, George Wells. Bias in measurement of outcomes or interventions: Isabelle Boutron, Asbjorn Hrobjartsson, David Moher, Lucy Turner. Bias in selection of the reported result: Doug Altman, Mohammed Ansari, Barney Reeves, An-Wen Chan, Jamie Kirkham, Jeffrey Valentine.

63 Summary Non-RCTs are required to answer some research questions, especially about important, long term or rare [unexpected] outcomes - typically, harms Review methods should be broadly similar Teams of review authors must include people with appropriate methodological and content expertise Protocol must pre-specify more details of methods Emerging consensus about an extended risk of bias tool Consider target RCT when appraising non-rcts Confounding, [selection biases] and selective reporting perceived to be major areas of difference to RCTs and concern

64

65 New Cochrane ROB for NRS The proposed response options for an overall RoB judgement are: (1) Low risk of bias (the study is comparable to a well-performed randomized trial); (2) Moderate risk of bias (the study provides sound evidence for a non-randomized study but cannot be considered comparable to a well-performed randomized trial); (3) Serious risk of bias (the study has some important problems); (4) Critical risk of bias (the study is too problematic to provide any useful evidence and should not be included in any synthesis); and (5) No information on which to base a judgement about risk of bias.

66 Risk of bias link to Forest Plot

67 Other SR Non Randomised Study Activities Cochrane ROBIS Harvard QE mtg November 2013 PICORI Translational Framework

68 Conclusion: Much is happening! Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views. Synthesis of the likely diversity of effectiveness-equity BUT: These require a whole new set of skills How do we do this right?

69

70 Other SR Non Randomised Study Activities Cochrane ROBIS Sept 13 Harvard QE mtg

71 ROBIS: Risk of Bias in Systematic Reviews Rachel Churchill,Julian Higggins et al

72 Other SR Non Randomised Study Activities Cochrane ROBIS Harvard QE mtg November 2013

73 Nov Inclusion of quasi-experimental studies in systematic reviews of health systems research Peter C. Rockers 1, John-Arne Røttingen 1,2,3, Ian Shemilt 4, Peter Tugwell 5, Till Bärnighausen 1,6 Article Discipline Definition of quasi-experimental King et al. (1995) Political Science An observational study with an exogenous explanatory variable that the investigator does not control Meyer (1995) Economics Studies in which there is a transparent exogenous source of variation in the explanatory variables that determine the treatment assignment Remler and Van Ryzin (2010) Public Affairs Studies that resemble... ran-domized field experiments... but that lack the researcher control or random assignment characteristic of a true experiment Shadish et al. (2002) Statistics An experiment in which units are not assigned to conditions randomly... Assignment to conditions is by means of self selection... or by means of administrator selection... or others decide which persons should get which treatment

74 Other SR Non Randomised Study Activities Cochrane ROBIS Harvard QE mtg November 2013 PICORI Translational Framework

75 Conclusion: Much is happening! Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views Synthesis of the likely diversity of effectiveness-equity BUT: These require a whole new set of skills How do we do this right?

76 Thank you!

77 Campbell and Cochrane Collaborations are continualy struggling with how to provide the SR evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views Synthesis of the likely diversity of effectiveness-equity Study Design should be fit for purpose

78 Campbell and Cochrane Collaborations are committed to providing the evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views Synthesis of the likely diversity of effectiveness-equity Study Design should be fit for purpose Are we randomistas ready to ditch the word pyramid

79 The proposed response options for an overall RoB judgement are: (1) Low risk of bias (the study is comparable to a well-performed randomized trial); (2) Moderate risk of bias (the study provides sound evidence for a nonrandomized study but cannot be considered comparable to a well-performed randomized trial); (3) Serious risk of bias (the study has some important problems); (4) Critical risk of bias (the study is too problematic to provide any useful evidence and should not be included in any synthesis); and (5) No information on which to base a judgement about risk of bias.