The Second David Sackett Symposium. Peter Tugwell. (Non-)randomized studies in Systematic Reviews including SRs on equity.

The Second David Sackett Symposium Peter Tugwell (Non-)randomized studies in Systematic Reviews including SRs on equity.

Campbell and Cochrane Collaborations are continualy struggling with how to provide the SR evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views Synthesis of the likely diversity of effectiveness-equity This has produced some major Creative tensions The Lightening Rod is Study Design Is Cochrane only for RCT Randomistas Let me take you through some of the debate with the other disciplines

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 1. 1. Lack of awareness that Non-Randomised Studies have arrived on the Cochrane Library!

Perhaps the most commonly cited Cochrane NRS Review. 2000

Pool fencing for preventing drowning of children Thompson DC, Rivara FP CDSR 2000 Objectives To determine if pool fencing prevents drowning in children (under 14 years of age). Selection criteria Studies evaluating pool fencing in a defined population and measuring the risk of drowning or near-drowning in fenced and unfenced pools. Main results Three case-control studies met the selection criteria. Pool fencing significantly reduces the risk of drowning. OR for the risk of drowning or near drowning is 0.27 (95% CI 0.16 to 0.47). Isolation fencing (enclosing pool only) is superior to perimeter fencing (enclosing property and pool); OR for the risk of drowning in a pool with isolation fencing compared to a pool with three-sided fencing is 0.17 (95% CI 0.07 to 0.44). Authors' conclusions Pool fences should have a dynamic and secure gate and should isolate the pool from the house. Legislation should require isolation fencing with secure, selflatching gates for all pools, public, semi-public and private. Legislation should require fencing of both newly constructed and existing pools and include enforcement provisions, in order to be effective.

Sunday 22 September 2013 Jos Verbeek 202 Cochrane reviews containing Non-Randomised Studies.. quasi-rcts (45%), CCT (55%), Controlled-before after studies (60%), interrupted time-series (52%), cohort-studies (37%), case-control studies (26%) Risk of bias assessment of NRS varies. Used a variety of checklists often self-constructed tools. Robust ROB urgently needed

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 2 Iain Chalmers points out that SRs are Observatiional Systematic Searching for studies relevant to the question is the hallmark of Cochrane IC uses them in his EBHC talks! Uses SIDS Back to Sleep example in his talks to emphasize the importance of Systematic Searching [with reproducible search] rather than one design..

Extract from publicity prepared for the UK Reduce the Risk Campaign in the early 1990s The risk of cot death is reduced if babies are NOT put on the tummy to sleep. Place your baby on the back to sleep...healthy babies placed on their backs are not more likely to choke.

Design. ITS: Interrupted Time Series

Reasons/events why the interest in expanding the scope of methods relevant to Health Policy and Systems Research in Cochrane, Campbell and others No 3 Cochrane EPOC (Effective Practice and Organisation of Care) Review Group Accept CCT, ITS, CBA.

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Relationship-Building :Cochrane/Campbell Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.;

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 4 4. Increasing Deliberative Dialogue :Cochrane Bridging with Classic Epidemiology,Social Scientists and Economics 2005 BMJ Chalmers, Glasziou, Vandenbroucke.; 2009 Freiburg Cochrane Colloquium final plenary. 2010 Ottawa Non Randomised Studies Meeting 2012 Bellagio PRISMA Equity meeting 2012 Montebello Complex Interventions Meeting

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 5 Public Health Review topics with no RCTs eg : Helmets vs no Helmets for motorcycles.

Helmets vs no Helmets for motorcycles Helmets are shown to reduce motorcyclist head injury and death Liu BC, Ivers R, Norton R, Boufous S, Blows S, Lo SK January 21, 2009 Motorcyclists are at high risk in traffic crashes, particularly for head injury. Main results: Sixty-one observational studies : Cohort, Case control and Cross Sectional. Despite methodological differences there was a remarkable consistency in results, particularly for death and head injury outcomes. Conclusion: helmets reduce the risk of head injury by around 69% and death by around 42%. The review supports the view that helmet use should be actively encouraged worldwide for rider safety -

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 6. Stop ignoring Adverse Effects: Cochrane Handbook,GRADE and Decision Aid Movement: Present Balance of Benefit and Harm Holger has covered GRADE The Cochrane Patient : Patient Decision Aid Movement is now linking with Cochrane

Require both Benefits and Harms Should be Balanced The Cochrane Patient 28

Cochrane Review Etanercept Decision Aid What are the benefits? BMJ 2011

Cochrane Review Etanercept Decision Aid : Side Effects and Harms BMJ 2011;343:d4027

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 7 New C1 and C2 Methods and Review Groups focussing on mixed methods Campbell and Cochrane Equity Methods Group.

Equity Mtgs

The Campbell and Cochrane Equity Methods Group http://equity.cochrane.org/

Campbell & Cochrane Equity Methods Group We aim to develop methods to improve the relevance and quality of systematic reviews for policy-makers, practitioners, patients, and the public by developing explicit methods for considering equity effects. The Equity Group also develops criteria for when these methods should be applied.

What is health inequity? Difference in Health Outcomes Unavoidable Potentially avoidable Acceptable Unacceptable and unjust Evans and Kawachi

Commission on Social Determinants of Health

www.equity.cochrane.org Erin.Ueffing:

7 crucial components for equityrelevant systematic reviews 1) developing a logic model; 2) defining disadvantage and for whom interventions are intended; 3) deciding on the appropriate study design(s); 4) identifying outcomes of interest; 5) process evaluation and understanding context; 6) analyzing and presenting data; and 7) judging the applicability of results.

41 Equity Extension to PRISMA Reporting Guidelines

42 Equity-focused systematic reviews Bellagio Face to face consensus meeting February 8-10, 2012: 20 participants from 12 countries

PRISMA-E 2012

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others NO 7 Contd 7. New C1 and C2 Methods and Review Groups focussing on mixed methods Campbell and Cochrane Equity Methods Group Campbell Collaboration International Development Group.

2012 While randomised control trials (RCTs) are probably the most famous example of causal studies, a range of other quasi-experimental techniques will usually also allow valid causal inference when well implemented [Shadish].

Events/reasons why the interest in expanding the scope of methods in Cochrane, Campbell and others No 8 2013 Oxford UK Cochrane Meeting Don t be too purist Professor Sally Davies Relevance: prioritise the question, not the method David Tovey EIC Cochrane Library

Responses to requests for NRS Methods

Responses 1. Cochrane Handbook being revised with a sense of urgency to addressing the increased demand from inside and out for these needs. Non Randomised Studies Methods Meeting Ottawa June 2010.

Ottawa Cochrane and Campbell Non Randomised Studies Meeting 2010

Ottawa 2010 workshop Objectives When including non-rcts, is it appropriate to use systematic review methods used for RCTs? Discuss how standard methods for reviewing evidence need to be adapted when including non-rcts Formulate methodological guidance for review authors. Participants from diverse perspectives: 6 Cochrane & Campbell methods groups, Cochrane editor-in-chief, Cochrane & Campbell coordinating editors, representatives from stakeholder organisations and funders (AHRQ, UK NICE, ICES Ontario, 3ie). 53 Four core sessions: study design and bias, confounding and metaanalysis, selective reporting, applicability/directness

Ottawa Cochrane and Campbell Non Randomised Studies Meeting Can we be more explicit in guiding authors on what might be gained or lost under different circumstances? i.e when selection bias introduced in NRS might be particularly egregious? or situations in which RCT generalizability is likely to be attenuated by strict exclusion criteria? Really, the bottom line is that for many of the potential public health review topics there are simply no RCTs, so: when do experimental and observational studies produce comparable estimates? how to deal with bias & confounding? Providing guidance to review authors may require not only a statistical perspective, but also epidemiologic as well as social science perspectives. transparency in our assumptions approach to conveying uncertainty

NRS Recommendations 1 NRS may be reasonably free of bias a sizeable body of evidence the only feasible evidence the only available evidence...so NRS has an important role in Cochrane reviews The Collaboration should be more openly explicitly that NRS may be included in Cochrane reviews Review teams should routinely consider whether NRS should be included All eligibility criteria for types of studies need to be justified, whether the include randomized trials only, NRS only or both Considerations may be different for intended effects and adverse effects

NRS Recommendations 2 Situations in which NRS might be considered include: interventions cannot be randomized important outcomes were not studied in randomized trials (e.g. rare, longterm, or not considered when trials were done) gaining understanding of bias or consistency among NRS (particularly NRS with different potential biases) Some common study design labels should be avoided (e.g. retrospective ) and others should be used only when clearly explained (e.g. cohort ) Certain study design labels are useful to us when they are clearly defined (e.g. interrupted time series ) Considerations of eligibility into a review must be based on study features that ensure bias is adequately reduced and not on naive use of study labels

Papers from NRS Meeting 1. Setting the scene: 2. Study design and bias: 3. Confounding and meta-analysis 4. Selective reporting: 5. Applicability 6. Summing-up and way forward:

Draft Algorithm for including NRS in SRs

New Cochrane Tool : Risk of Bias for NRS

Risk of bias link to Forest Plot

Extending the Cochrane Risk of Bias tool to assess risk of bias in non-randomized studies Edited by Jonathan Sterne, Julian Higgins and Barney Reeves Version 1.0.0: 13 September 2013 Contributors (Listed alphabetically within category) Core group: Julian Higgins, Barney Reeves, Jelena Savović, Jonathan Sterne, Lucy Turner. Bias due to confounding: Nancy Berkman, Miguel Hernan, Pasqualina Santaguida, Jelena Savović, Beverley Shea, Jonathan Sterne, Meera Viswanathan. Bias in selection of participants into the study: Nancy Berkman, Miguel Hernan, Pasqualina Santaguida, Jelena Savović, Beverley Shea, Jonathan Sterne, Meera Viswanathan. Bias due to departures from intended interventions: David Henry, Julian Higgins, Peter Juni, Lakho Sandhu, Pasqualina Santaguida, Jonathan Sterne, Peter Tugwell. Bias due to missing data: James Carpenter, Julian Higgins, Terri Piggott, Hannah Rothstein, Ian Shrier, George Wells. Bias in measurement of outcomes or interventions: Isabelle Boutron, Asbjorn Hrobjartsson, David Moher, Lucy Turner. Bias in selection of the reported result: Doug Altman, Mohammed Ansari, Barney Reeves, An-Wen Chan, Jamie Kirkham, Jeffrey Valentine.

Summary Non-RCTs are required to answer some research questions, especially about important, long term or rare [unexpected] outcomes - typically, harms Review methods should be broadly similar Teams of review authors must include people with appropriate methodological and content expertise Protocol must pre-specify more details of methods Emerging consensus about an extended risk of bias tool Consider target RCT when appraising non-rcts Confounding, [selection biases] and selective reporting perceived to be major areas of difference to RCTs and concern

New Cochrane ROB for NRS The proposed response options for an overall RoB judgement are: (1) Low risk of bias (the study is comparable to a well-performed randomized trial); (2) Moderate risk of bias (the study provides sound evidence for a non-randomized study but cannot be considered comparable to a well-performed randomized trial); (3) Serious risk of bias (the study has some important problems); (4) Critical risk of bias (the study is too problematic to provide any useful evidence and should not be included in any synthesis); and (5) No information on which to base a judgement about risk of bias.

Risk of bias link to Forest Plot

Other SR Non Randomised Study Activities Cochrane ROBIS Harvard QE mtg November 2013 PICORI Translational Framework

Conclusion: Much is happening! Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views. Synthesis of the likely diversity of effectiveness-equity BUT: These require a whole new set of skills How do we do this right?

Other SR Non Randomised Study Activities Cochrane ROBIS Sept 13 Harvard QE mtg

ROBIS: Risk of Bias in Systematic Reviews Rachel Churchill,Julian Higggins et al

Other SR Non Randomised Study Activities Cochrane ROBIS Harvard QE mtg November 2013

Nov 14 2013 Inclusion of quasi-experimental studies in systematic reviews of health systems research Peter C. Rockers 1, John-Arne Røttingen 1,2,3, Ian Shemilt 4, Peter Tugwell 5, Till Bärnighausen 1,6 Article Discipline Definition of quasi-experimental King et al. (1995) Political Science An observational study with an exogenous explanatory variable that the investigator does not control Meyer (1995) Economics Studies in which there is a transparent exogenous source of variation in the explanatory variables that determine the treatment assignment Remler and Van Ryzin (2010) Public Affairs Studies that resemble... ran-domized field experiments... but that lack the researcher control or random assignment characteristic of a true experiment Shadish et al. (2002) Statistics An experiment in which units are not assigned to conditions randomly... Assignment to conditions is by means of self selection... or by means of administrator selection... or others decide which persons should get which treatment

Other SR Non Randomised Study Activities Cochrane ROBIS Harvard QE mtg November 2013 PICORI Translational Framework

Conclusion: Much is happening! Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views Synthesis of the likely diversity of effectiveness-equity BUT: These require a whole new set of skills How do we do this right?

Thank you!

Campbell and Cochrane Collaborations are committed to providing the evidence that responds to the needs of users- the scope is evolving Synthesis of intervention effectiveness - what works Synthesis of implementation effectiveness - how it works Synthesis of resource effectiveness - at what cost/benefit? Synthesis of experiential effectiveness - users views Synthesis of the likely diversity of effectiveness-equity Study Design should be fit for purpose Are we randomistas ready to ditch the word pyramid

The proposed response options for an overall RoB judgement are: (1) Low risk of bias (the study is comparable to a well-performed randomized trial); (2) Moderate risk of bias (the study provides sound evidence for a nonrandomized study but cannot be considered comparable to a well-performed randomized trial); (3) Serious risk of bias (the study has some important problems); (4) Critical risk of bias (the study is too problematic to provide any useful evidence and should not be included in any synthesis); and (5) No information on which to base a judgement about risk of bias.