Running on Empty: The Cochrane Empty Reviews Project report of findings and consensus group feedback

Transcription

1 Running on Empty: report of findings and consensus group feedback Paul Montgomery, University of Oxford Joanne Yaffe, University of Utah Sally Hopewell, UK Cochrane Centre Lindsay Shepard, University of Utah The Empty Reviews Project is funded by the Cochrane Opportunities Fund

2 Empty Reviews Project Group Consensus Meeting Group Julian Higgins Cochrane Handbook Cambridge, UK David Moher Consensus Meetings/Guideline Design Ontario, Canada David Tovey Cochrane Editorial Unit London, UK Geraldine MacDonald DPLP Group High Nos. of Empty Reviews Belfast, NI Samantha Faulkner ENT Group Policy on Empty Reviews Oxford, UK Catherine McIlwain Consumers Oxford, UK Iain Chalmers Author of Empty Reviews Oxford, UK Sasha Shepperd EPOC Group Author Oxford, UK Apologies Claire Williams Heart Group Low No. of Empty Reviews London, UK Barney Reeves Non Randomised Studies Methods Group Bristol, UK Empty Reviews Project Core Team Jo Yaffe Paul Montgomery Sally Hopewell Lindsay Shepard Utah, USA Oxford, UK Oxford, UK Utah, USA

3 Summary of Session Presentation Introduction to the issue of empty reviews in The CDSR Update on the project to date Report on the consensus meeting Feedback and interactive discussion to move the project forward with your views

4 Definition An empty review is one that reports no studies eligible for inclusion.

5 How Do Systematic Reviews Come to Be Empty? Lack of high quality studies to address question posed and/or A very narrow population A very focused intervention Specific outcomes Many plan empty reviews to justify new trials in response to funding policies

6 Current Position As of 15 August 2010, 376 (8.7%) of Cochrane reviews contained no included studies Within The CDSR, the number of empty reviews is increasing and is inconsistent across Review Groups

7 Date of First Publication of Empty Reviews as at 15 August 2010

8 Introduction Of the 376 (8.7%) empty Cochrane reviews, 25.3% did not list excluded studies, 90.2% did not list ongoing studies, 96.0% did not list studies waiting assessment, and 23.4% did not list any studies other than what was discussed in their background section Proportions of empty reviews differed between groups, from 0% to 26.9% CRG editors differed somewhat in their perspectives on empty reviews and on editorial guidance provided to authors of these reviews

9 Acute Respiratory Airways Anaesthesia Back Bone/Joint/Muscle Breast Cancer Childhood Cancer Colorectal Cancer Consumers Cys Fib/Gen Dis Dementia Dep/Anx/Neurosis Devel/Psych/Learn Drugs/Alcohol Ear/Nose/Throat Effective Pract/Org Epilepsy Eyes/Vision Fertility Reg GYN Cancer Blood Cancer Heart Hepato-Biliary HIV/AIDS Hypertension Incontinence Infectious Disease Bowel Disorders Injuries Lung Cancer Menstrual/Subfert Metab/Endocrine Methods Movement Dis Multiple Sclerosis Musculoskeletal Neonatal Neuromuscular Oral Health Pain/Palliative Peripheral Vascular Pregnancy/Birth Prostate/Urol Canc Public Health Renal Schizophrenia STDs Skin Stroke Tobacco Upper GI/Pancreas Wounds Proportion of Empty Reviews across CRGs (as of 15 August 2010)

10 Importance Empty reviews are important as they: Tell us who is undertaking the review and thus interested in the topic Highlight major research gaps Indicate the state of the evidence at a point in time Justify further research and/or funding Highlight potential harms

11 Frustrations Policy-makers and practitioners report frustration in locating a review only to find that it has no included studies and is of limited utility for clinical decisions Lang, Edwards & Flieszer (2007) suggested that authors of empty reviews note observations based on excluded studies so that decision-makers are not left empty handed

12 Also In the absence of included studies some authors use evidence from excluded studies for reporting implications for practice: basing their recommendations on the available evidence not necessarily the best evidence At present there is little guidance for reporting excluded studies in the Cochrane Handbook for Systematic Reviews of Interventions There is no guidance for reporting of empty reviews

13 Aims of the Empty Reviews Project To examine the way in which empty Cochrane reviews are reported To review current guidelines for reporting of empty reviews To develop suggestions for guidance for the conduct and reporting of empty reviews

14 Consider Current Guidance in the Cochrane Handbook With thanks to Julian Higgins MRC Biostatistics Unit, Cambridge Co-Editor, Cochrane Handbook for Systematic Reviews of Interventions

15

16

17

18

19

20

21

22 Note 1/2 Cochrane reviews should be scientific reviews and as such must not draw conclusions based on data that are not part of the review Such evidence suffers from all the problems of narrative reviews that we strive so hard to avoid If no reliable evidence is found, this needs to be reported clearly, and not middied with unreliable evidence If it is realised after the fact that reliable evidence was excluded, the reviewers got their eligibility criteria wrong in the first place You can t fix that by post hoc cherry picking

23 Note 2/2 We do need to think about the relevance of reviews to decision makers, who often want the best available evidence rather than the available good evidence Perhaps a paradigm shift is necessary, and many are working on this (e.g. NRSMG) But to revolve the discussion around empty reviews is not helpful

24 Reporting of Empty Reviews Lang et al. (2007) Suggest guidelines needed Propose that reviewers could note observations from ineligible articles and abstracts Green et al. (2007) Note that specific structure could be helpful Argue that basing conclusions on studies which do not meet inclusion criteria increases the risk of bias

25 How Do CRGs (and Others) Handle Empty Reviews? Identify policies & procedures on Determining publication of empty reviews Practice of handling excluded studies across Cochrane and other providers Informing practice based on empty reviews and excluded studies

26 Methods Survey with closed and open-ended questions 53 Cochrane Collaboration Review Groups WHO, JBI, What Works Clearinghouse and SCIE Coded & analyzed responses

27 Cochrane Groups Publishing Empty Reviews N=53

28 % of responses Rationale for Publishing Empty Reviews N=46

29 Other Providers WHO No policy on empty reviews They search for the best available evidence, and then make a recommendation based on what they find SCIE As the evidence base in social care is very limited, their policy is similar WWCH Review focus is changed depending on the evidence base, thus empty reviews are not generally produced

30 CRG Views about Including Empty Reviews Include to instigate further research Our aim with empty reviews is to raise the profile of the question and hopefully generate research leading to high quality evidence which we can then report quickly in an updated review. Include if good case made There may be benefit in amending the TRF form to include the line: If you think this review might be empty, please give your reasons why you feel it is important to do the review.

31 CRG Reasons for Not Publishing Empty Reviews Not worth time and resources a colossal waste of everyone s time and energy Influence on group s impact factor May be misleading Empty reviews (< 3 studies) can put positive supportive evidence from initial, low quality studies behind treatment Some groups have minimum # of studies stipulation - e.g., 2-3 BUT, question can still matter may be exceptions where review questions appear of great public health relevance

32 Handling Excluded Studies Standard Policy? Permitted in Review? Unknown 4% Yes 21% No 75% Yes 58% No 31% Unknown 11%

33 % of responses Conditions for Providing Evidence Based on Excluded Studies N=31

34 Further Considerations for Whether to Allow Excluded Studies Excluded based on intervention or outcomes? RCTs feasible/ethical for topic area? If not, lower grade studies may be acceptable Guidelines may be needed for application of evidence from excluded studies We dissuade authors from providing evidence from excluded studies because of inconsistency, which would lead to provision of evidence that is arbitrary and unsystematic.

35 Guidance for Implications for practice Section N=53 For Yes responses (n=7): Extra guidance but on case-by-case basis (3) General advice (2) Refer to other empty reviews, EPICOT, & extra advice (1) Group Methodologists give guidance (1)

36 Views about Priority on Updating Empty Reviews No, topic-based priority only: Priority is only given on the basis of importance of the review topic, not on whether there are included studies or not. Only if new study emerges: Aim to update/review them around the same time as other reviews, but if we became aware of a study relevant to an empty review, it would probably prompt a more rapid update.

37 Content Analysis of Implications for Practice in Empty Reviews

38 What We Wanted to Know Despite the lack of high quality, or any, evidence included in empty reviews, how do authors report implications for practice?

39 Objectives of the Content Analysis To explore the following: How authors frame discussions of Implications for Practice? Do authors incorporate information from other research? If so, how and where? Do authors make recommendations for practice?

40 Conceptual Definition of Recommendation We defined recommendations for practice as statements which instruct readers, including consumers, practitioners, or policy makers to do something

41 One Sentence Wonders 117 (31.1%) "None identified from this systematic review. Mention lack of evidence and review topic: No randomized controlled trials, which assessed the effectiveness or safety of oral immunoglobulin preparations for the treatment of rotavirus diarrhoea in hospitalized low birthweight infants were found. Absence of evidence is not evidence of absence: "No relevant trials appear to have been published so this review concludes that there is no evidence of effect of leukotriene antagonists in bronchiectasis rather than evidence of no effect.

42 Vague Wording 26 (6.9%) - e.g. It is likely that clinicians will continue with their current practice, using clinical judgement and prescribing patterns to dictate treatment because there is no RCT-based evidence to help guide their choice of drug. It is difficult to know whether current practice is justified outside of a well designed, conducted and reported RCT. Currently policymakers have no RCT-based evidence upon which to base guidelines for HIVAN. They are likely to continue to rely on opinion and habit when making their recommendations. Funders of studies may wish to make this important subgroup of people a priority for future research.

43 Failure to Cite 50% of implications for practice sections make reference to evidence, only 16% of these cite its source The implications for fluid management in acute respiratory infections have not been studied in any RCTs to date. There is currently no evidence for or against increased fluids in acute respiratory infections. Nonexperimental (observational) data suggests that there may be a risk of symptomatic hyponatraemia due to increased antidiuretic hormone secretion in lower respiratory tract infections, particularly in children. The incidence in the primary care setting and the clinical significance of this observational data needs to be determined with further research, conducted as randomised controlled trials.

44 Targeted Discussions "1. For people with severe mental illness At present, the choice between dedicated supported housing schemes and outreach services is based on a combination of personal preference, professional judgement and availability of resources. Decisions of this nature should be made with the full understanding that no one intervention has been shown to be more effective than another in making a difference to symptoms, future use of services, quality of life or other measures of importance. Furthermore, the efficacy of supported housing remains untested. Participating in trials that test the effectiveness of such services should be encouraged. 2. For practitioners In the absence of evidence of their relative efficacy, decisions on the provision of alternative forms of accommodation and continued support for people with mental illness can only be based on a combination of professional judgement, patient preference and availability. This should be made clear to the patient or client who has to make this important decision. Practitioners may wish to actively support or participate in trials to test the effectiveness of supported housing schemes for people with severe mental illness. 3. For managers and policymakers Policies in favour of dedicated supported housing schemes should be viewed with some caution and should not be implemented without plans for evaluation using rigorous methods or should be delayed pending further evidence of their effectiveness. Forming alliances with researchers within this field may result in a fruitful collaboration that would not only inform local policies on this issue, but would also provide much needed evidence base on its effectiveness."

45 Implications for Practice with Recommendations 59 of 376 empty reviews (15.7%) appeared to contain recommendations Grouped as: generally supportive, recommending alternatives, or recommending either restricted use or non-use

46 Generally Favorable Recommendations Represent 23 of 59 (39.0%) empty reviews which appear to make recommendations 14 of these appear to have been based largely on other research cited in the background 7 seem to not be based on evidence, but on guidelines, theory, or absence of evidence of harm 2 seem to be based on excluded studies

47 Example of Generally Favorable Recommendation There is a need to raise awareness of the likely risks of waterpipe tobacco smoking worldwide. Particular attention should be devoted to examining empirically the potentially false perceptions about levels of dependence, disease and mortality associated with waterpipe tobacco smoking. Evidence-based information about waterpipe should be developed and disseminated in order to deglamourise and denormalise its use. Young people, especially women and girls in the Eastern Mediterranean Region, may be at higher risk of waterpipe use, and may need particular attention from healthcare workers. In the absence of policy research on waterpipe, and given what we know about its potential harmful effects, there is a need to include this tobacco use method in all tobacco control policy initiatives (e.g. bans on advertisement and sales to minors, taxation, warning labels, smokefree indoor air). Treatments of proven efficacy should be provided to help waterpipe smokers quit.

48 Recommendations to Use Something Else 8 (13.6%) of 59 empty reviews making recommendations Appear to be based on existing guidelines, practice standards, other research, or nothing readily apparent

49 Example of Recommendation to Use Something Else As there were no randomised or quasi-randomised controlled trials identified, we cannot draw any conclusions on the benefits (or otherwise) of regular administration of vitamin A in people with cystic fibrosis. Until further data are available, country or region specific guidelines (e.g. UK CF Trust Nutrition Guidelines (CF Trust 2002)) on the use and monitoring of vitamin A in people with cystic fibrosis should be followed.

50 Recommendations for Restricted Use 13 (22.0%) of 59 empty reviews making recommendations Limits on use include use in some circumstances (1), possible use in some circumstances (1), minimal use (1), or use only in the context of a controlled trial (10) Appear to be based largely on evidence of harm, possibility of harm, or no readily apparent basis

51 Examples of Recommendations Restricted Use for Restricted Use There is no evidence so far that subjective barriers reduce wandering, and the possibility of harm (particularly psychological distress) cannot be excluded. If used, then subjective barriers should form part of a diverse approach to problem wandering, which may include the identification and definition of the problem in the individual, preventative activities such as exercise classes or occupational therapies, and improved communication between carer and wanderer. Restrict to Trials 2. For clinicians In the absence of reliable evidence, clinicians prescribing calcium channel blockers for people with tardive dyskinesia must balance the possible benefits against the potential adverse effects of the treatment. Calcium-channel blockers lower people s blood pressure and may even cause symptoms of tardive dyskinesia to increase. These drugs should really only be used in a situation where their effects are closely monitored, i.e. within a randomised experimental design that can be written up and disseminated in order to inform practice.

52 Recommendations to Not Use 15 (25.4%) of 59 empty reviews making a recommendation Appear to be based on lack of evidence, evidence of harm, or ongoing research where trials had been aborted

53 Example of Recommendation to Not Use Given the lack of evidence comparing hydralazine versus placebo for primary (essential) hypertension, hydralazine should not be recommended as monotherapy. Clinicians who wish to recommend hydralazine for their patients should understand that while hydralazine may reduce blood pressure (based on non-randomized cross-over trials), to the best of our knowledge, there are no known clinical studies which have associated the use of hydralazine with a reduction in all cause mortality, myocardial infarction, or stroke. In addition, despite poor reporting of treatmentemergent adverse effects, clinicians must weigh the risks of potential serious side effects associated with the use of hydralazine, such as reflex tachycardia, hemolytic anemia, glomerulonephritis, vasculitis, as well as lupus-like syndrome against the potential benefits of blood pressure reduction with no proven beneficial effect on adverse cardiovascular outcomes.

54 Summary Implications for Practice sections differ widely across empty reviews Some Implications for Practice sections are very brief, while others are quite lengthy Some Implications for Practice sections are difficult to interpret or vaguely worded Many Implications for Practice sections appear to refer to research without a formal citation Whilst the majority of Implications for Practice sections do not make recommendations, some do

55 Some Good Practices Observed Clear statement on lack of included evidence Explanation of what that means, i.e., absence of evidence is not evidence of absence Appropriate comments on harms Description of limitations of excluded data Reference to related reviews (e.g., similar intervention/different population) with caveats to generalisability Consideration of economic issues

56 Where Has This Project Got to So Far? 1. Paper submitted outlining the issues surrounding empty reviews in The CDSR and the quantitative data presented here 2. Papers in preparation outlining the handling of empty reviews, how they are regarded and the qualitative data from CRGs and other systematic review producers 3. Website to collect views on the issues regarding need for guidance 4. Consensus meeting in June with stakeholders; feedback needed before proceeding further

57 Feedback and Guidance for Empty Reviews

58 Questions for Discussion How do we define an empty review? A systematic review that results in no included/eligible studies Empty reviews are suitable for/when : Important questions for practice or policy Trials are feasible Though may need to amend the protocol as part of an iterative process

59 Questions for Discussion Do we need guidance for conducting and reporting empty reviews? Yes, in these circumstances: 1- Reviews done and ready to report 2- Reviews in progress 3- Anticipating low evidence reviews 4- Updates Our Focus is #1, perhaps, with implications for #2-4

60 Completed Empty Reviews Should guidance be generic or organized by review question (e.g., parachute vs. RCTs unfound vs. side effects vs. trials inconceivable) or CRG? Guidance should be general to the whole collaboration

61 Which Sections of Empty Reviews Require Guidance? Abstract Main Results Brief sentence no evidence Author s Conclusion Brief sentence no evidence May highlight possible severe/prevalent adverse effects

62 Which Sections of Empty Reviews Require Guidance? Plain Language Summary Brief sentence no evidence Explain why empty, if appropriate, referring to eligibility criteria May highlight possible severe/prevalent adverse effects Summary of Findings Include even if empty

63 Which Sections of Empty Reviews Require Guidance? Background Objectives Methods Criteria for considering studies for this review Search methods for identification of studies Data collection and analysis Omit data collection/analysis sections Reference protocol

64 Which Sections of Empty Reviews Require Guidance? Results Description of studies Flow Diagram Results of the search Included studies Brief sentence no evidence On-going studies Excluded studies Usual text Do not report excluded results here Risk of bias in included studies Effects of interventions

65 Which Sections of Empty Reviews Require Guidance? Discussion Follow handbook subheadings Provide examples May discuss excluded study results, but Under heading Reference Discuss reason for exclusion Bookend with caveats Be brief

66 Which Sections of Empty Reviews Require Guidance? Author s conclusions Implications for practice Follow handbook no recommendations Add examples to Handbook Brief sentence no evidence Other evidence is not to be reported here General statements referring to local practice guidelines May flag severe/prevalent adverse effects Cite evidence Implications for research Signal for update (on-going, awaiting, etc.) General statement research needed/follow Handbook

67 Writing and Dissemination Strategies Standard Handbook applies Possible special addenda for empty reviews, section dependent (draft 2-3 pages to integrate) David Tovey editor training Colloquia workshops, editors meeting, etc. Website, webinars, etc. Other forums? Stand alone empty review reporting document?

68 Preliminary Considerations of Empty Reviews at Title Registration, at Protocol, in Progress, and at Update

69 Should Contingency Plans for Empty Reviews Be Defined in the Review TRF/Protocol? At TRF (Decision tree) 1. Trial feasible? Y/N 2. Aware of at least one possibly includable trial? Y/N 3. Important to review even if empty? At Protocol (Decision tree continued) Provide a contingency plan in case no eligible evidence?

70 Should empty reviews be flagged in the No Cochrane Library?

71 What are the views regarding the updating of empty reviews? Same general principles of updating apply

72 Any other points to consider?

73 The Cochrane Empty Reviews Project Funded by The Cochrane Opportunities Fund Paul Montgomery, University of Oxford Sally Hopewell, UK Cochrane Centre Joanne Yaffe, University of Utah Lindsay Shepard, University of Utah