How to Conduct a Systematic Review: A Workshop 24 th Annual Primary Care Research Methods & Statistics Conference, San Antonio, Texas Saturday, December 3, 2011 Robert M. Jacobson, M.D. Department of Pediatric and Adolescent Medicine Mayo Clinic, Rochester, Minnesota Paul M. Darden, M.D. Department of Pediatrics University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma (A) systematic review is an overview of primary studies which contains an explicit statement of objectives, materials, and methods and has been conducted according to explicit and reproducible methodology. Trisha Greenhalgh By providing a reliable synthesis of the available evidence on a given topic, systematic reviews adhere to the principle that science is cumulative and facilitate decisions considering all the evidence on the effect of an intervention. The Cochrane Handbook A systematic review is a standardized and objective review as opposed to the more traditional narrative review. A systematic review does not necessarily include a meta-analyses, which is a mathematical combining of the results of two or more primary studies. The primary studies do not have to be randomized controlled trials. Standards exist for conducting systematic reviews of observational studies, including studies that answer questions other than therapeutic ones. This workshop focuses on the process of conducting a systematic review rather than their value and place in patient-centered research. Table of Contents Page Steps in Creating a Systematic Review 2 Define the Review Question and Develop Criteria for Including Studies 2 Search for Studies 2 Select Studies and Collect Data 3 Assess Risk of Bias in Included Studies 3 The Cochrane Collaboration s Tool for Assessing Risk of Bias 4 Analyze Data and Undertake Meta-Analyses 5 Address Reporting Biases 5 Assemble Results 6 An Example of a Summary of Findings Table 7 Interpret Results and Draw Conclusions 8 PRISMA Flow Chart 9 PRISMA Checklist 9 The Cochrane Collaboration 11 Selected Readings 11 1
Steps in Creating a Systematic Review 1) Define the review question and develop criteria for including studies 2) Search for studies 3) Select studies and collect data 4) Assess risk of bias in included studies 5) Analyze data and undertake meta-analyses 6) Address reporting biases 7) Assemble results 8) Interpret results and draw conclusions Define the Review Question and Develop Criteria for Including Studies Stating your research question and selection criteria first before gathering data maintains your systematic review as a means for hypothesis testing. To engage the resources of the Cochrane Collaboration, you must contact the organization and agree on a review topic with a Cochrane Review Group (CRG). The Cochrane Collaboration has 52 CRGs, each with their own scope. Many CRGs have priorities for reviews of importance and will require the completion of a title registration form. A systematic review begins with the development of an answerable question. The question should have a PICO framework. The question should specify the Population or Participants or Patients, the Interventions and Comparisons, and the Outcomes. Several but not many outcomes should be included both beneficial and harmful (in the case of a therapeutic intervention). On the other hand the outcomes should be clinically meaningful and not trivial. The PICO question will drive both the detailed criteria for including studies and excluding others as well as the literature search itself. Exercise 1 Consider the postcard reminders veterinarians use to remind their clients of their pet s vaccinations that are coming due. Do you think they might work for humans? Write a PICO question and then draft a specific list of inclusion and exclusion criteria. Search for Studies Create a protocol for the search that includes three aspects 1) systematic of multiple, electronic databases, 2) a review of the reference lists of published studies found, and 3) consultation with experts. Use the PICO question to identify the key terms for your search. Usually use P and I connected by the Boolean Operator AND. Avoid using too many search concepts such as P, I, C, and O all at once. Identify synonyms for each of the terms to broaden the search. Goal is high sensitivity and low specificity. Use OR to link synonyms. Use free text as well as subject heading (MeSH or Medical Subject Headings). Use multiple electronic databases (MEDLINE or Pubmed for NLM, CINAHL for the nursing literature, EMBASE for Elsevier s database, PsychLit, Sociological Abstracts, CABHealth, 2
HealthSTAR, AIDSlINE, Occupational Safety and Health Database, Educational Research Index (ERIC), PsycINFO, Dissertation Abstracts, and Conference Papers Index). Google Scholar can provide access to the gray literature to generate reference lists. Goal is to include studies (rather than reports of studies) but the majority of studies are identified through the published reports. One can identify studies, if they are clinical trials, through trial registries as well. In this way, one can find trials that have not yet been published either because they are still underway or because the investigators chose not to publish them. Those working with the Cochrane Collaboration will be able to access the Cochrane Central Register of Controlled Trials (CENTRAL). Filters can be used to seek randomized trials. These are often specific to the database, however. For example, in PubMed, you can select limits to only include types of articles that are randomized controlled trials. Exercise 2 Using the agreed upon PICO question, design a search strategy complete with synonyms for a generic, electronic database. Select Studies and Collect Data These two tasks can be done simultaneously. Use at least two abstractors working independently. The team can use consensus to resolve differences between reviewers. Use a standardized abstraction form to evaluate each eligible study. The data collection form is key. Design it carefully to target the objectives of the systematic review. Pilot the form before launching the abstraction efforts. See the Zaza 2000 citation for detailed instructions as well as an example. Again, the unit of measure is the study and not the report. Abstractors must carefully recognize and connect studies that are reported more than once. The PRISM Flow Chart describes the flow of information through the different phases of a systematic review Assess Risk of Bias in Included Studies Conducting a systematic review requires that you not only collect appropriate studies but you assess them for bias. The inclusion and exclusion criteria took care of issues with generalizability (external reliability) but not with (bias) internal validity. You will want to use a standardized approach to assess risk of bias of individual studies You should also decide a priori how your assessment of bias for an individual study will affect how you report the results and use it for any data synthesis. You can choose from a variety of sources to obtain tools to assess the methodological quality of the studies. Avoid the temptation to use a numerical scale. You need a clear useful categorization of the studies. The Cochrane Collaboration prefers assessments of low risk, high risk, or unclear risk of bias. For parallel group trials, the Cochrane Collaboration asks for assessments of bias with regard to the following: 3
Sequence generation Allocation sequence concealment Blinding Incomplete outcome data Selective outcome reporting Assessments should include consideration of the effect of the bias in terms of over- or underestimating the true outcomes. Exercise 3 Listen to the reading of the methods of one of the selected studies and make the assessment using the following bias assessment tool: The Cochrane Collaboration s Tool for Assessing Risk of Bias Domain Description Review authors judgement Sequence generation. Describe the method used to generate the allocation sequence in sufficient detail to allow an assessment of whether it should produce comparable groups. Was the allocation sequence adequately generated? Allocation concealment. Blinding of participants, personnel and outcome assessors Assessments should be made for each main outcome (or class of outcomes). Incomplete outcome data Assessments should be made for each main outcome (or class of outcomes). Selective outcome reporting. Describe the method used to conceal the allocation sequence in sufficient detail to determine whether intervention allocations could have been foreseen in advance of, or during, enrolment. Describe all measures used, if any, to blind study participants and personnel from knowledge of which intervention a participant received. Provide any information relating to whether the intended blinding was effective. Describe the completeness of outcome data for each main outcome, including attrition and exclusions from the analysis. State whether attrition and exclusions were reported, the numbers in each intervention group (compared with total randomized participants), reasons for attrition/exclusions where reported, and any re-inclusions in analyses performed by the review authors. State how the possibility of selective outcome reporting was examined by the Was allocation adequately concealed? Was knowledge of the allocated intervention adequately prevented during the study? Were incomplete outcome data adequately addressed? Are reports of the study free of suggestion of selective outcome 4
Other sources of bias. review authors, and what was found. State any important concerns about bias not addressed in the other domains in the tool. If particular questions/entries were prespecified in the review s protocol, responses should be provided for each question/entry. reporting? Was the study apparently free of other problems that could put it at a high risk of bias? Analyze Data and Undertake Meta-Analyses A meta-analysis is the statistical combination of results from two or more studies. Such a combination can improve power to rule out a Type II error, provide improved precision to an outcome estimate, or address questions in subgroups not addressed in individual studies. On the other hand, a poorly done meta-analysis can incorrectly combine studies that should not be combined and drown important differences in a statistical pool. (Alvan R Feinstein) Not every systematic review permits a meta-analysis but when it is appropriate one should be conducted. The Cochrane Collaboration provides free software to download from the internet to combine studies. The software is called RevMan. Most statistical approaches to combining the results use a system of sample-size weighting of the effect size estimates from the individual studies. Studies that test as homogeneous can be combined and permit fixed-effects meta-analyses. Random effects meta-analyses permit combining heterogeneous studies but care must be taking in their interpretation. Sensitivity analyses can help in the evaluation of the results of a meta-analysis to determine how robust the results are. Sensitivity analyses may include fixed effects versus random effects analyses, methodological quality, inclusion/exclusion of unpublished studies, and characteristics of the intervention or study population. Choose outcomes to combine that are suitable for comparing intervention groups. For clinically relevant Numbers Needed to Treat calculations select outcomes where the patient or participant count forms the denominator. End points such as time to recovery or frequency of events lack the same clinical relevance. Exercise 4 As a large group, we will test a number of studies to combine first for heterogeneity and finding none, we will combine them using the Mantel-Haenzel Chi Square approach (a fixed effects method). Address Reporting Biases Reporting biases occur when the dissemination of a study s results are affected by the direction and magnitude of a study s results. Reporting biases include publication, time-lag, multiple publication, citation, and language biases. Investigations have shown that while there really appears to be a tendency for negative studies not to appear in the published literature, the fault appears to lay in the investigators efforts rather than the editors or peer-reviewers. 5
Not studied is the differential in availability by systematic searches depending on the journal that publishes the report of the study. However, a priori trial registries and changes in understanding of the need to publish small, equivocal studies are improving the situation. In conducting a systematic review, you must assess and addressing reporting biases. A funnel plot is a scatter plot of treatment effect against a measure of study size (such as standard error of the treatment effect). In general if all studies regarding a common treatment effect are published without bias to either the statistical significance of the result or the positivity of the result, then with treatment effect on the x axis and sample size on the y axis, an inverted funnel should form. Asymmetries can but not necessarily are due to publication bias. Methods exist for determining those asymmetries. Assemble Results Use figures and tables to help present the studies included in the systematic review and their findings. A forest plots is the standard method of illustrating the results of both he individual studies and the meta-analysis. The RevMan software provided freely by the Cochrane Collaboration can generate these. 6
A summary-of-findings table provides the key information concerning the quality of the evidence, the size of the effect, and the sum of available data on important outcomes. See the next page for an example of a Summary of Findings table. Actually, the Cochrane Handbook provides the complete table from which this partial table is extracted. An Example of a Summary of Findings Table Compression stockings compared with no compression stockings for people taking long flights Patients or population: Anyone taking a long flight (lasting more than 6 hours) Settings: International air travel Intervention: Compression stockings Comparison: Without stockings Outcomes Illustrative comparative risks* (95% CI) Assumed risk Corresponding risk Relative effect (95% CI) Number of participants (studies) Quality of the evidence (GRADE) Comments Without stockings With stockings Symptomatic deep vein thrombosis (DVT) See comment See comment Not estimable 2821 (9 studies) See comment 0 participants developed symptomatic DVT in these studies. Symptom-less Low risk population RR 0.10 2637 ++++ 7
deep vein thrombosis 10 per 1000 1 per 1000 (0 to 3) High risk population 30 per 1000 3 per 1000 (1 to 8) (0.04 to 0.26) (9 studies) High Superficial vein thrombosis 13 per 1000 6 per 1000 (2 to 15) RR 0.45 (0.18 to 1.13) 1804 (8 studies) +++Ο Moderate Oedema Post-flight values measured on a scale from 0, no oedema, to 10, maximum oedema. The mean oedema score ranged across control groups from 6 to 9. The mean oedema score in the intervention groups was on average 4.7 lower (95% CI 4.9 to 4.5). 1246 (6 studies) ++ΟΟ Low *The basis for the assumed risk is provided in footnotes. The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the intervention group and the relative effect of the intervention (and its 95% CI). CI: Confidence interval; RR: Risk ratio GRADE: GRADE Working Group grades of evidence (see explanations) Interpret Results and Draw Conclusions The Cochrane Collaboration provides a GRADE approach to rate the quality of each outcome, giving four levels of high, moderate, low, and very low. Do not describe the results simply in terms of the presence or absence of statistical significance but provide the central tendency as well as the confidence interval and exact P value. For clinical relevant results regarding benefits and harms, use Numbers Needed to Treat calculations and confidence intervals. Peer-reviewed journals recognize systematic reviews as original scientific contributions. The PRISMA criteria included in this syllabus provide a guide to insure that your manuscript means standard criteria for including important aspects of the systematic review. Exercise 5 Consider the control event rate and the experimental event rates for pharyngitis in examining two preventive strategies. The control event rate was 12%. The experimental event rate as 8%. Calculate the relative risk reductions, the absolute risk reduction, and the numbers needed to treat. 8
PRISMA Flow Chart Flow of information through the different phases of a systematic review Identification Screening # of records identified through database searching # of records after duplicates removed # of records screened # of additional records identified through other sources # of records excluded Eligibility # of full-text articles assessed for eligibility # of full-text articles excluded, with reasons Included # of studies included in qualitative synthesis # of studies included in quantitative synthesis (metaanalysis) PRISMA Checklist Checklist of items to include when reporting a systematic review or meta-analysis. Section/Topic # Checklist Item TITLE Title 1 Identify the report as a systematic review, meta-analysis, or both. ABSTRACT Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study 9
appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. INTRODUCTION Rationale 3 Describe the rationale for the review in the context of what is already known. Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). METHODS Protocol and registration Eligibility criteria Information sources 5 6 7 Search 8 Study selection 9 Data collection process 10 Data items 11 Risk of bias in individual studies Summary measures Synthesis of results Risk of bias across studies Additional analyses RESULTS 12 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators. List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis. 13 State the principal summary measures (e.g., risk ratio, difference in means). 14 15 16 Study selection 17 Study characteristics Risk of bias within studies Results of individual studies Synthesis of results Risk of bias across studies Additional 18 19 20 21 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I 2 ) for each meta-analysis. Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations. Present data on risk of bias of each study and, if available, any outcome-level assessment (see Item 12). For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group and (b) effect estimates and confidence intervals, ideally with a forest plot. Present results of each meta-analysis done, including confidence intervals and measures of consistency. 22 Present results of any assessment of risk of bias across studies (see Item 15). 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). analysis DISCUSSION Summary of 24 Summarize the main findings including the strength of evidence for each main 10
evidence Limitations 25 Conclusions 26 FUNDING Funding 27 outcome; consider their relevance to key groups (e.g., health care providers, users, and policy makers). Discuss limitations at study and outcome level (e.g., risk of bias), and at review level (e.g., incomplete retrieval of identified research, reporting bias). Provide a general interpretation of the results in the context of other evidence, and implications for future research. Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review The Cochrane Collaboration The Cochrane Collaboration is an international organization. Its primary aim is to help people make well-informed decisions about health care by preparing, maintaining and promoting the accessibility of systematic reviews of the evidence that underpins them. The Cochrane Collaboration was founded in 1993. It has grown to include over 15,000 contributors from more than 100 countries. The Cochrane Collaboration was named after British epidemiologist Archie Cochrane. Archie Cochrane once wrote, It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, updated periodically, of all relevant randomized controlled trials. You can find the Collaboration s web pages at www.cochrane.org. The Cochrane Handbook for Systematic Reviews of Interventions (the Handbook) provides guidance to authors for the preparation of Cochrane Intervention reviews (including Cochrane Overviews of reviews). Anyone can access the Handbook at www.cochrane.org/resources/handbook. Selected Readings Zaza S, Wright-De Aguero LK, Briss PA, Truman BI, Hopkins DP, Hennessy MH, Sosin DM, Anderson L, Carande-Kulis VG, Teutsch SM, Pappaioanou M, Task Force on Community Preventive Services Data Collection Instrument and Procedure for Systematic Reviews in the Guide to Community Preventive Services. Am J Prev Med 2000;18(1S):44 74. Greenhalgh T. How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses). BMJ 1997; 315: 672- Ndiaye SM. Hopkins DP. Smith SJ. Hinman AR. Briss PA. Task Force on Community Preventive Services. Methods for conducting systematic reviews of targeted vaccination strategies for the guide to community preventive services. American Journal of Preventive Medicine. 2005; 28:238-47. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. J Clin Epidemiol 2009; doi:10.1016/j.jclinepi.2009.06.005. The Cochrane Collaboration. URL: http://www.cochrane.org/. Accessed November 22, 2011. The Cochrane Collaboration. JPT Higgins and S Green, eds. The Cochrane Handbook for Systematic Reviews of Interventions. Edition 5.1.0. Updated March 2011. URL: http://www.cochrane-handbook.org/. Accessed November 22, 2011. 11