Introduction to systematic reviews/metaanalysis Hania Szajewska The Medical University of Warsaw Department of Paediatrics hania@ipgate.pl Do I needknowledgeon systematicreviews? Bastian H, Glasziou P, Chalmers I. PLoS Med. 2010;7:e1000326. 1
Do I needknowledgeon systematicreviews? Clarke M. PLoS Medicine 2004;1(2):e35. Clarke M et al. Lancet 2010;376:20-21. Cochrane database of systematic reviews >8500 November 2014 2
Objectives By the end of this session, you should be able to: Understand what is a systematic review/meta-analysis Know steps in conducting a systematic review Know how to interpret a forest plot How to use knowledge on systematic reviews To perform a systematic review yourself To understand a systematic review performed by others 3
Hierarchy of evidence for questions about the effectiveness of an intervention Systematic review/meta-analysis Strong RCT Cohort studies Case-control studies Case series studies Expert opinion, theories bases on physiology or plausibility, bench top research & animal studies Weak Guyatt, Rennie. User s guides to the medical literature. 2002 Level of evidence http://www.cebm.net/ 4
Systematic review vs meta-analysis Systematic review Meta-analysis RCT N 0 1 Analysis RCT N 0 1 RCT N 0 2 RCT N 0 3 RCT N 0 4 Analysis Analysis Analysis RCT N 0 2 RCT N 0 3 RCT N 0 4 Statistical techniques Pooled results of all RCTs RCT N 0 5 Analysis RCT N 0 5 RCT N 0 6 Analysis RCT N 0 6 Definition Systematic review A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies http://www.cochrane.org/glossary (accessed on 2 Jan 2011) 5
Why perform a meta-analysis? To increase power The chance to reliably detect a clinically important difference if one actuallly exist To improve precision in estimating effects Narrow the confidence interval around the effects Egger, Smith, Altman (ed.). Systematic review in health care. Meta-analysis in context. BMJ Books 2000. Steps in conducting a systematic review 1. Formulation of the review question (the problem) (PICO) 2. Searching for studies 3. Selecting studies and collecting data 4. Assessment of the risk of bias in the included trials 5. Analyzing the data and presenting the results http://www.cochrane.org/glossary (accessed on 30 July 2013) 6
Step 1: Define the question PICO Key components to a well-formulated question Population (participants) Intervention Comparison Outcome(s) Example Objective To compare the efficacy and safety of prophylactic enteral probiotics administration versus placebo or no treatment in the prevention of severe NEC or sepsis, or both, in preterm infants. P Population Preterm infants I Intervention Probiotics C Comparison Placebo or no treatment O Outcome Efficacy and safety (NEC, sepsis) AlFaleh, Anabrees. Cochrane review 2014 7
Step 2: Conduct literature search One databaseisneverenough! Three the most important sources MEDLINE EMBASE CENTRAL Cochrane Central Register of Controlled Trials Database overlap approx. 34% Of the 4,800 journals indexed in EMBASE, 1,800 are not indexed in MEDLINE. Similarly, of the 5,200 journals indexed in MEDLINE, 1,800 are not indexed in EMBASE. 8
Step 3: Selecting studies and collecting data Step 4: Assessment of the risk of bias in the included trials. Garbage in Garbage out Egger et al. BMJ Books 2003 9
Step 4: Assessment of the risk of bias in the included trials. Numerous tools are available The Cochrane Handbook We recommend against the use of scales yielding a summary score No gold standard for the true validity of a trial Cochrane Handbook of Systematic Reviews of Intervention 2011. Risk of bias assessments recommended by the Cochrane Collaboration Risk of bias graph: Risk of bias summary: Judgements about each risk of bias item presented as percentages across all included studies. Judgements about each risk of bias item for each included study. Conde-Agudelo A, Cochrane Database of Systematic Reviews 2011, Issue 3. 10
Step 5: Analyzing the data and presenting the results. To pool or not to pool? Determine methods of pooling of results Pool results (if appropriate) Explore heterogeneity Sensitivity and subgroup analysis Explore possibility of publication bias To pool or not to pool the results of individual trials? One of the most important decisions to be made by the authors of any systematic review 11
Why perform a meta-analysis? To increase power The chance to reliably detect a clinically important difference if one actuallly exist To improve precision in estimating effects Narrow the confidence interval around the effects Egger, Smith, Altman (ed.). Systematic review in health care. Meta-analysis in context. BMJ Books 2000. When to pool the results? If the studies are considered sufficiently homogenous in terms of the question and methods PICO 12
Take home message Systematic review It is always appropriate and desirable to systematically review data Not always Meta-analysis It may sometimes be inappropriate, or even misleading, to statistically pool results from separate studies Results of systematic review A 0.1 (0.07-0.3) 0.1 (0.05-0.2) 1.7 (1.1-2.9) 1.8 (1.1-3.1) Don t pool if the point estimates of all studies are not on the same side of the line of no effect! Favours treatment Favours control 13
Results of systematic review B 0.7 (0.4-1.1) 0.6 (0.2-1.7) 0.8 (0.5-1.3) 0.7 (0.3-1.3) Pool if There are similarities of point estimates CI overlap Favours treatment Favours control Results of systematic review C 0.7 (0.3-1.3) 1.3 (0.7-2.4) 0.8 (0.4-1.5) 1.3 (0.7-2.3) Pool if confidence intervals overlap Favours treatment Favours control 14
How to interpret forest plot? Forest plot A graphical representation of the individual results of each study included in a metaanalysis together with the combined metaanalysis result. Formulas containing hydrolysed protein for prevention of allergy The resultsof 7 RCTs are presented Osborn & Sinn. Cochrane Review 2006 15
Formulas containing hydrolysed protein for prevention of allergy Each horizontal line represents The black square the results in the The widthof the middlerepresents of one trial the horizontal line point estimateof the represents the 95% CI difference between for this estimate groups Osborn & Sinn. Cochrane Review 2006 Formulas containing hydrolysed protein for prevention of allergy Line of no effect Osborn & Sinn. Cochrane Review 2006 16
Formulas containing hydrolysed protein for prevention of allergy If the horizontal line for any trial does NOT cross the line of no effect, there is 95% chance that there is a real difference between the groups Line of no effect Osborn & Sinn. Cochrane Review 2006 Formulas containing hydrolysed protein for prevention of allergy Ifthehorizontallinefor anytrial doescross thelineof no effect, thiscanmeanthat either there is no sifnificant difference and/or that the sample sizewas toosmall Line of no effect Osborn & Sinn. Cochrane Review 2006 17
Formulas containing hydrolysed protein for prevention of allergy Combined results of all trials effect Centre of the diamond = pooled point estimate Horizontal tips = confidence interval (usually 95% CI) Osborn & Sinn. Cochrane Review 2006 Formulas containing hydrolysed protein for prevention of allergy Weight given to each study inthe combined results Osborn & Sinn. Cochrane Review 2006 18
Formulas containing hydrolysed protein for prevention of allergy Osborn & Sinn. Cochrane Review 2006 Doesitmatterhowa systematic review is done? Careful design, conduct, and analysis of a review is needed to prevent bias Bias A systematic error or deviation in results or inferences from the truth Egger, Smith, Altman (ed.). Systematic review in health care. Meta-analysis in context. BMJ Books 2000. 19
Critical appraisalof a systematic review Is the review valid? What are the results? Will the results help locally? http://www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme Rapidcritical appraisal of a systematic review Is the review valid? What are the results? Will the results help locally? 1. Did the review ask a clearly focused question? 2. Did the review included the right type of study? 3. Did the review try to identify all relevant studies? 4. Did the reviewers assess the quality of the included studies? 5. If the results of the studies have been combined, was it reasonable to do so? 6. How are the results presented and what is the main result? 7. How precise are these results? 8. Can the results be applied to the local population? 9. Were all important outcomes considered? 10. Should policy or practice change as a result of the evidence contained in this review? http://www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme 20
By now, you should be able to: Understand what is a systematic review/metaanalysis Know steps in conducting a systematic review Know how to interpret a forest plot A final comment What is true today is often not true tomorrow. Alpert JS. Am J Med 2012 Thank you for your attention 21
Additional reading Finding existing reviews Cochrane Collaboration http://www.cochrane.org NHS Centre for Reviews & Dissemination http://www.york.acul/inst/crd/welcome.htm Many leading journals Pediatrics J Pediatr Am J Clin Nutr 22
Publication http://www.prisma-statement.org PRISMA 2009 23
Problems & limitations in conducting systematic reviews Question With regard to meta-analysis, terms such as mega-silliness, statistical alchemy, and mixing apples and oranges are commonly used by the critics. What is your opinion? 1. Agree 2. Undecided 3. Disagree 24
A common criticism of a meta-analysis Mega-silliness Eysenck HJ. Am Psychol 1978;33:517 Shmeta-analysis Shapiro S. Am J Epidemiol 1994;140:771-8. Statistical alchemy Feinstein AR. J Clin Epidemiol 1995;48:71-9. Mixing apples and oranges Sharpe D. Clin Psychol Rev 1997;17:881-901 Mixing apples & oranges Dangerous or delicious? 25
Problems and limitations of a meta-analysis Failure to identify all relevant studies Unpublished data Quality of included trials Inconclusive systematic reviews Opposite conclusions Discrepancies between the results of a metaanalysis and a large RCT Failure to identify all relevant studies The core of SR/MA is the identification of every relevant evidence What is important Searching one database is never enough At least Medline, Embase, Cochrane Library No restriction on language In order to minimise bias, 2 or more reviewers The set of key words as complete as possible 26
Unpublished data To include or not to include? Controversial issue Unpublished data Arguments in favor Unpublished studies differ systematically from those that have been published Methodological weaknesses Smaller or no treatment effects revealed in these trials Often, but not always, industry sponsored Non-publications can lead to false assumptions regarding the efficacy of the treatment Inclusion reduces the risk of publication bias Cook et al. JAMA 1993;269:2749-53 Dickersin et al. JAMA 2003;290:516-23. Whittington et al. Lancet 2004;363:1341-5. 27
Unpublished data Arguments against Challenges Gaining access to the studies Obtaining sufficient information to evaluate the methodological quality Unpublished data may be of lower methodological quality Located trial data may be an unrepresentative sample of data from all unpublished studies Further introduce bias MacLean et al. J Clin Epidemiol 2003;56:44-51. Should unpublished data be included? Published data alone Indicated that benefits may outweigh the risks Published and unpublished data Suggested that the risks outweigh the benefits Whittington et al. Lancet 2004;363:1341-5. 28
Garbage in garbage out Egger et al. BMJ Books 20 Quality of included trials Adequate allocation generation Selection bias Allocation concealment Selection bias Blinding Perfomance bias Loss to follow-up Attrition bias Egger et al. BMJ Books 2003 29
How much loss to follow-up is acceptable? Fewtrell at al. Arch Dis Child 2008;93:458-61. Inconclusive systematic review No clear evidence Some evidence of a trend 30
Inconclusive systematic review Frustrating! But. Clearly demonstrating the inadequacy of existing evidence is an important objective of systematic reviews Should serve as a stimulus for conducting the appropriate and necessary trials Alderson & Roberts. Br Med J 2000;320:3 Opposite conclusions Systematic reviews addressing the same issue and performed at almost the same time have reached opposite conclusions Jadad et al. Can Med Ass J 1997;156:141 31
Discrepancies between the results of a meta-analysis and a large RCT No agreement what should be done Consider methodological quality of RCT and MA If both are of high quality, the results of the RCT are (probably) more reliable It is considered that a deviation in 5 percent of cases would be expected on the basis of chance alone Bailar C 3rd. NEJM 1997;337:559-61. Lelorier et al. NEJM 1997;337:536-42. Summary By now, you should be able to: Understand what is a systematic review/meta-analysis Know steps in conducting SR Read a forest plot Understand problems and limitations in conducting SR 32
A final comment.. The important thing is not to stop questioning. With regard to meta-analysis, terms such as megasilliness, statistical alchemy, and mixing apples and oranges are commonly used by the critics. What is your opinion? Strongly agree Agree Undecided Disagree Strongly disagree 33
How many systematic reviews (with or without meta-analysis) have you performed yourself? None One More than one More than three More than five Clinical trials should begin and end with systematic reviews of relevant evidence. Strongly agree Agree Undecided Disagree Strongly disagree 34