How to Conduct a Meta-Analysis Faculty Development and Diversity Seminar ludovic@bu.edu Dec 11th, 2017
Periodontal disease treatment and preterm birth We conducted a metaanalysis of randomized controlled trials to determine whether periodontal disease treatment with scaling and/or root planing during pregnancy may reduce preterm birth incidence Treatment resulted in significantly lower preterm birth incidence (odds ratio, 0.55, 95% confidence interval, 0.35 to 0.86; P=.008) Polyzos et al. Am J Obstet Gynecol. 2009 1/23
Periodontal disease treatment and preterm birth 11 trials (6558 women) were included Treatment had no significant effect on the overall rate of preterm birth (odds ratio 1.15, 95% confidence interval 0.95 to 1.40; P=0.15) Polyzos et al. BMJ. 2010 2/23
3/23 What is a meta-analysis? Estimation of a common effect size across studies Allows exploring heterogeneity Optional part of a systematic review SRs MAs
4/23 Meta-analysis is a two-stage process 1. an effect size and its variance is calculated for each study 2. a summary (pooled) effect estimate is calculated as a weighted mean of the effect sizes estimated in the individual studies To compute the weighted mean, we assign more weight to the more precise studies
5/23 Periodontal disease treatment and preterm birth Treatment No treatment Study x n x n Weight OR [95% CI] Sadatmansuri 2006 0 15 3 15 0.29% 0.12 [ 0.01, 2.45 ] Jeffcoat 2003 5 123 11 123 2.30% 0.43 [ 0.15, 1.28 ] Offenbacher 2006 9 35 14 32 2.57% 0.45 [ 0.16, 1.25 ] Lopez 2002 7 168 14 190 3.14% 0.55 [ 0.22, 1.39 ] Lopez 2005 18 563 17 282 5.91% 0.51 [ 0.26, 1.02 ] Tarranum 2007 53 99 68 89 6.89% 0.36 [ 0.19, 0.67 ] Oliveira 2011 27 116 31 117 7.70% 0.84 [ 0.46, 1.53 ] Michalowicz 2006 44 402 38 391 12.98% 1.14 [ 0.72, 1.81 ] Macones 2010 58 359 47 361 15.76% 1.29 [ 0.85, 1.95 ] Newnham 2009 52 538 50 535 16.37% 1.04 [ 0.69, 1.56 ] Offenbacher 2009 91 881 73 880 26.09% 1.27 [ 0.92, 1.76 ] IV, fixed effect model Q= 25.9, P value= 0.0038 ; I²= 61.4 % Favours treatment 100.00% 0.94 [ 0.80, 1.11 ] Favours no treatment 0.1 0.2 0.5 1.0 2.0 4.0 10.0 Odds Ratio (log scale)
6/23 Quantifying heterogeneity Differences across studies in a meta-analysis (statistical) heterogeneity = variation in the true effects underlying different studies which may manifest itself in more observed variation than expected by chance alone
Common way to quantify heterogeneity I 2 coefficient between 0% (no observed heterogeneity) and 100% Proportion of total variability explained by heterogeneity 0% to 40%: might not be important 30% to 60%: may represent moderate heterogeneity 50% to 90%: may represent substantial heterogeneity 75% to 100%: considerable heterogeneity Higgins and Thompson. Stat Med 2002 7/23
8/23 Fixed-effect meta-analysis Treatment No treatment Study x n x n Weight OR [95% CI] Sadatmansuri 2006 0 15 3 15 0.29% 0.12 [ 0.01, 2.45 ] Jeffcoat 2003 5 123 11 123 2.30% 0.43 [ 0.15, 1.28 ] Offenbacher 2006 9 35 14 32 2.57% 0.45 [ 0.16, 1.25 ] Lopez 2002 7 168 14 190 3.14% 0.55 [ 0.22, 1.39 ] Lopez 2005 18 563 17 282 5.91% 0.51 [ 0.26, 1.02 ] Tarranum 2007 53 99 68 89 6.89% 0.36 [ 0.19, 0.67 ] Oliveira 2011 27 116 31 117 7.70% 0.84 [ 0.46, 1.53 ] Michalowicz 2006 44 402 38 391 12.98% 1.14 [ 0.72, 1.81 ] Macones 2010 58 359 47 361 15.76% 1.29 [ 0.85, 1.95 ] Newnham 2009 52 538 50 535 16.37% 1.04 [ 0.69, 1.56 ] Offenbacher 2009 91 881 73 880 26.09% 1.27 [ 0.92, 1.76 ] IV, fixed effect model Q= 25.9, P value= 0.0038 ; I²= 61.4 % Favours treatment 100.00% 0.94 [ 0.80, 1.11 ] Favours no treatment 0.1 0.2 0.5 1.0 2.0 4.0 10.0 Odds Ratio (log scale)
9/23 Fixed-effect meta-analysis Treatment No treatment Study x n x n Weight OR [95% CI] Sadatmansuri 2006 0 15 3 15 0.29% 0.12 [ 0.01, 2.45 ] Jeffcoat 2003 5 123 11 123 2.30% 0.43 [ 0.15, 1.28 ] Offenbacher 2006 9 35 14 32 2.57% 0.45 [ 0.16, 1.25 ] Lopez 2002 7 168 14 190 3.14% 0.55 [ 0.22, 1.39 ] Lopez 2005 Tarranum 2007 Oliveira 2011 18 53 27 563 99 116 17 68 31 282 89 117 Random error 5.91% 0.51 [ 0.26, 1.02 ] 6.89% 0.36 [ 0.19, 0.67 ] 7.70% 0.84 [ 0.46, 1.53 ] Michalowicz 2006 44 402 38 391 12.98% 1.14 [ 0.72, 1.81 ] Macones 2010 58 359 47 361 15.76% 1.29 [ 0.85, 1.95 ] Newnham 2009 52 538 50 535 16.37% 1.04 [ 0.69, 1.56 ] Offenbacher 2009 91 881 73 880 26.09% 1.27 [ 0.92, 1.76 ] IV, fixed effect model True common effect size 100.00% 0.94 [ 0.80, 1.11 ] 0.1 0.2 0.5 1.0 2.0 4.0 10.0 Odds Ratio (log scale)
10/23 Random-effects meta-analysis Treatment No treatment Study x n x n Weight OR [95% CI] Sadatmansuri 2006 0 15 3 15 0.90% 0.12 [ 0.01, 2.45 ] Jeffcoat 2003 5 123 11 123 5.23% 0.43 [ 0.15, 1.28 ] Offenbacher 2006 9 35 14 32 5.63% 0.45 [ 0.16, 1.25 ] Lopez 2002 7 168 14 190 6.42% 0.55 [ 0.22, 1.39 ] Lopez 2005 18 563 17 282 9.12% 0.51 [ 0.26, 1.02 ] Tarranum 2007 53 99 68 89 9.77% 0.36 [ 0.19, 0.67 ] Oliveira 2011 27 116 31 117 10.24% 0.84 [ 0.46, 1.53 ] Michalowicz 2006 44 402 38 391 12.28% 1.14 [ 0.72, 1.81 ] Macones 2010 58 359 47 361 12.95% 1.29 [ 0.85, 1.95 ] Newnham 2009 52 538 50 535 13.07% 1.04 [ 0.69, 1.56 ] Offenbacher 2009 91 881 73 880 14.38% 1.27 [ 0.92, 1.76 ] IV, random effects model Q= 25.9, P value= 0.0038 ; I²= 61.4 % Favours treatment 100.00% 0.79 [ 0.58, 1.06 ] Favours no treatment 0.1 0.2 0.5 1.0 2.0 4.0 10.0 Odds Ratio (log scale)
11/23 Random-effects meta-analysis Treatment No treatment Study x n x n Weight OR [95% CI] Sadatmansuri 2006 Jeffcoat 2003 Offenbacher 2006 0 5 9 15 123 35 3 11 14 15 123 32 True effect underlying each study 0.90% 0.12 [ 0.01, 2.45 ] 5.23% 0.43 [ 0.15, 1.28 ] 5.63% 0.45 [ 0.16, 1.25 ] Lopez 2002 7 168 14 190 6.42% 0.55 [ 0.22, 1.39 ] Lopez 2005 Tarranum 2007 Oliveira 2011 18 53 27 563 99 116 17 68 31 282 89 117 Random error 9.12% 0.51 [ 0.26, 1.02 ] 9.77% 0.36 [ 0.19, 0.67 ] 10.24% 0.84 [ 0.46, 1.53 ] Michalowicz 2006 44 402 38 391 12.28% 1.14 [ 0.72, 1.81 ] Macones 2010 58 359 47 361 12.95% 1.29 [ 0.85, 1.95 ] Newnham 2009 52 538 50 535 13.07% 1.04 [ 0.69, 1.56 ] Offenbacher 2009 91 881 73 880 14.38% 1.27 [ 0.92, 1.76 ] Random effects model True mean effect size Distribution of effects θ i ~ N(θ, τ 2 ) 100.00% 0.79 [ 0.58, 1.06 ] 0.1 0.2 0.5 1.0 2.0 4.0 10.0 Odds Ratio (log scale)
12/23 Assessing the quality of a body of evidence Biases resulting from flaws in RCT design and conduct Small-study effect and reporting biases Publication bias Selective reporting of outcomes
Flaws in the design and conduct of trials 13/23
14/23 Effects of flaws in the design and conduct of trials Including biased trials will cause meta-analyses to be biased Change in average intervention effect (bias) Increase in between-trial heterogeneity Effect of bias will vary across meta-analyses
15/23 Incorporating risk of bias assessments into analysis Including all studies in a MA may produce a result with high precision but biased because of flaws in some of the studies Including only the studies at low risk of bias may produce an unbiased result but imprecise It is recommended to restrict the primary analysis to studies at low (or low and unclear) risk of bias
Subgroup analysis according to Risk of Bias Polyzos et al. BMJ. 2010 16/23
17/23 Funnel plot to assess small-study effect 0 Standard Error 1 2 3 0.1 0.33 0.6 1 3 Effect 10
18/23 Asymmetrical funnel plot 0 Standard Error 1 2 3 0.1 0.33 0.6 1 3 Effect 10
19/23 Asymmetrical funnel plot 0 Standard Error 1 2 Unpublished studies 3 0.1 0.33 0.6 1 3 Effect 10
20/23 Asymmetrical funnel plot 0 Standard Error 1 2 Small studies all finding positive effects 3 0.1 0.33 0.6 1 3 Effect 10
Small-study effect Polyzos et al. BMJ. 2010 21/23
22/23 Take-home messages Meta-analysis is the statistical combination of results from two or more separate studies Improve the estimation of treatment effects by pooling results of similar studies and exploring sources of heterogeneity Potential to mislead seriously if variation across studies, within-study biases and reporting biases are not carefully considered We must consider the degree of statistical heterogeneity, risk of bias and small-study effects when selecting the most appropriate statistical model to combine the evidence The best decision may be to focus on the best available evidence (ie, larger and methodologically robust trials)
23/23 Resources Cochrane Handbook http://training.cochrane.org/handbook Cochrane Review Manager http://community.cochrane.org/tools/review-productiontools/revman-5 OpenMeta http://www.cebm.brown.edu/openmeta/ PRISMA reporting guideline http://www.prisma-statement.org/