Contour enhanced funnel plots for meta-analysis Tom Palmer 1, Jaime Peters 2, Alex Sutton 3, Santiago Moreno 3 1. MRC Centre for Causal Analyses in Translational Epidemiology, Department of Social Medicine, University of Bristol 2. PenTAG/PenCLAHRC, Peninsula Medical School 3. Department of Health Sciences, University of Leicester 11 September 2009 Centre for Causal Analyses in Translational Epidemiology
Outline Introduction to funnel plots & contour enhanced funnel plots Moreno, Sutton, Turner, et al., 2009 BMJ example - Use with other bias assessment methods confunnel: syntax and options Discussion 1 / 13
Introduction to funnel plots Funnel plot with pseudo 95% confidence limits of log(or) 1.5 0 1.5.05.1.25.5 1 2 4 8 16 Odds ratio Plot of std error (y-axis) versus effect estimate (x-axis) Help assess small study reporting bias/publication bias Sterne & Harbord, 2004; metafunnel, funnel Same metric as Egger s test (Egger, Davey Smith, Schneider, & Minder, 1997) 2 / 13
Introduction to contour enhanced funnel plots Indicate regions of statistical significance on funnel plot Spiegelhalter, 2002, 2005; Peters, Sutton, Jones, Abrams, & Rushton, 2008 Funnel plot with pseudo 95% confidence limits 0 Studies 0 p < 1% 1% < p < 5% 5% < p < 10% of log(or).5 1.5 1 p > 10% 1.5.05.1.25.5 1 2 4 8 16 Odds ratio 1.5 4 2 0 2 4 Effect estimate 3 / 13
Moreno, Sutton, Turner, et al., 2009, BMJ, example Re-analysis of Turner, Matthews, Linardatos, Tell, & Rosenthal, 2008, NEJM Results of 74 trials of 12 antidepressant drugs Compare FDA results versus journal results 4 / 13
Moreno, Sutton, Turner, et al., 2009, BMJ, example Re-analysis of Turner et al., 2008, NEJM T h e n e w e ng l a nd j o u r na l o f m e dic i n Results of 74 trials of 12 antidepressant T h e drugs new engl and jour nal o f medicine Compare FDA results versus journal results Published, agrees with FDA decision Figure 2 (facing page). P Published, agrees with FDA decision Figure 2 (facing page). Publication S Published, Published, conflicts with with FDA FDA decision decision Regulatory Regulatory Decision by Decision Study and by b Not Not published Panel A Panel shows A the shows publication the publ statu studies. studies. Nearly every Nearly study every deemed st FDA (top row) was published in a w FDA (top row) was pub the FDA s judgment. By contrast, m Positive 37 deemed the negative FDA s (bottom judgment. row) or By q Positive (N=38) (97%) 37 dle row) deemed by the FDA negative either were (botto pub (N=38) (97%) 1 conflicted dle with row) the by FDA s the judgment FDA eith (3%) lished. Numbers shown in boxes ind Questionable 6 6 1 conflicted with the FDA (N=12) (50%) (50%) (3%) studies and correspond to the study lished. Numbers shown Questionable 6 6 Table A of the Supplementary Appe (N=12) (50%) (50%) shows the studies numbers and of correspond patients part Negative 5 16 individual Table studies A of indicated the Supplem in Pane (N=24) (21%) (67%) tients who shows participated the numbers in studies of dp Negative 53 16 the FDA were very likely to be publis (N=24) (12%) individual studies indic (21%) (67%) agreed with the FDA s judgment. By patients tients who participated who participated in studies 0 3 10 20 30 40 or questionable the FDA by were the FDA very tended likely (12%) published agreed or to be with published the FDA s in a wa j with the patients FDA s judgment. who participat 0 10 20 30 40 or questionable by the F published or to be publ Positive 7075 A list of the study-level effec (N=7155) (99%) with the FDA s judgmen in the above analyses derive 80 FDA reviews and the published 4 / 13
r B A Fixed effect meta-analysis journal data B FDA estimate Fixed effect meta-analysis journal data Fixed effect meta-analysis journal data Filled study Fixed effect trim and fill Significance level <1% Significance level 1-5% Significance level 5-10% Significance level >10% F DISCU The a and ad from as set exi tion o contou sion b size, wt datase fill met This Firstlyf are no Specifn maceu the FDt the me for thec not be there mo cant ones). Deb plots a tion bp some 5 / 13 m t s
Trim & fill: Duval & Tweedie, 2000b, 2000a C Fixed effect meta-analysis journal A data Fixed effect meta-analysis FDA data Filled study Fixed effect trim and fill B FDA estimate Fixed effect meta-analysis journal data Fixed effect meta-analysis journal data Filled study Significance level <1% Significance level 1-5% Significance level 5-10% Significance level >10% ones). T Deba plots an DISCU The a and ad from a set exi tion o contou sion b size, w datase fill me This Firstly are no Specif maceu the FD A the me for the not be there m cant ones). Deb plots a tion b some tion bia some q 6 / 13
r A B Significance level 1-5% Significance level 5-10% Significance level >10% FDA estimate FDA estimate A Fixed effect meta-analysis journal data Fixed effect meta-analysis journal data Filled study Fixed effect trim and fill Significance level <1% Significance level 1-5% Significance level 5-10% Significance level >10% DISCU The a and ad from a set exi tion o contou sion b size, w datase fill me This Firstly are no Specif maceu the FD the me for the not be there m cant ones). Deb plots a tion b some 7 / 13
some question their validity, including in this A FDA to journal change in effect Significance level <1% Significance level 1-5% Significance level 5-10% Significance level >10% rror FDA estimate Fixed effect meta-analysis unpublished B 8 / 13
r.0 te B B FDA estimate FDA estimate A Fixed effect meta-analysis unpublished Fixed effect meta-analysis journal data Fixed effect meta-analysis journal data Filled study Fixed effect trim and fill Significance level <1% Significance level 1-5% Significance level 5-10% Significance level >10% -1.0-0.5 0 0.5 1.0 Effect estimate DISCU The a and ad from a set exi tion o contou sion b size, w datase fill me This Firstly are no Specif maceu the FD the me for the not be there m cant ones). Deb plots a tion b some 9 / 13
Regression based bias adjustment methods: Shanget al., 2005; Moreno, Sutton, Ades, et al., 2009 D RESEARCH Fixed effect meta-analysis journal A data Fixed effect meta-analysis FDA data Regression line Fixed effect trim and fill B FDA estimate Fixed effect meta-analysis journal data -1.0-0.5 0 0.5 1.0 Fixed effect meta-analysis journal data Stand Significance level <1% Significance level 1-5% Significance level 5-10% Significance level >10% -1.0 DISCU The a and ad from a set exi tion o contou sion b size, w datase fill me This B Firstly are no Specif maceu the FD the me for the not be there m cant ones). Deb plots a tion b some 10 / 13
The confunnel command Peters et al., 2008 - investigation of 48 published Cochrane meta-analyses 11 / 13
The confunnel command Peters et al., 2008 - investigation of 48 published Cochrane meta-analyses Sterne et al., 2008, Cochrane Handbook, section 1.b 11 / 13
The confunnel command Peters et al., 2008 - investigation of 48 published Cochrane meta-analyses Sterne et al., 2008, Cochrane Handbook, section 1.b Palmer et al., 2008 (v1.) Palmer et al., 2009 (v1.0.5) 11 / 13
The confunnel command Peters et al., 2008 - investigation of 48 published Cochrane meta-analyses Sterne et al., 2008, Cochrane Handbook, section 1.b Palmer et al., 2008 (v1.) Palmer et al., 2009 (v1.0.5) 11 / 13
confunnel: syntax and options Syntax: confunnel logor selogor [, options ] 12 / 13
confunnel: syntax and options Syntax: confunnel logor selogor [, options ] Options: metric(se invse var invvar): different y-axes: variance, standard error & their inverses (Sterne & Egger, 2001) onesided(lower upper): one sided significance levels Other twoway options 12 / 13
Contour enhanced funnel plots discussion Funnel plots should be used with care (Lau, Ioannidis, Terrin, Schmid, & Olkin, 2006) Aid assessment of reporting biases Put other bias assessment methods in a context confunnel can be used with metan, metabias, metatrim 13 / 13
References I Duval, S., & Tweedie, R. L. (2000a). A nonparametric trim and fill method of accounting for publication bias in meta-analysis. Journal of the American Statistical Society, 95, 89 98. Duval, S., & Tweedie, R. L. (2000b). Trim and fill: A simple funnel plot based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56, 455 463. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315(7109), 629 634. Lau, J., Ioannidis, J., Terrin, N., Schmid, C. H., & Olkin, I. (2006). The case of the misleading funnel plot. British Medical Journal, 333(7568), 597 600. Moreno, S. G., Sutton, A. J., Ades, A. E., Stanley, T. D., Abrams, K. R., Peters, J. L., et al. (2009). Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 2. (published online 12 January 2009) Moreno, S. G., Sutton, A. J., Turner, E. H., Abrams, K. R., Cooper, N. J., Palmer, T. M., et al. (2009). Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. British Medical Journal, 339, 494 498. (b2981) Palmer, T. M., Peters, J. L., Sutton, A. J., & Moreno, S. G. (2008). Contour enhanced funnel plots for meta-analysis. The Stata Journal, 8(2), 242 254. Available from http://www.stata-journal.com/article.html?article=gr0033
References II Palmer, T. M., Peters, J. L., Sutton, A. J., & Moreno, S. G. (2009). Meta-Analysis in Stata: An Updated Collection from The Stata Journal. In J. A. C. Sterne (Ed.), (pp. 124 137). College Station, Texas: Stata Press. Available from http://www.stata.com/bookstore/mais.html Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61(10), 991 996. Shang, A., Huwiler-Müntener, K., Nartey, L., Jüni, P., Dörig, S., Sterne, J. A. C., et al. (2005). Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. The Lancet, 366(9487), 726 732. Spiegelhalter, D. J. (2002). Funnel plots for institutional comparison. Quality Safety in Health Care, 11(4), 390 391. Spiegelhalter, D. J. (2005). Funnel plots for comparing institutional performance. Statistics in Medicine, 24(8), 1185 1202. Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. Journal of Clinical Epdiemiology, 54(10), 1046 1055.
References III Sterne, J. A. C., Egger, M., & Moher, D. (2008, September). Cochrane handbook for systematic reviews of interventions version 5.. In J. P. T. Higgins & S. Green (Eds.), (chap. Chapter 10: Addressing reporting biases). The Cochrane Collaboration. Available from http://www.mrc-bsu.cam.ac.uk/cochrane/handbook/chapter 10/ figure 10 4 b contour enhanced funnel plots.htm Sterne, J. A. C., & Harbord, R. M. (2004). Funnel plots in meta-analysis. The Stata Journal, 4(2), 127 141. Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358(3), 252 260.