Choice of axis, tests for funnel plot asymmetry, and methods to adjust for publication bias

Transcription

1 Technical appendix Choice of axis, tests for funnel plot asymmetry, and methods to adjust for publication bias Choice of axis in funnel plots Funnel plots were first used in educational research and psychology, with effect estimates plotted against sample size. 1 It is now usually recommended that the standard error of the effect estimate be plotted on the vertical axis 2 because statistical power is determined by factors additional to sample size, such as the number of outcome events (dichotomous outcomes) and the standard deviation of responses (continuous outcomes). Ratio measures of intervention effect should be plotted on a logarithmic scale, so that effects of the same magnitude but opposite directions (eg, odds ratios of 0.5 and 2) are equidistant from 1.0. For outcomes measured on a continuous scale (such as blood pressure) intervention effects are usually measured as mean differences or standardised mean differences, which should be used as the horizontal axis. For mean differences (assuming the outcome standard deviation is similar across studies), the standard error is proportional to the inverse of the square root of sample size and is therefore an uncontroversial choice for the vertical axis. Tests for funnel plot asymmetry The table lists proposed tests for funnel plot asymmetry. Because these tests typically have low power, even when a test does not provide evidence of asymmetry, bias (including publication bias) cannot be excluded. There is insufficient evidence to recommend use of some tests. That proposed by Begg and Mazumdar 3 has the same statistical problems but lower power than the test of Egger et al, and is therefore not recommended. The test proposed by Tang and Liu 5 has not been evaluated in simulation studies, while that proposed by Macaskill et al 6 has lower power than more recently proposed alternatives. The test proposed by Schwarzer et al 10 avoids the mathematical association between the log odds ratio and its standard error but has low power relative to tests discussed above. The test proposed by Deeks et al 7 (which was not designed for meta-analyses of randomised trials) is likely to have lower power than more recently proposed alternatives.

2 Proposed tests for funnel plot asymmetry Reference Basis of test All outcomes Begg and Mazumdar Rank correlation between standardised intervention effect and its (1994) 3 standard error Linear regression of intervention effect estimate against its standard error, weighted by the inverse of the variance of the intervention Egger et al (1997) 4 effect estimate Linear regression of intervention effect estimate on 1 / N tot, with Tang and Liu (2000) 5 weights N tot Dichotomous outcomes only Linear regression of intervention effect estimate on N tot, with weights Macaskill et al (2001) 6 * S F/N tot Linear regression of log odds ratio on 1/ ESS with weights ESS, Deeks et al (2005) 7 * where effective sample size ESS = 4N E N C / N tot Modified version of the test proposed by Egger et al, based on the Harbord et al (2006) 8 * score (O E) and score variance (V) of the log odds ratio Linear regression of intervention effect estimate on 1/N tot, with Peters et al (2006) 9 * weights S F/N tot Rank correlation test, using mean and variance of the non-central Schwarzer et al (2006) 10 * hypergeometric distribution Test based on arcsine transformation of observed risks, with explicit Rücker et al (2008) 11 modelling of between-study heterogeneity N tot is the total sample size, N E and N C are the sizes of the experimental and control intervention groups, S is the total number of events across both groups and F=N tot S. *These tests were formulated to address the artefactual association between the log odds ratio and its standard error. They could potentially be modified to apply to other measures of intervention effect such as risk ratios, for which similar problems exist. For dichotomous outcomes with intervention effects measured as risk ratios or risk differences, and continuous outcomes with intervention effects measured as standardised mean differences, potential problems in tests for funnel plot asymmetry have been less extensively studied than for odds ratios, and firm guidance is not available. Meta-analyses of risk differences have been demonstrated empirically to display more heterogeneity than metaanalyses using a ratio measure of effect 12 and to be more likely to be correlated with the baseline risk. 13 Therefore, funnel plot asymmetry detected using risk differences may be artefactual. In such a case, the better metric is that for which the effects are more nearly equal across studies, often the risk ratio or odds ratio. Funnel plots and related tests using risk differences should seldom be of interest.

3 Methods to adjust intervention effect estimates for publication bias The trim and fill method aims both to identify and correct for publication bias The basis of the method is to (1) trim (remove) the smaller studies causing funnel plot asymmetry, (2) use the trimmed funnel plot to estimate the true centre of the funnel, then (3) replace the omitted studies and their missing counterparts around the centre (filling). As well as providing an estimate of the number of missing studies, an adjusted intervention effect is derived by performing a meta-analysis including the filled studies. The trim and fill method provides an estimate of the number of missing studies, and also provides an estimated intervention effect adjusted for the publication bias (based on the filled studies). However, because it is built on the strong assumption that there should be a symmetric funnel plot (it does not take into account reasons for funnel plot asymmetry other than publication bias), there is no guarantee that the adjusted intervention effect matches what would have been observed in the absence of publication bias. The method has not been shown to have desirable properties in the presence of substantial between study heterogeneity Therefore, corrected intervention effect estimates from this method should be interpreted with great caution. Publication and other reporting biases may lead to over-optimistic estimates of intervention effects. Several approaches to detect and correct for such biases have been proposed. 18 One family of tests tries to model the selection process that is, the process by which studies are selected for publication based on the P value for the effect of interest. Studies with a particular range of P values may be assumed to be more heavily represented in the evidence than studies with different ranges. For example, Dear and Begg 19 and Hedges 20 assumed there are different publication probabilities for studies with different ranges of P value ( , etc.). Ioannidis and Trikalinos proposed a test for an excess of significant findings in large meta-analyses. 21 All these approaches, like tests for funnel plot asymmetry, have limited applicability because they need a large number of studies to have reasonable power. Often, the best that can be done is to examine for bias in a whole field of research rather than in a small meta-analysis within this field However, one cannot be certain that this average bias affects all meta-analyses in this field equally Selection models have been extended to correct effect estimates for publication bias. These methods have not been widely used in practice, perhaps because of their complexity and the large number of studies needed. Their results may depend strongly on modelling assumptions: Begg argued that use of selection models should be restricted to identification of bias rather than correcting for it. 27 To address the difficulty in estimating model parameters

4 when the meta-analysis is not large, Copas proposed sensitivity analyses based on selection models, in which the value of the estimated treatment effect is computed under a range of assumptions about the severity of the selection bias A further problem with these approaches is that causes of funnel plot asymmetry other than publication bias may distort the observed range of P values and hence appear to provide evidence of publication bias. Simulation studies suggest that both extrapolation within funnel plots 32 and the Copas selection model outperform the trim and fill approach. 35 Adjusted effect estimates derived from any of these approaches should be interpreted with caution. 1 Light RJ, Pillemer DB. Summing up: the science of reviewing research. Harvard University Press, Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol 2001;54: Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994;50: Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997;315: Tang J-L, Liu JL. Misleading funnel plot for the detection of bias in meta-analysis. J Clin Epidemiol 2000;53: Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Stat Med 2001;20: Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. J Clin Epidemiol 2005;58: Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in metaanalyses of controlled trials with binary endpoints. Stat Med 2006;25: Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA 2006;295: Schwarzer G, Antes G, Schumacher M. A test for publication bias in meta-analysis with sparse binary data. Stat Med 2007;26: Rücker G, Schwarzer G, Carpenter J. Arcsine test for publication bias in meta-analyses with binary outcomes. Stat Med 2008;27:

5 12 Engels EA, Schmid CH, Terrin NT, Olkin I, Lau J. Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. Stat Med 2000;19: Schmid CH, Lau J, McIntosh MW, Cappelleri JC. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med 1998;17: Taylor SJ, Tweedie RL. Practical estimates of the effect of publication bias in metaanalysis. Australian Epidemiologist 1998;5: Duval S, Tweedie R. Trim and fill: a simple funnel plot based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000;56: Terrin N, Schmid CH, Lau J, Olkin I. Adjusting for publication bias in the presence of heterogeneity. Stat Med 2003;22: Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Stat Med 2007;26: Rothstein, HR, Sutton AJ, Borenstein M. Publication bias in meta-analysis: prevention, assessment and adjustments. Wiley, Dear KBG, Begg CB. An approach to assessing publication bias prior to performing a meta-analysis. Stat Sci 1992;7: Hedges LV. Modeling publication selection effects in meta-analysis. Stat Sci 1992;7: Ioannidis JP, Trikalinos TA. An exploratory test for an excess of significant findings. Clin Trials 2007;4: Moreno SG, Sutton AJ, Turner EH, Abrams KR, Cooper NJ, Palmer TM, et al. Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. BMJ 2009;339:b Kavvoura FK, McQueen MB, Khoury MJ, Tanzi RE, Bertram L, Ioannidis JP. Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer s disease. Am J Epidemiol 2008;168:

6 24 Hedges LV, Vevea JL. Estimating effect size under publication bias: small sample properties and robustness of a random effects selection model. J Educ Behav Stat 1994;21: Vevea JL, Hedges LV. A general linear model for estimating effect size in the presence of publication bias. Psychometrika 1995;60: Givens GH, Smith DD, Tweedie RL. Publication bias in meta-analysis: a Bayesian data-augmentation approach to account for issues exemplified in the passive smoking debate. Stat Sci 1997;12: Begg CB. Publication bias in meta-analysis. Stat Sci 1997;12: Copas J. What works? Selectivity models and meta-analysis. J R Statist Soc A 1999;162: Copas JB, Shi JQ. A sensitivity analysis for publication bias in systematic reviews. Stat Methods Med Res 2001;10: Henmi M, Copas JB, Eguchi S. Confidence intervals and P-values for meta-analysis with publication bias. Biometrics 2007;63: Vevea JL, Woods CM. Publication bias in research synthesis: sensitivity analysis using a priori weight functions. Psychol Methods 2005;10: Moreno SG, Sutton AJ, Ades AE, Stanley TD, Abrams KR, Peters JL, et al. Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Med Res Methodol 2009;9:2. 33 Carpenter JR, Schwarzer G, Rucker G, Kunstler R. Empirical evaluation showed that the Copas selection model provided a useful summary in 80% of meta-analyses. J Clin Epidemiol 2009;62: Schwarzer G, Carpenter J, Rucker G. Empirical evaluation suggests Copas selection model preferable to trim-and-fill method for selection bias in meta-analysis. J Clin Epidemiol 2010;63: Rucker G, Carpenter JR, Schwarzer G. Detecting and adjusting for small-study effects in meta-analysis. Biom J 2011;53: