Assessing publication bias in genetic association studies: evidence from a recent meta-analysis

Transcription

1 Psychiatry Research 129 (2004) Assessing publication bias in genetic association studies: evidence from a recent meta-analysis Marcus R. Munafò a, *, Taane G. Clark b, Jonathan Flint c a Department of Clinical Pharmacology, Cancer Research UK, General Practice Research Group, University of Oxford, Oxford OX2 6HE, UK b Centre for Statistics in Medicine, Cancer Research UK, University of Oxford, Oxford OX3 7LF, UK c Wellcome Trust Centre for Human Genetics, University of Oxford, Oxford OX3 7BN, UK Received 10 July 2003; received in revised form 12 June 2004; accepted 14 June 2004 Abstract Publication bias may exist when nonsignificant findings remain unpublished, thereby artificially inflating the apparent magnitude of an effect. This concern is not new, but it is particularly current in relation to genetic association studies. Data from a recent meta-analysis of association studies of personality were used to assess the potential of different graphical and statistical methods for assessing evidence of publication bias. The results suggest that no single method is sufficient for assessing evidence of publication bias, and that such methods may also offer insight into potential sources of heterogeneity, which may in turn guide the design of future studies. D 2004 Elsevier Ireland Ltd. All rights reserved. Keywords: Publication bias; Genetic association; Personality; Meta-analysis 1. Introduction * Corresponding author. Tel.: ; fax: address: marcus.munafo@clinpharm.ox.ac.uk (M.R. Munafò). Two recent meta-analyses of genetic association studies of personality (Kluger et al., 2002; Munafò, 2003) discuss the potential problem of publication bias against negative or nonsignificant results. This concern is not new and was raised almost 50 years ago in relation to psychiatric and psychological research (Sterling, 1959). One of these meta-analyses (Munafò, 2003) reported tentative evidence for a modest publication bias as contact with authors revealed unpublished data that were pertinent to the review, while the other meta-analysis (Kluger et al., 2002) concluded that the literature does not suffer from a serious publication bias, based on a visual inspection of a funnel plot of effect-size estimates plotted against corresponding standard errors. The funnel plot is a commonly used graphical test to assess publication bias in meta-analytic data sets (Egger et al., 1997). The rationale behind a funnel plot analysis is that if all studies come from a single population, then the plot should look like a funnel with the diameter of the funnel decreasing (i.e. effectsize estimate becoming more accurate) as sample size increases (Wang and Bushman, 1998). However, publication bias is not the only explanation for an /$ - see front matter D 2004 Elsevier Ireland Ltd. All rights reserved. doi: /j.psychres

2 40 M.R. Munafò et al. / Psychiatry Research 129 (2004) asymmetrical funnel plot. Other possibilities include true heterogeneity, methodological quality, multiple publication of small studies, poor design and analysis of small studies, confounding heterogeneity due to poor choice of effect measure, or chance (Petticrew et al., 1999). Furthermore, there are explanations for asymmetry that are specific to genetic studies, such as the violation of Hardy Weinberg equilibrium (Attia et al., 2003). Funnel plots are not ideal for the detection of publication bias for a number of reasons. First, visual inspection of the data can be unreliable when assessing whether the plot is symmetrical, especially when the number of studies is relatively small, as in the meta-analyses described. Second, funnel plots do not use the fact that the effect-size estimate in each study in a meta-analysis has an approximately Normal distribution if the study has a large enough sample size. Third, data can be distributed in a funnel shape even if the studies come from more than one population if the populations have the same mean but different variances (Wang and Bushman, 1998). Funnel plot methods require not only a large number of component studies, but also for these studies to range in sample size. An insufficient range of sample sizes means that there may not be sufficient large studies to form the apex of the predicted funnel. An alternative graphical test of publication bias may be derived by assessing the linearity of the Normal quantile plot (Wang and Bushman, 1998). This plot compares the quantiles of an observed distribution against the quantiles of the standard Normal distribution. In a meta-analysis, such a plot can be used to check the Normality assumption, investigate whether all studies come from a single population, and search for publication bias (Wang and Bushman, 1998). A range of statistical tests also exist that allow the hypothesis that a publication bias exists to be tested formally (Sutton et al., 2000), without the subjectivity inherent in a visual inspection of a graphical test. Rosenthal called publication bias the file drawer problem, (Rosenthal, 1979) and proposed that it could be assessed by calculating the fail-safe N : the number of negative studies (studies in which the treatment effect is zero) that would be needed to increase the P-value for the meta-analysis to above Unfortunately, the estimate of fail-safe N is highly dependent on the mean effect size that is assumed for the unpublished studies and, unlike other medical research methodology, does not focus on the size of the estimated effects and the associated confidence intervals. A number of alternative approaches are based on the assumption that an individual study s results affect its probability of publication. These methods are called selection models (Iyengar and Greenhouse, 1988; Dear and Begg, 1992; Hedges, 1992), and may be extended to estimate subgroup effects (Vevea and Hedges, 1995), corrected for the estimated publication bias. However, avoidance of strong assumptions about the nature of the selection mechanism means that a large number of studies are required so that a sufficient range of P-values is included. The complexity of the methods and the large number of studies needed probably explain why selection models have not been widely used in practice. In addition, results may depend heavily on the modeling assumptions used. Many factors may affect the probability of publication of a given set of results, and it is difficult, if not impossible, to model these adequately. Other approaches are statistical analogues of the funnel plot (Begg and Mazumdar, 1994; Egger et al., 1997). Begg and Mazumdar (1994) proposed an adjusted rank correlation method to examine the association between the effect estimates and their variances. Egger et al. (1997) introduced an approach that corresponds to a weighted regression of effect sizes on their standard errors, where the weights are inversely proportional to the variance of the effect size. The regression method is more sensitive than the rank correlation approach, but the sensitivity of both methods is generally low in meta-analyses based on fewer than 20 trials. An extension to the weighted regression method is to consider study size as one of several different possible explanations for heterogeneity between studies in multivariable meta-regression models. For example, the effects of study size, adequacy of randomisation, and type of blinding might be examined simultaneously. Finally, if evidence exists for a publication bias against nonsignificant findings in a particular domain of research, methods exist for estimating the true population effect size by adducing estimated effect sizes for the putative unpublished studies. The most common of these is the trim-and-fill method (Duval and Tweedie, 2000a,b). This method is based on

3 M.R. Munafò et al. / Psychiatry Research 129 (2004) adding studies to a funnel plot so that it becomes symmetrical. Smaller studies are omitted until the funnel plot is symmetrical (trimming). The trimmed funnel plot is used to estimate the true centre of the funnel, and then the omitted studies and their missing counterparts around the centre are replaced (filling). This approach provides an estimate of the number of missing studies and an adjusted treatment effect, including the filled studies. Simulation studies have found that the trim-and-fill method detects missing studies in a substantial proportion of metaanalyses, even in the absence of bias (Sterne et al., 2000). Thus, there is a danger that in many metaanalyses application of the method could mean adding and adjusting for non-existent studies in response to funnel plot asymmetry arising from nothing more than random variation (Egger et al., 2001). In general, the correction of effect estimates when publication bias is assumed to be present is problematic and a matter of ongoing debate. We report a case study where we assess the evidence for publication bias in genetic association studies of human personality, using published data reported in a recent comprehensive meta-analysis, by means of two graphical (funnel plot and Normal quantile plot) and two statistical (Begg and Egger) tests, and use the trimand-fill method to impute the theoretical values of suppressed or unpublished studies. For interested readers, a good summary of most aspects of publication bias can be found in Song et al. (2000). 2. Methods Data collected in a meta-analysis (Munafò, 2003) were used to test the null hypotheses that (a) the effectsize estimates reported in the included studies meet the Normality assumption, (b) the effect-size estimates are derived from a single population, and (c) there is no evidence for publication bias in the included literature. We investigated studies reporting data on the association of the 5HTT LPR polymorphism and avoidancerelated traits, and the DRD4 VNTR polymorphism and approach-related traits, as these are the two candidate genes and corresponding personality traits that have received the most empirical attention. Funnel and Normal quantile plots for the two data sets (5HTT LPR and DRD4 VNTR) were produced in S-Plus (version 6). To assess the linearity of the Normal quantile plot (and hence the Normality of the included data), we calculated the value of the Weisberg Bingham (WB) statistic, which is the square of the correlation between the order statistics and the Normal quantiles (Weisberg and Bingham, 1972). The order statistics were derived from the residuals of a linear regression of effect size on an intercept term, weighted by study size. Being a square correlation, the WB statistic will lie between 0 and 1: the smaller the statistic, the greater the evidence of non-normality. The statistical tests of Begg and Egger were implemented in S-Plus. Publication bias may be present if the rank correlation coefficient from the Begg test is close to 1, or the slope parameter from the Egger regression is high. Heterogeneity between studies was assessed using a chi-square test of homogeneity. The trim-and-fill values were estimated in a threestep process: (a) omitting small studies until the funnel plot was symmetrical, (b) estimating the centre of the funnel and (c) replacing the omitted studies and their missing counterparts around the centre. 3. Results Fig. 1 presents the funnel and Normal quantile plots calculated from the data for the 5HTT LPR and DRD4 VNTR polymorphisms. The funnel and Normal quantile plots show some evidence of a possible publication bias. The U-shaped Normal quantile plot for the DRD4 VNTR data indicates that the data have an asymmetric distribution that is skewed to the right. This shape implies that studies with nonsignificant results have been deleted. The Normal quantile plot for 5HTT LPR data shows five studies in the lower part that may be outliers, leading to potentially left skewness. There is no evidence to reject the null hypothesis of Normality for either the 5HTT LPR data (WB = 0.99, P = 0.72) or the DRD4 VNTR data (WB = 0.97, P = 0.38). All the points on the Normal quantile plot are well within the 95% confidence limits. There is evidence of heterogeneity for the 5HTT LPR ( P = 0.02) and DRD4 VNTR ( P < 0.001) data, implying that in both cases the studies may not come from a single population. In fact, the Normal quantile

4 42 M.R. Munafò et al. / Psychiatry Research 129 (2004) Fig. 1. Funnel and normal plots for avoidance and approach data sets. plot for the 5HTT LPR data may show two separate populations, evidenced by an apparent double bump in the plot distribution (Wang and Bushman, 1998), and there is a cluster of three studies with non- Caucasian samples (two Asian and one Afro-Caribbean) in the upper right. This is supported by the fact that the funnel plot does not converge to a single value as the study sample size increases (Wang and Bushman, 1998), unlike the DRD4 VNTR data. The Begg rank correlation coefficients for 5HTT LPR and DRD4 VNTR are 0.21 and 0.10, respectively. The Egger regression for 5HTT LPR produced a slope estimate (se) of 0.21 (0.08), which is not indicative of publication bias because of the small and negative coefficient. Similarly, the regression for DRD4 VNTR indicated no publication bias with a slope estimate (se) of 0.09 (0.15). The effect-size estimates for the hypothetical unpublished studies, using the trim-and-fill method (Egger et al., 2001), are represented as unfilled points in Fig. 1. We only applied the trim-and-fill method to the DRD4 VNTR data because its funnel plot looked asymmetrical, whilst the funnel plot for 5HTT LPR looked symmetrical. The summary effect size (se) for the DRD4 VNTR data before trim-and-fill was 0.10 (0.07) with a P-value of 0.11, while afterwards the effect size (se) was 0.01 (0.08), which was also nonsignificant ( P = 0.85).

5 M.R. Munafò et al. / Psychiatry Research 129 (2004) Discussion We found evidence for significant heterogeneity in the published data on the association between the 5HTT LPR polymorphism and avoidance behaviours, which may be due to these data being derived from two separate populations and, more tentatively, a publication bias against nonsignificant results. We also found similar evidence of significant heterogeneity in the published data on the association between the DRD4 VNTR polymorphism and approach behaviours, which may be due to a similar publication bias, although there was no evidence in this case to suggest that the data do not come from a single population. The clustering of effect size estimates in studies of the 5HTT LPR polymorphism in non-caucasian samples suggests that ethnicity may define the two populations indicated by the data. Unfortunately, it was not possible to perform a similar categorisation by sex as separate data on effect size estimates in men and women were not available, but one recent study indicates that sex may be an important moderator of this association (Du et al., 2000). Funnel plots should be used in most meta-analyses to provide a visual assessment of whether effect estimates are associated with study size. However, Normal quantile plots should be used to complement the funnel plots because, as we have found, they have the potential to detect other phenomena, such as whether there is more than one population in the data. Statistical methods that examine the evidence for funnel plot asymmetry are available, and meta-regression methods may be used to examine competing explanations for heterogeneity in effects between studies. The power of all these methods is limited, however, particularly for meta-analyses based on a small number of studies, and the results from such meta-analyses should therefore always be treated with caution (Egger et al., 2001). Methods that attempt to adjust the effect for potential publication bias, such as trim and fill, are effectively like sensitivity analyses, because untestable assumptions are usually made. However, it is important to test directly (rather than indirectly) some of the hypotheses that arise out the plots and analyses presented. Subsequently, the potential moderating effects of ethnicity and sex on genetic associations should be investigated using sensitivity analyses. Publication is just one of many biases that can occur when working with systematic reviews. Prevention of publication bias is better than applying methods for its cure. The main strategy for making metaanalyses less susceptible to publication bias is to make strenuous efforts to ensure that all studies, both published and unpublished, are found (Gilbody and Song, 2000). Acknowledgments Marcus Munafo holds a Cancer Research UK research fellowship. Taane Clark holds an NHS R&D fellowship. Jonathan Flint is supported by the Wellcome Trust. References Attia, J., Thakkinstian, A., D Este, C., Meta-analyses of molecular association studies: methodologic lessons for genetic epidemiology. Journal of Clinical Epidemiology 56, Begg, C.B., Mazumdar, M., Operating characteristics of rank correlation test for publication bias. Biometrics 50, Dear, K.B.G., Begg, C.B., An approach to assessing publication bias prior to performing a meta-analysis. Statistical Science 7, Du, L., Bakish, D., Hrdina, P.D., Gender differences in association between serotonin transporter gene polymorphism and personality traits. Psychiatric Genetics 10, Duval, S.J., Tweedie, R.L., 2000a. Trim and fill: a simple funnel plot based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56, Duval, S.J., Tweedie, R.L., 2000b. A non-parametric trim and fill method of assessing publication bias in meta-analysis. Journal of the American Statistical Association 95, Egger, M., Davey-Smith, G., Schneider, M., Minder, C., Bias in meta-analysis detected by a simple, graphical test. British Medical Journal 315, Egger, M., Davey-Smith, G., Altman, D., Systematic Reviews in Healthcare, 2nd ed. British Medical Journal Books, London. Gilbody, S.M., Song, F., Publication bias and the integrity of psychiatry research. Psychological Medicine 30, Hedges, L.V., Modeling publication selection effects in metaanalysis. Statistical Science 7, Iyengar, S., Greenhouse, J.B., Selection problems and the file drawer problem. Statistical Science 3, Kluger, A.N., Siegfried, Z., Ebstein, R.P., A meta-analysis of the association between DRD4 polymorphism and novelty seeking. Molecular Psychiatry 7,

6 44 M.R. Munafò et al. / Psychiatry Research 129 (2004) Munafò, M.R., Clark, T.G., Moore, L.R., Payne, E., Flint, J., Genetic polymorphisms and personality in healthy adults: a systematic review and meta-analysis. Molecular Psychiatry 8, Petticrew, M., Gilbody, S.M., Sheldon, T.A., Relation between hostility and coronary heart disease. British Medical Journal 319, Rosenthal, R., The File Drawer Problem and tolerance for null results. Psychological Bulletin 86, Song, F.C., Eastwood, A.J., Gilbody, S., Duley, L., Sutton, A.J., Publication and related biases. Health Technology Assessment 4, Sterling, T.D., Publication decision and their possible effects on inferences drawn from tests of significance or vice versa. Journal of the American Statistical Association 54, Sterne, J.A.C., Gavaghan, D., Egger, M., Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology 53, Sutton, A.J., Duval, S.J., Tweedie, R.L., Abrams, K.R., Jones, D.R., Empirical assessment of effect of publication bias on meta-analyses. British Medical Journal 320, Vevea, J.L., Hedges, L.V., A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, Wang, M.C., Bushman, B.J., Using Normal quantile plots to explore meta-analytic data sets. Psychological Methods 3, Weisberg, S., Bingham, C., An approximate analysis of variance test for non-normality suitable for machine calculation. Technometrics 17,