heterogeneity in meta-analysis stats methodologists meeting 3 September 2015

Transcription

1 heterogeneity in meta-analysis stats methodologists meeting 3 September 2015

2 basic framework k experiments (i=1 k) each gives an estimate x i of θ, with variance s i 2 best overall estimate is weighted (by w i =1/s i2 ) mean of the x i if the x i are really all estimating the same θ

3 The Combination of Estimates from Different Experiments by William G. Cochran Biometrics, Vol. 10, No. 1 (1954), pp , URL:

4 Quantifying heterogeneity in a meta-analysis by J. P. T. Higgins & S. G. Thompson Statist. Med. 2002; 21:

5 Quantifying heterogeneity in a meta-analysis by J. P. T. Higgins & S. G. Thompson Statist. Med. 2002; 21:

6 Assessing uncertainty in physical constants Max Henrion and Baruch Fischhoff Am. J. Phys. 54, 791 (1986);

7 Assessing uncertainty in physical constants Max Henrion and Baruch Fischhoff Am. J. Phys. 54, 791 (1986);

8 Assessing uncertainty in physical constants Max Henrion and Baruch Fischhoff Am. J. Phys. 54, 791 (1986);

9 if there is heterogeneity (interaction, overdispersion ) move from fixed effects model: to random effects

10 what is the problem with an additional additive component of variance in random effects M-A? no matter how big a trial is, its (statistical) weight is limited by τ which is particularly a problem if you are planning/reporting the next one

11 Trial sequential methods for meta analysis E Kulinskaya & J Wood RESEARCH SYNTHESIS METHODS 5(3): Sep 2014 MRC requires a comprehensive review of existing evidence before funding trials guidelines of (e.g.) JAMA and the Lancet state that reports of new clinical trials must explain how they affect previous research with direct reference to existing meta-analysis which suggests performing power calculations for sample size for the next trial however, for large estimated or assumed values of τ 2, several extra trials are needed, regardless of how large each may be

12 EXPLAINING HETEROGENEITY IN META-ANALYSIS: A COMPARISON OF METHODS S.G. THOMPSON & S.J. SHARP Statist. Med. 18, (1999) incorporating residual heterogeneity into a metaregression model: allow a multiplicative factor: include an additive between-study variance component τ 2 :

13 THOMPSON & SHARP (cont) The rationale for using a multiplicative factor for variance inflation is weak has little intuitive appeal leads to the same dominance of large studies over small studies that has been criticized in the context of fixed-effect meta-analysis we do not recommended them. The use of an additive component of variance to represent heterogeneity between studies is more intuitively appealing, and of course is the usual way of representing heterogeneity in meta-analysis

14 Confidence intervals for random effects metaanalysis by M. Henmi & J.B. Copas Stat Med Dec 20;29(29): fixed effects estimates are less sensitive to (publication) biases than random effects estimates, since they put relatively more weight on the larger studies and relatively less weight on the smaller studies. Whereas the DerSimonian-Laird interval is centred on a random effects estimate, we centre our confidence interval on a fixed effects estimate, but allow for heterogeneity by including an assessment of the extra uncertainty induced by the random effects setting

15 The Combination of Estimates from Different Experiments by William G. Cochran Biometrics, Vol. 10, No. 1 (1954), pp , URL: This paper discusses methods for combining a number of estimates x i of some quantity θ, made in different experiments It is important to find out whether the x i agree with one another within the limits of their experimental errors If they do not, the type of overall mean that will be useful for future action requires careful consideration.

16 [supporting info.]

17 full quote from Cochran 1954 This paper discusses methods for combining a number of estimates x i of some quantity θ, made in different experiments. For the ith estimate we have an unbiased estimate s 2 of its variance, based on n i degrees of freedom. It is important to find out whether the x i agree with one another within the limits of their experimental errors. If they do not, i.e. if interactions are present, the type of overall mean that will be useful for future action requires careful consideration. However, in most cases the problem of estimating the mean of the x i, at least over some subgroup of the experiments, will remain.

18 full quote from THOMPSON & SHARP (1999) The rationale for using a multiplicative factor for variance inflation is weak. The idea that the variance of the estimated effect within each study should be multiplied by some constant has little intuitive appeal, and leads to the same dominance of large studies over small studies that has been criticized in the context of fixed-effect meta-analysis. Thus, despite the fact that such analyses are easy to carry out, and might therefore be used as a quick and approximate way of assessing the impact of residual heterogeneity on the results, we do not recommended them in practice. The use of an additive component of variance to represent heterogeneity between studies is more intuitively appealing, and of course is the usual way of representing heterogeneity in meta-analysis without covariates as well as in many other situations

19 full quote from Henmi & Copas (2010) Stat Med Dec 20;29(29): doi: /sim Epub 2010 Oct 20. Confidence intervals for random effects meta-analysis and robustness to publication bias. Abstract The DerSimonian-Laird confidence interval for the average treatment effect in metaanalysis is widely used in practice when there is heterogeneity between studies. However, it is well known that its coverage probability (the probability that the interval actually includes the true value) can be substantially below the target level of 95 per cent. It can also be very sensitive to publication bias. In this paper, we propose a new confidence interval that has better coverage than the DerSimonian-Laird method, and that is less sensitive to publication bias. The key idea is to note that fixed effects estimates are less sensitive to such biases than random effects estimates, since they put relatively more weight on the larger studies and relatively less weight on the smaller studies. Whereas the DerSimonian-Laird interval is centred on a random effects estimate, we centre our confidence interval on a fixed effects estimate, but allow for heterogeneity by including an assessment of the extra uncertainty induced by the random effects setting. Properties of the resulting confidence interval are studied by simulation and compared with other random effects confidence intervals that have been proposed in the literature. An example is briefly discussed.