Estimation of effect sizes in the presence of publication bias: a comparison of meta-analysis methods

Transcription

1 Estimation of effect sizes in the presence of publication bias: a comparison of meta-analysis methods Hilde Augusteijn M.A.L.M. van Assen R. C. M. van Aert APS May 29, 2016

2 Today s presentation Estimation of effect sizes in the presence of publication bias: a comparison of meta-analysis methods: PET-PEESE (Stanley & Doucouliagos, 2013) P-uniform (van Assen, van Aert & Wicherts, 2015)

3 Key messages Lot of evidence of publication bias Traditional meta-analysis methods cannot assess effect sizes accurately when there is publication bias P-uniform currently only accurately assesses effect sizes when there is no heterogeneity PET-PEESE does not work when sample sizes are homogeneous Methods need to be improved

4 Overview presentation Publication bias Meta-analysis PET-PEESE P-uniform Simulation study Results Recommendations

5 Publication bias Impact on meta-analytical results Overestimated effect sizes Inaccurate estimates of heterogeneity (τ2)

6 Meta-analysis Objectives of meta-analysis Estimating True effect 2. Heterogeneity 3. Moderator effects

7 Meta-analysis methods Traditional methods Fixed effects meta-analysis Random effects meta-analysis New methods: PET-PEESE P-uniform

8 Meta-analysis: traditional methods Fixed effects: assumes one underlying effect size Weighted average Weight: precision of study: within-study variance (σi2) Random effects: assumes multiple true effect sizes Weighted average Weight: precision & heterogeneity: between- study variance (σi2 +τ2)

9 Methods dealing with publication bias A lot of methods have been developed. So far, none of them estimates accurately under all circumstances

10 Methods dealing with publication bias New methods: PET-PEESE (Stanley & Doucouliagos, 2013) P-uniform (van Assen, van Aert & Wicherts, 2015) & P-curve (Simonsohn, Nelson & Simmons, 2014 )

11 PET-PEESE When there is no publication bias, there is no relation between effect sizes and precision

12 PET-PEESE PEESE PET No publication bias Plot of 300 randomly generated studies, true ES = 0.5, sample sizes differ from 10 to 150.

13 PET-PEESE When there is no publication bias, there is no relation between effect sizes and precision Publication bias smaller studies report larger effect sizes

14 PET-PEESE No publication bias PEESE PET Standard Error Standard Error Plot of 300 randomly generated studies, true ES = 0.5, sample sizes differ from 10 to 150.

15 PET-PEESE When there is no publication bias, there is no relation between effect sizes and precision Publication bias smaller studies report larger effect sizes PET-PEESE uses the intercept of a regression line to estimate the effect size

16 PET-PEESE No publication bias PEESE PET Plot of 300 randomly generated studies, true ES = 0.5, sample sizes differ from 10 to 150.

17 PET-PEESE: homogeneous sample sizes X-axis: Standard error: related to sample size

18 PET-PEESE: homogeneous sample sizes No publication bias PET PEESE PEESE PET No publication bias Plots of 100 randomly generated studies, true ES= 0.5 with sample sizes differing between 30 and 50.

19 PET-PEESE: homogeneous sample sizes No publication bias PET PEESE PEESE PET No publication bias Plots of 100 randomly generated studies, true ES= 0.5 with sample sizes differing between 30 and 50.

20 P-uniform: Distribution of (conditional) p-values is uniform Uses only statistically significant studies Assumption: All significant studies are equally likely to be published Distribution of (conditional) p-values is uniform under the true effect Currently assumes a fixed effect: Does not provide accurate estimates when there is heterogeneity

21 Simulation study

22 Simulation study Independent variables True effect size (4 levels: Cohen s d of 0,, 0.5, ) Level of publication bias (4 levels: 0%, 75%, 95%, 100%) Sample size (2 levels: homogeneous & heterogeneous, average sample size = 100) Amount of heterogeneity (3 levels: τ2= 0, 1, 9; I2=0%, 50%, 90% when N = 100) Number of studies (1 level: Number of significant studies = 5) Total: 96 conditions, 10,000 iterations

23 Results: expectations Performance of fixed- and random-effects suffer from publication bias Performance of PET-PEESE suffers from homogeneous sample sizes Performance of P-uniform suffers from heterogeneity in effect sizes

24 Simulation study Dependent variables Bias in estimated effect size (Type I error rate) Power Coverage RMSE

25 Results: estimation of effect sizes Fixed & random Effects: overestimates effect sizes when there is publication bias

26 Results: estimation of effect sizes Fixed & random Effects ES d=0.5 d= level of publication bias d= d= ES RE: median estimates of ES FE: median estimates of ES level of publication bias

27 Results: estimation of effect sizes Fixed & random Effects: overestimates effect sizes when there is publication bias PET-PEESE: underestimates effect size when sample sizes are homogeneous

28 Results: estimation of effect sizes PET PEESE median estimates of ES Heterogeneous sample sizes level of publication bias ES d= d= d=0.5 d= ES Homogeneous sample sizes level of publication bias

29 Results: estimation of effect sizes Fixed & random Effects: overestimates effect sizes when there is publication bias PET-PEESE: underestimates effect size when sample sizes are homogeneous P-uniform: overestimates effect sizes when there is heterogeneity

30 Results: estimation of effect sizes P uniform median estimates of ES ES level of publication bias d= d= d=0.5 d= level of publication bias ES d= d= d=0.5 d= ES Large amount of heterogeneity Medium amount of heterogeneity No heterogeneity level of publication bias

31 Results: Power Ideal situation: power is at least.80 Fixed & Random effects: power is almost always close to 0 PET-PEESE: when sample sizes are homogeneous, power is almost always equal to.05 P-uniform: power is too low when the true effect size is small. Power increases when there is heterogeneity: due to overestimation of the effect.

32 Results: Power PET PEESE: Power Heterogeneous sample sizes level of publication bias power d= d= d=0.5 d= power Homogeneous sample sizes level of publication bias

33 Results: Power Ideal situation: power is at least.80 Fixed & Random effects: power is almost always close to 0 PET-PEESE: when sample sizes are homogeneous, power is almost always equal to.05 P-uniform: power is too low when the true effect size is small. Power increases when there is heterogeneity: due to overestimation of the effect.

34 Results: Coverage Ideal situation: 95% coverage FE/RE: coverage decreases when publication bias increases (up to 0% coverage) PET-PEESE: coverage 95% when sample sizes are homogeneous (large CI), lower when sample sizes are heterogeneous P-uniform: coverage = 95% when there is no heterogeneity in effect sizes, decreases when heterogeneity increases.

35 Results: RMSE Root-mean-squared-error: magnitude of error FE/RE: RMSE increases as publication bias increases PET-PEESE: Large RMSE values when sample sizes are homogeneous P-uniform: RMSE is only influenced by δ, not by heterogeneity and publication bias

36 Key messages Lot of evidence of publication bias Traditional meta-analysis methods cannot assess effect sizes accurately when there is publication bias P-uniform currently only accurately assesses effect sizes when there is no heterogeneity PET-PEESE does not work when sample sizes are homogeneous Methods need to be improved

37 Recommendations If you know there is no publication bias: Use fixed/random effects Do not use PET-PEESE if the sample sizes are homogeneous (or when there is a large amount of heterogeneity) Only use P-uniform if you know there is no or a small amount of heterogeneity Pay attention to new methods being published!

38 Questions/Remarks?