Author's response to reviews Title: A note on the graphical presentation of prediction intervals in random effects meta-analysis Authors: Charlotte Guddat (charlotte.guddat@iqwig.de) Ulrich Grouven (ulrich.grouven@iqwig.de) Ralf Bender (ralf.bender@iqwig.de) Guido Skipka (guido.skipka@iqwig.de) Version: 3 Date: 13 June 2012 Author's response to reviews: see over
IQWiG Im Mediapark 8 50670 Cologne Germany Systematic Reviews c/o BioMed Central 236 Gray's Inn Road London WC1X 8HB United Kingdom Director Jürgen Windeler, MD, PhD Address Im Mediapark 8 50670 Cologne, Germany Your contact Dr. Charlotte Guddat Medical Biometry Tel +49-221/3 56 85-461 Fax +49-221/3 56 85-10 Charlotte.Guddat@iqwig.de www.iqwig.de 13 June 2012 MS ID: 185516546070351 Revised Manuscript Dear Sir or Madam, Please find attached the revised version of our manuscript A note on the graphical presentation of prediction intervals in random effects meta-analysis. We would like to thank the two referees, Dan Jackson and Hannah Rothstein, for their helpful comments and suggestions. All changes in the new revised version are highlighted in the compare documents function. Below please find our point-by-point response to the referees. We hope that the revised version is accepted for publication in Systematic Reviews. With kind regards, Charlotte Guddat Page 1 of 4
-by-point response Referee 1 1. We are told that the random effects model is applied when the effects differ more than would be expected from random error alone (page 7). The authors mean from withinstudy sampling variation alone. However the model can be applied regardless -- the estimate of the between-study variance may be 0, so that the random effects analysis agrees numerically with a fixed effects analysis, but this is not a problem! 2. We are told that adding prediction intervals is one approach (page 4). What other approaches are there? CI for tau I presume? 3. We are told to assume normality for the random effects at the bottom of page 4. There are some papers that discuss other random effects distributions that might be worth mentioning as this assumption is questionable. 4. Page 5, Higgins et al "suggested" and did not "derive" the prediction interval. The prediction interval given is somewhat ad hoc and is not very well justified though it does have intuitive appeal. Applying a t- distribution "reflects" and does not "account for" uncertainty in tau in the random effects model for meta-analysis (it does in the IID normally distributed data textbook set up but things are a little more complicated here). Certainly, the random effects model can be applied when there is no between-study variation. We have modified the corresponding paragraph on page 3 / 4. Generally, there are various measures to quantify the heterogeneity between studies. In the revised version, we have acknowledged and as metrics of heterogeneity on page 5. In addition to presenting (one of) these numerical values, as a graphical presentation we suggest adding the predication interval to forest plots. Though the assumption of normally distributed effects is conventional, it is surely true that this may not always be justified. Hence, we have addressed this point on page 6. We agree with the linguistic corrections and have exchanged the terms on pages 5 and 6. Seite 2 von 4
-by-point response Referee 1 5. Crucial is that observed effect that the prediction interval is for is the true underlying effect for a new study, i.e. within-study sampling variation in the new study is not allowed for and this should be mentioned. One can think of an infinitely large new study to avoid within-study sampling variation for example, but this subtle point should be mentioned. When the authors say the "observed effect" on page 7 they may be revealing that they do not appreciate this point -- the true effect for the new study is an unobservable quantity (unless the new study is infinitely large), which makes it clear that the prediction interval is a rather abstract thing! 6. The simulation study in reference [12] is somewhat super-seeded by Biggerstaff BJ, Jackson D. The exact distribution of Cochran's heterogeneity statistic in one-way random effects metaanalysis Statistics in Medicine. 2008; 27:6093-6110. Certainly, the prediction interval gives a region for the true effect. We have corrected this point where necessary. We have added Biggerstaff and Jackson (2008) as a reference in addition to Mittlböck and Heinzl (2006). General comments: The manuscript has been edited by a native speaker. All changes to the previous version are indicated in the Compare documents function. Seite 3 von 4
-by-point response Referee 2 1. I do, however, believe that they are in error when they state that FE and RE metaanalyses are treated the same way in practice. I don't believe that this is generally the case. 2.The authors need to acknowledge that other metrics of heterogeneity, including I squared, Q, tau and tau squared exist, and are often used in meta-analyses. The authors need to describe the unique information provided by the predictive interval. 3. The authors need to explain why they use a rectangle to depict the predictive interval. My understanding of the predictive interval is that it generally describes a normal distribution of effects around the mean effect, when the standard RE model is used. Borenstein et al. (2009) use a normal distribution superimposed over the mean overall effect and 95% CI to depict the width of the predictive interval. The authors do not acknowledge this; perhaps they are unfamiliar with it. 4. Long before Higgins described the predictive interval, Hunter and Schmidt (2004) describe a very similar entity which they called a credibility value. This work deserves acknowledgement as the first work to call attention to the problem of quantifying the range of (true) effects. We have revised the statement that FE and RE meta-analyses are treated in the same way. We meant that the graphical representation of the results is the same and have revised the corresponding paragraph. We have acknowledged, and as metrics of heterogeneity on page 5. We have extended the description of the information provided by the prediction interval on page 4. We have explained why a rectangle may be chosen to depict the prediction interval on page 8. We have acknowledged the presentation by Borenstein et al. (2009) (see pages 4 and 7) We have acknowledged the work by Hunter and Schmidt (2004) on page 8 / 9. General comments: The manuscript has been edited by a native speaker. All changes to the previous version are indicated in the Compare documents function. Seite 4 von 4