22-12-2017 Dear Dr. Villanueva, We would like to thank you for your interest in our paper and the opportunity to resubmit our manuscript Living network meta-analysis for reducing research waste: an empirical study for publication in the British Medical Journal. We are thankful for all reviewers and editors time and effort reviewing our paper. Below you will find our point-by-point responses to the comments. Thank you very much for considering our revised manuscript for publication. Sincerely yours, Adriani Nikolakopoulou, Matthias Egger and Georgia Salanti Institute of Social & Preventive Medicine, University of Bern, Switzerland On behalf of all the authors 1
REFEREE COMMENTS Reviewer: 1 Dear editor. I have carefully reviewed the revised manuscript and the letter outlining responses to peerreview and editorial assessment. I find revisions and comments appropriate, with a particular eye on the elements I reacted to in my peer-review. The limitations are now more clearly outlined and several issues more appropriately discussed. One final suggestion from my side on a minor wording issue: Title now includes timely recommendations, reflecting a role for clinical practice guidelines. I would add "timely guideline recommendations" as this is more precise. I would still support publication of this paper in The BMJ. Best regards, Per Olav Vandvik Please enter your name: Per Olav Vandvik Thank you very much for your positive feedback. We agree that the potential of timely recommendations illustrated in our empirical study primarily applies to guidelines. However, as we think that recommendations is a broader term than guidelines, we would prefer not to change the title unless the editor feels strongly about this suggestion. 2
Reviewer: 2 The authors have adequately addressed the comments made and adjusted the revised manuscript accordingly. Please enter your name: angela wade Thank you for your supportive review of our paper. 3
Reviewer: 3 The authors have taken into account all my comments and modifying accordingly the manuscript. They have completed the Selection of comparison of interest section specifying who were the clinical experts and their tasks, and how the comparison of interest has been chosen. They clarified the term strong evidence replacing it by strong evidence against the null hypothesis, and in the discussion section they acknowledge that strong evidence against the null hypothesis does not necessarily translate into strong recommendations (48). As suggested, they added an appendix (Appendix Figure 3) with the monitoring graphs for all comparisons. Moreover, they also provided additional information about the construction of monitoring boundaries with a new appendix section explaining the methods and link our calculations to a GitHub repository. Please enter your name: CREQUIT Perrine Thank you very much. 4
Reviewer: 4 I commend the authors for the strengths of the revision, they addressed most of the points raised, they have considerably expanded the appendix and provided some R code for the monitoring boundaries. Thank you very much for your positive feedback and the constructive comments throughout the peer review process. Please note we have updated our R library in GitHub to further improve its functionality (see appendix N for details). Considering these strengths, though, I have still found few areas in which I would have appreciated a more balanced presentation of the findings. The paper could be further improved: - The results rely completely on the choice of the true effect in each NMA. The authors used the final NMA estimate. Appendix L discusses this point. But the paper, in particular, the abstract, falls short on highlighting this limitation. First, the findings rely on this choice; if we were to pick a different true effect to implement a prospective living NMA, it is not clear if the conclusion of the abstract would hold. Second, it would make perfect sense to pick a different value for the true effect, because taking the final NMA value is impossible in practice, as the final NMA estimate is not available yet when one starts the living NMA. I invite the authors to highlight this limitation in the abstract and key points. Thank you for this comment. Investigating the impact of taking the final NMA effects in the conclusions in the first revision of the paper, we have performed a sensitivity analysis where the anticipated treatment effect to detect was estimated as the final summary effect from pairwise metaanalysis (Appendix table 3). Indeed, as we discuss in Appendix L, in a real prospectively planned living NMA it is not possible to define the final NMA effect as the anticipated treatment effect to detect. We have added in the abstract that We defined a significance level α=5%, power of 90% (β=10%) and an anticipated treatment effect to detect equal to the final estimate from the network meta-analysis. to clarify the assumptions underlying our method. However, we decided to keep the key points section short and focused on the main message rather than adding methodological considerations. 5
-The authors note in the discussion [we] excluded only five networks with evidence of inconsistency. It would be clearer to mention that one can expect about 1 in 8 networks to show evidence of inconsistency [Veroniki et al Int J Epidemiol 2013], in which case the method would not be applied. We have now added One in eight networks was previously found to show evidence of inconsistency using the design-by-treatment test; this means that our methods would not be applicable in, on average, one in eight networks (44).. - I still recommend changing the label strong evidence for statistical significance. I have noted the authors clarification when they state on page 9 that strong evidence refers to strong evidence against the null hypothesis. But I would rather call like it is throughout the manuscript, in particular the abstract. Thank you for this comment and for noting that we have clarified that with the term strong evidence we are referring to strong evidence against the null hypothesis. We do mention statistical significance in the abstract ( We constructed monitoring boundaries of statistical significance and considered the evidence against the null hypothesis as strong when the monitoring boundaries were crossed. ) and in methods (We considered that a pairwise or network meta-analysis provided strong evidence against the null hypothesis (the hypothesis that there is no difference between the interventions) when the accumulated information crossed the monitoring boundaries of statistical significance, constructed as described here and in (18).). We agree that we could alternatively change the label to statistical significance as we do not take into account the width of (repeated) confidence intervals for constructing Table 3. We do however also show the relative advantage of network compared to pairwise meta-analysis in terms of precision in Appendix Figure 2 and we show how continuous meta-analysis updates can be visualised in a repeated confidence intervals forest plot and interpreted based on the uncertainty surrounding treatment effects in Appendix M. Most importantly, we are concerned that changing the term to statistical significance would potentially confuse readers that we are making inferences based on p-value cutpoint 0.05; which is not true as threshold changes over the accumulation of the evidence. Please enter your name: Ludovic Trinquart 6