Authors response. Editorial. Responses to referees: Impact of BCG, DTP and measles-containing vaccines on childhood mortality: a systematic review

Transcription

1 Responses to referees: Impact of BCG, DTP and measles-containing vaccines on childhood mortality: a systematic review We are very grateful for all of the comments, and we respond to each below. In a version of the paper we have highlighted the main changes in response to these. Authors response Editorial E.1 We thought your study addresses an interesting and potentially important research question. However, after discussion, we felt that it did not add enough to take it further towards publication in The BMJ. We had the following concerns: E.2 We were impressed with your effort to synthesise the evidence on a globally important question. But after all that effort we struggled to discern clear and actionable messages. The results seem mixed (trials do not report significant effects on mortality, and while observational analyses do, they are at high risk of bias). E.3 We were slightly puzzled about the use of all cause mortality. Surely that includes mortality from vaccine specific diseases (eg measles). If there is enough measles about, then reducing measles mortality should reduce all cause mortality. So that outcome doesn't tell us much about non specific effects of vaccines. E.4 We thought your study was a well conducted and comprehensive systematic review and meta-analysis, performed by leading experts in the meta-analysis field. All the analysis methods look spot-on, but our statistical advisor had some suggestions: E.5 1) There are no assessment of small-study effects (potential publication bias). Can the authors do this, or justify why not? E.6 2) The key measure of interest should be the hazard ratio. The authors say they prioritised this, but if they could not get it, they seeks ORs and RRs. I agree this is sensible, but I want to know how different the HR results are from the other results. After all, the HR results account for the whole follow-up No response indicated. Although the available evidence mixed and with limitations, we draw two clear conclusions: (i) there is insufficient evidence to justify a change to the vaccination recommendations at this point in time (contrary to prominent calls in the literature); and (ii) there is sufficient concern about the evidence so-far accrued to justify large independent randomized trials of aspects of the sequencing of vaccines. We have tried to make these conclusions clearer in the revision. We have added more discussion of this issue. While the review is motivated by non-specific effects, we focussed on overall mortality because of substantial limitations of the evidence for non-specific mortality. We collected data on deaths due to diseases other than those the vaccines were designed to prevent. We now comment on several studies of measles vaccines in the Discussion. There were no data for DTP vaccine, about which there is most concern, and only three studies for BCG. We additionally now cite a recent Science paper by Mina and colleagues which provides further evidence that the effects are not due to measles mortality alone. No response indicated. We now explain that the numbers of studies were insufficient to use these methods (and we provide a reference to support this). However, we note in the Discussion that the similarity of point estimates from fixed-effect and random-effects meta-analysis provides some reassurance against the presence of small-study effects. We now illustrate the effect measures in the plots to allow visual examination of differences. In fact we had misleadingly used the terms used by the study authors in some of the supplementary tables. We now consistently use HR for results from Cox regressions, and it can be seen that a large majority of 1

2 period and should have less heterogeneity than the pooling of other estimates, which are specific to particular time-points and do not deal with censoring adequately. Therefore, can we see in the forest plots which estimates relate to HRs please, and give us in a table meta-analysis results for just the HRs please, and discuss this issue in the results? E.7 3) I think the HR issue should have been included in the risk of bias assessment. Was it? That is, if studies did not detail how they accounted for censoring, or did not report HRs, then it should be classed as at least moderate risk of bias (unless follow-up as short and no loss to follow-up)? E.8 4) The authors say in the abstract that there is limited evidence that the vaccines do better than expected. What does the word expected mean here? What RR was expected? What does limited mean? E.9 Please view the comments of the independent reviewers which are included at the end of this . You will see that the reviewers were more positive than we were about the paper s suitability for the BMJ. They did not, however, persuade us that we should accept the paper. Reviewer: 1 the studies used hazard ratios rather than risk ratios or rate ratios (note that rate ratios also account for different times of observation). It can be seen in the plots that there is no particular relationship between effect measure and effect size, and we do not think that presenting post hoc subgroup analyses bases on effect measure is an important addition to the paper. In fact the approach to censoring was not always appropriate in these studies. Some studies censored children on receipt of a subsequent vaccine, and such informative censoring may introduce bias, since subsequent vaccination may be associated with both prior vaccine and mortality. We removed the word limited. The word expected relates to the rest of the sentence (omitted by the editor here). We said more than expected through their effects on the diseases they prevent. We have edited the text to make our meaning clearer. We are hugely grateful to the referees for their thorough examination of our manuscript and many helpful comments and corrections. We have attempted to fix any errors and to address points that were not clear. Some of the reviewers argued that particular studies or results should be excluded from primary analyses. We carefully reviewed these comments against the risk of bias assessments that were conducted for each included result from each study. None of the points raised by the referees addressed issues that had not been covered in the original risk of bias assessment. We are concerned that excluding studies based on the reviewers comments would risk: (1) undermining the consistency of the original risk of bias assessments, and (2) the possibility that studies were excluded based on knowledge of their results as well as their methods. We have therefore not excluded additional studies based on the reviewers suggestions. n/a n/a Frank Shann Professorial Fellow, Royal Children's Hospital, Melbourne Competing interests: None. R1.1 A huge amount of effort by the authors and the members of the WHO Working Group on the Non-Specific Effects (NSE) of Vaccines has gone into this analysis. This paper and the NSE of vaccines are very important indeed, with the potential to make substantial further reductions in child mortality at minimal cost using the current EPI vaccines. The statistical expertise of the analysis is impressive, however, the authors overlook some aspects of the n/a No response indicated. 2

3 pre-specified hypotheses relating to the non-specific (heterologous) effects of vaccines (Fine, TMIH 2007;12:1-4; Shann J Inf Dis 2011;204:182-4) - this is understandable, given that only one member of the WHO Working Group and no member of SAGE has contributed original data testing the NSE of vaccines. I note that the project was funded by WHO, and that the conclusions strongly support the current WHO immunisation schedule. R1.2 The main comparisons are (1) BCG compared to unvaccinated, (2) DTP given after BCG compared to BCG alone, and (3) MCV given after the last dose of DTP compared to DTP. Other important comparisons are (4) DTP with BCG compared to DTP after BCG, (5) DTP with MCV compared to DTP before MCV, and (6) DTP after MCV compared to DTP before MCV. BCG represents BCG ± OPV and DTP represents DTP ± OPV throughout. R The big picture: opposite effects within studies The main analysis in this manuscript assesses the effects of each vaccine (BCG v none, DTP v BCG, and MCV v DTP) separately. The authors find that both observational studies (with a high potential for bias) and clinical trials suggest a beneficial effect of BCG on mortality ; that among the observational studies (with a high potential for bias) the majority suggest a potential deleterious effect of DTP on mortality ; and that four clinical trials pointed towards a beneficial effect of MCV and 18 observational studies (with a high potential for bias) consistently estimated the effect of measlescontaining vaccines on mortality within the first 2-5 years of life to be beneficial. However, it is very important to analyse the effects of DTP on all-cause mortality compared to the effects of both BCG and MCV *within studies* as well as between studies. Comparison within studies greatly reduces the potential for bias. I attach a forest plot of the data from all six observational studies from five different countries that provide estimates for BCG and DTP and MCV (and were classified as not having too high risk of bias). In five of the six studies, the MRR following DTP was higher than 1.00 (it increased mortality relative to BCG), and the MRR for both BCG and MCV was less than 1.00 (BCG reduced mortality relative to no vaccination, and MV reduced mortality relative to DTP); the results of the sixth study (PNG ) also found that mortality after MCV was approximately half the rate after DTP, but it reported a non-significant reduction in mortality when DTP was given after BCG. It is highly unlikely that these findings in the six studies of all three vaccines are caused entirely by bias. For bias to explain the findings of the between-study and within-study comparisons, it is necessary to postulate bias that would cause BCG to approximately *halve* mortality compared to no vaccination, then cause DTP to approximately *double* mortality compared to BCG in the same community, then MCV to approximately *halve* morality compared to DTP in the same community. This is a crucial point that should be strongly 3 No response indicated. We have created the proposed plot and introduced a section on these findings into the Discussion.

4 emphasised in the discussion. In addition to the RCT evidence (with low bias) of the effect of BCG on neonatal mortality in Guinea-Bissau (Guinea-Bissau early and late), these within-study comparisons strongly suggest that, until the next vaccine is given, DTP may be associated with higher mortality than BCG and MCV. I suggest that a version of my funnel plot should be included in the paper to illustrate the importance of examining the *differential* effects of BCG, DTP and MCV within single studies. In addition, the paper should emphasise the fact that bias is unlikely to be the sole explanation for these systematic differences within studies because the effects go in opposite directions from BCG to DTP, and from DTP to MCV. R The analysis of DTP-after-BCG versus BCG-only The WHO schedule is for BCG to be given at birth, and the first dose of DTP (DTP1) at 6 weeks of age; so this is the timing that should be tested. (There is evidence that mortality may be less when DTP1 is given at the same time as BCG: Hirve, Vaccine 2012;30:7300-8, and Aaby, Int J Epi 2004;33:367-73). R1.5 In Burkina Fasso , the mean age for BCG was 4.8 months and DTP1 6.3 months, so many (probably most) infants will have received DTP1 *with* BCG. In addition, there was survival bias in this study. It should be removed from the analysis of DTP. R1.6 The DTP-after-BCG versus BCG-only data for Bangladesh are from an unpublished re-analysis by Aaby and Ravn (ref 31). Dr Aaby kindly provided me with a copy of the re-analysis. It states, If there were problems in registering subsequent vaccinations after the initial registration some children who died after having received the second or third dose may not have had these vaccinations registered and therefore ended up as a death in the first group; in other words mortality would be too high in the first group and too low in the subsequent vaccine dose groups. That something similar might have happened is suggested by an analysis of mortality in relation to 4 We agree in principle. However, in only one of the observational studies of BCG (Guinea-Bissau ) was BCG given within the first week of life for all or most of the children. For DTP, most studies had median vaccination dates at about the scheduled time. The mean vaccination date of 6.3 months in Burkina Faso is difficult to interpret, since the distribution is likely to be skewed. We have elected to keep the study in the analysis. We know that Most of the vaccinated children received either BCG followed by DTP or the vaccines simultaneously, but it is unclear how often each happened. The mean different ages at BCG vaccination (4.8 months) and at DTP vaccination (6.3 months) are inconclusive regarding this issue. A particularly awkward issue is that of survival bias. Having read repeated criticisms of some of the studies as suffering from this important problem, we looked very hard at the reports of these studies to find evidence that survival bias was a problem. We could find nothing specific. This does not mean that we believe it isn t present, just that we are unable to provide any concrete evidence. This puts us in a difficult position. We think it would be inappropriate for us to diagnose survival bias on the basis only of claims from others in the literature, especially given the strong beliefs about effects that are obviously held by some research groups. Problems can be found in all of the observational studies we included. Given the well-known position of the authors of this Bangladesh study report, we are concerned that the authors are here providing a post hoc explanation for findings that do not agree with their previous findings.

5 vaccination status and distance from the hospital (Supplementary table 1); mortality among unvaccinated children did not differ by distance from the hospital but mortality for the BCG-only group was much higher with distance from the hospital and even higher than mortality in the unvaccinated groups. With such potential registration problems comparisons of the subsequent vaccinations might be biased, e. g. DTP1 (after BCG-first) versus BCG-first or DTP2 versus BCG+DTP1. For these reasons, the main comparison in the present analysis is between the three ways of starting the vaccination schedule. Bangladesh should be removed from the analysis of DTP-after-BCG compared to BCG-alone, given that the investigators who calculated the MRR of 0.52 ( ) state that it is a biased estimate. R1.7 For PNG (Int J Epi 2005;34: ), the HR of 0.48 ( ) is used from the 29d-5m section of Table 6, however, we are not told what group is being compared to DTP1 (we know only that all the children in Table 6 received BCG by 6m, but Table 3 shows that BCG was often given with or after DTP1); however, Table 3 says mortality for 29d-5m was 16/ (deaths/person days) with DTP-after-BCG, and 12/ with BCG-only, giving a MRR of 0.67 ( ) (N = 2374). This is the estimate that should be used in Figure 3, as we know that DTP-after-BCG is being compared to BCG-only (and propensity scores are not required, as we are told on p.147 that adjustment made no difference when the analysis was confined to children who had received BCG). R1.8 In the 1970s and 1980s, I was the paediatrician responsible for the Tari region of PNG where this information was collected, and the data are highly suspect: vaccines were given only at clinics, and sick children were not vaccinated (page 147); the 1-5mo unvaccinated children had a mortality rate of 230/1000py (Table 3) compared to an infant mortality rate in Tari of 68/1000 (page 147), so there was very large frailty bias among children who were unvaccinated or vaccinated late (had BCG, but had not yet had DTP). A strong case could be made for excluding this study for too high bias R1.9 The DTP estimate (in Figure 3) for Guinea-Bissau (BMJ 2000;321: ) should be 2.50 ( ) which is adjusted for nutrition, and not 1.74 ( ). R1.10 With these changes, for DTP-after-BCG versus BCG-only, we have Benin, GB , GB (RR 2.50, ), GB , India, Malawi, PNG (0.67, ), and Senegal; I2 is 33.7% with fixed effects MRR 1.85 ( ), and random effects MRR 1.86 ( ). R A surprising lack of concern about the possible adverse effects of DTP The authors are remarkably complacent about the evidence that DTP given on-schedule after BCG may increase child mortality until MCV is given. The 5 The referee is correct that the authors found that adjusting for propensity scores did not have any effect. However, neither of these results adjust for propensity scores, while the one we selected adjusts for baseline covariates. There is a difficult trade-off between the strategy to prefer adjusted HR over unadjusted MRRs, and the strategy to prefer data with some degree of coadministration over data with no co-administration. Our judgement was to favour the former strategy, so we stick with the data we selected. This would appear to be reinforced by the referee s claim that there is very large frailty bias in this study (see point R1.8). We share the referee s concern here, although have not opted to downgrade the study based only on the referee s comment. We recognize the high likelihood of frailty bias in many of these studies, and have added a general comment about this in the summary of risks of bias across the studies. Our risk of bias table for this study states High risk of bias due to confounding (no adjustment for SES or child s health). We extracted the adjusted estimate from a letter submitted by the Guinea- Bissau team to BMJ in 2002, two years after the paper was published. Therefore, we picked the result from most recent paper, as determined by our algorithm. We are unclear why the referee has selected the most extreme result from a multiplicity of analyses presented in the paragraph to which they refer. For reasons outlined above, we do not feel it would be appropriate for us to concur with the referee s preferred selection of results. We agree and have made a number of edits throughout the manuscript to address this.

6 within-study comparisons (above) suggest that bias is unlikely to be the sole explanation for this increase in mortality. In the existing Figure 3, the point estimates for MRR for DTP-after-BCG are fixed effects 1.27 and random effects and the revised estimates are fixed effects MRR 1.85 and random effects Importantly, the MRR is approximately 1.75 in fixed and random models in the within-studies comparison. These are extraordinary figures! If DTP increases all-cause mortality by approximately 27% (let alone 50% or 86%), then it is responsible for approximately 1.3 million excess under-5 deaths per year (there were 6.3 million under-5 deaths in 2013: 6.3/1.27 = 5.0, and = 1.3); this will over-estimate deaths in high-income countries, but most under-5 deaths occur in low-medium income countries. The confidence intervals for these estimates are wide, but the available evidence (especially from the within-study comparisons) strongly suggests that there may well be a very substantial problem. This should result in *urgent* action to obtain reliable evidence that DTP is safe. *Millions* of lives are at stake. It is inappropriate to downplay the importance of this signal in a misguided attempt to protect the immunisation programme. Immunisation is, of course, extremely important - but only because it saves lives. If one component of the programme may have a harmful effect, we need reliable estimates of its extent and how to minimise it while still protecting children against diphtheria, tetanus and pertussis. Any hint of a cover-up will do far more damage to immunisation in the long-term than urgent action now to determine the actual effect of DTP on all-cause mortality and minimise the problem if it is real. This should not be seen as a threat to immunisation, but rather as an exciting opportunity to further reduce child mortality at minimal cost - by exploiting the beneficial NSE of BCG and MCV, and minimising any potential harmful effects of DTP. Throughout this paper the implicit assumption is that strong evidence is needed that DTP is harmful *before* urgent action is taken - the available evidence does not support a change. This complacent attitude to the evidence that DTP may be causing over a million excess deaths a year in high-mortality countries is in striking contrast to the suspension of the administration of RotaShield vaccine in the USA after only 15 cases of intussusception (MMR 1999;48:1007); there is a worrying inconsistency here. The onus of proof is on those of us who advocate administration of DTP to every child in the world to provide evidence it is safe: we need robust evidence of safety, and we do not have that evidence. R1.12 Not enough emphasis is given in this review to the importance of Guinea- Bissau 2004 (Aaby, Int J Epi 2004;33:374-80), which is the *only* study of adequate size that has described the effect on all-cause mortality when DTP was first introduced - the MRR was 1.92 ( ). This suggests that there 6 We have incorporated a comment about this study into the Discussion.

7 was an increase in mortality even in the absence of herd immunity from previous use of DTP. R1.13 Given that the majority of studies indicated a potentially deleterious effect of DTP on mortality (page 7, line 11), it is inappropriate to say that 8 observational studies found a tendency for DTP to have a more beneficial effect in boys than girls. The correct wording is, a tendency for DTP to have a more harmful effect in girls than boys. R DTP with oral polio vaccine (OPV) The statement in the Abstract that there is insufficient evidence about DTP in part because it was almost always administered with OPV should be removed. First, BCG is also usually given with OPV. Second, if DTP with OPV may be harmful (as suggested by the majority of studies ), then the harm is no less important than if it is caused by DTP alone. As is done for BCG and MCV, the Abstract should report the MRR for DTP (from studies with a high risk of bias), but say that this effect was also seen in within-study comparisons with much less risk of bias, and that randomised trials are urgently needed to obtain reliable information about the effects of DTP on allcause mortality. R1.15 The evidence suggests that OPV may have mildly beneficial NSE (Aaby, Vaccine 2004;22: ); Aaby, Vaccine 2005;23: ), so that DTP alone may be even worse than DTP with OPV. Further, IPV may have harmful NSE (Aaby Ped Inf Dis J 2007;26:247-52), so the proposed change from OPV (possible beneficial NSE) to IPV (possible harmful NSE) may substantially increase all-cause mortality in high-mortality countries. R Risk of bias The sources of bias considered in the analysis should be spelt out, and an indication given as to whether each one will give a conservative or exaggerated estimate. Studies with marked survival bias should be excluded. What evidence is there that selection after a vaccine has been given causes bias? It should be emphasised that bias is very unlikely to explain why BCG reduces mortality, then DTP increases mortality, then MCV reduces mortality (especially within individual studies). 7 Agreed. We have corrected the wording. We have removed the link between the comment about OPV and our statement that we are unable to draw strong conclusions: DTP was almost always administered with oral polio vaccine. There is insufficient evidence to draw firm conclusions about the effect of DTP vaccine (with or without oral polio vaccine) on all-cause mortality. A corresponding edit has been made in the Discussion. A fair point. However, we do not feel that it is our place to speculate in this way. We have extended the Discussion to address this important issue in detail. The risk of bias tool used in our study allows reviewers to predict the direction of bias for each bias domain from each study. We attempted to do this, but concluded that most of the time it was not meaningful to do so. In some circumstances, it is possible to predict the direction of bias with some certainty. For example, random (non-differential) error in measuring intervention status is likely to bias the estimated effect of intervention towards the null. And if sick children at high risk of death are less likely to be vaccinated then the benefit of vaccination is likely to be overestimated. For this reason, we commented in the Discussion section of our original manuscript that The benefits of BCG and measles vaccination could have been exaggerated by the baseline confounding. However In most circumstances the direction of bias cannot be predicted with certainty. For example, it is possible that in some settings vaccination could be targeted at deprived social groups, which would lead to underestimation of the benefit of vaccination. We do not claim that selection after a vaccine has been given causes bias ; just that it creates a risk of bias: it is clear that selecting prevalent users of an

8 R1.17 The nomenclature for risk of bias is clumsy (low, moderate, high, too high), why not say low, moderate, high, and very high? R1.18 The results for studies that have very high (too high) risk of bias should not be reported in Figures 2-4, and data from these studies should not be used in Figures for example: for Figure 6, India (BCG), GB (DTP), India (DTP), Philippines (DTP), Ghana (MCV), India (MCV), and Senegal (MCV). R BCG versus unvaccinated The combined results for the only two studies in the analysis with a low risk of bias (GB 2002 early and late) should be presented separately, as RR 0.52 ( ) from Ped Inf Dis J 2012;31:306-8 Table 1 - they provide strong evidence for the use of BCG-Denmark in low-birth-weight babies, which is not current practice (note, however, that different strains of BCG should not be assumed to be equivalent, Comstock, Clin Inf Dis 2000;30[Suppl 3]:S250-3 and Favorov, PLoS ONE 2012;7[3]:e32567). R1.20 Most of the studies of BCG in Figure 2 were not confined to the neonatal period, and this phrase should be removed from page 6 line 50. R Effect of vaccine sequence Figure 5 provides evidence that DTP should not be given with or after MCV; in fact this effect is important in girls but not in boys (Aaby, BMJ Open 2012:2:e000707). This should change policy, and girls should not be given DTP once they have received MCV. It also suggests an urgent need for randomised trials of the effect on all-cause mortality of the booster dose of DTP in the second year of life (because it is given after MCV). R1.22 In Figure 5, the estimate from Senegal (TRSTMH 2015;109:77-84) should be 2.54 ( ) for DTP-with-MCV from the Cox model and not 1.96 ( ), and 2.74 ( ) for DTP-after-MCV from the Cox model and not 2.40 ( ). This will change the risk ratios. R Effect of sex This should be analysed using female/male mortality ratios (which greatly reduce bias) - in addition to comparing vaccinated to unvaccinated children, which introduces a high risk of frailty bias (as shown by the fact that for DTP the three inconsistent studies had frailty or survival bias and showed benefit for girls, while all the other studies showed harm for girls). For female/male MR data, see Aaby, BMJ Open 2012:2:e Figure 6 should present deaths/children as in Figures 2-5 so the nature of the comparisons is clear. intervention can in some circumstances lead to serious bias (Am J Epidemiol 2012; 175: ). We have changed this as suggested. We used the phrase too high risk of bias at the request of the SAGE working group during the project. We are happy to use the suggested terms in the paper, which we also prefer. We do not think it would be appropriate to remove these data from the plot. The distinction between high risk and very high risk of bias is a difficult and subjective judgement, and we wish to be transparent about all the results we collected and the judgements we made about them. This allows the reader to see the findings, in case she or he has different judgements about the risk of bias. We have performed this additional suggested analysis, and report it in the main text. Done. We agree with the need for randomized trials, and we emphasize this at the end of the paper in the revision, along with more specific comments on the questions to be addressed. We have added a note of explanation to the plot. We selected results from the full sample rather than the sample aged 9-24 months. We computed them manually from the MRRs in Table 1 of the paper, using the Greenland & Longnecker approach to obtain the SEs; hence the estimates are also adjusted. We have added numbers of deaths and children to the forest plots of sex interactions. We decided from the outset to restrict investigation of differential vaccine effects by sex (and vitamin A supplementation) to data that allow us to look at interaction. This is the standard approach to looking at these effects in epidemiological research. We do not agree that female/male mortality ratios are appropriate, or that they reduce bias. Female/male mortality ratios will reflect primarily sex differences rather than vaccine effects. R Effect of age at vaccination This is explained in the main results section about risk of bias: our sentence 8

9 In Figure 7, Bangladesh is listed as having too high bias in Figure 2 (BCG effects). Why is it included here? R1.25 Under MCV, I suggest adding Aaby, Ped Inf Dis J 1989;8: (see Table 2). R Short-term follow up The paper refers frequently to short-term follow up and implies that this is a shortcoming of these studies. In fact, it is an inevitable consequence of the hypothesis that the main NSE of a vaccine are seen only until a different vaccine is given (Fine, TMIH 2007;12:1-4; Shann, Clin Ther 2013;35:109-14). The WHO schedule is for BCG at birth; DTP at 6, 10 and 14 weeks (so BCG NSE occur from 1 to 5 weeks of age); MCV at 9 months (so DTP NSE occur months); and DTP booster at 18 months (so MCV NSE occur 9-17 months). This should be explained, and the insinuations removed that that short-term follow up is inappropriate. Note that, in 2013, 4.6 million (73%) of the 6.3 million under-5 deaths occurred in the first year of life, and an even higher proportion occurred in the first 18 months. R The Discussion Page 8, line 52 should say, Receipt of BCG and measles-containing vaccine was associated with a reduced risk of all-cause mortality, but DTP was associated with an increased risk of mortality in 7 studies and a decrease in 1 study. In comparisons within studies, DTP was associated with a higher risk of mortality than BCG and MCV. (The estimate for Bangladesh is regarded as too biased by its originators, and a high proportion of infants in Burkina Fasso received DTP-with-BCG - so these estimates should not be used). R1.28 Page 9, line 56, Evidence concerning modification of the effects of any of the three vaccines on the risk of all-cause mortality by vitamin A status, or age at vaccination was generally insufficient to allow conclusions to be drawn. The data suggested that DTP given with or after MCV or BCG is likely to be associated with a higher risk of mortality in girls. In girls, given the F/M ratios and Aaby, BMJ Open 2012:2:e R1.29 Page 9, line 18, Baseline confounding may have exaggerated the benefits of BCG and measles vaccination and underestimated the harm from DTP. R1.30 Page 9, line 26, While the majority of studies of DPT vaccine suggested that vaccine recipients were at increased risk of all-cause mortality until they 9 now reads We regard the estimates of interaction (e.g. for differences by sex) to be much less affected by bias, because we expect that the biases affecting direct estimates of vaccine effects are likely to be similar across subgroups within a study (e.g. similar in boys and girls). Although the studies were judged to be at very high risk of bias for the effects of vaccines, the same reasoning does not apply for looking at interactions. To judge the study as at very high risk of bias for the interaction, we would need to suspect that the bias is differential between different age groups, and we have no particular reason to believe this this to be the case. This study was excluded as it did not meet our eligibility criteria. It compares seroconversion vs lack of seroconversion after measles vaccination rather than vaccination vs non-vaccination. Thank you for picking up this issue. We did not intend to convey criticism of the studies the main limitation is the number of events (as we had conveyed for the results for measles: Due to the short follow-up, numbers of deaths were low and the findings inconclusive ). We have inserted a qualification so that it now reads Follow-up periods were often of necessity short. Changed as suggested. See response to R1.23. Done. We do agree that the evidence available strongly motivates further investigations.

10 received MCV, we have little confidence in attributing the observed higher mortality to DPT vaccine. However, within-study analysis also suggested that DTP may increase all-cause mortality, and analysis of vaccine sequences suggested that mortality may be increased when DTP is given after BCG or MCV. Please justify why do you have little confidence. Bias cannot explain why BCG reduces, then DTP increases, and then MCV reduces mortality between and (especially) within studies, and the sequence of vaccination analysis (Figure 5) also suggests increased mortality when DTP is the most recent vaccine. Confidence is at least moderate, and the findings should lead to urgent investigation. R1.31 Page 9, line 34. Follow-up periods were often short, particularly in observational studies, mostly to less than 12 months of age for BCG and to less than 9 months of age for DTP - this should be removed, as explained above. We do not fully agree that bias cannot explain the findings. There are potential biases that are too difficult to assess from written reports and which cannot be easily argued in a scientific paper without evidence (specifically those associated with selective reporting and (non)reporting). See R1.26. R1.32 [Page 9,] Line 36, A substantial proportion of the studies were conducted in a single country, and it is unclear whether the results are generalizable to infants and children in other settings - this should be removed, see Sankoh, Int J Epi 2014;43:645-53, and note that the six important within-study comparisons come from five different countries. R1.33 Page 9, line 38, The studies typically took place in communities with many years of use of these vaccines. As a result, a combination of direct vaccine effects and herd immunity means that these communities had low incidences of the diseases targeted by the vaccines, so that the substantial net benefits of routine use of these vaccines may not have been apparent - this is pure speculation, and is contradicted by the evidence in GB (Aaby, Int J Epi 2004;33:374-80) where introduction of DTP was associated with a MR of 1.92 ( ). The entire paragraph should be deleted. R1.34 Page 9, line 48, provide some support for the hypothesis. Why not substantial support? Two RCTs for BCG in neonates, and the majority of observational studies showing benefit from BCG and MCV but harm from DTP (separately and within studies, and with interactions) - exactly as described in BMJ 2000;321: R1.35 Page 10, line 5, We do not believe that the available evidence supports a change in either the choice of vaccines or the timing or sequence of immunizations routinely administered to infants and children. We believe calls to alter the immunization schedule so as to maximize any possible beneficial effects of BCG and measles vaccine or to minimize any possible deleterious effects of DTP vaccine on all-cause mortality are premature. The results of this analysis (especially after the corrections) suggest that BCG should be given to all babies at birth, rather than being withheld from all low-birth-weight babies and only given to term babies when there are enough babies present 10 Removed as suggested. We have revised our phrasing to avoid implying that all studies were conducted in communities with many years of vaccine use, and removed the speculation implied by the term the substantial net benefits. We have removed the word some so as to be more positive about the findings We have added a remark about the evidence for low birth weight infants to the Discussion. We also agree with the need for further randomized trials, which need to be adequately powered to address the concerns raised about sex differences. We do not feel the evidence is strong enough to introduce differential strategies for girls and boys, which would be challenging to implement. While we do not disagree with the other statements of the referee, we do not feel that the evidence is strong enough to include such calls in our paper.

11 (policy should be changed to suggest opening a vial even if only one baby needs BCG). The policy that MCV not be given after 12 months should be changed (Fisker, Lancet Glob Health 2014;2:478-87), and the main indicator for EPI should be the proportion of children given MCV before 12 months of age (rather than the proportion given DTP3). The significant interactions between DTP and MCV suggest that it should be policy that DTP be given before MV, and not with or after MCV in girls. In addition, the analysis suggests an *urgent* need to test the effects of DTP on all-cause mortality, perhaps beginning with randomised trials of the booster dose of DTP (which is given in some countries but not in others). R Minor issues Done. Please could the reference numbers be included in Figures 2-7? It is a laborious process to identify a study in a Figure, then look in the appropriate section of Appendix 2 to find the reference number for that study, then look in the References of included articles section at the end of Appendix 2 to find the citation. This is made even more difficult by the fact that some studies are incorrectly named: R1.37 Guinea-Bissau should be Guinea-Bissau Fixed - thanks. R1.38 Guinea-Bissau should be Guinea-Bissau This is a label we give to a larger birth cohort, including the paper by Roth et al (Paed Infec Dis 2004; 23: ) of children born in the period (we use data from this paper in the BCG analysis). We have renamed the label to to correctly reflect two papers (Int J Epidemiol 2003; 32: and Vaccine 2004; 22: ). R1.39 Ghana should be Ghana Our label is based on the information that At enrollment, children aged 6 90 mo were randomly assigned to receive VAS or placebo as stated in one of the trial reports. This implies that children could have been born up to 5 years before the start date of the trial in R1.40 Reference 12 is Trop Med Hyg Int Health 2013;18: Fixed. Reference 15 is Trans R Soc Trop Med Hyg 2015;109: R1.41 Sex and gender are used interchangeably throughout the manuscript. It is Done. normal practice to use sex to refer to biological effects (as in this paper), and use gender in cultural or social contexts (Oxford English dictionary). I suggest using sex throughout. R Changes to data See R1.18. Delete all analyses of studies with too high bias in Figures 2-7. R1.43 Present the combined BCG estimate for the low bias RCTs GB : We now present this result in the main text. MRR 0.52 ( ). R1.44 In Figure 3, delete Burkina Fasso (many children got DTP with See R1.5. BCG). R1.45 In Figure 3, delete Bangladesh (too high bias in reanalysis). See R1.6. R1.46 In Figure 3, PNG should be 0.67 ( ), N=2374 (DTP-after- BCG vs BCG only). See R

12 R1.47 In Figure 3, GB should be 2.50 ( ) which is adjusted for See R1.9. nutrition. R1.48 In Figure 5, Senegal should be 2.54 ( ) for DTP with MCV (Cox See R1.22. model). In Figure 5, Senegal should be 2.74 ( ) for DTP after MCV (Cox model). R1.49 For sex, add female/male mortality ratio analysis. See R1.23. R Abstract The Results and Conclusions need to be revised. The abstract and conclusions have been extensively revised. Results We identified results from 32 birth cohorts. Most evidence was from observational studies assessed as at high risk of bias, with some from clinical trials. Most studies reported on all-cause (rather than non-specific) mortality. BCG vaccine appeared to reduce all-cause mortality: the average relative risks (RR) were 0.70; 95% confidence interval (CI) 0.49 to 1.01 from five clinical trials and 0.47; 95% CI 0.32 to 0.69 from nine observational studies at high risk of bias. DTP appeared to increase all-cause mortality (RR 1.86; 95% CI 1.31 to 2.64) from eight observational studies at high risk of bias; this effect appeared stronger in girls than boys. MCV appeared to reduce allcause mortality (RR 0.74; 95% CI 0.51 to 1.07 from four clinical trials, RR 0.51; 95% CI 0.41 to 0.63 from 18 observational studies at high risk of bias): this effect appeared stronger in girls than boys. Conclusions and implications of key findings Bias is unlikely to be the sole explanation for the observed reduction in mortality after BCG, followed by an increase in mortality after DTP, then a reduction in mortality after MCV. There is limited evidence to suggest that BCG and MCV reduce overall mortality by more than expected through their effects on the diseases they prevent, and that DTP may increase all-cause mortality when there is herd immunity to diphtheria, tetanus and pertussis. The available evidence suggests that BCG should be given to as many babies as possible soon after birth, including low-birth-weight babies; that DTP should not be given to girls with or after MCV; and that children should be given MCV even if it has not been given by 12 months of age. Further research is urgently required on the possible non-specific effects of immunizations on the immune system and on mortality in low income settings. R What this study adds Through a comprehensive systematic review and risk of bias assessment, we found few randomized trials, and that many types of bias may have influenced the results of the numerous observational studies. 12 We have followed the suggestion for point 2, and added a third point corresponding to our conclusions.

13 We found limited evidence to suggest that BCG and measles-containing vaccines reduce overall mortality by more than expected through their effects on the diseases they prevent, and limited evidence that DTP may increase all-cause mortality when there is herd immunity to diphtheria, tetanus and pertussis. Bias is unlikely to be the sole explanation for a reduction in mortality after BCG, followed by an increase in mortality after DTP, then an reduction in mortality after MCV. Reviewer: 2 n/a Ane Fisker, Post Doc, Research Centre for Vitamins and Vaccines, Bandim Health Project, Statens Serum Institut Competing interests: I work at the Bandim Health Project, in the research field of non-specific effects. In Denmark Bandim Health Project is based at the Statens Serum Institut (SSI) which is a vaccine producer- However, neither Bandim Health Project nor any of the research conducted by BHP employees is funded by the SSI, but rely on external grants. R2.1 Originality and Importance of work to general readers The review summarises a huge amount of data from a geographically wide area and over a span of many years and assesses the effects of interventions widely used and with potential public health implications. Given its potential impact, it should matter to the general reader. The review will provide important information for identifying the future research questions, but I think the review could do so much more strongly than it does now. R2.2 Research Question - clearly defined and appropriately answered? I thought the research question is clearly defined (to evaluate the non-specific and all-cause effects of BCG, DTP and MV) but the interpretation that the effect of DTP could not be assessed because it is almost always coadministered with OPV, seems to indicate that it should have been clearer. The observations made by our group and the hypotheses formulated in 2007 (Fine PE. 'Non-specific effects of vaccines'--an important analytical insight, and call for a workshop. Trop Med Int Health 2007; 12(1): 1-4.) are based on DTP +/- OPV. The data points available and the question which can be assessed is DTP+/-OPV, and that should be assessed. R2.3 The secondary questions: to examine any modifying effects of sex, age at vaccination, sequence in which vaccines are given, and prior or coadministration of vitamin A receive little attention in the presentation and I have struggled to assess whether these questions are appropriately n/a No response indicated. We agree that the available data should be interpreted as the effects of DTP+/-OPV. Indeed we did not penalize the DTP studies in our risk of bias assessment when there was co-intervention of OPV with DTP. We have edited the Abstract and Discussion section to make it clearer that we are unable to separate the effects of DTP and OPV rather than implying that it is a major limitation of the evidence. See R

14 answered. For the sex-differential effects I do not understand the rationale for looking at merely at vaccine estimates in girls versus the same estimates in boys, thereby excluding studies which have reported the female-male mortality ratio among children by the most recent vaccine. R2.4 Scientific reliability Design and Participants studied The organisation of the data into birth cohorts to have independent observations has clearly been a difficult exercise. I have below pointed out a couple of places where I believe that papers have been grouped wrongly together. R2.5 Methods - adequately described? The algorithm for selecting the estimates is described and the estimates presented for the three vaccines but not for interactions and sequence of vaccines analysis. R2.6 Results - answer the research question? Clear presentation of the main results in the text, but not in the abstract (see below). R2.7 The results for the analysis of effect modification (details below) could be clearer (see specific comments below). R2.8 For the specific data points extracted, the direction of bias is not presented/discussed, but would mostly serve to dismiss a result. However, there are many examples where the bias would lead to presenting more conservative estimates, and in these instances I think it should be emphasised that the bias does not explain away a finding. For example, on page 65, it is reported that there is high risk of bias due to retrospective collection of vaccination data. I presume that this means that a child surviving is more likely to have a vaccine registered and therefore that any vaccine would be associated with a beneficial effect. R2.9 Interpretation and conclusions A preliminary report which has summarised the review has been available online since April When comparing the results from the review previously made available, there is a general down toning of the results. The previously published review it was presented that there was consistent evidence of beneficial effects of BCG and MV: The results indicated a beneficial effect of BCG on overall mortality in the first 6-12 months of life and There was consistent evidence of a beneficial effect of measles vaccine, although all observational studies were assessed as being at risk of bias and the GRADE rating was of low confidence. In the previous version the contrast to the description of the DTP results: The findings were inconsistent, with a majority of the studies indicating a detrimental effect of DTP, and two studies indicating a beneficial effect. With the present wording the contrasting results between the live BCG and MV and the inactivated DTP vaccine does not convey this results. There is insufficient evidence to draw 14 Thank you for these valuable insights. We have added a comment that the same algorithm was used, and have additionally added information about the source of each data point below the risk of bias table for each corresponding study. The abstract has been extensively revised. See specific comments below. See R1.16. We have changed the tone of our results to better convey the findings of the DTP studies. We also introduced a new plot and discussion of the withinstudy differences in BCG/MCV vs DTP findings.

15 firm conclusions about the effect of DTP vaccine on all-cause mortality, in part because it was almost always administered with oral polio vaccine. The abstract therefore does not convey the picture of the opposite effects of the vaccines which is shown in the results section: BCG: The clinical trial results, including two from low birth weight infants and two in native American children in the 1930s and 40s, suggested a beneficial effect of BCG on mortality (average RR 0.70; 95% CI 0.49 to 1.01). The average for the nine observational studies (follow-up mostly within the first year of life) was RR 0.47; 95% CI 0.32 to 0.69; inconsistency (I2) = 63%, although all these studies were considered at high risk of bias. DTP: The majority of studies indicated a potential deleterious effect of DTP on mortality, and three individual results had 95% confidence intervals that excluded no effect (one beneficial, two harmful). The average effect was RR 1.38 (95% CI 0.92 to 2.08). MV: Directions of effect in these trials, as well as a fourth clinical trial in Nigeria, pointed towards a beneficial effect of measles containing vaccine (RR 0.74; 95% CI 0.51 to 1.07; I2 = 0%). The 18 observational studies that were not excluded consistently estimated the effect of measles-containing vaccine on mortality within the first 2 5 years of life to be beneficial, with average halving of mortality risk (RR 0.51; 95% CI 0.41 to 0.63; I2 = 62%). In the present paper, that the majority of the estimates are studies classifying to have a high risk of bias, cannot explain such opposing effects of the different antigens in studies using the same methodology to estimate the effect of BCG, DTP and MV and for many of the studies within the same cohorts. This should be discussed, and reported in the abstract and in the what this paper adds. R2.10 It is concluded that further research is warranted, but only randomised trials: we favour limiting future studies to those randomized trials that can be ethically and practically conducted. These views were supported by SAGE recommendations in April 2014 (13). To me it is not clear which document is referred to in ref 13 in oundpaper_final.pdf?ua=1 R2.11 Under research agenda observational studies are still in play: There was a view that further observational studies with inherent and substantial risk of bias would be unlikely to provide conclusive evidence about putative non-specific mortality effects. However, if observational studies are to be contemplated, their design and analysis should mimic what would be undertaken if it were to be a randomised controlled trial. Future studies should draw upon a broad investigator pool and from a wide range of geographic locations and burden of disease settings. The development of standardized protocols for both RCTs and observational studies of mortality 15 Fixed (we now name the specific document) We would like to see randomized trials because the results of the observational studies are so difficult to interpret. We continue to believe that further observational studies such as those we reviewed are unlikely to resolve the debate. We have deleted the unnecessary phrase about trials being ethically and practically conducted.