Title:Bounding the Per-Protocol Effect in Randomized Trials: An Application to Colorectal Cancer Screening

Transcription

1 Author's response to reviews Title:Bounding the Per-Protocol Effect in Randomized Trials: An Application to Colorectal Cancer Screening Authors: Sonja A Swanson (sswanson@hsph.harvard.edu) Oyvind Holme (oyvind.holme@medisin.uio.no) Magnus Loberg (magnus.loberg@medisin.uio.no) Mette Kalager (mkalager@hsph.harvard.edu) Michael Bretthauer (michael.bretthauer@medisin.uio.no) Geir Hoff (Geir.Hoff@kreftregisteret.no) Eline Aas (eline.aas@medisin.uio.no) Miguel A Hernan (mhernan@hsph.harvard.edu) Version:3Date:19 October 2015 Author's response to reviews: see over

2 October 19, 2015 Trials Dear Editors: Please find enclosed a resubmission of our manuscript entitled Bounding the Per-Protocol Effect in Randomized Trials: An Application to Colorectal Cancer Screening (manuscript ID: ). We are grateful for the constructive feedback from the editors and three reviewers. Below we detail our point-by-point responses to the specific comments. Reviewer 1 1. In the section titled Bounding the per-protocol effect within compliance types, you are performing a principal stratification based on treatment non-compliance. There are many papers exploring this type of principal stratification and, as summarized and explained by VanderWeele (2011), non-compliance is one of the causal issues principal stratification handles quite well. While I feel the authors did an excellent job in this section, the fact that they did not acknowledge this as a principal stratification, nor cite any of the works within that literature, should be changed. Likewise, the passage in the discussion indicating these types of bounds are of questionable interest could be softened. We thank the reviewer for bringing up these important points. In the Results section, we now connect the terminology between principal strata and compliance types. We have also broadened our Discussion section to cite the important work in this area and the merits and limitations of this approach in the context of trials (page 15). In that same paragraph, we also expand upon the issues with solely using the complier subgroup effect to inform clinical or policy decision-making. 2. Page 8 Line 4: It should be clarified that Pr[X=1 Z=0]=0, as previously stated in the text. This has now been clarified. 3. Table 3: The use of the word block in the text is useful but Table 3 itself should be divided into these blocks to help the reader quickly find each block. The tables have now been reformatted to emphasizing the blocking. 4. Pages 8 and 9: The formulas for LB and UB are somewhat confusing, perhaps separating the different values with commas. We have added commas at the end of each line to signal that each of these expressions are a maximum/minimum of eight separate values. The presentation is similar to that presented in Judea Pearl s Causality textbook, although Dr. Pearl introduces a slightly more compact (and denser) notation for probability statements. 5. Page 16: While the authors acknowledge the focus on identification, there are some simple formulas for confidence intervals for sensitivity bounds that might strengthen this paper, see Vansteelandt, Goetghebeur, Kenward, Molenberghs (2006) and related texts.

3 We have expanded our discussion section to describe some of the methods proposed and on-going areas of statistical research for bounds. On page 17 we now write: Finally, our example and discussion has focused on identification, but there is a growing body of literature on how to incorporate random variability [21]. Specifically, there has been recent development in methods for estimating confidence intervals around the bounds [22-26] as well as estimating confidence intervals for the partially identified treatment effect itself [27, 28]. Incorporating random variability into the presentation of partial identification results in randomized trials is critical, however more research is needed as there is currently no consensus in the statistical literature on or readily available software for the optimal, valid approach. Reviewer 2 1. As the authors correctly acknowledge, the bounds need to be supplemented to account for sampling uncertainty. Furthermore, since there is more than one outcome of interest, it seems that the bounds should also be supplemented to account for multiple comparisons. Please extend on these important aspects. We thank the reviewer for bringing up these interesting points. Regarding sampling uncertainty, please see our response to Reviewer 1 Comment 5 above. Regarding multiple comparisons, we note that the outcomes used in the current set of analyses were those pre-specified in the study protocol for this trial. We clarify this point on page 5. As such, we do not adjust for multiple comparisons. 2. Please provide further guidance on how practitioner should weigh the different assumptions considered. In the discussion section, we now have an expanded description of how and when the assumptions across different trial designs may be reasonable. Weighing the reasonableness of each condition of course depends on both the study design and subject matter expertise on the particular treatment and outcome of interest. We have also made some changes to the final column of Table 2 to clarify these points. Investigators considering employing these methods in randomized trials with point interventions and dichotomous outcomes should consider how features of their particular study design may affect which sets of assumptions we describe in Table 2 are reasonable. The instrumental conditions are expected to hold in placebo-controlled, double-blinded randomized trials of point interventions where there is no loss to follow-up, no placebo effect, and double-blinding is successfully maintained, but the instrumental conditions are suspect in head-to-head randomized trials and whenever double-blinding is not successfully maintained or there is a possible placebo effect. The homogeneity conditions, on the other hand, are not expected to hold based on any study design feature and thus should be weighed judiciously when applied to the analysis of any randomized trial. A similar caveat applies to conditions about the distribution of or effects within compliance types when there is non-compliance in both treatment arms [5]. 3. Please explore whether the bounds vary by gender. We include these gender-stratified results in Supplemental Table 4.

4 4. Please clarify how are the instrument conditions different to the classical instrumental variables conditions established in Angrist, Imbens and Rubin (1996). The primary difference between the terminology used by Angrist, Imbens, and Rubin in 1996 versus the terminology used the current manuscript (and in Balke and Pearl 1997, Hernan and Robins 2006, Pearl 2009, Richardson and Robins 2010, Swanson and Hernan 2013, Imbens 2014, and elsewhere) is whether monotonicity (no defiers) is an instrumental condition. We introduce assumptions about compliance type distributions separately because we are not only targeting the effect in the compliers. In the partial identification literature, there is also some differences in whether marginal or joint exchangeability is assumed. Because this is not relevant to the discussion of trials, we mention this only briefly in our text (page 9). 5. Page 5: there should be a space between outcomes and [9]. 6. Page 8: in *the* year age group. 7. Page 9: there should be a space before [9] and only one space after it. 8. Consider citing Manski s work on partial identification earlier in the manuscript. We now reference Manski in the second paragraph of the Background (page 4) and again in the Results on page 9. Reviewer 3 1. Page 4: Is the per protocol effect - the effect that would have been observed had all trial participants followed the trial protocol a standard term? I think it may raise confusion vs. the standard (biased) per protocol analysis. It may be good to give a bit more on this effect vs. the standard use of the term, and the ITT effect. As the reviewer raises here, we are aware that there is often confusion between terminology in effects of interest versus the terminology for specific analytic strategies (presumably targeting those effects). Our paper explicitly defines the per-protocol effect (the estimand) and differentiates it from the naïve perprotocol analysis in the first paragraph of the introduction. 2. Page 4: An alternative to estimating the per-protocol effect estimate There is one too many estimate in that sentence. 3. Page 6: The main concern I have with the paper is in presentation of the assumptions. The list of assumptions is long, and in the table it is hard to follow. It is subsequently hard to follow in the figures;

5 one has to cross-check with Table 2 regularly in order to follow the figures. In particular, the instrumental conditions should be described in the text. We have made a number of changes in the text, tables, and figure descriptions to clarify the presentation of assumptions. In particular: Table 2 now more clearly states the first three conditions are the instrumental conditions (previously this appeared in a footnote); the description in Figures 2 and 3 now reminds readers what the three instrumental conditions are (relevance, exclusion restriction, and exchangeability); and the text was revisited to clarify the three conditions that comprise the instrumental conditions (pages 7-8). 4. Page 10/11: give some context for the per-protocol effect in never-takers. This effect is, essentially, what we d observe if we could force them to follow the protocol. Some readers may have trouble making sense of this parameter, as the standard is to think of there being no effect in the never-takers. We now state: For the never-takers, we can obtain a point estimate for the risks under no screening, but we have (by definition) no information on what would have happened to them had we forced them to follow the protocol in the screening arm and therefore been screened. Therefore, we can only achieve wide bounds for the per-protocol effect in the never-takers because we have limited information in the data on what would have happened to this subgroup had they followed the protocol. 5. Page 8/9: Is there any intuition that might help describe the upper and lower bound for the per protocol risk difference? It s a bit hard to follow. Unfortunately, the intuition for the bounds is limited. In our special setting with one-sided noncompliance, there is some intuition as to the point identification of the RD for compliers and the bounds for the RD for never-takers, and this expression would reduce to a weighted average of these RD. In more complicated settings, the work by Richardson and Robins (2010) demonstrating the geometry of the bounds provides, to our knowledge and in our opinion, the most intuitive presentation of where these expressions come from. However, their work is also a dense read and requires a comfort with mathematics beyond that of our intended audience at Trials. In the text, we point readers to the Richardson and Robins reference for further discussion, including some intuition for these complicated expressions (page 9), and explain that the bounds for the full study population are a weighted average of the bounds for the compliers and never-takers in our special case (page 11). 6. Table 3: consider a text separator indicating the fact that you can point identify the bottom three sets of parameters. We have added a footnote clarifying the point identification results. Editorial requests 1. Please include a Discussion section after the Results. The Results and discussion may be combined into a single section or presented separately. Results of statistical analysis should include, where appropriate, relative and absolute risks or risk reductions, and confidence intervals. The Results and discussion sections may also be broken into subsections with short, informative headings.

6 Our manuscript is formatted with a Results and Discussion section with subsections in the Results. We include both relative and absolute measures of risk. Please see our response to Reviewer 1 Comment 5 for why we feel confidence intervals are not appropriate at this time. 2. Please mention each author individually in your Authors Contributions section. We suggest the following kind of format (please use initials to refer to each author's contribution): AB carried out the molecular genetic studies, participated in the sequence alignment and drafted the manuscript. JY carried out the immunoassays. MT participated in the sequence alignment. ES participated in the design of the study and performed the statistical analysis. FG conceived of the study, and participated in its design and coordination and helped to draft the manuscript. All authors read and approved the final manuscript. This formatting has now been adopted. Thank you for your consideration of our work. Please be in touch if any questions arise. Sincerely, Sonja A. Swanson Department of Epidemiology Erasmus MC Rotterdam, the Netherlands s.swanson@erasmusmc.nl on behalf of co-authors