Observed survival (%) time women men

Transcription

1 Manuscript ID BMJ entitled "Development and validation of risk prediction equations to estimate survival in patients with colorectal cancer: cohort study." Thank you for sending us your paper. We sent it for external peer review and discussed it at our Editors meeting. We recognise its potential importance and relevance to general medical readers, but I am afraid that we have not yet been able to reach a final decision on it because several important aspects still need clarifying. The key issue for us is the clinical utility of this tool, especially as used by GP's. We would like you to revise the paper and include in your analysis the end-point of colorectal cancer mortality. You will see that some of the reviewers have highlighted this in their comments below. Please also respond to another issue that was raised in relation to whether treatment for colorectal cancer can be incorporated. Authors response: thank you for giving us chance to revise the paper. In summary, - we have changed the main model from a Cox model for all-cause mortality to a competing risks model which also accounts for clustering by practice. This has enabled us to include colorectal cancer mortality as an additional end-point - we have incorporated chemotherapy in the model. - We have presented additional analyses (tables 1 & 2) and justification for our decision to present separate models for men and women - We have tested for and included more interaction terms in the model - including the interactions between chemotherapy and stage - We have amended the calibration graphs as suggested by the reviewers - We have updated the literature review to refer to additional studies - We have addressed all the other issues raised by the reviewers as we describe in detail below. In your response please provide point by point replies to all the comments made by The BMJ editors and the external peer reviewers, explaining how you have dealt with them in the article. Editorial committee John Fletcher (chair), Gary Collins (stats), Elizabeth Loder, Jose Merino, Kristina Fister, Georg Roeggla, Tiago Villanueva, Daoxin Yin, Rubin Minhas A-Use of 2 different models in men/women is questionable. - Authors response: In our protocol, we decided to present separate models for men and women since we considered it likely there would be systematic differences between the sexes for baseline characteristics and also the extent to which factors are likely to affect allcause mortality. This was supported by the analyses and we have updated tables 1 & 2 and the text in the results (section 3.2) to highlight this. For example, table 1 shows differences in tumour location % of women in the QResearch derivation cohort had colon cancer compared with 59.7% of men. Use of chemotherapy tended to be higher in men than women (33.0% vs 27.9%) and also radiotherapy (16.8% vs 11.8%). Table 2 shows that men

2 are more likely to be ex-smokers than women (40.1% vs 24.9%). Men are also more likely than women to have cardiovascular disease (23.3% vs 15.5%) and type 2 diabetes (14.8% vs 11.2%). Men are more likely to be prescribed statins (29.7% vs 21.6%) and aspirin (24.5% vs 17.0%). Women were more likely than men to be anaemic at diagnosis (28.3% vs 20.3%) and have raised platelets (11.8% vs 6.6%). Observed survival rates also differ between men and women as shown in the table below with men having a higher 1-year survival but a lower 10-year survival. The revised models (table 4 and 5) already have quite complex two-way interaction terms and three-way interaction terms (to include sex) would have been even more complex to interpret compared with a two separate models for men and women. Observed survival (%) time women men (69.26 to 69.64) (72.00 to 72.33) (43.94 to 44.37) (43.52 to 43.91) (32.04 to 32.50) (30.11 to 30.53) - Reviewer concerns regarding lack of colorectal cancer mortality endpoint seem highly significant. - Authors response: We had chosen overall survival as our main outcome rather than colorectal cancer mortality as we think this is the key outcome of most interest to patients and doctors. It is a meaningful, unambiguous outcome, which is understandable by and directly relevant to patients. Overall survival is also the gold standard outcome for the demonstration of clinical benefit in cancer drug trials and is the primary endpoint used in other colorectal cancer prediction models 1 2 including the models MKSCC which reviewer 4 refers to below 1. We think this endpoint is likely to be more relevant and may be more reliable than colorectal cancer specific mortality based on information recorded in death registers (especially when no post-mortem has been performed). - However, we agree that there is a potential clinical utility to having both the overall survival and colorectal cancer mortality since this may give patients and clinicians additional information to help their decision regarding cancer treatments including surgery, chemotherapy and radiotherapy. For example, as reviewer 4 points out, patients with a high risk of dying from colorectal cancer and a low risk of dying from other causes, may be more inclined to consider more aggressive treatments such as chemotherapy compared with those whose risk of death is predominantly due to other causes. - We have therefore revised our analysis to use a competing risks model which enables us to calculate estimates of both all-cause mortality and colorectal mortality. We have updated the abstract, methods (sections 2.5, 2.7, 2.10), results (section 3.3, 3.4, tables 4-6) and discussion (section 4.1) accordingly. - The PHE validation cohort data could be better described regarding data completeness - Authors response: We agree and have added additional text to the paper to make it clear what information is available and what is not available on the PHE cancer registry. Page 2

3 -For example, section 2.1 now states The PHE cancer registry, however, does not include the cause of death or other variables such as smoking, alcohol, body mass index, co-morbidities or prescribed medication. - Section 2.6 has new headings to identify variables from the PHE cancer registry and those from the linked GP record. -section 2.10 has more detail on how missing values on the PHE cancer registry were handled when applying the prediction model. - Section 4.3 includes a new paragraph discussing the limitations of the completeness of the variables in the PHE cancer registry dataset - Lack of consideration of chemotherapy. - Authors response: The PHE cancer registry has binary variables for treatment with chemotherapy and radiotherapy within 12 months of diagnosis of colorectal cancer. -So we have included these in the analysis and updated the methods, results and discussion accordingly. - Chemotherapy met the criteria for inclusion in the model so is now incorporated in the final model along with interaction terms between stage and chemotherapy. - We have undertaken a landmark analysis restricted to patients who had survived at least a year following diagnosis so that the hazard ratios can be compared with the main models to assess the potential for survival bias (see section 2.9 methods and for the results). - We have discussed the potential strengths and limitations of inclusion of chemotherapy in the discussion and compared the magnitude of the effect size with meta-analyses of RCTs of chemotherapy for colorectal cancer. We think this has now strengthened our main analyses and thank the reviewers accordingly. - What is the value of the outcome (total mortality) for general practice. The authors could say more about what this study adds. Its unclear how thus study would be used in primary care. - Authors response: We have added the what is known and what this study adds section which has the following text. We have also added more detailed text along these lines to the introduction and discussion. What is known Realistic estimates of overall survival are important for patients diagnosed with colorectal cancer who need to make decisions regarding the risks and benefits of surgery, chemotherapy, radiotherapy or palliative care There is a lack of robust information on survival which takes account of patient characteristics and likely effect of different treatments. Prognostic models which include more variables tend to produce more accurate predictions than those simply based on stage of cancer at diagnosis. What this study adds We have developed and validated new prognostic models for colorectal cancer which predict both all-cause mortality and colorectal mortality. They include additional clinically relevant variables which are readily available in clinical practice and can be applied to all patients with colorectal cancer. Page 3

4 They include the facility to update the survival estimates conditional on the number of years of survived since diagnosis. They predict survival over a longer period of times and have better discrimination than other models. Decision : Request revised paper. Decision would be 'reject' as this currently stands but willing to review a paper aligned in response to the reviewers and editorial committees comments. Reviewer: 1 Min Lian Washington University School of Medicine, Internal Medicine from Washington University The investigators developed and validated a risk prediction model of absolute and conditional survival of colorectal cancer (CRC) patients. This is an interesting and important study. The study purpose is significant for the improvement of clinical practice and management of CRC patients. The large sample size across a large geographic area is the major strength of the study. The analyses are well-described and easy to follow. There are several concerns on variables selection and statistical modelling for the further improvement of the manuscript. Authors reply: thank you for these comments. Is the Townsend deprivation score an individual-level variable, or area-level variable? Although a literature has been cited, the authors may need a brief description on how this score was computed and what it means it will help general readers to get a better understanding. While individual characteristics and treatment-relevant information are well-considered, some other potential confounders might have been ignored, including the timeliness of treatments, and the use of surveillance lower endoscopic examinations. The treatment delay is an important risk factor associated with CRC survival. Additionally, little information has been provided about the social influences on CRC survival. Geographic accessibility or travel distance to the medical service providers might have impacted on the regular check-up and treatments of CRC patients. The authors may examine, at least discuss this potential issues. Authors reply: The Townsend deprivation score is an area level variable which is assigned to an individual based on characteristics of the population registered postcode (equivalent to a zip code). Originally developed by Townsend 3, it includes unemployment (as a percentage of those aged 16 and over who are economically active); non-car ownership (as a percentage of all households); non-home ownership (as a percentage of all households) and household overcrowding. These variables are measured for a given area of approximately 120 households, via the 2011 census, and combined to give a Townsend score for that area. A greater Townsend score implies a greater level of deprivation. We have made this clear in the section We have included the useful points about other potential confounders (timeliness of treatments, use of surveillance, geographical access etc) in section 4.3 of our discussion as suggested. Page 4

5 The sample size was identified from 947 practices (developing model) and 305 practices (one of validating models). It is possible there is a variation of CRC survival across different practices. How was such practice-specific variation controlled or considered in the development of the risk prediction model? The practice characteristics might be potential confounders for the risk prediction model. If practice-level variation exists, the risk prediction model may need to be built under a multilevel framework. Single-level Cox proportional hazards models could not account for the heterogeneity across the practices and areas, and the correlation within a specific practice and/or a specific area. The investigators may be able to examine the data for the geographic heterogeneity in CRC survival to address if it impacted on the precision of risk prediction models. Authors reply: We have re-run the models using robust variance estimates to allow for clustering of patients within general practices and have updated the methods (section 2.7) and results (tables 4, 5) accordingly. Our focus was on characteristics of individual patients rather than practice characteristics which might not be known to patients or secondary care clinicians wishing to use the risk prediction equations. Reviewer: 2 Doug Altman This is an important paper, extending the outputs from the QResearch database using what is now rather well standardised procedures. The models show very good predictive ability and excellent calibration and I believe they will be clinically valuable. The existence of web tools is commendable. The paper is well written and admirably brief, but inevitably I have wondered whether some more information could be provided in some places, especially for results. Authors reply: thank you for these comments Some specific comments: 1. 8/40, I wasn t clear which comments related to the imputation model and which to the developed prediction model. Authors reply: We have clarified in section 2.7 which comments relate to the imputation model and which to the prediction model in the text. 2. 9/3-9: I m not sure I understand this, so nor will most readers. Authors reply: We have clarified this by adding a new heading called Landmark analysis (section 2.9) with the following text Patients were classified as having received chemotherapy or colorectal surgery if this was undertaken within a year of the date of diagnosis of cancer. However, the precise date of cancer treatment was unavailable. We therefore undertook a landmark analysis 4 to avoid immortal time bias (which would tend to over-estimate the benefit of treatment) 5. We assigned a landmark date which was 365 days following cancer diagnosis. Deaths which occurred between the diagnosis date and landmark date were then excluded from subsequent analyses /26: it is stated that table 3 shows the model. It shows the predictions from the model. In fact nowhere is the model shown beyond a list of main effects in the text and interaction terms in footnote to table. I understand that the full model is very complex, and Page 5

6 also that it might well be updated, but I do think that the reader should be given some insight into which are the most important predictors. In fact some information is given on p12 HRs for a few variables but perhaps they could give adjusted HRs for the most important predictors in each model. Also, it would be interesting to know if the models that are conditional on survival to one year have substantially the same coefficients this information is given for just one variable and it is stated that no such large changes were seen for other variables. This is thus one area where the authors do not in fact adhere fully to the TRIPOD recommendations. (Another is in relation to quantifying the amount of missing data which also is partially reported.) Perhaps more detailed results of this kind could be given in an appendix. Authors reply: We have presented the adjusted hazard ratios for the landmark analysis (i.e. restricted to those who survived at least a year after diagnosis) supplementary table 3. The full algorithm for the main competing risks models is published here (and when the paper is published the need to login will be removed ie it will be accessible without a need to login) URL Username reviewer Password mrsjanetprice We had included this in the original submission but it seems that the link wasn t working. We keep this on the website itself as this means that when people try to implement it, they will automatically find subsequent updates. This is the same approach we have used for all of the various prediction models we published in the BMJ in the last 10 years and ensures transparency but also access to updates and is consistent with advice in TRIPOD 6 for such models. We have added additional information on missing data as recommended by TRIPOD 6. This including a new supplementary table 1 which compares the characteristics of patients with complete data with those with one or more missing values /3: It s unclear if this means predictions can be conditional on surviving any number of years. Results are shown conditional on 1 year survival (7/56; 8/44) but my brief look at the web tool showed that in fact one can condition on survival for up to 10 years see also Fig 3. This issue should be clarified. Authors reply: Thank you for highlighting this. We have clarified the method for producing estimates of conditional survival for overall survival (section 2.8.1) and also cause specific mortality (section 2.8.2). The text now reads Overall survival We calculated overall survival estimates conditional on having survived a given number of years after diagnosis (X) for patients who had already survived Y years since diagnosis by dividing the absolute survival at X years point by the absolute overall survival estimates at Y years 7. Cause specific mortality Page 6

7 To calculate cause specific mortality estimates at a given number of years after diagnosis (X) for patients who had already survived Y years since diagnosis, we calculate the cumulative risk at X years minus the cumulative risk at Y years and divided this difference by the overall survival at Y years i.e. (cumulative risk at X years) (cumulative risk at Y years) overall survival at Y years. In this way, it is possible to calculate predictions for overall survival as well as cause specific mortality conditional on survival for any number of years for each year until 10 years. 5. Figure 1 made me wonder about the distribution of age. Presume very few under age 30 or indeed at the upper end. Would be nice for Fig 1 to show where the bulk of the data were e.g. as a rug plot. For example, while the inclusion criteria specified an age range of this isn t informative about the actual distribution. Nor are the mean and SD alone very informative. Similar comments apply to the other continuous predictors. Authors reply: We have updated table 1 to include the numbers of colorectal cases in each cohort by 10-year age band and the numbers by band of body mass index /16: It isn t obvious how one could replace the deprivation scores by different scores suitable for another country. If this variable is omitted using the tool presumably some mean value is inserted? Authors reply: We have provided more information in section methods on the components of the Townsend score in response to reviewer 1. Whilst it may be possible for some countries to generate their own equivalent deprivation scores, in practice we suspect this will be not be done. Practically, if the postcode field on the web calculator is left blank, then a mean value is used in the calculation. We have clarified this in the text and also added a sentence to section 4.3 of the discussion about how the resulting scores may be affected (i.e. survival may be over-estimated in deprived and under estimated in affluent patients). Minor points 7. I think relative net survival could be explained, and distinguished from absolute survival. Specifically what does net imply? Authors reply: we have added an explanation in the introduction to clarify that relative survival is net of the competing cause of death. 8. 5/28: individualised 5 year absolute survival clarify that this is predicted or estimated. Authors reply: we have clarified in the text that this is predicted /29: I don t think algorithm should be used as a synonym for equation (or model ) Authors reply: we have changed algorithm to models throughout /58: Likewise parameters would be better replaced by variables Authors reply: We agree and have changed parameters to variables Page 7

8 Reviewer 3 Juliet Usher-Smith 1. Thank you for inviting me to review this paper. It is well written and clearly describes the development and validation of risk models for men and women to estimate survival for patients with colorectal cancer taking account of demographic and clinical factors. The authors are established within the field of developing and validating risk models and the methods used are similarly robust to those used in their previous publications. The topic itself is important and, as the authors describe, providing dynamic survival estimates has the potential to inform decision making for both patients and physicians. Authors reply: thank you for these comments 2. However, this is not the first risk model which has been developed to do this. A number of risk models have been published and are available online. These include: Weiser MR, Gönen M, Chou JF, Kattan MW, Schrag D. Predicting survival after curative colectomy for cancer: individualizing colon cancer staging. J Clin Oncol Dec 20;29(36): doi: /JCO Epub 2011 Nov ACCENT-Based Web Calculators to Predict Recurrence and Overall Survival in Stage III Colon Cancer (L.A. Renfro et al., JNCI 106(10), 2014). Although these typically relate to only sub-groups of patients with a diagnosis of colorectal cancer, there is currently no mention of other work in this area within the manuscript which is a weakness. Authors reply: Thank you - we have added these references and others to the introduction where we have also summarised the key features and limitations (restricted to sub groups either undergoing curative intent surgery or with stage 3 disease; failure to take account of co-morbidities; inability to take account of conditional survival and poor discrimination). We have also included a paragraph in section 4.2 of the discussion on how our study improves on earlier work in the area by addressing these limitations and producing a model with substantially better discrimination for example C statistic 0.78 compared with Where this risk model has the potential to improve on these others is the inclusion of additional clinical characteristics included in primary care records. However, of the 15 variables included in the final model, it appears that only 5 were available in the external PHE validation dataset and yet the performance of the model in that dataset and the QResearch validation cohort (in which all 15 variables were present) was similar. This is surprising given the hazard ratios presented in Table 4 but I think Page 8

9 needs some discussion. It is also questionable whether including only 5 variables from a risk model with 15 is really considered validation of that model. Authors response: We have added more information on the completeness of the PHE cohort to the methods and results and included more on the limitations of this in the discussion. 4. Given that the novelty of this work is around the inclusion of additional clinical characteristics, I wonder if some additional discussion around the hazard ratios reported in Table 4 would add to the paper. For example, do the authors have any thoughts about why family history of bowel cancer appears to be more protective in women than men? Authors response: We have added a new table (supplementary table 2) which compares characteristics in men and women with and without a family history of colorectal cancer. We have added some text on this to section 3.2 and of the results and section 4.1 of the discussion. Whilst we agree that the inclusion of additional variables is novel, we also think that our study improves on other models because (a) the resulting model can be applied to all patients with colorectal cancer (rather than one or two selected subgroups); (b) it includes overall survival and colorectal mortality; (c) it includes the facility to update the survival estimates conditional on the number of years of survived since diagnosis; (d) predicts survival over a longer period of time (10 years) rather than 5 years and (e) the discrimination of our model is substantially better as noted above. We have highlighted these points in our discussion and what this study adds Minor comments 5. Page 7 line 17/18 - Did the authors also consider raised ALP or low albumin in the category of abnormal LFTs? Authors response: we used the same definition of LFTs that we had used in other studies 8 i.e. either GGT, ALT or bilirubin more than 3 times normal. We included this definition of abnormal liver function tests as it a definition already implemented in primary care computer systems and is specific to liver dysfunction. We didn t include alkaline phosphatase or low albumin as these can reflect other conditions such as metabolic bone disease or nephrotic syndrome. 6. Page 11 line 5 I think this should read Table 4 rather than Table 3. The same applies throughout the rest of that paragraph. Authors: thank you we have corrected this error. 7. The authors selected blood tests up to 12 months after diagnosis. It would perhaps be interesting to know if the same results would have been found if they had limited the dates up to diagnosis (to remove the effect of post-operative results). Authors: the reason we included blood tests within a year of diagnosis is that we thought it better to use the patients own values within this time period rather than imputation. Page 9

10 Reviewer: 4 Andrew Vickers Memorial Sloan Kettering Cancer Center There is a lot of interesting material here, but several choices about the endpoint, modeling approach and evaluation weaken enthusiasm for the paper. 1. It is unusual to choose overall survival as the endpoint for a study of cancer patients. This is because the endpoint is difficult to use in treatment decision making. Normally we look at either disease specific or other cause mortality. Aggressive therapy is indicated by a high risk of the former and low risk of the latter. It is unclear how a patient would make a treatment decision based on risk of overall mortality. This mortality could either result from cancer (in which case the patient should consider more aggressive treatment) or from other causes (in which case the patient should avoid cancer therapy). Indeed, the relatively high discrimination is likely due to the effect of age on other cause mortality. - Authors respond: We had chosen overall survival as our main outcome rather than colorectal cancer mortality as we think this is the key outcome of most interest to patients and doctors. It is a meaningful, unambiguous outcome, which is understandable by and directly relevant to patients. Overall survival is also the gold standard outcome for the demonstration of clinical benefit in cancer drug trials and is the primary point used in other colorectal cancer prediction models 1 2 including the models MKSCC to which the reviewer refers 1. We think this end-point is likely to be more relevant and may be more reliable than colorectal cancer specific mortality based on information recorded in death registers (especially when no post-mortem has been performed). - However, we agree that there is a potential clinical utility to having both the overall survival and colorectal cancer mortality as outcomes since this may give patients additional information to help their decision regarding cancer treatments including surgery, chemotherapy and radiotherapy. For example, as this reviewer points out, patients with a high risk of dying from colorectal cancer and a low risk of dying from other causes, may be more inclined to consider more aggressive treatments such as chemotherapy compared with those whose risk of death is predominantly due to other causes. - We have therefore revised our analysis to use a competing risks model which enables us to calculate both all-cause mortality and colorectal mortality. We have updated the abstract, methods (sections 2.5, 2.7, 2.10), results (section 3.3, 3.4, tables 4-6) and discussion (section 4.1). 2. It is very standard to adjust models for time of follow-up. As a simple example, the Memorial Sloan Kettering Cancer Center prediction models for recurrence after prostate cancer surgery includes an input for number of months since surgery ( But the approach of the MSKCC prediction model and most others is to apply a straightforward formula: if the risk of failure between time 0 and time n is x, then the risk of failure between time s and n is some function of x and s. The authors make an extremely unusual decision, which is to create a new model at one particular follow-up time. There is no particular reason to believe that predictors would predict differently at time zero vs. one year. Moreover, the authors approach restricts the analysis to that one year Page 10

11 follow-up time. What if a patient is 2 years after diagnosis and wants to know their risk? Author s response: We actually did the conditional survival calculations as professor Vickers suggests but realise our description must have been unclear. We have therefore updated the text in the methods on producing estimates of conditional survival and added a new heading (section 2.8) to make this clearer. The text now reads Overall survival We calculated overall survival estimates conditional on having survived a given number of years after diagnosis (X) for patients who had already survived Y years since diagnosis by dividing the absolute survival at X years point by the absolute overall survival estimates at Y years 7. Cause specific mortality To calculate cause specific mortality estimates at a given number of years after diagnosis (X) for patients who had already survived Y years since diagnosis, we calculate the cumulative risk at X years minus the cumulative risk at Y years and divided this difference by the overall survival at Y years i.e. (cumulative risk at X years) (cumulative risk at Y years) overall survival at Y years. In this way, it is possible to calculate predictions for overall survival as well as cause specific mortality conditional on survival for any number of years for each year until 10 years. Separately to the conditional survival analyses, we undertook what we now refer to as landmark analysis (as we described above in response to reviewer 2 and reviewer 5 who raised a similar point). This analysis was done to assess potential bias in our main analyses since it avoids immortal time bias for cancer treatments. To make this clearer, we have added a new section now called Landmark analysis (section 2.9) with the following text Patients were recorded as having received chemotherapy or colorectal cancer if this was undertaken within a year of the date of diagnosis of cancer. However, the precise date of cancer treatment was unavailable. We therefore undertook a landmark analysis 4 to avoid immortal time bias (which would tend to over-estimate the benefit of treatment) 5. We assigned a landmark date which was 365 days following cancer diagnosis. Deaths and other censored events which occurred between the diagnosis date and landmark date were then excluded from subsequent analyses. 3. The authors make another unusual choice in presenting a different model for men and women. This suggests that there is an interaction between sex and predictor variables (e.g. diabetes has a different effect on mortality for men and women). But eyeballing, there doesn t seem to be. Why not just have one model and include sex as a predictor? Page 11

12 Author s response: In our protocol, we decided to present separate models for men and women since we considered there would be systematic differences between the sexes for baseline characteristics and also factors which are likely to affect all-cause mortality. We have updated the tables 1 & 2 and text in the results (section 3.2) to highlight this. For example, table 1 shows differences in tumour location % of women in the QResearch derivation cohort had colon cancer compared with 59.7% of men. Use of chemotherapy tend to be higher in men than women (33.0% vs 27.9%) and radiotherapy (16.8% vs 11.8%). Table 2 shows that men are more likely to be ex-smokers than women (40.1% vs 24.9%). Men are also more likely than women to have cardiovascular disease (23.3% vs 15.5%) and type 2 diabetes (14.8% vs 11.2%). Men are more likely to be prescribed statins (29.7% vs 21.6%) and aspirin (24.5% vs 17.0%). Women were more likely than men to be anaemic at diagnosis (28.3% vs 20.3%) and have raised platelets (11.8% vs 6.6%). Observed survival rates also differ between men and women as shown in the table below with men having a higher 1-year survival but a lower 10-year survival. time women men (69.26 to 69.64) (72.00 to 72.33) (43.94 to 44.37) (43.52 to 43.91) (32.04 to 32.50) (30.11 to 30.53) The revised models (tables 4 and 5) already have quite complex interaction terms and this would have been even more complex to interpret with a single model incorporating both men and women. 4. The authors present a model that includes a large number of variables, many of which are subject to considerable missingness. Might they consider a simplified model with a smaller number of variables? Author s response: The completeness of the data in the derivation cohort was good as shown in table 1. For example, smoking was recorded in 94% of patients, body mass index in 83%, platelets in 72% and LFTs in 73% of patients; stage was recorded in just over 80% of patients and grade in 77%. The completeness of data was similar for the QResearch validation cohort. In the derivation cohort, 40.0% has complete data for the all variables compared with 37.6% in the validation cohort. We have added a supplementary table 1 which compares the characteristics of patients with complete data with those with one or more missing values as recommended by TRIPOD 6. Overall, we think the levels of completeness in the QResearch data are reasonable and any missing data have been replaced by multiple imputation. However, the PHE cancer validation cohort did not contain many of the variables derived from the linked GP record (i.e. the variables listed in Table 2). As described above, we have updated the methods, results and discussion to highlight this. We have thought about whether to present a simpler model and have decided to continue with the full model because it maximises the use of the available data and Page 12

13 includes variables which are well known to affect mortality (such as smoking, CVD, diabetes etc). In the UK (and other parts of the world) many clinicians can now access data from both primary and secondary care and such access is likely to increase in future. This means that the models can be implemented and the implementation itself is then likely to drive improvements in completeness of recording of additional information over time. We have also included a new figure 4 which illustrates how risks change when information from the linked GP record is included. 5. The authors evaluate their models by D, ROC and R2. It is unusual to use multiple different measures of discrimination and I can t see the benefit to doing so. ROC is the most common and I would recommend that the authors use that. However, the authors do not explain how they calculated time dependent ROC. Did they use Kaplan Meier methods or only include patients will complete data at 5 and 10 years? Why not use Harrel s C index instead? Author s response: We used Royston s D statistic 9 as a measure of discrimination as it takes account of the survival nature of the data and improves on the ROC statistic. We also included the related R 2 statistic 10 as a measure of explained variation. Also previous methodological articles in the BMJ have highlighted the suitability of the D statistic for survival data 11. Many other papers validating prognostic models (both our own papers and those by both by Professors Collins and Altman ) have presented D, R 2 and ROC. The ROC has often been included to enable comparison with other papers where only the ROC has been used. However, we agree that Harrell s C is preferable to the ROC statistic to account for the survival nature of the data and so have calculated this instead of the ROC statistic now and have updated the methods (section 2.10) and results (section 3.5, table 6) accordingly. 6. The authors claim to follow TRIPOD, but actually several key pieces of information are missing. Author s response: Thank you for highlighting this as described above, we have added additional information on missing data in line with table 4 from the TRIPOD paper 6 this can be found in supplementary table 1. We have also added the numbers of patients in the landmark analysis (section 3.4.3) and included a link to the publication of the full algorithm coefficients as open source software under the LGPL license ( ). 7. The authors state that: research is needed to evaluate the clinical benefits and cost effectiveness of using these risk equations in clinical practice. But this sort of evaluation doesn t need to await further research, because decision analytic methods can be used on the authors data set. See Steyeberg for a general overview. For one specific decision analytic measure that has been recommended by many leading journals including JAMA, Annals and BMJ see Author s response: We agree and have amended the conclusions of the paper accordingly in the abstract and main paper. Page 13

14 Minor comments 1. Table 3 is not very informative. If could be reformatted so that it is easier to see whether there are any important differences between cohorts. Author s response: we agree and have reformatted table 3 as suggested 2. The calibration plots are small and do not follow a standard presentation in which predicted risk on the x axis is plotted against observed risk on the y axis. Author s response: Thank you for this suggestion. We have updated the calibration plots to enlarge them and have present predicted risk on the x axis and observed risk on the y axis. We have included the 45-degree line as requested by reviewer On page 30, the authors present what appears to be a screen shot of an online tool. This does not appear to be referenced in the paper and is highly problematic in that it does not follow basic good practices in risk communication to patients. Author s response: The reviewer is correct, this is a screen shot of an online tool. We had included a link in the results and after the abstract but have now included a new heading web calculator to make this clearer. We have also substantialy revised the website to focus on a simpler graphical representation which we think will be more accessible. We have retained the table of values though the user (intended primarily for clinicians and academic researchers) but the user has to click on a button to display this. Reviewer: 5 Lindsay Renfro Mayo Clinic, Health Sciences Research The purpose of the paper by Hippisley-Cox and Coupland is to develop and validate clinical prediction models for overall survival among patients with colorectal cancer. Data utilized for model development and validation includes a very large number of patients from English general practice contained in the QResearch database and the national cancer registry. As a result, the scope of this project is quite large, with the authors constructing models to predict overall survival across all 4 stages of disease. A few major and minor considerations would greatly strengthen this paper: MAJOR: 1. While the methodology used to build the models is quite strong (Cox regression modeling), the methods used to evaluate and validate the models are less clear. As is the standard in the literature for these types of clinical calculators, C-indexes for survival data should be included as a measure of discrimination, and these C-indexes should be corrected for optimism, as described in E Steyerburg's book "Clinical Prediction Models", Frank Harrell's book "Regression Modeling Strategies", and as implemented in the R package "rms". Also, as measures of calibration, the calibration plots presented should be much larger so that the individual groups and Page 14

15 their intervals can be read with respect to a 45-degree line. For each calibration plot, the x and y axis for predicted versus observed survival should each range from 0-100%, and the 45-degree line should be plotted with the calibration groups superimposed (see above references for explanation, code, and examples). Otherwise it is extremely difficult to visually tell whether the models are wellcalibrated; in fact, in the plots presented, it appears there are several cases where the models are *not* well calibrated, but because they are so small, this is very hard to determine. Authors response: Thank you for these suggestions. We have re-run the analyses using Harrell s C statistic 31 which takes account of the censored nature of the data and presented this instead of the ROC statistic. We have also updated the calibration graphs as suggested to make the larger and include the 45-degreee line as described above. 2. I would strongly recommend that instead of building 2 different models (one for men, and one for women) across all 4 stages of cancer, instead, the authors build a separate model for each cancer stage (I,II,III,IV). It is well known that patient sex is not a strong prognostic factor at any stage, so there is no reason to separate models based on sex. However, prognostic factors tend to vary strongly by stage of disease (some reading by the authors of the clinical literature on prognostic factors by stage should underscore this). Therefore, it makes far more clinical sense to build separate models by stage, where it is possible that different factors will appear in the final models for each stage. These authors certainly have enough data to do this, and it could greatly strengthen the paper to take this approach. Authors response: Thank you for these points which we have given careful consideration. As we mention above, since there are differences between men and women - in baseline survival, prevalence of risk factors and in the magnitude of the adjusted hazard ratios - we have presented separate models for men and women as per our protocol. We think that testing for interactions by stage is preferable to having 4 separate models by stage, and 8 in total with separate ones for men and women. Otherwise there are a lot of models, numbers get smaller so estimates would be less precise. Also, there is some missing data for stage which we have imputed, but with separate models by stage we d probably have to drop people with missing stage (since otherwise imputed values would vary across the imputed datasets) so losing information and potentially having a less representative cohort. 3. It is unclear why the authors limited tests of 2-way interaction to age*factor across all other factors, when it is quite likely that other meaningful 2-way interactions exist. I suggest setting a very conservative rule for allowing 2-way interactions into the final model, and then looking at all possible interactions (e.g., between grade and sex within a model for overall survival in stage II patients). Page 15

16 Authors response: We agree and have now tested for many 2 way interactions and used a conservative rule (p < 0.001) for allowing interactions into the final model. We have updated the methods (section 2.7) to say we tested for additional 2-way interactions between age and each predictor; stage*each predictor; combinations of stage, grade, surgery, chemotherapy. We included significant interactions. (p < 0.001). We have updated the results text, tables and graphs accordingly 4. While I applaud the authors for considering that the effects of continuous variables might be non-linear on the log relative hazard scale, I wonder if their choice of nonlinear modeling approach is optimal. For example, it has been published by the ARCAD group (Lieu, Renfro, et al., JCO) that risk of death is *highest* for the youngest patients with stage IV disease, lowest for middle-aged patients, and increases again for elderly patients. I was surprised not to see the same pattern in these data, but perhaps this is because the non-linear modeling technique used did not allow for this shape to occur. I recommend restricted cubic splines for age and other continuous variables within Cox models -- the references provided above show how this can be done -- it is extremely straightforward in the R package "rms". Authors response: Using fractional polynomial (FP) modelling of age with 2 FP terms we would be able to detect a U or J shaped association such as highest risks in younger and older patients if there was one. A simulation study comparing the two approaches found they often select similar explanatory models when there is a large amount of information and that FP modelling can perform better than spline modelling for several criteria 32. They say With larger sample sizes, however, MFP often selects a function close to the truth, provided the true function has a limited number of extrema or inflection points. Spline models can be better for very complex functions. Looking at the Lieu paper 33 they found much smaller increases with age than we did for stage 4. Also, although they found a non-linear association with age in univariate analysis, we think for the multiple variable analysis they did age wasn t significant at all (p=0.08) for overall survival, and they say the contribution of age was found to be linear, with (marginally significant) increased risk of death for older patients. 5. I do think this could be a VERY strong addition to the literature with corrected methodology as described above, particularly if implemented online for clinical use once validated. I think you should include a statement of your plans to put online, or perhaps try to have it put online by the time of publication, assuming the final models are in fact well-calibrated with good discrimination. Authors response: Thank you for these encouraging comments. To clarify, we have developed an online calculator already and had included a link to an online web calculator in our original submission. However since only one of the reviewers found it, we have now highlighted this link more prominently in our revised manuscript by adding some new text and a heading (section 3.6). Page 16

17 MINOR: 1. Other "competing" colon prognostic calculators, such as those built by Renfro et al / Mayo and the MSKCC group, which are currently online and published in JNCI and JCO, respectively, should be acknowledged somewhere in the paper. Authors response: We have added an evaluation of these papers to the introduction (section 1, 3 rd paragraph) and section 4.2 of the discussion. 2. The exact methodology for producing estimates of conditional survival are unclear. Do these estimates utilize the models for overall survival from diagnosis, or do the authors actually subset to those patients who survive at least one year and build new models to see what then influences survival? It seems like the latter approach would be more reasonable with fewer assumptions based on the full models with survival from diagnosis. Authors response: We have clarified the method for producing estimates of conditional survival in section 2.8 as described above in response to reviewers 2 and 4. We calculated conditional survival estimates over X years (for example, the 5- year survival) for patients who had already survived Y years since diagnosis by dividing the absolute survival at X years point by the absolute survival estimates at Y years 7. In this way, the model is able to calculate predictions conditional on survival for any number of years for each year until 10 years. 3. The discussion of predictor variables (page 6-7) should include more precise definitions as well as explanations of how the categorical variables were categorized (e.g., mention all possible levels of categorical variables in parentheses so the level of detail contained in each variable is perfectly clear). Authors response: We have updated the section on predictors (section 2.6) to include precise definitions and categories as requested. 4. Be more clear about your criteria for including interactions in final models, beyond "improved model fit", which is vague. Authors response: We have clarified this in the section The text now reads We tested for additional 2-way interactions between age and each predictor; stage* each predictor; combinations of stage, grade, surgery, chemotherapy. We included significant interactions (where P < 0.001) 5. Cite the delayed entry approach mentioned on page 8 for those not familiar with it (if it continues to be used given comments above). Authors response: We have included a new section 2.9 entitled landmark analysis which includes a more detailed description of what was done and the rationale. 6. On top of page 11, it seems you mean to refer to Table 4, not Table 3. Authors response: We have corrected this. References Page 17