Dear Editorial Committee and Reviewers, Many thanks for your helpful comments and suggestions. We have addressed these below:

Transcription

1 Dear Editorial Committee and Reviewers, Many thanks for your helpful comments and suggestions. We have addressed these below: The committee was interested in the topic of your research. The following concerns were mentioned: The committee shared the reviewers concerns. They have fairly serious concerns about important items missing from these databases, particularly operative items such as blood loss, likely to have a powerful influence on risk of acute kidney injury. One of the reviewers asks searching questions about the influence of trauma. We agree that peri-operative factors such as blood loss may have an important influence on risk of AKI. However, our aim was to develop a pre-operative risk score which can be used by clinicians pre-operatively to predict whether patients are at risk of developing post-operative AKI and thus lead to modifications in peri- and postoperative management (as described in our discussion on page 17). It is for this reason we did not include peri-operative variables. We have now made this clearer in the manuscript (page 4-5). Lack of data on co morbidity worries us. We did not include data on co-morbidity as we had concerns regarding underreporting in the database for our orthopaedic surgery population (peripheral vascular disease 0.8%, hypertension 10.4%, liver disease 0.5% and congestive cardiac failure 0.8%). We have added a reference for this in the manuscript (page 15). However, to further check our results we assessed whether the comorbidity data we do have available (albeit under-reported) added something to the model. Although hypertension and liver disease were statistically significant, they did not add to the predictive performance of the model and the c-statistic did not change (0.74). In the interest of space, we decided not to add this additional analysis to the manuscript, although this could of course be changed if so desired. As a proxy for comorbidities we did include number of medicines in our model which has been shown to correlate well with co-morbidities (Perkins AJ, Kroenke K, Unutzer J, et al. Common comorbidity scales were similar in their ability to predict health care costs and mortality. Journal of clinical epidemiology 2004;57(10):1040-8) as described in our discussion (page 15). Does AKI have long-term deleterious consequences? Or are we over diagnosing AKI (as when we use the CKD scales)? As described in the introduction, there is a growing evidence base showing the long term consequences of AKI. Our paper also adds to this by demonstrating the strong effects on mortality, even longer term. These epidemiological studies have all been adjusted for multiple factors but it remains uncertain whether AKI leads to these consequences or people who develop AKI are sicker. We have now mentioned this in the discussion (page 15). Nevertheless, there is a global focus on prevention of AKI. We feel that our simple risk score would be a potentially low cost method of preventing AKI with significant long term health and financial benefits. 1

2 A clearer read than most papers reporting prediction models, and we get a screen shot of the actual score at work. Thank you for this compliment. Please find the statistical issues in Gary Collins review. We have addressed all his points below. REFEREES COMMENTS Reviewer: 1 Recommendation: Comments: Dear Editors, I read with great interest the manuscript from Bell and colleagues regarding risk of AKI following orthopaedic surgery. The manuscript report results of a large retrospective cohort study. Adult patients who underwent orthopaedic surgery between 2005 and 2011 in three hospitals were included in this study. Data were extracted from several administrative and local databases. As acknowledge by the authors several relevant variables, including co-morbidities, severity or sepsis at time of surgery, were missing. Patients with missing pre or post surgery creatinine or with end-stage renal disease requiring renal replacement therapy were excluded. Two cohorts were analyzed, a development cohort (Patients hospitalized in two hospitals) and a validation cohort (patients admitted to a third participating hospital). Baseline creatinine was defined as the latest pre-surgery creatinine for patients with elective surgery and as the latest creatinine prior to hospitalization in patients with emergency surgery. AKI was defined according to the creatinine criteria of the KDIGO definition. Diuresis criteria was not used for this study. Overall, incidence of AKI was of 10.8% in the development cohort (672 of 6220 patients) and 6.7% in the validation cohort (295 of 4395 patients; P< when compared to the development cohort). Risk factors of AKI after adjustment were gender, age, diabetes, number of prescribed drugs at time of surgery, use of ACE or ARB, chronic kidney disease, and ASA score. Calibration of the model was good in the development cohort but was suboptimal (as consequences of the lower than expected AKI incidence) in the validation cohort. 2

3 AKI was associated with a higher short- and long-term mortality. The manuscript is clear and well written. This manuscript report results of a large database analyzed thoroughly. The authors have to be congratulated for this very nice work. Provided information are clinically relevant although of confirmatory nature. Hence, incidence of AKI is within previously reported ranges (Kateros BMC Nephrol 2012, Abelha Crit Care 2009, ) and this study confirms both influence of usual risk factors for AKI and prognostic impact of this later (Kheterpal et al. Anesthesiology 2009). Development of a prediction model may help in developing individualized strategies but this model is insufficiently evaluated and seems to have a limited positive predictive value. Last generalizability of the results is questionable and may deserve to be discussed through the manuscript. We thank the reviewer for his positive comments and have addressed his concerns below. Please find enclosed my main comments: Major comments: 1- First, several methodological limits may deserve to be more clearly discussed through the manuscript: a. Definition of AKI: As consequences of the retrospective nature of the study, only serum creatinine was used to assess renal dysfunction. Previous studies in ICU demonstrated that use of urinary output criteria may tremendously increase rate of AKI, especially for higher AKI stage (Joannidis et al. Intensive care med 2009). We agree that we could have even higher rates of AKI if we were able to use urine output to define AKI. However, urine output measures are very poorly measured and/or recorded outwith the ICU setting and so the majority of other hospital populations utilise the KDIGO creatinine based criteria to define AKI. Due to lack of studies examining urine output criteria in these populations, validation of the KDIGO urine output criteria is limited. We have addressed this issue more extensively both in the methods and the discussion (page 6 & 16), and as far as this could be considered a limitation we have now acknowledged this in the discussion with the reference provided (page 16). b. Baseline creatinine was defined by the admission serum creatinine in most of the patients. Although this surrogate for baseline creatinine is perfectly acceptable, its use may significantly underestimate AKI incidence (Siew et al. Kidney international 2010). Many thanks for drawing this to our attention. There is an absence of an agreed definition for baseline renal function therefore we used the admission creatinine which is the most readily available in clinical practice. We have acknowledged that this may underestimate AKI in the discussion and referenced the above paper (page 15). 3

4 c. Important data regarding comorbidities, sepsis at time of surgery, trauma or hemodynamic status are missing. Every of these variables are also well known risk factor of AKI unaccounted for in this study. Indeed, certain variables that might improve performance of the model were not included for the following reasons: We examined co-morbidities using hospital coding and found that they were far lower in this orthopaedic surgery population than would be expected (peripheral vascular disease 0.8%, hypertension 10.4%, liver disease 0.5% and congestive cardiac failure 0.8%). This is likely to be due to problems with under-reporting. We therefore used number of pre-admission medicines which has been shown to correlate well with co-morbidities (Perkins AJ, Kroenke K, Unutzer J, et al. Common comorbidity scales were similar in their ability to predict health care costs and mortality. Journal of clinical epidemiology 2004;57(10):1040-8) as described as a limitation and referenced within the discussion (page 15). Our aim was to develop a preoperative risk score which can be used by clinicians pre-operatively to predict whether patients are at risk of developing post-operative AKI and thus lead to modifications in peri and post-operative management, as we have described in our introduction (page 4) and discussion (page 17). It is for this reason we did not include peri-operative variables. We have made this clearer in the manuscript (page 4) and have now acknowledged in the discussion that inclusion of peri-operative factors may improve model performance (page 15), although the model would then serve a different purpose. We did include whether the operation was elective, emergency or expedited in our model. As the emergency operations comprise mainly of trauma, this variable serves as a proxy for trauma. It did not, however, remain in our final model. We have included a comment on this in the methods (page 8). Finally, we would like to point out that not all known aetiological risk factors are necessarily strong predictors within a full multivariable model. For instance, some comorbidities may be an aetiological risk factor, but adjusted for medications which are prescribed as a consequence of having that comorbidity, they may well be no longer predictive for mortality. Furthermore, the lack of certain variables does not make these results invalid; it merely means predictive performance could potentially be improved, which we have now acknowledged more distinctly in the discussion. d. Patients from the development cohort were included in two hospitals raising concerns regarding generalizability of the results. This limit is further underlined by the significantly lower incidence of AKI in the validation cohort. We agree that ideally every risk prediction score would be developed in a large dataset covering diverse populations, for example multiple regions or hospitals to enhance generalizability. In practice, however, this ideal situation is extremely hard to achieve. That is why external validation is so important before risk scores should be applied in practice. Our large dataset with > 10,000 patients covering not one but two hospitals and a third hospital for external validation makes a good effort towards 4

5 the ideal in our opinion. More external validation in different settings is necessary before implementing our score in practice, as we have stressed in the discussion (page 16). The fact that the validation cohort had a lower incidence of AKI indeed underlines the importance of external validation, and actually makes our external validation even more significant: the true test is whether the model still performs well in such a different setting. Indeed, it does, after recalibrating to this incidence, which is a simple adjustment as we have explained in the manuscript. Thus, we feel the significantly lower incidence of our validation cohort is actually a strength of our study, and hope the reviewer can appreciate our point of view. 2- It may be relevant to report length of stay in studied patients. Additionally, since AKI was defined as a serum creatinine elevation during the first 7 days AKI may be viewed as a time dependent variable and should perhaps be analyzed as such. We agree with the reviewer that length of stay may indeed be important and have therefore added it to the description of our cohorts (Table 1). As AKI was defined across a period of 7 days a concern could be that patients without AKI with a shorter stay (0.9%) could be underdiagnosed. However, patients with rising serum creatinine values would be unlikely to be discharged. Therefore, we feel it is safe to assume our definition captures at least the large majority of AKI cases in this setting and thus can be analysed as a dichotomous variable. 3- As stated above, most of the findings mainly confirm those of previous studies in this field. Interestingly, despite differences regarding AKI definition, risk factors and variables included in the regression model are close to the one reported by the study of Kheterpal et al. (Anesthesiology 2009). In the discussion of our manuscript we compared our study with the one described above (page 13/14). Indeed, the list of predictors included in the model by Kheterpal shows similarities to our list, which we have now noted more explicitly (page 13). However, we would like to stress that our model predicts an, in our view, more relevant and up-to-date outcome (KDIGO), also including milder AKI, in a time frame more likely related to the actual surgery (7 days vs 30 days). Hence, although partly similar predictors proved relevant, the models are different in goal and essence: a prediction model is after all defined by predictors, outcome and time frame. An interesting analysis would be to assess the predictive performance of the Kheterpal et al. general surgery model for predicting our different outcome in the orthopaedic setting. However, as a number of the variables in that model are missing in our dataset a head-to-head comparison is unfortunately not feasible. For further detail, see also our reply to the second comment of reviewer 4 (page 11 of this rebuttal). 4- Additional data might deserve to be reported in way to help in assessing potential interest of the prediction model. a. Reporting in a specific table, at pre-defined level of risk score (quartile or according to risk of AKI), the predicted vs. observed risk of AKI along with the number of patients at risk may be informative. We thank the reviewer for this suggestion which helps the reader interpret the model results. We have added Table 4 to the manuscript on page 22 which shows the AKI incidence across different risk categories in both the development and validation 5

6 cohorts. Results in the validation cohort again show the model over-predicts actual AKI incidence, hence the need for recalibration. We have added a few lines referencing the table in our results section on page 12. b. Figures 1a to 1d, may suggest that the developed model have a high negative predictive value and a low positive predictive value. If so, this may greatly limit relevancy of this prediction score to develop specific strategies in preventing AKI which may deserve to be discussed. Conversely this model may help in identifying patients Indeed, figures 1a to 1d show the model over-predicts risk of AKI in the external validation cohort, especially in the highest risk categories, and spread of predicted risk is limited, resulting in limited positive predictive value, as is also clear from the added Table 4 as described in our reply above. Nevertheless, the incidence in the higher risk categories is still quite substantial, and worth targeting preventive measures, in our opinion, especially since non-invasive, relatively low cost measures are available for preventing AKI (see also page 17 of the manuscript). Also, please note that though our PPV may be limited, it is substantially higher than that of the only other general surgery risk score for AKI by Kheterpal et al. (2009) (although this is of course partly influenced by the difference in incidence). c. Comparison to other models developed in general surgery may be of interest. As we have clarified in the discussion, there is only one published risk score in general surgical patients that we know of, that is the one by Kheterpal et al. (2009). We have compared our study to this general surgery risk score in the discussion (page 13) see also our reply to your comment 3 above. Interestingly, our model shows similar, if not higher, PPV as this model (see also our replies above), and also other performance measures are comparable. However, the Kheterpal et al. model is only validated internally using a split-sample approach: a method that tends to show over optimistic results because of similarity of development and validation sets as a result of the random split. In contrast, we have validated our model both internally through bootstrapping and externally in a separate hospital. Besides this general surgery model, risk scores for predicting post-operative AKI have been developed in different surgical specialties. Predictors in these risk scores however are generally very specific to the particular specialties, rendering them not applicable to our orthopaedic population. Nevertheless, we have added a paragraph in the discussion describing other risk scores for post-operative AKI, to give the reader a broader impression of this research field (page 13). Minor: Several of the variables included in the logistic regression model are likely to be correlated (age and number of medications, ACE/ARB and diabetes or chronic kidney disease...). Although this could very well be the case, the fact that they were retained in the final model after the backward selection procedure means these variables have 6

7 independent predictive value. Thus, although they are correlated, they have added value for the model independent of the others. However, the reviewer is perhaps worried about potential statistical problems arising from multicollinearity. Therefore, we determined variance inflation factors. Variance inflation factors (VIFs) over 4 indicate some multicollinearity, VIFs over 10 indicate serious multicollinearity that may need to be resolved. The VIFs of most of the variables in our model are well below 4, only CKD show signs of multicollinearity with VIFs between 4 and 8 for the different categories. The fact that CKD stage is correlated with things like age and sex seems plausible. However, multicollinearity would not be expected to bias the coefficients from the equations but only inflate standard errors which would widen the confidence intervals and reduce the p-value. And even though CKD is thus correlated with the other variables with the model, inflating the standard error and reducing its significance, it was still a significant predictor on top of the others. When VIFs are very high, the ability to distinguish which of the variables contributes to the improved prediction is reduced. However, as we are not interested in the aetiology of specific risk factors, but in the prognostic performance of the model as a whole, and as none of the VIFs are worryingly high, we feel it is safe to retain all variables in our final model. Reviewer: 2 Comments: A very well designed work and very well presented. We thank the reviewer for his positive comments. Reviewer: 3 Recommendation: Comments: Thank you for the opportunity to review this well written manuscript. The paper addresses a very important topic and the prediction model objective is particularly interesting. There are no effective treatments for acute kidney injury. Thus, risk stratification and prevention are paramount to decreasing the burden of AKI on public health. The availability of a valid prediction model would therefore have great importance for patient care and future research. For my review, I will address each aim of the manuscript separately. Prediction model: The authors have conducted a rigorous study that has a number of strengths: the study has a very large sample size with many outcome events (over 600 in the derivation cohort). This results in nearly 60 events per variable entered into the prediction model, which is very important for prediction model building. They used a validated, international consensus outcome measure (KDIGO); and their model building procedures were statistically sound. Of particular note is the use of both 7

8 internal validation (bootstrapping estimates of optimism) and external validation. In addition, their methods to assess discrimination and calibration were appropriate. However, the most important determinants of any prediction model s performance are the variables entered into the model how they are defined, measured, and or collected. The authors utilize several databases and electronic health records as data sources; a number of which seem to be unique to the United Kingdom. This has major implications for the expected performance of this model when applied to health systems from other geographic areas. Would model performance be similar if applied to other database resources (e.g. administrative claims data; or other EMR data such as the THIN database)? The authors should address these issues in their discussion. We thank the reviewer for this valid point and have added this to the discussion (page 16). We now stress that validation in other geographical areas is necessary before implementing the score in other populations, as databases were used that may be typical of the UK. Other types of databases could indeed have consequences for model performance, although we feel the predictors included in our model are fairly objective and probably well documented in various types of databases, especially in the (orthopaedic) surgery setting the model was developed in and intended for. The THIN database, for example, does not seem to lend itself for applying our model for predicting post-operative AKI, as it concerns a GP database(and happens to be a UK-database as well). In addition, greater detail regarding the data sources for key variables might improve the likelihood that other investigators could implement the model successfully. We apologise for the lack of detail and have included more detail regarding the data sources as requested (page 6). The model showed moderate discrimination and sub-optimal calibration. Recalibration improved performance, however, it is unclear from the manuscript which model is put forth as the final model (i.e in the Excel spreadsheet). Indeed, calibration was good in the development cohort but suboptimal in the validation cohort. As explained in the manuscript, this was mainly a consequence of the difference in AKI incidence in the different cohorts, that is, a difference in baseline risk. By adjusting the intercept of the model to the situation the model will be used in, it can be recalibrated to that situation. Alternatively, the second recalibration method can be used. We chose to present the calculator for the development cohort (and thus not recalibrated) as an example, which we have now better clarified in the manuscript by adjusting the Appendix 3 title. The formula can of course be easily adjusted to the recalibrated models, if so desired. In addition, the authors do not address how or whether other health systems wanting to use this model should pursue re-calibration. These issues need to be clarified. Many thanks for this comment, which also relates to reviewer 1 s comment 1d (page 4 of this rebuttal). We have now clarified in the discussion (page 16) how to deal with a differing AKI incidence. 8

9 The derivation and validation cohorts were composed of both elective and emergent orthopaedic surgery cases, the latter likely including trauma patients. The trauma population is very unique in terms of baseline characteristics (generally young males without comorbidity) and acute physiology. It seems therefore that AKI risk could be substantially different in the emergent vs. non-emergent populations, which would have important implications for model performance. Moreover, the relationship between predictor variables and outcome might differ in emergent vs. non-emergent populations. The prevalence of this covariate differed substantially between the internal vs. external validation cohorts. Would the model perform better if the derivation cohort were limited to only one type (e.g. only non-emergent or only emergent)? Did the authors test the urgency variable for interaction? We did test the type of surgery variable for interaction. We apologise for the lack of clarity and have clarified this in the text (page 11). Nevertheless, we repeated our analysis in non-emergent and in emergency patients only and results were similar in the two strata. In both models now only six predictors were retained: number of medicines was no longer in the final models. Parameter estimates were quite similar, both in direction and order of magnitude. Performance was similar as in the total cohort, for example the c-statistic in the development cohort was slightly higher (0.77) in emergency and slightly lower (0.72) in nonemergent patients than in the total cohort (0.74). In the validation cohort, this result was reversed: in emergency patients the c-statistic was 0.68 and in non-emergent patients 0.71, where the total validation cohort had a c-statistic of Calibration showed a similar pattern: in non-emergent patients the calibration slope and calibration-in-the-large in the validation cohort improved slightly compared to the total cohort. The calibration slope was now 0.88 (vs 0.79 in the total cohort) and the calibration-in-the-large was 6.7% vs 10.3% (instead of 6.6% vs 10.4% in the total cohort. For emergency patients the results were reversed: calibration performance was somewhat worse than in the total cohort. Hence, even though the non-emergent model seems more stable than the total model and its performance slightly higher, the difference is only marginal. In the interest of space, we chose not to add these results to the manuscript, but this could of course be changed, if the reviewer preferred. Effect of AKI on mortality In a secondary objective, the authors examined the effect of acute AKI (within 7 days post-op) on both short and long term mortality. This objective is novel, in that there are few data examining this question in the orthopedic surgery population. However, the association between AKI and mortality is clearly established. Thus, the results of the study are not new. Showing that the association holds in a previously unstudied population does further the validity of this association. However, like nearly all studies examining AKI and mortality, the observation nature makes it impossible to determine if the excess mortality is due to the development of AKI; or whether sicker patients that have a higher mortality risk at baseline develop AKI. 9

10 We appreciate that the reviewer acknowledges the added value of these corroborating results in a different population to further the knowledge base regarding this association. We agree that the causal direction of this association is not evident and have now commented on this in the discussion (page 15). The authors should explain their method of confounder selection in the cox-model. Were variables retained in the model based on p-values, AIC values, or changes in the point estimates? Lastly, it appears that the potential confounding variables in the survival model were the same variables chosen for the AKI prediction model based on their relevance to AKI prediction. The authors should comment on there prognostic importance for survival. Indeed, from our manuscript our method of confounder selection was unclear and we thank the reviewer for pointing this out. We have now clarified in the manuscript that confounders for this aetiological analysis were selected based on literature and clinical expertise, and thus independent of estimation results. The fact that the list of confounders shows overlap with the list of final predictors is mainly a result of the fact that the number of variables available in the data is limited. Nevertheless, it is interesting that known confounding factors for survival after AKI are also strong prognostic factors for getting AKI in the first place. However, as we aim for a clear distinction between the predictive exercise of a risk score for AKI and the aetiological analysis of survival and to avoid confusion and aetiological interpretation of our prognostic results, we have refrained from discussing this in the current version of our manuscript and hope the reviewer agrees with this approach. Reviewer: 4 Recommendation: Comments: This is a good study in terms of conduct and the reporting is generally ok, needs to minor tweaks here and there to ensure they are adhering to the TRIPOD guideline for prediction models ( - STROBE is not relevant here. We have completed the TRIPOD checklist and attached this. Furthermore, we have added references to the TRIPOD guideline in our methods section (page 5), and made some changes to further improve adherence to the guideline, such as more detail on eligibility criteria and data sources, stratifying baseline characteristics by outcome and adding a worked example to explain in more detail how to use the model (pages 5, 6, 20, appendix). Sample size (and number of outcome events), AKI is large in both the development and validation cohorts. The carry out bootstrapping to internally validate the model too - all good practice. They generally look to be doing all the right things (handling of missing data, sensitivity analyses, continuous predictors examined carefully). 10

11 Thank you, that is at least what we tried to do. (page 4) The authors mention there are no externally validated pre-operative risk scores, but have there been any that have been developed? If so, this should be mentioned, ideally compared (validated) in a head-to-head comparison with their model. We apologise for the confusion we have created with our choice of words. What we meant was we have described the only published pre-operative risk score in general surgery (by Kheterpal et al. 2009), and additionally this score was also not externally validated. Thus, no other risk scores for predicting post-operative AKI have been developed in general surgery that we know of. We have now made this clearer in the manuscript (page13). Although the model by Kheterpal et al. predicts a different outcome than our model (only more severe AKI and over a longer time period), it would be interesting to assess whether this model also performs well for predicting our outcome. However, as a number of variables in the Kheterpal et al. model were not available in our database, and some (such as intraperitoneal surgery and ascites) not relevant in our orthopaedic setting, we were unable to perform a head-to-head comparison, which we have also clarified in the manuscript (page 14). Nevertheless, a possibility would be to use particular medications as proxies for the lacking comorbidity variables. This would not constitute an actual head-to-head comparison but more of a model updating exercise, validating the set of predictors from Kheterpal et al. to some extent. As this exercise would be rather labour-intensive, and we are not sure whether it would not be too farfetched and distract too much from the message of the article, we have chosen not to perform this analysis for now, but could of course upon request go ahead with it anyway. Additionally, we have added a paragraph on risk scores developed in other surgical subspecialty settings (page 13), to broaden the field and paint a more comprehensive picture. In this context, please also note our replies to reviewer 1 s comments 3 and 4c (pages 5 and 6 of this rebuttal). (page 5) Under Study Population, the eligibility criteria are brief - is that correct? If I want to use this model on my patients, then will what is written, i.e. aged 18years and older, who underwent orthopaedic surgery the only eligibility criteria needed to apply the model? The eligibility criteria are patients over 18 who underwent orthopaedic surgical procedures named in Appendix 1. We have added this for clarification (page 5). Candidate predictors (page 6), the candidate predictors were chosen based on the literature, are there any references for this? 11

12 We apologise for omitting the references and have now inserted them (page 7). multivariate used incorrectly, should be multivariable. Deciles is incorrectly used, should be tenths (see Thank you for pointing this out. We have corrected this. I don t understand in the methods (page 8; line 17), the model s stability was ensured using forward selection - no idea what this means and what it entailed. Does it mean they repeated the analysis using forward selection and the same predictors were identified? If so, this could be made clearer. We agree our choice of words was not entirely clear. This has now been clarified in the methods (page 8). Table 1, ideally should be reported by with and without outcome for both cohorts and overall for each cohort (as per TRIPOD recommendations) Table 1 has been amended as suggested. Figure 1 - should ideally be supplemented with loess regression line which is a smoothed regression line (using the calibrate or val.prob functions in the rms package which the authors are using). [see TRIPOD Statement E&E paper (annals.org/article.aspx?articleid= ). We have constructed the loess plots as recommended and the results are below: 12

13 0.6 1a: calibration development cohort 0.6 1b: calibration validation cohort observed probability observed probability predicted probability predicted probability 0.6 1c: After recalibration method d: After recalibration method 2 observed probability observed probability predicted probability predicted probability As the reviewer can see from the figures, even after recalibration the model overpredicts the risk in the highest few percent of patients, which was less pronounced from the tenths of risk scatter plot. We are concerned these figures enlarge this feature of the model disproportionally. Even though it only concerns a very small part of the population (2-3% of patients), because of the spread in predicted probabilities the diverging line segment is relatively big. Clinically, this is not problematic, as it would mean an over-prediction for the already high risk patients, only confirming they need extra attention. However, this may be hard to explain to the reader and could distract perhaps too much from the strengths of the model. We have kept the original figures in the manuscript for now, and would highly appreciate the reviewer s advice in this matter: should the loess plots be added to the calibration plots as suggested, or do they paint a distorted picture of the truth and should a different solution be chosen? Figure 2: ROC curves in the context of prediction models are uninformative, what is of interest is the area under the ROC curve, i.e. the c-statistic which measures the discrimination. If clinically relevant risks (from the model) are labelled on the curve so that sensitivity and specificity can be read off, then maybe there is some merit to the 13

14 curve, but I would recommend omitting this (this particular issue is discussed in the TRIPOD E&E paper). We have removed the ROC curve as suggested. I find the survival analysis a slight distraction and not sure on the relevance particular as the main aim of the paper is the risk score and the survival analysis doesn t use anything from the risk score. We included the survival analysis to stress the importance and clinical relevance of the outcome we are predicting, especially in the clinical context where we developed and validated our model. Long term survival data following AKI is limited. We feel that our data actually strengthen the paper by showing the consequences of failing to prevent AKI. We feel that illustrating this motivation adds to the paper and the relevance of the model. As the other reviewers seemed to agree with this approach, we have chosen to keep it in for now, and hope the reviewer can appreciate its value. What would add to the paper, is a clear presentation of the model, table 2, lists all the relevant regression coefficients, but putting this into an actual equation would aid presentation, with a clear description on how to use the model, including some hypothetical worked patients examples. Many thanks for this suggestion. We have included a worked example with computational details in appendix 2 and also describe the result of this example in our discussion (page 17). 14