Clinical Prediction Models of Patient and Graft Survival in Kidney Transplant Recipients: A Systematic Review and Validation Study

Transcription

1 Clinical Prediction Models of Patient and Graft Survival in Kidney Transplant Recipients: A Systematic Review and Validation Study by Sunita Kaur Shah Singh A thesis submitted in conformity with the requirements for the degree of Master of Science Institute of Health Policy, Management and Evaluation University of Toronto Copyright by Sunita Kaur Shah Singh 2016

2 Clinical Prediction Models of Patient and Graft Survival in Kidney Transplant Recipients: A Systematic Review and Validation Study Abstract Sunita Kaur Shah Singh Master of Science Institute of Health Policy, Management and Evaluation University of Toronto 2016 There have been an increasing number of clinical prediction models (CPM) of patient and graft survival in kidney transplant recipients (KTR). The performance of these models in Canadian KTR is unknown. The purpose of this thesis is: To systematically review CPM of patient and graft survival in KTR and externally validate CPM in a cohort of Canadian KTR. CPM were assessed for performance, bias risk and clinical usefulness. Selected CPM were externally validated in a cohort of 1,326 Canadian KTR. The results of this work shows that four of 18 studies that developed CPM were found to have a low risk of bias, were externally validated and clinically usable. CPM showed modest discrimination (c statistics: 0.67 to 0.74) and poor calibration (P values <0.001) in the validation cohort. This work highlights the importance external validation of CPM prior to implementation and using novel strategies to improve the performance of CPM. ii

3 Acknowledgments I would like to thank the members of my thesis committee for their guidance, support and mentorship. Dr. David Naimark has been an outstanding supervisor. I am grateful for his patience, insight and expertise, which have been invaluable in the completion of this work. I have greatly benefited from the knowledge of Charles Victor, during my MSc training and during the course of this thesis work. His clear, organized and thoughtful approach has been critical in my understanding of biostatistics and epidemiology. Dr. Joseph Kim has had a tremendous impact on my training to date. He has served as an outstanding mentor and role model over the past five years. I am inspired by his meticulous and rigorous approach to research, enthusiasm, creativity, and vision. I am grateful for his constant support, kindness and patience. I would also like to thank Ani Orchanian-Cheff from the UHN libraries and Segun Famure for his help with data management and database support. Lastly, I would like to thank my friends and family for their encouragement and support. In particular, I dedicate this thesis to my parents, Nicole and Surinder, who have always encouraged me to think critically, instilled in me the importance of hard work and supported me during the course of my training. iii

4 Table of Contents 1 Introduction The burden of chronic kidney disease Benefits of kidney transplantation for ESRD Ongoing challenges in kidney transplantation: Patient and graft survival Clinical prediction models in medicine: A brief overview Clinical prediction models in kidney transplant recipients Purpose of thesis work Systematic Review of Clinical Prediction Models of Patient and Graft Survival Background Methods Study design Study selection Data extraction Results Clinical prediction models of death with graft function in deceased donor kidney transplant recipients Clinical prediction models of graft failure in deceased donor kidney transplant recipients Clinical prediction models of graft failure in living donor kidney transplant recipients Discussion Validation Study in a Cohort of Canadian Kidney Transplant Recipients Background Validation of clinical prediction models iv

5 3.1.2 Rationale for external validation Considerations for external validation of a Cox proportional hazards model Methods Creation of the external validation cohort Model selection Predictor variables Outcome variables Statistical analysis Sensitivity analysis Results Characteristics of the external validation cohort Outcomes of interest Comparison of derivation and validation cohorts Performance measures: Discrimination Performance measures: Calibration Discussion Conclusions and future directions References Appendices Copyright acknowledgments v

6 List of Tables Table 1: Study characteristics of clinical prediction models of death with graft function in deceased donor kidney transplant recipients Table 2: Performance measures of clinical prediction models of death with graft function in deceased donor kidney transplant recipients Table 3: Risk of bias and clinical usefulness of clinical prediction models...18 Table 4 Study characteristics of clinical prediction models of graft failure in deceased donor kidney transplant recipients Table 5: Performance measures of clinical prediction models of graft failure in deceased donor kidney transplant recipients Table 6: Study characteristics of clinical prediction models of graft failure in living donor kidney transplant recipients...23 Table 7: Performance measures of clinical prediction models of graft failure in living donor kidney transplant recipients Table 8: Advantages and disadvantages of internal validation methods..30 Table 9: Variables from prediction models not captured in CoReTRIS database...38 Table 10: Characteristics of prediction models selected for external validation..42 Table 11: Baseline survival functions for clinical prediction models selected for external validation Table 12: Baseline characteristics of external validation cohort and derivation cohorts of models selected for external validation Table 13: Baseline characteristics of external validation cohort by donor subtype..51 vi

7 Table 14: Cumulative incidence (%) of death with graft function, total graft failure and deathcensored graft failure by donor subtype Table 15a: Number of patients experiencing death with graft function, death censored graft failure and loss to follow up in the first year after transplantation, by donor subtype.54 Table 15b: Number of patients experiencing death with graft function or death censored graft failure after one year post transplant, by donor subtype Table 16: External validation c statistics for death with graft function Table 17: External validation c statistics for total and death-censored graft failure 57 vii

8 List of Figures Figure 1: Adjusted hazard ratio of death of deceased donor kidney transplant recipients in comparison to waitlisted patients on dialysis.. 2 Figure 2: Relative number of clinical prediction models published over time Figure 3: Criteria for determining risk of bias and clinical usefulness of clinical prediction models Figure 4: Flow diagram of systematic review study selection...15 Figure 5: Comparison of patient survival between Canadian and US kidney transplant recipients Figure 6: 5- and 10- year graft survival rates by country Figure 7a: Model selection of clinical prediction models of death with graft function in recipients of deceased donor kidney transplant recipients...39 Figure 7b: Model selection of clinical prediction models of graft failure in recipients of deceased donor kidney transplant recipients..40 Figure 7c: Model selection of clinical prediction models of graft failure in recipients of living donor kidney transplant recipients Figure 8: Creation of external validation cohort Figure 9a: Cumulative incidence of death with graft function by donor subtype...52 Figure 9b: Cumulative incidence of total graft failure by donor subtype. 53 Figure 9c: Cumulative incidence of death-censored graft failure by donor subtype Figure 10a: Predicted vs. observed probabilities by risk quintile for death with graft function..58 viii

9 Figure 10b: Predicted vs. observed probabilities by risk quartile for total graft failure...59 Figure 10c: Predicted vs. observed probabilities by risk quartile for death-censored graft failure. 60 ix

10 List of Appendices Appendix 1: Glossary of terms. 78 Appendix 2a: Search strategy of clinical prediction models of patient and graft survival in Ovid MEDLINE.. 79 Appendix 2b: Search strategy of clinical prediction models of patient and graft survival in Ovid MEDLINE in-process and non-indexed citations.. 80 Appendix 2c: Search strategy of clinical prediction models of patient and graft survival in Embase...81 Appendix 3a: Predictor variables included in clinical prediction models of death with graft function in deceased donor kidney transplant recipients...82 Appendix 3b: Predictor variables included in clinical prediction models of graft failure in deceased donor kidney transplant recipients..83 Appendix 3c: Predictor variables included in clinical prediction models of graft failure in living donor kidney transplant recipients Appendix 4: Baseline characteristics of patients in development cohort of clinical prediction models of death with graft function...86 Appendix 5: Baseline characteristics of patients in development cohort of clinical prediction models of total and death-censored graft failure...87 Appendix 6a: Cumulative incidence curves for death with graft function (DWGF) for validation 1 cohort, stratified by quintiles of predicted probabilities.88 Appendix 6b: Cumulative incidence curves for total graft failure (TGF) for validation 1 cohort, stratified by quintiles of predicted probabilities 89 x

11 Appendix 6c: Cumulative incidence curves for death-censored graft failure (DCGF) for validation 1 cohort, stratified by quintiles of predicted probabilities 90 Appendix 7: Notification of University Health Network (UHN) REB Approval 91 Appendix 8: Notification of University of Toronto REB Approval.92 xi

12 1 Chapter 1 1 Introduction 1.1 The burden of chronic kidney disease Chronic kidney disease (CKD), defined as the presence of abnormalities of kidney structure or function for a period of greater than 3 months (1), is one of the most important public health concerns in Canada and worldwide (2-6). A study evaluating the prevalence of CKD in Canada between 2007 and 2009 estimated that approximately 3 million Canadian adults, or 12.5% of the population, are living with CKD (2). There are many causes of CKD, however a large proportion of cases can be attributed to the aging Canadian population, as well as to the increase in the prevalence of obesity and diabetes over the past several decades (1, 7, 8). It is well recognized that CKD is associated with an increased risk of cardiovascular disease and death, as well as progression to kidney failure or end-stage renal disease (ESRD) (9-12). Among those with CKD, only a small subset (i.e., less than 1%) will progress to ESRD (2, 7). However, the development of ESRD results in multiple complications including dysregulation of calcium and phosphate balance (13), left ventricular dysfunction and anemia (14, 15), as well as an increased predisposition to infection (16, 17), hospitalization (18) and cardiovascular disease (19). As a result, patients with ESRD suffer from a significant decrease in their quality of life as well as a marked increase in morbidity and mortality in comparison to their counterparts without ESRD (20-23). Furthermore, although the proportion of patients with ESRD is small relative to the number of patients with CKD, the presence of ESRD and its sequelae results in significant direct and indirect costs to the health care system (24). At present, the incidence of ESRD has reached a plateau, however, the prevalence of ESRD continues to rise with over 40,000 patients currently living with ESRD in Canada, of which over 23,000 are on dialysis (25). 1.2 Benefits of kidney transplantation for ESRD The treatment options for patients with ESRD include renal replacement therapy in the form of dialysis (i.e., hemodialysis or peritoneal dialysis) or kidney transplantation. It has now been well established that kidney transplantation is the renal replacement therapy of choice for patients with ESRD (26, 27). Wolfe et al. conducted a landmark retrospective cohort study comparing the

13 2 relative hazard of death of 23,275 recipients of deceased donor kidney transplants to 46,164 patients who remained on the waiting list for transplantation. Although the risk of death was higher among transplant recipients in the early period post-transplantation, kidney transplantation was associated with a significant survival advantage in comparison to remaining on dialysis over the long-term after adjustment of important confounders (Figure 1) (27). Figure 1: Adjusted hazard ratio of death of deceased donor kidney transplant recipients in comparison to waitlisted patients on dialysis Relative Risk of Death Risk equal Survival equal Days since Transplantation Patients on maintenance dialysis (referent group) are represented by the dashed horizontal line. Reproduced with permission from Wolfe et al. Comparison of mortality in all patients on dialysis, patients on dialysis awaiting transplantation, and recipients of a first cadaveric transplant. N Engl J Med. 1999;341(23): Copyright Massachusetts Medical Society. (27) Furthermore, in comparison to remaining on maintenance dialysis, kidney transplantation has also been shown to improve quality of life, emotional and psychosocial wellbeing and permits

14 3 the return to the workforce in certain subsets of transplant recipients (26, 28, 29). Finally, in addition to the significant benefits to patients, the cost savings to the Canadian health care system are notable. It is estimated that in Canada, kidney transplantation results in savings of $250,000 per-patient over a 5-year period of time in comparison to remaining on dialysis (30). This estimate accounts for the cost associated with the procedure itself (i.e., hospitalization, donor and recipient surgeries, induction immunosuppression etc.). A number of other studies have confirmed the benefit of both living and deceased donor kidney transplantation (Appendix 1) on patient survival and quality of life (31-34). Typically, patients with ESRD will first initiate dialysis and subsequently undergo kidney transplantation if they are deemed medically eligible, although a subset of patients undergo the preferred approach of preemptive kidney transplantation (Appendix 1) prior to the initiation of dialysis (35). Given the ongoing gap between the demand and supply of deceased donor kidneys for transplantation, living donor kidney transplantation has become a more common transplant option for patients with ESRD over the past decade (25). In fact, for individuals with ESRD who have identified a suitable living donor, this approach is encouraged for a number of reasons. First, graft survival is generally superior in recipients of living donor kidney transplants in comparison to deceased donor kidney transplants (36-39). In Canada, 1- and 5-year graft survival for living donor transplant recipients is 97.5% and 90.9% respectively, in comparison to 93.3% and 83.2% in recipients of deceased donor transplants (25). Second, the probability of post-operative complications such as primary non-function (PNF) and delayed graft function (DGF) (Appendix 1) is less likely in the setting of living donor kidney transplantation. DGF and PNF is an uncommon occurrence in recipients of living donor kidney transplants (i.e., less than 5 percent of cases), in contrast to DGF rates of 20 to 40% in recipients of deceased donor kidney transplants. Furthermore, the occurrence of these complications is not insignificant, potentially resulting in early or late graft loss, an increased risk of acute rejection, and death (40-42). Third, if a living donor can be identified and evaluated in a timely fashion, patients with ESRD may benefit from pre-emptive transplantation. As a result, these patients may avoid the potential deterioration in health and the increased risk of death that may occur while waiting on dialysis for a kidney transplant (43). At the population-level, living donor kidney transplantation results in an increase in the pool of deceased donor kidneys and a decrease in waiting time for individuals with ESRD who do not have a suitable living donor.

15 4 1.3 Ongoing challenges in kidney transplantation: Patient and graft survival Despite the overall survival benefit conferred by transplantation, kidney transplant recipients still have a significantly higher mortality than their age-matched counterparts without kidney disease (44). While overall patient survival has improved over the past several decades in kidney transplant recipients, death accounts for over half of kidney transplant failures (45, 46). In particular, cardiovascular disease remains the leading cause of death with a functioning graft. It has been hypothesized that kidney transplantation, while improving the cardiovascular risk profile of recipients, cannot completely reverse the burden of disease accumulated during the development of ESRD. Therefore, death from cardiovascular disease continues to be an important problem (44). Furthermore, the need for life-long maintenance immunosuppression after kidney transplantation increases the risk of death from infection or malignancy (47, 48). Once kidney transplantation is performed, one of the primary goals of post-transplant care is to maximize allograft survival. Although there has been substantial improvement in short-term graft survival, late graft failure (defined as graft loss beyond 1-year post-transplantation) continues to be an important problem (49). Graft loss can occur for a number of reasons including acute rejection, chronic rejection (also referred to as transplant glomerulopathy), recurrence in the allograft of the original disease causing ESRD, drug toxicity and viral infections (e.g., polyoma nephrophathy) (50). At the patient level, graft failure results in the return to chronic dialysis or the need for re-transplantation. Once this occurs, the survival advantage offered by transplantation is lost (51). At a population level, given the ongoing shortage of organs, increasing graft survival time maximizes the use of a scarce resource especially in countries such as Canada where the median waiting time for a kidney transplant is 3.5 years (25). The importance of improving patient and graft survival in kidney transplant recipients is clear but remains a significant challenge for the nephrologist. Although multiple studies have identified individual risk factors influencing patient and graft survival, our understanding of how individual risk factors interact to affect long-term outcomes is limited. A major challenge in improving long-term patient and graft survival is the lack of tools to identify, at an early stage, those at risk for mortality and graft failure.

16 5 1.4 Clinical prediction models in medicine: A brief overview Prediction or forecasting of a future event has been widely used for decades in many areas outside of medicine such as economics or meteorology. However, until recently, prediction research in medicine has received little attention in the literature (52). Clinicians commonly estimate the risk or the probability of an outcome after integrating data from a clinical history, physical examination and/or a constellation of investigations. In combination with the experience of the clinician, the risk of an outcome in an individual, such as death for example, is estimated. However, clinical prediction can be challenging and inaccurate. As a result, clinical prediction models have emerged in the medical literature, especially over the past decade, in order to provide physicians with tools to guide estimation of risk at the patient level (Figure 2) (53). Figure 2: Relative number of clinical prediction models published over time Relative number of studies "prognostic model" or "prediction model" Year of publication Reproduced with permission from Steyerberg, E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating Copyright Springer-Verlag New York. (53)

17 6 Clinical prediction models, also referred to as prognostic models, decision rules, risk scores, nomograms or prediction rules, have become important and useful tools for clinical decision making. One of the most commonly used clinical prediction models in medicine is the Framingham risk score which was developed to estimate the probability of cardiovascular disease using clinical and laboratory data in asymptomatic individuals in order to implement preventive measures based on the estimated risk profile of an individual (54). Other well-known clinical prediction models include the MELD score (55), which estimates the mortality risk in patients with end-stage liver disease and the CHADS2 score which estimates the risk of stroke in patients with atrial fibrillation to guide decision-making regarding the initiation and type of anticoagulation (56). Clinical prediction models traditionally include a combination of demographic, laboratory or physical exam data into a multivariable regression model to predict the occurrence of disease or an outcome of interest (53). Regression models are also used for etiologic research, however, prediction models and etiological models differ in various ways. First, the purpose of an etiological model is to provide an estimate of the relative risk of a particular exposure or risk factor on the outcome of interest after accounting for confounders. In contrast, prediction models integrate available data to estimate the absolute risk of an event or disease occurring, most commonly expressed as a probability. Second, in etiological models, variables included in the multivariable regression model are generally known to be causally related (either based on biological plausibility or prior research findings) to the outcome of interest. However, in prognostic models, variables included in the model do not necessarily need to be causally associated with the outcome. Third, the performance of a prediction model is evaluated using measures of calibration and discrimination, which are not evaluated in etiological models. Finally, a major objective of prediction research is to derive a model that is generalizable and performs well in an external population, whereas etiological research uses the data available to best estimate a measure of association, which may not be reproducible in another population (52, 57-59). Generally, after a clinical prediction model has been developed in a population of patients, the model should be tested, or validated, in a separate cohort of patients to demonstrate performance and portability of a prediction model. Ideally, impact studies should also be performed to

18 7 determine if integration of a clinical prediction model into practice results in an improvement in patient outcome (52, 58-60). 1.5 Clinical prediction models in kidney transplant recipients The ability to predict patient and graft survival is of significant value to the transplant nephrologist. Patients identified to be at high risk for mortality and graft failure may warrant closer monitoring or intervention. For example, given the known importance of immunologic injury in chronic allograft dysfunction, titration of immunosuppression may be appropriate in individuals at high-risk for graft failure (61). A patient at high-risk for post transplant mortality may be more likely to undergo investigations for ischemic heart disease without waiting for clinical manifestations. Furthermore, identification of these high-risk individuals will allow for further prospective study of the potential benefit of these interventions on patient outcomes (62, 63). Most importantly, patients and their families can be informed of the likelihood of graft loss so the initiation of chronic dialysis or, ideally, re-transplantation can be appropriately planned. Kidney transplant recipients sufficiently differ from the general and chronic kidney disease populations to limit the usefulness of clinical tools for risk prediction from the latter cohorts (64, 65). For example, it has been shown that the Framingham risk score performs poorly in predicting cardiovascular mortality among kidney transplant recipients (66). To address the shortcomings of applying models from non-transplant populations to transplant patients, a number of clinical prediction models developed in cohorts of kidney transplant recipients have been published in the literature. However, their clinical use has been limited due to several drawbacks. First, the performance of existing clinical prediction models is modest and many have not been externally validated. External validation of a clinical prediction model is essential, as it demonstrates that a prediction model can be applied in a population other than the one from which it was derived (58). Secondly, many clinical prediction models have been derived from large US registries. The generalizability of these models in the Canadian context is unclear due to the differences in patient demographics, listing practices, access to transplantation, delivery of health care, and post-transplant mortality in Canada vs. the US (67). Finally, several clinical prediction models include a number of variables that are often not readily or commonly captured in clinical or research databases.

19 8 1.6 Purpose of this thesis The purpose of this thesis is twofold. The number of published clinical prediction models of patient and graft survival far exceeds the number of models routinely utilized by clinicians as a risk stratification tool in kidney transplant recipients. However, identification of individuals at increased risk for these events is a critical step in improving transplant outcomes. A better understanding of the clinical prediction models that currently exist can allow clinicians to determine how to best utilize these tools in the kidney transplant population, determine whether new models are needed or if existing models should be refined prior to clinical use. Therefore, the first objective of this thesis is to systematically review existing clinical prediction models of patient and graft survival in kidney transplant recipients in order to evaluate their strengths, limitations, performance metrics, potential for bias and clinical usefulness. Amongst the existing clinical prediction models published in the literature, external validation is not consistently performed and, in particular, is rarely performed in Canadian kidney transplant recipients. As a result, it is unclear if clinical prediction models perform well in this population. Due to the notable differences in the delivery of health care, demographics of recipients, access to transplantation and transplant outcomes by country, external validation is an important step prior to implementation of a model into clinical practice. Therefore, the second objective of this thesis is to externally validate selected clinical prediction models of patient and graft survival in a contemporary cohort of Canadian kidney transplant recipients. This will be done using a single-centre cohort of kidney transplant recipients from Toronto General Hospital.

20 9 Chapter 2 2 Systematic Review of Clinical Prediction Models of Patient and Graft Survival 2.1 Background The development of clinical prediction models in kidney transplant recipients has become increasingly more common in order to assist clinicians in personalizing the care of the transplant patient (68). There are several advantages to using prediction models. First, predictions made using clinical judgment alone are often imprecise, inaccurate and highly variable amongst physicians (69, 70). In contrast, a clinical prediction model will yield the same probability of an event, irrespective of the skill or experience of the user (69, 71). Second, the human brain is only capable of integrating a limited number of variables. Thus, if many variables are predictive of an outcome, a clinical prediction model can better integrate multiple data elements in order to yield a consistent estimated probability of an outcome (69). Although it is not known if clinical prediction models improve predictions in kidney transplant recipients as compared to physician judgment, there are a number of prediction models in the non-transplant setting that have been shown to predict outcomes better than clinical judgment alone. (72-74). For example, Bandiera et al. evaluated the performance of the Canadian C-spine rule versus physician judgment in predicting the probability of a cervical spine fracture. This study demonstrated that first, the interobserver agreement was fair when clinical judgment was used, and second, that the Canadian C-spine rule was superior at detecting clinically important injuries as compared to clinical judgment (73). However, despite its advantages, the integration of prediction models into clinical use has been limited. Various reasons may explain why adoption of these models has been poor to date. It has been hypothesized that some clinicians may not fully appreciate the additional value of a prediction model, or alternatively, clinicians may simply trust their own clinical judgment over prediction model estimates. In addition, many models are not user friendly or parsimonious, and as a result are cumbersome to use at the bedside. Finally, one of the major factors that may explain why clinicians are not using clinical prediction models is that, for a specific problem, there are simply too many clinical prediction models available, thus making selection of the best

21 10 model challenging. Furthermore, the differences in the methodologies used across various models add an additional level of complexity in selecting the best model for use (69, 75, 76). Amongst clinical prediction models derived in the kidney transplant population, differences in study design and analysis are common. For example, clinical prediction models have been developed using single centre databases as well as large transplant registries. Multivariable regression models, as well as more novel machine learning techniques, have been used for model development. The time point after kidney transplantation at which the model can be used also varies, as does the number and type of variables included in the final model. Finally, not all models are externally validated, thereby limiting generalizability to other populations. A comparison and critical appraisal of available clinical prediction models published in the literature would be useful in order to guide clinicians in selecting the best and most appropriate model to use in the care of kidney transplant recipients. Therefore, the purpose of this study is to systematically review existing clinical prediction models of patient and graft survival in kidney transplant recipients, summarize their performance, evaluate the studies for potential areas of bias and assess their clinical usefulness. 2.2 Methods Study design Both the Embase and Ovid MEDLINE (including Ovid MEDLINE in-progress) databases were searched. References of selected articles and review articles identified were also hand searched to identify any potential clinical prediction models that should be included in the systematic review. A librarian at the University Health Network conducted the search. The timeframe for the search was from January 1 st, 1990 to February 4 th, The start date of 1990 was selected as prediction models started to appreciably emerge in the medical literature after this time (Figure 2) (53). Furthermore, patient and graft outcomes of kidney transplant recipients prior to 1990 differed significantly to that of the modern era of transplantation, and thus, a bibliographic review prior to 1990 was not conducted. The search included clinical prediction models developed using statistical methods (e.g., regression modeling) as well as alternative machine learning methods such as artificial neural

22 11 networks (ANN) and classification and regression tree (CART) methods. The search included listings in all languages but was limited to human subjects. This systematic review included clinical prediction models of the following outcomes of interest: (1) Death with graft function, defined as death from any cause that occurs in a patient with a functioning graft (2) Total graft failure, defined as death with graft function, return to chronic dialysis or pre-emptive re-transplantation (3) Death-censored graft failure, defined as the return to chronic dialysis or pre-emptive re-transplantation Evidence-based filters developed to maximize sensitivity when searching for clinical prediction models in Embase and Ovid MEDLINE were used (77, 78). Citations were imported into EndNote reference manager (Version 7.0.1). The complete search strategy is shown in Appendix 2a, 2b and 2c Study selection Citations imported into EndNote were screened based on the presence of the following key words in the title or abstract: score, risk, rule(s), decision rule prediction, prediction model, model, nomogram, tool or probability. Key words were selected after careful screening of a number clinical prediction models in the literature and reviewed with a librarian and panel of experts. Retained citations subsequently underwent full-text review. If there was any uncertainty, the citation was retained for full-text review. Studies were eligible for inclusion if the prediction model was derived in a cohort of at least 100 patients and included at minimum 3 variables in the model. Studies were also only eligible for inclusion if they reported at least one traditional measure of model performance, such as discrimination or calibration Data extraction Detailed data were extracted from all the studies by Sunita Singh. A standardized data extraction sheet was used. The following data elements were extracted where available for each study included in the systematic review: (1) The study population in which the prediction model was

23 12 developed, including the proportion of patients in the cohort that were recipients of living donor kidney transplants; (2) The timeframe when the cohort was assembled; (3) The database used for derivation of the model (i.e., single centre database vs. national registry); (4) The total number of patients in the cohort; (5) The outcome of interest (i.e., death with graft function, total graft failure or death-censored graft failure) and the number of events; (6) The number of clinical prediction models derived; (7) The number of predictors included in the models; (8) The duration of follow-up and (9) The time of assessment (i.e., the time point at which the clinical prediction model can be used for risk estimation in a clinical setting). Reported measures of model performance such as discrimination, calibration and reclassification were also extracted. Discrimination is typically reported as a c statistic and indicates how well a model can differentiate those with and without the event of interest. This rank statistic, in the setting of a survival analysis, represents the proportion of concordant pairs among all evaluable pairs. A concordant pair is one in which the subject with the higher predicted risk had the shorter observation time. Generally, a c statistic of 0.70 to 0.79 indicates modest discrimination, a c statistic of 0.80 to 0.89 indicates good discrimination and a c statistic of 0.90 to 1.00 indicates excellent discrimination (53, 79). Calibration refers to the agreement between predicted probabilities derived by a prediction model and the observed probabilities of an event. A model that calibrates well is one where predicted and observed probabilities are similar. Calibration can be assessed both graphically as well as statistically (80), where a non-statistically significant result indicates a model that calibrates well (i.e., good agreement between predicted and observed probabilities). Newer measures of model performance, such as the net reclassification index, were also extracted. Reclassification refers to the movement of patients from one risk category to another when an additional predictor is added to an existing set of predictors in a model. Ideally, a new predictor will result in the movement of patients to higher risk categories among those experiencing the event of interest and movement to lower risk categories among those who do not experience the event of interest (79, 81). The clinical prediction models included in the systematic review were evaluated for risk of bias and clinical usefulness based on a framework developed by Tangri et al., (82). This framework was used in a systematic review of clinical prediction models of mortality, cardiovascular events and risk of ESRD in patients with CKD and adapted from Hayden et al. (83). The following domains were assessed: (1) study participation; (2) study attrition; (3) prognostic factor selection;

24 13 (4) prognostic factor measurement; (5) outcome measurement; (6) statistical analysis; (7) reporting of model performance; (8) clinical utility and (9) clinical usability. The areas considered in the evaluation of each of these domains are showed in Figure 3. Finally, the prediction models were evaluated to determine if internal and/or external validation was performed and performance measures were reported. Figure 3: Criteria for determining risk of bias and clinical usefulness of clinical prediction models Potential Bias Risk of bias Study participation Study attrition Prognostic factor selection Prognostic factor measurement Outcome measurement Statistical analysis Reporting of model performance Usefulness Clinical utility Clinical usability Areas to Be Considered The study population is adequately described for key characteristics. The sampling frame and recruitment are adequately described, possibly including methods to identify the study sample and the period and place of recruitment. Inclusion and exclusion criteria are adequately described. There is adequate participation in the study by eligible individuals. The baseline study sample is adequately described for key characteristics. The response rate (i.e., the proportion of the study sample completing the study and providing outcome data) is adequate. Attempts to collect information on participants who withdrew from the study are described. Reasons for loss to follow-up are provided. Participants lost to follow-up are adequately described for key characteristics. There are no important differences in key characteristics and outcomes between participants who completed the study and those who did not. There are clear definitions of all candidate predictors, including how and when they were measured. The selection method for the final predictors in the model is described. The method of measurement is given for each prognostic factor. A description of how each prognostic factor was handled in model development is given (i.e., continuous vs. discrete). A clear definition of each outcome of interest is given, including ascertainment, duration of follow-up, and level and extent of the outcome construct. The primary modeling method is clearly described (e.g., logistic regression or survival analysis) and appropriate for the type of data. Alternative methods are discussed when appropriate (competing risk models or multiple events models). Discrimination, calibration, and goodness of fit are reported. Reclassification is reported when models are compared with simpler models or standard of care. The rationale for choosing risk categories for reclassification analyses is well-described. There is discussion about a threshold risk or risk category above which a diagnostic or therapeutic decision is made. There are clinical trial data to suggest that using the model improves processes of care or outcomes (preferred). There is an easy-to-use risk calculator or nomogram that facilitates risk prediction at the bedside. There is an electronic decision aid enabling calculation via a smartphone or Web-based calculator (preferred). * Partially adapted from reference 13. Items should be evaluated as low risk ( ), high risk ( ), or unknown risk (?). Items should be evaluated as yes ( ) or no ( ). Reproduced with permission from Tangri, N et al. Risk Prediction Models for Patients With Chronic Kidney Disease: A Systematic Review. Ann Intern Med. 2013:158: Copyright The American College of Physicians. (82)

25 Results The study flow diagram for the systematic review as per the PRISMA statement (84) is shown in Figure 4. There were 9,045 records identified by EMBASE, 7,809 records identified by Ovid MEDLINE and 1 record identified by hand searching. After exclusion of 2,134 duplicate citations, there were 14,001 citations that were screened, of which 13,706 were excluded. There were 295 full-text articles reviewed for inclusion in the systematic review. After review, 277 articles were excluded for the following reasons: (1) non-english (Arabic) language article due to lack of resources for translation into English (n=1); (2) duplicate publication (n=43); (3) review or editorial article (n=38); (4) study where no clinical prediction model was derived (i.e., single exposure-outcome study or score with no reported performance measures) (n=42); (5) clinical prediction model of outcome other than patient or graft survival (n=46); (6) validation study of an existing clinical prediction model (n=25); (7) prediction model using histological or echocardiographic data only (n=13); (8) prediction model derived for kidney allocation purposes only (n=5); (9) decision tree for clinical decision making (not derived from a regression model or machine learning method) (n=1); (10) clinical prediction model derived from a cohort of less than 100 patients (n=9); (11) clinical prediction model with less than 3 predictors included in the model (n=38); (12) clinical prediction model derived using logistic regression, which is an inappropriate regression method given that all outcomes of interest are time-to-event outcomes (n=6); (13) nested model (i.e., a clinical prediction model derived from a subset of patients where another clinical prediction model was derived and included in the systematic review) (n=1) and (14) clinical prediction model derived in a non-kidney transplant population (n=9). There were 18 studies included in the systematic review. Five studies derived clinical prediction models of death with graft function and 11 studies derived clinical prediction models of graft failure (total and/or death-censored) in recipients of deceased donor kidney transplants. Among these studies two studies derived prediction models of death with graft function and graft failure in the same publication. Finally, two studies derived clinical prediction models of graft failure in recipients of living donor kidney transplants only. There were no clinical prediction models of death with graft function derived in a population of exclusively living donor kidney transplant recipients.

26 15 Figure 4: Flow diagram of systematic review study selection Records identified by Ovid Medline (n=7,089) Records identified by EMBASE (n=9,045) Records identified by hand searching (n=1) Duplicate citations excluded (n=2,134) Citations screened (n= 14,001) Citations excluded (n= 13,706) Full-text articles reviewed for eligibility (n= 295) Full-text articles excluded (n=277) Not English language: n=1 Duplicate publication: n=43 Review or editorial: n=38 No prediction model: n=42 Other outcomes: n=46 Validation study: n=25 Echo or histological based score: n=13 Allocation model: n=5 Clinical decisional tree: n=1 Less than 100 patients: n=9 Less than 3 predictors: n=38 Logistic regression model: n=6 Nested model: n=1 Non-kidney transplant population: n=9 Studies included in systematic review (n=18)

27 Clinical prediction models of death with graft function in deceased donor kidney transplant recipients Among the 18 studies included in the systematic review, there were 5 studies identified which developed clinical prediction models of death with graft function in a population of primarily deceased donor kidney transplant recipients (85-89). The characteristics of these studies are shown in Table 1. One study included recipients of kidney-pancreas transplants in addition to kidney-only transplant recipients (89). The proportion of living donor kidney transplant recipients was reported in four of the five studies, and consisted of 0 to 22% of the total study cohort. Two studies utilized the United States Renal Data System for derivation of their prediction models and thus, included a large number of patients (i.e., 25,270 to 57,389 patients) (87, 88). The other three studies derived prediction models in smaller cohorts of Spanish transplant recipients or in a cohort of patients included from a multi-centre randomized controlled trial evaluating the use of fluvastatin on cardiovascular events and mortality in kidney transplant recipients (range 1,293-4,928 patients) (89). All prediction models were derived using multivariable regression models with the exception of one study, which used both Cox proportional hazards models as well artificial neural networks (87). Each study developed between 1 and 20 models of death with graft function, and the number of variables included in the models ranged from 6 to 23 variables. The variables included in the models are shown in Appendix 3a. Follow up time was reported in three of the five studies, and ranged from 4 and 6.8 years. Models were derived to be used at various time points: Two models were to be used at the time of transplantation, the other three models were to be used at the time of hospital discharge, at 6 months post transplant or at 1 year post transplant. The performance measures of the clinical prediction models of death with graft function are shown in Table 2. All models reported discrimination and calibration, while none reported reclassification. Although all models calibrated well, discrimination was modest with c statistics of 0.69 to 0.80 in the models derived using regression models. The models derived using artificial neural networks had c statistics of depending on the outcome (i.e., death with graft function at 1-,3-,5- or 7-years post-transplant) All models were internally validated, but none were validated externally.

28 17 Table 3 summarizes the assessment of the risk of bias and clinical usefulness of the studies included in the systematic review. One study was identified as having a high risk of bias, since this model included a large number of comorbidities as predictor variables that were likely subject to significant measurement error (88). Most studies adequately reported methods of prognostic factor selection and measurement as well as outcome measurement. The analysis was appropriate in all studies, as was the reporting of traditional measures of model performance (e.g., discrimination and calibration). However, four of the five studies did not report study attrition (e.g., loss to follow-up, response rate) and two of the five studies did not comprehensively report study participation (e.g., inclusion and exclusion criteria). When clinical utility was assessed, only one study discussed a threshold for risk based on the results of a prediction model (85), while three models were deemed clinically usable (i.e., a nomogram or online calculator for risk estimation was developed) (85, 86, 89). Table 1: Study characteristics of clinical prediction models of death with graft function in deceased donor kidney transplant recipients (85-89)! Study Machnicki et al Hernandez et al Soveri et al Lin et al. 2008* Hernandez et al Population and timeframe Adult, primary kidney transplant recipients Kidney transplant recipients Kidney and kidney pancreas transplant recipients Kidney transplant recipients Adult, single kidney transplant recipients alive at 1 year post transplant % Living Donor Database Patients, n 0 USRDS 25,270 NR 22 University Hospital of the Canary Islands Database, Spain ALERT RCT Dataset Events, n 4,689 (Total deaths) 3284 (DWGF) Number of models, n Number of predictors, n 1,293 NR 1 8 2, Follow-up Time of assessment NR At transplant Median 4 years (IQR years) Mean 6.7 years Post transplant (At hospital discharge) At minimum 6 months post transplant NR USRDS 57,389 8, NR At transplant transplant centres in Spain 4, Mean 6.8 (IQR 4-8.6) 1 year post transplant *Denotes the use of machine learning techniques and Cox proportional hazards model for development of prediction model. NR=Not reported; IQR=Interquartile range; USRDS=United States Renal Data System; ALERT=Assessment of Lescol in Renal Transplantation (90); RCT=Randomized controlled trial; DWGF=Death with graft function.

29 18 Table 2: Performance measures of clinical prediction models of death with graft function in deceased donor kidney transplant recipients (85-89) Study Discrimination Calibration Reclassification Validation Machnicki et al Hernandez et al Soveri et al Lin et al Hernandez et al Yes No Internal 0.69 (derivation); 0.60 (validation) Yes No Internal 0.73 (derivation); 0.72 (validation) Yes No Internal Yes No Internal 0.75 (derivation); 0.74(validation) Yes No Internal Yes=Calibration assessed; No=Reclassification not assessed Table 3: Risk of bias and clinical usefulness of clinical prediction models (85-89, )! Study Machnicki et al Hernandez et al Soveri et al Lin et al Hernandez et al Tiong et al Akl et al Rao et al Kasiske et al Foucher et al Goldfarb-Rumyantev et al Moore et al Krikov et al Brown et al Shabbir et al Watson et al Schnitzler et al Thorogood et al Study Participation Study Attrition Prognostic Factor Selection Prognostic Factor Measurement Outcome Measurement Analysis Reporting of Model Performance Clinical Utility Clinical Usability Low? Low High Low Low Low No No?? Low Low Low Low Low Yes Yes Low Low Low Low Low Low Low No Yes??? Low Low Low Low No No Low? Low Low Low Low Low No Yes Low? Low Low Low Low Low No Yes???? Low Low Low No Yes Low? Low Low Low Low? No Yes Low? Low Low Low Low Low No Yes Low Low Low Low Low Low? Yes Yes Low Low Low Low Low? Low No No Low? Low Low Low Low Low No Yes Low? Low Low Low Low Low No No High? Low Low Low Low? No No Low Low Low Low Low Low Low No Yes Low? Low Low Low Low? Yes No Low? Low Low Low Low? No No High? Low Low? Low? No No Low= Low risk of bias;?=unknown risk of bias; High=High risk of bias

30 Clinical prediction models of graft failure in deceased donor kidney transplant recipients Among the 18 studies included in the systematic review, there were 13 studies identified which developed clinical prediction models of graft failure in a population of primarily deceased donor kidney transplant recipients. The characteristics of these studies are shown in Table 4. Six studies derived models for total graft failure (88, 93, 94, 99, 101, 102), whereas four studies developed models of death-censored graft failure (87, 95, 96, 98). Two models derived models for both outcomes (97, 100), and one study did not specify the outcome in the manuscript (103). Two studies included recipients of kidney-pancreas transplants in addition to kidney-only transplant recipients (96, 98). The proportion of living donor recipients was reported in eleven of the thirteen studies, and consisted of 0 to 53% of the total study cohort. Eight studies utilized large US registries (United States Renal Data System or Scientific Registry of Transplant Recipients) for derivation of their prediction models. The sample sizes for these studies ranged from 7,348 to 92,844 patients. The other studies used data from European transplant recipients (UK and France) with sample sizes of 651 to 8,154 patients. Four prediction models were derived using artificial neural networks or tree based models (87, 96, 98, 99). There were between 1 and 20 models developed in each study, and the number of variables included in the models ranged from 6 to 42 variables. The variables included in the models are shown in Appendix 3b. Follow up time was reported in four studies, and ranged from 3 to 5.1 years. Models were derived to be used at various time points: Eight models were to be used at the time of transplantation, four models were to be used at 1-year post transplant and one model was to be used at the time of transplantation as well as at 7-days and 1-year posttransplant. The performance measures of the clinical prediction models of death with graft function are shown in Table 5. All studies, with the exception of two, reported discrimination, and nine of the 13 studies reported calibration. All models calibrated well. Discrimination was variable, with c statistics of 0.61 to 0.90 where regression models were used for analysis and 0.59 to 0.82 where machine learning methods were used. Seven of the 13 models were externally validated.

31 20 Three studies were identified as having a high risk of bias based on the description of the study population used to derive the model. One study was noted to have a high risk of bias, as this model included a large number of comorbidities as predictor variables, which were likely subject to significant measurement error (88). One study derived a model in a population of transplant recipients assembled between 1984 and 1987, which is an era of transplantation where there were marked differences in outcomes and immunosuppressive agents when compared to the current era of transplantation (103). The other study excluded a large number of transplant recipients from the cohort where outcome data was missing (99). Baseline characteristics of the excluded patients were not provided. Studies were found to have a low risk of bias with respect to prognostic factor selection and measurement, outcome measurement and analysis. Similarly to models developed for death with graft function, study attrition was poorly reported limiting bias assessment. Furthermore, six models only reported one traditional measure of model performance (i.e., either discrimination or calibration). When clinical utility was assessed, only two studies discussed a threshold for risk based on the results of a prediction model while five studies were deemed clinically usable (i.e., a nomogram or online calculator for risk estimation was developed).

32 21 Table 4: Study characteristics of clinical prediction models of graft failure in deceased donor kidney transplant recipients (87, 88, ) Study Rao et al TGF Machnicki et al TGF Kasiske, et al TGF Foucher et al DCGF Goldfarb- Rumyantev et al. 2003* Population and timeframe Adult, primary kidney transplant recipients Adult, primary kidney transplant recipients Adult, kidney transplant recipients Adult kidney transplant recipients alive at 1 year post transplant Kidney and kidneypancreas transplant recipients % Living Donor Database Patients, n Events, n Number of models, n Number of predictors, n Follow-up Time of assessment 0 SRTR 69,440 19, NR At transplant 0 USRDS 25,270 7, NR At transplant 0 USRDS 0 DIVAT (France) 43,743, 57, 603 & 59,091 NR 3 11, 8 & 6 NR 2,169 NR 1 8 Mean 5.1 years (SD 2.7 years) At transplant, 7 days post, & 1 year post transplant 1 year post transplant 0 SRTR 37,407 NR 2 22 & 17 Mean 3 years At transplant! DCGF Moore et al TGF, DCGF Adult, kidney transplant recipients with functioning graft at 1 year LOTESS study database (UK) 2, graft failures, 196 deaths 2 6 & 7 Mean 4 years (SD year post transplant years)! Krikov et al. 2007^ DCGF Brown et al. 2012^ TGF Shabbir et al TGF, DCGF Watson et al TGF Schnitzler et al TGF Lin et al. 2008* DCGF Thorogood et al. 1991? Kidney or kidneypancreas transplant recipients Adult, primary kidney transplant recipients Adult kidney transplant recipients Adult kidney transplant recipients Kidney transplant recipients alive at 1 year post transplant Kidney transplant recipients Kidney transplant recipients USRDS & UNOS 92,844 32, NR At transplant 0 USRDS 7,348 NR 2 42 NR At transplant Queen Elizabeth Hospital, UK UK Transplant Registry graft failures, 31 deaths 2 6 & 7 NR At 1 year post transplant 7,620 NR 1 5 NR At transplant 53 USRDS 87,575 NR 1 20 Median 4.3 years At 1 year post transplant NR USRDS 57,389 7, NR At transplant NR 52 Eurotransplant centres 8,154 NR 2 10 & 6 NR At transplant *Denotes the use of machine learning techniques and Cox proportional hazards model for development of prediction models. ^Denotes the use of machine learning techniques for development of prediction model. TGF=Total graft failure; DCGF=Death-censored graft failure; NR=Not reported; SD=Standard deviation; SRTR=Scientific Registry of Transplant Recipients; USRDS=United States Renal Data System; DIVAT=Données Informatisées et Validées en Transplantation; LOTESS=Long Term Efficacy and Safety Surveillance; UK=United Kingdom.

33 22 Table 5: Performance measures of clinical prediction models of graft failure in deceased donor kidney transplant recipients (87, 88, ) Study Discrimination Calibration Reclassification Validation Rao et al No No Internal Machnicki et al Yes No Internal Kasiske et al ; 0.67; 0.72 (internal) ; ; (external) Yes No Internal, External Foucher et al No No External Goldfarb-Rumyantev et al Yes No Internal, External Moore et al TGF: 0.73 (internal), 0.70 (external) DCGF: 0.87 (internal), 0.83 (external) Yes Yes Internal, External Krikov et al , 0.64, 0.72, 0.83, 0.90 Yes No Internal Brown et al Shabbir et al , 0.60 (internal) 0.63 (external) (DCGF) (TGF) No No Internal, External Yes Yes External Watson et al No No Internal Schnitzler et al NR Yes No Internal, External Lin et al Yes No Internal Thorogood et al NR Yes No Internal External validation for model derived by Goldfarb-Rumyantev et al. found in a separate publication (104); Yes=Calibration and/or reclassification assessed; No=Calibration and/or reclassification not assessed Clinical prediction models of graft failure in living donor kidney transplant recipients Among the 18 studies included in the systematic review, there were 2 studies identified which developed clinical prediction models of graft failure in a population of living donor kidney transplant recipients (91, 92). There were no published studies of models for death with graft function in this population. The characteristics of these studies are shown in Table 6. One study utilized a large US registry for derivation of their prediction model using regression modeling, which included 20,805 patients (91). The other study was derived in a cohort of 1,819 patients from a single centre in Egypt using both regression models as well as artificial neural networks. Each study developed 2 models, which included between 8 to 21 variables. Akl et al. (92) derived models for total graft failure to be used at 3 months post-transplant whereas Tiong et al.

34 23 (91) derived models for death-censored graft failure to be used at transplant and at 6 months post-transplant. The variables included in the models are shown in Appendix 3c. Follow up time was not reported in either study. The performance measures of the clinical prediction models of death with graft function are shown in Table 7. Both models reported discrimination and calibration, while none reported reclassification. Both models calibrated well. Discrimination was variable with c statistics of 0.71 to 0.78 in the models derived using regression models and c statistics of in models derived using artificial neural networks. All models were internally validated, but none were validated externally. Neither study was identified as having a high risk of bias, however, the model by Akl et al. (92) did not comprehensively report study participation, attrition, prognostic factor selection and measurement, thereby limiting bias assessment. Study attrition was not reported in the study by Tiong et al. (91). Outcome measurement, analysis and reporting of performance measures were appropriate in both studies. When clinical utility was assessed, neither study discussed a threshold for risk based on the results of a prediction model, while both studies derived models that were clinically usable (i.e., a nomogram and/or online calculator for risk estimation was developed). Table 6: Study characteristics of clinical prediction models of graft survival in living donor kidney transplant recipients (91, 92)! Study Tiong et al DCGF Akl et al. 2008* TGF Population and timeframe Kidney transplant recipients Kidney transplant recipients alive at 3 months post transplant % Living Donor Database Patients, n Events, n Number of models, n Number of predictors, n Follow up 100 UNOS 20, , 21 NR 100 Single centre in Egypt 1819 NR 2 8, 11 NR Time of assessment At transplant and 6 months post transplant 3 months post transplant *Denotes the use of machine learning methods and Cox proportional hazards model for development of prediction model. NR=Not reported; UNOS=United Network for Organ Sharing

35 24 Table 7: Performance measures of clinical prediction models of graft survival in living donor kidney transplant recipients (32, 33) Study Discrimination Calibration Reclassification Validation Tiong et al Akl et al Yes No Internal 0.77 (derivation Cox model); 0.94 (derivation ANN) 0.72 (validation Cox model); 0.88 (validation ANN) Yes No Internal ANN=Artificial neural network; Yes=Calibration assessed; No=Reclassification not assessed 2.4 Discussion The results of this systematic review show that there have been several clinical prediction models developed to predict death with graft function as well as graft failure in kidney transplant recipients. However, only four of the 18 studies identified, all of which developed models for graft failure, were found to have a low risk of bias, were externally validated and were found to be clinically usable by developing a risk calculator or tool for use by the clinician (94, 95, 97, 100). These models were generally parsimonious, including between six and 11 predictors. Although three of the studies developing clinical prediction models for death with graft function were found to be clinically usable and parsimonious (including six to eight predictors) (85, 86, 89), none were externally validated. The importance of external validation of a clinical prediction model has been well described in the literature (58, 105, 106) Without external validation, the performance and generalizability of a model in another population is unknown (58, 68). Specifically, assessing the performance based on internal validation alone may lead to an overly optimistic assessment of a model. Furthermore, performance in an external population may be poor in circumstances where the original model is over-fitted, particularly in a smaller cohort of patients or where a large number of predictors are included in the model (53, 107, 108). Additionally, significant differences in the characteristics of an external cohort of patients may also result in the poor performance of a model. However, despite the importance of external validation, it is performed inconsistently in

36 25 the studies included in this systematic review, which is also a common problem amongst prediction models published in non-kidney transplant populations (82, 109, 110). It is unclear why external validation was not performed in these studies, however there are a number of reasons which may influence the likelihood of external validation that warrant mention. First, external validation requires accessibility to data from an external cohort, which is often not readily available to investigators. Additionally, when data from external cohorts are available, the variables required for validation may not be completely or systematically captured. Finally, external validation requires additional funds for acquisition and/or data analysis that may be a barrier to external validation. In this systematic review, where external validation was performed, only one study validated its model in a cohort of Canadian kidney transplant recipients (100). These findings underscore the importance of external validation of prediction models, in particular amongst kidney transplant recipients, where country-specific differences in patient demographics and outcomes have been well documented (67, 111, 112). Most of the studies included in this systematic review derived prediction models from large US or European databases. For models predicting death with graft function, recipient age, gender, race, cause of ESRD, comorbidity data (such as the presence of diabetes) and recipient renal function were the most commonly included predictors. In addition to these variables, models predicting graft failure also included a number of donor variables such as age, race, renal function, body mass index, comorbidities such as hypertension and diabetes, as well as transplant factors such as HLA mismatches or the presence of delayed graft function or acute rejection after transplantation. The inclusion of these variables in prediction models are not surprising, given that these factors have been associated with an increased risk of death and graft failure in prior etiological studies(34, 40-42, 113, 114). The number of variables included in the models was highly variable. Models for death with graft function included between 8 and 23 variables and models for graft failure included six to 42 variables. Simple, easy to use and parsimonious models are more likely to be integrated into clinical practice. Prediction models that are available in a web-based or mobile application format are also more likely to be used by clinicians (106). This systematic review identified ten studies that developed models with 15 or more predictors, many which were derived using

37 26 machine learning methods such as artificial neural networks or CART. Although in general, models derived using these methods had slightly better c statistics, these models were also the most likely to include a large number of variables. Furthermore, a number of models included variables that are unlikely to be easily measured and available in most transplant centres, creating an additional challenge to clinical usability. Although the number of prediction models developed using methods such as artificial neural networks or CART have increased in the literature, it has not been convincingly shown that machine learning methods used for the purposes of prediction modeling provide a substantial advantage over traditional statistical methods. In the cardiovascular literature, studies by Austin et al. which compared traditional regression models to CART both for predicting the probability of heart failure with preserved ejection fraction (versus reduced ejection fraction) (115) and predicting short term mortality in hospitalized patients with myocardial infarction or heart failure (116) found that machine learning methods were not superior to conventional logistic regression. Conversely, a study comparing artificial neural networks and decision trees to traditional regression methods found that machine learning methods were superior in predicting survival in patients with breast cancer (117). Whether it is advantageous to use novel machine learning methods for prediction rather than traditional statistical in kidney transplant recipients remains unclear and warrants further study. The performance of the models included in this systematic review was variable. Calibration was generally good when reported. However, in four studies, calibration was not assessed altogether or graphical evaluation in lieu of formal statistical tests of calibration was done (i.e., comparing Kaplan-Meier survival curves in the development and test populations). Although evaluation of calibration is important, the best way to do so is unclear, particularly where large databases are used and small differences between observed and predicted risk may be statistically significant. In most models, discrimination was modest, with c statistics generally between 0.65 and 0.75, although a few select models achieved higher discrimination with c statistics between 0.81 and 0.94 for predicting graft failure (87, 92, 97, 98, 100). Only three studies were identified as having a high risk of bias, although study attrition was rarely or poorly reported. Only two studies reported reclassification, which is a useful metric in determining the incremental improvement in the performance of a model when additional predictors are added.

38 27 Discrimination poorly captures small improvements in model performance, especially if the model performs well already, and therefore the use of a reclassification measure such as the net reclassification index (NRI) could provide meaningful information about the incremental benefit of a predictor (108). In particular, in the transplant setting, identification of novel biomarkers to help detect graft injury at an early stage is the subject of extensive research. Reclassification measures can be useful in determining the incremental predictive value of including novel biomarkers in existing prediction models, and whether the additional cost is justified (118). An important consideration when using reclassification indices is to determine what constitutes clinically meaningful risk categories (i.e., clinical utility), which may be variable from clinician to clinician and is generally infrequently reported in studies included in this systematic review (119). The strengths of this systematic review include a sensitive search strategy to identify studies for inclusion. As a result, a large number of citations were reviewed and it is unlikely that a prediction model was missed. Furthermore, in addition to detailed reporting of study characteristics and performance metrics, studies were extensively screened for bias and clinical utility using an established framework (82). Finally, the models selected in this systematic review were stratified by donor subtype (i.e., derived in a cohort of living donor or primarily deceased donor kidney transplant recipients) given the significant variability in transplant activity by donor type that exists across transplant centres and/or jurisdictions. However, there are some limitations. First, one non-english prediction model was identified but was not included in the final systematic review, and as such, the performance of this model is unknown. Second, the objective was to systematically review clinical prediction models of patient and graft survival in kidney transplant recipients. Therefore, models predicting specific post-transplant patient outcomes, such development of coronary artery disease, or post-transplant graft outcomes, such as the development of transplant glomerulopathy, where not included. Furthermore, prediction scores which were based uniquely on histological or imaging findings as predictors were also excluded. Finally, this systematic review did not review impact studies (i.e., studies to determine if the models when implemented results in improvement in patient outcome, cost-effectiveness, physician decisions etc.).

39 28 In conclusion, this systematic review has identified a several clinical prediction models of patient and graft survival with generally modest performance. However, only four studies were both externally validated and clinically usable. Further study is needed to better understand the barriers to external validation and to develop novel strategies to improve upon existing prediction research in kidney transplant populations.

40 29 Chapter 3 3 Validation Study in a Cohort of Canadian Kidney Transplant Recipients 3.1 Background Validation of Clinical Prediction Models Validation, which refers to the evaluation of the performance of a model, can be classified into two categories: Internal or external validation. The purpose of internal validation is to assess the reproducibility or internal validity of a model, which is done by evaluating the model in the same sample of patients that was used to develop the model (53). In addition, internal validation provides a threshold of performance that one could expect from a model in other (external) populations. There are various methods that can be used to assess internal validation. A summary of the advantages and disadvantages of internal validation methods is shown in Table 8. Commonly, internal validation is performed using the split-sample procedure, which divides the cohort of patients into two groups (typically a 50/50 or 67/33 sample split), where one group is used as a development sample to derive the model, and the remaining group is used as the test sample for validation. Although this method is simple from a methodological perspective, it has a number of drawbacks. The number of patients in the development sample is significantly reduced due to splitting and thus, model results may be less stable. In addition, the test sample may by random chance, contain significant imbalances in distributions of predictors or the outcome, resulting in a biased assessment of performance when only a part of the data is used for validation. Generally, the limitations of split-sample validation are less concerning in circumstances where sample sizes are larger. Newer validation methods such as cross-validation and bootstrap validation are more commonly used in order to overcome some of the limitations of split-sample validation. In cross-validation, the cohort of patients undergoes splitting as in split-sample validation. However, the test sample generally contains a much smaller proportion of the total sample (i.e., approximately 10%) but

41 30 this splitting exercise is repeated multiple times. The performance metric of a model is an average of all validations. As a result, all patients are used to test the model and a larger sample size can be used for model development. Bootstrap validation first requires that a bootstrap sample be created for model development, usually by sampling patients with replacement, from the original sample. This procedure is usually repeated a number of times (i.e., 100 to 200 times) and prediction models are developed in each bootstrap sample. The model is then validated in the original sample. This is now the preferred method for validation as model performance estimates are more stable when sufficient bootstrap samples are created (53, 105). Table 8: Advantages and disadvantages of internal validation methods Method Procedure Advantages Disadvantages Apparent validation Testing of model using 100% of development sample Sample size not reduced for the purposes of validation Results in overly optimistic estimates of model performance Simple to perform Split-sample validation Testing of model in subset of randomly selected patients (i.e, 33% to 50%) Simple to perform Reduction of development sample size for purposes of validation Performance of model not based on full sample of patients (i.e., small subset which may randomly show poor performance) Cross-validation Testing of model in subset of randomly selected patients (i.e., 10%) but testing repeated numerous times on random subsets Larger sample can be used for model development All patients included in testing of the model Computer-intensive May not reflect all sources of model uncertainty Bootstrap validation Model development in subsamples of data generated using sampling with replacement, validation in original dataset Uses whole data set for development and validation Estimates with low variability and minimal bias Computer-intensive

42 31 In contrast to internal validation, external validity is used to determine the generalizability of a model to other populations (53). Temporal validation is a subtype of external validation, which refers to validation of a model in a more recent cohort of patients similar to the development sample (i.e., from the same centre) (105, 120). For example, a prediction model could be developed in a sample of transplant patients at Toronto General Hospital from 2000 to 2005 and subsequently temporally validated in transplant patients at Toronto General Hospital from 2006 to Temporal validation can result in excellent estimates of performance if definitions of predictors and outcomes remain the same in the development and testing samples, and the demographics of patients in both samples are similar. Conversely, changes in the incidence of an outcome and/or definitions of predictors may account for a poorly performing model with temporal validation. Furthermore, changes in treatments or delivery of care that occur over time, as well as technological innovations may also account for poorly performing models. Fully independent validation is the preferred method for external validation in order to demonstrate portability of a model and requires testing of a model in a completely different sample of patients (53) Rationale for external validation There are a number of reasons why a prediction model may not perform in an external population, thereby highlighting the importance of external validation. From a statistical perspective, the original model may not perform as well if a predictor, which is important in the new population, is missing from the model, due to the variable selection methods used during model development. Alternatively, the modeling strategies used to derive the original model may result in a poorly fitted model, and therefore the performance in a new population may be poor. From a clinical perspective, the model may not perform well due to the significant variation in case-mix between patients used for model development and external validation. Differences in patient demographics, delivery of health services, patient outcomes, measurement of predictors and socio-economic factors may result in a poorly performing model that is not transportable. Conversely, a robust model will perform well despite differences in derivation and validation populations. These differences are of particular importance in kidney transplant recipients, where geographical variations in post-transplant outcomes have been well established. Ojo et al.

43 32 conducted a retrospective cohort study, which compared the outcomes of kidney transplant recipients from Spain to those from the US (112). The results of this study showed that although US and Spanish transplant recipients had similar 10-year death-censored graft survival, US kidney transplant recipients had a two-fold increase in their hazard of death with graft function. Another study by Kim et al. compared mortality after kidney transplantation between the US and Canada (67). In this study, mortality in the first year post-transplant was similar between both populations. However, in the long term, post-transplant mortality was significantly higher in US vs. Canadian kidney transplant recipients, with a step-wise increase in relative hazard based on the duration of dialysis prior to transplantation. (Figure 5) Figure 5: Comparison of patient survival between Canadian and US kidney transplant recipients Figure 1: Unadjusted Kaplan-Meier post-transplant survival curves for Canada and the United States. Patients include all recipients of a kidney transplant between January 1, 1991 and December 31, 1998, with follow-up until December 31, The Kaplan-Meier curves are significantly different based on the logrank test (p < 0.005). Figure 2: Post-transplant mortality hazard ratios (United States/Canada) by pre-transplant years on dialysis. Hazard ratios are adjusted for age, sex, race, cause of ESRD, year of transplant and donor source. Bars represent hazard ratios, while the range denotes the 95% confidence interval. All hazard ratios are significantly different from 1.00 (p < 0.005) except for U.S. and Canadian patients with less than 1 year of pre-transplant time on dialysis (p = 0.41). Reproduced with permission from Kim, S.J. et al. Mortality after kidney transplantation: a comparison between the United States and Canada. Am J Transplant. 2006;6(1): Copyright John Wiley and Sons. (67)

44 33 These observations are likely due to a number of factors including differences in dialysis care and vascular access prior to transplantation (21), geographic variations in patient characteristics (i.e., a higher proportion of patients of African descent in the US) and differences in the prevalence of comorbid conditions. Furthermore, cross-country variation in the delivery of health services and financial coverage for immunosuppressive medications are well documented and likely contributes to differences in outcomes. A recent editorial by Gill and Tonelli highlighted the differences in graft survival rates post-transplantation by country and showed, that in the US where immunosuppressive medications are only covered for the first 3 years post transplant, 5- and 10-year graft survival was significantly lower than Canada, Australia and the UK, where recipients receive lifetime drug coverage (111) (Figure 6). Figure 6: 5- and 10- year graft survival rates by country Kidney-Transplant Survival and Immunosuppressive Coverage Policies for Selected Countries (for Recipients of a First Kidney-Only Transplant from a Deceased Donor).* Country 5-Yr Survival percent 10-Yr Survival Government-Funded Immunosuppressive Coverage Australia Lifetime for all recipients Canada Lifetime for all recipients United Kingdom Lifetime for all recipients United States Lifetime for recipients >65 yr of age or with workrelated disability; 3 yr for all other recipients * Data include patients whose kidney transplants failed because they died. These data were obtained from the ANZDATA Registry Report, 2010 (Australia and New Zealand Dialysis and Transplant Registry), the Canadian Organ Replacement Register Report, 2011 (Canadian Institute for Health Information), the National Health Services Blood and Transplant Annual Report, (National Health Services), and the USRDS 2011 Annual Data Report: Atlas of Chronic Kidney Disease and End-Stage Renal Disease in the United States (United States Renal Data System). Reproduced with permission from Gill, J and Tonelli, M. Penny wise, pound foolish? Coverage limits on immunosuppression after kidney transplantation. N Engl J Med. 2012;366(7): Copyright Massachusetts Medical Society. (111)

45 34 The vast majority of clinical prediction models of patient and graft survival have been derived in US or European transplant populations and there were no studies identified in the systematic review which developed prediction models in Canadian kidney transplant recipients. Only one study, which developed a prediction model from a cohort of kidney transplant recipients from the UK, validated their prediction model in a Canadian cohort of transplant recipients from Halifax, Nova Scotia (100). The geographical differences between transplant populations discussed, as well as potential statistical limitations of existing prediction models, underscore the importance of externally validating these models in a contemporary cohort of Canadian kidney transplant recipients prior to implementation of these models in the clinical setting. 3.2 Considerations for validation of a Cox proportional hazards model Methods for validation of prediction models developed using logistic regression models have been well described in the literature (120, 121). However, certain considerations warrant discussion when validating a prediction model for a time-to-event outcome. First, in a logistic regression model, discrimination is assessed by the c statistic or area under the ROC curve, which compares those with and without the event of interest and represents the proportion of pairs where the predicted probability is higher for the patient who experienced the event. However, in a survival analysis, certain individuals will be censored during the follow-up period for a number of reasons, such as loss to follow-up, the end of the study period or the occurrence of a competing risk. The varying survival times between individuals as well as censoring of patients cannot be accounted for when using traditional logistic regression model validation methods. As a result, newer metrics such as Harrell et al. s AUC (122) statistic have been developed in order to take into account the fact that individuals will experience the event of interest at different times. Thus, discrimination in the setting of survival analysis determines the models ability to separate those with longer event-free survival from those with shorter eventfree survival over a certain time horizon (109, 123). Second, the Cox proportional hazards model allows for estimation of the hazard function, which is the probability that an individual will experience the outcome in the next increment of time, given that the outcome has not occurred up to the beginning of that time. h(t) is the hazard function at a certain time, t, and h 0 (t) is represents the baseline hazard function (the equivalent of

46 35 the intercept of a logistic regression model), which represents the hazard of an event when all the covariates included in the regression model are set to their referent values. (β 1 *X 1 + β 2 *X 2 + β 3 *X β p *X p ) represents the linear predictor. h(t)=h 0 (t)*exp(β 1 *X 1 + β 2 *X 2 + β 3 *X β p *X p ) Therefore, for the purposes of external validation (specifically to assess model calibration), the baseline hazard function from the original data is needed. Unfortunately, the baseline hazard of a model is infrequently reported as it is considered a nuisance parameter by most statistical packages. An alternative rough, but imperfect assessment of calibration can be done by using individual covariate values, mean covariate values, regression coefficients and an estimate of the survival probability using the Kaplan-Meier survival curves to estimate an individual patient risk when the baseline hazard function is not available (109). However, these estimated risk predictions may be imprecise and as such, using the baseline hazard, if available, is ideal. 3.3 Methods Creation of the external validation cohort The second objective of this thesis is to externally validate selected clinical prediction models of patient and graft survival in a contemporary cohort of Canadian kidney transplant recipients. The Comprehensive Renal Transplant Research Information System (CoReTRIS) database (124) was used to create the cohort for external validation of the selected clinical prediction models. CoReTRIS is an in-centre research database which houses an extensive set of recipient, donor, transplant, laboratory, pathology, treatment, and follow-up data on all patients receiving kidney transplants at the Toronto General Hospital, University Health Network, since 1 Jan All data housed in CoReTRIS have been abstracted from patient charts (electronic and paper), entered into the database, and audited for completeness and accuracy. To ensure that the data entered into CoReTRIS is of high quality, a number of educational resources have been developed (i.e., training manuals, development of standard operating procedures and interactive e-learning tools) in order to ensure adequate training of all individuals involved in abstracting data. Quality control measures (validation checks) have been built into the CoReTRIS data platform and Stata statistical software was used to develop statistical codes used to detect any

47 36 potential errors in data fields. The CoReTRIS database has resulted in a number of high-quality peer-reviewed publications. The external validation cohort was created by including all incident kidney transplant recipients included CoReTRIS from 1 Jan 2000 to 30 Apr This time frame was selected as it reflected the modern era of transplantation, where there have been no major changes in immunosuppression protocols or delivery of care at the Toronto General Hospital. Patients who were wait-listed for kidney transplantation but had not yet received a kidney transplant were not included in the cohort. In order to create a cohort of patients similar to the development cohorts of the models to be validated, the following exclusion criteria were applied: (1) The clinical prediction models selected for validation were developed in a cohort of kidneyonly transplant recipients. Therefore, multi-organ transplant recipients were excluded. (2) The selected clinical prediction models were derived in cohorts of adult kidney transplant recipients. As a result, pediatric transplant recipients (i.e., less than 18 years of age at the time of transplantation) were excluded. (3) Kidney transplant recipients with missing age at the time of transplantation were excluded given that age is a major determinant of patient survival and is included in the clinical prediction models of patient and graft survival selected for validation. (4) Recipients with missing outcome status (i.e., unknown if died or still alive with graft function) without a last follow-up date in the database were excluded. (5) Recipients with missing kidney donor subtype information (i.e., living or deceased donor) were excluded. (6) The clinical prediction models of death with graft function, total graft failure and deathcensored graft failure selected for validation were all developed in a cohort of patients who had survived to 1 year post-transplantation with a functioning kidney transplant and are intended to be used in the clinical setting at this time point (see section below for a detailed description of model selection). As a result, patients who experienced death with graft function or graft failure in the first year after transplantation were excluded, as well as patients who had not yet accrued 1 year of follow-up time after transplantation.

48 Model selection Among the many studies identified in the systematic review of clinical prediction models of patient and graft survival, the following approach was implemented in selecting models for external validation in a Canadian cohort of kidney transplant recipients: (1) Models were not selected for external validation if the study had a high-risk of bias (based on the framework described in Chapter 2); (2) Models were not selected for external validation if they were derived using machine learning methods (i.e., artificial neural networks or CART); (3) Models were not selected for external validation if the variables included in the original model were not available or comprehensively captured in the Comprehensive Renal Transplant Research Information System (CoReTRIS) database. As described above, CoReTRIS houses an extensive set of donor, recipient and transplantspecific data elements that are commonly used in clinical practice and for research purposes. Some variables are not collected in CoReTRIS as they are not applicable to the Canadian health care system model (i.e., insurance type, transplant centre volume) or they are not routinely used for clinical-decision making (i.e., donor adrenaline use). Other variables are not collected comprehensively in all patients, but rather only if clinically indicated or at the discretion of an individual physician (i.e., urine albumin to creatinine ratio, 24 hour urine collection for proteinuria, renal biopsy or echocardiogram findings). Other variables, although available in clinical notes and electronic medical records, are not systematically collected for the purposes of CoReTRIS given that they are of limited utility for current research purposes (i.e., living donor nephrectomy type). Table 9 summarizes the variables from the clinical prediction models that are not captured or incompletely captured in CoReTRIS.

49 38 Table 9: Variables from prediction models not captured or incompletely captured in CoReTRIS database Models of death with graft function in deceased donor transplant recipients Left ventricular hypertrophy Vascular calcification Current smoker Primary insurance/source of pay for treatment Acute tubular necrosis Models of graft failure in deceased donor kidney transplant recipients Donor adrenaline use Urinary albumin-creatinine ratio Primary insurance/source of pay for treatment Proteinuria at 1 year post-transplant (in grams per day) Transplant centre volume Transplant procedure Serum albumin at listing Models of graft failure in living donor kidney transplant recipients Time to diuresis Acute tubular necrosis Living donor nephrectomy type Flow diagrams depicting model selection for external validation are shown in Figures 7a, 7b and 7c.

50 39 Figure 7a: Selection of clinical prediction models of death with graft function in recipients of deceased donor kidney transplant recipients 5"CPM"(Cox"model)"&"1"CPM" (ANN)"of"DWGF"in"DD"KTR" Exclusions:" " High"risk"of"bias"(N=1)" " M.L."model"(N=1)" " Variables"not"available"(N=3)" 1"CPM"of"DWGFin"DD" KTR"for"external" validaoon"in"coretris" M.L.=Machine learning; CPM=Clinical prediction model; DWGF=Death with graft function; DD KTR=Deceased donor kidney transplant recipients; ANN=Artificial neural network. There were six clinical prediction models of death with graft function developed in a cohort of primarily deceased donor kidney transplant recipients (from five studies). One study was identified as having a high-risk of bias (88). One model was derived using machine learning methods (87), three models were derived using variables not available or captured in a comprehensive way in CoReTRIS (85, 89) (87). The model derived by Hernandez et al. was selected for external validation (86) (Figure 7a).

51 40 Figure 7b: Selection of clinical prediction models of graft failure in recipients of deceased donor kidney transplant recipients 2"CPM"(CART),"2"CPM"(ANN)"&" 11"(Cox"model)"of"GF"in"DD" KTR" Exclusions:" " High"risk"of"bias"(N=3)" " M.L."model"(N=3)" " Variables"not"available"(N=7)" " Model"validaPon"underway"(N=1)" " 1"CPM"of"GF"in"DD"KTR" for"external"validapon"in" CoReTRIS" M.L.=Machine learning; CPM=Clinical prediction model; GF=Graft failure; DD KTR=Deceased donor kidney transplant recipients; ANN=Artificial neural network; CART=Classification and regression tree. There were 15 clinical prediction models of graft failure developed in a cohort of deceased donor kidney transplant recipients (from thirteen studies). Three studies were found to have a high risk of bias (88, 99, 103). Three models were derived using machine learning methods (87, 96, 98), seven models included variables that were not captured in CoReTRIS (87, 94-96, ). One model was not selected as a validation study in the same external validation cohort was already being done for this model (93). The clinical prediction model derived by Moore et al. which can be used to predict death-censored graft failure as well as total graft failure (two models developed by the authors) was selected for external validation in CoReTRIS (97). (Figure 7b)

52 41 Figure 7c: Selection of clinical prediction models of graft failure in recipients of living donor kidney transplant recipients 2"CPM"(Cox"model)""&"1"CPM" (ANN)"of"GF"in"LD"KTR" Exclusions:" " M.L."model"(N=1)" " Variables"not"available"(N=2)" " 0"CPM"of"GF"in"LD"KTR" for"external"validajon"in" CoReTRIS" M.L.=Machine learning; CPM=Clinical prediction model; GF=Graft failure; LD KTR=Living donor kidney transplant recipients. There were three clinical prediction models of graft failure developed in a cohort of living donor kidney transplant recipients (from two studies). One model was derived using machine learning methods, and two models were derived using variables that were not available in the CoReTRIS database (91, 92). Thus, no prediction models were selected for validation from this population. (Figure 7c).

53 Predictor variables The characteristics of the selected prediction models for external validation, including the predictor variables and their beta coefficients in each model, are shown in Table 10. Table 10: Characteristics of prediction models selected for external validation (86, 97) Model Outcome Derivation1population Variables1included1(recipient)1 Beta1coefficient Hernandez(et(al.( Death(with Adult(kidney(transplant(recipients Age(<40 Referent Transplantation graft(function (from(1990a2002(alive(with( Age(40A ;(88:803A809 functioning(graft(at(1(year(postatransplant Age(50A Living(Donor(KTR:(1.2%( Age(> PreAtransplant(diabetes(mellitus((Y/N) Database:(38(Spanish(Transplant(Centres Positive(HCV(antibody(positivity((Y/N) NODAT(at(1st(year(postAtransplant Serum(Creatinine(at(1st(year(postAtransplant((per(mg%) Proteinuria(>1g(at(1st(year(postAtransplant((Y/N)( 0.99 Tacrolimus(use(at(1st(year(postAtransplant((Y/N) A0.476 MMF(use(at(1st(year(postAtransplant((Y/N) A0.782 Moore(et(al.( Adult(kidney(transplant(recipients Age(> AJKD Total(graft(failure( (from(1995a1998(alive(with( Male(gender((Y/N)( ;(57(5):(744A751 functioning(graft(at(1(year(postatranpslant Serum(albumin(at(1(year(<(4.0g/L((Y/N) Living(Donor(KTR:(6.9% Serum(urea(nitrogen(at(1(year(>40mg/dL((Y/N) egfr(<35ml/min(at(1(year((y/n) Database:(LOTESS(Study((UK) %(decrease(in(egfr(>30%((y/n) Moore(et(al.( Same(as(above Age(< AJKD DeathAcensored Black(race((Y/N) ;(5(5):(744A751 graft(failure Acute(rejection((Y/N) Serum(albumin(at(1(year(<3.5g/dL((Y/N) egfr(at(1(year(<35%((y/n) %(decrease(in(egfr(>20%((y/n) KTR=Kidney transplant recipients; HCV=Hepatitis C virus; NODAT=New onset diabetes mellitus; MMF=Mycopheolate Mofetil; egfr=estimated glomerular filtration rate; Y/N=Yes/No. In order to externally validate the selected models, the predictor variables were manipulated in order to match the original models. For validation of the clinical prediction model of death with graft function published by Hernandez et al.(86), predictor variables were categorized as follows: (1) age at transplantation < 40 years, years, years, >60 years; (2) presence of pretransplant diabetes mellitus (Yes/No); (3) positive hepatitis C antibody (Yes/No); (4) new-onset diabetes mellitus after transplantation developed in the first year post-transplantation (Yes/No); (5) serum creatinine at 1 year post-transplantation (measured in mg/dl); (6) proteinuria greater than 1 gram at 1 year post-transplantation (Yes/No); (7) tacrolimus use at 1 year posttransplantation (Yes/No) and (8) mycophenolate mofetil use at 1 year post-transplantation

54 43 (Yes/No). In the transplant program at Toronto General Hospital, proteinuria is measured by urinary dipstick or by 24 hour urine collection in circumstances where deemed appropriate by the transplant nephrologist. If urinary dipstick was used, patients with 3+ proteinuria or greater (or Moderate or Severe proteinuria) were categorized as having greater than 1 gram of proteinuria per day. Urinary dipstick assessments only provide an approximation of urinary protein excretion. However, in clinical practice, the presence of 3+ proteinuria is generally reflective of at least a moderate amount of protein excretion, and as such, this cutoff was used to categorize patients as having greater than 1g of proteinuria per day where applicable. For validation of the clinical prediction models of total graft failure by Moore et al. (97), predictor variables were categorized in CoReTRIS as follows: (1) recipient age >40 years (Yes/No); (2) male gender (Yes/No); (3) serum albumin at 1 year <4.0 g/l (Yes/No); (4) serum urea nitrogen at 1 year >40 mg/dl (Yes/No); (5) egfr <35 ml/min at 1 year calculated using the 4 variable Modification of Diet in Renal Disease (MDRD) Study equation (Yes/No); (6) percent decrease in egfr from time of transplantation to 1-year post-transplantation of greater than 30 percent (Yes/No). For validation of the clinical prediction models of death-censored graft failure, predictor variables were categorized as follows: (1) recipient age <40 years (Yes/No); (2) black race (Yes/No); (3) acute rejection in first year post-transplantation (Yes/No); (4) serum albumin at 1 year post-transplantation <3.5g/dL (Yes/No); (5) egfr <35ml/min at 1 year calculated using the 4 variable Modification of Diet in Renal Disease (MDRD) Study equation (Yes/No); (6) percent decrease in egfr from time of transplantation to 1-year post-transplantation of greater than 20 percent (Yes/No) Outcome variables Outcomes of interest were defined in the same way in CoReTIRS and in the original models selected for validation. Specifically, death with graft function was defined as death from any cause with a functioning transplant. A kidney transplant is considered to be functioning if the patient does not require dialysis, irrespective of the serum creatinine or estimated glomerular filtration rate of that individual. Total graft failure was defined as the need for chronic dialysis, preemptive re-transplantation or death with graft function. Death-censored graft failure was defined as the need for chronic dialysis or preemptive re-transplantation.

55 Statistical analysis Baseline characteristics of the external validation cohort where assessed using parametric and non-parametric methods where appropriate. The Kaplan-Meier product limit method was used to assess time to total graft failure, death-censored graft failure and death with graft function. Performance metrics of discrimination and calibration were used to assess the model in the external validation cohort. Discrimination indicates the proportion of times that a prediction model will correctly identify, within a pair of individuals, the individual that has the longest survival time. Calibration refers to the agreement between predicted probabilities derived by the prediction model and the actual observed probabilities in the external validation cohort (53, 79). The regression model used for external validation of all selected clinical prediction models was the Cox proportional hazards model. The concordance statistic (c statistic) was used to evaluate discrimination in the entire external validation cohort, as well as in subsets of living and deceased donor kidney transplant recipients. In order to assess calibration, the original regression model was used to estimate a predicted probability of the event of interest for each individual in the external validation cohort. This predicted probability was then compared to the observed probability of the event of interest in the external validation cohort. Calibration was assessed both graphically and statistically. Graphical assessment was done by plotting predicted and observed probabilities by quartile or quintile using a bar graph and visually assessed for calibration. Differences between predicted and observed probabilities were assessed using the Nam and D Agostino X 2 statistic. A P value > 0.05 for this test indicates a model that calibrates well. The estimated probabilities for each individual in the external validation cohort were derived using the following method: (1) The sum of the regression model (x) was calculated for each individual in the cohort based on the value of the predictors (X) for that individual and the regression (β) coefficients from the original model as shown by the formula below: x= β1*x 1 + β 2 *X 2 + β 3 *X β p *X p

56 45 (2) Subsequently, y is generated as follows: y=exp(x) (3) The estimated probability for each individual is then generated: Probability = 1 S o (t) (y) Where S o (t) is equal to the baseline survival function at a specified time point, which represents the survival rate for an individual with the mean value of covariates in the risk equation. This survival function is estimated from the baseline cumulative hazard function H o (t), obtained from the original Cox model based on the following formula: S o (t)=exp(-h o (t)) The baseline survival function (or hazard function) is derived from the development data. As a result, in order to validate a prediction model (i.e., specifically determine calibration) in an external cohort of patients, this function must be reported by the authors. The baseline survival function for the model of death with graft function (Hernandez et al.) (86) was reported in the manuscript and shown in Table 11. However, this was not published for the models predicting total and death-censored graft failure (Moore et al.) (97). The corresponding author was contacted on multiple occasions by myself and Dr. David Naimark (telephone and communication). Unfortunately, the authors did not respond to our communication. A clinical prediction model of total and death-censored graft failure was derived in a similar cohort of adult kidney transplant recipients from the UK from , and baseline survival functions for total and death-censored graft failure were reported in this manuscript (100). Given the similarities of the transplant populations, these baseline survival functions were used for external validation of the selected models.

57 46 Table 11: Baseline survival functions for clinical prediction models selected for external validation Model So(t) Time point DWGF years DCGF * 5 years TGF * 5 years DWGF=Death with graft function; DCGF=Death-censored graft failure; TGF=Total graft failure. Obtained from alternate publication (Shabir, S. et al) (100) Sensitivity analysis A number of patients in the external validation cohort had one or more missing data elements. For the model predicting death with graft function, the following sensitivity analyses were conducted: (1) If data on immunosuppressive drug use was missing at 1-year post transplant (i.e., patient on tacrolimus or MMF), discrimination was assessed based on the following assumptions: (a) Patients were on tacrolimus and MMF or (b) Patients were not on tacrolimus and MMF or (c) Patients were on tacrolimus and not MMF. (2) If data on proteinuria or serum creatinine was missing at 1-year post transplant, discrimination was assessed using the last lab data if available within the prior 6 months. (3) If data on hepatitis C antibody positivity was missing, discrimination was assessed assuming that a missing result indicated a negative hepatits C antibody status. For the validation of the model of total graft failure and death-censored graft failure, the following sensitivity analyses were conducted:

58 47 (1) If data on BUN, serum albumin and egfr were missing, discrimination was assessed using the last available lab data if available within the prior 6 months. 3.4 Results Characteristics of the external validation cohort A flow diagram representing inclusion and exclusion criteria applied in the creation of the external validation cohort is shown in Figure 8. The original cohort included 2,023 incident kidney transplant recipients between 1 Jan 2000 and 30 Apr After exclusion of multiorgan transplant recipients (N=292), pediatric transplant recipients (N=74), recipients with missing age at transplantation (N=48), recipients with missing outcome status (N=21) and missing donor subtype (N=6), recipients with death with graft function or graft failure prior to 1- year post-transplantation (N=79) and recipients with less than 1 year follow-up time (N=177), the final cohort for external validation consisted of 1,326 kidney transplant recipients. Amongst the multi-organ transplant recipients that were excluded from the cohort, the specific non-kidney organ transplants were as follows: pancreas transplant (N=233), liver transplant (N=49), lung transplant (N=4), heart transplant (N=3) and other organ transplant (N=3).

59 48 Figure 8: Creation of the external validation cohort All#incident#kidney#transplant# recipients#in#coretris#database# between#jan#1#2000# #Apr#30# 2013# (N=2,023)# Cohort#of#incident#kidney# transplant#recipients## (N=1,582)# Final#cohort#of#incident#kidney# transplant#recipients#alive#with#a# funchoning#grav#at#1#year#posti transplantahon## (N=1,326)# Exclusions:## # MulHIorgan#transplant###### recipient#(n=292)# Age#<18#years#at#transplantaHon# (N=74)# Missing#age#at#transplantaHon# (N=48)# Missing#outcome#status#(N=21)# Missing#donor#subtype#(N=6)# Exclusions:# # Death#with#graV#funcHon#or# grav#failure#prior#to#1#year#post# transplantahon#(n=79)# FollowIup#Hme#less#than#1#year# postitransplantahon#(n=177)# The baseline characteristics of the external validation cohort are shown in Table 12. Amongst the 1,326 patients included in the external validation cohort, the mean age was 49.2 ±13.1 years, and the majority of patients were male (62.4%) and Caucasian (66.7%). Glomerulonephritis was the most common cause of ESRD. Approximately half of the cohort had received a living donor kidney transplant (51.4%). Table 13 shows the baseline characteristics of the external validation cohort stratified by donor subgroup. In comparison to recipients of living donor kidney transplants, recipients of deceased donor kidney transplants were older (mean age in years 52.5 ± 12.6 vs. 46 ± 12.8), were more likely to be Black and have diabetes as the cause of ESRD. In addition, there were higher proportions of patients with a history of pre-transplant diabetes mellitus, coronary heart disease, stroke and peripheral vascular disease amongst recipients of deceased donor kidney transplants. Only 0.3% of deceased donor transplant recipients underwent

60 49 pre-emptive kidney transplantation in comparison to 21.2% of living donor kidney transplant recipients. In addition, a larger proportion of deceased donor kidney transplant recipients (vs. living donor kidney transplant recipients) were on conventional intermittent hemodialysis prior to undergoing kidney transplantation. Among recipients of living donor kidney transplants, the vast majority received an organ from a relative (i.e., parent, sibling, other relative). In recipients of deceased donor kidney transplants, 29.3% received a kidney from an expanded criteria donor (ECD) and 11.5% received a kidney from a donor after circulatory death (DCD) (Appendix 1).

61 50 Table 12: Baseline characteristics of external validation cohort and derivation cohorts of models selected for external validation Recipient(Factors Baseline(Characteristics KT(Recipients((((((((((((( (N(=1,326) (Derivation(cohort((((((((( DWGF(Model((((((((((((( (N(=2,542) (Derivation(cohort( TGF/DCGF(Models((((((((( (N(=1,934) Age(at(transplant((years,(mean(±(SD) 49.2± ± ±14.0 Female(sex((%) Race((%)((( White Black South+Asian N/A East+Asian Other Cause(ESRD((%)( Diabetes Glomerulonephritis Polycystic+kidney+disease 13.6 N/A N/A Other Missing 0.6 Hepatitis(C(Antibody((%)(((((((((((((((((Positive Negative N/A Missing 3.5 N/A Hepatitis(B(Core(Antibody((%)((((((+Positive Negative N/A Missing 34.3 N/A Hepatits(B(Surface(Antibody((%)((Positive Negative 34.2 N/A N/A Missing 7.7 Recipient(Blood(Type((%)(((((((((((((((((O A B 13.4 N/A N/A AB Missing 2.3 PreUtransplant(Diabetes((%)(((((((((((Yes No ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((Missing 4.8 N/A N/A Hemodialysis(modality(((( Intermittent+HD Home+HD Nocturnal+HD 5.2 N/A Peritoneal+Dialysis 22 N/A Missing+or+Preemptive 13.6 N/A PreUtransplant(Coronary(Heart(Disease((%)( Yes No 77.5 N/A N/A Missing 5.5 PreUtransplant(Stroke((%) Yes No N/A N/A Missing 5.5 PreUtransplant(Peripheral(Vascular(Disease((%) Yes No N/A N/A Missing 5.4 Transplant(Factors( Living(Donor((%) Transplant(number(((((((((((((((((((((((((((((((((((+First+transplant ReStransplant Peak(panel(reactive(antigen((%);(median((IQR)( 2+(30) (Mean±SD)+11.2±23 N/A Missing+N=102+(7.7%) HLA(mismatches(((N);(median((IQR) 4+(2) (Mean±SD)+3.1±1.2 (Mean±SD)+2.6± Missing+N=316+(23.8%) Transplant(year((%)((( S (prior+to+year+2000) S S

62 51 Table 13: Baseline characteristics of external validation cohort by donor subtype Recipient(Factors Baseline(Characteristics LD(KT(Recipients((((((((((( (N(=681) DD(KT(Recipients(((((((((( (N(=645) Age(at(transplant((years,(mean(±(SD) 46± ±12.6 Female(sex((%) Race((%)((( White Black South+Asian East+Asian Other Cause(ESRD((%)( Diabetes Glomerulonephritis Polycystic+kidney+disease Other Missing Hepatitis(C(Antibody((%)(((((((((((((((((Positive Negative Missing Hepatitis(B(Core(Antibody((%)((((((+Positive Negative Missing Hepatits(B(Surface(Antibody((%)((Positive Negative Missing Recipient(Blood(Type((%)(((((((((((((((((O A B AB Missing PreOtransplant(Diabetes((%)(((((((((((Yes No ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((Missing Hemodialysis(modality(((( Intermittent+HD Home+HD Nocturnal+HD Peritoneal+Dialysis PrePemptive+transplantation Missing PreOtransplant(Coronary(Heart(Disease((%)( Yes No Missing PreOtransplant(Stroke((%) Yes No Missing PreOtransplant(Peripheral(Vascular(Disease((%) Yes No Missing Transplant(Factors( Transplant(number(((((((((((((((((((((((((((((((((((+First+transplant RePtransplant Peak(panel(reactive(antigen((%);(median((IQR)( 0+(20) 5+(40) Missing+ N=64+(9.4%) 38+(5.9%) HLA(mismatches(((N);(median((IQR) 3+(3) 5+(1) Missing+ N=200+(29.4%) N=116+(18%) Transplant(year((%)((( P P P Living(Donor(Relation(((((((((((((((((((((((((((((((((Parent Sibling Other+relative 21.4 N/A Unrelated Missing/Unknown 7.1 Donor(Age((years);(median((IQR) N/A 49+(20) Missing N=50+(7.8%) Cold(Ischemic(Time((hours);(median((IQR) N/A 13.2+(8.4) Missing N=187+(28.9%) Donor(Creatinine((mmol/L);(median((IQR) N/A 67+(30) Missing+ N=101+(15.6%) Donor(Cause(of(Death((((((((((((((((((((((((((((((((((CVA Anoxia Trauma N/A ICH Other Missing 9.6 Donor(History(of(hypertension((%)(((((((((++Yes N/A No 52.9 (((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((+++++Missing 18.6 Expanded(Criteria(Donor(Kidney((%)((((((((Yes N/A Unknown 7.8 Donation(after(Cardiac(Death((%)( N/A 11.5 Double(kidney(transplant((%)( N/A 9.5

63 Outcomes of interest The cumulative incidence curves for death with graft function, total graft failure and deathcensored graft failure are shown in Figures 9a, 9b and 9c, stratified by donor subtype. The cumulative probability of death with graft function is slightly higher for recipients of deceased donor subtype (log rank P=0.04). Graft survival also differs by donor subtype, with recipients of deceased donor kidney transplants having a higher cumulative incidence of total graft failure (log rank P= 0.01) and death-censored graft failure (log rank P=0.16). The cumulative incidences of death with graft function, total graft failure and death-censored graft failure at 3-, 5- and 10-years post-transplant are shown in Table 14. Figure 9a: Cumulative incidence of death with graft function by donor subtype Cumulative Probability of DWGF Deceased Donor Living Donor 0.00 Number at risk Deaceased Donor Living Donor Years Post Transplant

64 53 Figure 9b: Cumulative incidence of total graft failure by donor subtype 1.00 Deceased Donor Living Donor Cumulative Probability of TGF Number at risk Deaceased Donor Living Donor Years Post Transplant Figure 9c: Cumulative incidence of death-censored graft failure by donor subtype 1.00 Deceased Donor Living Donor Cumulative Probability of DCGF Number at risk Deaceased Donor Living Donor Years Post Transplant

65 54 Table 14: Cumulative incidence (%) of death with graft function, total graft failure and death-censored graft failure by donor subtype Deceased'Donor'KTR Outcome 3'years' 5'years 10'years DWGF DCGF TGF Living'Donor'KTR Outcome 3'years' 5'years 10'years DWGF DCGF TGF A higher proportion of recipients of deceased donor kidney transplant recipients experienced death with graft function and graft failure in the first year after transplant in comparison to recipients of living donor kidney transplant recipients (Table 15a). The final external validation cohort of 1,326 patients consisted of 645 recipients of deceased donor kidney transplants and 681 recipients of living donor kidney transplants. Similarly, over follow-up, there were a higher proportion of deceased donor kidney transplant recipients in comparison to living donor transplant recipients who experienced death with graft function and graft failure, as shown in Table 15b. Table 15a: Number of patients experiencing death with graft function, death censored graft failure and loss to follow up in the first year after transplantation, by donor subtype Total&Cohort&(N) DD&KTR&(N) LD&KTR&(N) Kidney&transplant&recipients&at&baseline 1, DWGF&<&1&year&post&transplant DCGF&<&1&year&post?transplant Follow?up&time&<&1&year Alive&with&functioning&graft&at&1&yr&post&transplant 1, KTR=0Kidney0transplant0recipient;0DD=0Deceased0donor;0LD=0Living0donor DCGF=0DeathGcensored0graft0failure;0DWGF=00Death0with0graft0function

66 55 Table 15b: Number of patients experiencing death with graft function or death censored graft failure after one-year post transplant, by donor subtype Total&Cohort&(N) DD&KTR&(N) LD&KTR&(N) Alive&with&functioning&graft&at&1&yr&post&transplant 1, DWGF&>&1&year&&post&tranxplant DCGF&>&1&year&postDtransplant DD=-Deceased-donor;-LD=-Living-donor Comparison of derivation and validation cohorts Baseline characteristics of patients in the external validation cohort and the derivation cohorts for the models selected for external validation are shown in Table 12, Appendix 4 and Appendix 5. In comparing the external validation cohort to the cohorts used to derive the original prediction models, there are several notable differences. The clinical prediction model of death with graft function published by Hernandez et al. was derived in a cohort of Spanish kidney transplant recipients from an earlier era of transplantation ( ) in comparison to the external validation cohort. In addition, the derivation cohort included very few (1.2%) recipients of living donor kidney transplants. Additionally, recipients were younger (46.3 years vs years), were more likely to be on hemodialysis prior to transplantation (90% vs. 64.4%) and were less likely to be diabetic prior to transplantation (5.7% vs. 22.7%) in comparison to the external validation cohort. A higher proportion of patients were hepatitis C positive (13.9% vs.1.8%). Recipient race and other comorbidities (CHD, CVD and PVD) were not reported in the derivation cohort. The mortality rate was reported to be 12.3% in the development cohort. Median follow up was 82 months (IQR 48 to 103 years). Kaplan-Meier survival curves for the cohort and cumulative incidence rates of the event of interest were not shown/reported. The clinical prediction model of total graft failure and death-censored graft failure published by Moore et al. was derived in a cohort of kidney transplant recipients from the UK. The LOTESS (Long Term Efficacy and Safety Surveillance) database was used for derivation of this prediction model, which is a Novartis-funded pharmacovigilance project that included kidney transplant recipients form 1995 to As a result, all patients were treated with cyclosporine and azathioprine. This is in contrast to the current practice at the Toronto General Hospital where the vast majority (i.e., > 90%) of patients are treated with tacrolimus and MMF. A total of 6.9% of

67 56 the development cohort included recipients of living donor kidney transplants. In comparison to the external validation cohort, patients in the development cohort were younger (46.2 vs years) and the vast majority were Caucasian (92.5% vs. 66.7%). 13.9% of patients had pretransplant diabetes mellitus. The death-censored graft failure rate was 8.1% and total graft failure rate was 15.2% in the development cohort, which is similar to the external validation cohort. Mean follow-up was 4.0 +/- 1.3 years Performance measures: Discrimination The c statistics of the models when validated in the external cohort are shown in Table 16 and Table 17. Overall, the prediction model of death with graft function in the entire cohort of kidney transplant recipients showed modest discrimination with a c statistic of 0.69 (95% CI ). In recipients of deceased donor and living donor kidney transplants, the c statistic was 0.65 (95% CI ) and 0.73 (95% CI ) respectively. Results of the sensitivity analyses show similar c statistics to the primary analyses (c statistics of ). Table 16: External validation c statistics for death with graft function Validation)for)DWGF Cohort)size)(N) Events)(N) C)Statistic) 95%)CI Validation)1) 913/ G0.75 Validation)2) 456/ G0.74 Validation)3) 456/ G0.81 Validation)4) 1098/ G0.77 Validation)5 1098/ G0.79 Validation)6 913/ G0.76 Validation)7 1148/ G0.78 Validation)8 1148/ G0.79 Validation)1:)Entire)cohort)of)KTR Validation)2:)Cohort)of)deceased)donor)KTR Validation)3:)Cohort)of)living)donor)KTR Validation)4:)Entire)cohort)KTR;)assumption)that)missing)Tac/MMF)values)indicate)these)patients)are)on)Tac/MMF Validation)5:)Entire)cohort)KTR;)assumption)that)missing)Tac/MMF)values)indicate)these)patients)are)not)on)Tac/MMF Vaidation)6:)Entire)cohort)KTR)with)last)available)urine)protein)and)creatinine)data)for)patients)with)missing)urine/creatinine)data Validation)7:)Entire)cohort)KTR;)assumption)of)validation)4)and)validation)6;)assumption)missing)HCV)indicates)negative)result Validation)8:)Entire)cohort)KTR;)assumption)that)missing)Tac/MMF)values)indicate)that)patients)are)on)Tac)and)not)on)MMF; validation)6)assumption;)assumption)missing)hcv)indicates)negative)result Similarly, the prediction models of total graft failure and death-censored graft failure also showed modest discrimination. For total graft failure, the c statistic for the entire cohort of total

68 57 graft failure was 0.67 (95% CI ). In the subset of recipients of deceased donor kidney transplant recipients, the c statistic was 0.69 (95% CI ) and 0.62 (95% CI ) in recipients of living donor kidney transplant recipients. There were no significant changes to the c statistics for each validation when the last available values of egfr, BUN and albumin were used to impute missing data. For death-censored graft failure, discrimination was slightly better with a c statistic of 0.74 (95% CI ) for the entire cohort, and a c statistic of 0.79 (95% CI ) in recipients of deceased donor kidneys and 0.68 (95% CI ) in recipients of living donor kidneys. Similarly, the c statistics did not change significantly when the last available values of egfr and albumin were used to impute missing data. Table 17: External validation c statistics for total and death-censored graft failure Validation)for)TGF Cohort)size)(N) Events)(N) C)Statistic) 95%)CI Validation)1) 968/ H0.73 Validation)2) 477/ H0.77 Validation)3) 491/ H0.70 Validation)4) 983/ H0.72 Validation)5 480/ H0.77 Validation)6 503/ H0.68 Validation)1:)Entire)cohort)of)KTR Validation)2:)Deceased)donor)KTR Validation)3:)Living)donor)KTR Validation)4:)Entire)cohort)KTR)with)last)available)albumin,)BUN)and)eGFR)data) Validation)5:)Deceased)donor)KTR)with)last)available)albumin,)BUN)and)eGFR)data Vaidation)6:)Living)donor)KTR)with)last)available)albumin,)BUN)and)eGFR)data Validation)for)DCGF Cohort)size)(N)) Events)(N) C)Statistic) 95%)CI Validation)1) 1105/ H0.82 Validation)2) 556/ H0.88 Validation)3 549/ H0.8 Validation)4 1123/ H0.82 Validation)5) 560/ H0.88 Validation)6) 563/ H0.80 Validation)1:)Entire)cohort)of)KTR Validation)2:)Deceased)donor)KTR Validation)3:)Living)donor)KTR Validation)4:)Entire)cohort)KTR)with)last)available)albumin)and)eGFR)data) Validation)5:)Deceased)donor)KTR)with)last)available)albumin)and)eGFR)data Vaidation)6:)Living)donor)KTR)with)last)available)albumin)and)eGFR)data TGF= Total graft failure; DCGF=Death-censored graft failure

69 58 Cumulative incidence curves were also derived for the cohort of patients included in Validation 1 for all three outcomes of interest (death with graft function, total graft failure and death-censored graft failure), with the cohort stratified by predicted probabilities/prognostic index into quintiles. These curves are shown in Appendix 6a, 6b and 6c Performance measures: Calibration Calibration was assessed both graphically and statistically. Bar graphs comparing the predicted and observed event rates are shown in Figure 10a, 10b and 10c. For death with graft function, there was agreement for between observed and predicted probabilities for the first and second quintile, however there was significant discordance in the third to fifth quintiles. Statistically, the model did not calibrate well in our external validation cohort, with a P value < Figure 10a: Predicted vs. observed probabilities by risk quintile for death with graft function 10! 9! Probability*of*event,*%* 8! 7! 6! 5! 4! 3! 2! 1! 0! 1! 2! 3! 4! 5! Predicted*risk*quintile* Predicted! Observed!! Quintile (N) Predicted/prob/(mean) Q Q Q Q Q Predicted/prob/(range) 0.28A A A A4.43 Observed/prob/(mean) A A A10.13 Observed/prob/(95%/CI) 0.12A A A11.34

70 59 For total graft failure, the model did not calibrate well, both graphically and statistically (P value < 0.001) in all quartiles. Figure 10b: Predicted vs. observed probabilities by risk quartile for total graft failure Probability*of*event,*%* 90" 80" 70" 60" 50" 40" 30" 20" 10" 0" 1" 2" 3" 4" Predicted*risk*quar7le* Predicted Observed Quartile (N) Predicted/prob/(mean) Q Q Q Q Predicted/prob/(range) 2.15? ? ? ?99.84 Observed/prob/(mean) Observed/prob/(95%/CI) 2.51? ? ? ?31.82 For death-censored graft failure, the model showed good calibration in the third and fourth quartiles, but did not calibrate well in the first two quartiles. Similarly, the model did not calibrate well statistically (P value < 0.001).

71 60 Figure 10c: Predicted vs. observed probabilities by risk quartile for death-censored graft failure 25! 20! Probabilty*of*event,*%* 15! 10! 5! Predicted! Observed! 0! 1! 2! 3! 4! Predicted*risk*quartile*! Quartile (N) Predicted/prob/(mean) Q Q Q Q Predicted/prob/(range) Observed/prob/(mean) Observed/prob/(95%/CI) Discussion The results of this external validation study show that the selected prediction models of death with graft function, total graft failure and death-censored graft failure did not perform well in a large contemporary cohort of Canadian kidney transplant recipients. Discrimination was modest, with c statistics of 0.69 and 0.67 for models of death with graft function and total graft failure, respectively. However, discrimination did appear to be slightly better for the model predicting death-censored graft failure with a c statistic of 0.74, which suggests that prediction of patient death is more challenging than graft failure in this population. Calibration was poor for all models selected for external validation, with discordance between observed and predicted probabilities for all models, although calibration did appear, at least graphically, to be acceptable in higher risk groups (quartiles 3 and 4) for death-censored graft failure. There are several reasons that may account for the suboptimal performance of these models in this population. First, the differences between the derivation and validation cohorts may have

72 61 influenced the performance of these models. For example, the models predicting total and deathcensored graft failure were derived in a population where the majority of patients were on a combination of cyclosporine and azathioprine as the immunosuppressive regimen of choice, which is a reflection of the transplant era when the cohort was assembled as well as the fact that the cohort was assembled as part of a Novartis-funded pharmacovigilance project. This is in contrast to the external validation cohort where patients assembled from year 2000 onwards are mostly on a combination of tacrolimus and MMF. The potential improvements in prognosis that may have occurred with changes in transplant care over time, including the changes in immunosuppressive regimens, may explain why a model does not perform well. Similarly, for the model predicting death with graft function, patient factors in the external validation cohort such as older age, higher proportion of patients of African descent and a higher comorbidity burden (i.e., the presence of diabetes), which all can result in inferior patient and graft outcomes, may also explain suboptimal external validation. If a model is predictive, differences in case-mix between derivation and validation cohorts should not influence the performance of a model in an external validation study. Rather, there may be a number of other factors or variables (that are often unknown or unmeasured) due to case-mix differences that may modify the relationship between the predictors in the model and the outcome of interest resulting in poorer performance. Second, although the cohort used for validation in this study is the largest single-centre cohort of kidney transplant recipients in Canada, the number of events is relatively small, and the sample size is particularly reduced when the cohorts were stratified by donor subtype for validation. Third, from a statistical perspective, the original models may have been overfitted or alternatively, a specific predictor may be of significant importance in the Canadian population of transplant recipients but less predictive in the derivation population and thus, omitted from the original model. This is difficult to ascertain but both situations can result in poor external validation. Finally, differences in the measurement of the predictors between development and validation cohorts may have occurred (i.e., various definitions for the diagnosis of diabetes exist). This information bias may also be a determinant of suboptimal external validation (53). In this external validation study, discrimination was modest and calibration was poor. There is a tendency when models do not perform well, to simply derive another model, often in an even smaller dataset that was used for derivation of the original model. However, other strategies, which may be more appropriate, such as updating of an existing model in a new population,

73 62 particularly if the model discriminates well but calibrates poorly should be considered. Recalibration (i.e., updating the intercept of a regression model and relative weights of the predictors) or model revision (i.e., potential inclusion of new predictors) should be attempted to improve performance (60, 125, 126). There are instances where the differences in case-mix are too extreme so recalibration may not be possible. It is important to note that the c statistic, which is a measure of discrimination, is a rank statistic and does not provide any actual information regarding the actual probability of an outcome. Therefore, a model may discriminate well, but calibrate poorly by providing a predicted probability that is far from the observed probability of an event. This is of particular importance in the kidney transplant setting, where clinicians largely use prediction models to make decisions based on a risk threshold (i.e., low- vs. highrisk) obtained from a predicted probability, which may in turn lead to changes in care or therapy (122). This is the first study, to our knowledge, which validates models of patient and graft survival in a large contemporary cohort of Canadian kidney transplant recipients. Shabir et al. (100) validated a prediction model derived in UK transplant recipients in a smaller cohort of 475 Canadian transplant recipients from Halifax, Canada. The results of this study highlight the importance of external validation of a model prior to clinical use, particularly in the Canadian transplant population. There are some limitations to this validation study that warrant discussion. First, one researcher selected studies and performed the data abstraction. Future work could involve study selection and data abstraction of a random subset of citations by a second selector and abstractor. Second, for validation of the clinical prediction models of total graft failure and death censored graft failure, the baseline hazard function for assessment of calibration was not available and therefore the baseline hazard function from another cohort was used. Although the populations were similar, using an alternate baseline hazard function may have resulted in poor calibration by deriving predicted probabilities that were inaccurate. Currently, the reporting of the baseline hazard function is inconsistent in the literature. However, this is critical element to permit an assessment of calibration, and should be included in published prediction models (109). Third, there were certain variables that were not completely captured in the validation cohort (i.e., proteinuria data) which resulted in a reduction of sample size for validation. Missing data is a common problem in prediction research, and problematic when individuals with missing data are systematically different to those with complete observations. However, in order to account for

74 63 this, multiple sensitivity analyses were performed which demonstrated that the results of the primary analyses were robust to the study assumptions. Fourth, the small number of events in the external validation cohort may have resulted in over-fitting, particularly where validation was done on subsets of the external validation cohort (i.e., living donor or deceased donor kidney transplant recipients). Fifth, although the predictor variables were manipulated in order to match the original models, there were certain assumptions made that may have resulted in information bias. For example, in the validation of the clinical prediction model of death with graft function, the predictor variable of proteinuria of >1g was included in the model. If quantification of proteinuria was not available in the external validation cohort, an approximation of the degree of proteinuria was obtained from a dipstick test, which is only a semi-quantitative assessment of proteinuria and subject to potential error. Finally, this study only focused on validation of selected clinical prediction models as described in section As a result, it is still unknown if other clinical prediction models would have performed well in this cohort of Canadian kidney transplant recipients if the necessary variables were available. Thus, conclusions about the performance of clinical prediction models in this external validation cohort are limited to the models that were actually validated. In summary, the results of this external validation study show that selected clinical prediction models of patient and graft survival perform suboptimally in a large Canadian cohort of kidney transplant recipients, although performance metrics were slightly better for death-censored graft failure than total graft failure or death with graft function. Efforts to update or recalibrate these models to achieve better prediction should be attempted prior to implementation of these models into clinical practice.

75 64 Chapter 4 4 Conclusions and future directions It is not surprising that there has been a significant increase in the number of clinical prediction models in the literature. The advantages of using these models to personalize patient care and achieve more consistent predictions in comparison to clinical judgment alone are appealing. However, despite the growth in the number of models, implementation into clinical practice in both the transplant and non-transplant setting has been limited due to a number of challenges, which have been highlighted in this thesis. The systematic review of clinical prediction models of patient and graft survival in kidney transplant recipients demonstrated that very few models have been externally validated and have been found to be clinically usable. Given the importance of external validation in establishing the generalizability of a model, it is difficult to justify clinical use without this step. Furthermore, published studies frequently did not report the baseline hazard function to allow for proper calibration of a model in an external population. Statistical methods to estimate a smooth baseline hazard function for the Cox proportional hazards models are beyond the scope of this thesis but are discussed extensively elsewhere (109). The performance of these models is modest, and in particular, there has not been a significant improvement in the predictive ability of these models over time. In particular, despite the many prediction models that continue to be developed and published in the literature, it appears that with current prediction modeling strategies, it is difficult to achieve significant improvement in the modest c statistics reported for prediction models of death and graft failure. Perhaps it is unrealistic to expect that prediction models, which are designed to be used in the early stages after transplantation, will yield high-quality predictions of events that occur over several years of follow-up. The post-transplant period is highly variable between patients and the occurrence of post-transplant events such as a rejection episode or infectious complications may have a significant impact on prognosis. Existing models do not currently account for the timedependency of risk factors over follow-up, but this is likely an area that warrants further study in

76 65 order to move the field of prediction modeling forward, particularly in the setting of kidney transplantation (122, 127). Interestingly, better performing models do not always translate into clinical practice. In the transplant population, the kidney donor risk index, which uses a combination of 15 donor and transplant variables, is commonly, although not ubiquitously, used to predict the risk of graft failure in kidney transplant recipients. However, this model has a c statistic of 0.67 and calibration was not reported (93). Another example is the Framingham risk score, which also has modest discrimination, but is widely used in multiple populations. A number of determinants such as the use of a model in the literature (i.e., in research studies), external validation of a model, the ease of use (i.e., a simple, parsimonious model with commonly available, easy to measure predictors) and the availability of a web-based or mobile application are likely to influence the use of a prediction model in clinical practice. Further study is needed to determine what factors drive the uptake of one model over another in the clinical setting, such as the availability of a clinical tool integrated into an electronic medical record for example. In addition, a better understanding on how (and when) to use more novel approaches (i.e., tree based models or neural networks) for prediction is needed, given that many clinicians have little familiarity with these methodologies. Focus groups with clinicians regularly caring for kidney transplant recipients would be helpful in providing insights into the factors that determine the integration of a clinical prediction model into their practice. Not infrequently, clinical prediction models cannot be validated due to the use of predictor variables in clinical models that are not routinely or systematically collected in external validation populations. Particularly, in this thesis work, three models of death with graft function and nine models of graft failure could not be validated due one or more variables missing from the external validation cohort. Due to the lack of external validation in CoReTRIS, it is not possible to ascertain whether these models would have performed well in this cohort. Amongst individual transplant centres, there is no consensus on which variables are important to collect for research purposes in kidney transplantation. A consortium or working group within the kidney transplant community, akin to the Chronic Kidney Disease Prognosis Consortium (CKD- PC), which focuses on data sharing efforts in order to develop and validate prediction models in larger cohorts, would be of benefit to overcome the existing limitations with respect to data collection and availability.

77 66 External validation of selected clinical prediction models in a large Canadian cohort of kidney transplant recipients showed modest discrimination and poor calibration. The reasons for this modest performance are unclear, but may be related to the differences in case-mix between the derivation and validation cohorts. In particular, the delivery of post-transplant care may modify transplant specific outcomes by centre. For example, the frequency of follow-up visits, laboratory monitoring and diagnostic testing (i.e., protocol biopsies, screening ultrasounds) are known to vary by physician, transplant centre, jurisdiction and/or country. These differences in practice, which are not readily measurable or completely captured in databases, may account for the suboptimal performance of clinical prediction models in an external cohort. Furthermore, predictors that may be important in a cohort of Canadian kidney transplant recipients may be omitted from the original derivation cohorts, which in this thesis work, consisted of patients from Spain and the United Kingdom. Finally, this thesis has focused on systematically reviewing existing models that have already been developed along with external validation of a subset of these models. However, the final but often overlooked step in prediction modeling is to conduct an impact study (60). Do clinical prediction models improve patient outcomes? Do they predict risk better than clinical judgment alone? Do they result in the provision of more cost-effective care? Impact studies are challenging, as they require a control group to determine if the implementation of a prediction model in practice (i.e., the intervention ) results in better outcomes. Furthermore, if patient outcomes or cost is studied, a follow-up period is required (60). Currently, impact studies are infrequently performed but are an important step in the adoption of prediction models for clinical decision-making, particularly when a model has been shown to perform reasonably well in an external validation. The integration of a clinical prediction model into practice may be even more attractive if impact studies can demonstrate that there is an economic advantage to using these models. Although the integration of a model early on would require additional resources, these costs may be justified if it can be shown that in the long-term, these models reduce health care costs due to the implementation of preventive or treatment strategies in individuals predicted to be at high risk of a disease or adverse outcome. Much of the emphasis of prediction modeling in kidney transplantation has been placed on model development. As a result, new prediction models for the same outcomes are commonly published in the literature, however, without much improvement in the performance of these

78 67 models and with infrequent implementation into clinical practice. The emphasis of prediction modeling has been misplaced. Alternate strategies to develop better predicting models (i.e., machine learning methods, models which can account for time-dependency of risk) should be the focus of prediction research in order to move the field forward, rather than continuing to use the same methods to develop yet another prediction model with modest performance. However, although better performing models are certainly desirable, there is no agreement on what constitutes a good performing model. From the perspective of discrimination, it is accepted that a model with a c statistic of greater than 0.5 is better than flipping a coin. However, the c statistic is a rank statistic, and a model that calibrates well is likely of grater importance as physicians commonly use probabilities of an event occurring to drive clinical decisions and counsel patients. However, whether prediction models are useful in the clinical context extends beyond performance metrics and rather, is a function of their performance in comparison to clinical judgment alone. If the current prediction models that exist, despite their seemingly modest performance are superior to current clinical judgment without a prediction model, then they are likely valuable. Conversely, if clinical judgment alone is superior or equivalent to using a prediction model, then the integration of a prediction model is unlikely to provide any added value in the clinical setting. In kidney transplant recipients, comparisons of predictions generated by clinicians in comparison those generated by prediction models have not, to my knowledge been done. This is a critical point that will likely influence the integration of models into practice and warrants further study.

79 68 References 1. Group KDIGOKCW. KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease.. Kidney Int Suppl.3: Arora P, Vasa P, Brenner D, Iglar K, McFarlane P, Morrison H, et al. Prevalence estimates of chronic kidney disease in Canada: results of a nationally representative survey. Cmaj. 2013;185(9):E Chadban SJ, Briganti EM, Kerr PG, Dunstan DW, Welborn TA, Zimmet PZ, et al. Prevalence of kidney damage in Australian adults: The AusDiab kidney study. J Am Soc Nephrol. 2003;14(7 Suppl 2):S Coresh J, Selvin E, Stevens LA, Manzi J, Kusek JW, Eggers P, et al. Prevalence of chronic kidney disease in the United States. Jama. 2007;298(17): Zhang L, Zhang P, Wang F, Zuo L, Zhou Y, Shi Y, et al. Prevalence and factors associated with CKD: a population study from Beijing. Am J Kidney Dis. 2008;51(3): Eknoyan G, Lameire N, Barsoum R, Eckardt KU, Levin A, Levin N, et al. The burden of kidney disease: improving global outcomes. Kidney Int. 2004;66(4): Coresh J, Astor BC, Greene T, Eknoyan G, Levey AS. Prevalence of chronic kidney disease and decreased kidney function in the adult US population: Third National Health and Nutrition Examination Survey. Am J Kidney Dis. 2003;41(1): Levey AS, Stevens LA, Schmid CH, Zhang YL, Castro AF, 3rd, Feldman HI, et al. A new equation to estimate glomerular filtration rate.[erratum appears in Ann Intern Med Sep 20;155(6):408]. Ann Intern Med. 2009;150(9): Astor BC, Matsushita K, Gansevoort RT, van der Velde M, Woodward M, Levey AS, et al. Lower estimated glomerular filtration rate and higher albuminuria are associated with mortality and end-stage renal disease. A collaborative meta-analysis of kidney disease population cohorts. Kidney Int. 2011;79(12): Gansevoort RT, Matsushita K, van der Velde M, Astor BC, Woodward M, Levey AS, et al. Lower estimated GFR and higher albuminuria are associated with adverse kidney outcomes. A collaborative meta-analysis of general and high-risk population cohorts. Kidney Int. 2011;80(1): Chronic Kidney Disease Prognosis C, Matsushita K, van der Velde M, Astor BC, Woodward M, Levey AS, et al. Association of estimated glomerular filtration rate and albuminuria with all-cause and cardiovascular mortality in general population cohorts: a collaborative meta-analysis. Lancet. 2010;375(9731): van der Velde M, Matsushita K, Coresh J, Astor BC, Woodward M, Levey A, et al. Lower estimated glomerular filtration rate and higher albuminuria are associated with all-cause

80 69 and cardiovascular mortality. A collaborative meta-analysis of high-risk population cohorts. Kidney Int. 2011;79(12): Reslerova M, Moe SM. Vascular calcification in dialysis patients: pathogenesis and consequences. Am J Kidney Dis. 2003;41(3 Suppl 1):S Middleton RJ, Parfrey PS, Foley RN. Left ventricular hypertrophy in the renal patient. J Am Soc Nephrol. 2001;12(5): Levin A. Anemia and left ventricular hypertrophy in chronic kidney disease populations: a review of the current state of knowledge. Kidney International - Supplement. 2002(80): Berman SJ, Johnson EW, Nakatsu C, Alkan M, Chen R, LeDuc J. Burden of infection in patients with end-stage renal disease requiring long-term dialysis. Clin Infect Dis. 2004;39(12): Sarnak MJ, Jaber BL. Mortality caused by sepsis in patients with end-stage renal disease compared with the general population. Kidney Int. 2000;58(4): Banerjee D, Ma JZ, Collins AJ, Herzog CA. Long-term survival of incident hemodialysis patients who are hospitalized for congestive heart failure, pulmonary edema, or fluid overload. Clin J Am Soc Nephrol. 2007;2(6): Collins AJ, Foley RN, Herzog C, Chavers BM, Gilbertson D, Ishani A, et al. Excerpts from the US Renal Data System 2009 Annual Data Report. Am J Kidney Dis. 2010;55(1 Suppl 1):S1-420, A Osthus TB, von der Lippe N, Ribu L, Rustoen T, Leivestad T, Dammen T, et al. Healthrelated quality of life and all-cause mortality in patients with diabetes on dialysis. BMC Nephrol. 2012;13: Goodkin DA, Bragg-Gresham JL, Koenig KG, Wolfe RA, Akiba T, Andreucci VE, et al. Association of comorbid conditions and mortality in hemodialysis patients in Europe, Japan, and the United States: the Dialysis Outcomes and Practice Patterns Study (DOPPS). J Am Soc Nephrol. 2003;14(12): USRDS: the United States Renal Data System. Am J Kidney Dis. 2003;42(6 Suppl 5): Slinin Y, Foley RN, Collins AJ. Calcium, phosphorus, parathyroid hormone, and cardiovascular disease in hemodialysis patients: the USRDS waves 1, 3, and 4 study. J Am Soc Nephrol. 2005;16(6): McFarlane P, Komenda P. Economic considerations in frequent home hemodialysis. Semin Dial. 2011;24(6): Canadian Institute for Health Information. Canadian Organ Replacement Register Annual Report: Treatment of End-Stage Organ Failure in Canada, 2003 to Ottawa, ON: CIHI;

81 Laupacis A, Keown P, Pus N, Krueger H, Ferguson B, Wong C, et al. A study of the quality of life and cost-utility of renal transplantation. Kidney Int. 1996;50(1): Wolfe RA, Ashby VB, Milford EL, Ojo AO, Ettenger RE, Agodoa LY, et al. Comparison of mortality in all patients on dialysis, patients on dialysis awaiting transplantation, and recipients of a first cadaveric transplant. N Engl J Med. 1999;341(23): Cameron JI, Whiteside C, Katz J, Devins GM. Differences in quality of life across renal replacement therapies: a meta-analytic comparison. Am J Kidney Dis. 2000;35(4): Matas AJ, Lawson W, McHugh L, Gillingham K, Payne WD, Dunn DL, et al. Employment patterns after successful kidney transplantation. Transplantation. 1996;61(5): Rabeau Y. The Economics of Kidney Failure. The Kidney Foundation of Canada Snoeijs MG, Schaubel DE, Hene R, Hoitsma AJ, Idu MM, Ijzermans JN, et al. Kidneys from donors after cardiac death provide survival benefit. J Am Soc Nephrol. 2010;21(6): Merion RM, Ashby VB, Wolfe RA, Distant DA, Hulbert-Shearon TE, Metzger RA, et al. Deceased-donor characteristics and the survival benefit of kidney transplantation. Jama. 2005;294(21): Rao PS, Merion RM, Ashby VB, Port FK, Wolfe RA, Kayler LK. Renal transplantation in elderly patients older than 70 years of age: results from the Scientific Registry of Transplant Recipients. Transplantation. 2007;83(8): Port FK, Bragg-Gresham JL, Metzger RA, Dykstra DM, Gillespie BW, Young EW, et al. Donor characteristics associated with reduced graft survival: an approach to expanding the pool of kidney donors. Transplantation. 2002;74(9): Crawford PW, Lerma EV. Treatment options for end stage renal disease. Prim Care. 2008;35(3):407-32, v. 36. Mandal AK, Snyder JJ, Gilbertson DT, Collins AJ, Silkensen JR. Does cadaveric donor renal transplantation ever provide better outcomes than live-donor renal transplantation? Transplantation. 2003;75(4): Hariharan S, Johnson CP, Bresnahan BA, Taranto SE, McIntosh MJ, Stablein D. Improved graft survival after renal transplantation in the United States, 1988 to N Engl J Med. 2000;342(9): Cecka JM. Kidney transplantation in the United States. Clin Transpl. 2008: Gjertson DW, Cecka JM. Living unrelated donor kidney transplantation. Kidney Int. 2000;58(2):491-9.

82 Tapiawala SN, Tinckam KJ, Cardella CJ, Schiff J, Cattran DC, Cole EH, et al. Delayed graft function and the risk for death with a functioning graft. J Am Soc Nephrol. 2010;21(1): Yarlagadda SG, Coca SG, Formica RN, Jr., Poggio ED, Parikh CR. Association between delayed graft function and allograft and patient survival: a systematic review and meta-analysis. Nephrol Dial Transplant. 2009;24(3): Singh SK, Cole E, Kim SJ. Delayed Graft Function and Kidney Transplantation in Kidney Transplantation: Practical Guide to Managment. New York: Springer Science and Business Media; Meier-Kriesche HU, Port FK, Ojo AO, Rudich SM, Hanson JA, Cibrik DM, et al. Effect of waiting time on renal transplant outcome. Kidney Int. 2000;58(3): Collins AJ, Foley RN, Chavers B, Gilbertson D, Herzog C, Ishani A, et al. US Renal Data System 2013 Annual Data Report. Am J Kidney Dis. 2014;63(1 Suppl):A Meier-Kriesche HU, Ojo AO, Port FK, Arndorfer JA, Cibrik DM, Kaplan B. Survival improvement among patients with end-stage renal disease: trends over time for transplant recipients and wait-listed patients. J Am Soc Nephrol. 2001;12(6): Pascual M, Theruvath T, Kawai T, Tolkoff-Rubin N, Cosimi AB. Strategies to improve long-term outcomes after renal transplantation. N Engl J Med. 2002;346(8): Chapman JR, Webster AC, Wong G. Cancer in the transplant recipient. Cold Spring Harb Perspect Med. 2013;3(7). 48. Fishman JA. Infection in solid-organ transplant recipients. N Engl J Med. 2007;357(25): Remuzzi G, Grinyo J, Ruggenenti P, Beatini M, Cole EH, Milford EL, et al. Early experience with dual kidney transplantation in adults using expanded donor criteria. Double Kidney Transplant Group (DKG). J Am Soc Nephrol. 1999;10(12): El-Zoghby ZM, Stegall MD, Lager DJ, Kremers WK, Amer H, Gloor JM, et al. Identifying specific causes of kidney allograft loss. Am J Transplant. 2009;9(3): Rao PS, Schaubel DE, Jia X, Li S, Port FK, Saran R. Survival on dialysis post-kidney transplant failure: results from the Scientific Registry of Transplant Recipients. Am J Kidney Dis. 2007;49(2): Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? Bmj. 2009;338:b Steyerberg EW. Clinical Predicition Models: A Practical Approach to Development, Validation, and Updating: Springer Science and Business Media; 2010.

83 Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97(18): Kamath PS, Wiesner RH, Malinchoc M, Kremers W, Therneau TM, Kosberg CL, et al. A model to predict survival in patients with end-stage liver disease. Hepatology. 2001;33(2): Gage BF, van Walraven C, Pearce L, Hart RG, Koudstaal PJ, Boode BS, et al. Selecting patients with atrial fibrillation for anticoagulation: stroke risk stratification in patients taking aspirin. Circulation. 2004;110(16): Tripepi G, Jager KJ, Dekker FW, Zoccali C. Testing for causality and prognosis: etiological and prognostic models. Kidney Int. 2008;74(12): Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. Bmj. 2009;338:b Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. Bmj. 2009;338:b Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. Bmj. 2009;338:b Halloran PF, Melk A, Barth C. Rethinking chronic allograft nephropathy: the concept of accelerated senescence. J Am Soc Nephrol. 1999;10(1): Kent DM, Rothwell PM, Ioannidis JP, Altman DG, Hayward RA. Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal. Trials. 2010;11: Hayward RA, Kent DM, Vijan S, Hofer TP. Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis. BMC Med Res Methodol. 2006;6: Kasiske BL, Chakkera HA, Roel J. Explained and unexplained ischemic heart disease risk after renal transplantation. J Am Soc Nephrol. 2000;11(9): Kasiske BL. Cardiovascular disease after renal transplantation. Semin Nephrol. 2000;20(2): Silver SA, Huang M, Nash MM, Prasad GV. Framingham risk score and novel cardiovascular risk factors underpredict major adverse cardiac events in kidney transplant recipients. Transplantation. 2011;92(2): Kim SJ, Schaubel DE, Fenton SS, Leichtman AB, Port FK. Mortality after kidney transplantation: a comparison between the United States and Canada. Am J Transplant. 2006;6(1):

84 Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol. 2008;61(11): Liao L, Mark DB. Clinical prediction models: are we building better mousetraps? J Am Coll Cardiol. 2003;42(5): Kong DF, Lee KL, Harrell FE, Jr., Boswick JM, Mark DB, Hlatky MA, et al. Clinical experience and predicting survival in coronary disease. Arch Intern Med. 1989;149(5): Gandara E, Wells PS. Diagnosis: use of clinical probability algorithms. Clin Chest Med. 2010;31(4): Sanders S Fau - Sanders S, Doust J Fau - Doust J, Glasziou P Fau - Glasziou P. A Systematic Review of Studies Comparing Diagnostic Clinical Prediction Rules with Clinical Judgment. PLoS One. 2015;10(6): Bandiera G, Stiell IG, Wells GA, Clement C, De Maio V, Vandemheen KL, et al. The Canadian C-spine rule performs better than unstructured physician judgment. Ann Emerg Med. 2003;42(3): Grove WM, Zald DH, Lebow BS, Snitz BE, Nelson C. Clinical versus mechanical prediction: a meta-analysis. Psychol Assess. 2000;12(1): Yealy DM, Auble TE. Choosing between clinical prediction rules. N Engl J Med. 2003;349(26): Adams ST, Leveson SH. Clinical prediction rules. Bmj. 2012;344:d Wong SS, N.L. W, R.B. H, Ramkissoonsingh R. Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. Proc AMIA Symp. 2003: Holland JL, Wilczynski NL, Haynes RB, Hedges T. Optimal search strategies for identifying sound clinical prediction studies in EMBASE. BMC Med Inf Decis Mak. 2005;5: Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7): D'Agostino R, Nam B. Evaluation of the Performance of Survival Analysis Models: Discrimination and Calibration Measures in Handbook of Statistics. San Francisco: Elsevier; Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology. 2014;25(1): Tangri N, Kitsios GD, Inker LA, Griffith J, Naimark DM, Walker S, et al. Risk prediction models for patients with chronic kidney disease: a systematic review. Ann Intern Med. 2013;158(8):

85 Hayden JA, Cote P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med. 2006;144(6): Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e Hernandez D, Rufino M, Bartolomei S, Lorenzo V, Gonzalez-Rinne A, Torres A. A novel prognostic index for mortality in renal transplant recipients after hospitalization. Transplantation. 2005;79(3): Hernandez D, Sanchez-Fructuoso A, Gonzalez-Posada JM, Arias M, Campistol JM, Rufino M, et al. A novel risk score for mortality in renal transplant recipients beyond the first posttransplant year. Transplantation. 2009;88(6): Lin RS, Horn SD, Hurdle JF, Goldfarb-Rumyantzev AS. Single and multiple time-point prediction models in kidney transplant outcomes. J Biomed Inform. 2008;41(6): Machnicki G, Pinsky B, Takemoto S, Balshaw R, Salvalaggio PR, Buchanan PM, et al. Predictive ability of pretransplant comorbidities to predict long-term graft loss and death. Am J Transplant. 2009;9(3): Soveri I, Holme I, Holdaas H, Budde K, Jardine AG, Fellstrom B. A cardiovascular risk calculator for renal transplant recipients. Transplantation. 2012;94(1): Holdaas H, Fellstrom B, Jardine AG, Holme I, Nyberg G, Fauchald P, et al. Effect of fluvastatin on cardiac outcomes in renal transplant recipients: a multicentre, randomised, placebo-controlled trial. Lancet. 2003;361(9374): Tiong HY, Goldfarb DA, Kattan MW, Alster JM, Thuita L, Yu C, et al. Nomograms for predicting graft function and survival in living donor kidney transplantation based on the UNOS Registry. J Urol. 2009;181(3): Akl A, Ismail Am Fau - Ghoneim M, Ghoneim M. Prediction of graft survival of livingdonor kidney transplantation: nomograms or artificial neural networks? Transplantation. 2008;86(10): Rao PS, Schaubel DE, Guidinger MK, Andreoni KA, Wolfe RA, Merion RM, et al. A comprehensive risk quantification score for deceased donor kidneys: the kidney donor risk index. Transplantation. 2009;88(2): Kasiske BL, Israni AK, Snyder JJ, Skeans MA, Peng Y, Weinhandl ED. A simple tool to predict outcomes after kidney transplant. Am J Kidney Dis. 2010;56(5): Foucher Y, Daguin P, Akl A, Kessler M, Ladriere M, Legendre C, et al. A clinical scoring system highly predictive of long-term kidney graft survival. Kidney Int. 2010;78(12):

86 Goldfarb-Rumyantzev AS, Scandling JD, Pappas L, Smout RJ, Horn S. Prediction of 3-yr cadaveric graft survival based on pre-transplant variables in a large national dataset. Clin Transplant. 2003;17(6): Moore J, He X, Shabir S, Hanvesakul R, Benavente D, Cockwell P, et al. Development and evaluation of a composite risk score to predict kidney transplant failure. Am J Kidney Dis. 2011;57(5): Krikov S, Khan A, Baird BC, Barenbaum LL, Leviatov A, Koford JK, et al. Predicting kidney transplant survival using tree-based modeling. Asaio J. 2007;53(5): Brown TS, Elster EA, Stevens K, Graybill JC, Gillern S, Phinney S, et al. Bayesian modeling of pretransplant variables accurately predicts kidney graft survival. Am J Nephrol. 2012;36(6): Shabir S, Halimi JM, Cherukuri A, Ball S, Ferro C, Lipkin G, et al. Predicting 5-year risk of kidney transplant failure: a prediction instrument using data available at 1 year posttransplantation. Am J Kidney Dis. 2014;63(4): Watson CJ, Johnson RJ, Birch R, Collett D, Bradley JA. A simplified donor risk index for predicting outcome after deceased donor kidney transplantation. Transplantation. 2012;93(3): Schnitzler MA, Lentine KL, Axelrod D, Gheorghian A, You M, Kalsekar A, et al. Use of 12-month renal function and baseline clinical factors to predict long-term graft survival: application to BENEFIT and BENEFIT-EXT trials. Transplantation. 2012;93(2): Thorogood J, Houwelingen JC, Persijn GG, Zantvoort FA, Schreuder GM, van Rood JJ. Prognostic indices to predict survival of first and second renal allografts. Transplantation. 1991;52(5): Tang H, Hurdle JF, Poynton M, Hunter C, Tu M, Baird BC, et al. Validating prediction models of kidney transplant outcome using single center data. Asaio J. 2011;57(3): Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med. 2000;19(4): Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Perel P, Schroter S, et al. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e Nunez E, Steyerberg EW, Nunez J. [Regression modeling strategies]. Rev Esp Cardiol. 2011;64(6): Hendriksen JM, Geersing GJ, Moons KG, de Groot JA. Diagnostic and prognostic prediction models. J Thromb Haemost. 2013;11 Suppl 1: Royston P, Altman DG. External validation of a Cox prognostic model: principles and methods. BMC Med Res Methodol. 2013;13:33.

87 Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med. 2006;144(3): Gill JS, Tonelli M. Penny wise, pound foolish? Coverage limits on immunosuppression after kidney transplantation. N Engl J Med. 2012;366(7): Ojo AO, Morales JM, Gonzalez-Molina M, Steffick DE, Luan FL, Merion RM, et al. Comparison of the long-term outcomes of kidney transplantation: USA versus Spain. Nephrol Dial Transplant. 2013;28(1): Meier-Kriesche HU, Ojo AO, Hanson JA, Cibrik DM, Punch JD, Leichtman AB, et al. Increased impact of acute rejection on chronic allograft failure in recent era. Transplantation. 2000;70(7): Meier-Kriesche HU, Arndorfer JA, Kaplan B. The impact of body mass index on renal transplant outcomes: a significant independent risk factor for graft failure and patient death. Transplantation. 2002;73(1): Austin PC, Tu JV, Ho JE, Levy D, Lee DS. Using methods from the data-mining and machine-learning literature for disease classification and prediction: a case study examining classification of heart failure subtypes. J Clin Epidemiol. 2013;66(4): Austin PC, Lee DS, Steyerberg EW, Tu JV. Regression trees for predicting mortality in patients with cardiovascular disease: what improvement is achieved by using ensemble-based methods? Biometrical Journal. 2012;54(5): Delen D, Walker G, Kadam A. Predicting breast cancer survivability: a comparison of three data mining methods. Artificial Intelligence in Medicine. 2005;34(2): Pencina MJ, D'Agostino RB, Pencina KM, Janssens AC, Greenland P. Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol. 2012;176(6): Pepe MS. Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol. 2011;173(11): Miller ME, Hui SL, Tierney WM. Validation techniques for logistic regression models. Stat Med. 1991;10(8): Harrell FE, Jr. Regression Modeling Strategies, with Applications to Linear Models, Logistic Regression, and Survival Analysis. New York: Springer Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1): Pencina MJ, D'Agostino RB, Sr., Song L. Quantifying discrimination of Framingham risk functions with different survival C statistics. Stat Med. 2012;31(15):

88 Famure O, Phan NA, Kim SJ. Health information management for research and quality assurance: the Comprehensive Renal Transplant Research Information System. Healthc Manage Forum. 2014;27(1): Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol. 2008;61(1): van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Stat Med. 2000;19(24): Chambless LE, Diao G. Estimation of time-dependent area under the ROC curve for long-term risk prediction. Stat Med. 2006;25(20):

89 78 Appendices Appendix 1: Glossary of terms Living Donor Kidney Transplantation: Donation of a kidney from a related or unrelated individual for transplantation to a suitable/compatible individual with ESRD. Deceased Donor Kidney Transplantation: Donation of a kidney from a deceased individual for transplantation to a suitable/compatible individual with ESRD. Pre-emptive kidney transplantation: Kidney transplantation that occurs in an individual prior to initiation of dialysis for ESRD. Typically, most commonly occurs with living donor kidneys due to the ongoing shortage of deceased donor kidneys for transplantation. Primary non-function: Permanent loss of allograft function starting immediately after transplantation. Causes include, but not limited to, venous or arterial thrombosis or hyperacute rejection of the allograft. Delayed graft function: Dysfunction of the allograft with or without oliguria in the early postoperative period. Many definitions exist, but most commonly, delayed graft function is defined as the need for dialysis in the first 7 days post-transplantation. Expanded Criteria Donor: A donor who dies of primary brain death but who, at the time of death is: Aged > 60 years OR aged 50 to 59 years with 2 of the following 3 criteria: Serum creatinine >1.5mg/dl, history of hypertension or death from cerebrovascular accident. Donation after Cardiac Death: Donor who does not meet criteria for primary brain death but where cessation of cardiac function has occurred prior to organ procurement.

90 79 Appendix 2a: Search strategy of clinical prediction models of patient and graft survival in Ovid MEDLINE Database: Ovid MEDLINE(R) <1946 to January Week > Search Strategy: Kidney Transplantation/ (77465) 2 (kidney* adj2 transplant*).mp. (80804) 3 (kidney* adj2 allotransplant*).mp. (200) 4 (kidney* adj2 graft*).mp. (3368) 5 (kidney* adj2 homotransplant*).mp. (97) 6 (kidney* adj2 retransplant*).mp. (107) 7 (renal adj2 transplant*).mp. (36559) 8 (renal adj2 allotransplant*).mp. (416) 9 (renal adj2 graft*).mp. (2794) 10 (renal adj2 homotransplant*).mp. (323) 11 (renal adj2 retransplant*).mp. (81) 12 or/1-11 (85749) 13 exp survival analysis/ (175031) 14 surviv*.mp. (848022) 15 kaplan-meier.mp. (48693) 16 (product-limit adj2 method*).mp. (363) 17 (hazard* adj2 model*).mp. (52376) 18 (cox adj2 model*).mp. (9742) 19 Survival/ (3835) 20 exp Mortality/ (271207) 21 mo.fs. (402453) 22 mortality.mp. (446782) 23 death*.mp. (563415) 24 fatal*.mp. (134694) 25 exp Death/ (112213) 26 (graft* adj1 fail*).mp. (7855) 27 (transplant* adj1 fail*).mp. (778) 28 Graft Survival/ (38424) 29 Graft Rejection/ (49673) 30 (graft* adj1 loss).mp. (4354) 31 or/13-30 ( ) and 31 (40943) 33 [Clinical Prediction Guides Sensitive Filter from Haynes et al] (0) 34 predict*.mp. (856734) 35 scor*.tw. (464015) 36 observ*.mp. ( ) or 35 or 36 ( ) and 37 (8326) 39 animals/ not (animals/ and humans/) ( ) not 39 (7768) 41 limit 38 to humans (7758) or 41 (7768) 43 from 42 keep (4999) 44 remove duplicates from 43 (4977) 45 from 42 keep (2766) 46 remove duplicates from 45 (2766) or 46 (7743) 48 limit 47 to yr="1990 -Current" (6787)

91 80 Appendix 2b: Search strategy of clinical prediction models of patient and graft survival in Ovid MEDLINE in-process and non-indexed citations Database: Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations <February 04, 2014> Search Strategy: (kidney* adj2 transplant*).mp. (1586) 2 (kidney* adj2 allotransplant*).mp. (1) 3 (kidney* adj2 graft*).mp. (128) 4 (kidney* adj2 homotransplant*).mp. (0) 5 (kidney* adj2 retransplant*).mp. (2) 6 (renal adj2 transplant*).mp. (1535) 7 (renal adj2 allotransplant*).mp. (5) 8 (renal adj2 graft*).mp. (110) 9 (renal adj2 homotransplant*).mp. (0) 10 (renal adj2 retransplant*).mp. (0) 11 surviv*.mp. (45210) 12 kaplan-meier.mp. (2597) 13 (product-limit adj2 method*).mp. (21) 14 (hazard* adj2 model*).mp. (1712) 15 (cox adj2 model*).mp. (871) 16 mortality.mp. (32614) 17 death*.mp. (32232) 18 fatal*.mp. (6474) 19 (graft* adj1 fail*).mp. (380) 20 (transplant* adj1 fail*).mp. (43) 21 (graft* adj1 loss).mp. (248) 22 predict*.mp. (94657) 23 scor*.tw. (43798) 24 observ*.mp. (212371) 25 or/1-10 (2687) 26 or/11-21 (99225) or 23 or 24 (318123) and 26 and 27 (302)

92 81 Appendix 2c: Search strategy of clinical prediction models of patient and graft survival in Embase Database: Embase <1974 to 2014 February 04> Search Strategy: exp kidney transplantation/ (112459) 2 (kidney* adj2 transplant*).mp. (114384) 3 (kidney* adj2 allotransplant*).mp. (8614) 4 (kidney* adj2 graft*).mp. (65209) 5 (kidney* adj2 homotransplant*).mp. (97) 6 (kidney* adj2 retransplant*).mp. (654) 7 (renal adj2 transplant*).mp. (52471) 8 (renal adj2 allotransplant*).mp. (510) 9 (renal adj2 graft*).mp. (4934) 10 (renal adj2 homotransplant*).mp. (322) 11 (renal adj2 retransplant*).mp. (112) 12 or/1-11 (133989) 13 exp survival/ (589580) 14 surviv*.mp. ( ) 15 kaplan meier method/ (31460) 16 kaplan-meier.mp. (59654) 17 (product-limit adj2 method*).mp. (571) 18 proportional hazards model/ (41550) 19 (hazard* adj2 model*).mp. (63253) 20 (cox adj2 model*).mp. (17280) 21 exp mortality/ (658391) 22 mortality.mp. (900305) 23 death*.mp. (833302) 24 fatal*.mp. (206479) 25 exp death/ (456031) 26 graft failure/ (18605) 27 (transplant* adj1 fail*).mp. (1275) 28 (graft* adj1 loss).mp. (7490) 29 or/13-28 ( ) and 29 (50024) 31 [Clinical Prediction Guides Sensitive Filter from Haynes et al] (0) 32 predict:.tw. ( ) 33 exp methodology/ ( ) 34 validat:.tw. (342972) or 33 or 34 ( ) and 35 (13160) 37 limit 36 to yr="1990 -Current" (12205) 38 limit 37 to (book or book series or conference abstract) (2680) not 38 (9525) 40 (exp animals/ or exp animal experimentation/ or nonhuman/) not ((exp animals/ or exp animal experimentation/ or nonhuman/) and exp human/) ( ) not 40 (9332) 42 limit 39 to human (9226) or 42 (9332) 44 from 43 keep (5000) 45 from 43 keep (4332) 46 remove duplicates from 44 (4804) 47 remove duplicates from 45 (4241) or 47 (9045)

93 82 Appendix 3a: Predictor variables included in clinical prediction models of death with graft function in deceased donor kidney transplant recipients Machnicki et al Recipient age -Recipient gender -Recipient race -Recipient ethnicity -Recipient BMI -Cause of ESRD -Time on dialysis -Recipient comorbidities (OPTN, CCS, Charlson, Elixhauser) -Donor age -Donor race -Donor ethnicity -Donor BMI -Donor comorbidities (OPTN, CCS, Charlson, Elixhauser) -Transplant year -Number of HLA mismatches -Sensitization of recipient -CMV sero-pairing Hernandez et al Recipient age -Pretransplant cardiovarscular disease -Serum Creatinine >2.5mg/dL at discharge -Left ventricular hypertrophy -Vascular calcifications -Diabetes before transplantation -Time on dialysis > 48 months -Acute tubular necrosis Soveri et al Recipient age -Recipient coronary heart disease -Recipient a previous smoker -Recipient a current smoker -Recipient Serum Creatinine -Recipient diabetes mellitus -Time on dialysis Lin et al Recipient age -Recipient gender -Recipient race -Recipient height -Recipient weight -Cause of ESRD -History of hypertension -History of diabetes mellitus -History of cardiovascular disease -Time between graft failure date and retransplantation (if applicable) -Dialysis modality prior to transplant -Predominant ESRD service -Primary source of pay for treatment -Donor type -Donor age -Donor gender -Donor race -Donor height -Donor weight -Donor cause of death -Number of HLA mismatches -Cold storage time -Procedure type Hernandez et al Recipient age -Pretransplant diabetes -Positive Hepatitis C antibiodies -NODAT at the first year -Serum Creatinine at the first year -Proteinuria >1g at the first year -Use of tacrolimus at the first year -Use of MMF at the first year ESRD=End-stage renal disease; CMV=Cytomegalovirus; BMI=Body mass index; HLA=Human leukocyte antigen; NODAT=New onset diabetes after transplantation

94 83 Appendix 3b: Predictor variables included in clinical prediction models of graft failure in deceased donor kidney transplant recipients Rao et al Donor age -Donor race -Donor hypertension -Donor diabetes -Donor serum creatinine -Donor cause of death -Donor height & weight -Donation after cardiac death -Donor Hepatitis C -Number of B HLA & DR HLA mismatches -Cold ischemic time -Enbloc and/or double kidney transplant Foucher et al Recipient gender -Donor creatinine -Recipient age -Creatinine at 3 months post transplant -Creatinine at 1 year post transplant -Proteinuria at 1 year post transplant -Acute Rejection in first year post transplant -Number of transplants Machnicki et al Recipient age, gender, race, ethnicity, BMI -Cause of ESRD -Time on dialysis -Recipient comorbidities (OPTN) -Donor age, race, ethnicity, BMI, comorbidities (OPTN, CCS, Elixhauser, Charleston) -Transplant year -Number of HLA mismatches & sensitization of recipient -CMV sero-pairing Kasiske et al Model at transplant -Donor age -Recipient race, age -Duration of renal replacement therapy -Primary cause of ESRD -Recipient hepatitis C Antibody positive -Donor history of hypertension -Primary insurance -Donor cause of death -Number of HLA mismatches Goldfarb-Rumyantev et al. Goldfarb-Rumyantev et al TBM Traditional statistical model -Recipient race -Donor age -Recipient weight -Cold ischemic time -Recipient height -Previous number of transplants -Recipient age -Number of HLA matches -Donor race -Cause of ESRD -Recipient gender -Number of HLA mismatches -Recipient BMI -Presence of diabetes mellitus in donor -Presence of hypertension in donor -Donor height -Donor/recipient BMI -Donor age, BMI -Recipient age, BMI -Number of HLA matches -Cold ischemic time -Recipient gender -Donor gender -Terminal serum creatinine -Number of transplants -Donor race -Recipient race -Dialysis modality -Donor hypertension -Donor diabetes -Duration of diabetes -Duration of hypertension -SPK -Transplant procedure -Transplant centre volume -Number of transplants for ESRD diagnosis Kasiske et al Model at 7 days post transplant -egfr at hospital discharge -DGF -Donor age -Primary cause of ESRD -Recipient race -Recipient age -Duration of renal replacement therapy prior to transplant -Retransplantation Moore et al DCGF -Recipient age -Recipient race -Acute rejection in first year after transplant -Serum albumin -Serum urea nitrogen -egfr -Percent decrease in egfr in first year after transplant TGF -Recipient age -Male recipient -Serum albumin -Serum urea nitrogen -egfr rate at 1 year post transplant -Percent decrease in egfr in first year after transplant Kasiske et al Model at 1 year post transplant -egfr at 1 st year post transplant -Recipient race -Hospitalization during first year post transplant -Primary cause of ESRD -Recipient age -Primary insurance Krikov et al Recipient race, gender, age, height, weight -Prior transplant & number or transplants -Time on waitlist -RRT modality -Percentage of time on PD -Number of RRT modalities -Combination of RRT modalities -Recipient comorbidity score -History of cardiovascular disease, unstable angina, diabetes, hypertension -Presence of Hepatitis B core antibodies & Hepatitis C antibodies -Peak & last PRA, number of HLA matches -Primary source of pay for medical service -Donor race, gender, age, height, weight -Donor type, CIT, MMF use

95 84 Brown et al Age at diabetes onset -Recipient age at transplant -Recipient BMI -Cardiac arrest -Cold ischemic time -Recipient Creatinine decline by >25% in first 24hrs -Donor blood type, age, BMI, cause of death, creatinine -Primary diagnosis -Donor diabetes, hypertension, cigarette use -Recipient diabetes -Dialysis at listing -Length of time on dialysis -Donation after cardiac death -Donor drug use, cocaine use, race & gender -Warm ischemic time -DGF -Graft thrombosis -Induction use, pump use -Procedure type -Maintenance immunotherapy -Number of HLA mismatches -Discharge creatinine -Creatinine at listing -Last PRA -Pretransplant dialysis -Recipient blood type, race, gender -Recurrent disease -Creatinine at transplant -Recipient hypertension -Anastomotic time Shabbir et al Urine ACR -egfr at 1 year post transplant -Acute rejection in first year -Recipient ethnicity -Recipient gender -Recipient age -Serum albumin* Thorogood et al Primary transplants -Female donor to male recipient -Number of HLA-B mismatches -Number of HLA-DR mismatches -Highest PRA -Recipient diabetes -Cold ischemic time -Donor age -Recipient age -Recipient blood group -Transplant centre Watson et al Donor age -Donor hypertension -Donor weight -Donor days in hospital -Donor adrenaline use Thorogood et al Re-transplants only -Female donor to male recipient -Number of HLA-B mismatches -Number of HLA-DR mismatches -Highest PRA -Donor age -Graft survival of first graft Shnitzler et al egfr at 1 year post transplant -Recipient age -Recipient gender -Pre-transplant diabetes -Recipient BMI -Cause of ESRD -Recipient diabetes -Prior transplant -Peak PRA -Number of HLA mismatches -Serum albumin at listing -Recipient race -Recipient year of transplant -Donor age -Donor hypertension -Donor CMV positive -Donor weight -Donor race -Donor cause of death -Acute rejection in 1 st year post transplant Lin et al Recipient age -Recipient gender -Recipient race -Recipient height -Recipient weight -Cause of ESRD -History of hypertension -History of diabetes mellitus -History of cardiovascular disease -Time between graft failure date and retransplantation (if applicable) -Dialysis modality prior to transplant -Predominant ESRD service -Primary source of pay for treatment -Donor type -Donor age -Donor gender -Donor race -Donor height -Donor weight -Donor cause of death -Number of HLA mismatches -Cold storage time -Procedure type

96 85 Appendix 3c: Predictor variables included in clinical prediction models of graft failure in living donor kidney transplant recipients Tiong et al Recipient age -Recipient gender -Recipient race -Recipient BMI -Donor age -Donor gender -Donor race -Donor BMI -Donor serum creatinine -Nephrectomy type -Number of HLA mismatches -Cause of ESRD -Use of depleting antibodies -Use of IL2 receptor antibodies -Azathioprine use -MMF use -Rapamycin use -CNI use -DGF* -Treatment of rejection in first 6 months* -egfr at 6 months post transplant! Akl et al Recipient age -Donor age -Number of HLA mismatches -Time to diuresis -Total steroid dose during first 3 moths post transplant -Primary immunosuppression -Acute tubular necrosis -Number of acute rejections during first 3 months post transplant -HLA A & B mismatch^ -HLA DR mismatch^ -Number of blood transfusions^ *Denotes additional variables included for model predicting graft survival at 6 months post transplant; ^Denotes additional variables included in artificial neural network (ANN). BMI=Body mass index; HLA=Human leukocyte antigen; ESRD=End-stage renal disease; IL2=Interleukin-2; MMF=Mycophenolate mofetil; CNI=Calcineurin inhibitor; DGF=Delayed graft function; egfr=estimated glomerular filtration rate

97 86 Appendix 4: Baseline characteristics of patients in development cohort of clinical prediction model of death with graft function TABLE 1. Patient characteristics of the modeling and testing population Modeling population, n 2542 Testing population, n 2476 Pretransplantation Donor characteristics Age (yr) Living donors (%) Non heart beating donor (%) Cardiovascular death (%) Recipient characteristics Age (yr) Male gender (%) Hemodialysis modality (%) Mean time on dialysis (mo) Pretransplant diabetes (%) Hypertension (%) Mean blood pressure (mm Hg) Dyslipidemia (%) Retransplant (%) Mean peak PRA (%) Mean CIT (hr) Hepatitis B virus (%) Hepatitis C virus (%) Mean HLA mismatches (A B DR) Mean weight at transplantation (kg) Posttransplantation DGF (%) Acute rejection (%) CMV infection at the first year (%) Smoking status at the first year (%) NODAT at the first year (%) Hypertension at the first year (%) Dyslipidemia at the first year (%) Use of statins at the first year (%) Use of ACEI/ARB II at the first year (%) Induction therapy with antibodies (%) Use of CsA at the first year (%) Use Tacrolimus at the first year (%) (Continued) P TABLE 1. Continued Modeling population, n 2542 Testing population, n 2476 Use of Aza at the first year (%) Use of MMF at the first year (%) Use of sirolimus at the first year (%) Mean Scr at the third month (mg/dl) Mean Scr at the first year (mg/dl) Mean weight at the first year (kg) Mean proteinuria at the first year (g/d) Proteinuria 1 g at the first year (%) 3-yr survival (%) Number of trasplants by year (1990, 1994, 1998, and 2002) 420, 572, 774, , 562, 756, Conversion factors to SI units: creatinine 88.4 ( mol/l). PRA, panel reactive antibody; CIT, cold ischemia time; CMV, cytomegalovirus; DGF, delayed graft function; HLA, human leukocyte antigen; NODAT, new onset of diabetes after transplantation; Scr, serum creatinine; MMF, mycophenolate mofetil; CsA, cyclosporine A; Aza, azathioprine; ACEI/ ARB II, angiotensin-converting enzyme inhibitors/angiotensin II receptor blockers. P Reproduced with permission from Hernandez, D. et al. A Novel Risk Score for Mortality in Renal Transplant Recipients Beyond the First Posttransplant Year. Transplantation. 2009;88: Copyright Wolters Kluwer Health, Inc. (86).

98 87 Appendix 5: Baseline characteristics of patients in development cohort of clinical prediction models of total and death-censored graft failure Reproduced with permission from Moore, J. et al. Development and evaluation of a composite risk score to predict kidney transplant failure. AJKD. 2011;57: Copyright Elsevier. (97)

99 88 Appendix 6a: Cumulative incidence curves for death with graft function (DWGF) for validation 1 cohort, stratified by quintiles of predicted probabilities Q2 Q3 Q4 Q5 Cumulative Probability of Death Number at risk Q Years post transplant Quintile 1 (Q1) Quintile 3 (Q3) Quintile 5 (Q5) Quintile 2 (Q2) Quintile 4 (Q4) Quintiles generated from predicted probabilities/prognostic index, with Q1 representing the lowest predicted probabilities and Q5 representing the highest predicted probabilities.

100 89 Appendix 6b: Cumulative incidence curves for total graft failure (TGF) for validation 1 cohort, stratified by quintiles of predicted probabilities Q2 Q3 Q4 Q5 Cumulative Probability of TGF Number at risk Q Years post transplant Quintile 1 (Q1) Quintile 3 (Q3) Quintile 5 (Q5) Quintile 2 (Q2) Quintile 4 (Q4) Quintiles generated from predicted probabilities/prognostic index, with Q1 representing the lowest predicted probabilities and Q5 representing the highest predicted probabilities.

101 90 Appendix 6c: Cumulative incidence curves for death-censored graft failure (DCGF) for validation 1 cohort, stratified by quintiles of predicted probabilities Q2 Q3 Q4 Q5 Cumulative Probability of DCGF Number at risk Q Years post transplant Quintile 1 (Q1) Quintile 3 (Q3) Quintile 5 (Q5) Quintile 2 (Q2) Quintile 4 (Q4) Quintiles generated from predicted probabilities/prognostic index, with Q1 representing the lowest predicted probabilities and Q5 representing the highest predicted probabilities.

102 91 Appendix 7: Notification of University Health Network (UHN) REB approval

103 92 Appendix 8: Notification of University of Toronto REB approval PROTOCOL REFERENCE # February 19, 2014 Dr. Joseph Kim DEPT OF HEALTH POLICY, MANAGEMENT & EVALUATION FACULTY OF MEDICINE Dr. Sunita K. S. Singh DEPT OF HEALTH POLICY, MANAGEMENT & EVALUATION FACULTY OF MEDICINE Dear Dr. Kim and Dr. Sunita K. S. Singh, Re: Administrative Approval of your research protocol entitled, "Clinical prediction models of patient and graft survival in waitlisted patients with end stage renal disease and kidney transplant recipients" We are writing to advise you that the Office of Research Ethics (ORE) has granted administrative approval to the above-named research protocol. The level of approval is based on the following role(s) of the University of Toronto (University), as you have identified with your submission and administered under the terms and conditions of the affiliation agreement between the University and the associated TAHSN hospital: Graduate Student research - hospital-based only Storage or analysis of De-identified Personal Information (data) This approval does not substitute for ethics approval, which has been obtained from your hospital Research Ethics Board (REB). Please note that you do not need to submit Annual Renewals, Study Completion Reports or Amendments to the ORE unless the involvement of the University changes so that ethics review is required. Please contact the ORE to determine whether a particular change to the University's involvement requires ethics review. Best wishes for the successful completion of your research. Yours sincerely, Daniel Gyewu REB Manager OFFICE OF RESEARCH ETHICS McMurrich Building, 12 Queen's Park Crescent West, 2nd Floor, Toronto, ON M5S 1S8 Canada Tel: Fax: ethics.review@utoronto.ca