Prediction of acute renal failure after cardiac surgery: retrospective cross-validation of a clinical algorithm

Transcription

1 Nephrol Dial Transplant (2003) 18: Original Article Prediction of acute renal failure after cardiac surgery: retrospective cross-validation of a clinical algorithm Bjørn O. Eriksen 1, Kristel R. S. Hoff 2 and Steinar Solberg 3 1 Section of Nephrology, Department of Medicine, University Hospital of North Norway, 2 Faculty of Medicine, University of Tromsø and 3 Department of Thoracic and Cardiovascular Surgery, University Hospital of North Norway, Tromsø, Norway Abstract Background. Acute renal failure (ARF) after cardiac surgery is associated with high costs and a poor prognosis. Based on the results of a large US study, an algorithm has been developed for predicting ARF from pre-operative risk factors. The aim of this study was to cross-validate this algorithm in a patient population from Europe, and to assess its usefulness as a clinical tool. Methods. All coronary bypass and valvular surgery patients from a 5-year period were included. Data on pre-operative risk factors for all patients who developed dialysis-dependent ARF and for a random sample of patients without ARF were retrospectively obtained from hospital databases and medical records. For each patient, a risk score for ARF was calculated on the basis of the algorithm. The sensitivity, specificity, positive and negative predictive values and area under receiver operating characteristic (ROC) curve of the score s ability to predict ARF were estimated. Results patients were included. The risk of ARF was 1.9%, and the area under the ROC curve For a risk score of 6 or higher, the sensitivity was 0.53, the specificity 0.71, the positive predictive value 0.03 and the negative predictive value Conclusions. The validity of the algorithm was confirmed in a population differing in several aspects from the US populations where it was developed. Although it is useful for estimating the risk of ARF in groups of patients, the low risk of ARF limits the algorithm s ability to predict the outcome for individual patients. Keywords: cardiac surgical procedures; coronary artery bypass; kidney failure acute; post-operative complications; receiver operating characteristic curve; sensitivity and specificity Correspondence and offprint requests to: Bjørn Odvar Eriksen, Section of Nephrology, Department of Medicine, University Hospital of North Norway, 9038 Tromsø, Norway. medboe@rito.no Introduction Acute renal failure (ARF) is a serious complication of cardiac surgery. In different studies, the risk of developing a need for dialysis in the post-operative period has varied between 0.5 and 15% [1 9]. The prognosis of ARF in this setting is poor, with mortality rates ranging from 28 to 64% [2,3,5,7,8]. These patients often have long stays in intensive care units and high costs [10]. Changes in the treatment of post-operative ARF have had little impact on the outcome, possibly because improvements in therapy have been balanced by increasing pre-operative comorbidity [11]. For this reason, the possibility of identifying patients with a high risk of ARF before surgery has been investigated. In 1997, Chertow et al. [3] published an algorithm for predicting ARF from pre-operative data based on a study of patients in the Veterans Administration Continuous Improvement in Cardiac Surgery Program (CICSP). The algorithm estimates the risk of ARF from a severity score obtained by adding points for separate risk factors. It was validated in an independent sample of patients and demonstrated good discrimination. In a later study, Fortescue et al. [4] validated the algorithm in the Quality Measurement and Management Initiative (QMMI) cohort. Again, the algorithm performed well in discriminating for the development of ARF. Cross-validation of risk-stratification algorithms is necessary before they can be accepted as useful in clinical practice [12,13]. So far, the CICSP algorithm has only been validated in patients in the US in the first half of the 1990s. The aim of the present study was to validate the algorithm in patients operated on at the University Hospital of North Norway between 1995 and Also, we wanted to compare risk factors for ARF after cardiac surgery with those found in the US cohorts and to assess the algorithm s usefulness by estimating the positive and negative predictive values of its predictions of ARF. # 2003 European Renal Association European Dialysis and Transplant Association

2 78 B. O. Eriksen et al. Subjects and methods Subjects All patients having undergone coronary artery bypass grafting anduor valvular surgery at the University Hospital of North Norway between the 1 January 1995 and the 31 December 1999 were identified retrospectively from the hospital s registers. Patients who had been operated more than once in the period were considered to be at risk for each new operation, except for re-operations taking place within 30 days of the primary operation. Patients who had simultaneously been subjected to other surgical procedures, who were receiving renal replacement therapy or who had active endocarditis at the time of operation were excluded. In order to be consistent with the CICSP study, patients with a serum creatinine )265 mmolul (3 mgudl) pre-operatively were also excluded [3,4]. Patients in need of renal replacement therapy within 30 days of the operation were considered to have experienced ARF as a complication. These patients were identified by review of medical records. A 33% random sample of the patients without ARF was drawn as controls, but because of financial limitations only 32% were actually examined. These 32% still constitute a valid random sample, and no bias was introduced by omission of the last 1%. Randomization was performed with a pseudo-random number generator. Data Age, gender and serum creatinine were obtained directly from hospital databases. Other data necessary to implement the CICSP algorithm were extracted from medical records by a medical student (K.R.S.H.) under the supervision of a nephrologist (B.O.E.). As in the validation based on QMMI data, left-ventricular ejection fraction -35% was used as a proxy for cardiomegaly [4]. For the other variables in the algorithm, we applied the definitions of risk factors from the CICSP study [3]. Creatinine clearance was estimated from serum creatinine, age, weight and gender by the formula of Cockcroft and Gault [14]. Operative mortality was defined as death from any cause within 30 days of surgery. Statistical methods A multivariate logistic regression analysis with ARF as the dichotomous dependent variable and the risk factors of the CICSP algorithm as the independent variables was performed with the SAS statistical package (SAS Institute, Cary, NC). Calibration of the model was assessed using the Hosmer Lemeshow goodness-of-fit test [15]. The CICSP additive severity score was calculated for each patient by adding points for each risk factor present, as specified in the study of Fortescue et al. [4]. The points associated with each type of risk factor were based on the odds-ratios (OR) found in the logistic regression analysis of risk factors in the CICSP study [3]. The score s sensitivity, specificity and positive and negative predictive value for predicting ARF for different cut-off levels of the score were calculated according to standard formulae as follows [16]: sensitivity~ number of patients with ARF and score cut-off total number of patients with ARF specificity~ number of patients without ARF and score - cut-off total number of patients without ARF positive predictive value~ number of patients with ARF and score cut-off total number of patients with score cut-off negative predictive value~ number of patients without ARF with score - cut-off total number of patients with score - cut-off For the calculation of the predictive values, the number of patients without ARF with a given score was estimated from the number of those randomly drawn by dividing this number by 0.32, i.e. the proportion of randomly drawn patients without ARF. A receiver operating characteristic curve (ROC curve) was constructed by plotting the sensitivity vs (1 specificity) for all possible cut-off levels. The ROC curve area was calculated by the non-parametric technique of Hanley and McNeill [17]. Ninety-five per cent confidence intervals (CI) of proportions were calculated as exact binomial CI. Missing data for dichotomous variables were coded as absent, and average values were substituted for missing continuous variables. Analyses were also repeated without patients with missing information. Statistical significance was set at Results In the study period, there were 2154 operations for aortocoronary bypass anduor valvular surgery. 117 operations were excluded for the following reasons: concurrent performance of other procedures (93), re-operations within 30 days of previous surgery (two), patients on renal replacement therapy (four) or pre-operative serum creatinine )265 mmolul (3 mgudl) (18). In all, 2037 operations were included in the study. Among these, seven patients had had two operations and were considered to be at risk of ARF for both. Thirty-eight of the operations were complicated by ARF, giving a risk of 1.9% (95% CI %). For patients with coronary bypass only, the risk was 1.2% (95% CI %), and for patients with valvular surgery with or without coronary bypass 4.9% (95% CI %). The 30-day mortality for patients with ARF was 45% (95% CI 29 61%), for patients without ARF 2.6% (95% CI %). Of the patients without ARF, 640 were randomly drawn as controls. A comparison between the baseline

3 Acute renal failure after cardiac surgery Table 1. Baseline characteristics of patients with and without ARF according to surgery type 79 Coronary artery bypass only Valvular surgery with or without coronary bypass ARF No ARF ARF No ARF Total 21 (100) 545 (100) 17 (100) 95 (100) Female sex 5 (24) 106 (19) 7 (41) 40 (42) Left ventricular ejection fraction (24) 24 (4) 2 (12) 10 (11) Chronic kidney disease 0 (0) 9 (2) 1 (6) 1 (1) Cerebrovascular disease 3 (14) 31 (6) 0 (0) 6 (6) Chronic obstructive pulmonary disease 4 (19) 20 (4) 0 (0) 6 (6) Diabetes mellitus 2 (10) 41 (8) 0 (0) 7 (7) NYHA functional class 4 4 (19) 51 (9) 2 (12) 13 (14) Peripheral vascular disease 2 (10) 40 (7) 2 (12) 8 (8) Prior heart surgery 0 (0) 16 (3) 5 (29) 9 (9) Prior percutaneous transluminal 2 (10) 24 (4) 0 (0) 1 (1) coronary angioplasty Prior myocardial infarction 15 (71) 313 (57) 5 (29) 14 (15) Pulmonary rales 5 (24) 50 (9) 7 (41) 9 (9) Intra-aortic ballon pump prior to surgery 0 (0) 5 (1) 1 (6) 4 (4) Estimated creatinine clearance mlumin P100 2 (10) 156 (29) 2 (12) 12 (13) (24) 150 (28) 5 (29) 15 (16) (38) 156 (29) 2 (12) 39 (41) (24) 73 (13) 6 (35) 26 (27) (5) 10 (2) 2 (12) 3 (3) Mean age Mean systolic blood pressure Mean diastolic blood pressure characteristics of patients with and without ARF according to type of surgery is presented in Table 1. A multivariate logistic regression analysis of ARF with the same independent variables as in the final logistic model of the CICSP study was performed (ns678). The following risk factors were associated Table 2. Multivariate logistic regression analysis of risk factors for ARF a with a higher risk of ARF (P-0.05): valvular surgery, left-ventricular ejection fraction -0.35, the presence of pulmonary rates and chronic obstructive pulmonary disease. From Table 2, it can be seen that the other risk factors found in the CICSP study were not statistically significant in our analysis. This was to be expected, as Present study CICSP Risk score Odds ratio 95% CI Odds ratio 95% CI Valvular surgery 4.1 ( ) 2.0 ( ) 3 Estimated creatinine clearance mlumin ) ( ) 1.3 ( ) ( ) 1.8 ( ) ( ) 3.4 ( ) ( ) 5.8 ( ) 9 Intra-aortic ballon pump prior to surgery 0.7 ( ) 3.2 ( ) 5 Prior heart surgery 3.5 ( ) 1.9 ( ) 3 NYHA functional class ( ) 1.6 ( ) 2 Peripheral vascular disease 1.4 ( ) 1.5 ( ) 2 Left ventricular ejection fraction ( ) 1.5 ( ) 2 Pulmonary rales 4.2 ( ) 1.4 ( ) 2 Chronic obstructive pulmonary disease 3.6 ( ) 1.3 ( ) 2 Systolic blood pressure ( ) 1.3 ( ) and valvular surgery 0.5 ( ) 1.4 ( ) and CABG 1.0 ( ) 0.7 ( ) 0 P160 and valvular surgery 1.1 ( ) 1.0 ( ) 0 P160 and CABG 0.7 ( ) 2.0 ( ) 3 a Comparison with results from the CICSP study.

4 80 B. O. Eriksen et al. Table 3. Sensitivity, specificity, positive and negative predictive values for different levels of the additive risk score for predicting ARF Score Number of patients with score Prediction of ARF for value equal to or greater than score in the study population Cases Controls Sensitivity Specificity Positive predictive value a Negative predictive value a Total a The predictive values were calculated after taking into account that the controls represent a random sample of the patients without ARF. the present study had insufficient statistical power to identify all the risk factors. In Table 2, the OR found in the CICSP study are listed for comparison. With one exception (pulmonary rales), the CICSP ORs fell within the 95% CI of the ORs in the present study. The Hosmer Lemeshow test demonstrated good calibration of the model (x 2 s8.40, dfs8, Ps0.40). When the analysis was repeated without patients with missing data, the results were not appreciably different. The points used for calculating the CICSP additive severity score are listed in Table 2. Table 3 shows its distribution. The score s sensitivity and specificity for predicting ARF were estimated for cut-off levels corresponding to each value of the score and plotted in a ROC curve shown in Figure 1. The area under the curve, which is a measure of the score s discriminatory power, was 0.71, compared with 0.76 in the CICSP study and 0.72 for the QMMI cohort [3,4]. In Table 3, the positive and negative predictive values of the score have been calculated for all possible cut-off levels. A cut-off level of 6, i.e. taking a score of 6 or higher to predict ARF, represents a reasonable compromise between sensitivity (0.53) and specificity (0.71). The positive predictive value for this cut-off was 0.03 and the negative predictive value The maximum positive predictive value for any cut-off was than patients [3]. Even so, cross-validation is a crucial step in the development of algorithms for clinical prediction. In the present study, the CICSP additive severity score algorithm was validated in a patient population from a different geographical region and time period than in the original study. Other important differences were a higher percentage of female patients and a lower percentage of patients with diabetes and previous cerebrovascular disease in our cohort. The fact that the algorithm performed well in spite of these differences is an indication of its robustness. The data for the cross-validation were collected retrospectively partly through medical record review. Although this design has the limitation that the model s performance may be favourably biased by differential retrieval of data, an analysis excluding patients with missing data gave no indication that this was the case. Discussion The risk of ARF after cardiac surgery has been the subject of numerous studies. Most of them have tried to assess pre-operative risk factors, but have included too few patients to be able to do this with sufficient statistical power. So far, the only study with adequate power has been the CICSP study, which included more Fig. 1. ROC curve for the prediction of ARF after cardiac surgery by the CICSP additive severity score.

5 Acute renal failure after cardiac surgery The algorithm has previously been validated in the QMMI cohort [4]. However, this cohort did not include valvular surgery. In the present study, 17% of the patients had undergone valvular surgery with or without surgery on the coronary arteries. Compared with the CICSP cohort, there was a higher risk of ARF among patients with valvular surgery in our hospital. The logistic regression showed that the OR for this type of surgery was higher (4.1 vs 2.0), although the 95% CI included the CICSP OR (Table 2). Another difference was a higher mortality in the CICSP cohort (for patients with ARF 64 vs 45%; for patients without ARF 4.3 vs 2.6%). These differences were both statistically significant. Since ARF and death are competing risks in this situation, one could speculate that the different risk of ARF for valvular surgery could be explained by a higher mortality for the US patients. Unfortunately, the mortality for valvular surgery was not given for the CICSP cohort [3]. As a part of the cross-validation, we wanted to explore the potential for clinical use of the algorithm. A clinical decision based on the algorithm would have to fix a cut-off level of the additive score for deciding which patients were eligible for some intervention, as e.g. withholding surgery or preventive measures. The probability of developing ARF for patients with a score higher than the cut-off is conventionally referred to as the positive predictive value, and the probability of not developing ARF with a lower score as the negative predictive value [16]. In Table 3, these probabilities have been calculated for all possible cutoff levels of the score. As can be seen, the maximum positive predictive value was 0.14, i.e. if the score was greater than the cut-off it was still far more likely that the patient would not develop ARF than the opposite. On the other hand, the negative predictive value was greater than or equal to 0.98 for all cut-off levels. These results indicate that the algorithm will not be very useful as a tool for making decisions about individual patients. The reason for this is the very low risk of ARF in both the CICSP- and QMMI-cohorts and in our material. As well as being functions of the sensitivity and specificity of the algorithm, the positive and negative predictive values depend on the risk of ARF. To identify these few patients accurately, an algorithm with almost perfect discriminatory power would be needed, which is rarely seen in this kind of clinical decision-making. On the other hand, the algorithm will be useful for estimating risk in groups of patients, e.g. when comparing results from different hospitals. We conclude that the CICSP algorithm for predicting post-operative ARF was valid for patients undergoing cardiac surgery at our hospital. The risk factors for ARF in the present study were consistent with the risk factors found in studies with greater statistical power. Although the low risk of ARF precludes the prediction of ARF for individual patients, the CICSP algorithm is useful as a tool for risk stratification in populations. In addition, the identification of risk factors lays the foundation for further research into the pathophysiological mechanisms of the condition and possible preventive measures. Acknowledgements. We are grateful to Jie Yang, Department of Clinical Research, and to Åshild Halvorsen, Department of Clinical Chemistry, both of the University Hospital of North Norway, for extracting data from hospital databases. References 1. Zanardo G, Michielon P, Paccagnella A et al. Acute renal failure in the patient undergoing cardiac operation. Prevalence, mortality rate, and main risk factors. J Thorac Cardiovasc Surg 1994; 107: Ostermann ME, Taube D, Morgan CJ, Evans TW. Acute renal failure following cardiopulmonary bypass: a changing picture. Intensive Care Med 2000; 26: Chertow GM, Lazarus JM, Christiansen CL et al. Preoperative renal risk stratification. Circulation 1997; 95: Fortescue EB, Bates DW, Chertow GM. Predicting acute renal failure after coronary bypass surgery: cross-validation of two risk-stratification algorithms. Kidney Int 2000; 57: Conlon PJ, Stafford-Smith M, White WD et al. Acute renal failure following cardiac surgery. Nephrol Dial Transplant 1999; 14: Suen WS, Mok CK, Chiu SW et al. Risk factors for development of acute renal failure (ARF) requiring dialysis in patients undergoing cardiac surgery. Angiology 1998; 49: Andersson LG, Ekroth R, Bratteby LE, Hallhagen S, Wesslen O. Acute renal failure after coronary surgery a study of incidence and risk factors in 2009 consecutive patients. Thorac Cardiovasc Surg 1993; 41: Frost L, Pedersen RS, Lund O, Hansen OK, Hansen HE. Prognosis and risk factors in acute, dialysis-requiring renal failure after open-heart surgery. Scand J Thorac Cardiovasc Surg 1991; 25: Schmitt H, Riehl J, Boseila A et al. Acute renal failure following cardiac surgery: pre- and perioperative clinical features. Contrib Nephrol 1991; 93: Mangano CM, Diamondstone LS, Ramsay JG et al. Renal dysfunction after myocardial revascularization: risk factors, adverse outcomes, and hospital resource utilization. The Multicenter Study of Perioperative Ischemia Research Group. Ann Intern Med 1998; 128: Aronson S, Blumenthal R. Perioperative renal dysfunction and cardiovascular anesthesia: concerns and controversies. J Cardiothorac Vasc Anesth 1998; 12: Halstenberg WK, Goormastic M, Paganini EP. Validity of four models for predicting outcome in critically ill acute renal failure patients. Clin Nephrol 1997; 47: Lins RL, Elseviers M, Daelemans R, De Broe ME. Problems in the development, validation and adaptation of prognostic models for acute renal failure. Nephrol Dial Transplant 2001; 16: Cockcroft DW, Gault MH. Prediction of creatinine clearance from serum creatinine. Nephron 1976; 16: Hosmer DW, Lemeshow S. Applied Logistic Regression. John Wiley & Sons, New York: Sackett DL, Haynes RB, Tugwell P. Clinical Epidemiology A Basic Science for Clinical Medicine. Little, Brown & Co., Boston, MA: Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: Received for publication: Accepted in revised form: