Cardiac surgery remains a very complex area for outcome

Transcription

1 Mortality Prediction in Cardiac Surgery Patients Comparative Performance of Parsonnet and General Severity Systems J. Martínez-Alario, MD; I.D. Tuesta, MD; E. Plasencia, MD; M. Santana, MD; M.L. Mora, MD, PhD Background Our purpose was to assess the performance of general severity systems (Acute Physiology and Chronic Health Evaluation [APACHE], Simplified Acute Physiology Score [SAPS], and Mortality Models [MPM]) and to compare them with the Parsonnet score to predict mortality after cardiac surgery. Methods and Results This was a prospective observational study of 465 cardiac surgery patients in a tertiary referral center. Probabilities of hospital death for patients were estimated by applying the 4 models and were compared with actual mortality rates. Performance of the 4 systems was assessed by evaluating calibration with the Hosmer-Lemeshow goodness-of-fit test and discrimination with receiver operating characteristic (ROC) curves. 2 values were 3.71 for Parsonnet, 4.52 for MPM II 0, 4.30 for MPM II 24, 5.16 for SAPS II, and for APACHE II. The area under the ROC curve was for Parsonnet, for MPM II 0, for MPM II 24, for SAPS II, and for APACHE II. Conclusions In our experience, the Parsonnet score performs very well, with calibration and discrimination very high, better than general severity systems, and it is an appropriate tool to assess mortality in cardiac surgery patients. In our experience, the general severity systems perform well to predict mortality after cardiac surgery, with high calibration of MPM II 24, MPM II 0, and SAPS II; minor calibration for APACHE II; and high discrimination for 3 general systems, but not as well as the Parsonnet score. (Circulation. 1999;99: ) Key Words: surgery mortality risk factors Cardiac surgery remains a very complex area for outcome prediction. However, there are important reasons for prognostic scoring in these patients. Outcome prediction models can be broadly classified as disease-specific or general. General severity of illness systems for critically ill patients, such as Acute Physiology and Chronic Health Evaluation (APACHE I, II, and III), Simplified Acute Physiology Score (SAPS I and II), and Mortality Models (MPM I and II) have matured through 3 generations in the past 2 decades. 1 7 Among the specific outcome prediction models, in 1989 Parsonnet et al 8 elaborated a method of uniform risk stratification for evaluation of the results of surgery in acquired adult heart disease. Although the patients undergoing cardiac surgery were excluded from the general severity systems, the aim of this work was to assess the performance of general severity systems in cardiac surgery patients and to compare these systems with the Parsonnet score to predict mortality after cardiac surgery. Methods The study was conducted in the intensive care unit of a tertiary referral center. A total of 465 consecutive patients undergoing cardiac surgery were studied during a period of 17 months from January 1, 1997, to May 31, Clinical and physiological data for general systems and Parsonnet score were prospectively collected according to the criteria and definitions described by the developers. Data were entered into a computerized database. hospital mortality rates were calculated by use of the logistic regression model suggested in the original articles, with 2 significant digits. The main outcome measurement was hospital mortality, defined as death occurring before hospital discharge. Probabilities of hospital death were compared with the actual outcome. The relationships of risk estimates among models was tested by linear regression analysis. Specific approval by the Ethics Committee was not required because data used for the study had already been collected for clinical purposes. Statistical Analysis The performances of the severity-of-illness scoring systems (APACHE II, SAPS II, MPM II 0, and MPM II 24 ) and the Parsonnet score in cardiac surgery patients were assessed by evaluation of calibration and discrimination. Calibration evaluates the degree of correspondence between the probabilities of mortality estimated by the severity system and the actual mortality experience. Calibration was assessed by the Hosmer-Lemeshow goodness-of-fit test (C statistic), which compares the number of observed and predicted deaths in deciles of risk covering the entire range of probabilities of death. 9,10 The expected or predicted number of nonsurvivors was obtained by summation of predicted mortality risks of all individuals in the decile; the expected number of survivors was the total number of individuals in the decile minus the expected nonsurvivors. 2 equals the sum of the squared difference between observed and expected numbers divided by the expected number [ (E O) 2 /E]. The smaller this value, the better the calibration. Received September 24, 1998; revision received February 16, 1999; accepted February 16, From the Department of Critical Care (J.M.-A., E.P., M.S., M.L.M.) and the Department of Cardiovascular Surgery (I.D.T.), Hospital Universitario de Canarias, Spain. Correspondence to Dr Ignacio Díaz de Tuesta, Servicio de Cirugía Cardiaca, Hospital Universitario de Canarias, La Laguna, SC Tenerife, Spain. tuesta@usa.net 1999 American Heart Association, Inc. Circulation is available at

2 Martínez-Alario et al May 11, TABLE 1. Hosmer-Lemeshow Goodness-of-Fit Test C for Parsonnet Score TABLE 3. II 24 Score Hosmer-Lemeshow Goodness-of-Fit Test C for MPM and Frequency Goodness-of-fit 2, ; df, 8;P For assessing discrimination, or the ability of the model to discriminate between patients who live and patients who die, we used 2 2 classification tables with decision criteria of 10%, 50%, and 90% and the area under the receiver operating characteristic (ROC) curve, computed by a modification of the Wilcoxon statistics, as proposed by Hanley and McNeil. 11 The areas under the ROC curves were compared by use of the z statistic, with correction for the correlation introduced by studying the same sample. 12 As a general rule, the larger the area under the ROC curve, the better the discriminatory capability of the model. This method is available for scores and probabilities but is meaningful only after the model has been shown to calibrate well. Results The operations performed were 241 (51.8%) CABG, 162 (34.8%) valve surgery, 48 (10.3%) double valve surgery, and 14 (3%) congenital heart disease. Patients undergoing aortic dissection were excluded from this analysis. There were 26 deaths, giving an overall hospital mortality rate of 5.59%. The sex ratio was 36.9% women (n 172) to 63.1% men (n 293), and the mean age was years. The mean Parsonnet score was The mean APACHE II score was TABLE 2. II 0 Score Hosmer-Lemeshow Goodness-of-Fit Test C for MPM and Frequency Goodness-of-fit 2, ; df, 8;P and Frequency Goodness-of-fit 2, ; df, 7;P The mean APACHE III score was The mean SAPS II score was The mean MPM II 0 was The mean MPM II 24 was Tables 1 through 5 show the Hosmer-Lemeshow goodnessof-fit test for Parsonnet, MPM II 0, MPM II 24, SAPS II, and APACHE II, respectively. The goodness-of-fit tables show 10 groups (deciles of risk), with increasing risk of mortality, which are distributed as expected survivors and expected nonsurvivors, as well as observed survivors and observed nonsurvivors. Low values of the Hosmer-Lemeshow C statistic and the corresponding high P values indicate good agreement between observed and expected number of deaths. The calibration of systems was , df 8, P for Parsonnet; , df 8, P for MPM II 0 ; , df 7, P for MPM II 24 ; , df 8, P for SAPS II; and , df 8, P for APACHE II. Table 6 presents the classification tables for the severity systems using decision criteria of 10%, 50%, and 90%. When a decision criterion of, for example, 10% was applied, predicted mortality risks 10% were considered as predict- TABLE 4. II Score Hosmer-Lemeshow Goodness-of-Fit Test C for SAPS and Frequency Goodness-of-fit 2, ; df, 8;P

3 2380 Mortality Predictors in Cardiac Surgery TABLE 5. Hosmer-Lemeshow Goodness-of-Fit Test C for APACHE II Score and Frequency Goodness-of-fit 2, ; df, 8;P ing hospital mortality, whereas predicted mortality risks 10% were considered as predicting survival. For each decision criterion, the true-positive rate or sensitivity (the proportion of the observed deaths correctly predicted to die), false-positive rate (the proportion of observed survivors incorrectly predicted to die), and the overall correct classification rate (proportion of patients correctly classified as survivors or nonsurvivors) are presented. With a decision criterion of 10%, sensitivity was 96% for Parsonnet, 65% for MPM II 0, 81% for MPM II 24, 65% for SAPS II, and 100% for APACHE II; the false-positive rate was 51%, 26%, 43%, 26%, and 94%, respectively; and the overall correct classification was 74%, 85%, 77%, 85%, and 19%, respectively. With a decision criterion of 50%, sensitivity was 11% for Parsonnet, 31% for MPM II 0, 46% for MPM II 24, 38% for SAPS II, and 54% for APACHE II; the false-positive rate was 0%, 1%, 1%, 1%, and 4%, respectively; and the overall correct classification was 95%, 95%, 96%, 95%, and 96%. With a decision criterion of 90%, sensitivity was 0% for Parsonnet, 8% for MPM II 0,0%for MPM II 24, 15% for SAPS II, and 8% for APACHE II; the false-positive rate was 0% for all systems; and the overall correct classification was 94%, 94%, 94%, 94%, and 95%, respectively. The Figure shows the area under the ROC curve. For Parsonnet, this was 0.857; for MPM II 0, 0.783; for MPM II 24, 0.796; for SAPS II, 0.771; and for APACHE II, TABLE 6. Classification Tables for Parsonnet and General Severity Systems With Decision Criteria of 10%, 50%, and 90% Parsonnet MPM II 0 MPM II 24 SAPS II APACHE II Decision Criteria Alive Dead Alive Dead Alive Dead Alive Dead Alive Dead 10% alive dead Sensibility Specificity Overall correct classification Positive predictive value Negative predictive value False-positive rate % alive dead Sensibility Specificity Overall correct classification Positive predictive value Negative predictive value False-positive rate % alive dead Sensibility Specificity Overall correct classification Positive predictive value Negative predictive value False-positive rate

4 Martínez-Alario et al May 11, Comparison of areas under ROC curves for Parsonnet (0.857), MPM II 0 (0.783), MPM II 24 (0.796), SAPS II (0.771), and APACHE II (0.803). Discussion Prognostic indices developed for a specific disease have a conceptual advantage over general systems because they focus on predictors that are specific or peculiar to that entity. A general prediction model is intended to estimate prognosis for a broad variety of diseases. For most diseases, physiological abnormalities are not limited to a single organ system. The large databases and computer analyses used in developing general prognostic systems have permitted testing of multiple predictor variables and of their empirical importance by use of regression analysis. This methodology allows these systems to include a minimum number of variables, to simplify data collection, and yet to maintain prognostic accuracy. The outcome to be measured must be relevant to clinicians, easily recognized, and well defined so as to be free of ascertainment bias. Hospital mortality is the outcome most commonly measured by currently available prognostic systems and meets all of these criteria. Hospital mortality is, and will remain, a highly relevant outcome for most physicians and patients. Conversely, the prognostic factors used to calculate the outcome are usually clinical and epidemiological information that the researcher identifies empirically or on the basis of previous studies as outcome-related factors. They should be obtained by objective methods and avoid any distortion by the observer. In case of subjective information, as symptoms described by the patient, efforts must be made to classify the information on the basis of standard criteria. Usually, the prognostic factors are easily obtained from the available clinical information of the patient and do not require special diagnostic procedures. Patients undergoing CABG were excluded from the APACHE data collection, and all cardiac surgery patients were excluded from the MPM and SAPS data collection. However, to predict hospital outcome, complications, and length of stay, recent articles analyze these systems in isolation or by combination of preoperative, intraoperative, and postoperative variables in cardiac surgery patients A variety of models to predict mortality after cardiac surgery have been developed from analysis of outcome, most of them in coronary artery surgery. Each model has been validated at the originating institution In general, predictive models perform better in the original setting than when transposed to other patient populations. 24 In our work, we assess the performance of general severity of illness scoring systems (APACHE II, SAPS II, and MPM II) for cardiac surgery patients and compare these systems with the Parsonnet score to obtain a good estimate of severity of illness and probability of hospital mortality. APACHE III was not included in the study because the equations of the model are not in the public domain and remain subject to copyright. For the Parsonnet score, the agreement between observed and expected in the 2 columns of survivors and nonsurvivors corresponds to a small value of the C statistic (good calibration) and therefore indicates that the Parsonnet score provided an adequate estimation of the probability of mortality in cardiac surgery patients. The Parsonnet score calibrates very well ( , df 8, P ). The discrimination of the Parsonnet score was very high; the area under the ROC curve was The evaluation of the performance of general severity of illness measures showed an adequate estimation of the mortality experience in cardiac surgery patients, but not as well as the Parsonnet score. For MPM II 24, MPM II 0, and SAPS II systems, low values of the Hosmer-Lemeshow C statistic and the corresponding high P value indicate good agreement between observed and expected number of deaths, and these systems also calibrate very well ( and P , and P , and P , in descending order of calibration). However, the APACHE II system did not calibrate as well with a higher 2 value ( and P ). The discrimination of the 3 general severity systems was good: the area under the ROC curve was for MPM II 0, for MPM II 24, for SAPS II, and for APACHE II. These differences may be clinically insignificant. However, the number of deaths was small, possibly weakening the predictive ability. Prognostic scoring is important because the physician and the patient need an idea of likely risk of hospital mortality rate. This information prepares the physician for complications and helps to stratify the patient. It also helps the patient and family to weigh the risk and benefits of surgery and clarifies their expectations. Accurate outcome data will result in better communication with patients and relatives, and the treatment is more likely to be consistent with the patient s value system. Conversely, as healthcare costs have increased, outcome assessment has become a major priority. Accurate and objective outcome prediction will allow for financial and human resources to be allocated appropriately. Resources will need to be dedicated to patients who are likely to benefit, and cost containment will be a constant pressure. Resolution of these conflicts will require specialists to be not only excellent physicians but also excellent managers. A prognostic system that establishes a predicted mortality rate for each unit based on a representative database and a patientby-patient measurement of risk allows comparison of observed

5 2382 Mortality Predictors in Cardiac Surgery versus predicted outcomes. The difference between actual and predicted death rates provides an outcome-based measure of quality of care and provides insight into means for improving performance. For many clinicians, the most important question regarding prognostic scoring systems is, how can they help with individual patient care decisions? Many physicians believe that group statistics do not apply to individuals. Although individual patients do have unique characteristics, they also share many common features with previous patients, and consideration of these similarities permits us to anticipate the patients responses and predict their outcomes. We do use past experiences every day when we choose one therapy over another, and we frequently base our decisions on the relative probability that a particular treatment will be successful in an individual patient. Statistical predictions of outcome produced by prognostic scoring systems are apparently at least as accurate as clinical predictions and in most cases are more reliable. These findings suggest that the predictions available from prognostic scoring systems could eventually be useful in aiding or supporting clinical judgment in decision making for individual patients. We can foresee a very interesting consequence from our results: whereas the Parsonnet model does not observe the effects of the surgical procedure, general-purpose systems do. However, the Parsonnet score, a specific-purpose system that considers only patient preoperative conditions, performs better than general-purpose systems (MPM, SAPS, and APACHE) that include not only risk factors but also information about the patient situation after the procedure. Although surgery generates the most significant and probably most aggressive change in patient evolution, in a standard institution the results may be better estimated through an accurate evaluation of the preoperative conditions of the patient, focused on the particular disease to be treated, rather than through a general evaluation of the patient and his or her situation after the procedure. In summary, in an institution with acceptable standards, the most important consideration to predict the outcome is the previous conditions of the patients, and not the immediate consequences of the procedure. In our experience, the Parsonnet score performs very well, with calibration and discrimination very high, better than general severity systems, and it is an appropriate tool to assess severity of illness in cardiac surgery patients, with applications to clinical practice and clinical research. In our experience, the general severity systems perform well to predict mortality after cardiac surgery, with high calibration of MPM II 24, MPM II 0, and SAPS II, in descending order, and minor calibration for APACHE II and high discrimination for 3 general systems, but not as well as the Parsonnet score. References 1. Knaus WA, Zimmerman JE, Wagner DP, Draper EA, Lawrence DE. APACHE: Acute Physiology and Chronic Health Evaluation: a physiologically based classification system. Crit Care Med. 1981;9: Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13: Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, Sirio CA, Murphy DJ, Lotring T. APACHE III prognostic system. Chest. 1991;100: Le Gall JR, Loirat P, Alperovitch A, Glaser P, Granthil C, Mathiev D, Mercier P, Thomas R, Villers D. A simplified acute physiology score for ICU patients. Crit Care Med. 1984;12: Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA. 1993;270: Lemeshow S, Teres D, Pastides H, Avrunin JS, Steingrub JA. A method for predicting survival and mortality of ICU patients using objectively derived weights. Crit Care Med. 1985;13: Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J. Mortality Models (MPM II) based on an international cohort of intensive care unit patients. JAMA. 1993;270: Parsonnet V, Dean D, Bernstein AD. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation. 1989;79(suppl I):I-3 I Lemeshow S, Hosmer DW. A review of goodness of fit statistics for use in the development of logistic regression models. Am J Epidemiol. 1982; 115: Hosmer DW, Lemeshow S. Applied Logistic Regression. New York, NY: John Wiley & Sons; Hanley J, McNeil B. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143: Hanley J, McNeil B. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148: Turner JS, Mudaliar YM, Chang RW, Morgan CJ. Acute Physiology and Chronic Health Evaluation (APACHE II) scoring in a cardiothoracic intensive care unit. Crit Care Med. 1991;19: Becker RB, Zimmerman JE, Knaus WA, Wagner DP, Seneff MG, Draper EA, Higgins TL, Estafanous FG, Loop FD. The use of APACHE III to evaluate ICU length of stay, resource use, and mortality after coronary artery bypass surgery. J Cardiovasc Surg Torino. 1995;36: Turner JS, Morgan CJ, Thakrar B, Pepper JR. Difficulties in predicting outcome in cardiac surgery patients. Crit Care Med. 1995;11: Hannan EL, Kilburn H Jr, O Donnell JF, Lukacik G, Shields EP. Adult open heart surgery in New York State: an analysis of risk factors and hospital mortality rates. JAMA. 1990;264: Higgins TL, Estafanous FG, Loop FD, Beck GJ, Blum JM, Paranandi L. Stratification of morbidity and mortality outcome by preoperative risk factors in coronary artery bypass patients: a clinical severity score. JAMA. 1992;267: O Connor GT, Plume SK, Olmstead EM, Coffin LH, Morton JR, Maloney CT, Nowicki ER, Levy DG, Tryzelaar JF, Hernandez F, Adrian L, Casey KJ, Bundy D, Soule DN, Marrin CA, Nugent WC, Charlesworth DC, Clough R, Katz S, Leavitt BJ, Wennberg J. Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery. Circulation. 1992;85: Tuman KJ, McCarthy RJ, March RJ, Najafi H, Ivankovich AD. Morbidity and duration of ICU day after cardiac surgery: a model for preoperative risk assessment. Chest. 1992;102: Edwards FH, Clark RE, Schwartz M. Coronary artery bypass grafting: the Society of Thoracic Surgeons National Database experience. Ann Thorac Surg. 1994;57: Hannan EL, Kilburn H Jr, Racz M, Shields EP, Chassin MR. Improving the outcomes of coronary artery bypass surgery in New York State. JAMA. 1994;271: Orr RK, Maini BS, Sottile FD, Dumas EM, O Mara P. A comparison of four severity-adjusted models to predict mortality after coronary artery bypass graft surgery. Arch Surg. 1995;130: Tu JV, Jaglal SB, Naylor CD, and the Steering Committee of the Provincial Adult Cardiac Care Network of Ontario. Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery. Circulation. 1995;91: Díaz de Tuesta I, Rufilanchas JJ, Cortina J, Renes E, Rodríguez E, Molina L, Pérez de la Sota E, Carrascal Y, Marolo L, Guillén F. A method for the predictive estimation of the surgical risk in adult cardiac pathology. Rev Esp Cardiol. 1995;48: