The Society of Thoracic Surgeons 30-Day Predicted Risk of Mortality Score Also Predicts Long-Term Survival

Transcription

1 ADULT CARDIAC The Society of Thoracic Surgeons 30-Day Predicted Risk of Mortality Score Also Predicts Long-Term Survival John D. Puskas, MD, Patrick D. Kilgo, MD, Vinod H. Thourani, MD, Omar M. Lattouf, MD, Edward Chen, MD, J. David Vega, MD, William Cooper, MD, Robert A. Guyton, MD, and Michael Halkos, MD Clinical Research Unit, Division of Cardiothoracic Surgery, Emory University School of Medicine, Atlanta, Georgia Background. The Society of Thoracic Surgeons Predicted Risk of Mortality (PROM) score is a well-validated predictor of 30-day mortality after cardiac procedures. This study investigated the ability of PROM to predict longer-term survival. Methods. From January 1, 1996, to December 31, 2009, 24,222 patients with PROM scores underwent cardiac procedures at an academic center. Long-term all-cause mortality was determined from the Social Security Death Index. Logistic and Cox survival regression analyses evaluated the long-term predictive utility of the PROM. Area under the receiver operator characteristic curve measured the discrimination of PROM at 1, 3, 5, and 10 years. Kaplan-Meier curves were stratified by quartiles of PROM risk to compare long-term survival. All analyses were performed for the whole sample and for 30-day survivors. Results. The overall 30-day mortality was 2.78% (674 of 24,222). PROM predicted 30-day mortality extremely well (area under the receiver operator characteristic, 0.794) and predicted longer-term survival almost as well. Among all patients and 30-day survivors, area under the receiver operator characteristic values for PROM at 1, 3, 5, and 10 years were remarkably similar to the 30-day end point for which PROM is calibrated. PROM was highly predictive of Kaplan-Meier survival for patients surviving beyond 30 days. Among 30-day survivors, each percent increase in PROM score was associated with a 9.6% increase (95% confidence interval, 9.3% to 10.0%) in instantaneous hazard of death (p < 0.001). Conclusions. The PROM algorithm accurately predicts death at 30-days and during 14 years of follow-up with almost equally strong discriminatory power. This may have profound implications for informed consent and for longitudinal comparative effectiveness studies. (Ann Thorac Surg 2012;93:26 35) 2012 by The Society of Thoracic Surgeons The Society of Thoracic Surgeons (STS) 30-day risk models were developed to provide clinicians and hospitals with a tool to evaluate risk-adjusted outcomes and to guide quality improvement initiatives. The scores themselves are simply predicted probabilities (ranging from 0 to 1) calculated from a multivariable logistic regression model calibrated on STS data within fixed time periods. The model coefficients are periodically updated to make the predictions commensurate with evolving technology and generally improved outcomes over time. The STS Predicted Risk of Mortality (PROM) score, most recently calibrated by Shroyer and colleagues [1], discriminates well between 30-day survivors and nonsurvivors (c index 0.78) and has a high degree of agreement between predicted and observed mortality (calibration). Accepted for publication July 21, Presented at the Forty-seventh Annual Meeting of The Society of Thoracic Surgeons, San Diego, CA, Jan 31 Feb 2, Address correspondence to Dr Puskas, Division of Cardiothoracic Surgery, Emory University, Emory University Hospital Midtown, Medical Office Tower, 6th Flr, 550 Peachtree St, Atlanta, GA 30308; john.puskas@emoryhealthcare.org. The PROM score can be calculated for five different procedures, including (1) isolated primary coronary artery bypass grafting (CABG), (2) isolated aortic valve replacement (AVR), (3) isolated mitral valve repair (MV repair) or replacement (MVR), (4) CABG combined with AVR, and (5) CABG and MV repair or MVR. The weighting of various risk factors is recalibrated with each new version of the STS Adult Cardiac Database according to the most recent data uploaded to the STS National Cardiac Database (NCDB) by the more than 900 participating cardiac surgical programs in the United States. PROM is used for analysis and comparison of clinical outcomes in comparative effectiveness research, in quality assurance initiatives, and recently, in various pay-forperformance programs. Other predictive algorithms have been widely applied to predict short-term outcomes after cardiac operations, including the Parsonnet score [2] and European System for Cardiac Operative Risk Evalu- Dr Thourani discloses that he has financial relationships with Medtronic and Maquet by The Society of Thoracic Surgeons /$36.00 Published by Elsevier Inc doi: /j.athoracsur

2 Ann Thorac Surg PUSKAS ET AL 2012;93:26 35 STS PROM SCORE PREDICTS LONG-TERM SURVIVAL ation (EuroSCORE) [3], but none have proved as accurate or been as rigorously recalibrated as the STS PROM score, which has become the global standard. Interestingly, although a great deal of effort has been focused on predicting short-term outcomes after cardiac operations driven by demands for improvement in operative and perioperative care processes there has been relatively little effort to develop a statistical algorithm to predict long-term survival after cardiac procedures. The additive and logistic EuroSCORE was predictive of longterm survival in 180 patients undergoing MV operations [4]; the additive but not the logistic EuroSCORE was predictive of midterm survival in 233 patients who had AVR and CABG [5]. The role of PROM in predicting longer-term survival has not been systematically investigated to date at least partly because the STS NCBD has not included data on survival beyond 30 days. Nonetheless, a reliable predictor of long-term survival would be enormously useful and would have powerful implications for patients, physicians, administrators, and society at large as decisions are made about individual treatments, alternative therapies, and health care funding. Further, such a score might be useful in risk adjustment when evaluating the longterm effects of different treatments. The goal of this study was to test and validate the PROM score as a predictor of long-term survival. Material and Methods This study was approved by the Emory University Institutional Review Board. Patients and Sample From January 1, 1996, to December 31, 2009, 30,636 patients underwent cardiac operations at Emory University hospitals. Of these, 24,222 (79.1%) had one of the five procedures for which PROM models have been developed: (1) isolated primary CABG, (2) isolated AVR, (3) isolated MV repair or MVR, (4) CABG combined with AVR, and (5) CABG combined with MV repair or MVR. MV repair patients were included only from 2008 forward because PROM was not calibrated for these patients before that date. Measurements Before analysis, the 30 preoperative risk factors used to calculate the PROM score were identified and harvested from the Emory University institutional STS Adult Cardiac Database. For descriptive purposes, each variable was summarized in the exact manner in which it is included in the PROM predictive model for each procedure type (Table 1). Patient age was used as a continuous and a dichotomous measure (age 66, age 66). Race was represented by three dichotomous variables: black, Hispanic, or other. Chronic lung disease, which had been measured by different scales during the study period, was summarized dichotomously in this study. Ejection fraction was dichotomized as less than 0.50 or 0.50 or greater. Status was measured as elective or nonelective. Previous incidence of sternotomy was measured dichotomously across two variables: first reoperation and multiple reoperations. New York Heart Association classification score was measured dichotomously as class IV or not class IV. Short-term operative mortality (30-day) was measured directly and extracted from the institutional STS database. Long-term all-cause mortality was determined by referencing the Death Master File via the U.S. National Social Security Death Index by a mechanism compliant with the Health Insurance Portability and Accountability Act. On March 14, 2010, the survival for each patient in this study was verified by querying the Social Security Death Index. Patients still alive on this date were considered censored in survival analyses. The sensitivity of the Social Security Death Index (92.2%) is comparable to that of the National Death Index among American-born individuals (87% to 98%) [6]. For analysis purposes, PROM was treated as a percentage (0 to 100) rather than a probability (0 to 1) so that meaningful interpretations of unit increases in risk could be posited. To evaluate the relationship between PROM and long-term survival end points, and to validate it for use as a predictor, a variety of analytic approaches were performed using logistic and survival regression methods. Model Performance Survival to fixed points (30 days, 1, 3, 5, and 10 years) was analyzed separately for patients who underwent operations early enough in the study period to observe the end point. In this manner, each eligible patient was classified as dead or alive at the specified time. Logistic regression models were constructed that related survival as a function of PROM for each time point and procedure combination and for all procedures combined. In all, 30 models were evaluated (5 time points 5 procedures all procedures combined), although the 30-day mortality models were only included for comparison purposes. Each model was evaluated with respect to discrimination and calibration. DISCRIMINATION. Discrimination refers to the model s ability to separate survivors and nonsurvivors and was assessed using the area under the receiver operating characteristic curve (AUROC). AUROC ranges from 0.50 to 1.00; higher values portend better discrimination, whereas values closer to 0.5 indicate that the model s discrimination is essentially random, like flipping a coin. One useful interpretation of the AUROC is as follows: If a randomly selected survivor is paired with a randomly selected nonsurvivor, then the AUROC is the probability that the nonsurvivor will have a higher model-predicted risk of death than the survivor. CALIBRATION. Calibration refers to the degree of agreement between observed and predicted outcomes. Normally, the Hosmer-Lemeshow statistic is recommended to evaluate calibration; however, the Hosmer-Lemeshow statistic is underpowered and overly sensitive to large sample 27 ADULT CARDIAC

3 ADULT CARDIAC 28 PUSKAS ET AL Ann Thorac Surg STS PROM SCORE PREDICTS LONG-TERM SURVIVAL 2012;93:26 35 Table 1. Preoperative Predictive Risk of Mortality Characteristics by Procedures Variables a Isolated Procedures CABG CABG AVR MV b AVR MV b (n 20,014) (n 1,781) (n 945) (n 1,059) (n 423) PROM percentage, mean (SD) 2.24 (3.32) 4.22 (4.90) 4.84 (6.51) 5.89 (5.49) 9.60 (8.26) Patient age Mean (SD) years 62.9 (10.9) 63.5 (14.6) 58.2 (14.3) 70.3 (10.1) 66.2 (10.5) 66 7,900 (39.5) 830 (46.6) 306 (32.4) 714 (67.4) 216 (51.1) Aortic stenosis c 249 (1.2) 877 (49.2) 17 (1.80) 588 (55.5) 9 (2.1) Black race a 2,894 (14.7) 278 (15.8) 221 (23.6) 99 (9.5) 72 (17.2) Body surface area, c mean (SD) m (0.26) 1.98 (0.28) 1.89 (0.26) 1.99 (0.25) 1.95 (0.28) Congestive heart failure 3,617 (18.1) 887 (49.8) 586 (62.0) 505 (47.7) 275 (65.0) Chronic lung disease 1,456 (7.3) 68 (3.8) 57 (6.0) 53 (5.0) 34 (8.0) Cerebrovascular accident 1,707 (8.5) 141 (7.9) 107 (11.3) 106 (10.0) 57 (13.5) Diabetes 7,249 (36.2) 404 (22.7) 152 (16.1) 331 (31.3) 124 (29.3) Ejection fraction 0.50 c 6,237 (35.4) 385 (26.7) 164 (21.0) 290 (32.2) 175 (47.3) Elective status 15,071 (75.3) 1,398 (78.5) 716 (75.8) 824 (77.8) 279 (66.0) First reoperation 1,140 (5.7) 259 (14.5) 203 (21.5) 140 (13.2) 57 (13.5) Hispanic race c 142 (0.7) 22 (1.3) 16 (1.7) 13 (1.3) 4 (1.0) Dyslipidemia 8,454 (42.2) 615 (34.5) 216 (22.9) 496 (46.8) 206 (48.7) Hypertension 15,485 (77.4) 1,239 (69.6) 537 (56.8) 830 (78.4) 331 (78.3) Pre-op IABP 678 (3.4) 4 (0.2) 13 (1.4) 13 (1.2) 20 (4.7) Immunosuppressive agent 548 (2.7) 88 (4.9) 48 (5.1) 45 (4.3) 18 (4.3) Left main 50% 4,723 (23.6) 25 (1.4) 8 (0.9) 144 (13.6) 60 (1.2) Male 14,355 (71.7) 1,075 (60.4) 386 (40.9) 765 (72.2) 251 (59.3) Mitral insufficiency c 4,716 (50.7) 847 (75.0) 799 (95.8) 514 (75.4) 376 (97.4) Multiple reoperations 95 (0.5) 35 (2.0) 51 (5.4) 9 (0.8) 5 (1.2) NYHA class IV c 2,619 (28.0) 218 (18.8) 133 (22.2) 142 (20.9) 109 (39.9) Other race c 428 (2.2) 17 (1.0) 28 (3.0) 13 (1.3) 11 (2.6) Myocardial infarction 10,282 (51.4) 245 (13.8) 108 (11.4) 305 (28.8) 218 (51.5) PTCA 6 hrs c 64 (0.3) 0 (0.0) 0 (0.0) 2 (0.2) 0 (0.0) Peripheral vascular disease 1,945 (9.7) 119 (6.7) 36 (3.8) 128 (12.1) 55 (13.0) Renal failure with dialysis 370 (1.9) 76 (4.3) 46 (4.9) 36 (3.4) 20 (4.7) Cardiogenic shock 263 (1.3) 12 (0.7) 18 (1.9) 9 (0.9) 18 (4.3) Smoker 5,196 (26.0) 290 (16.3) 167 (17.7) 164 (15.5) 110 (26.0) Triple-vessel disease c 12,379 (66.8) 105 (7.0) 50 (6.6) 363 (37.7) 195 (50.6) a Data are presented as number (%), unless otherwise indicated. b Includes mitral valve repair or mitral valve replacement. c Denotes that the variable contains some missing data. AVR aortic valve replacement; CABG coronary artery bypass graft; IABP intraaortic balloon pump; MV mitral valve; NYHA New York Heart Association; PROM Predictive Risk of Mortality; PTCA percutaneous transluminal coronary angioplasty; SD standard deviation. sizes [7, 8]. Instead, calibration curves, similar to those originally reported for PROM, were visually inspected for each model. Each curve is a scatter plot of observed and predicted probabilities of death averaged by decile of PROM. The connected points should closely track with the line of identity (y x). For each model, the odds ratio with its 95% confidence interval (CI) was noted, which represents the increase in odds of death at a fixed time point for each unit increase in PROM percentage. The point biserial correlation between survival (dichotomous) and PROM (numeric) was also reported for each model to observe how the correlation increased with the length of the fixed time point. Predictive Validation Validation in a large sample with only one predictor can be taken for granted in most cases because the real danger of a predictive model formulation is over-fitting (which by definition requires more than one predictor). However, to demonstrate that PROM is internally valid, two general approaches were used a bootstrapping validation and a split-sample validation [9p90 7]. The bootstrapping approach was used for each time point/procedure subset (hereafter, the original sample ) and involved repeated sampling with replacement of the eligible patients used for each model. A total of 1,000 bootstrap samples of size n (where n is the number of

4 Ann Thorac Surg PUSKAS ET AL 2012;93:26 35 STS PROM SCORE PREDICTS LONG-TERM SURVIVAL eligible patients for that model) were collected. For each bootstrap sample, a logistic regression model was fit and the model estimates (intercept and slope) were collected and applied to the original sample of size n to estimate predicted probabilities of death for each patient. These predicted probabilities were used as independent variables in the original sample and model performance statistics (AUROC and point biserial correlation) were computed. This process was completed 1,000 times, each time collecting the performance statistics. After all 1,000 bootstrap samples were analyzed and applied to the original sample, the 500th ordered value of each performance statistic was considered the best estimate of the true value of the statistic and the 25th and 975th ordered values served as 95% confidence bounds. Also, bootstrapped estimates of the model variables were collected for reporting purposes. The split-sample validation approach for each model consisted of dividing the original sample into two halves in a random fashion. Unlike bootstrapping, split-sample validation is not a resampling algorithm. The first sample (half of the original sample), called the test set, was used to fit a logistic regression model that related dichotomous survival to PROM. The model variables were collected and applied to the second half of the data, called the holdout set, and predicted probabilities of death were calculated for each patient in the holdout sample. These predicted probabilities were used to calculate model performance estimates were calculated, including AUROC and point biserial correlation, and calibration curves were generated for each model [9p78]. Because of the nature of the approach, no CIs were calculated for the performance statistics. Once estimates of the model performance statistics were calculated using the two validation approaches, they were compared with the analogous measures from the original sample. Survival Analysis To evaluate PROM as a predictor of long-term survival, PROM was divided into deciles of risk and 10 Kaplan- Meier curves were constructed by decile. Similar curves were constructed separately for 30-day survivors to assess PROM s predictive validity apart from the early deaths for which PROM was originally intended. The PROM score was further evaluated in a Cox proportional hazards regression model for each procedure and for all procedures combined (6 models total). Associated hazard ratios (HR) and 95% CI were computed. The proportional hazards assumption was checked by examining the correlation between ranked survival time and the Schoenfeld residuals for uncensored patients [10p151 3]. Adjusted survival estimates were generated from the Cox models, and median survival was estimated for various values of PROM [10p103 7]. A smoothing algorithm from a quadratic regression equation was used to create curves where estimated median survival was calculated for each value of PROM. Results The patient sample included 24,222 patients who underwent isolated CABG, 20,014 (82.6%); isolated AVR, 1,781 (7.4%); isolated MV procedures, 945 (3.9%); CABG MVR, 423 (1.8%); and CABG AVR, 1,059 (4.4%). Preoperative characteristics that informed PROM are listed in Table 1 by procedure. Patients undergoing concomitant CABG with valves procedures had the highest average PROM scores, tended to be older, and exhibited more preoperative comorbidities. Overall 30-day mortality was 2.8% (674 of 24,222). As expected, PROM discriminated 30-day mortality moderately well overall (AUROC 0.794). In the isolated CABG group, PROM exhibited comparable discrimination of 30-day mortality (AUROC 0.769) to that of the original STS cohort on which the PROM score was calibrated (AUROC 0.780) [1]. PROM also discriminated well in the other procedure subgroups of isolated AVR (AUROC 0.763), isolated MV (AUROC 0.816), CABG AVR (AUROC 0.749) and CABG MV (0.717). These results suggest that this patient population is generally representative of the larger, national population with respect to early survival risk (Table 2). For longer-term survival, the PROM score showed excellent face validity, as demonstrated in Kaplan-Meier analysis. After classifying PROM into deciles of risk, Kaplan-Meier curves were generated for each decile. The 10 curves were remarkably sequential, with the first decile having the best survival and each subsequent decile having survival lower than the previous (Fig 1). Further, the same analysis was performed separately for 30-day survivors, the end point for which PROM was originally designed. Strata of PROM deciles exhibited similar sequentiality in 30-day survivors, suggesting a predictive robustness of PROM beyond its original intent (Fig 2). Survival estimates, overall and by procedure, are reported in Table 3. Patients undergoing concomitant CABG procedures generally had poorer long-term survival than their counterparts undergoing isolated operations. When survival to fixed times of 1, 3, 5, and 10 years was considered, PROM demonstrated an only slightly diminished ability to discriminate between survivors and nonsurvivors. For all patients, AUROC was for survival to 30 days, to 1 year, to 3 years, to 5 years, and to 10 years. This similarity remained intact in the procedure subgroups of CABG and AVR, with the lowest AUROC at more distant time points seldom meaningfully lower (and sometimes higher) than AUROC at 30 days in isolated CABG cases (0.769 to 0.755), isolated AVR (0.763 to 0.790), and CABG AVR (0.749 to 0.728). Where MV procedures are performed without CABG, there is a more pronounced decline in discrimination for isolated MV (0.816 to 0.741), and CABG MV (0.717 to 0.626). PROM is weakest as a predictor of long-term survival in CABG MV patients and is a relatively poor discriminator of longer-term survival in these patients (Table 2). For each unit increase in PROM percentage, the additional increase in odds of death at each fixed time 29 ADULT CARDIAC

5 ADULT CARDIAC 30 PUSKAS ET AL Ann Thorac Surg STS PROM SCORE PREDICTS LONG-TERM SURVIVAL 2012;93:26 35 Table 2. Survival and Predictive Ability by Procedure for Fixed Postoperative Points in Time Fixed Survival End Point by Procedure Deaths/total (%) a OR (95% CI) for Death for Each Unit Increase in PROM Percentage AUROC Spearman Rank Correlation Isolated CABG 30 days 423/20,014 (2.1) 1.12 ( ) year 1,037/19,053 (5.4) 1.16 ( ) years 1,696/16,384 (10.4) 1.21 ( ) years 2,244/13,777 (16.3) 1.29 ( ) years 2,368/6,781 (34.9) 1.89 ( ) Isolated AVR 30 days 76/1,781 (4.3) 1.11 ( ) year 154/1,582 (9.7) 1.15 ( ) years 205/1,273 (16.1) 1.23 ( ) years 231/1,013 (22.8) 1.31 ( ) years 175/465 (37.6) 1.44 ( ) Isolated MV b 30 days 49/945 (5.2) 1.12 ( ) year 90/852 (10.6) 1.12 ( ) years 124/682 (18.2) 1.13 ( ) years 148/572 (25.9) 1.15 ( ) years 124/305 (40.7) 1.15 ( ) CABG AVR 30 days 83/1,059 (7.8) 1.12 ( ) year 158/983 (16.1) 1.17 ( ) years 193/788 (24.5) 1.20 ( ) years 204/595 (34.3) 1.30 ( ) years 173/282 (61.4) 1.43 ( ) CABG MV b 30 days 43/423 (10.2) 1.08 ( ) year 77/381 (20.2) 1.10 ( ) years 95/281 (33.8) 1.09 ( ) years 97/222 (43.7) 1.08 ( ) years 70/111 (63.1) 1.06 ( ) All procedures 30 days 674/24,222 (2.8) 1.13 ( ) year 1,516/22,851 (6.6) 1.16 ( ) years 2,313/19,408 (11.9) 1.20 ( ) years 2,924/16,179 (18.1) 1.26 ( ) years 2,910/7,944 (36.6) 1.54 ( ) a For each end point, only those patients with operation dates early enough to observe the survival end points were studied. b Mitral valve includes mitral valve replacement or repair. AUROC area under the receiver operating characteristic curve; AVR aortic valve replacement; CABG coronary artery bypass graft; CI confidence interval; MV mitral valve; OR odds ratio; PROM predicted risk of mortality. point was calculated. For the entire sample, the OR generally increased for longer-term survival end points, being at its highest at 10 years. This was also true in the AVR and CABG procedures. The ORs at 1 and 10 years, respectively, were 1.12 and 1.89 for isolated CABG, 1.11 and 1.44 for isolated AVR, and 1.12 and 1.43 for CABG AVR. Changes in the OR over time were not necessarily sequential, nor as pronounced, in the MV subgroups, at 1 and 10 years, respectively, 1.15 and 1.15 for isolated MV, and 1.08 and 1.06 for CABG MV (Table 2). Results from the internal validation techniques confirmed the discriminative ability of PROM for fixed survival time points. Results from the split-sample validation are consistent with the original sample and the bootstrap results with respect to the AUROC and the Spearman coefficients. In addition, in the split-sample approach, calibration curves were fit to each model to assess observed and predicted rate agreement. Calibration for each fitted model was moderate to good for the entire cohort and generally moderate for procedure subgroups. The inclusion of a

6 Ann Thorac Surg PUSKAS ET AL 2012;93:26 35 STS PROM SCORE PREDICTS LONG-TERM SURVIVAL Fig 1. Long-term Kaplan-Meier survival estimates are shown by decile of the predictive risk of mortality (PROM) score. 31 ADULT CARDIAC squared PROM term into the model markedly improved calibration when applied to the holdout sample, while not affecting discrimination. This suggests a curvilinear association between PROM and the log odds of survival at each end point, probably because PROM was originally built to address early perioperative risk, which is known to be higher due to the trauma of the procedure (Fig 3). A more thorough and statistically comprehensive discussion of the validation and calibration algorithms used is presented in an unabridged version of this manuscript at: PROM was further evaluated as a predictor of longterm survival in Cox proportional hazards models where it was determined to be highly associated with survival (HR, 1.066; 95% CI, to 1.070, p 0.001). This relationship persisted even among 30-day survivors (HR, 1.065; 95% CI, to 1.070, p 0.001). Median survival curves across all levels of PROM generally revealed higher median survival for isolated CABG and MV procedures and markedly lower long-term survival for AVR. Figure 4 allows the surgeon to predict the median survival of any patient undergoing one of the five procedures for which PROM models have been developed. Comment For many legitimate reasons, much attention and study has been focused on early outcomes after cardiac operations. Risk-adjusted comparisons of early outcomes between groups of patients undergoing specific cardiac procedures have been a fundamental part of clinical research, comparative effectiveness studies of alternative techniques and therapies, quality improvement initiatives, as well as institutional, local, regional, and national benchmarking in cardiac operations. Moreover, the ability to predict with reasonable accuracy the short-term risk of death after cardiac procedures is essential to the process of informed consent, which is a foundation of the surgeon-patient relationship. However, as the United States population ages, decisions regarding acute surgical intervention depend increasingly on our imperfect ability to predict long-term survival after cardiac operations. This applies to the individual patient and to society as a whole as difficult choices are made individually and on a societal level. To our knowledge, there has been no user-friendly, effective tool to predict long-term survival for specific patients after specific cardiac procedures. We sought to test the hypothesis that the STS PROM algorithm could not only predict the likelihood of death Fig 2. Long-term Kaplan-Meier survival estimates are shown by decile of the predictive risk of mortality (PROM) score among 30-day survivors.

7 ADULT CARDIAC 32 PUSKAS ET AL Ann Thorac Surg STS PROM SCORE PREDICTS LONG-TERM SURVIVAL 2012;93:26 35 Table 3. Survival Estimates by Procedure Procedure Survival, % 30 days 1 year 3 years 5 years 10 years Isolated CABG Isolated AVR Isolated MV a CABG AVR CABG MV All procedures a Includes mitral valve repair or replacement. AVR aortic valve replacement; CABG coronary artery bypass graft; MV mitral valve. within 30 days, as it was designed and calibrated to do, but could also predict long-term survival, for which it has never been intended or used. Not surprisingly, the STS PROM was a powerful predictor of 30-day survival after all of the procedures for which it was calibrated. However, the extraordinary power of the STS 30-day PROM to predict survival during 14 years of postoperative follow-up was an unexpected finding. Indeed, PROM performed very nearly as well as a predictor of long-term survival as it did predicting 30-day mortality. Although intuitively it is clear that comorbid conditions influencing early death after cardiac operations will also affect long-term survival, it is not intuitive that the algorithmic weighting of these risk factors, calibrated for 30-day events, should so precisely predict long-term survival. This essentially indicates that the 30 different preoperative patient risk factors incorporated into the STS PROM mathematic model affect long-term survival in a manner almost identical to the manner in which they affect a patient s likelihood of surviving 30 days after a specific procedure. We acknowledge that the demographic and comorbidity variables of patients undergoing cardiac operations have evolved during the 14 years in which patients were prospectively enrolled into the Emory University institutional STS database from which our analyses were performed. However, this does not limit the relevance of the Fig 3. Calibration curves for all procedures combined with squared predictive risk of mortality (PROM) score term for (A) 1 year, (B) 3 years, (C) 5 years, and (D) 10 years. Note that these curves closely follow the line of identity.

8 Ann Thorac Surg PUSKAS ET AL 2012;93:26 35 STS PROM SCORE PREDICTS LONG-TERM SURVIVAL Fig 4. Median estimated survival by predictive risk of mortality (PROM) percentages. (AVR aortic valve replacement; CABG coronary artery bypass grafting.) STS PROM for each patient, irrespective of the year of operation for any given patient. This is because the STS PROM is recalibrated every 6 months. Short-term and long-term survival for each patient is compared with that patient s own contemporary PROM score. Thus, changes in demographics and care processes over time are accommodated by the use of the contemporaneous PROM score for each patient. Interestingly, the recalibration of the STS database is based on 30-day outcome data; nonetheless, the iterative recalibrations have yielded a mathematic algorithm that predicts long-term survival over 14 years with remarkable consistency. Moreover, the model predicts long-term survival for patients undergoing each of the five different procedures for which the STS PROM has been calibrated, especially patients undergoing CABG and AVR procedures. It is not intuitively obvious why a group of different mathematic models calibrated to predict patients ability to survive specific procedures should perform so uniformly well in also predicting long-term survival. This is a single-center, retrospective study, which is a potentially serious limitation because our patient population and their average risk profiles might not represent those of other centers. Although these analyses include more than 24,000 individuals, the generalizability of these findings to the entire nation is not conclusively demonstrated. The STS developed PROM algorithms for these different cardiac procedures; however, there are many combinations and permutations of complex cardiac procedures for which there exists no such predictive model. Analytically, these findings were not externally validated; internal validation, although important, is almost assured when using a model with just a single predictor variable. Also, the cause of death in this study is unknown, and noncardiac deaths were assumed to be equally distributed among the patient subgroups. In conclusion, the STS PROM algorithm developed to predict death within 30 days of a cardiac operation accurately predicts death at 30 days and during 14 years of follow-up with almost equally strong discriminatory power for most procedure subgroups. Thus, these mathematic models, based on the same preoperative risk variables routinely collected for hundreds of thousands of United States patients annually, can be used to estimate the likelihood of long-term survival for specific patients and to adjust survival estimates in comparative effectiveness studies after specific cardiac procedures. This may have profound implications for the informed consent process, comparative effectiveness studies, and health care policy. References 1. Shroyer ALW, Plomondon ME, Grover FL, Edwards FH, for The Society of Thoracic Surgeons National Database Committee. The 1996 coronary artery bypass risk model: the Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg 1999;67: Parsonnet V, Dean D, Berstein AD. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 1989;79: Nashef SAM, Roques F, Michel P et al. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999;16: Heikkinen J, Biancari F, Satta J, et al. Predicting immediate and late outcome after surgery for mitral valve regurgitation with EuroSCORE. J Heart Valve Disease 2007;16: Kobayashi KJ, Williams JA, Nwakanma LU, et al. Euro- SCORE predicts short- and mid-term mortality in combined aortic valve replacement and coronary artery bypass patients. J Card Surg 2009;24: Schisterman EF, Whitcomb BW. Use of the social security administration death master file for ascertainment of mortality status. Popul Health Metr 2004;2:2. 7. Allison PD. Logistic regression using the SAS system: theory and application. Cary, NC: The SAS Institute 1999: Collett D. Modelling binary data. 2nd ed. Boca Raton: Chapman and Hall/CRC, 2000:88 9. Harrell FE. Regression modeling strategies. New York: Springer; 2001:78, Kleinbaum DG, Klein M. Survival analysis: a self-learning text. 2nd ed. New York: Springer; 2005:103-7, ADULT CARDIAC DISCUSSION DR FRED H. EDWARDS (Jacksonville, FL): Dr Mathisen and Dr Reed, it is a real privilege to discuss this paper from the Emory group. The Society for Thoracic Surgery (STS) cardiac surgery risk models have served us well for more than 2 decades now, and these models are designed, as you know, to predict shortterm outcomes, but today we have seen that there may be another dimension to these models. The need for longitudinal follow-up has been a high priority for the STS database since day one, and it is particularly important in this era of formal comparative effectiveness research.

9 ADULT CARDIAC 34 PUSKAS ET AL Ann Thorac Surg STS PROM SCORE PREDICTS LONG-TERM SURVIVAL 2012;93:26 35 The central feature of your study, John, is the link between the STS data and the Death Master File data, but I must say even after having read the manuscript and hearing your presentation, I am still not clear exactly how that link was established. It appears to be probabilistic matching, but we need to know if that is in fact the case, and if probabilistic matching was used, it is important to know the accuracy of that, the sensitivity, and the specificity. DR PUSKAS: Thank you, Fred. We certainly agree that this is more timely and more important than ever before as we require longitudinal follow-up to do effectiveness comparison studies. In terms of how we did this by a Health Insurance Portability and Accountability Act (HIPAA)-compliant mechanism, within one institution it is actually pretty simple. To do it within the whole national database with 1,000 institutions, this will be logistically, politically, and legally challenging. But our task was relatively simple. As many of you know, the federal government maintains a database that is publicly accessible and from which, for a small fee, you can find out whether any given Social Security number has been turned in, meaning the individual who had that number has passed away in America and has been recorded as dead. And this file is remarkably updated, thorough, and accurate, and it is so because it has to do with the government paying money: Social Security payments cease when you die. And so this is a matter of the federal government maintaining transparency in the expenditure of tax revenues, and that is why this database exists on the Internet and why it is public. So contrary to a lot of conversation about the notion of a Social Security number being inherently privileged information, it is actually a very public number. It is privileged when it is linked to a person, but you can get on-line, anyone of you right now, with a wireless device and find out whether your Social Security number is alive or any other number is associated with a live person. It will cost you about 20 cents per query. So what we did with our Emory database is we took the 24,000 patients, which, of course, are in the STS format in a software vendor we happen to use the Automated Reporting Management Information System (ARMIS) and we asked ARMIS to strip from the database just the names, the Social Security numbers, dates of birth, and then apply another unique identifier. We gave a pseudo, a record ID number, if you will, and created an ordered, sorted list of those 24,000 patients. Then we made a second list of just the Social Security numbers from those patients in the same order. We took that list of Social Security numbers and uploaded it on the Internet to the death master file, and we received back an indication whether each of those numbers was associated with a live or dead person. And then we reconstituted the list of numbers in the same order that we had the list of numbers linked to patients and we put them back together. We did in fact check with a variety of macro checks, essentially a spreadsheet, using simple programming, to ensure that those two lists did in fact mesh accurately. Once we had the two lists linked back together, we repopulated that piece of information, dead or alive and date of death, into our Emory institutional STS database. So we didn t ever actually send the Social Security numbers outside of our institution linked to a patient, but the numbers, of course, go up on-line to the database. They are just not identified at the time that they leave our institution. DR EDWARDS: The models used in this analysis were published in 1999, and certainly, it is appropriate to use those models for the patients who underwent operations in the earlier parts of your population, but I am surprised to see that they were used for the more current population. That leads to the second question. Were the more current STS risk models used for the more current members of the surgical population? And if not, I think it would be helpful to know why this was not done, especially since the STS database, of course, contains just the most current risk models. DR PUSKAS: That is a very good question and an insightful one. All the predicted risk of mortality scores, the PROM scores, were contemporaneous. Each patient had a PROM score calculated when they were entered into our database. So they are the contemporaneous risk predictor scores for each patient over the course of the roughly 16 years of the analysis. So, yes, they are updated for the recent patients and are contemporaneous for the remote patients. DR T. BRUCE FERGUSON (Greenville, NC): John, that was a very nice study, and we also have done, using the same technology, a similar type of matching with Social Security Death Index data. The question that I have is how specific do you think this really is for Atlanta, Georgia, and Emory s patient population? In practical terms, I have a hard time believing that all of the follow-up mortality is contained in the risk factors and procedural issues that are calculated before somebody has an operation. For example, the notion that there isn t any substantial influence of change in socioeconomic status and other health disparity issues on those patients as they move through time postsurgically is challenging. The ability to do this matching is critically important. My second comment is that the matching fidelity has to be extremely high to draw meaningful conclusions from the matching analyses. Your study emphasizes the value of being able to do these kinds of things at the local level and to drill down to a level of granularity locally that likely is not achievable to this degree at the national level. DR PUSKAS: Thank you. I do think that there are factors that impact survival that are not included in the PROM risk algorithm, but it is amazing how accurate and powerful this is as a long-term predictor. Keep in mind that it does include such things as race, which is, to some extent, a socioeconomic indicator, but it does not include ZIP code or family income. But the data don t lie, and it is 24,000 patients. PROM is a very reliable predictor of long-term survival, and we could simply take that slide with the multicolored curves on it and plug in Mrs. Smith s PROM score, and you can tell her likelihood of being alive at any given time point within a confidence interval. This is the median likelihood of being alive with a similar PROM score. DR JOSEPH F. SABIK (Cleveland, OH): John, congratulations on some very nice work. Whenever we have done long-term follow-up studies, we have found that risk factors that occur early, within the first 30 to 60 days, are often very different than risk factors that affect long-term survival. For example, if we just take coronary artery disease, we know that status, whether this is an emergent case, whether the patient is hemodynamically unstable, is very important for early risk. Once you get beyond the early risk, however, the survival curves of stable patients and unstable patients become parallel, suggesting that that has no long-term effect because we have remedied that problem with our operation. However, there are other factors, such as extent of coronary artery disease, or single-, double- or triple-vessel disease, that have no effect early on; however, over the long run those have been shown to be very important in terms of decreasing survival, with patients with less extensive disease having a better survival than patients with three-vessel disease.

10 Ann Thorac Surg PUSKAS ET AL 2012;93:26 35 STS PROM SCORE PREDICTS LONG-TERM SURVIVAL I worry a little bit about just using a model that is developed on short-term outcomes and one of the questions I have is how that might affect your long-term results and did you do anything to try to remedy that? Again, we have tended to break things up into early and long-term models. And the second question I have, is an area under the curve of 0.75 really a very good predictive model? Thank you. DR PUSKAS: Thanks, Joe. We were surprised at how well an algorithm calibrated on 30-day data could predict 10- and 14-year survival, but it does. And an area under the receiver operating characteristic curve of 0.75 or 0.79 is a very powerful predictor; it is the best predictor we have presently. I think the honest truth is that a lot of those same things that drive a person to an operation and impact their perioperative risk also determine long-term survival in ways that I don t think we fully had appreciated. Patients bring a lot to the table when they come to the operating table, and those same things impact their longerterm survival in what appears to be a rather predictable way. Thank you. 35 ADULT CARDIAC