The Society for Cardiothoracic Surgery in Great Britain

Original Article Performance of the EuroSCORE Models in Emergency Cardiac Surgery Stuart W. Grant, MRCS; Graeme L. Hickey, PhD; Ioannis Dimarakis, PhD, MRCS; Graham Cooper, MD, FRCS (CTh); David P. Jenkins, MS, FRCS (CTh); Rakesh Uppal, FRCS (CTh); Iain Buchan, MD, FFPH; Ben Bridgewater, PhD, FRCS (CTh) Background Accurate risk-adjustment models are useful for clinical decision making and are important for minimizing any tendency toward risk-averse clinical practice. In cardiac surgery, emergency patients are potentially at greatest risk of inappropriate risk-averse clinical decisions. UK cardiac surgery outcomes are currently risk-adjusted with EuroSCORE models. The objective of this study was to assess the performance of the EuroSCORE models in emergency cardiac surgery. Methods and Results The National Institute for Cardiovascular Outcomes Research database was used to identify adult cardiac surgery procedures performed in the United Kingdom between April 2008 and March 2011. A subset of procedures (July 2010 March 2011) was used for EuroSCORE II validation. The outcome measure was in-hospital mortality. Model calibration (Hosmer Lemeshow test, calibration plots, calculation of calibration intercept and slope) and discrimination (area under receiver-operating characteristic curve [area under the curve]) were assessed. In total, 109 988 cardiac procedures at 41 hospitals were included, of which 3342 were defined as emergency procedures. Compared with performance in all cardiac surgery and nonemergency cardiac surgery, the logistic EuroSCORE and EuroSCORE II models had poorer discrimination (area under the curve, 0.703 and 0.690, respectively) and poorer calibration for emergency surgery. The EuroSCORE risk factors of female sex, chronic pulmonary disease, neurological disease, active endocarditis, unstable angina, recent myocardial infarction, and pulmonary hypertension were not identified as important risk factors for emergency cardiac surgery. Conclusions Both EuroSCORE models demonstrated poor calibration and comparatively poor discrimination for emergency cardiac surgery. This has important implications when these models are used for clinical decision making or to adjust governance analyses. (Circ Cardiovasc Qual Outcomes. 2013;6:178-185.) Key Words: cardiac surgery clinical governance emergency surgery EuroSCORE risk prediction The Society for Cardiothoracic Surgery in Great Britain and Ireland (SCTS) has published mortality rates for cardiac surgery by named hospital since 2001 and by named surgeon since 2005. 1 The publication of these mortality rates was prompted by the public inquiry into pediatric cardiac surgery at Bristol between 1984 and 1995, 2 although the SCTS had held a national database for benchmarking purposes since 1994. 1 The Society of Thoracic Surgeons in the United States has published mortality rates and other outcomes by named surgical group and hospital since 2010. Previously, publication of results for specific procedures was limited to statewide initiatives, for example, the New York Heart Association report cards. 3,4 The SCTS-published mortality rates are for all National Health Service (NHS) and some private hospitals in the United Kingdom, and when mortality rates are higher than expected, a process for notifying and investigating divergence is initiated. 5 This clinical governance program has been associated with improved mortality outcomes despite increasing numbers of high-risk patients undergoing surgery. 6 Risk adjustment of mortality rates and clinical acceptance of the risk-adjustment process are integral to this process if valid comparisons are to be made and risk-averse clinical decisions avoided. The SCTS has used both the logistic EuroSCORE and a modified EuroSCORE model for risk-adjustment purposes 7,8 ; however, both models are now poorly calibrated for contemporary cardiac surgery. 9 The EuroSCORE II is now being considered for future SCTS risk-adjustment purposes after a successful contemporary validation exercise. 10,11 Despite these regular updates of the risk-adjustment process, there are concerns that governance programs that publish mortality results by named hospital or surgeon might encourage risk-averse rather than risk-proportionate clinical decision making. 12 15 Patients undergoing emergency cardiac surgery generally have the highest mortality but can have the most to gain from successful surgery. Because of the high risk of mortality in Received September 20, 2012; accepted December 28, 2012. From the Manchester Academic Health Science Centre, Department of Cardiothoracic Surgery, University Hospital of South Manchester (S.W.G., I.D., B.B.) and Manchester Academic Health Science Centre, the Northwest Institute for BioHealth Informatics (G.L.H., I.B.), University of Manchester, Manchester, UK; Department of Cardiothoracic Surgery, Northern General Hospital, Sheffield, UK (G.C.); Department of Cardiothoracic Surgery, Papworth Hospital NHS Foundation Trust, Papworth Everard, Cambridge, UK (D.P.J.); Department of Cardiothoracic Surgery, Barts and the London Chest Hospital, London, UK (R.U.); and William Harvey Research Institute, Barts and the London School of Medicine, London, UK (R.U.). Correspondence to Ben Bridgewater, PhD, Department of Cardiothoracic Surgery, University Hospital of South Manchester, Southmoor Rd, Manchester, UK. E-mail ben.bridgewater@uhsm.nhs.uk 2013 American Heart Association, Inc. Circ Cardiovasc Qual Outcomes is available at http://circoutcomes.ahajournals.org DOI: 10.1161/CIRCOUTCOMES.111.000018 178

Grant et al EuroSCORE Models in Emergency Cardiac Surgery 179 WHAT IS KNOWN Risk models are used for clinical decision making and to characterize the quality of cardiac surgery. Any risk model used should be accurate so that riskaverse clinical decisions are minimized and inappropriate conclusions about quality are avoided; this is particularly important for emergency cardiac surgery. WHAT THE STUDY ADDS Compared with performance in all cardiac surgery and nonemergency cardiac surgery, both the logistic EuroSCORE and EuroSCORE II demonstrate relatively poor discrimination for emergency cardiac surgery. Both the logistic EuroSCORE and EuroSCORE II are poorly calibrated for emergency cardiac surgery and tend to underpredict risk in the lower-risk patients and to overpredict risk in the higher-risk patients. A number of the original EuroSCORE risk factors do not seem to be important for emergency cardiac surgery. emergency patients, inappropriate risk-averse clinical decisions are potentially more likely. It is therefore important that any models that could potentially be used for risk adjustment perform well for this group of patients. The objective of this study was to assess the performance of the logistic EuroSCORE and EuroSCORE II models in emergency cardiac surgery. For comparison, the performance of the models was also assessed for all cardiac surgery and nonemergency cardiac surgery. Methods Data Collection, Validation, and Cleaning Prospectively collected data for cardiac surgery procedures performed at all NHS and some private hospitals in the United Kingdom between April 1, 2008, and March 31, 2011 (inclusive), were extracted from the National Institute for Cardiovascular Outcomes Research National Adult Cardiac Surgery Audit database. The data were cleaned by resolving transcriptional discrepancies, numeric irregularities, and clinical conflicts. Only procedures that were either the first or the only cardiac procedure within a single admission were included. The following records were removed: duplicate records, records with missing in-hospital mortality data, records with unreliable date information, records with no indicator of operative urgency, and records with absent hospital identifier information. Because transplantations, trauma, or primary ventricular assist device procedures are not included in SCTS clinical governance analyses, these records were also excluded. To ensure that the performance of each model was assessed with data prospective to that used for model development, EuroSCORE II was validated only with the subset of procedures performed between July 26, 2010, and March 31, 2011 (inclusive). In addition to the above exclusion criteria, any patient older than 95 years of age was removed, in keeping with development of EuroSCORE II. 10 Emergency cardiac procedures were defined as any unscheduled patient with ongoing refractory cardiac compromise with no delay in surgery regardless of the time of day (SCTS-defined emergency surgery) or any patient requiring cardiopulmonary resuscitation en route to the operating theater or before the induction of anesthesia (SCTSdefined salvage surgery). Other variable definitions are available at http://www.ucl.ac.uk/nicor/audits/adultcardiacsurgery/datasets. The outcome measure for the study was in-hospital mortality, defined as death attributable to any cause during admission to the base hospital for cardiac surgery. When there were irresolvable conflicts or missing in-hospital mortality data, record linkage to the Office for National Statistics database, which records details of all deaths in the United Kingdom, was used to backfill mortality status. Data were returned to each contributing hospital for local validation before analysis. Statistical Analysis The logistic EuroSCORE and EuroSCORE II were calculated for each relevant record. 7,10 If risk factor data required for calculation of the models were missing for a record, they were imputed with the baseline value. The percentage of missing data was calculated for EuroSCORE and EuroSCORE II risk factors as an indicator of data quality. The minimum data set required for submission to the SCTS database does not include all EuroSCORE II risk factors, meaning that a number of risk factor assumptions were required. These risk factor assumptions have been justified and published previously. 11 For each model and data set, model performance was evaluated in 3 groups: all cardiac surgery, nonemergency surgery, and emergency surgery. Model performance was summarized by evaluating model calibration and discrimination. Model calibration was evaluated with the following range of methods: (1) calculation of the calibration intercept and slope parameters and testing of unreliability, (2) the Hosmer Lemeshow test, 16 (3) comparison of observed and expected mortality in groups stratified by clinically chosen thresholds, (4) calculation of the overall observed-toexpected (O:E) ratio, and (5) graphical inspection of calibration plots. To calculate the calibration intercept and slope parameters, a logistic regression model was fit with the dependent variable set as the observed outcomes and the independent variable set as the log-odds transformed model predictions. Two parameters were estimated from the model: an intercept and a slope parameter. If the model is perfectly calibrated, the intercept and slope equal 0 and 1, respectively. The intercept is a measure of overall model calibration, which is whether predictions agree on average with observed probabilities. 17 A χ 2 test (on 2 df) for unreliability (null hypothesis: intercept=0 and slope=1) was also performed. 18 The Hosmer Lemeshow test allows the detection of departures from calibration within different groups. Contributing χ 2 statistics from each decile group were summed. Because this is an external model validation, the overall test statistic was evaluated with a χ 2 distribution on 10 df to test the hypothesis of whether the model predictions match the observed probabilities within the subgroups. 16 The Hosmer Lemeshow test decile stratification is arbitrary. Alternative approaches are (1) to group data together and (2) to group data into groups based on clinically defined thresholds. For option 1, calibration was summarized overall by calculating the O:E ratio (the number of observed in-hospital deaths compared with the number of predicted deaths). If a model is well calibrated on average, then the O:E ratio should be close to 1; departures above or below are indicative of underprediction and overprediction, respectively. Option 2 was performed only for EuroSCORE II because it is the most contemporarily calibrated model. For each validation cohort, records were split into a low-risk group (bottom 50% of cohort) and the remaining 50% of records were divided into 2 equal groups, medium risk and high risk, based on their predicted probability of inhospital mortality. The mean observed and predicted mortality was then calculated in each of these categories and compared by use of a χ 2 test on 1 df. Calibration plots are graphical representations of the Hosmer Lemeshow test. The calibration plots presented show for each model the mean predicted probability of outcome against the observed proportion of outcomes for 10 approximate equally sized groups on the basis of the ranked predicted risks calculated according to the models. Overlaid is the line of equality that would indicate a model with

180 Circ Cardiovasc Qual Outcomes March 2013 Table 1. Frequency and Percentage of Logistic EuroSCORE Risk Factors for All, Nonemergency, and Emergency Cardiac Surgery Procedures Performed Between April 1, 2008, and March 31, 2011 (Overall Cohort), After Imputation Risk Factors All Cardiac Surgery (n=109 988) perfect calibration and a nonparametric locally weighted scatterplot smoothing curve for each model showing the general relationship. 19 Approximately 95% confidence intervals (CIs) for the observed mortality proportions are shown as error bars and are estimated from the Agresti Coull approximation. 20 Model discrimination was evaluated by deriving the receiveroperating characteristic curve and calculating the area under the curve (AUC) summary statistic. 21 The AUC is interpretable as follows: If 2 patients are drawn at random from the group of interest, one who experiences an event and the other who does not, then the AUC is the probability that the model-estimated probability of the event is higher for the patient who actually experienced the event. The AUC ranges from 0.5, indicating that the model performs no better than chance, to 1.0, indicating that the model discriminates perfectly. The threshold for acceptable model performance is generally considered to be 0.7, with a threshold of 0.8 for excellent discrimination. 16 DeLong s method for estimating the variance of the AUC was used to calculate 95% CIs and to determine whether the difference between emergency and nonemergency AUCs is statistically signifcant. 22 To investigate which of the logistic EuroSCORE risk factors are important for emergency surgery, an updated EuroSCORE model was developed on the complete data set using multiple logistic regression. Because the model may be overfitted in this unique cohort of patients, model fit and complexity were balanced by use of a backward model selection based on the Akaike Information Criterion. 23 All coefficients with values of P<0.05 were considered significant. All statistical analyses were performed with the R statistical computing software (version 2.14.2; R Development Core Team, Vienna, Austria). 24 Calculations of calibration coefficients and corresponding hypothesis tests were performed with the rms package. 25 Discrimination analysis was performed with the proc package. 26 Nonemergency (n=106 646) Emergency (n=3342) Frequency Percentage Frequency Percentage Frequency Percentage Absolute Difference, % Missing Records, % Age, y* 66.94 11.61 67.07 11.50 62.67 14.24 4.40 0.00 Female 30 270 27.52 29 240 27.42 1030 30.82 3.40 0.01 Chronic pulmonary disease 14 713 13.38 14 294 13.40 419 12.54 0.87 0.54 Extracardiac arteriopathy 13 207 12.01 12 742 11.95 465 13.91 1.97 0.53 Neurological disease 2635 2.40 2412 2.26 223 6.67 4.41 1.46 Previous cardiac surgery 6890 6.26 6539 6.13 351 10.50 4.37 0.00 Creatinine >200 μmol/l 2760 2.51 2404 2.25 356 10.65 8.40 1.78 Active endocarditis 1606 1.46 1229 1.15 377 11.28 10.13 2.32 Critical preoperative state 4999 4.55 3376 3.17 1623 48.56 45.40 0.00 Unstable angina 5348 4.86 4202 3.94 1146 34.29 30.35 0.37 LV function 3.03 Moderate 25 539 23.22 24 653 23.12 886 26.51 3.39 Poor 6614 6.01 6076 5.70 538 16.10 10.40 Recent MI 19 576 17.80 18 455 17.30 1121 33.54 16.24 5.01 Pulmonary hypertension 2158 1.96 2046 1.92 112 3.35 1.43 0.02 Emergency 3342 3.04 0.00 Other than isolated CABG 50 703 46.10 48 494 45.47 2209 66.10 20.63 0.06 Surgery on thoracic aorta 4455 4.05 3592 3.37 863 25.82 22.45 0.43 Postinfarct septal rupture 216 0.20 107 0.10 109 3.26 3.16 0.00 CABG indicates coronary artery bypass graft; LV, left ventricular; and MI, myocardial infarction. Absolute differences between nonemergency and emergency risk factor percentages and the percentage of missing data for each risk factor prior to imputation are shown. *Mean (SD). Results For the overall cohort, 111 011 records were extracted across 41 hospitals, of which 110 476 corresponded to unique admissions. A total of 337 records were removed because of missing hospital identifier, outcome, or operative urgency status. A further 151 records that represented transplantations, trauma, or primary ventricular assist device procedures were removed. The percentages of missing risk factor data ranged from 0.0% (age, previous cardiac surgery, critical preoperative state, septal rupture) to 5.0% (recent myocardial infarction). With the exception of 5 risk factors (neurological disease, creatinine >200 μmol/l, active infective endocarditis, left ventricular function, and recent myocardial infarction), all had <1% missing data (Table 1). Of the 109 988 records included, 3342 were emergency procedures and 106 646 were nonemergency. The mean age in the emergency subgroup was 62.7 years (SD, 14.2 years), and the majority of patients (69.2%) were men. The prevalence of each logistic EuroSCORE risk factor in the study population is shown in Table 1. In total, there were 3449 (3.1%; 95% CI, 3.0 3.2) in-hospital deaths, of which 2860 (2.7%; 95% CI, 2.6 2.8) were in the nonemergency cohort and 589 (17.6%; 95% CI, 16.4 19.0) were in the emergency cohort. The EuroSCORE II validation subset included 23 624 records, of which 22 862 were nonemergency procedures

Grant et al EuroSCORE Models in Emergency Cardiac Surgery 181 Table 2. Frequency and Percentage of EuroSCORE II Risk Factors for All, Nonemergency, and Emergency Cardiac Surgery Procedures Performed Between July 26, 2010 and March 31, 2011 After Imputation Risk Factors All Cardiac Surgery (n=23 624) Nonemergency (n=22 862) Emergency (n=762) Frequency Percentage Frequency Percentage Frequency Percentage Absolute Difference, % Missing Records, % Age, y* 67.13 11.78 67.28 11.67 62.45 13.79 4.83 0.00 Female 6543 27.70 6301 27.56 242 31.76 4.20 0.02 Renal impairment 4.19 CC, 50 85 9527 40.33 9281 40.60 246 32.28 8.31 CC 50 1706 7.22 1616 7.07 90 11.81 4.74 Dialysis 138 0.58 119 0.52 19 2.49 1.97 Extracardiac arteriopathy 2735 11.58 2620 11.46 115 15.09 3.63 0.56 Poor mobility 611 2.59 554 2.42 57 7.48 5.06 0.78 Previous cardiac surgery 1550 6.56 1473 6.44 77 10.10 3.66 0.00 Chronic pulmonary disease 3226 13.66 3133 13.70 93 12.20 1.50 0.50 Active endocarditis 332 1.41 254 1.11 78 10.24 9.13 0.97 Critical preoperative state 1154 4.88 773 3.38 381 50.00 46.62 0.00 Diabetes mellitus on insulin 1264 5.35 1231 5.38 33 4.33 1.05 0.55 NYHA class 1.02 II 9914 41.97 9736 42.59 178 23.36 19.23 III 6635 28.09 6499 28.43 136 17.85 10.58 IV 1266 5.36 1090 4.77 176 23.10 18.33 CCS class IV angina 2008 8.50 1799 7.87 209 27.43 19.56 0.74 LV function 3.08 Moderate 4960 21.00 4769 20.86 191 25.07 4.21 Poor 1268 5.37 1158 5.07 110 14.44 9.37 Very poor 109 0.46 100 0.44 9 1.18 0.74 Recent MI 4643 19.65 4372 19.12 271 35.56 16.44 4.59 PA systolic pressure, mm Hg 0.04 31 55 1425 6.03 1376 6.02 49 6.43 0.41 55 614 2.60 582 2.55 32 4.20 1.65 Operative urgency 0.00 Urgent 6823 28.88 6823 29.84 0 0.00 29.84 Emergency 694 2.94 0 0.00 694 91.08 91.08 Salvage 68 0.29 0 0.00 68 8.92 8.92 Weight of procedure 0.29 Single non-cabg 4961 21.00 4763 20.83 198 25.98 5.15 2 Procedures 4754 20.12 4576 20.02 178 23.36 3.34 3+ Procedures 1514 6.41 1391 6.08 123 16.14 10.06 Surgery on thoracic aorta 1010 4.28 807 3.53 203 26.64 23.11 0.51 CABG indicates coronary artery bypass graft; CC, creatinine clearance; CCS, Canadian Cardiovascular Society; LV, left ventricular; MI, myocardial infarction; NYHA, New York Heart Association; and PA, pulmonary artery. Absolute differences between nonemergency and emergency risk factor percentages and the percentage of missing data for each risk factor prior to imputation are shown. *Mean (SD). The actual percentage of true missing data is 95.9% because serum creatinine has only recently been collected. However, creatinine clearance classification could be estimated robustly from other available renal function data. 14 and 762 were emergency procedures. Table 2 shows the prevalence and percentage of missing data for each risk factor in the EuroSCORE II model. In total, for this subset, there were 744 (3.1%; 95% CI, 2.9 3.4) in-hospital deaths: 622 (2.7%; 95% CI, 2.5 2.9) were in the nonemergency cohort and 122 (16.0%; 95% CI, 13.6 18.8) were in the emergency cohort. Model Performance for All Cardiac Surgery The calibration plots (Figure 1, left) demonstrate that the logistic EuroSCORE is poorly calibrated for all cardiac surgery, significantly overpredicting mortality, which is confirmed by the O:E ratio of 0.43. The EuroSCORE II is better calibrated in this group with an O:E ratio of 0.92. The calibration slopes were 0.876 and 0.968 for the logistic EuroSCORE

182 Circ Cardiovasc Qual Outcomes March 2013 All cardiac surgery Non emergency Emergency 0.3 Model Logistic EuroSCORE EuroSCORE II Model Logistic EuroSCORE EuroSCORE II 0.8 Model Logistic EuroSCORE EuroSCORE II Observed mortality proportion 0.2 0.1 Observed mortality proportion 0.2 0.1 Observed mortality proportion 0.6 0.4 0.2 0.0 0.0 0.1 0.2 0.3 Predicted mortality proportion 0.0 0.0 0.1 0.2 Predicted mortality proportion Figure 1. Calibration plots for the logistic EuroSCORE and the EuroSCORE II. Black dashed line is the line of equality that represents perfect calibration. Blue and red lines represent nonparametric smoothing curves. Vertical lines represent 95% binomial confidence intervals of the observed mortality proportion. 0.0 0.0 0.2 0.4 0.6 0.8 Predicted mortality proportion and EuroSCORE II, respectively (Table 3). The intercept for the logistic EuroSCORE ( 1.247) suggested an overall loss of average calibration compared with the EuroSCORE II ( 0.187). The observed versus predicted mortality in each clinical group for EuroSCORE II was 0.6% versus 1.0% (P<0.001; low risk), 2.7% versus 2.4% (P=0.191; medium risk), and 8.7% versus 9.3% (P=0.084; high risk). Despite EuroSCORE II being well calibrated for 2 of 3 clinical groups, the Hosmer Lemeshow test rejected the null hypothesis of calibration across the decile subgroups for both models. The receiveroperating characteristic curves for the models are shown in Figure 2. Both models demonstrated excellent discrimination with AUCs of 0.806 and 0.811 for the logistic EuroSCORE and EuroSCORE II, respectively. Model Performance for Nonemergency Surgery For nonemergency surgery, the calibration plots (Figure 1, center) and O:E ratios (Table 3) demonstrate that the logistic EuroSCORE is poorly calibrated but the EuroSCORE II is well calibrated. The calibration slopes were close to 1, with the intercept for the logistic EuroSCORE ( 1.220) suggesting an overall loss of average calibration compared with the EuroSCORE II ( 0.085). For EuroSCORE II, the observed versus predicted mortality in each clinical group was 0.6% versus 1.0% (P<0.001; low risk), 2.3% versus 2.3% (P=0.829; medium risk), and 7.3% versus 7.8% (P=0.141; high risk). Despite EuroSCORE II being well calibrated for 2 of 3 clinical groups, the Hosmer Lemeshow test rejected the null hypothesis of calibration across the decile subgroups for both models. Both models demonstrated good discrimination, with an AUC of 0.786 for the logistic EuroSCORE and 0.799 for EuroSCORE II. Model Performance for Emergency Surgery Both the logistic EuroSCORE and EuroSCORE II demonstrated poor calibration for emergency surgery (Figure 1, right). For the emergency cohort, the slope parameters were similar for both the logistic EuroSCORE and EuroSCORE II models (0.52 and 0.54, respectively), indicating, along with the calibration plots, that the models are poorly fitted for this cohort. Both models tended to underpredict risk in the lower-risk patients and to overpredict risk in the higher-risk patients. The intercept parameters were also markedly different from the optimal value of 0 ( 1.16 and 0.64, respectively), indicating miscalibration. The logistic EuroSCORE significantly overpredicted the mortality rate in the emergency cohort with a mean expected mortality rate of 31.0%, giving an O:E ratio of 0.57. The EuroSCORE II closely predicted the overall mortality rate for emergency surgery with a mean score of 15.6% (O:E, 1.03). For EuroSCORE II, the observed versus predicted mortality in each clinical group was 8.9% versus 4.4% (P<0.001; Table 3. Model Performance Summary Statistics for the Logistic EuroSCORE and EuroSCORE II Models (Applied to the Appropriate Data Subsets) Across the 3 Groups: All Cardiac Surgery, Nonemergency Cardiac Surgery, and Emergency Cardiac Surgery Group Model HL Test Calibration Curve Discrimination χ 2 P Intercept Slope P O:E AUC 95% CI All cardiac surgery EuroSCORE 3209.6 <0.001 1.247 0.876 <0.001 0.43 0.803 0.796 0.810 EuroSCORE II 31.2 <0.001 0.187 0.968 0.022 0.92 0.811 0.796 0.826 Nonemergency EuroSCORE 2876.7 <0.001 1.220 0.903 <0.001 0.41 0.786 0.778 0.793 EuroSCORE II 31.8 <0.001 0.085 1.011 0.016 0.90 0.799 0.782 0.815 Emergency EuroSCORE 671.3 <0.001 1.157 0.516 <0.001 0.57 0.703 0.679 0.726 EuroSCORE II 40.0 <0.001* 0.643 0.535 <0.001 1.03 0.690 0.638 0.743 AUC indicates area under the receiver-operating characteristic curve; CI, confidence interval; HL, Hosmer and Lemeshow; and O:E, observed-to-expected ratio. *Because of fewer records in the EuroSCORE II cohort, 5 approximate equal groups were determined by the quintiles of the predictions to achieve sufficiently sized expected values.

Grant et al EuroSCORE Models in Emergency Cardiac Surgery 183 Sensitivity 0.0 0.2 0.4 0.6 0.8 1.0 low risk), 15.8% versus 13.9% (P=0.459; medium risk), 30.4% versus 39.5% (P=0.010; high risk), meaning that for emergency surgery, EuroSCORE II was only well calibrated for 1 clinical group. The Hosmer Lemeshow test rejected the null hypothesis of calibration across the decile subgroups for both models. The AUCs were 0.703 for logistic EuroSCORE and 0.690 for EuroSCORE II. The difference between the AUC values for the nonemergency and emergency cohorts was statistically significant for both models (P<0.001). Table 4. Risk Factors for In-Hospital Mortality After Emergency Cardiac Surgery Derived From an Updated Model Fit of the Logistic EuroSCORE Model Using Multiple Logistic Regression With Backward Model Selection Applied Risk Factor All cardiac surgery Non emergency Emergency 0.0 0.2 0.4 0.6 0.8 1.0 1 Specificity Figure 2. Receiver-operating characteristic curves for the logistic EuroSCORE and EuroSCORE II in 3 groups of patients: all cardiac surgery, nonemergency surgery, and emergency surgery. Grey line represents the line of equality. Logistic EuroSCORE Odds Ratio Logistic EuroSCORE EuroSCORE II Updated Model Fit Odds Ratio 95% CI P Age 1.07 1.04 1.02 1.05 <0.001 Extracardiac arteriopathy 1.93 1.38 1.07 1.78 0.013 Previous cardiac surgery 2.73 1.68 1.27 2.20 <0.001 Creatinine >200 μmol/l 1.92 1.44 1.10 1.89 0.008 Critical preoperative state 2.73 2.07 1.67 2.57 <0.001 LV function Moderate (LVEF 30% 49%) 1.52 1.10 0.87 1.40 0.422 Poor (LVEF <30%) 2.99 2.29 1.76 2.98 <0.001 Other than isolated CABG 1.72 2.82 2.16 3.71 <0.001 Surgery on thoracic aorta 3.19 1.83 1.44 2.34 <0.001 Postinfarct septal rupture 4.31 2.23 1.45 3.42 <0.001 CABG indicates coronary artery bypass graft; CI, confidence interval; LV, left ventricular; and LVEF, left ventricular ejection fraction. Original odds ratios are obtained by exponentiating the appropriate logistic EuroSCORE model coefficients. Updated Model Fit Risk factors for in-hospital mortality after emergency cardiac surgery derived from an updated model fit of the logistic EuroSCORE are shown in Table 4. Patient age (transformed as per the EuroSCORE transformation), extracardiac arteriopathy, previous cardiac surgery, creatinine >200 μmol/l, critical preoperative state, left ventricular function (moderate and poor), surgery other than isolated coronary artery bypass graft, surgery on the thoracic aorta, and postinfarct septal rupture were all important risk factors. Female sex, chronic pulmonary disease, neurological disease, active endocarditis, unstable angina, recent myocardial infarction and pulmonary hypertension were not retained in the model selection routine for in-hospital mortality after emergency cardiac surgery. Discussion This study demonstrates that the performance of both EuroS- CORE models in emergency cardiac surgery is inadequate compared with all cardiac surgery and nonemergency surgery. Both models demonstrated acceptable discrimination for all cardiac surgery and nonemergency procedures. However, the discrimination in the emergency cohort was substantially lower. As expected, the logistic EuroSCORE was poorly calibrated for contemporary cardiac surgery within the United Kingdom for both the overall cardiac surgery and the nonemergency cohort; therefore, it is not surprising that it was also poorly calibrated in the emergency cohort. The EuroSCORE II model, on the other hand, was well calibrated in the overall and nonemergency cardiac surgery cohorts but less well calibrated in the emergency surgery cohort. In general, for emergency cardiac surgery, the EuroSCORE models tended to underpredict risk in the lower-risk patients and to overpredict in the higher-risk patients. Although the emergency surgery group contributes only a relatively small number of the total number of patients, they are clinically high risk and have much to gain from surgery because they will usually die without it. This study suggests that if any EuroSCORE model is used for clinical decision making in emergency cardiac surgery, model predictions are likely to be unreliable and should be interpreted with caution. Denying a patient emergency cardiac surgery simply because the patient has a high EuroSCORE would certainly be inappropriate, especially given the tendency of both models to overpredict risk in higher-risk emergency patients. In addition to the implications for clinical decision making, this study has important implications for cardiac surgery governance analyses. Performance in emergency or highrisk surgery is likely to be an important indicator of quality; however, including emergency procedures in such analyses is potentially inappropriate because of inadequate risk adjustment. It has previously been demonstrated that inappropriate risk adjustment leads to flawed clinical governance analyses. 27 This is unlikely to be a limitation of model performance exclusive to the EuroSCORE models and may apply to other widely used models such as those developed by the Society of Thoracic Surgeons. The Society of Thoracic Surgeons models could not be validated in this study because required

184 Circ Cardiovasc Qual Outcomes March 2013 risk factor data were not routinely collected by the SCTS. 28 Risk prediction models specific to emergency surgery should therefore be developed for emergency cardiac surgery governance analyses. This study is based on a large clinical data set that has been validated locally at each contributing hospital. The number of emergency cardiac surgical procedures performed in the United Kingdom not captured by this data set is negligible because almost all emergency cardiac surgery is performed in NHS hospitals. As with any study based on clinical data, some data were missing. However, patients excluded from the study because of missing data represented only 0.31% of the overall cohort. The percentage of missing data for each risk factor individually was generally small, and there were no significant differences in missing data between the subgroups analyzed in this study. An imputation approach was applied to complete the remaining missing data, and this imputation approach is justified by an understanding of how clinicians complete data as part of the SCTS database program. A thorough assessment of model performance has been performed. Multiple approaches to assess model performance, including clinical face validity, are required to determine the usefulness of a model. In this study, model calibration was assessed through the use of recalibration intercept and slopes, the Hosmer Lemeshow test, and calibration plots. In this study, the Hosmer Lemeshow test rejected the hypothesis in all models and clinical groups. However, despite its widespread use in clinical studies, it is no longer advocated for discerning model performance because of known shortcomings. 29,30 The nonparametric calibration curves are a useful diagnostic tool for examining the nature of any miscalibration. The O:E ratio statistic is a single summary statistic and thus can average out opposing regions of miscalibration to suggest that a model is well calibrated. This was the case for the EuroSCORE II model in the emergency cohort in which the O:E ratio was close to 1, attributable to the averaging out of underpredicting and overpredicting. Model discrimination has been assessed with the widely accepted method of AUC; however, this must be taken into account with the other performance measures. 31 The EuroSCORE, which was published in 1999 as an additive model and then superseded by a logistic model in 2003, has been used for clinical governance analyses in the United Kingdom and has been widely accepted throughout Europe. 7,32 Since publication of the EuroSCORE, the predicted mortality now significantly exceeds observed mortality for most patient groups, which is presumably attributable to overall improvements in quality of care. 6 To the best of our knowledge, this is the first detailed assessment of the performance of EuroSCORE risk models in emergency cardiac surgery. EuroSCORE model performance, however, has previously been assessed in a number of different high-risk cardiac surgery populations. 31 40 Whereas the focus of this study was on the performance of existing risk models in emergency cardiac surgery, this study also represents one of the largest analyses to date of which logistic EuroSCORE risk factors are important for in-hospital mortality after emergency cardiac surgery. A number of logistic EuroSCORE risk factors were not found to be important for emergency surgery, and the weightings of the remaining risk factors were found to be different from the original EuroSCORE weightings. It is likely that the overall clinical state of the patient (including pathogenesis and clinical presentation) associated with emergency status is the most important determinant of postoperative outcome. For these reasons, a simple recalibration of the EuroSCORE to improve model performance for emergency cardiac surgery would be inappropriate. Although the findings of this study relate directly to cardiac surgery, it is likely that governance analyses will become more common in other surgical and interventional medical specialties. There is an increasing public appetite for comparative clinical outcomes data, and this will require robust analyses to allow safe comparisons to be made. 5 This study raises important generic issues about the use of risk prediction models for emergency patients, which may potentially be reproduced across different subspecialty patient groups. Acknowledgments We acknowledge all members of the Society for Cardiothoracic Surgery in Great Britain and Ireland who contribute data to the SCTS database. The National Institute for Cardiovascular Outcomes Research, University College London, London, UK, provided the data for this study. We would also like to thank Norman Stein for his work in cleaning the SCTS database. Sources of Funding Part of this research was funded by a grant from Heart Research United Kingdom (grant RG2583). None. Disclosures References 1. Bridgewater B; Society for Cardiothoracic Surgery in GB and Ireland. Cardiac registers: the adult cardiac surgery register. Heart. 2010;96: 1441 1443. 2. Kennedy I. The Inquiry into the management of care of children receiving complex heart surgery at the Bristol Royal Infirmary. Bristol, UK: the Bristol Royal Infirmary Inquiry, 2001. http://www.bristol-inquiry.org.uk/. Accessed 06 March 2012. 3. Ferris TG, Torchiana DF. Public release of clinical outcomes data: online CABG report cards. N Engl J Med. 2010;363:1593 1595. 4. Hannan EL, Kilburn H Jr, O Donnell JF, Lukacik G, Shields EP. Adult open heart surgery in New York State: an analysis of risk factors and hospital mortality rates. JAMA. 1990;264:2768 2774. 5. Bridgewater B, Cooper G, Livesey S, Kinsman R; Society for Cardiothoracic Surgery in Great Britain & Ireland. Maintaining Patients Trust: Modern Medical Professionalism. Henley-on-Thames, UK: Dendrite Clinical Systems Ltd; 2011. 6. Bridgewater B, Keogh B, Kinsman R, Walton P. Demonstrating Quality: The Sixth National Adult Cardiac Surgery Database Report. Henley-on- Thames, UK: Dendrite Clinical Systems Ltd; 2009. 7. Roques F, Michel P, Goldstone AR, Nashef SA. The logistic EuroSCORE. Eur Heart J. 2003;24:881 882. 8. Choong CK, Sergeant P, Nashef SA, Smith JA, Bridgewater B. The EuroSCORE risk stratification system in the current era: how accurate is it and what should be done if it is inaccurate? Eur J Cardiothorac Surg. 2009;35:59 61. 9. Siregar S, Groenwold RH, de Heer F, Bots ML, van der Graaf Y, van Herwerden LA. Performance of the original EuroSCORE. Eur J Cardiothorac Surg. 2012;41:746 754.

Grant et al EuroSCORE Models in Emergency Cardiac Surgery 185 10. Nashef SA, Roques F, Sharples LD, Nilsson J, Smith C, Goldstone AR, Lockowandt U. EuroSCORE II. Eur J Cardiothorac Surg. 2012;41:734 44; discussion 744. 11. Grant SW, Hickey GL, Dimarakis I, Trivedi U, Bryan A, Treasure T, Cooper G, Pagano D, Buchan I, Bridgewater B. How does EuroSCORE II perform in UK cardiac surgery: an analysis of 23 740 patients from the Society for Cardiothoracic Surgery in Great Britain and Ireland National Database. Heart. 2012;98:1568 1572. 12. Schneider EC, Epstein AM. Influence of cardiac-surgery performance reports on referral practices and access to care: a survey of cardiovascular specialists. N Engl J Med. 1996;335:251 256. 13. Burack JH, Impellizzeri P, Homel P, Cunningham JN Jr. Public reporting of surgical mortality: a survey of New York State cardiothoracic surgeons. Ann Thorac Surg. 1999;68:1195 1200. 14. Apolito RA, Greenberg MA, Menegus MA, Lowe AM, Sleeper LA, Goldberger MH, Remick J, Radford MJ, Hochman JS. Impact of the New York State Cardiac Surgery and Percutaneous Coronary Intervention Reporting System on the management of patients with acute myocardial infarction complicated by cardiogenic shock. Am Heart J. 2008;155:267 273. 15. Omoigui NA, Miller DP, Brown KJ, Annan K, Cosgrove D 3rd, Lytle B, Loop F, Topol EJ. Outmigration for coronary bypass surgery in an era of public dissemination of clinical outcomes. Circulation. 1996;93:27 33. 16. Hosmer D, Lemeshow S. Applied Logistic Regression. New York: John Wiley; 1989. 17. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128 138. 18. Harrell FE Jr. Regression Modeling Strategies. New York: Springer- Verlag; 2001. 19. Cleveland WS, Devlin SJ. Locally weighted regression: an approach to regression-analysis by local fitting. J Am Stat Assoc. 1988; 83:596 610. 20. Agresti A, Coull BA. Approximate is better than exact for interval estimation of binomial proportions. Am Stat. 1998; 52:119 126. 21. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29 36. 22. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837 845. 23. Akaike H. New look at statistical-model identification. IEEE Trans Autom Control. 1974; AC19:716 723. 24. R Development Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2011. http://www.r-project.org/. Accessed 14 September 2012. 25. Harrell FE Jr. rms: regression modeling strategies. R package version 3.5-0, 2012. http://cran.r-project.org/package=rms. Accessed 14 September 2012 26. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Müller M. proc: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12:77. 27. Grant SW, Grayson AD, Jackson M, Au J, Fabri BM, Grotte G, Jones M, Bridgewater B. Does the choice of risk-adjustment model influence the outcome of surgeon-specific mortality analysis? A retrospective analysis of 14,637 patients under 31 surgeons. Heart. 2008;94:1044 1049. 28. Shahian DM, Edwards FH. The Society of Thoracic Surgeons 2008 cardiac surgery risk models: introduction. Ann Thorac Surg. 2009; 88:S1. 29. Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit Care Med. 2007;35:2052 2056. 30. Marcin JP, Romano PS. Size matters to a model s fit. Crit Care Med. 2007;35:2212 2213. 31. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115:928 935. 32. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999;16:9 13. 33. Michel P, Roques F, Nashef SA; EuroSCORE Project Group. Logistic or additive EuroSCORE for high-risk patients? Eur J Cardiothorac Surg. 2003;23:684 687. 34. Ranucci M, Castelvecchio S, Menicanti LA, Scolletta S, Biagioli B, Giomarelli P. An adjusted EuroSCORE model for high-risk cardiac patients. Eur J Cardiothorac Surg. 2009;36:791 797. 35. Bhatti F, Grayson AD, Grotte G, Fabri BM, Au J, Jones M, Bridgewater B; North West Quality Improvement Programme in Cardiac Interventions. The logistic EuroSCORE in cardiac surgery: how well does it predict operative risk? Heart. 2006;92:1817 1820. 36. Wendt D, Osswald BR, Kayser K, Thielmann M, Tossios P, Massoudy P, Kamler M, Jakob H. Society of Thoracic Surgeons score is superior to the EuroSCORE determining mortality in high risk patients undergoing isolated aortic valve replacement. Ann Thorac Surg. 2009;88: 468 474. 37. Barmettler H, Immer FF, Berdat PA, Eckstein FS, Kipfer B, Carrel TP. Risk-stratification in thoracic aortic surgery: should the EuroSCORE be modified? Eur J Cardiothorac Surg. 2004;25:691 694. 38. Ghanta RK, Shekar PS, McGurk S, Rosborough DM, Aranki SF. Nonelective cardiac surgery in the elderly: is it justified? J Thorac Cardiovasc Surg. 2010;140:103 9, 109.e1. 39. Nissinen J, Biancari F, Wistbacka JO, Loponen P, Teittinen K, Tarkiainen P, Koivisto SP, Tarkka M. Is it possible to improve the accuracy of EuroS- CORE? Eur J Cardiothorac Surg. 2009;36:799 804. 40. Goodwin AT, Ooi A, Kitcat J, Nashef SA. Outcomes in emergency redo cardiac surgery: cost, benefit and risk assessment. Interact Cardiovasc Thorac Surg. 2003;2:227 230.