ARTICLE IN PRESS. Assessing clinical performance in cardiac surgery. Does a specialised clinical database make a difference?

Transcription

1 doi: /icvts Interactive CardioVascular and Thoracic Surgery 5 (2006) Institutional report - Cardiac general Assessing clinical performance in cardiac surgery. Does a specialised clinical make a difference? a a b c d e Daniela Fortuna, Maria Vizioli, Andrea Contini, Italo Ghidoni, Bruno Biagi, Davide Pacini, f a, Giannantonio Carnelos, Roberto Grilli * a Department of Clinical Governance, Agenzia Sanitaria Regionale, Viale Aldo Moro, 21, Bologna, Italy b Division of Cardiology, Azienda Ospedaliera, Parma, Italy c Department of Cardiac Surgery, Hesperia Hospital, Modena, Italy d Department of Cardiac Surgery, Villa Maria Cecilia, Lugo, Ravenna, Italy e Department of Cardiac Surgery, Azienda ospedaliera, Bologna, Italy F Department of Cardiac Surgery, Villa Salus, Reggio Emilia, Italy Received 24 May 2005; received in revised form 13 December 2005; accepted 14 December 2005 Abstract We compared mortality rates league tables for six cardiac surgery centres developed using an administrative (integrated with information on patients EuroSCORE) with those drawn from a specialised clinical. Data from 4017 patients undergoing cardiac surgery over the period January 1st December 31st 2003, and identified both s were used. Case mix adjusted in-hospital mortality rates were estimated relying on information provided by each, and league tables were drawn from both. The correlation between the two league tables was assessed through the Spearman correlation coefficient. League tables drawn from the two sources identified the same best and worst centres and the Spearman correlation coefficient confirmed a high level of agreement between the two rankings (rs0.89; P-0.02). Use of the logistic EuroSCORE instead of the additive one did not change the results. An administrative and a clinical specialised provided similar league tables. However, this finding by no means implies that clinical s should be abandoned. While administrative data allow a more efficient performance assessment, clinical s may more properly satisfy the legitimate demand of surgical staff of being directly involved in quality monitoring, rather than being mere passive objects of external assessment Published by European Association for Cardio-Thoracic Surgery. All rights reserved. Keywords: Administrative data; Clinical ; Cardiac surgery; League table; Case-mix adjustment 1. Introduction Comparative assessment of hospital performance, usually in the form of league tables, is currently one of the most frequently addressed topics in health services research, and surely one of those relevant to cardiac surgery. When providers are compared, a number of problems arise concerning the statistical techniques used in outlier identification and the broad impact that these initiatives may have on individual hospitals and their staff w1x. Leaving the policy implications of public disclosure aside w2x, one of the technical issues at stake is the relevance of the source of information on which the performance assessment is based. Indeed, information source s role is crucial for the assessment of clinical outcomes, as detailed information is needed if individual patient s complexity and surgical risk has to be properly taken into account. While offering the obvious advantage of being easily available at low cost, administrative s are by their very nature usually considered inadequate when it comes to adjusting for case-mix w3x. Thus, specialised clinical *Corresponding author. Tel.: q ; fax: q address: rgrilli@regione.emilia-romagna.it (R. Grilli) Published by European Association for Cardio-Thoracic Surgery s, based upon a prospective collection of detailed information on clinical characteristics of individual patients, could be considered a valuable alternative. However, their organisation and management can be complex and resource demanding. Both those alternatives are currently being experimented in Emilia-Romagna, an Italian region of 4 million residents with six cardiac surgery centres. In this regional context, the system of continuous monitoring of hospitals clinical performance, which has been in place since 1998 w4x, relies on the regional administrative of hospital admissions (SDO) as the main source of information for the assessment of mortality rates for any type of cardiac surgery. A remarkable feature of this system is the inclusion of a description of patient s surgical risk expressed through the EuroSCORE w5x. This information has no administrative value and it is included only for quality assessment purposes. On April 2002, as part of a broader effort to improve our quality monitoring system, a regional registry of patients undergoing cardiac surgery was launched, with individual centres required to collect detailed information on patient characteristics and outcomes. In undertaking this initiative, it was assumed that this more clinically oriented source of

2 124 D. Fortuna et al. / Interactive CardioVascular and Thoracic Surgery 5 (2006) information would have improved case-mix adjustment procedures, thus providing more accurate mortality figures for each centre and, overall, a fairer assessment of their performance. The coexistence of the two s made it possible to assess the extent to which that assumption held true. In this paper we compare mortality rates league tables for six cardiac surgery centres, developed using the administrative of hospital admissions and the specialised clinical (the registry), respectively. In particular, the aim was to assess whether the additional information provided by the specialised clinical had any impact on the performance assessment of the individual centres, as compared to the same effort undertaken simply relying on the administrative data integrated with information on patients EuroSCORE. 2. Materials and methods 2.1. General characteristics of the s The regional hospital admissions (SDO) was established in 1991, and since 1995 it has been routinely used for administrative purposes, including the attribution of each individual episode of care to the corresponding DRG. Primary and secondary diagnoses, as well as diagnostic and surgical procedures are recorded using the ICD-9-CM classification system. In short, information on patient age, gender, co-morbidity, complications and vital status at discharge is available. For patients undergoing cardiac surgery, information on their surgical risk according to the EuroSCORE is also available. In April 2002, a specialised registry for cardiac surgery procedures was established, with the aim of providing individual centres with detailed feedback on their activity and on the clinical outcomes of their patients. The registry collects more detailed information on the clinical characteristics of patients undergoing cardiac surgery, and on those of the surgery performed. While both sources include the EuroSCORE, this information is reported differently. While in the administrative only the overall score attributed to each patient by the surgical staff is explicitly reported, in the clinical each clinical characteristic concurring with the EuroSCORE calculation is reported, and the score is then estimated centrally Statistical analysis For the purposes of this analysis, 4017 of 4175 patients undergoing cardiac surgery at one of the 6 regional centres over the period January 1st December 31st 2003 were considered. Patients identified in the SDO but not in the registry were excluded (ns158). Case-mix adjusted in hospital mortality rate for each centre was assessed using both administrative and clinical data. Adjustment was performed using logistic regression models, applied separately to each source of information. In particular, the relationship between death and patient clinical characteristics was assessed by univariate analyses, and variables emerging as statistically significant were then included in a logistic regression model, whose calibration and prediction were assessed with the Hosmer-Lemeshow test, and the c statistic, respectively w6x. The results of the two models were used to estimate individual patient s likelihood of death, and the overall number of expected events for each centre. To evaluate the statistical significance of the difference between observed and expected events (O-E), the standard error of the observed rate was calculated according to the 1/2 following formula: wp = (1 P)ynx, where n is the total number of patients, and P is the expected mortality rate. Then a z statistic was calculated, dividing O-E by the standard deviation w7x. When z is Fy1.96 or G1.96, the difference can be considered statistically significant at a P-value F0.05. Risk-adjusted death rates were then calculated by multiplying the observedyexpected ratio by the overall regional mortality rate over the study period. The two different league tables thus obtained ranked centres according to the risk-adjusted rate estimated. In addition, the correlation between the two rankings was assessed with the Spearman correlation coefficient. Case volume of individual centres (overall and by type of surgery) and crude in-hospital mortality rates are reported in Table Results A description of the general characteristics of the 4017 patients considered, as drawn from the two s, is reported in Table 2. Prevalence of risk factors (hypertension, diabetes, obesity) was higher when assessed in the clinical. As for the surgical risk according to EuroSCORE, 67% of the patients scored G6 (i.e. high surgical risk) in the clinical, while the same figure drawn from the SDO was 48% (Table 2). Table 3 outlines the best fitting logistic regression models drawn from the two s. As shown, while the SDO model included only EuroSCORE and type of surgery, the one applied to the clinical additionally identified two variables not considered in the administrative (dialysis and priority). Nevertheless, the two models did not differ in terms of predictive ability, as expressed by the c statistic (78% vs. 77%). The implications of the use of the two regression models for estimating case-mix adjusted in-hospital mortality rates for individual centres are reported in Table 4. Adjusted mortality rates estimated from the two approaches differed substantially for two centres (by 1.9% and 1.6% for centres C and A, respectively), but were similar in the remaining cases. Nevertheless, one hospital (A) was identified by both models as a negative outlier (i.e. observed events exceeded the number predicted by both models, and the difference was statistically significant), while another (D) emerged as a positive outlier (i.e. number of observed events was significantly lower than expected) in the SDO analysis, but not in the clinical analysis.

3 D. Fortuna et al. / Interactive CardioVascular and Thoracic Surgery 5 (2006) Table 1 Individual centre caseload and in-hospital mortality for cardiac surgery, overall and by type of surgical procedure Cardiac surgery Centers In-hospital mortality rate A B C D E F Total by procedure n (%) n (%) n (%) n (%) n (%) n (%) n (%) Isolated CABG 182 (65.76) 424 (60.85) 157 (61.81) 651 (57.12) 433 (64.38) 251 (54.3) 2098 (52.23) 2.1 Isolated valve surgery 48 (17.14) 165 (22.18) 37 (11.64) 235 (19.93) 149 (20) 218 (29.03) 852 (21.21) 3.05 ValveqCABG 13 (4.64) 89 (11.96) 42 (13.21) 89 (7.55) 74 (9.93) 65 (8.66) 372 (9.26) 8.06 Other cardiac surgery 37 (13.21) 66 (8.87) 82 (25.79) 204 (17.3) 89 (11.95) 217 (28.89) 695 (17.3) 7.77 Total In-hospital mortality rate by centre Table 2 Information made available by clinical and administrative s Patient Clinical SDO (administrative characteristics ) % % EuroSCORE low risk: medium risk: high risk G missing 2.0 Priority elective 78 urgent 18.6 emergency 3.4 Previous operations** CABG valve surgery PTCA Body mass index underweight F normal weight: overweight: obese: G30* missing Hypertension Diabetes Dialysis Chronic pulmonary disease Bacterial endocarditis Serum creatinine (preoperative) G2 Mg 1.62 LV disfunction (ejection fraction) poor: -30% 3.76 moderate: 30 49% good: G50% missing 2.91 * Only severe obesity is reported in the administrative. ** Only previous surgery occurring within one year before the index episode of care is reported in the administrative. Table 3 Results of the logistic regression model prediction of in-hospital mortality for cardiac surgery, drawn from administrative and specialised clinical registry, respectively Risk factors Odds ratio (CI 95%): Odds ratio (CI 95%): administrative clinical EuroSCORE continuous 1.26 ( ) 1.24 ( ) variable Dialysis No 1.0 Yes 4.42 ( ) Priority Elective 1.0 Urgent 1.86 ( ) Emergency 2.07 ( ) Procedures Isolated CABG Isolated valve surgery 1.15 ( ) 1.26 ( ) ValveqCABG 2.71 ( ) 2.83 ( ) Other cardiac surgery 2.34 ( ) 4.42 ( ) Model fitting Hosmer Lemeshow test 2 x s6.3 Ps x s4.92 Ps0.77 c-statistic

4 126 D. Fortuna et al. / Interactive CardioVascular and Thoracic Surgery 5 (2006) Table 4 Predicted and risk-adjusted in-hospital mortality rates, drawn from clinical and administrative s, respectively Centre Observed Expected in-hospital mortality rates Risk adjusted in-hospital mortality rate in-hospital mortality rate Administrative Clinical Administrative Clinical % z score % z score % % A B y y C D y y E y y F Overall in-hospital mortality rate: 3.8%. League tables drawn from the two approaches are shown in Table 5. In both cases, the same best (i.e. rank 1) and worst (i.e. rank 6) centres were identified. Spearman correlation coefficient confirmed a high level of agreement between the two rankings (rs0.89; P-0.02). The use of the logistic EuroSCORE in alternative to the additive one did not change the result. 4. Discussion Exploring the implications of assessing clinical performance through different sources of information is not merely a technical exercise. Given the current emphasis on health care providers public accountability and on the value of performance indicators as a policy tool for improving quality of care, it is highly relevant to explore the extent to which those goals could be achieved by simply relying on sources of information already available or, alternatively, by developing specific and much more demanding tools. Thus, it is not surprising that this issue has been the object of a number of studies, especially in the area of cardiac surgery w8 13x. However, the results are quite conflicting, ranging from those convincingly supporting the use of administrative s for quality assessment purposes w9,10,12x, to those claiming the superiority of clinical s when it comes to developing mortality rates report cards or league tables w8,11,13x. According to our findings, individual centre mortality rates obtained from both sources do show some differences. For two centres in particular, adjusted mortality rates differed by about 2% (Table 4). However, the practical relevance of Table 5 Ranking of cardiac surgery centres based on case-mix adjusted in-hospital mortality rates for all types of surgery, drawn from administrative and clinical data, respectively Centre Ranking Administrative data Clinical data: Clinical data: logistic additive EuroSCORE EuroSCORE A B C D E F Spearman correlation coefficient: rs0.89, Ps0.02. those discrepancies was minimal, as the same centres were identified as outliers in both s, and had no substantial effect on their ranking. The same findings held true when the analyses were performed separately by type of surgical procedure: as shown in Table 6, rankings of case-mix adjusted mortality rates for isolated coronary artery bypass graft drawn from both s were strikingly similar. In addition, the same degree of consistency between clinical and administrative was confirmed when the analysis was repeated considering 30-day, rather than in-hospital, mortality, and using data from year 2004 (data not shown). Nevertheless, it is questionable whether outlier identification relying on mortality rates should be the only goal of performance monitoring. Monitoring the performance of cardiac surgery centres may well focus also on outcomes other than mortality, and should be done in such a way that surgical staff are not just passive objects of the evaluation, but actively involved in the assessment process. In this perspective, clinical s may have an obvious advantage over administrative ones. As shown in our study, they provide more complete information on outcomes overlooked by administrative data (i.e. complication rates) and, by their very nature, require the active participation of surgical teams in the data collection process. Another advantage offered by a clinical is the opportunity to rely on logistic rather than additive EuroSCORE in case mix adjustment. Indeed, the logistic EuroSCORE has been shown to perform better than the additive one in predicting mortality in high risk patients w14x. However, reliance on logistic EuroSCORE did not provide any difference in the ranking of our six centres (see Table 5). Table 6 Ranking of cardiac surgery centres based on adjusted in-hospital mortality rates for isolated CABG, drawn from the administrative and clinical, respectively Centre Ranking Administrative data A 6 6 B 2 2 C 1 1 D 3 3 E 5 5 F 4 4 Clinical data

5 D. Fortuna et al. / Interactive CardioVascular and Thoracic Surgery 5 (2006) In comparing administrative vs. clinical s, it is worth noting that the former can be partially modified to allow their exploitation for the assessment of clinical performance. In Emilia-Romagna region, the administrative of hospital admissions (SDO) has the peculiar feature of including key clinical information not relevant to administrative purposes, but deliberately collected to be used for assessing clinical performance taking into account patient complexity. In fact, since 1998, in cardiac surgery centres, the information describing individual patient characteristics in the SDO includes the surgical risk profile expressed with the EuroSCORE system. As others have already pointed out w10x, a key issue supporting the value of clinical s is their greater face validity to health care providers and health professionals. Indeed, if the objective of clinical performance monitoring is the development of report cards to be made available to the public, the degree of trust that health professionals place on the source of information used for the assessment has a crucial role in making the initiative acceptable. Nevertheless, although it is important, source of information is just one of the aspects affecting the credibility of performance monitoring initiatives for health professionals, and not necessarily the most relevant. Others include the extent to which the main objectives of these efforts are shared with health care providers, the relevance of the indicators used in monitoring clinical performance and, in general, the extent to which these efforts are perceived by health professionals to be supportive rather than punitive w15x. 5. Conclusions In ranking cardiac surgery centres according to their casemix adjusted mortality rates, an administrative including key clinical information provided the same results as a more complex and resource-demanding specialised. However, as in the case of our study, showing that inexpensive administrative s may reach the same conclusions as clinical s, in labelling individual hospitals as outliers, does not mean that specialised clinical s should necessarily be abandoned. While there is a large consensus on the need for monitoring hospital performance, it should be noted that the choice of a source of information over another implicitly implies also a choice between different approaches to quality improvement and models of accountability w15x. In such a context, the main added value of clinical s is the degree of direct participation of surgical teams required by their implementation. Such participation represents the necessary premise to expanding the scope of performance monitoring beyond the mere identification of bad apples by some external agency, rather emphasising quality assessment as a specific responsibility of individual surgical teams. Acknowledgements Funding: support for this study was provided by Agenzia Sanitaria Regionale, Bologna (Italy). References w1x Marshall M, Shekelle P, Brook R, Leatherman S. Dying to know: public release of information about quality of health care. London (UK): Nuffield Trust and RAND, w2x Treasure T. Mortality in adult cardiac surgery. Br Med J 2005;330: w3x Iezzoni LI. Assessing quality using administrative data. Ann Intern Med 1997;127: w4x Nobilio L, Fortuna D, Vizioli M, Berti E, Guastaroba P, Taroni F, Grilli R. Impact of regionalisation of cardiac surgery in Emilia-Romagna, Italy. J Epid Community Health 2004;58: w5x Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999;16:9 13. w6x Hosmer DW, Lemeshow S. Applied Logistic Regression. New York: John Wiley & Sons, w7x Goldfield N, Pine M, Pine J. Measuring and managing health care quality. Procedures, techniques, and protocols. Gaithersburg, MD: Aspen Publishers, w8x Hartz AJ, Kuhn EM. Comparing hospitals that perform coronary artery bypass surgery: the effect of outcomes measures and data sources. Am J Pub Health 1994;84: w9x Geraci JM, Johnson ML, Gordon HS, Petersen NJ, Shroyer AL, Grover FL, Wray NP. Mortality after cardiac bypass surgery. Prediction from administrative versus clinical data. Med Care 2005;43: w10x Ghali WA, Rothwell DM, Quan H, Brant R, Tu JV. A Canadian comparison of data sources for coronary artery bypass surgery outcome report cards. Am Heart J 2000;140: w11x Hannan EL, Racz MJ, Jollis JG, Peterson ED. Using Medicare claims data to assess provider quality for CABG surgery: does it work well enough? Health Serv Res 1997;31: w12x Landon B, Iezzoni LI, Ash AS, Shwartz M, Daley J, Hughes JS, Mackiernan,YD. Judging hospitals by severity-adjusted mortality rates: the case of CABG surgery. Inquiry 1996;33: w13x Hannan EL, Kilburn H, Jr, Lindsey ML, Lewis R. Clinical versus administrative data bases for CABG surgery. Does it matter? Med Care 1992;30: w14x Shanmigam G, Wets M, Berg G. Additive and logistic EuroSCORE performance in high risk patients. Interact CardioVasc Thorac Surg 2005;4: w15x Shahian DM, Normand SL, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, Dreyer PI. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 2001;72: