Comparison of Pediatric Cardiac Surgical Mortality Rates From National Administrative Data to Contemporary Clinical Standards

Transcription

1 Comparison of Pediatric Cardiac Surgical Mortality Rates From National Administrative Data to Contemporary Clinical Standards Karl F. Welke, MD, Brian S. Diggs, PhD, Tara Karamlou, MD, MS, and Ross M. Ungerleider, MD, MBA Division of Cardiothoracic Surgery and Department of Surgery, Oregon Health and Science University, Portland, Oregon; and Section of Cardiac Surgery, University of Michigan, School of Medicine, Ann Arbor, Michigan Background. Despite the superior coding and risk adjustment of clinical data, the ready availability, national scope, and perceived unbiased nature of administrative data make it the choice of governmental agencies and insurance companies for evaluating quality and outcomes. We calculated pediatric cardiac surgery mortality rates from administrative data and compared them with widely quoted standards from clinical databases. Methods. Pediatric cardiac surgical operations were retrospectively identified by ICD-9-CM diagnosis and procedure codes from the Nationwide Inpatient Sample (NIS) and the Kids Inpatient Database (KID) Cases were grouped into Risk Adjustment for Congenital Heart Surgery, version 1 (RACHS-1) categories. In-hospital mortality rates and 95% confidence intervals were calculated. Results. A total of 55,164 operations from the NIS and 10,945 operations from the KID were placed into RACHS-1 categories. During the 18-year period, the overall NIS mortality rate for pediatric cardiac surgery decreased from 8.7% (95% confidence interval, 8.0% to 9.3%) to 4.6% (95% confidence interval, 4.3% to 5.0%). Mortality rates by RACHS-1 category decreased significantly as well. The KID and NIS mortality rates from comparable years were similar. Overall mortality rates derived from administrative data were higher than those from contemporary national clinical data, The Society of Thoracic Surgeons Congenital Heart Surgery Database, or published data from pediatric cardiac specialty centers. Although category-specific mortality rates were higher in administrative data than in clinical data, a minority of the relationships reached statistical significance. Conclusions. Despite substantial improvement, mortality rates from administrative data remain higher than those from clinical data. The discrepancy may be attributable to several factors: differences in database design and composition, differences in data collection and reporting structures, and variation in data quality. (Ann Thorac Surg 2009;87:216 23) 2009 by The Society of Thoracic Surgeons As medicine enters the pay for performance era, governmental agencies, insurance companies, and consumer groups are increasingly interested in measuring the quality of pediatric cardiac surgery programs. Despite the superior coding and risk adjustment of clinical data, the ready availability, national scope, and perceived unbiased nature of administrative data make it the choice of these parties for evaluating heath-care quality and outcomes. Because administrative databases are being used to measure the quality of heath care and judge outcomes, it is imperative that one understand the information that they contain. However, there is a paucity of published pediatric cardiac surgical mortality standards from administrative data available for comparisons. Existing published reports of pediatric cardiac surgical mortality Accepted for publication Oct 14, Presented at the Fifty-fourth Annual Meeting of the Southern Thoracic Surgical Association, Bonita Springs, FL, Nov 7 10, Address correspondence to Dr Welke, Division of Cardiothoracic Surgery L353, Oregon Health and Science University, 3181 SW Sam Jackson Park Rd, Portland, OR ; welkek@ohsu.edu. rates from administrative data are either out of date or are based on regional populations [1, 2]. In addition, how mortality rates from administrative data compare with rates from contemporary clinical data is unknown. The purpose of this investigation is to calculate pediatric cardiac surgical mortality rates from administrative data and compare them with widely quoted standards from clinical databases. Material and Methods This study was designed as a retrospective cohort analysis. The Oregon Health and Science University Institutional Review Board approved this study. Owing to the nature of the study, patient consent was not required. We obtained data from the Nationwide Inpatient Sample (NIS) and the Kids Inpatient Database (KID) [3, 4]. Both databases were derived from the Uniform Hospital Discharge Data Set, a uniform, minimum dataset that allows investigation of cost and quality of short-term hospital services across regional and national populations. Each record represents a stay in an inpatient facility and 2009 by The Society of Thoracic Surgeons /09/$36.00 Published by Elsevier Inc doi: /j.athoracsur

2 Ann Thorac Surg WELKE ET AL 2009;87: SURGICAL MORTALITY RATES 217 contains data from the 1992 Uniform Bill hospital discharge abstract. The NIS is the largest all-payer inpatient care database in the United States [3]. The database is a stratified, cross-sectional sample that includes approximately 20% of all community (nonfederal) hospital discharges in the United States. The NIS data are available from 1988 to 2005, during which time the number of states in the NIS has grown from 8 to 37. In 2005, the database contained data on approximately 8 million hospital stays at 1,054 hospitals in 37 states. The sampling frame for the 2005 NIS is a sample of hospitals that comprises approximately 90% of all hospital discharges in the United States. To ensure the representative nature of the database, the NIS is stratified by geographical region, urban versus rural location, teaching status, hospital ownership, and hospital bed size. We have previously published a description of the characteristics of NIS hospitals that performed pediatric cardiac surgery [4]. The KID was specifically designed for research on issues related to the health of children [5]. Like the NIS, the large size and national scope of the KID make it well suited for study of national trends in health-care utilization, access, charges, quality, and outcomes. The KID is available for 1997, 2000, and The scope of the database has increased from data on patients 18 years of age and younger in 22 states in 1997 to data on patients 20 years of age and younger in 36 states in Each year the KID includes 2 million to 3 million pediatric discharges sampled from 2,500 to 3,500 American Hospital Association designated community hospitals. The sampling strategy for the KID differs from that for the NIS. To ensure an accurate representation of each hospital s pediatric case mix, the discharges are sorted by state, hospital, diagnosis-related group (DRG), and a random number within each DRG. Systematic random sampling is used to select 10% of uncomplicated in-hospital births and 80% of complicated in-hospital births and other pediatric cases from each hospital for which data are available. As with the NIS, sampling weights are provided so that the KID can be used to produce national estimates. For this study we combined data from the 1988 through 2005 NIS and used data from the 2003 KID. Congenital cardiac surgical procedures performed on patients younger than 18 years of age were identified by International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) diagnosis and procedure codes. For a patient to be included in this study, the procedure code had to match to a plausible diagnostic code. Operations were categorized by the Risk Adjustment for Congenital Heart Surgery (RACHS-1) method [1]. This risk stratification system groups the varied congenital cardiac surgical case mix into six categories based on similar expected short-term mortality rates and was developed to compare the mortality for groups of patients undergoing congenital cardiac surgery. The Aristotle basic complexity score can also be used for this purpose in clinical data; however, because it has not been linked to ICD-9-CM codes, it is not usable with administrative data [6]. Category 1 has the lowest risk of death and category 6 the highest. Category 1 contains atrial septal defect repair, patent ductus arteriosis closure on patients older than 30 days of age, and coarctation repair on patients older than 30 days of age. Category 2 contains operations such as ventricular septal defect repair, pulmonary valve replacement, total repair of tetralogy of Fallot, and Glenn shunt. Category 3 includes operations such as aortic valve replacement, Fontan procedure, and arterial switch. Category 4 includes complex neonatal surgery such as repair of transposition with ventricular septal defect and repair of truncus arteriosis. Category 5 includes tricuspid valve repositioning for Ebstein s anomaly at 30 days of age and combined repair of truncus arteriosis and interrupted arch. Category 6 includes Norwood procedure and Damus-Kaye-Stansel procedure. The RACHS-1 method is a widely used risk stratification methodology for congenital heart surgery [7 11]. The methodology has been validated and is included in The Society of Thoracic Surgeons (STS) Congenital Heart Surgery Database reports [12]. Owing to the small number of patients in RACHS-1 category 5 and to allow for comparison with the STS Congenital Heart Surgery Database, risk categories 5 and 6 were collapsed into a single group. To decrease the year-to-year variability, the 18-year time span of the NIS was subdivided into 6 groupings of 3 years each. Analyses were performed using SAS version 9.1 (SAS Institute, Cary, NC). Mortality was defined as in-hospital mortality as indicated by the discharge disposition. The numerator was the number of deaths and the denominator was the number of patients. We calculated case mix and in-hospital mortality rates for each RACHS-1 category. Comparisons of case mix and mortality rates between year groupings were made using 95% confidence intervals. We then compared the NIS and KID case mix and mortality rates with published mortality rates from contemporary clinical databases. We chose the Congenital Heart Surgeons Society (CHSS) cohort reported by Welke and colleagues [10] and the STS cohort reported by Jacobs and associates [13]. The CHSS cohort contained 12,672 RACHS-1 categorized cases from 2001 to 2004 voluntarily submitted by 11 institutions with CHSS member surgeons [10]. The STS cohort contained 45,635 RACHS-1 categorized operations from 2002 to 2005 voluntarily submitted by North American Congenital Heart Surgery Centers. The number of submitting centers increased from 20 in 2002 to 52 in 2005 [13]. Results From 1988 to 2005, 124,087,005 discharges from 3,948 hospitals were recorded in the NIS. Of these discharges, 55,164 were patients who underwent a congenital cardiac procedure coded in RACHS-1. These operations occurred at 307 hospitals. During the 18-year study period, the case mix of NIS hospitals performing pediatric cardiac surgery remained similar with the notable exception of category 5 and 6 (Table 1). The percentage of category 5 and 6 cases in the

3 218 WELKE ET AL Ann Thorac Surg SURGICAL MORTALITY RATES 2009;87: Table 1. Nationwide Inpatient Sample Case Mix and Mortality Rates by Risk Adjustment for Congenital Heart Surgery, Version 1, Category ( ) a Case Mix Mortality RACHS-1 Category Number of Cases Percentage 95% CI Number of Deaths Percentage 95% CI , , , , , , , , , , , , , , , , , , , & Total , , , , , , a Case counts and deaths are raw data from the Nationwide Inpatient Sample. Percentages and confidence intervals are based on weighted values. CI confidence interval; RACHS-1 Risk Adjustment for Congenital Heart Surgery, version 1. NIS increased steadily from 0.51% to 3.32% (p 0.05). This significant increase likely reflects the acceptance of the Norwood procedure (a RACHS-1 category 6 operation) as treatment for hypoplastic left heart syndrome and other single-ventricle anomalies. During the course of the study period, the overall in-hospital mortality rate for congenital cardiac surgical procedures performed at NIS hospitals decreased from 8.7% to 4.6% (p 0.05). Mortality rates in all RACHS-1 categories declined during the study period as well

4 Ann Thorac Surg WELKE ET AL 2009;87: SURGICAL MORTALITY RATES 219 Table 2. Comparison of Administrative and Clinical Data Case Mix and Mortality Rates by Risk Adjustment for Congenital Heart Surgery, Version 1, Category a Case Mix Mortality RACHS-1 Category Number of Cases Percentage 95% CI Number of Deaths Percentage 95% CI 1 NIS STS [5] NIS CHSS [4] KID 2003 [1] NIS STS [5] NIS CHSS [4] KID 2003 [1] NIS STS [5] NIS CHSS [4] KID 2003 [1] NIS STS [5] NIS CHSS [4] KID 2003 [1] &6 NIS STS [5] NIS CHSS [4] KID 2003 [1] Total NIS STS [5] NIS CHSS [4] KID 2003 [1] Numbers in brackets are reference citations. a Case counts and deaths are raw data from the Nationwide Inpatient Sample. Percentages and confidence intervals are based on weighted values. CI confidence interval; CHSS Congenital Heart Surgeons Society; KID Kids Inpatient Database; NIS Nationwide Inpatient Sample; STS The Society of Thoracic Surgeons. (Table 1). Owing in part to the small number of cases performed in the early years, the decrease in category 5 and 6 mortality from 40.5% to 29.1% did not reach statistical significance. In the KID 2003, 1,905,797 discharges from 2,521 hospitals were recorded. Of these discharges, 10,945 were patients who underwent a congenital cardiac procedure coded in RACHS-1. These operations occurred at 151 hospitals. Thirty of these hospitals were children s general hospitals, 62 children s units in general hospitals, and 59 were not identified as children s hospitals. The mortality rates for the KID were similar to contemporary NIS data (Table 2). Although there were slight differences in case mix between the administrative and clinical databases, there was no consistent pattern of discrepancy (Table 2). The slight disparities may reflect the differences in the hospital compositions of the cohorts. The NIS includes a broad range of hospitals chosen to be representative of national practice. The STS and CHSS cohorts include groups of hospitals that are more likely to contain pediatric cardiac specialty centers.

5 220 WELKE ET AL Ann Thorac Surg SURGICAL MORTALITY RATES 2009;87: Overall mortality in the NIS administrative data was higher than mortality rates from contemporary clinical databases (Table 2). Category-specific mortality rates from the STS 2002 to 2005 and CHSS 2001 to 2004 clinical cohorts were similar to the NIS administrative data for category 1, 2, 4, and 5 and 6 cases. Mortality rates from the clinical databases were significantly lower than those from the NIS for category 3 cases. Comment In this analysis of 18 years of national administrative data, we found that pediatric cardiac surgical mortality rates have decreased substantially. However, compared with contemporary clinical databases, overall mortality rates in administrative data were higher. The discrepancy may be attributable to several factors: differences in database design and composition, differences in data collection and reporting structures, and variation in data quality. This comparison is timely as governmental agencies, insurance companies, and other stake holders attempt to define the data sources and metrics that will be used to evaluate providers in terms of cost and quality in the pay for performance movement [14]. Administrative databases were designed for the collection of data about claims and billing. Subsequently, these data were used for calculating publicly reported surgical rates of mortality and for profiling of providers. Two administrative databases have applicability to congenital cardiac surgery, the NIS and the KID. Administrative databases are relatively inexpensive and readily available, and include large groups of patients from state or national areas. In addition, they are often available for a number of years, facilitating longitudinal studies. Because of their large size, these databases can generate sample sizes often not available in single-center or even multiinstitutional databases. This large volume of data is especially helpful for the study of rare diagnoses and procedures. The large size of administrative databases also may mitigate, in part, coding inaccuracies. Because they were designed for billing purposes, administrative databases excel as sources of financial data not available from other sources [15]. Administrative databases are inclusive by design. They either include all hospitals within a specified geographic area, such as state-level databases, or use a stratified sampling design that allows a fixed percentage of hospitals to accurately represent the entire sampling universe. Findings from within the sample can, therefore, be readily generalized to the larger population from which the sample was selected. By including information from both high- and low-performing and high- and low-volume hospitals, administrative data can be used to evaluate the practice of hospitals that are less likely to participate in large, voluntary, clinical databases. The broader scope of administrative data is emphasized by the observation that, using the NIS, we identified 307 hospitals in the United States at which pediatric heart surgery was performed. This is in contrast to the 122 centers identified by the 2005 Society of Thoracic Surgeons Congenital Heart Surgery Practice and Manpower Survey [16]. This suggests that a minority of hospitals performing congenital heart surgery in the United States consider themselves specialty centers. The low surgical volumes at many centers support this assumption [4]. Inclusiveness also provides a unique opportunity to study trends and patterns among regional or national geographic areas. Despite these advantages, administrative data have important limitations [17]. Many of these are a result of the documentation of the clinical status of the patients using codes from the ICD-9-CM. Although in general these codes from the ICD-9-CM capture a great amount of detail about diagnoses and procedures, multiple areas exist in which the codes are nonexistent or lack the desired granularity [18, 19]. For example, there is no procedure code for the Norwood operation. To select Norwood operations from an administrative database, one must construct a composite coding algorithm that contains individual procedural elements encompassing the Norwood operation. In addition, diagnostic codes from the ICD-9-CM do not capture many findings from physical examination, diagnostic findings, laboratory values, and hemodynamic measurements that have prognostic value and importance in risk models. The complex case mix of pediatric cardiac surgery and the structure of the collection of administrative data lead to considerable variation in data quality [20, 21]. The administrative coding personnel who obtain the 1992 Uniform Bill data from abstraction of the chart are skilled in coding, but they are not clinicians and they have no contact with the clinical team or the patient; their abstractions are derived solely from what is explicitly stated in the medical record. Variation in the quality of administrative data may also result in part from the agenda for coding being financially driven. A greater impact likely comes from a combination of the coders limited knowledge of pediatric cardiac surgery, their restricted ability to clarify conflicts in the data and fill in missing data, and poor or inconsistent documentation in the medical record [22]. In addition, inasmuch as procedures in similar categories may be performed in both the operating room and the cardiac catheterization laboratory, miscoding of interventional procedures as surgical procedures occurs. On the other hand, coding personnel are unbiased. They have no stake in the hospital outcomes and, therefore, have no incentive to favorably report data. Clinical databases are maintained by several groups: professional organizations, such as the STS, the European Association for Cardio-Thoracic Surgery, and the Pediatric Cardiac Care Consortium; states, such as the New York Cardiac Surgery Reporting System; hospitals; and private groups [23, 24]. Many of these databases, such as the STS Congenital Heart Surgery Database, were designed for quality improvement. Clinical data are collected by clinical personnel, who have a better knowledge of cardiac surgery than do administrative coding personnel. In circumstances in which they are not intimately involved in the process of care of the patient, they can identify where preoperative patient-level comorbidi-

6 Ann Thorac Surg WELKE ET AL 2009;87: SURGICAL MORTALITY RATES 221 ties are recorded, review the operative notes for surgical data, and follow the patient daily to track complications. As a result, clinical data may be more accurate than administrative data; however, they too suffer from a limited number of elements of data, inaccuracies of coding, and the lack of long-term follow-up. The close association between those who collect the data and those invested in how the data are used has led to the presumption that physicians and other health-care providers will game the system through the data collection process. This concern has increased with the implementation of reporting of medical and surgical outcomes to the public and governmental pay-for-participation programs, along with the potential implementation of initiatives involving pay for performance, and has led some to question the validity of clinical data. Although there is no evidence that STS hospitals game the system by underreporting mortality, the STS data quoted in this study were not validated. The STS has since (2007) implemented a site visit data validation program to address this concern. Participation in most clinical databases is voluntary, resulting in important biases regarding outcomes. Hospitals with limited resources or those with lower performance may abstain from participation, thus weakening the comparative power of clinical databases and potentially inflating favorable results derived from participants. In particular, hospitals that choose to participate must have the infrastructure in place to collect data, which may differentiate them from nonparticipating hospitals. They may be more likely to also engage in other practices that lead to better outcomes. Rather than being broad cohorts of all comers like the NIS and the KID, the clinical databases used in this study were select subsets that could be expected to contain a greater proportion of institutions specializing in congenital cardiac disease. The CHSS cohort was a group of hospitals that were presumed to be committed to excellence in congenital heart disease based on surgeon membership in the Congenital Heart Surgeons Society. The STS cohort was a group that voluntarily chose to participate in the STS Congenital Heart Surgery Database. Although coding errors are widely perceived to be mainly a drawback of administrative data rather than clinical data, such a perception is oversimplified and potentially dangerous, as both are likely to contain some level of inaccuracy. The clinical data used in this study were submitted by the responsible surgeons or centers and were not validated. In a recent study, the Toronto Cardiovascular Surgery Database for Congenital Heart Surgery was deliberately seeded with three types of errors: errors of omission of data, errors of miscoding of outcomes such as alive or dead, and the miscoding of procedures [25]. Expectedly, random errors had little effect, but rates of mortality calculated from the seeded database and the pristine database were sensitive to even small levels of miscoding of procedures and outcomes. The impact of these coding errors on an analysis varies depending on the focus of the investigation. If coding errors are random within the population, their impact is diminished by the large sample sizes available with administrative and, to a certain extent, clinical data. The result of such random miscoding would be to bias the results of an analysis toward the null hypothesis. However, in a small subset of the data, potential errors may not be randomly distributed and may confound the findings. When studying operations, errors from miscoded data can be reduced by including records in which the procedural code matches to a plausible diagnostic code as was done in the present investigation. For example, a patient would have to both have the diagnosis of tetralogy of Fallot and undergo the procedure of repair of tetralogy of Fallot to be included in a cohort. Using this same strategy one can also reduce the number of patients with associated lesions, such as atrioventricular septal defect with tetralogy of Fallot, from being included in isolated lesion studies. Administrative and clinical databases each have strengths that make them suited for different, but complementary, purposes. Administrative data are widely available to payers, governmental agencies, and groups that produce Internet and other reports that reach patients, their families, and providers. Such data provide an unbiased, national picture of the practice of medicine. High- and low-performing hospitals are included, allowing one to evaluate the practice of hospitals that are less likely to participate in voluntary, clinical databases. Because they were designed for billing, administrative databases are rich in cost data, making them useful for economic analyses. Clinical databases have more detailed operative and complication data. They are the choice for investigations and risk stratifications that rely on accurate comorbidity and risk factor data, such as determining performance benchmarks. For such efforts to be most useful, the validity of both administrative and clinical data must be improved. Ideally, future work to take advantage of the strengths of both types of data and to minimize their weaknesses will lead to the best understanding of the practice and outcomes of pediatric cardiac care. References 1. Jenkins KJ, Gauvreau K, Newburger JW, Spray TL, Moller JH, Iezzoni LI. Consensus-based method for risk adjustment for surgery for congenital heart disease. J Thorac Cardiovasc Surg 2002;123: Bazzani LG, Marcin JP. Case volume and mortality in pediatric cardiac surgery patients in California, Circulation 2007;115: HCUP Nationwide Inpatient Sample (NIS). Healthcare Cost and Utilization Project (HCUP) Agency for Healthcare Research and Quality, Rockville, MD. Available at Accessed March 1, Welke KF, Diggs BS, Karamlou T, Ungerleider RM. The relationship between hospital surgical case volumes and mortality rates in pediatric cardiac surgery: a national sample, Ann Thorac Surg 2008;86: HCUP Kids Inpatient Database (KID). Healthcare Cost and Utilization Project (HCUP) Agency for Healthcare Research and Quality, Rockville, MD. Available at www. hcup-us.ahrq.gov/kidoverview.jsp. Accessed November 5, 2007.

7 222 WELKE ET AL Ann Thorac Surg SURGICAL MORTALITY RATES 2009;87: Lacour-Gayet F, Clark D, Jacobs J, et al. The Aristotle score: a complexity-adjusted method to evaluate surgical results. Eur J Cardiothorac Surg 2004;25: Jenkins KJ, Gauvreau K. Center-specific differences in mortality: preliminary analyses using the Risk Adjustment in Congenital Heart Surgery (RACHS-1) method. J Thorac Cardiovasc Surg 2002;124: Boethig D, Jenkins KJ, Hecker H, Thies WR, Breymann T. The RACHS-1 risk categories reflect mortality and length of hospital stay in a large German pediatric cardiac surgery population. Eur J Cardiothorac Surg 2004;26: Kang N, Cole T, Tsang V, Elliot M, de Leval M. Risk stratification in paediatric open-heart surgery. Eur J Cardiothorac Surg 2004;26: Welke KF, Shen I, Ungerleider RM. Current assessment of mortality rates in congenital cardiac surgery. Ann Thorac Surg 2006;82: Al-Radi OO, Harrell FE Jr, Caldarone CA, et al. Case complexity scores in congenital heart surgery: a comparative validation study of the Aristotle Basic Complexity score and the Risk Adjusted Congenital Heart Surgery (RACHS-1) system. J Thorac Cardiovasc Surg 2007;133: The Society of Thoracic Surgeons Congenital Heart Surgery Database 2007 Congenital Report Executive Summary. Available at Congenital_Report_Executive_Summary_All_Patients.pdf. Accessed November 5, Jacobs JP, Jacobs ML, Lacour-Gayet FG, et al. Case-mix complexity adjustment using both RACHS-1 and Aristotle applied to operations in the STS congenital database. Pediatr Cardiol In press. 14. Rosenthal MB, Fernandopulle R, Song HR, Landon B. Paying for quality: providers incentives for quality improvement. Health Affairs 2004;23: Glance LG, Dick AW, Osler TM, Mukamel DB. Does date stamping ICD-CM codes increase the value of clinical information in administrative data? Heath Serv Res 2006;41: Jacobs ML, Mavroudis C, Jacobs JP, Tchervenkov CI, Pelletier GJ. Report of the 2005 STS congenital heart surgery practice and manpower survey: a report from the STS work force on congenital heart surgery. Ann Thorac Surg 2006;82: Shahian DM, Silverstein T, Lovett AF, Wolf RE, Normand SLT. Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards. Circulation 2007;115: Iezzoni LI. Assessing quality using administrative data. Ann Intern Med 1997;127: Mack MJ, Herbert M, Prince S, et al. Does reporting of coronary artery bypass grafting from administrative databases accurately reflect actual clinical outcomes? J Thorac Cardiovasc Surg 2005;129: Cronk CE, Malloy ME, Pelech AN, et al. Completeness of state administrative databases for surveillance of congenital heart disease. Birth Defects Res A Clin Mol Tetratol 2003;67: Frohnert BK, Lussky RC, Alms MA. Validity of hospital discharge data for identifying infants with cardiac defects. J Perinatol 2005;25: Finlayson EVA, Birkmeyer JD, Stukel TA, Siewers AE, Lucas FL, Wennberg DE. Adjusting surgical mortality rates for patient comorbidities: more harm than good? Surgery 2002; 132: Moller JH, Powell CB, Joransen JA, Borbas C. The pediatric cardiac care consortium revisited. Jt Comm J Qual Improv 1994;20: Pediatric Congenital Cardiac Surgery in New York State Albany: New York State Department of Health, Gallivan S, Stark J, Pagel C, Williams G, Williams WG. Dead reckoning: can we trust estimates of mortality rates in clinical databases? Eur J Cardiothorac Surg 2008;33: DISCUSSION DR JOHN EDMUND MAYER (Boston, MA): Thanks very much and congratulations on a nice presentation and very nice work comparing this data. I think you have correctly emphasized the importance of clinical databases versus those that are based on administrative claims data. You may or may not have seen a paper in Circulation earlier this year that compared the coronary bypass, numbers of cases, in Massachusetts compared to those derived from administrative claims data. Frighteningly, there was a 27% difference in simply the number of cases counted as isolated coronary artery bypass, which I think makes the point that the administrative claims data is at least potentially flawed. I think that the real strength is actually if one can merge the clinical data with the financial administrative claims data that is contained in the HCUP (Healthcare Cost and Utilization Project) or other databases, and I am sure you are aware that in Virginia, at least, the Virginia Cardiac Surgical Quality Initiative actually was able to do that and then was actually able to find out what the savings were if one reduced complication rates as a for instance. I think where the real power is going to come from is the actual ability to merge the clinical data, which I think we all feel is more likely to be accurate, with the financial data that comes out of the hospital data sets. So those considerations lead me to two questions, one of which you already alluded to, and one is, what were the number of cases, or give us some idea about the percentage of cases that you could not classify using RACHS to risk adjust? What percentage couldn t be captured by that system? In the early years of RACHS, there were as many as 12% to 15% of the cases that couldn t be classified. So that is question one. And then the second question is whether or not you have actually gone ahead and started to look at some of that financial data in that HCUP database and whether or not you found any relationships between clinical outcomes and cost? DR WELKE: Dr Mayer, thank you very much for your questions. I will address the Risk Adjustment for Congenital Heart Surgery 1 (RACHS-1) question first. The percentage of cases in a database that are classified by RACHS-1 depends on the scope of the database and the denominator that one chooses. There are specific cases that were excluded from RACHS-1 when it was developed because of concerns that the mortality from these cases was not directly related to the operation. Two notable operations are ligation of patent ductus arteriosis in patients under 30 days of age and heart transplantation. When choosing a denominator, one can include all cardiac cases or only cases that utilized cardiopulmonary bypass. Additionally thoracic cases can be included or excluded. Depending on how the denominator is chosen, the percentage of cases in a database that will be captured by RACHS-1 can range from 80% to 90%. The Aristotle Basic Complexity Score, which is also used in The Society of Thoracic Surgeons Congenital Heart Surgery Database, captures a higher percentage of operations but is not yet coded for use in administrative data. We are working on that. To answer your second question, there has been a lot of discussion at this meeting about the flaws of administrative data;

8 Ann Thorac Surg WELKE ET AL 2009;87: SURGICAL MORTALITY RATES 223 however, there are areas where administrative data are very rich. Since administrative databases were designed to capture billing information, they are very rich in financial data. We have not yet examined cost data; however, we plan to do so. As you suggested, combining the strengths of administrative data and clinical data will give us the best overall picture of our practice. The first step toward this end is to compare the coding of variables in the two databases. The Society of Thoracic Surgeons is working with the Agency for Healthcare Research and Quality to compare The Society of Thoracic Surgeons short lists of congenital diagnosis and procedure codes to International Classification of Diseases, 9th Clinical Modification (ICD-9-CM) codes used in administrative databases so that we can begin to scrutinize the discrepancies between clinical data and administrative data. After this work has been done, we can engage in research that takes advantage of the superior coding of clinical data in a clinical database and the billing data available in an administrative database. Thank you. DR FREDERICK GROVER (Denver, CO): Karl, this is a very important paper and well presented and analyzed, and I think John s points are very important. And I would just like to emphasize that our Washington office is constantly trying to push quite a bit against administrative data as a measure of quality or a measure of how we are working, but, exactly, trying to take advantage of its stronger points, which is the financial data, and, potentially with CMS (Centers for Medicare and Medicaid Services), late mortality data. It is a way that we can get into long-term follow-up quickly before we do our own late follow-up. But just to give you an idea, when Jeff Rich did the Virginia demonstration project, if you used administrative data to identify the best users of internal mammary artery, which is one of the measures that adult cardiac surgeons are so-called judged by, an institution would have been awarded on their administrative data by being the best when it was really the worst. There are just certain things that you can t count on in administrative data. But the problem is it is easy and it is cheap, and even though a lot of it is erroneous, it will be pushed, and this is why it is so important, and I think other specialties will figure this out when they get report cards or whatever, that if you don t have your own clinical database where you are really collecting the important things that you are liable to be judged inappropriately. You mentioned that you would like to improve the quality of the data in the administrative databases, and I guess my question is, since most of this is billing and coding data and it is motivated to bill as much as you can and get as much credit, how would you do that? How would you envision improving the quality of the administrative data collected? DR WELKE: Thank you, Dr Grover. The first step in improving the quality of administrative data is to look for coding discrepancies by cross coding the diagnoses and procedure codes for administrative data with those from clinical data as we are doing in concert with the Agency for Healthcare Research and Quality. We have to see how accurate and complete the data are before we can move forward with investigations of billing correctness and cost. It is exciting that those who have the data, the Agency for Healthcare Research and Quality, want to work with us to check the quality of their data. Of course, they are hoping the administrative data look good. We think that the clinical data are going to look good. We will probably end up somewhere in the middle, but that is the place to start. Hopefully this effort will result in changes that will improve the coding and quality of administrative data and everyone will benefit.