In two articles in this supplement [1, 2], the conceptual

Transcription

1 Criteria by Which to Evaluate Risk-Adjusted Outcomes Programs in Cardiac Surgery Jennifer Daley, MD Health Services Research and Development and Department of Medicine, Brockton/West Roxbury Veterans Affairs Medical Center, West Roxbury, and Division of General Medicine and Primary Care, Department of Medicine, Beth Israel Hospital, Harvard Medical School, Boston, Massachusetts Risk adjustment models for hospitalized patients are most advanced for the assessment of the clinical outcome of cardiac procedures, and for coronary artery bypass grafting in particular. The goal of being able to use outcomes as a credible indicator of quality of care has stimulated the development of several programs that use reliable, valid patient data collected during the surgical episode to adjust outcomes for the severity of illness. Several criteria that are useful in the assessment of risk adjustment methods for outcome and quality-of-care investigations are discussed in detail and five of these programs are compared. The programs have more similarities than differences and identify many of the same patient characteristics predictive of a higher likelihood of mortality in the period immediately after operation. Whether persistent differences in mortality after risk adjustment across institutions or individual surgeons, or both, may ultimately be attributed to the process and structure of care needs further study and investigation. Similar methods should be applied to other outcomes of importance to patients, their families, and their physicians, such as surgically related morbidity, functional status, quality of life, costs, and patient-reported perceptions of the nontechnical aspects of their care. (Ann Thorae Surg ) In two articles in this supplement [1, 2], the conceptual basis of risk adjustment from the standpoints of quality of care and outcomes research and the dimensions of risk and outcomes to be considered in assessing a risk adjustment system have been outlined. In this article, I discuss several issues that must be addressed in developing or evaluating a risk adjustment system for medical outcomes. Criteria are offered and questions are posed that clinicians, administrators, and methodologists may want to consider when assessing risk adjustment systems. Because cardiac surgical treatment for atherosclerotic cardiovascular disease is so common in the United States and is associated with significant mortality, morbidity, and costs, it has been the subject of intensive investigation into the patient risk factors that affect outcome, particularly surgically related mortality [3-17]. The criteria and questions for assessing risk adjustment systems are applied to each of five risk adjustment programs for coronary artery bypass grafting currently in use in the United States. The patient-level variables used in these five risk models are also compared. Criteria by Which to Evaluate Risk-Adjusted Outcomes Models Narrowly defined, the purpose of risk adjustment is to account for patient characteristics that influence patient Presented at the National Symposium on Using Outcomes Data to Improve Clinical Practice: Building on Models From Cardiac Surgery, Keystone, CO, June 6-7, Address reprint requests to Dr Daley, West Roxbury VAMC, 1400 VFW Pkwy, West Roxbury, MA by The Society of Thoracic Surgeons outcomes. Broadly defined, risk adjustment seeks to account for all factors other than the processes and structures of care that may explain variation in patient outcomes. Three factors may account for observed differences in outcomes: differences in significant risk factors among patients, random variation, or differences in the processes or structures of care. Focusing on outcomes adjusted for risk is an indirect way of studying the differences in processes or structures of care. Random variation is taken into account through the application of a variety of statistical techniques, a description of which is beyond the scope of this article, but they have been discussed elsewhere [18,19]. Designing or evaluating an adequate risk adjustment system to assess the contribution of patient risk factors requires consideration of several criteria that are outlined in the following sections. Five programs that use risk adjustment methods for cardiac surgery have been evaluated according to these criteria, and the findings are summarized in Table 1. Each program was assessed through a review of the published literature about each program. This assessment was sent to the primary developers of each program for their comments and revisions. Table 1 represents the state of each risk adjustment program as of June What Are the Outcomes of Interest? In-hospital or hospitalization-associated mortality is the outcome that has been most frequently studied. In the field of cardiac surgery, mortality and survival are clearly important outcomes. The bypass grafting procedure carries with it a small, but significant, risk of death and, for some subsets of the population of patients with athero /94/$7.00

2 1828 OUTCOMES DATA DALEY Ann Thorae Surg Table 1. Comparison of Five Programs Using Risk-Adjusted Outcomes in Coronary Artery Bypass Grafting Characteristic Outcome Time frame of outcome(s) Model validation New York State Unit(s) of analysis Patient, medical center, surgeon Data source Prospective Data collectors Standardized definitions Data entry Data reliability Sample population Time frame of data collection for model development Size of data set Analytic technique Attributional validity Mortality In -hospital Surgical teams PC-based data entry system Sample of charts reviewed by utilization review coordinators All cardiac surgery programs in New York State Split sample cross validation Northern New England 57,000 3,404 Multivariate logistic Multivariate logistic Site visit and chart review Mortality In-hospital Patient, medical center, surgeon Prospective Surgical teams Patient Parsonnet Mortality In-hospital Paper forms Five clinical centers in Existing database northern New England (Maine, NH, Vermont) 7/1987-4/1989 Split sample cross validation Chart review and site visit The Society of Thoracic Surgeons Mortality In-hospital Patient Retrospective from Prospective and/or existing retrospective chart database review Archivists, RN Data managers clinicians, researchers, perfusionists ("not senior surgeons") ,935 (1) Additive multiple (2) Logistic Prospective validation at seven New Jersey hospitals ; training programs available for data managers Paper forms and data entry into computer software Random chart review by New Jersey Department of Health Academic medical centers, military hospitals, community hospitals with cardiothoracic and thoracic surgery programs ,000 Bayesian model Split sample cross validation Veterans Health Administration (CICSS) Mortality, morbidity Mortality: any death within 30 days postoperatively and any postoperative death directly related to operation Morbidity: selected postoperative complications Patient, VAMC Prospective Before 1991, surgical team; after 1991, nurse data collector, training programs and help line for data collectors Paper forms and data entry into computer software (75%) Interrater reliability checked periodically 43 VAMCs performing all cardiac operations invha ~ 35,000 Multivariate logistic Split sample cross validation Site visits and chart reviews by Cardiac Consultants Committee continued

3 Ann Thorac Surg OUTCOMES DATA DALEY 1829 Table 1. Continued Characteristic Performance of prediction models Treatment of missing data How often are new models generated? References Comments New York State Observed/expected death rates by different ranges of predicted probabilities of death 5,6 Veterans Health Northern New The Society of Administration England Parsonnet Thoracic Surgeons (CICSS) ROC curves (C index) Observed/expected Observed/expected ROC curves (C Hosmer-Lerneshow ratios by ranges death rates by index) statistic of preoperative quintiles of risk score mortality; observed/ expected rates by ranges of predicted probabilities Chart is not used If more than 20% of «2% of data missing, hospitals) Yearly variable not used; cases with missing values are not used New models have been developed three times with change in data forms since ,16,17 Includes patients Valve operation with CABG and model separate valve operation CABG = coronary artery bypass grafting; CICSS ~ Continuous Improvement in Cardiac Surgery Study; PC ~ personal computer; RN = registered nurse; ROC = receiver operating characteristic; VAMC = Veterans Affairs Medical Center; VHA = Veterans Health Administration. sclerotic cardiovascular disease and angina refractory to medical therapy, the procedure is performed to prevent death. Complications and potentially preventable morbidity after the grafting procedure are also important outcomes. Imprecise definitions of these events and the human tendency to underreport these events in medical records may limit the reliable identification of these nonfatal outcomes. For patients and their families, outcomes after discharge, such as the ability of patients to function independently, to interact with families and friends, and to return to work and daily routines, are critically important. Unfortunately, as the interval after the surgical procedure lengthens, the ability to collect information about patients' functional status and quality of life becomes more difficult. The science of measuring these outcomes has made considerable strides in the past 10 years, but measurement instruments that are both widely used and accepted by the clinical community are just emerging. As a result, hospital-associated mortality is the most widely used outcome, partially because it is easy to measure and readily available. Ideally, a range of pertinent outcomes (ie, mortality, morbidity, changes in functional status and quality of life, cost of care, and patient-reported perceptions of the nontechnical aspects of care) should be reported for each procedure. Within What Time Frame Should Each Outcome Be Measured? The time frame, or window of observation, should be determined for each outcome. Some options include the hospitalization, a fixed period (eg, 30 days) after the procedure, the episode of care (eg, from the onset of symptoms through the time of cardiac rehabilitation), or a variable period initiated by the patient's admission, by the performance of a procedure, or by the diagnosis of a condition. The window of observation may vary according to each outcome. For example, information on hospital mortality and morbidity may be collected during the hospitalization, whereas patient functional status and quality of life may be assessed at several different times in the weeks to months after operation. Collecting inhospital mortality data may be biased by the variability in discharge practices in different hospitals and geographic regions of the country. For example, the mortality rate noted for a hospital that rapidly discharges its patients to cardiac rehabilitation hospitals after coronary artery procedures may appear to be lower than the rate noted for a hospital that keeps its patients in the hospital so that they can undergo cardiac rehabilitation. It is therefore better to identify a fixed period in terms of days or weeks after the procedure for determining postprocedural morbidity and mortality to eliminate this bias. The time frame for each outcome also influences the nature of significant patient risk factors. For example, the patient risk factors for 3D-day mortality may be significantly different from the risk factors for functional status at 6 months postoperatively. Acute clinical stability is the most important component of risk for studies involving short time frames. Chronic disability, physical functioning, and various nonclinical factors increase in importance as the time window expands. There is little systematic information available about sources of long-term patient risks or outcomes in cardiac surgery. What Are the "Units ofanalysis" of Interest? Inherently a comparative exercise, the most appropriate comparisons in outcomes assessment depend on who will be using the risk-adjusted outcome information and

4 1830 OUTCOMES DATA DALEY Ann Thorac Surg their purposes. Depending on the context, the unit of observation may be the individual patient, or it may be groupings of patients according to individual surgeons, groups of surgeons, a hospital, or groups of hospitals. These units are not mutually exclusive, in the sense that smaller units are nested within larger units. In cardiac surgery, individual surgeons or groups of surgeons as well as hospitals are compared. The use of the individual patient as a unit of observation deserves special attention. Some investigators advocate the use of these risk models as clinical prediction rules. Diamond [20, 21] argues that, for example, the predicted probability of death obtained by entering an individual patient's risk factors into the risk model generates information that may be helpful to that patient and his or her physician in making decisions about whether to pursue cardiac surgical treatment. Although this predicted probability of death may be one of many helpful pieces of clinical information available to the patient and surgeon, caution should be used in interpreting this predicted probability of death. First, the predicted probability of death generated by these risk adjustment models reflects the likelihood of death for that patient if he or she were operated on in the average hospital by the average surgeon of all the hospitals and surgeons in the database from which the models were developed, and not the predicted probability with respect to the specific hospital where the patient is or with respect to the surgeon whom the patient has consulted. Second, the uncertainty about the predicted probability of death for an individual patient is usually great. These predicted probabilities of death should be shared with patients cautiously and with the stated understanding that the accuracy of the prediction is quite limited. From What Sources Are the Data Collected? How Much Does It Cost in Terms of Human Resources, Time, and Dollars to Collect the Data? Three sources of data are available for risk adjustment: discharge abstract information, retrospective chart review, and prospective primary data furnished by the medical care team or others in the hospital. The strengths and weaknesses of each of these approaches have been described [22,23]. The skepticism of clinicians about the validity of discharge abstract data has been responsible for engendering many of the risk adjustment systems currently used in cardiac surgery. Both retrospective chart audits and primary data collection by the medical care teams are more costly than obtaining discharge abstract information, but are more reliable. Either prospective data collection or retrospective chart review, or some combination of the two, is used for most of the current cardiac surgery programs. In prospective primary data collection, the completeness of the data is improved and some of the problems with missing data are eliminated, as information cannot be determined after the procedure on retrospective chart review (eg, the failure to record in the chart the ejection fraction or number of diseased vessels noted at catheterization). Given the overall costs of coronary artery bypass graft- ing, the marginal cost of collecting risk adjustment information and outcomes is very small [17]. Who Will Collect the Data? How Much Training and Clinical Background Do They Need to Collect the Data in an Accurate, Reliable, and Timely Way? VlThat Are the Pros and Cons of Having the Patient Care Team Collect the Data? Hammermeister [1] argues that those closest to the care of the patient, the patient care team, are the most appropriate people to collect the data. The care providers are most familiar with the patient and his or her history and care. Because the use of risk-adjusted outcome information may be perceived by the care providers as either irrelevant to the care of the patient or as part of a punitive exercise in quality assurance, resistance to collecting this information may be entrenched among care providers. Some hospitals use clinical personnel who are independent of the care team to collect the data and others have assigned a particular member of the care team (eg, the perfusionist) to collect the risk information. Other participants in this conference have discussed the "gaming" of these risk systems [24]. Some argue that members of the clinical care team familiar with the risk adjustment model may report that patients have risk factors that they do not have in order to inflate the risk profile of their patients and thus improve their risk-adjusted outcomes. Anecdotal accounts of this type of behavior have been reported for some systems, particularly systems such as the one in New York State where the data are released to the public and the media. Random audits of institutions rarely confirm a systematic bias in these systems, although an occasional institution or provider has been asked to recode cases. Do Standardized Definitions of the Data to Be Collected Exist? Are Standardized Definitions Incorporated Into Data Collection Software or Is There a Standard Manual That Can Be Used Easily? How Often Is It Updated? Although prospective data collection may solve some problems relating to the variability in definitions, standardization, reliability, and validity, one problem with the application of risk adjustment in cardiac surgery has been the lack of standardization of common risk variables and outcomes. Doctor Robert Jones of Duke University is heading a task force of those developing databases in cardiac surgery to standardize definitions for coronary artery bypass grafting (personal communication, 1994). Training, standard reference manuals of terms and definitions, and readily available resources for assistance in the form of summary sheets, help screens in computer software, or networks of data collectors who can answer questions all help to standardize the collection of risk factor and outcome data. Newsletters and regular meetings or conference calls may help to communicate changes in data collection systems, and frequent updates on new variables and definitions help to improve the reliability of data collection. Particular atten-

5 Ann Thorae Surg OUTCOMES DATA DALEY 1831 tion should be paid to the training of new data collectors to continue the standardization of data collection. How Are the Data Collected and Entered Into a Database? Can Data Entry Be Easily Incorporated Into Clinical Care? Several methods of data collection are currently in use. In some systems, the clinical risk variables and outcomes are recorded on paper-and-pencil forms. Developers emphasize that keeping this form to one piece of paper facilitates data collection across many sites. These paperand-pencil forms may be forwarded to a central data center and the data on them entered into a database, or such data may be entered into a database on a small personal computer at the study site and an extract of those data transferred by disk or by electronic means to the central data center. Some sites have the data entered directly into computers on the clinical care floors or in the operating room and then transmit the information to the central data center. Use of data entry software at the local site permits the development of data entry software edits that ask the data collector whether a patient variable appears implausible at the time of data entry. These data entry edits help to improve the reliability of the data and reduce the time needed to "clean" the database before analysis. How Is Data Reliability Checked? If Data Entry Software Is Used, Are There Range Checks and Data Edits Built Into the Software? Are Some Cases Abstracted Twice by Two Different People to Check Interrater Reliability? Clinicians and administrators who find that they have higher-than-expected risk-adjusted outcomes will immediately question the accuracy of both the patient risk factors and the outcomes. Thus, the reliability of the data needs to be assessed. Data reliability and its assessment is a field of measurement of its own, and the reliability of risk adjustment systems has been reviewed extensively [25]. The most important aspect of these reliability assessments is to establish interrater reliability. Interrater reliability tests determine whether two trained data collectors find the same values of risk and outcome variables in the risk models on independent reviews of the patient and his or her care. The measures of interrater reliability vary according to the type of risk factor or outcome variable (ie, dichotomous variable or continuous variable). Although data reliability is usually excellent in systems that make use of prospective data collection, trained data collectors, and standardized definitions, some variability (usually underreporting) in outcomes such as complications has been noted. What Is the Sample Population on Which the Predictive Models Are Built? Predictive risk-adjusted models for coronary artery bypass grafting have been developed in voluntary regional or state alliances, state-mandated systems, national voluntary systems, and nationally mandated comprehensive systems (see Table 1). These systems have both common features and important differences. Given the different motivations for the development and use of the data systems, each system uses a somewhat different sample of patients. Most of the systems strive to collect data on every coronary artery bypass graft procedure performed within each hospital. Some of the systems restrict themselves to isolated coronary artery bypass graft procedures; others include patients who undergo coronary artery bypass grafting and valve replacement, although the data from these patients may be analyzed separately. From What Period Were the Data Used to Build the Predictive Models Obtained? Given that it takes several years to build these databases because of the large number of cases needed to create stable models, the data in the database should be recent enough to represent both the current population of patients undergoing cardiac operations and the current practice of cardiac surgery. The introduction of new therapies for the medical treatment of acute and chronic ischemic heart disease and the rise of the use of interventional cardiac procedures such as percutaneous transluminal angioplasty have changed the practice of cardiac surgery so that it now focuses on populations of patients with more involved coronary arteries, higher risk profiles, and histories of prior coronary artery bypass graft procedures. The databases should contain sufficient numbers of such patients from recent years to reflect these increasing risk profiles. The surgical treatment of these patients has also changed, and, despite the higher risk profiles, the mean mortality rates have declined. Use of the Collaborative Study in Cardiac Surgery risk adjustment models would, for example, be inappropriate, as so much has changed in clinical practice since the publication of the findings from those studies [3,4]. How Large Is the Database Used to Develop Predictive Models? The appropriate size of the database used to develop risk adjustment models depends on the incidence of the outcome of interest, the number of patient risk factors to be included in the model, and the statistical modeling approach. In studies of operative mortality after cardiac operations, stepwise logistic analysis as well as other methods should be performed, provided the number of deaths is at least ten times greater than the number of predictor variables [26]. As a rule of thumb, more than 2,000 cases are necessary to develop stable logistic models for comparing hospitals or cardiac surgeons for a procedure such as coronary artery bypass grafting with a baseline mortality rate of 3% to 5% and a reasonable number (ie, seven to fifteen) of potential predictive variables. What Analytic Technique Was Used to Develop the Models? Multiple analytic issues must be considered in developing risk adjustment models: data reduction, the structure of continuous variables such as age, clinical and statisti-

6 1832 OUTCOMES DATA DALEY Ann Thorac Surg cal criteria for the selection of important risk factors for each outcome to be studied, and the choice of the most appropriate multivariate modeling techniques. A detailed discussion of these issues may be found elsewhere [27]. Multivariate techniques such as logistic that permit a risk equation to be specified that yields a predicted probability of death for each patient with the use of a hand-held calculator [28] are preferred in some programs. Bayesian approaches do not lend themselves to the production of patient-level predictions at the bedside orin the office. The caveats about the margins of error in these patient-level predictions discussed earlier should be kept in mind when using such equations to make patient-physician decisions about treatment. How Were the Models Validated? Validity in risk adjustment systems is a multidimensional concept. Some dimensions of validity have to do with the accuracy of the data and the precision of the measures, and others have to do with the justifiability of the inferences drawn from the data [29]. In assessing the validity of risk adjustment methods, the most important types of validity are face validity, content validity, criterion or construct validity, attributional validity, and predictive validity. FACE VALIDITY. Face validity is often examined to find out whether a measure appears to measure what it claims to measure. Although face validity is not a rigorous concept, its importance cannot be underestimated. Acceptance of a risk adjustment system by clinicians rests on how their internal standard of how sick a patient is compares with the probability of death predicted by the risk prediction equations. For example, from the standpoint of cardiac valve replacement procedures, most models do not include the presence of acute endocarditis because it occurs too infrequently to have an important impact on the model predictions, but most cardiac surgeons will question the validity of a model that does not include endocarditis as a risk factor. One of the challenges in developing risk adjustment systems is the complicated task of translating the risk structure implicit in a particular model into something understandable to clinicians, so that they can "test it" against their clinical experience. Another problem is that risk models include the independent effects of several different risk factors. If some clinical risk factors are highly correlated, only one may be present in the final models. Clinicians may question the absence of one of a pair of highly correlated risk factors without understanding the interrelationship among different risk factors. CONTENT VALIDITY. Content validity is the extent to which the risk factors incorporated in the risk adjustment system include the universe of risk factors that should be included. A risk adjustment system can always include more risk factors. An examination of the content validity asks whether any important risk factor is missing. Information for judging this dimension is typically drawn from the clinical literature and experts. Table 2 includes all of the clinical risk factors, as revealed by multivariate analyses, that are included in the risk prediction models being used by the five cardiac surgery risk-adjusted outcomes assessment programs. Each program usually collects more clinical data than are included in the models, but these are the patient characteristics that have been shown to be statistically significant by the multivariate predictive models in each program. Considerable overlap exists among the five programs in terms of the most significant patient characteristics predictive of mortality (eg, age, sex, recent myocardial infarction, prior open heart operation, extent of coronary artery disease, and left ventricular ejection fraction). CONSTRUCT VALIDITY. Construct or correlation validation assesses how well the risk adjustment method correlates with other measures of the same concept. To determine this validity, hypotheses are generated about how the levels of risk measured will correlate with other measures or variables. Confirmation of these hypotheses provides evidence of the measure's validity. One strategy for establishing construct validity is to examine the correlations between a risk rating and the determinations of risk by a panel of expert physicians. ATTRIBUTIONAL VALIDITY. Arguably, the most important type of validity is attributional validity. In the context of risk adjustment for outcome and quality-of-care studies, the goal of risk adjustment is to account for patient characteristics so as to be able to attribute the remaining variation in outcomes, if it exists, to the processes and structures of care. Attributional validity is the extent to which the risk adjustment is sufficient for one to conclude that variations were not related to intrinsic patient characteristics. Surprisingly little attention has been paid to this issue. Most of the research in the attributional validity of risk adjustment models has been conducted in the area of cardiac surgery using some of the systems described here. Many administrators and policymakers are eager to attribute persistent differences in outcome after risk adjustment to the process and structure of care in institutions and cardiac surgery programs. They should be reminded that risk-adjusted outcomes are an indirect means of assessing the process of care. A direct examination of the process and structure of care is the most appropriate response to the finding that a particular service or institution has a higher- (or lower-) than-expected riskadjusted mortality rate in its cardiac surgery population. PREDICTIVE VALIDITY. Predictive validity is defined as how well the risk prediction equation in question predicts the outcome of interest. When considering predictive validity, it is important to distinguish between the data set used to develop the model and the data set used for model validation. Risk adjustment models typically perform better when predicting outcomes in the data set used to develop the approach than they do in the context of other data sets. The most important test of a model is how well it works on data sets that were not used in its development. Independent validation of the prediction

7 Ann Thorac Surg OUTCOMES DATA DALEY 1833 Table 2. Risk Factors Predictive of Operative Death in Multivariate Models of Coronary Artery Bypass Grafting From Five Programs Using Risk-Adjusted Outcomes" The Society of Northern New Thoracic Veterans Health Risk Factor New York State England Parsonnet Surgeons Administration (CICSS) Patient demographics Age ; in 5-year intervals ; in 10-year intervals Sex No Cardiac history and symptoms Recent myocardial In the prior 7 <6 hours, <24 In <7 days infarction days hours, <3 weeks, >3 weeks Prior heart operation 1 prior. 1 or more prior 1 prior 1 JS:ior operation: 1 or more prior operations operation: operations operation: 1 pn!>r ~2 prior ~2 prior operation operations operations Angina Unstable Unstable Stable/unstable New York Heart II versus IV Association functional class Valvular heart disease Aortic; mitral Current tobacco use >100 pack-years smoking Diabetes Hypertension Family history of coronary artery disease Comorbid illnesses Morbid obesity Peripheral vascular disease Cerebrovascular disease Chronic obstructive pulmonary disease Dialysis dependent Catastrophic states" Comorbidity score Charlson index [31] Extent of coronary artery disease Coronary artery disease LMCA >90% LMCA >90% LMCA disease 1 vesse~ 2 vesse~ 3 vessel; LMCA Physical examination, laboratory, and left ventricular function Body surface area Cardiomegaly Congestive heart failure Ventricular aneurysm Cardiogenic shock Ventricular function EF ~ 0.40; ; <0.20 EF ~0.50; ; <0.30 EF in intervals of 0.05 Prior treatment/emergency status Percutaneous transluminal coronary angioplasty emergency Intravenous inotropic agent support Intravenous nitroglycerin Surgical priority Disaster Elective, Elective, urgent, urgent, emergent emergent Intraaortic balloon pump preoperatively Concomitant valve procedure Other concomitant operation (not cardiac) a All programs collect more patient risk factors; the risk factors in this table are those predictive of operative death in models developed recently in each program. Data are obtained from the references noted in Table 1. b Catastrophic states include "acute structural instability, acute renal failure, cardiogenic shock, etc." CICSS = Continuous Improvement in Cardiac Surgery Study; EF = ejection fraction; LMCA = left main coronary artery.

8 1834 OUTCOMES DATA DALEY Ann Thorae Surg equations on an independent data set or on part of the developmental data set that has been put aside and not used to develop the model is critical to establishing the validity of the risk adjustment method. The area under the receiver operating characteristics curve or C index is commonly used to evaluate the ability of a prediction model to discriminate between the occurrence and nonoccurrence of a dichotomous outcome, such as survival versus death. A second type of predictive validity is known as "model calibration." This is commonly assessed by dividing the patient population into equal groups (eg, deciles) according to the predicted outcome and comparing the mean observed outcome rate with the mean predicted outcome rate. The Hosmer-Lerneshow test is often used to assess the strength of this relationship. Detailed discussions of these tests and techniques and their appropriate interpretation are available elsewhere [29]. If Patient Data Are Missing, How Are Missing Data Treated in the Development of the Models? In any reasonably large data set, the values of some variables will be missing. This problem is encountered most frequently with data collected by retrospective medical record reviews, but also occurs with information that is collected prospectively. Given that most clinical risk factors specified by the risk adjustment programs for coronary artery bypass graft procedures are measured routinely, how should missing values be interpreted? Some information is less consistently and reliably recorded in medical records, but has important predictive value as a risk factor. Indicators of acute clinical stability may be manipulated by medical interventions-indeed, the immediate goal of therapy is often to resolve acute physiologic abnormalities, even if the underlying disease remains. In addition to the implications of missing values in terms of clinical and data collection, how missing values are handled may dictate how many cases will be available for analysis in some modeling techniques. For example, many statistical routines drop cases with any missing value among any of the variables in the model. Even if each variable is missing for only a few cases, if many variables are considered, many cases will have at least one missing value. Data sets with many missing values may then lose a significant proportion of cases, and this may bias the models toward that subset of patients with complete data who are unlikely to be represent a random sample. Given the possible confounding of patient risk with data quality, availability, and practice patterns, how missing data is handled is an important matter. Generally, the more complete the data set, the more valid the prediction model. Therefore, it is axiomatic that the first recording of the data should be as complete as possible. However, given that the problem of missing data will occur, there are two general approaches to dealing with this situation. The first is to impute the missing data, which means inserting a value for the missing data item. Imputation techniques include simply using the mean of the known values for that variable and estimating the missing value on the basis of the other characteristics of the patient using multivariate. The second general approach to dealing with missing data is to use a statistical model that does not drop the case if the value of one of the variables is missing. These techniques include classification and trees and the Bayesian model [12]. How Often Are New Models Generated? Given the changes in the risk profile of patients undergoing cardiac operations and the changes in the process of care, risk models need to be refined periodically. The optimal timing for recalibrating the models is not known. Significant changes in patient risk profiles or practice innovations (eg, the advent of percutaneous transluminal coronary angioplasty) should stimulate reexamination and potential revision of the models. Comments Table 1 compares five risk adjustment programs using these criteria and reveals more similarities than differences among the programs. Despite some differences in the data collection methods and modeling techniques, each program has addressed most of the issues outlined here. Review of the patient-level variables also reveals a striking amount of overlap in the significant patient characteristics used by each program to predict mortality after coronary artery bypass grafting (see Table 2). Of note, each of the systems uses some predictor variables that are not solely patient characteristics. The use of variables such as surgical priority as well as information on medical (intravenously administered nitroglycerin and inotropic agents) and interventional procedures (eg, emergency percutaneous transluminal coronary angioplasty or insertion of an intraaortic balloon pump) incorporates both patient characteristics and process of care. Decisions by the medical or surgical teams, or both, before the patient is taken to operation influence the patient's outcome. Thus, from the perspective of patient risk factors, with the inclusion of these variables, these risk adjustment models are partially confounded by the process of care. If the purpose of the intended analysis is to explore the effectiveness or quality of care from the time the patient enters the operating room, this is not problematic. If the purpose is to assess the care during the patient's entire hospitalization, the process of care before operation has been confounded with patient risk. Summary Risk adjustment models for hospitalized patients with a particular condition or who undergo a particular procedure is most advanced for cardiac surgery, and especially for coronary artery bypass grafting. This is partially because of the prevalence of coronary artery disease and the need for revascularization among populations of patients with advanced disease. The procedure is performed frequently and involves considerable mortality, morbidity, and cost. The

9 Ann Thorae Surg OUTCOMES DATA DALEY 1835 release of mortality data by the Health Care Financing Administration based on administrative data with insufficient adjustments for patient risk factors has stimulated the development of several risk adjustment programs that use reliable, valid patient data collected during the surgical episode. Five of these systems are evaluated according to several criteria useful in the assessment of risk adjustment methods for outcome and quality-of-care investigations. These systems have more similarities than differences and identify many of the same patient characteristics predictive of a higher likelihood of mortality in the period immediately after operation. Risk adjustment for patient characteristics does not eliminate all of the variation in mortality rates [7-9, 16, 17]. Whether persistent differences in mortality after risk adjustment across institutions or individual surgeons are due to the process and structure of care needs further study and investigation. Similar methods should be applied to other outcomes of importance to the patients, their families, and their physicians, such as the surgically related morbidity, the patient's functional status and quality of life, the costs, and patient-reported perceptions of the nontechnical aspects of their care. Doctor Daley is a Senior Research Associate in the Department of Veterans Affairs Health Services Research and Development Career Development Award Program. I express my appreciation to the developers of these cardiac surgery risk adjustment programs for their review of the program summaries in Tables 1 and 2. References 1. Hammermeister KE. Participatory continuous improvement. Ann Thorac Surg 1994;58: lezzoni LI. Using risk-adjusted outcomes to assess clinical practice: an overview of issues pertaining to risk adjustment. Ann Thorac Surg 1994;58: Kennedy JW, Kaiser GC, Fisher LD, et al. Multivariate discriminant analysis of the clinical and angiographic predictors of operative mortality from the Collaborative Study in Cardiac Surgery (CASS). J Thorac Cardiovasc Surg 1980; 80: Kennedy JW, Kaiser GC, Fisher LD, et al. Clinical and angiographic predictors of operative mortality from the Collaborative Study in Coronary Artery Surgery (CASS). Circulation 1981;63: Hannan EL, Kilburn H, O'Donnell JF, Lukachik G, Shields EP. Adult open heart surgery in New York State. An analysis of risk factors and hospital mortality rates. JAMA 1990; 264: Hannan EL, Kilburn H Jr, Racz M, Shields E, Chassin MR. Improving the outcomes of coronary artery bypass surgery in New York State. JAMA 1994;271: O'Connor GT, Plume SK, Olmstead EM, et al. A regional prospective study of in-hospital mortality associated with coronary artery bypass grafting. JAMA 1991;266: O'Connor GT, Plume SK, Olmstead EM, et al. Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery. Northern New England Cardiovascular Disease Study Group. Circulation 1992;85: Kaspar JF, Plume SK, O'Connor GT. A methodology for QI in the coronary artery bypass grafting procedure involving comparative process analysis. QRB Qual Rev Bull 1992;18: Disch DL, O'Connor GT, Birkmeyer JD, et al. Changes in patients undergoing coronary artery bypass grafting: Ann Thorac Surg 1994;57: Parsonnet V, Dean D, Bernstein AD. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 1989;79(Suppl 1): Edwards FH, Albus RA, Zajtchuk R, et al. Use of a Bayesian statistical model for risk assessment in coronary artery surgery. Ann Thorac Surg 1988;45: Edwards FH, Albus RA, Zajtchuk R, Graeber GM, Barry M. A quality assurance model of operative mortality in coronary artery disease. Ann Thorac Surg 1989;47: Edwards FH, Clark RE, Schwartz M. Coronary artery bypass grafting: The Society of Thoracic Surgeons National Database experience. Ann Thorac Surg 1994;57: Clark RE. The Society of Thoracic Surgeons National Database status report. Ann Thorac Surg 1994;57: Grover FL, Hammermeister KE, Burchfield C, cardiac surgeons of the Department of Veterans Affairs. Initial report of the Veterans Administration Preoperative Risk Assessment Study for cardiac surgery. Ann Thorac Surg 1990;50: Hammermeister KE, Johnson R, Marshall G, Grover FL. Continuous assessment and improvement in quality of care. A model from the Department of Veterans Affairs. Ann Surg 1994;219: Blumberg MS. Risk adjusting health care outcomes: a methodologic review. Med Care Rev 1986;43: Steen PM. Approaches to predictive modeling. Ann Thorac Surg 1994;58: Diamond GA. Future imperfect: the limitation of clinical prediction models and the limits of clinical prediction. J Am ColI Cardiol 1989;14:12A-22A. 21. Diamond GA. What price perfection? Calibration and discrimination of clinical prediction models. J Clin Epidemiol 1992;45: Iezzoni LI. Data sources and implications: administrative data bases. In: Iezzoni LI, ed. Risk adjustment for measuring health care outcomes. Ann Arbor, MI: Health Administration Press, 1994: Iezzoni LI. Data sources and implications: medical records. In: Iezzoni LI, ed. Risk adjustment for measuring health care outcomes. Ann Arbor, MI: Health Administration Press, 1994: Edwards FH, Clark RE, Schwartz M. Practical considerations in the management of large multiinstitutional databases. Ann Thorac Surg 1994;58: Hughes JS, Ash AS. Reliability of risk-adjustment methods. In: Iezzoni LI, ed. Risk adjustment for measuring health care outcomes. Ann Arbor, MI: Health Administration Press, 1994: Marshall G, Grover FL, Henderson WG, Hammermeister KE. Assessment of predictive models for binary outcomes: an empirical approach using operative death from cardiac surgery. Stat Med (in press). 27. Daley J, Schwartz M. Developing risk-adjustment methods. In: Iezzoni LI, ed. Risk adjustment for measuring health care outcomes. Ann Arbor, MI: Health Administration Press, 1994: Nugent WC, Schults We. Playing by the numbers: how collecting outcomes data changed my life. Ann Thorac Surg 1994;58: Donabedian A. The definition of quality and approaches to its assessment. Explorations in quality assessment and monitoring. Vol I. Ann Arbor, MI: Health Administration Press, Ash AS, Swartz M. Evaluating the performance of risk adjustment methods: dichotomous measures. In: Iezzoni LI, ed. Risk adjustment for measuring health care outcomes. Ann Arbor, MI: Health Administration Press, 1994: Charlson ME, Pompei P, Ales KI, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis 1987:40: