Review of Methods for Measuring and Comparing Center Performance After Organ Transplantation

Transcription

1 LIVER TRANSPLANTATION 16: , 2010 REVIEW ARTICLE Review of Methods for Measuring and Comparing Center Performance After Organ Transplantation James Neuberger, Sue Madden, and David Collett Directorate for Organ Donation and Transplantation, National Health Service Blood and Transplant, Bristol, United Kingdom The assessment of outcomes after transplantation is important for several reasons: it provides patients with data so that they can make informed decisions about the benefits of transplantation and the success of the transplant unit; it informs commissioners that resources are allocated properly; and it provides clinicians reassurance that results are acceptable or, if they are not, provides early warning so that problems can be identified, corrections can be instituted early, and all interested parties can be reassured that scarce resources are used fairly. The need for greater transparency in reporting outcomes after liver transplantation and for comparisons both between and within centers has led to a number of approaches being adopted for monitoring center performance. We review some of the commonly used methods, highlight their strengths and weaknesses, and concentrate on methods that incorporate risk adjustment. Measuring and comparing outcomes after transplantation is complex, and there is no single approach that gives a complete picture. All those using analyses of outcomes must understand the merits and limitations of individual methods. When used properly, such methods are invaluable in ensuring that a scarce resource is used effectively, any adverse trend in outcomes is identified promptly and remedied, and best performers are identified; they thus allow the sharing of best practices. However, when they are used inappropriately, such measurements may lead to inappropriate conclusions, encourage risk-averse behavior, and discourage innovation. Liver Transpl 16: , VC 2010 AASLD. Received October 18, 2009; accepted June 22, In the last few years, we have seen an increasing demand for transparency and accountability in all aspects of medicine; patients, funders, health care administrators, and other interested parties all have a legitimate interest in ensuring that limited resources are used as effectively as possible, in informing patient choice, and in ensuring that outcomes of medical interventions match a similar standard nationally and indeed internationally. With respect to liver transplantation, for which grafts from deceased donors are a national, shared life-saving resource that cannot meet the overall demand, there is an even greater need to demonstrate that outcomes do not fall below an acceptable standard. The need for greater transparency has led to a requirement for clear yet comprehensive methods for measuring outcomes that take into account the complexity of the clinical situation. There are a variety of ways in which outcomes after liver transplantation can be compared both between centers and between individuals. This is an important area for many reasons: patients and their families need to know the outcomes after any procedure so that they can make suitably informed decisions about undergoing transplantation. Furthermore, comparisons of center performance may affect the decision to refer patients to one center rather than another. Commissioners and patients, in particular, also have a vested interest in knowing how outcomes may vary between centers so that resources (financial resources and donated organs) are used effectively; transplant organizations that are involved in allocation need to be reassured that there is no inequity. Clinicians also need evidence that their outcomes are acceptable and, should Abbreviations: CI, confidence interval; CUSUM, cumulative sum; ECD, extended criteria donor; O-E, observed minus expected. Address reprint requests to James Neuberger, M.D., National Health Service Blood and Transplant, Fox Den Road, Stoke Gifford, Bristol BS34 8RR, United Kingdom. Telephone: ; FAX: ; james.neuberger@nhsbt.nhs.uk DOI /lt View this article online at wileyonlinelibrary.com. LIVER TRANSPLANTATION.DOI /lt. Published on behalf of the American Association for the Study of Liver Diseases VC 2010 American Association for the Study of Liver Diseases.

2 1120 NEUBERGER ET AL. LIVER TRANSPLANTATION, October 2010 outcomes fall below acceptable levels, need early warning so that any problems can be identified quickly and corrected. In the early years of liver transplantation, centers published their 1- or 5-year patient and graft survival data usually on an ad hoc basis and occasionally after a good run. Such relatively crude analyses have rightly largely fallen into disrepute for many reasons but primarily because of the lack of rigor and the failure to account for risk factors. Increasingly, health care administrations are now assuming responsibility for undertaking such analyses and making the findings publicly available. In this review, we discuss the various approaches for comparing outcomes after liver transplantation and their strengths and weaknesses (Table 1). These approaches should be used in conjunction with prospective on-site monitoring of deaths and other adverse outcomes and not as replacements. 1 WHAT DATA ARE NEEDED? Relevant outcome variables will usually include the survival time from diagnosis, referral, listing, transplantation, or retransplantation to death or graft failure, with censored times used when the endpoint has not been observed. Interest may center on survival rates at different time points on the transplant journey and may be center-, region-, or country-specific, disease-specific, or age-specific. Different factors (eg, donor, recipient, and surgical factors) may have different impacts on survival at different times. For example, the impact of recurrent hepatitis C infection is seen mainly several years after transplantation, whereas the impact of anesthesia is likely to affect short-term outcomes rather than long-term ones. RISK ADJUSTMENT Because there will be variations in the health of recipients and in the quality of the grafts (ie, the likelihood that grafts will function), outcome measures should be adjusted in light of the risk: variations in outcomes will reflect, at least in part, variations in the case mix rather than variations within the center. Risk adjustment is carried out with statistical models based on either a Cox proportional hazards model or a logistic regression model as appropriate. A Cox model is appropriate for analyzing survival times, whereas a logistic model is appropriate when interest centers on whether an event (eg, death) has occurred in a given time period. Hierarchical models that include variation at more than 1 level (eg, center and patient levels) may also be used, although our experience is that both models lead to similar conclusions. 2 Models for risk adjustment are based on a judicious combination of statistical results and clinical relevance (eg, a factor such as recipient age may be included in a model if it is considered to be of clinical relevance even if the effect does not have sufficient statistical significance to meet the accepted inclusion criteria). TABLE 1. Advantages and Potential Risks of Comparisons of Outcomes Advantages Provide greater transparency Allow informed patient choice Enable early recognition and correction of possible structural or temporary problems Support excellence Risks May be misleading to patients and commissioners May lead to risk-averse behavior with consequently worse outcomes for patients May discourage innovation May lead to reduced training opportunities May lead to inappropriate withdrawal of support The identification of risk factors and the relative importance of these factors in determining outcomes are based on analyses of previous cohorts of transplant patients and usually include donor, recipient, and transplant factors. Donor factors include the age, sex, primary disease and associated diseases, serum creatinine, sodium, and albumin levels, and ventilation status; graft-specific factors include the donor cause of death, type of graft, appearance of the graft, and degree of steatosis; transplant factors include the cold ischemia time; and recipient factors include the age and gender, primary, secondary, and associated diseases, and factors that might be relevant to each organ. Of course, such analyses can be undertaken only with the data collected; it remains a concern that not all relevant data will have been identified when the database is established, and some data will not be collected or will not be readily quantifiable. Some important risk factors may not be included in the database because they are difficult to identify, quantify, or validate (eg, organ perfusion). Furthermore, variations between laboratories in the measurement of analytes and the lack of standardized definitions will be further limitations to the robustness of risk adjustment. The failure to include all relevant data may produce misleading conclusions. 3 For example, a steatotic graft, especially in association with a prolonged cold ischemia time, is associated with a greater risk of graft failure. 4 In practice, other than histology, there is no ready and universal approach to defining the degree of steatosis, but the failure to consider steatosis (whether by direct measures such as histological assessment or by indirect surrogate methods such as the body mass index or waist circumference) may lead to misleading conclusions when there is great variation in the proportion of steatotic grafts used. Risk may also change over time, and so re-evaluation is needed. 5 Outcomes unadjusted for risk factors also give valuable information when center performance is being compared either with past performance or with the performance of other centers. For example, if a center accepts extended criteria donor (ECD) grafts that are declined by

3 LIVER TRANSPLANTATION, Vol. 16, No. 10, 2010 NEUBERGER ET AL Figure 1. Unadjusted and risk-adjusted O-E plots of the 90-day patient mortality rates for adult elective liver transplants performed between January 1, 2004 and September 30, 2009 at the 7 liver transplant centers in the United Kingdom. other centers, outcomes may be poor when they are unadjusted, but when they are adjusted for risk factors, patient survival rates may be similar to those of other centers. However, the patient who is given that ECD graft may have had a higher survival probability if the ECD liver had been declined and the patient had waited for another graft. Thus, the priority for the patient is the option providing the greatest probability of survival rather than the greatest risk-adjusted survival. That said, our own data from the United Kingdom suggest that the differences between observed and risk-adjusted outcomes is relatively modest (Fig. 1), and similar conclusions may be drawn in other clinical situations. 6,7 This illustrates the distinction between absolute outcomes and relative outcomes. Thus, in contrast to the recipient, who is usually concerned primarily with the absolute outcome, those interested in comparing outcomes both within and between centers and between different treatment options will be also interested in the relative risks of mortality or morbidity, These are complementary rather than alternative measures. It is inevitable that not all relevant data will be available for all patients, and this too leads to potential problems. For monitoring purposes, observations with missing values for any of the risk factors included in the model may be assigned the value of that factor for which the risk of mortality is lowest. Not only is this method straightforward to implement, but it also has the advantage over other methods of the imputation that the expected number of deaths is underestimated; the observed mortality rates will then compare less favorably when centers fail to return data, and this will stimulate the return of a complete data set. There are many other approaches to dealing with missing data (including the use of median or most commonly occurring values and methods based on statistical models for variables with missing values), but these too may lead to conclusions that patients or others may find misleading. If there is a systematic failure to measure or return data from one center, then the missing variable itself may have a prognostic impact. METHODS FOR COMPARING PERFORMANCE When methods are being selected, it is essential to define what question or questions are to be addressed

4 1122 NEUBERGER ET AL. LIVER TRANSPLANTATION, October 2010 because methods for monitoring outcome data may address different aspects of surgeon, center, or provider performance. Each method has its own advantages and limitations. Furthermore, there may be variations in how some of the components of the models are quantified. For example, there are variations in assay methods used by different laboratories that can affect the value reported, 8 and for some clinical components (eg, cardiac risk), there may be no standardized approach or definition. Methods for comparing outcomes are applicable to survival or mortality outcome data or indeed any outcome measure at different provider levels as long as the numbers are sufficient. However, when the numbers are too small, interval estimates for summary outcomes will be so wide that conventional statistical methods will have little practical value. The performance of a center can be compared with the performance of other comparable centers or with its own past performance, and significance levels and confidence intervals (CIs) can be generated. 9 The distinction between statistical and clinical significance is important; large numbers may show a difference that is statistically significant but clinically not important. For example, a large data set may indicate that donor gender has a significant impact on transplant outcome, but this is unlikely to be clinically relevant. 10 Most health care clinicians and patients are familiar with this type of analysis. Graphical representation is clear and simple, and the patient and clinician can easily understand, for example, that a risk-adjusted 60-day graft survival rate of 95% in center A is better than a rate of 90% in center B (even if this difference is not statistically significant, perhaps because of a small number of transplants in one center), and a fall in survival from 95% to 90% may indicate a problem within the center. Such figures will usually be obtained from a Kaplan-Meier estimate of the survivor function, which accounts for patients lost to follow-up (rare in short-term survival analysis) or incomplete data return. However, there are other methodologies, described later, that can be used for measuring outcomes and that allow the detection of variations within and between centers at early stages; this allows a center to examine the reasons for the variations and, when it is appropriate, to take corrective action. As can be seen in Fig. 1, a center that is underperforming may have crude outcomes on the survival curves that do not appear to be greatly different from those of other centers. The application of a combination of methods ensures that different aspects of performance are monitored. Within-center monitoring can detect a deterioration in performance with respect to that center s own past performance. If, however, a center s performance remains unchanged over time but falls behind the advancing field, this divergent performance will be highlighted by between-center monitoring. WITHIN-CENTER MONITORING TECHNIQUES Cumulative Sum (CUSUM) Charts CUSUM methodology is used to monitor how a center is performing with respect to its past performance through the comparison of current mortality rates with center-specific expected mortality rates. Hence, even the very best provider is expected to maintain the same high standards, and the poorest performers do not signal excessively. It is important that all users understand the process because errors are not uncommon. 11 CUSUM Chart Interpretation There are 2 complementary charts used in CUSUM monitoring: observed minus expected (O-E) CUSUM and tabular CUSUM. Both charts can be based on risk-adjusted or unadjusted survival rates. The O-E chart is formed from the CUSUM of the difference between each observed outcome and the expected mortality rate. In unadjusted analysis, the expected mortality rate is the center-specific mortality rate, and it is uniform for all patients at a given center. In risk-adjusted analysis, the expected mortality rate is dependent on the individual s specific risk score, which is determined from a survival model fitted to national data. In both unadjusted and riskadjusted analysis, the expected rate is determined from the baseline period when performance is known to be satisfactory (however that is defined). Thus, in our practice, expected rates for liver transplantation are determined from the 4 years prior to the start of the monitoring period. In liver transplantation, expected mortality rates are fairly low, and the observed outcome is equal to 0 for a success and 1 for a failure. Hence, for each successful outcome (patient or graft survival), the O-E chart goes down a small amount (the expected mortality rate), whereas for each negative outcome (patient death or graft failure), the chart goes up a larger amount (1 minus the expected mortality rate). The step sizes reflect that there is a small probability of death or graft failure (the expected mortality rate). If current center performance is as expected, the overall O-E CUSUM will be approximately 0 with no trend. If current performance is improving with respect to the expected rate, then there will be a downward trend, whereas an upward trend indicates that observed center performance has deteriorated with respect to the center s past performance. The tabular CUSUM plots the cumulative values of a statistic (a log-likelihood ratio) that summarizes the extent to which observed outcomes are consistent with the expected rate. The construction of the tabular CUSUM has been described in detail elsewhere. 12,13 The larger the value is of this statistic, the stronger the evidence is that there has been a change in the underlying rate. The tabular CUSUM will signal when the value of the plotted statistic crosses a

5 LIVER TRANSPLANTATION, Vol. 16, No. 10, 2010 NEUBERGER ET AL predefined threshold, with a simulation method used to determine a chart threshold with the required properties. Similarly to the O-E CUSUM, the tabular CUSUM decreases by a small value for each observed success and increases by a large value for every failure. To prevent a center from accruing credit after a long run of successes, the tabular CUSUM is constrained to be nonnegative. After a signal, if the cause of the adverse outcomes is known and the problem is rectified, then the tabular CUSUM is reset to 0, and monitoring recommences. However, if the cause of the signal remains unexplained after a clinical review, we can implement a more responsive system by resetting the tabular CUSUM to a point half-way between 0 and the threshold. This is known as a head-start CUSUM, 14 and the CUSUM signals more quickly if outcomes remain inconsistent with the expected rate. Although the O-E CUSUM provides a useful tool for observing center performance over time, the tabular CUSUM is designed to be able to detect significant changes in a center s performance. The values plotted in both CUSUM charts can be calculated as soon as outcome data become available for each transplant. Consequently, performance can be monitored in real time, and this allows the prompt detection of an increase in adverse outcomes. The CUSUM charts for a particular liver center in the United Kingdom are displayed in Fig. 2. The O-E chart (Fig. 2A) shows a sharp upward trend indicating an increase in the number of adverse outcomes. After the fifth death in 20 transplants, the tabular CUSUM crosses the threshold causing a signal to occur, and this suggests that the performance was no longer in line with the expected rate. The signal is superimposed on the O-E CUSUM chart (marked by a dot and the transplant date). After internal investigations, some amendments were made to the transplant program (in this case, a change in the protocol for the use of split liver grafts). The outcomes were then closely monitored with the head-start CUSUM. No further signals were identified, and the tabular CUSUM returned to 0; this suggested that the changes made were effective and that the observed mortality rate had reverted to the expected rate. CUSUM analyses are not designed to compare performance across centers; the methodology described in the following section addresses between-center comparisons. BETWEEN-CENTER MONITORING TECHNIQUES Regression Modeling Cox regression models can be used to identify the extent of differences in survival times between centers that are risk-adjusted for all relevant factors. For short-term outcomes or other types of outcome variables (eg, the survival rate at a particular time point or Figure 2. (A) O-E and (B) tabular CUSUM plots for 1 liver transplant center. the serum bilirubin level), logistic regression or general linear models can also be used to compare centers. These methods assess overall performance in a specific time frame and can therefore detect overall significant differences between centers. With established risk-adjusted mortality models, a center effect is included as an additional variable, and the model is compared to the model without a center effect. When the number of liver centers is relatively small, a model with fixed center effects can be used in which the effect of each center with respect to a chosen selected center can be estimated. However, if the number of centers is large, center effects are better modeled with a random effects model so that center effects are assumed to be drawn from an underlying normal distribution. If the model that includes the center effect shows a significant improvement with respect to the model without such an adjustment, then there is a significant difference in center outcomes with adjustments for all risk factors. Significant between-center variation may not necessarily be due to the divergence of one center but may instead stem from an accumulation of relatively small differences between centers, each of which has an acceptable level of performance. We illustrate the application of logistic regression analysis by using our own data to compare the riskadjusted 90-day mortality rates between centers. The

6 1124 NEUBERGER ET AL. LIVER TRANSPLANTATION, October 2010 TABLE 2. Logistic Regression Model of the 90-Day Patient Mortality Rates for All First Adult Elective Liver-Only Transplants: Comparison of Center Performance from January 1, 2004 to December 31, 2008 with Adjustments for Other Risk Factors Factor Level Analyzed Patients (n) Odds Ratio 95% CI P Value Transplant center Center E (baseline) Center A Center B Center C Center D Center F <0.01 Center G Overall results are presented in Table 2. Center E has the lowest mortality rate and has therefore been selected as the baseline center. There is a significant difference in overall performance between the centers (P ¼ 0.02) after we take account of the relevant risk factors. The risk of death in center F is 3 times that in center E, and the risk of death is more than doubled in centers B and G versus center E. Funnel Plots Funnel plots 15 provide a useful visual tool for comparing unadjusted and risk-adjusted overall mortality rates. Funnel plots can present either unadjusted or risk-adjusted outcome rates with respect to the national average. Performance is monitored over a specific time frame (eg, in our data set). Overall mortality rates are plotted against the center size and alongside the national rate and its 95% and 99.8% CIs. Any point falling outside the 95% CIs may suggest that the center s outcome is not consistent with the national rate, whereas centers whose rates lie above the upper 99.8% CI are considered to have significantly worse mortality than the national rate. The funnel shape of the CIs reflects the number of transplants performed at each center; estimated mortality rates for the larger centers will be more precise, and the CIs will be narrower. Figure 3 displays a funnel plot for 90-day mortality data after first adult elective liver transplantation. Unadjusted 90-day mortality rates are presented together with the risk-adjusted rate for each center. All centers fall below the upper 99.8% limit for the national rate, and this signifies that no center is experiencing a mortality rate significantly higher than the national rate. However, there is evidence suggesting that center E has a significantly lower mortality rate versus the national rate. It is worth noting that divergent centers may unduly influence the national rate estimate and therefore mask their own divergence. Cross-Validation Method An alternative method for comparing performance across centers is the cross-validation method, 16 which provides a measure of how divergent a center is with respect to the remaining centers. This technique has a number of advantages over other between-center comparisons. The method uses formal hypothesis tests to assess the divergence of each center; this can be beneficial when formal action is taken as a result of divergence. Also, the issues with masking seen in the funnel plots are avoided, so center divergence can be identified more quickly. The method compares the observed number of deaths over the monitoring period for a given center with the number of expected deaths on the basis of data from remaining centers. The expected outcome for each individual at a particular center is obtained by simulation from risk-adjusted mortality models incorporating data from all other centers to estimate the total expected number of deaths. This number is then compared to the actual observed number of deaths. This simulation is repeated numerous times (eg, 1000 times) for each center, and the P value is determined by the calculation of the proportion of simulations in which the observed number of deaths exceeds that expected on the basis of all remaining Figure 3. Funnel plot of the 90-day patient mortality rates for all first adult elective deceased donor liver-only transplants in the United Kingdom between January 1, 2004 and December 31, 2008.

7 LIVER TRANSPLANTATION, Vol. 16, No. 10, 2010 NEUBERGER ET AL TABLE 3. Cross-Validation Results Comparing the 90-Day Mortality Rates After First Adult Elective Deceased Donor Liver Transplantation from January 1, 2004 to December 31, 2008 Transplant Center Patients (n) Observed Deaths (n) Expected Deaths (n) P Value Center A Center B Center C Center D Center E Center F <0.01 Center G Total centers. A center that has substantially more observed deaths than predicted will have a P value close to 0, and a center that has far fewer deaths than predicted will have a P value close to 1. Any center with a P value < 0.1 is considered significantly divergent from all other centers and merits further investigation. As the cross-validation method compares each individual center with the others, an adjustment is required to account for the multiple testing. 16 If interest is focused solely on the worst performer (ie, the center with the lowest P value), then the Bonferroni correction method is an adequate adjustment technique for the multiple testing: this method controls the expected proportion of times that a center is found to be significantly divergent when it is not (a type II error). If an overall significance level of a is desired for identifying any one of n centers, each center is compared to the rest with a significance level of a/n. Analternative method of allowing for multiple testing is based on the false discovery rate, which is the expected proportion of times that a center is incorrectly identified as divergent among all centers found to be divergent. This method allows a group of divergent centers to be identified instead of the most divergent. As an illustration, the results from the cross-validation comparison of 90-day mortality data after elective liver transplantation are presented in Table 3. A P value has been calculated for each center. Generally, the P values are high, and this indicates that the observed performance in most centers is in line with the performance of the others. However, the 90-day mortality rate at center F is greater than expected in comparison with the other centers. Adjusting the P value with the Bonferroni correction by simple multiplication of the observed P value by the number of centers being compared results in a value of 0.07, which remains significant with respect to the level indicating divergence (0.1). In practice, such findings would prompt an evaluation of possible reasons for the discordance of this center. DISCUSSION Potentially Adverse Impacts of Transparency The publication of outcomes is to be welcomed, but unless the limitations of the various approaches are fully understood, such data can have a deleterious impact on transplantation. 17 A simplistic approach to the interpretation and use of data can potentially be more harmful than nonpublication of such analyses and lead to risk-averse behavior with a lower transplant benefit and a lack of innovation. Although the identification of a center or surgeon appearing to be performing poorly may indicate a potential problem, it may merely reflect the normal pattern of variation. However, experience with the publication of outcomes of cardiac surgery suggests that these problems can be overcome, at least in part. 18 Risk-Averse Behavior May Be Encouraged According to the health care system, the surgeon has the responsibility for either accepting the offered graft for an individually named recipient or selecting the recipient from the center s own list. In either case, some degree of matching has to take place, and many of the factors are not readily captured or quantified: not only will the surgeon assess the balance of risks of using the graft for that patient versus waiting for another and possibly more suitable graft, but he will also consider logistical issues such as the cold ischemia times, the capacity in the operating room and intensive therapy unit, and the current activity of the transplant team. From an institutional point of view, a center that is or appears to be performing poorly may lose because patients may decide to go elsewhere or commissioners may decide to withdraw support or designation. 3 The increasing shortfall between the number of organs available and the number of patients who might benefit from transplantation has stimulated surgeons to look for ways of increasing the number of usable organs: these include the use of split livers, the use of livers from non heart-beating donors, and the greater use of ECDs. The use of these grafts is associated with an increased risk of graft failure. Undue emphasis on outcomes after transplantation means that transplant figures will be better if the offer is declined for an ill recipient and the death occurs on the waiting list rather than after transplantation. This can be overcome, at least in part, if we also compare outcomes from the date of listing rather than transplantation. This approach will be valid when there is a common point of listing. Thus, for some center X,

8 1126 NEUBERGER ET AL. LIVER TRANSPLANTATION, October 2010 Figure 4. Kaplan-Meier survival curves showing (A) patient survival on the liver transplant list and (B) 3-year posttransplant survival after liver transplantation. the outcomes of patients after transplantation are better than average (Fig. 4B), but when survival on the transplant list is assessed, patients in that center have a much worse survival rate in comparison with the national average. An explanation for such a scenario could be that center X had a higher rate of refusing offered organs that were subsequently used in other centers. Quantifiable measures such as the Model for End- Stage Liver Disease score generally do not take into account those patients for whom factors other than parenchymal disease affect prognosis (eg, those with primary liver cancer or hepatopulmonary syndrome) or who undergo transplantation for reasons other than end-stage disease (eg, polycystic disease or intractable encephalopathy). For those with liver cancer, whose prognosis may be determined by the cancer rather than the degree of parenchymal damage, it is possible to give additional points to give an equivalent prognostic score. As the other indications are relatively few, their inclusion or exclusion from the analysis will make little difference. It is naive to ignore the potential impact of the publication of outcomes on risk taking from a financial point of view. Liver transplantation requires and therefore attracts considerable resources, and whatever health care policies are in place, hospitals have to balance their books. Transplant teams are necessarily large and resource-intensive and cannot readily be formed or disbanded: if a center underperforms or has poor outcomes, centers may become nonviable because patients or commissioners look elsewhere for the provision of services. Innovation and Research May Be Inhibited The history of liver transplantation is remarkable for the degree of surgical innovation and the use of new techniques to allow more patients to receive grafts and to make more grafts suitable for clinical use. However, the introduction of new techniques is often associated with a worse outcome in the early phases (a learning curve). If a surgeon or center is looking to develop or introduce new techniques, then the fear of worse outcomes may inhibit such developments. Likewise, the use of transplantation for novel indications may be associated with a transient reduction in outcome. These concerns are felt by clinicians 17 and may persist despite assurances to clinicians that their concerns are taken seriously and that payers are sensible in their response to signals. 19 One solution to this dilemma is to exclude such patients from the analysis, but this should be determined before the introduction of the new procedure. Models Are of Limited Power All models of outcomes are dependent on the provided data; however, not all data are captured, and many may be subjective and not readily or meaningfully quantified. Furthermore, some significant risk factors may not have been recognized when databases were established; allowances have to be made for missing data, variations between laboratories, and different approaches to measuring laboratory and other variables. There is not always agreement on definitions of important variables; for instance, centers may not always agree on definitions of variables such as the cold ischemic time, and there are variations in the ways in which centers evaluate and define the diagnosis of comorbid diseases. Not all measures are readily and objectively quantified: experienced surgeons will assess the graft not only by measurable and quantifiable factors (eg, the age of the donor, cause of death, and degree of steatosis) but also by unquantifiable

9 LIVER TRANSPLANTATION, Vol. 16, No. 10, 2010 NEUBERGER ET AL measures (eg, the texture, appearance, and performance of the graft). Matching donor organs with the most appropriate recipient is often a challenge. Some of the factors that are associated with poor initial graft function (eg, graft perfusion) will not be apparent until surgery has started. There is also an interaction between the donor and recipient: although surgeons may match a relatively fit recipient with an ECD graft, this approach may not be associated with the best outcome for the recipient, who may have a greater survival probability with a less marginal graft. 20 Models are derived from retrospectively collected data: there are obvious concerns about the completeness and accuracy of the data. Furthermore, such models may not take into account changes in practice, such as changes in perfusion fluids; therefore, extrapolation from retrospective data may be inappropriate. Risk adjustment is based on collected data, and critical factors may not be included. Therefore, reviewing used models on a regular basis and close collaboration with the clinicians are essential. Half the Centers Will Perform Below the Median: How Much Variation Is Acceptable? Any analysis will identify centers above and below the median, and there will be some variation. Comparisons of outcomes must take into account acceptable variations that, especially in the case of low-volume centers or surgeons, may lead to alarm signals. Inevitably, some arbitrary value will need to be drawn for determining when deviations from the norm may no longer be attributed to random chance but require further investigation. This is best done by statisticians working in collaboration with the clinicians and commissioners, and the results should be reviewed and revised in the light of experience. Training May Be Harmed The training of new surgeons is clearly essential to the continuation of existing programs and the establishment of new programs. Outcomes of procedures performed by trainees may be inferior to the outcomes of procedures performed by more experienced surgeons. Especially when a center is performing below average, there will be the temptation for the procedures to be performed by the more experienced surgeon. In most (if not all) centers, the assessment, surgery, and follow-up are done by a team, and indeed, most units encourage teamwork. If teamwork is to be more than a theoretical approach, outcomes (good or other) should be ascribed to the team rather than the individual. It is always difficult to draw a balance between the overenthusiastic investigation of a center s or surgeon s performance with the associated consequences and the failure to identify, investigate, and correct poorly performing centers and surgeons. Therefore, a variety of methodologies should be used to assess performance to ensure that any decisions taken are fully informed. THE WAY FORWARD Historically, performance monitoring has considered only posttransplant outcomes. There is a movement toward analyzing outcomes from the point of listing; this will provide additional information on the transplant process as a whole. However, this is a relatively new area, and the work of defining the most clinically relevant ways of analyzing and presenting the data is still in development. Analyses will also need to take into account that, even with a common point of listing, not all patients will be listed at a similar time with respect to their risk of death and that death, whether before or after transplantation, may be due to factors unrelated to the disease or its treatment (eg, a road traffic accident). Regardless of the outcome measure, the monitoring of center performance and the publication of comparisons with other centers are real and appear to be associated with improved performance in other areas of transplantation. 6,19,20,21 A clear understanding of the strengths and limitations of the employed methodologies is needed so that a response to a signal can be constructive. There are concerns that uninformed media will have adverse effects on the patients, centers, and overall confidence in the whole transplant process. ACKNOWLEDGMENT The authors are grateful to Kerri Barber for help with Fig. 4. REFERENCES 1. Poloniecki J, Sismanidis C, Bland M, Jones P. Retrospective cohort of false alarm rates associated with a series of heart operations: the case for hospital mortality monitoring groups. BMJ 2004;328: Cohen ME, Dimick JB, Bilimoria KY, Ko CY, Richards K, Hall BL. Risk adjustment in the American College of Surgeons National Surgical Quality Improvement Program: a comparison of logistic versus hierarchical modeling. J Am Coll Surg 2009;209: Weinhandl ED, Snyder JJ, Israni AK, Kasiske BL. Effect of comorbidity adjustment on CMS criteria for kidney transplant center performance. Am J Transplant 2009;9: Feng S, Goodrich NP, Bragg-Gresham JL, Dykstra DM, Punch JD, DebRoy MA, et al. Characteristics associated with liver graft failure: the concept of donor risk index. Am J Transplant 2006;6: Hall BL, Hamilton BH, Richards K, Bilimoria KY, Cohen ME, Ko CY. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg 2009;250: Mehta RH, Liang L, Karve AM, Hernandez AF, Rumsfled JS, Fonarow GC, Peterson ED. Association of patient case-mix adjustment, hospital process performance rankings and eligibility for financial incentives. JAMA 2008;300: Novick RJ, Fox SA, Stitt LW, Forbes TL, Steiner S. Direct comparison of risk-adjusted and non-risk-adjusted

10 1128 NEUBERGER ET AL. LIVER TRANSPLANTATION, October 2010 CUSUM analyses of coronary artery bypass surgery outcomes. J Thorac Cardiovasc Surg 2006;132: Cholongitas E, Marelli L, Kerry A, Senzolo M, Goodier DW, Nair D, et al. Different methods of creatinine measurement significantly affect MELD scores. Liver Transpl 2007;13: Dickinson DM, Shearon TM, O Keefe J, Wong HH, Berg CL, Rosendale JD, et al. SRTR center-specific reporting tools: post-transplant outcomes. Am J Transplant 2009; 6(pt 2): Clinical Effectiveness Unit of the Royal College of Surgeons. UK and Ireland liver transplant audit of patients who received a first liver transplant between 1 October 2008 and 30 September Biau DJ, Resche-Rigon M, Godiris-Petit G, Nizard RS, Porcher R. Quality control of surgical and interventional procedures: a review of the CUSUM. Qual Saf Health Care 2007;16: Steiner SH, Cook RJ, Farewell VT, Treasure T. Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 2000;1: Collett D, Sibanda N, Pioli S, Bradley JA, Rudge C. The UK scheme for mandatory continuous monitoring of early transplant outcome in all kidney transplant centres. Transplantation 2009;88: Lucas JM, Crosier RB. Fast initial response for CUSUM quality control schemes: give your CUSUM a head start. Technometrics 1982;24: Spiegelhalter DJ. Funnel plots for comparing institutional performance. Stat Med 2005;25: Ohlssen DI, Sharples LD, Spiegelhalter DJ. A hierarchical modelling framework for identifying unusual performance in health care providers. J R Stat Soc A 2007; 170: Abercassis MM, Burke R, Klintmalm GB, Matas AJ, Merion RM, Millman D, et al. American Society of Transplant Surgeons transplant center outcomes requirements a threat to innovation. Am J Transplant 2009;9: Bridgewater B, Grayson AD, Jackson M, Brooks N, Grotte GJ, Keenan DJ, et al. North West Quality Improvement Programme in Cardiac Interventions. BMJ 2003;327: Hamilton TE. Accountability in health care-transplant community offers leadership. Am J Transplant 2009;9: Khuri SF, Henderson WG, Daley J, Jonasson O, Jones RS, Campbell DA, et al. Successful implementation of the Department of Veterans Affairs National Surgical Quality Improvement Program in the private sector: the patient safety in surgery study. Ann Surg 2008;248: Axelrod DA, Kalbfleisch JD, Sun RJ, Guidinger MK, Biswas P, Levine GN, et al. Innovations in the assessment of transplant center performance: implications for quality improvement. Am J Transplant 2009;9: