The Art and Science of Increasing Authorization to Donation

The Art and Science of Increasing Authorization to Donation OPO Metrics: The Good, The Bad, and The Maybe Charlotte Arrington, MPH Arbor Research Collaborative for Health Alan Leichtman, MD University of Michigan Kidney Epidemiology and Cost Center

Disclaimer We re describing today s practices. Expect change as metrics and flagging thresholds are evaluated and revised by CMS HRSA OPTN SRTR the OPO and transplant center communities

Quality Assurance vs Quality Improvement Quality assurance: the use of standards that define acceptable or unacceptable levels of performance. failure to meet these standards may identify systems or processes that need to be changed. Quality improvement: the use of data to monitor processes and continuously improve outcomes

Types of Measures Metric Target/goal - level an OPO might wish to achieve Flagging criteria target with teeth Operational or clinical relevance e.g. if all OPOs below the mean on OTPD improved to the current mean, potential 3000 incremental organs per year

Types of Statistics Descriptive: summaries that may display patterns and distribution of data Mean, median, frequency, range, variance, standard deviation Inferential: inferences about a population based on your sample, for the purposes of generalization or prediction Regression Appropriate uses and conclusions from each

Available Metrics (SRTR) Descriptive Organs recovered per donor Organs transplanted per donor Number of organs transplanted Observed donation (conversion) rate Time to transplant Inferential: Expected donation (conversion) rate Expected organs transplanted per donor

Different OPO Outcomes Might Reflect: Case mix of donors Medical management practices Other OPO practices (placement efficiency, etc) Random chance

Understanding Differences in Outcomes Due to case mix? Use adjusted outcomes. Due to random chance? Ignore non-significant differences. Unimportant? Ignore differences that are not operationally or clinically meaningful. Due to unmeasured case mix? Consider unique circumstances within DSAs/UNOS regions. Practices or program quality? Adapt.

Why Compare Observed to Expected? Allows fair comparison of outcomes even when centers treat different types of patients OPOs with an unfavorable case mix might be doing a good job even if their observed outcomes are worse than average OPOs with a favorable case mix might be doing a poor job even if their observed outcomes are better than average Failure to properly adjust for case mix can lead to misinterpretation of OPO quality and perhaps even the recommendation and adoption of inferior practices

Risk Adjustment What rate would be expected for the donors at this OPO, if they had outcomes comparable to the national experience for similar donors? Similar defined by characteristics affecting event rates, such as: demographics cause of death medical history Differences between observed and expected are not due to adjustment factors because models have accounted for these factors

Conversion Rate (Observed) Your Donors (65% conversion) All U.S. Donors (70% conversion)

Conversion Rate (Observed) Your Donors (65% conversion) Your DSA performance is worse than the national average. If your case mix is the same. All U.S. Donors (70% conversion)

Compared to the Nation (Adjusted) Your Donors (65% conversion) Case mix is not the same. So we adjust for the differences. All U.S. Donors (70% conversion)

Compared to the Nation (Expected) Your Donors (65% conversion) Compared to the national average for donors like yours, your conversion rate is better than expected. U.S. Donorslike yours (60% conversion)

Observed vs. Expected: The Importance of Adjustment Observed Yield per 100 Donors 400 380 360 340 320 300 280 260 240 220 200 Dots represent DSAs

Watch the green dots Observed Yield per 100 Donors 400 380 360 340 320 300 280 260 240 220 200 2 nd highest observed yield 3rd lowest observed yield

Comparing Observed to Expected Yield 400 Observed Yield per 100 Donors 380 360 340 320 300 280 260 240 220 200 200 220 240 260 280 300 320 340 360 380 400 Expected Yield per 100 Donors

Observed vs. Expected Aggregate Yields 400 Observed Yield per 100 Donors 380 360 340 320 300 280 260 240 220 Higher than expected Lower than expected 200 200 220 240 260 280 300 320 340 360 380 400 Expected Yield per 100 Donors Dots represent DSAs

Observed vs. Expected Aggregate Yields 400 Observed Yield per 100 Donors 380 360 340 320 300 280 260 240 220 3rd lowest observed yield 2 nd highest observed yield 200 200 220 240 260 280 300 320 340 360 380 400 Expected Yield per 100 Donors Dots represent DSAs

Observed vs. Expected Aggregate Yields 400 Observed Yield per 100 Donors 380 360 340 320 300 280 260 240 220 Low observed yield does not mean lower-thanexpected performance 200 200 220 240 260 280 300 320 340 360 380 400 Expected Yield per 100 Donors Dots represent DSAs

Observed vs. Expected Aggregate Yields Observed Yield per 100 Donors 400 380 360 340 320 300 280 260 240 220 Same case mix (expect 308 organs from 100 donors), Different observed rates (75 more organs per 100 donors) 200 200 220 240 260 280 300 320 340 360 380 400 Expected Yield per 100 Donors Dots represent DSAs

Adjustments: When and Why Include variables that are statistically significant or near-significant Include variables that are clinically important and increase the face validity of the model Reject variables that adjust for practice patterns when creating standards May be ok to look at in models if you are testing for the effect of a practice Reject variables that produce unstable values Coefficients reflect biological effect of each factor on outcomes empirically

Data Quality What is collected Missingness Auditing? Gaming What is the definition of a donor? Organs recovered? Organs transplanted?

Individual Metrics Compare Practices Rather than OPOs Overall Data compares practices and outcomes No metrics for best OPO

Quality improvement vs flagging SRTR provides data on their website that are used by a variety of audiences Can be used for quality improvement by looking at outcomes over time Regulators (CMS, MPSC) use those same data to set standards and flag centers and OPOs for further review

CMS Metrics

Outcome Requirements Include three measures of organ donation rates and yield Reported for each OPO either by year (Measures 1 and 3) by each aggregate 18 month period in a 36 month interval (Measure 2) Based on data submitted to the OPTN by OPOs

Three Outcome Requirements 1. Observed donation rate 2. Observed donation rate compared to expected rate 3. Organ yield measures i. Transplants per ECD ii. Transplants per SCD iii. Organs for Research per Donor

Donor Types for CMS Metrics Donation After Cardiac Death Donors (DCD) Non-heart beating donors Expanded Criteria Donors (ECD) Heart-beating donors age 60+, and Heart-beating donors aged 50-59 meeting two of the following three conditions: died of a stroke, had a history of hypertension, or had a serum creatinine > 1.5 Standard Criteria Donor (SCD) Donors under age 60 who are not DCD or ECD

Definitions: Donors and Organs Recovered Donors include any deceased donor from whom at least one solid organ was recovered for the purpose of transplantation, regardless of whether the organ was transplanted or not. Up to 8 organs can be recovered from each donor 1 heart 2 lungs 2 kidneys 1 pancreas 1 liver 1 intestine

Measure 1: Donation Rate Regulation The OPO s donation rate of eligible donors as a percentage of eligible deaths is no more than 1.5 standard deviations below the mean national donation rate of eligible donors as a percentage of eligible deaths averaged over the four years of the recertification cycle. Both the numerator and the denominator of an individual OPO s donation rate ratio are adjusted by adding a 1 for each donation after cardiac death and each donor over the age of 70.

Measure 1: Donation Rate Donor and Death Definitions Eligible Deaths OPTN definition (heart-beating potential donor declared brain dead, age <71, not having any exclusionary conditions) Eligible Donors Donors at the OPO meeting eligible death criteria Additional Donors Donors at the OPO that are not from among eligible deaths (eg. aged 71+ or DCD or among excluded causes of death)

Measure 1: Donation Rate Rate Definitions Observed Donation Rate Eligible donors per 100 eligible deaths Adjusted Donation Rate* Eligible donors per 100 eligible deaths including additional donors Number of additional donors added to both the numerator and denominator *CMS definition; not case mix adjusted

Measure 1: Donation Rate Cutoff Value National Mean Donation Rate Average of observed OPO donation rates Cutoff Value 1.5 standard deviations below the national mean donation rate Statistically unadjusted Both the observed and adjusted donation rates compared to the same cutoff value

Measure 1: Donation Rate Summary Identify those OPOs that do not meet the donation rate cutoff with and without the additional credit This measure is designed to identify OPOs with donation rates that are substantially below the norm

Measure 2: Donation Rate vs. Expected Regulation The observed donation rate is not significantly lower than the expected donation rate for 18 or more months of the 36 months of the data used for recertification as calculated by the SRTR. Jan 2008 Jun 2009 Feb 2008 Jul 2009 Mar 2008 Aug 2009 Apr 2008 Sep 2009 May 2008 Oct 2009 An OPO in compliance for any of the 18-month periods within the recertification cycle is considered in compliance with the entire measure.

Measure 2: Donation Rate vs. Expected Rate Definitions Observed Donation Rate Eligible donors per 100 eligible deaths Expected Donation Rate Eligible donors expected per 100 eligible deaths based on SRTR conversion rate model

Risk Adjustment Factors in SRTR Conversion Rate Model Donor Age Donor Gender Donor Race White, Black, Hispanic, Asian, Other Donor Cause of Death Stroke, Anoxia, Head Trauma, Other

Measure 2: Donation Rate vs. Expected Summary Identify those OPOs with donation rates significantly (p<0.05) less than that expected This measure is designed to identify OPOs with donation rates that are below the norm after adjustment for characteristics of the eligible death population in the OPO

Measure 3: Organ Yield Regulation for Contiguous States At least 2 out of 3 of the following yield measures are no more than 1 standard deviation below the national mean averaged over the four years of the recertification cycle. i. Number of organs transplanted per SCD, including pancreata used for islet cell transplantation. ii. iii. Number of organs transplanted per ECD, including pancreata used for islet cell transplantation. Number of organs used for research per donor, including pancreata used for islet cell research.

Requirements for Non-Contiguous States/Territories (Hawaii, Puerto Rico) Measure 1: same Measure 2: same Measure 3: similar but with different components i. Kidneys transplanted per SCD ii. Kidneys transplanted per ECD iii. Organs for research also includes pancreas islet cells transplanted

Counting Organs Transplanted Each organ or organ segment recovered at the OPO that results in a transplant counts as one organ transplanted Includes organs exported to another OPO for transplant Organs divided into segments count as more than one organ transplanted if segments go to different recipients Each organ in a multiple organ transplant counts as one organ transplanted A pancreas transplanted for islet cells is counted as one organ transplanted

Organs for Research per Donor A donor must have at least one organ recovered intended for transplant to be included in this statistic Organs for research include organs (from these donors) that are recovered not for transplant, but instead for research, for pancreas islet cells, or for extra-corporeal liver for transplant, but submitted for research or for extra-corporeal liver for pancreas islet transplant (applies to Hawaii and Puerto Rico OPOs only)

National Means Measure 3: Organ Yield Cutoff Values Separate national average for each yield measure Contiguous OPO yield measures: Averages calculated for contiguous OPOs only Non-Contiguous (Puerto Rico, Hawaii) OPO yield measures: Averages include all OPOs Cutoff Values Separate cutoff value for each yield measure 1 std deviation below the respective national mean

Measure 3: Organ Yield Summary Identify those OPOs that do not meet the cutoff for at least 2 out of the 3 organ yield measures Transplants per SCD Transplants per ECD Organs for research per donor Yield measures are different for contiguous and non-contiguous OPOs

Strengths of CMS Metrics Includes measures for multiple steps in the donation and transplantation process Using an adjusted expected rate for comparison is appropriate and provides a better basis for evaluation of performance than does the use of the crude unadjusted rate

Limitations of CMS Metrics Benchmarks (mean, SD, expected) are only available after the fact and will change in each reporting period not easy for OPOs to know what target they are shooting for Measures 1 and 3 are unadjusted and do not account for case mix Measure 2 has limited adjustment variables due to data collection constraints

Limitations of CMS Metrics(2) Self-reported data (particularly eligible deaths and organs for research) may not be consistently collected across OPOs Uses binary ECD kidney definition across all organs Designed to predict KI graft outcome, not donation Inconsistent effect on OPOs of varying sizes Measures 1 and 3 will tend to flag small OPOs due to chance variation Measure 2 is less likely to flag small OPOs due to insufficient sample size

Potential Alternative OPO Metrics 1. Adjusted Yield 2. Donor profile index 3. Donors per million population 4. HRSA Donation and Transplantation Community of Practice (DTCP) goals

Alternative 1: Adjusted Yield Currently out for public comment for use by the OPTN Membership and Professional Standards Committees Developed by the Arbor Research as the SRTR contractor in conjunction with the OPTN OPO and Membership and Professional Standards Committees Uses both observed and expected yield to evaluate performance

Observed Yield DSA performance is currently assessed using organs transplanted per donor (OTPD, or observed yield) Sum of all organs transplanted from donors at the DSA Number of donors within the DSA Compared to what? Does not consider the relative quality of donors Reported in Table 1 of the OSRs on the SRTR website

National Performance National OTPD = 3.0 Individual DSAs range from 2.2 to 3.9 Low observed yield = lower performance High observed yield = higher performance

Expected Yield per Donor Based on the characteristics of an actual donor and using the national experience with donors of that type, the model equation calculates the number of organs that would be expected to be transplanted from that donor. Aggregate expected yield (0-8 organs) and also organ-specific expected yields (lung, kidney, heart, liver, pancreas)

Expected Yield of DSAs The expected counts for all of the donors at a DSA are summed. Sum of organs expected to be transplanted from donors at the DSA Number of donors within the DSA The observed and expected total counts can be reported as observed and expected yields per 100 donors.

Analytical Approach The OPO Committee identified characteristics of donors as they are presented to OPOs, not under the control of OPOs, in order to identify covariates appropriate for case mix adjustment. Does not include pumping, cold ischemic time, etc. Ordinal logistic regression: Aggregate count (0-8 organs) and kidney models (0-2 kidneys) Logistic regression (yes-no): Lung, liver, heart, and pancreas Organ-specific models complement the aggregate yield model to help pinpoint successes or opportunities that are suggested by the aggregate model.

Donor Characteristics in the Model (1) Aggregate Lung Kidney Heart Liver Pancreas Age Yes Yes Yes Yes Yes Yes Race Yes Yes No Yes Yes Yes Sex Yes Yes No Yes No No Blood type Yes Yes Yes Yes Yes Yes BMI Yes Yes No Yes Yes Yes Cardiac arrest Yes Yes Yes Yes Yes No DCD Yes Yes Yes No Yes Yes DCD controlled No No No No Yes No CDC high risk Yes Yes Yes Yes Yes Yes Outside of US Yes Yes Yes Yes No Yes Cause of death Yes Yes Yes Yes Yes Yes Circ. of death Yes Yes Yes No Yes Yes

Donor Characteristics in the Model (2) Mech. of death Aggregate Lung Kidney Heart Liver Pancreas Yes Yes Yes Yes No Yes Infection Yes Yes Yes Yes Yes No Hx of cancer Yes Yes Yes No No Yes Hx of cigarette Yes Yes Yes Yes Yes No past 6 mos. No Yes No No No No Hx of cocaine No Yes Yes No No Yes past 6 mos. Yes No No Yes Yes No Hx other drug No Yes No Yes Yes No Hx of diabetes Insulin depend. Hx of heavy alcohol Yes No Yes Yes Yes No Yes Yes Yes No Yes No Yes No Yes No Yes Yes

Donor Characteristics in the Model (3) Hx of hypertension Aggregate Lung Kidney Heart Liver Pancreas Yes No Yes Yes No Yes po 2 on FiO 2 Yes Yes No Yes Yes Yes Serum Creatinine HBV + (surface antigen) HBV + (core antibody) Yes Yes Yes Yes No Yes Yes No No No No No Yes Yes Yes Yes Yes Yes HCV + Yes No Yes No Yes No

O/E Ratio Ratio = observed yield expected yield Shows how many organs are transplanted per donor relative to how many are expected to be transplanted per donor Ratio of 1.0 means the observed yield is the same as the expected yield Ratio < 1.0 indicates that observed yield is lower than expected, ratio > 1.0 indicates that observed yield is higher than expected

Adjusted yield: Compare and contrast organizations # of donors O/E ratio % single organ donors 175 0.91 24% 36% 49 0.91 12% 45% 43 0.92 14% 76% 104 1.08 17% 59% 55 1.09 9% 41% 142 1.10 13% 16% % exported

Observed vs. Expected Aggregate Yields 400 Observed Yield per 100 Donors 380 360 340 320 300 280 260 240 220 Statistical significance vs. clinical significance 200 200 220 240 260 280 300 320 340 360 380 400 Expected Yield per 100 Donors Dots represent DSAs

Proposed MPSC Flagging Methodology Two year cohort Three criteria must be met to be flagged: A difference of at least 11 fewer observed organs per 100 donors than expected yield (Observed per 100 donors-expected per 100 donors < -10) A ratio of observed to expected yield less than 0.90 (O/E<0.90) A two-sided p-value less than 0.05

Strengths and Limitations of Adjusted Yield Model Strengths: Adjusted metric incorporates many differences in donor case mix Provides better benchmark for OPO performance than a single arbitrary unadjusted cutoff Limitations: Some covariates that are strongly related to yield that could not be included in the models e.g, hearts from DCD donors Geographic characteristics of DSAs that are unique to one or two DSAs cannot be included in the model e.g., area of DSA, distance from non-mainland DSAs to regional or national centers

Alternative 2: Donor Risk Index Continuous kidney donor risk index (KDRI) for deceased donor kidneys, combining donor and transplant variables to quantify graft failure risk Includes 14 donor and transplant factors, each found to be independently associated with graft failure or death The KDRI reflects the rate of graft failure relative to that of a healthy 40-year-old donor. Rao et al. Transplantation; 88(2): 231-236, 2009

FIGURE 2. Adjusted* graft survival by kidney donor risk index (KDRI) quintile. 100% Graft Survival 75% Median Graft KDRI Quintile Lifetime (years) 50% 0.45-<0.79 13.6 0.79-<0.96 12.6 25% 0.96-<1.15 10.8 1.15-<1.45 9.2 1.45+ 7.5 The curves are ordered, top to bottom, as quintile 1, quintile 2, quintile 3, quintile 4, quintile 5. Extrapolation was used for the first and second quintile. *Adjusted to a reference 50-year-old recipient. 0% 0 1 2 3 4 5 6 7 Time Since Transplant (Years) 8 9 10 Each survival pertains to a recipient who is aged 50 years, nondiabetic, and at the reference level of all other recipient factors. Rao et al. Transplantation; 88(2): 231-236, 2009

FIGURE 5. Expanded criteria donor (ECD) status by kidney donor risk index (KDRI) category. % of transplants 100 90 80 70 60 50 40 30 20 10 0 0.4-<0.6 0.6-<0.8 0.8-<1.0 1.0-<1.2 1.2-<1.4 1.4-<1.6 1.6-<1.8 Non-ECD 1.8-<2.0 Percentages of patients in ECD and non-ecd groups, by KDRI level. 2.0-<2.2 2.2-<2.4 ECD 2.4-<2.6 2.6-<2.8 2.8-<3.0 Kidney Donor Risk Index 3.0-<3.2 3.2-<3.4 3.4-<3.6 3.6-<3.8 4.0-<4.2 Rao et al. Transplantation; 88(2): 231-236, 2009

Strengths and Limitations Strengths: Greater detail than current dichotomous ECD definition Enhance decision-making about organ use Weaknesses: Unmeasured donor factors may also contribute May underestimate the risk of high risk organs if practices shift to use more of these type of organs

Alternative 3: Donors per million? Formerly used and dropped, but some have reproposed this as an alternative to the current conversion rate Strengths: Avoids self-reported data Consistent data collection Limitations: Deceased in numerator and living in denominator is inconsistent Unadjusted; does not account for differing case mix in populations Does not reflect donor potential per million

Alternative 4: HRSA DTCP Metrics 2013 goals 75% conversion rate 3.75 organs transplanted per donor 10% DCD

DTCP Conversion Rate, 2010 100.0 90.0 80.0 MEAN 70.0 1.5 STD DEV RATE 60.0 50.0 40.0 30.0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 OPO 76.1% 65.4% *CMS and HRSA Collaborative both use the methodology of adding all DCD donors and all donors over age 70 to both the numerator and denominator when calculating the conversion rate.

DTCP Organs Transplanted Per Donor, 2010 4.0 OTPD 3.80 3.60 3.40 3.20 3.0 2.80 2.60 2.40 2.20 MEAN 1 STD DEV 2.0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 OPO 3.15 2.85

Strengths and Limitations of DTCP Metrics Strengths: Gives clear targets Limitations: Unadjusted; does not account for differing case mix in populations May not be feasible for all OPOs to achieve Increasing DCD while simultaneously pushing OTPD may be challenging

Summary No single metric can define the quality of an OPO Individual metrics can provide insight into aspects of an OPO s performance Risk adjustment provides a common ground for description Each metric has strengths and weaknesses Metrics are evolving Different regulators have different performance thresholds