Hierarchical Generalized Linear Models for Behavioral Health Risk-Standardized 30-Day and 90-Day Readmission Rates

Transcription

1 Hierarchical Generalized Linear Models for Behavioral Health Risk-Standardized 30-Day and 90-Day Readmission Rates Allen Hom PhD, Optum, UnitedHealth Group, San Francisco, California Abstract The Achievements in Clinical Excellence (ACE) program encourages excellence across all behavioral health network facilities by promoting those that provide the highest quality of care. Two key benchmarks of outcome effectiveness in the ACE program are the risk adjusted 30-day readmission and risk adjusted 90-day readmission rates. Risk adjustment was performed with hierarchical general linear models (HGLM) to account for differences across hospitals in patient demographic and clinical characteristics. One year of administrative admission data (June 30, 2013 to July 1, 2014) from patients for 30-day (N=78,761, N Hospitals=2,233) and 90-day (N=74,540, N Hospital =2,205) time frames were the data sources. HGLM simultaneously models two levels 1) Patient level models log-odds of hospital readmission using age, sex, selected clinical covariates, and a hospital-specific intercept, and 2) Hospital level a random hospital intercept that accounts for within-hospital correlation of the observed. PROC GLIMMIX was used to implement a HGLM with hospital as a random (hierarchical) variable separately for substance use disorder (SUD) admissions and mental health (MH) admissions and pooled to obtain a hospital-wide risk adjusted readmission rate. The HGLM methodology was derived from Centers for Medicare & Medicaid Services (CMS) documentation for the 2013 Hospital-Wide All-Cause Risk-Standardized Readmission Measure SAS package. This methodology was performed separately on 30-day and 90-day readmission data. The final metrics were a hospital-wide risk adjusted 30-day readmission rate percent and a hospital-wide risk adjusted 90-day readmission rate percent. HGLM models were cross-validated on new production data that overlapped with the development sample. Revised HGLM models were tested in April, 2015, and the outcome statistics were extremely similar. In short, the test of the revised model cross-validated the original HGLM models, because the revised models were based on different samples. Background of Hospital Readmissions Reduction Program Section 3025 of the Affordable Care Act added section 1886(q) to the Social Security Act establishing the Hospital Readmissions Reduction Program, which requires Centers for Medicare & Medicaid Services (CMS) to reduce payments to hospitals with excess readmissions, effective for discharges beginning on October 1, Hospitals with greater than expected readmission rate are subject to financial penalty. Performance was based on 30- day readmission metrics for three conditions that started in acute myocardial infarction, heart failure, and pneumonia. CMS contracted with Yale New Haven Health Services Corporation/Center for Outcomes Research and Evaluation (YNHHSC/CORE) to develop a claims-based, risk-adjusted hospital- 1

2 wide readmission (HWR) measure for public reporting that reflects the quality of care for hospitalized patients in the United States. The hospital-wide risk-standardized readmission rate (RSRR) is a summary score derived from the weighted geometric mean of five statistical models built for groups of admissions that are clinically related: medicine, surgery/gynecology, cardiorespiratory, cardiovascular, and neurology. For each specialty, an index admission is the hospitalization to which the readmission outcome is attributed and includes admissions criteria for patients in the CMS study: Must meet Peer Review Enrolled in Medicare fee-for-service (FFS); Aged 65 or over; Discharged from non-federal acute care hospitals; Without an in-hospital death; Not transferred to another acute care facility; and, Enrolled in Part A Medicare for the 12 months prior to the date of the index admission. Excluded were admissions without at least 30 days post-discharge enrollment in FFS Medicare, patients who leave against medical advice, admissions for medical cancer treatment, admission for primary psychiatric diagnoses, and admissions for rehabilitation. Despite encouraging results from Medicare quality improvement interventions since 2008, the overall national readmission rate remains high, with a 30-day readmission following nearly 20% of discharges. RISK ADJUSTMENT METHODOLOGY Readmission Outcome Definition Thirty-Day and Ninety-Day Timeframe The two outcomes are unplanned all-cause 30-day readmission and unplanned all-cause 90- day readmission. A 30-day readmission is a subsequent inpatient admission to any acute care facility which occurs within 30 days of the discharge date of an eligible index admission. A 90- day readmission is a subsequent inpatient admission to any acute care facility which occurs within 90 days of the discharge date of an eligible index admission. All-Cause Readmission Admissions for acute illness or for complications of care are not planned. Any procedure completed during an admission for an acute illness is not likely to have been planned, even if that procedure is usually planned in other non-acute cases. The CMS methodology for Hospital-Wide Risk-Standardized Readmission Metrics was applied to unplanned behavioral health acute inpatient readmissions within 30 and 90-days of discharge. 2

3 Definition of Eligible Admissions Patient must have continuous eligibility with the health plan for 12 months prior to the initial admission. Eligibility at least one day after the discharge date is required to ensure that the absence of a readmission is not simply due to loss of the behavioral health benefit. Must meet Peer Review For 30-day readmission, the readmission had to be greater than or equal to two days and less than or equal to thirty days. For 90-day readmission, the readmission had to be greater than or equal to two days and less than or equal to ninety days. Exclusion Criteria for 30-day readmission: In-hospital death Patient is without at least 30 days post discharge continuous enrollment (because the 30 day readmission outcome cannot be assessed in this group). Patient is without at least 365 days of continuous enrollment prior to admission start date Transferred to another acute care facility Discharged against medical advice (AMA) Same day discharges For 90-day readmission: In-hospital death Patient is without at least 90 days post discharge continuous enrollment (because the 90 day readmission outcome cannot be assessed in this group). Patient is without at least 365 days of continuous enrollment prior to admission start date Transferred to another acute care facility Discharged against medical advice (AMA) Same day discharges Substance Use Disorder Cohort Behavioral health admissions were divided into two categories that represent major divisions of acute inpatient treatment programs within hospitals: Substance Use Disorder (SUD) and Mental Health (MH) therapy. A substance use disorder describes a problematic pattern of using alcohol or another substance that results in impairment in daily life or noticeable distress. A person with this disorder will often continue to use the substance despite negative consequences. The SUD sample was 19% of the 30-day readmission sample (14,786 out of 78,761 patients, tables 1 and 2) and the 90-day readmission sample (13,993 out of 74,540 patients, tables 3 and 4). Behavioral Health Cohort The mutually exclusive MH group was composed of index admissions for all other DSM-5 primary diagnosis categories that were not SUD such as: Depressive Disorders, Bipolar and Related Disorders, Schizophrenia Spectrum and Other Psychotic Disorders, Neurocognitive 3

4 Disorders, Anxiety Disorders, Disruptive, Impulse-Control and Conduct Disorders, Neurodevelopmental Disorders, Feeding and Eating Disorders, and Personality Disorders. The MH sample was 81% of the 30-day readmission sample (63,975 out of 78,761 patients, tables 5 and 6) and the 90-day readmission sample (60,547 out of 74,540 patients, tables 7 and 8). In short, the MH sample was five times larger than the SUD sample. Data Sources Primary and secondary DSM-5 mental health diagnoses were extracted for a year prior to the index admissions from a hospital administrative claims data warehouse. If the primary and secondary prior diagnosis were from the same DSM-5 category, they were combined to create a single indicator DSM-5 risk variable for the past year (see Table 1). Modelling Approach Risk adjustment was performed with hierarchical general linear models (HGLM) that accounts for age, gender, current admissions, mental health diagnoses in the past year, product type (Commercial, Medicaid, Medicare), involuntary admission to acute mental health hospitalization, and current electro-convulsive therapy. Logistic regression was used to screen variables and obtain estimated odd ratios for variables. The goal of risk adjustment is to account for differences across hospitals in patient demographic and clinical characteristics that might be related to the outcome. Hospital-level 30-day and 90-day all-cause Behavioral Health Risk Standardized Readmission Rates (RSRR) for SUD and MH readmissions were estimated using Hierarchical Generalized Linear Models (HGLM) to adjust for patient clustering (hierarchically correlated) effects within hospitals. HGLM simultaneously models two levels (patient and hospital) - Patient level models log-odds of hospital readmission using age, sex, selected clinical covariates, and a hospital-specific intercept. - Hospital level a random hospital intercept for Hierarchical General Linear Model (HGLM) that accounts for within-hospital correlation of the observed outcomes and models the assumption that underlying differences in quality among the health care facilities being evaluated lead to systematic differences in outcomes. Steps in Calculating the Behavioral Health Risk-Standardized 30-Day Readmission Rate A logistic regression (PROC LOGISTIC in SAS ) was performed to screen variables prior to HGLM analysis, obtain fit statistics, odd ratio (OR) estimates, OR 95% confidence limits, and residual statistics. PROC GLIMMIX was used to implement a HGLM with hospital as a random (hierarchical) variable separately for SUD admissions and MH admissions. The HGLM methodology and SAS code were derived from CMS documentation for the 2013 Hospital-Wide All-Cause Risk-Standardized Readmission Measure SAS package that was developed by the 4

5 Yale-New Haven Health Services Corporation/Center for Outcomes Research & Evaluation. For each model, the GLIMMIX procedure implements an HGLM that outputs the predicted number of admissions for index admissions and the number of expected admissions that will be used in the next step to compute the Standardized Readmission Ratio. Standardized Readmission Ratio The Standardized Readmission Ratio (SSR) is calculated as the ratio of the predicted number of admissions to the number of expected readmissions for a given hospital. For each hospital, the numerator of the SRR is the number of readmissions within 30 days predicted based on the hospital s performance with its observed case mix and service mix. The denominator is the number of readmissions expected based on the performance of an average hospital with similar case mix and service mix. This approach is analogous to a ratio of observed to expected used in other types of statistical analyses. SRR Cj = predict Cj / expect Cj c = admissions in cohort (SUD, MH) j = hospital Two SRRs are computed: SRR_SUD and SRR_MH. 1) A national raw readmission rate for the all SUD admissions is computed. 2) A national raw readmission rate for the all MH admissions is computed. 3) A program-wide RSRR is computed separately for each SUD and MH where: - SUD RSRR = SUD SSR x SUD national raw readmission rate. - MH RSRR = MH SSR x MH national raw readmission rate. To report a single SRR for Hospital-Wide Readmission (SRR_HWR), the SUD and MH SRRs were pooled to compute a composite hospital volume (admissions) weighted logarithmic mean: SRR j = exp ( ( Σ m cj log(r cj )) / Sm cj ) c = admissions in cohort (SUD, MH) j = hospital m cj = admissions per cohort for hospital R cj = SRR for the condition A SRR less than one indicates a lower-than-expected readmission rate (better quality), while a SRR greater than one indicates a higher-than-expected readmission rate (worse quality). The corresponding SAS code takes into account that some hospitals have a SUD or MH program, but not both: mod_30day_readmit_admit_cnt = volume_sud + volume_mh; IF volume_sud >0 THEN SUD_NUM = volume_sud * LOG(SRR_SUD); ELSE SUD_NUM=0; IF volume_mh >0 THEN MH_NUM = volume_mh * LOG(SRR_MH); ELSE MH_NUM=0; total_num = sum(sud_num, mh_num); SRR _HWR = exp(total_num/mod_30day_readmit_admit_cnt); SRR_HWR is the Standardized Readmission Ratio (SRR) for Hospital-Wide Readmission (HWR) 5

6 Risk Adjusted 30-Day and 90-Day Readmission Percent A national unadjusted average readmission rate is obtained from the combined overall SUD and MH samples. The composite Facility-wide SRR_HWR is multiplied by the national average readmission rate to produce hospital-wide Risk Adjusted 30-day Readmission Percent (RSK_ADJ_30day_readmit_pct). RSK_ADJ_30day_readmit_pct = (SRR _HWR * &HWYBAR); &HWYBAR is a SAS macro variable that contains the national unadjusted average readmission rate. The Risk Adjusted 30-day Readmission Percent (e.g., RSK_ADJ_30day_reamit_pct) is equivalent to the CMS metric that is known as Risk Standardized Readmission Rate (RSRR) The entire Risk Adjustment Methodology section (page 2-5) is repeated for 90-day readmission to obtain a hospital-wide Risk Adjusted 90-day Readmission Percent (RSK_ADJ_90day_readmit_pct). RESULTS Variables in Substance Use Disorder HGLM Analysis Updated 30 and 90-day readmit models were created based upon the new mental health DSM- 5 variables, after DSM-4 became obsolete. Index admissions for the current development data were from June 30, 2013 to July 1, The new models were cross-validated on new production data, which spanned a year from January 1, 2014 to December 31, There is, however, an overlap of half year of 2014 between the two samples. Slightly more than half of the production sample had patients in common with development sample. Tables 1, 2, 3 and 4 show the variables in the validation production data for Substance Use Disorder HGLM analysis sorted by F values. The top four variables in the 30-day SUD readmit HGLM were: 1. or secondary SUD diagnosis in the past year (F=71.03). 2. or secondary bipolar and related disorder diagnosis in the past year (F=58.54). 3. or secondary depressive disorder diagnosis in the past year (F=49.26). 4. Patient is covered by public sector health insurance - Medicaid/Medicare (F=25.76). The same variables are in top four for the 90-day SUD readmit HGLM, with different ranking: 1. or secondary SUD diagnosis in the past year (F=119.24). 2. Patient is covered by public sector health insurance - Medicaid/Medicare (F=45.78). 3. or secondary bipolar and related disorder diagnosis in the past year (F=45.07). 4. or secondary depressive disorder diagnosis in the past year (F=41.16). The odds ratio indicates that patients with a substance-related and addictive disorder diagnosis in the past year are twice (OR=2.19 and OR=2.25) as likely to have an inpatient readmission. In 6

7 addition, the importance of bipolar and depressive disorders suggests that patients may be selfmedicating to control negative mood swings. There is also the issue of co-occurring mental illness and substance use disorders. In 2014, a national survey on drug use and health in the United States from the Substance Abuse and Mental Health Services Administration (SAMHSA) found that among adults with diagnosis of SUD in the past year, 39.1% also had a co-occurring diagnosis for mental illness in the past year. % Acute Logistic N=14,786 Admissions Regression HGLM HGLM Variable Description For Variable OR (95% CL) T Value F Value Substance Abuse Bipolar Disorder Risk Depressive Public Sector or secondary substance-related and addictive disorder DX in the past year or secondary bipolar and related or secondary depressive disorder DX in the past year 1= Medicare / Medicaid 0=Commercial 72.70% 2.19 (1.86, 2.58) % 1.85 (1.58, 2.18) % 1.64 (1.45, 1.86) % 1.69 (1.46, 1.95) Gender female =1 male= % 0.77 (0.68, 0.87) Anxiety Schizophrenia Age Category or secondary anxiety disorder DX in the past year or secondary schizophrenic spectrum and other psychotic 1=age <=12, 2=13-17, 3=18-25, 4=26-64, 5= >= % 1.46 (1.26, 1.69) % 1.37 (1.09, 1.72) See Table 2 Table 1: Production Facility Readmit 30-Day Model for SUD 0.82 (0.73, 0.92) Age Category Frequency Percent Less than or equal to 12 years % years old % years old 4, % years old 9, % Greater than or equal to 65 years % Total 14,786 Table 2: Age Breakdown for Readmit 30-Day Model for SUD 7

8 N= 13,993 % Acute Admissions Logistic Regression HGLM HGLM Variable Description For Variable OR (95% CL) T Value F Value / Substance secondary substance-related Abuse 72.49% 2.25 (2.00, 2.53) and addictive disorder DX in the past year Public Sector Bipolar Disorder Risk Depressive Age Category Anxiety Schizophrenia Other Mental 1= Medicare or Medicaid 0=Commercial / secondary bipolar and related / secondary depressive 1=age <=12, 2=13-17, 3=18-25, 4=26-64, 5= >=65 or secondary anxiety disorder DX in the past year / secondary schizophrenic spectrum and other psychotic disorder DX in past year / secondary other mental 18.26% 1.73 (1.55, 1.94) % 1.58 (1.38, 1.80) % 1.47 (1.33, 1.61) See Table (0.67, 0.80) % 1.43 (1.27, 1.60) % 1.39 (1.15, 1.67) % 1.54 (1.20, 1.99) Gender female =1 male=0 37.5% 0.92 (0.84, 1.01) Neurocognitive / secondary neurocognitive Table 3: Production Facility Readmit 90-Day Model for SUD 0.81% 1.06 (0.69, 1.65) Age Category Frequency Percent Less than or equal to 12 years % years old % years old 4, % years old 9, % Greater than or equal to 65 years % Total 13,993 Table 4: Age Breakdown for Facility Readmit 90-Day Model for SUD 8

9 Variables in Mental Health Production HGLM Analysis Tables 5, 6, 7 and 8 show the variables in the validation production data for Mental Health HGLM analysis sorted by F values. The F values are larger (than SUD HGLM) and odd ratio confidence limits are narrower (and therefore more confidence/reliability) because the MH sample is five times larger than the SUD sample. The top five variables in the 30-day MH readmit HGLM were: 1. or secondary bipolar and related disorder diagnosis in the past year (F=187.65). 2. or secondary schizophrenic spectrum and other psychotic disorder diagnosis in the past year (F=181.30). 3. or secondary depressive disorder diagnosis in the past year (F=176.18). 4. Patient is covered by public sector health insurance - Medicaid/Medicare (F=168.84). 5. or secondary substance-related and addictive disorder DX in the past year (F=106.23). The same variables are in top five for the 90-day MH readmit HGLM, but with different rankings: 1. or secondary depressive disorder diagnosis in the past year (F=363.23). 2. Patient is covered by public sector health insurance - Medicaid/Medicare (F=339.63). 3. or secondary schizophrenic spectrum and other psychotic disorder diagnosis in the past year (F=338.61). 4. or secondary bipolar and related disorder diagnosis in the past year (F=305.03). 5. or secondary substance-related and addictive disorder DX in the past year (F=159.68). For both SUD and MH 90-day readmission models, the second most important variable is public sector (i.e., Medicare/Medicaid) health insurance. For 30-day readmission models public sector insurance is also in one of the top four predictors. The MH models also show that after mental health problems (bipolar, schizophrenia, and depressive disorders) substance abuse is also an important issue in readmission. 9

10 N= 63,975 % Acute Admissions Logistic Regression HGLM HGLM Variable Description For Variable OR (95% CL) T Value F Value Bipolar Disorder Risk Schizophrenia Depressive / secondary bipolar and related disorder DX in the past year / secondary schizophrenic spectrum and other psychotic disorder DX in the past year or secondary depressive disorder DX in the past year 25.64% 1.60 (1.51, 1.70) % 1.55 (1.45, 1.64) % 1.46 (1.38, 1.54) Public Sector Substance Abuse Other Mental Anxiety Age Category Trauma or Stress-Related Feeding and Eating Disorder Risk 1= Medicare / Medicaid 0=Commercial / secondary substance-related and addictive or secondary other mental disorder DX in the past year or secondary anxiety disorder DX in the past year 1=age <=12, 2=13-17, 3=18-25, 4=26-64, 5= >=65 / secondary trauma or stressor-related secondary feeding and eating disorder DX in the past year 33.53% 1.57 (1.48, 1.66) % 1.44 (1.36, 1.52) % 1.48 (1.37, 1.61) % 1.17 (1.10, 1.25) See Table (0.89, 0.95) % 1.19 (1.11, 1.27) % 1.54 (1.30, 1.83) Gender female =1 male= % 0.91 (0.86, 0.95) Personality Index Schizophrenia Impulse- Control and Conduct or secondary other personality schizophrenic spectrum and other psychotic disorder DX or secondary impulse-control and conduct disorder DX in the past year 2.68% 1.30 (1.14, 1.47) % 1.44 (1.15, 1.80) % 1.16 (1.03, 1.29) Table 5: Production Facility Readmit 30-Day Model for MH 10

11 N= 63,975 % Acute Admissions Logistic Regression HGLM HGLM Variable Description For Variable OR (95% CL) T Value F Value Index Obsessive Compulsive Disorder Risk Index Trauma or Stress-Related Index Anxiety Index Bipolar Index Impulse- Control and Conduct Index Feeding and Eating Index Neurocognitive Index Neurodevelopmental Index Depressive Index Personality Index Other Mental obsessive-compulsive and related disorder DX trauma or stressor-related disorder DX anxiety disorder DX bipolar and related disorder DX impulse-control and conduct disorder DX feeding and eating disorder DX neurocognitive disorder DX neurodevelopmental disorder DX depressive disorder DX personality disorder DX other mental disorder DX 0.20% 1.95 (1.15, 3.29) % 0.83 (0.63, 1.09) % 0.77 (0.56, 1.07) % 1.18 (0.94, 1.47) % 1.25 (0.90, 1.73) % 0.82 (0.53, 1.28) % 0.87 (0.63, 1.19) % 1.16 (0.79, 1.71) % 0.98 (0.79, 1.22) % 1.04 (0.63, 1.72) % 0.99 (0.56, 1.74) Table 5 - Production Facility Readmit 30-Day Model for MH, continued Age Category Frequency Percent Less than or equal to 12 years 2, % years old 12, % years old 12, % years old 30, % Greater than or equal to 65 years 6, % Total 63,975 Table 6: Breakdown of Age for Facility Readmit 30-Day Model for MH 11

12 N= 60,547 % Acute Admissions Logistic Regression HGLM HGLM Variable Description For Variable OR (95% CL) T Value F Value Depressive Public Sector Schizophrenia Bipolar Disorder Risk Substance Abuse Age Category Other Mental Trauma or Stress-Related Anxiety Feeding and Eating Personality Index Schizophrenia Involuntary admission Impulse- Control and Conduct / secondary depressive disorder DX in the past year 1= Medicare or Medicaid 0=Commercial / secondary schizophrenic spectrum and other psychotic / secondary bipolar and related / secondary substance-related and addictive disorder DX in the past year 1=age <=12, 2=13-17, 3=18-25, 4=26-64, 5= >=65 / secondary other mental / secondary trauma or stressorrelated disorder DX in the past year / secondary anxiety disorder DX in past year / secondary feeding and eating / secondary personality disorder DX in the past year schizophrenic spectrum and other psychotic disorder DX Involuntary admission to acute MH hospitalization / secondary impulse-control and conduct disorder DX in the past year 50.32% 1.58 (1.51, 1.65) % 1.68 (1.60, 1.77) % 1.69 (1.61, 1.78) % 1.60 (1.53, 1.69) % 1.50 (1.42, 1.57) See Table 8 Table 7: Production Facility Readmit 90-Day Model for MH 0.88 (0.85, 0.90) % 1.43 (1.33, 1.54) % 1.19 (1.13, 1.26) % 1.17 (1.11, 1.23) % 1.61 (1.39, 1.86) % 1.35 (1.21, 1.51) % 1.57 (1.29, 1.90) % 0.93 (0.89, 0.97) % 1.15 (1.04, 1.26)

13 N= 60,547 % Acute Admissions Logistic Regression HGLM HGLM Variable Description For Variable OR (95% CL) T Value F Value Gender female =1 male= % 0.94 (0.91, 0.99) Index Trauma or Stress-Related Electro-Convulsive Therapy Index Bipolar Index Neurocognitive Index Anxiety Index Feeding and Eating Disorder Risk Index Obsessive Compulsive Neurodevelopmental Index Impulse- Control and Conduct Disorder Risk Index Personality Index Depressive Index Other Mental Index Neurodevelopmental trauma or stressorrelated disorder DX Presence of electroconvulsive therapy bipolar and related disorder DX neurocognitive disorder DX anxiety disorder DX feeding and eating disorder DX other obsessivecompulsive and related disorder DX or secondary neurodevelopmental disorder DX in the past year impulse-control and conduct disorder DX other personality disorder DX depressive disorder DX other mental disorder DX neuro-developmental disorder DX 3.17% 0.77 (0.61, 0.97) % 1.17 (0.98, 1.40) % 1.21 (1.00, 1.47) % 0.80 (0.60, 1.06) % 0.85 (0.65, 1.11) % 0.79 (0.55, 1.13) % 1.39 (0.86, 2.25) % 1.03 (0.96, 1.11) % 1.16 (0.87, 1.53) % 1.10 (0.73, 1.68) % 1.00 (0.82, 1.20) % 0.94 (0.58, 1.54) % 0.99 (0.71, 1.38) Table 7: Production Facility Readmit 90-Day Model for MH, continued 13

14 Age Category Frequency Percent Less than or equal to 12 years 2, % years old 12, % years old 11, % years old 28, % Greater than or equal to 65 years 5, % Total 60,547 Table 8: Breakdown of Age for Facility Readmit 90-Day Model for MH Index admissions for the development data were from June 30, 2013 to July 1, The new models were cross-validated on new production data, which spanned a year from January 1, 2014 to December 31, Thus, there is an overlap of half year of 2014 between the two samples. SUD and MH Model Performance (Logistic Regression) Table 9 and 10 shows the performance for the logistic regression models (SUD and MH) and the unadjusted readmission rate versus the risk adjusted readmission rate (from HGLM analysis). Summary statistics were computed to assess model performance: calibration (a measure of over fitting), discrimination in terms of predictive ability, discrimination in terms of c statistic (equivalent to area under the receiver operating curve [ROC]), distribution of residuals, and model chi square. Over-fitting refers to the phenomenon in which a model describes the relationship between predictive variables and outcome well in the development dataset, but fails to provide valid predictions in new patients. Since the γ0 in the validation sample is close to zero and the γ1 is close to one in each of the models, there is little evidence of over-fitting. Discrimination in predictive ability measures the ability to distinguish hig h-risk subjects from lowrisk subjects. It appears that the 90-day SUD model is better is more discriminating in terms of high risk subjects (larger range of difference) than the 30-day SUD model. The 90-day MH model also shows the same difference versus the 30-day MH model, but to a lesser extent. The c statistic is a measure of how accurately a statistical model is able to distinguish between a patient with and without an outcome. For binary outcomes the c statistic is identical to the ROC. A c statistic of 0.50 indicates random prediction, implying all patient risk factors are useless. A c statistic of 1.0 indicates perfect prediction, implying patients outcomes can be predicted completely by their risk factors, and physicians and hospitals play no role in patients outcomes. While higher c statistic is desirable, we do not want to maximize it by adjusting for factors that should not be adjusted for. For example, we do not want to include in -hospital complications as a risk factor. The range of c statistic results is to for all models which is in line with results we have seen for other 30-day and 90-day readmission measures and the CMS models. The Pearson Residuals show that about 90% are within the range of -2 to 0, which is consistent with the CMS model results. 14

15 Table 11 shows unadjusted readmission rates, the risk adjusted readmission rates, and deciles for combined HGLM models that were pooled over SUD and MH per hospital. Table 12 shows the descriptive statistics for mean and median national observed (unadjusted) readmission rate and risk adjusted rates for hospitals with 25 or more admits broken down by overall 30 and 90- days models, SUD 30 and 90-day models, and MH 30 and 90-day models. Quartile and decile statistics, and the total number of admissions and total number of hospitals are also displayed. Indices Development SUD 30 Day Validation Production SUD 30 Day Development SUD 90 Day Validation Production SUD 90 Day Number of hospital stays 15,532 14,786 14,487 13,993 Number of hospitals 1,080 1,068 1,061 1,046 Number of hospitals with 25 or more admits Unadjusted rate for hospitals with GE 25 admits Risk adjusted rate for hospitals with GE 25 admits (HGLM) % 8.3% 18.2% 18.3% 8.5% 9.0% 19.2% 19.3% Calibration (γ0, γ1) (0.096, 1.030) (0.087, 1.029) (0.035, 1.015) (0.057, 1.026) Discrimination Predictive Ability (lowest decile %, highest decile %) Among Facilities with 25 or More Admits Unadjusted readmission rate 1.6% % 2.0% % 6.7% % 7.9% % Risk adj readmission rate 5.3% % 6.0% % 13.5% % 13.6% % Discrimination C statistic Breakdown of Distribution of Pearson Residuals Less than to < to < Greater Than or Equal to Model Likelihood χ2 (DF) (8) (8) (10) (10) R-Square Max Rescaled R-Square Table 9: SUD Model Performance (Logistic Regression) 15

16 Indices Development MH 30 Day Validation Production MH 30 Day Development MH 90 Day Validation Production MH 90 Day Number of hospital stays 66,788 63,981 62,566 60,552 Number of hospitals 1,934 1,895 1,906 1,873 Number of hospitals with 25 or more admits Unadjusted rate for hospitals with 25 or more admits Risk adjusted rate for hospitals with 25 or more admits (HGLM) % 11.9% 20.8% 20.3% 12.1% 12.1% 21.4% 20.5% Calibration (γ0, γ1) (0, 1) (0, 1) (0, 1) (0, 1) Discrimination Predictive Ability (lowest decile %, highest decile %) Among Facilities with 25 or More Admits Unadjusted readmission 5.0% % 5.0% % 11.5% % 11.3% % rate Risk Adj readmission rate 10.8% % 10.5% % 19.4% % 18.6% % Discrimination C statistic Breakdown of Distribution of Pearson Residuals Less than to < to < Greater Than or Equal to Model Likelihood χ2 (DF) (25) (25) (28) (28) R-Square Max Rescaled R-Square Table 10: MH Model Performance (Logistic Regression) 16

17 Indices Development Overall 30 Day Validation Production Overall 30 Day Development Overall 90 Day Validation Production Overall 90 Day Number of hospital stays 82,320 78,761 77,053 74,545 Number of hospitals 2,247 2,233 2,215 2,205 Number of hospitals with GE 25 admits Unadj rate hospitals with GE 25 admits 11.0% 11.1% 20.2% 19.8% Risk adj rate hospitals with GE 25 admits 11.4% 11.5% 20.9% 20.2% Discrimination Predictive Ability (lowest decile %, highest decile %) Among Facilities with 25 or More Admits Unadjusted readmission rate 3.9% % 4.0% % 10.6% % 10.3% % Risk Adjusted readmission rate 9.9% % 9.7% % 18.6% % 17.9% % Table 11 - Overall (Pooled SUD and MH per hospital) HGLM Model Performance Mean SD Min 10th Per c entile 25th Per c entile 50th Per c entile 75th Per c entile Overall 30-Day Readmissions N Admissions = 78,761 N Hospitals = 2,233 90th Per c entile 100 Per c entile Unadjusted 30-Day 11.1% 6.0% 0.0% 4.0% 7.1% 10.7% 14.3% 18.6% 50.0% Risk adjusted 30-Day 11.5% 2.0% 5.6% 9.7% 10.4% 11.3% 12.3% 13.5% 35.9% Overall 90-Day Readmissions N Admissions = 74,540 N Hospitals = 2,205 Unadjusted 90-Day 19.8% 7.8% 0.0% 10.3% 14.4% 19.2% 24.7% 30.1% 52.5% Risk adjusted 90-Day 20.2% 2.7% 9.9% 17.9% 18.9% 20.0% 21.1% 22.8% 46.5% SUD 30-Day Readmissions N Admissions = 14,786 N Hospitals = 1,068 SUD Unadj 30 Day 8.3% 6.3% 0.0% 2.0% 3.7% 7.1% 11.6% 15.9% 50.0% SUD Risk adj 30 Day 9.0% 3.2% 4.2% 6.0% 6.8% 8.1% 10.2% 12.7% 27.0% SUD 90-Day Readmissions N Admissions = 13,993 N Hospitals = 1,046 SUD Unadj 90-Day 18.3% 9.1% 2.1% 7.9% 11.1% 16.9% 24.6% 30.8% 50.0% SUD Risk adj 90-Day 19.3% 5.4% 9.0% 13.6% 15.4% 18.0% 22.9% 27.3% 42.1% MH 30-Day Readmissions N Admissions = 63,975 N Hospitals = 1,895 MH Unadj 30-Day 11.9% 5.8% 0.0% 5.0% 7.8% 11.4% 15.0% 19.4% 40.0% MH Risk adj 30-Day 12.1% 1.5% 7.7% 10.5% 11.1% 11.9% 13.0% 14.0% 20.1% MH 90-Day Readmissions N Admissions = 60,547 N Hospitals = 1,873 MH Unadj 90-Day 20.3% 7.5% 0.0% 11.3% 15.4% 19.5% 24.7% 30.3% 52.5% MH Risk adj 90-Day 20.5% 1.6% 15.9% 18.6% 19.5% 20.4% 21.5% 22.7% 26.3% Table 12: Distribution of Unadjusted and Risk Adjusted Rates for Hospitals with 25 or More Admits 17

18 Evaluation of Hospital Risk Adjusted Readmissions Overall, the effect of risk adjustment methodology was to reduce the range and the variability of the original distribution. For 30-day readmissions, the range dropped from 50% to 34%, while the standard deviation dropped from 6% to 2%. For 90-day readmissions, the ranged dropped from 52.5% to 36.6% (Table 12). To estimate the confidence interval around hospital (a random intercept for HGLM) means, CMS used a bootstrap procedure because of complications with determining the standard error for the random effect in the HGLM model. Creating Confidence Intervals with Bootstrap Procedure The bootstrap procedure was adapted from a CMS SAS macro for repeated resampling of hospitals (second level data) with replacement Bootstrap programs were created separately for 30-day hospital RSRR and 90-day hospital RSRR. Any hospital with less than five inpatient admissions were excluded and resample size was increased to 6000 hospitals for each replication to minimize problems with the GHLM solution. Hospitals represent the hierarchical random effect and a random multiplier (to SE of random effect) was also computed during each replication replications each were run for 30-day and 90-day RSRR bootstrap - To eliminate extreme outliers (an artifact of bootstrapping), the 95% percentile for each overall bootstrap distribution was used as a cutoff value to trim the distributions. After the 5% trim, the bootstrap means were equivalent to the respective RSRR means of the original sample of hospitals with 25 or more admits (Table 13). RSRR Model Facilities Mean SD Min 25th Percentile 50th Percentile 75th Percentile Current 30-Day % 1.98% 5.6% 10.4% 11.3% 12.2% 35.9% Bootstrap 30-Day 4,036, % 5.61% 0.0% 7.3% 11.1% 15.3% 26.2% Max Current 90-Day % 2.68% 9.9% 18.9% 20.0% 21.1% 46.5% Bootstrap 90-Day 4,048, % 7.33% 0.2% 14.8% 20.0% 25.0% 37.5% Table 13: 30-day and 90-day RSRR Bootstrap Means Compared to Validation Production Sample of Hospitals with 25 or More Admits Hospital Specific CI. Each hospital had more than 2,900 values for RSRR. The 2.5 th and 97.5 th percentiles were extracted for each hospital distribution to obtain the lower confidence limit (LCL) and upper confidence limit (UCL) for the 95% CI. The average regional RSRR was computed from the original data for four United States regions: Central, Northeast, Southeast, and West. A hospital s confidence limits were evaluated against their average regional RSRR. A hospital received a passing score if one of two conditions were met: - 1) If the UCL is lower than the corresponding average regional RSRR, or - 2) If the average regional RSRR is within the confidence interval 18

19 Table 14 shows that the 95% CI versus the average regional RSRR passed 20% more hospitals than the regional median method for both the 30-day and 90-day RSRR. Due to the timeconsuming nature of the bootstrap procedure, the regional median method was retained. Measure Evaluation Frequency Percent 30-Day Readmission 90-Day Readmission Current Pass - Regional Median RSRR % CI Pass - Regional Average RSRR % CI Pass - National Average Readmission Current Pass - Regional Median RSRR % CI Pass - Regional Average RSRR % CI Pass - National Average Readmission Table 14: Evaluations Based on Hospitals Without and With 95% Confidence Interval Summary and Conclusions Based upon the CMS methodology for Hospital-Wide Risk-Standardized Readmission Metrics, a robust hierarchical general linear case-mix model for 30-day and 90-day readmission was created based on behavioral health admissions. The case-mix models allow for comparison across hospitals by adjusting for patient demographic and clinical characteristics for actual types of cases that it handles. RSRR metrics are incorporated with other measures (behavioral health cost, 7 and 30-day follow-up rates, peer review rate, and a length of stay metric) to assign a facility to a performance tier level. The highest tier rating has benefits such as increased recognition, more referrals, additional market support, and streamlined clinical review. REFERENCES CMS Hospital-Wide All-Cause Unplanned Readmission Measure: 2013 SAS Pack Software Documentation, Lin, Zhenqiu and Grady, Jacqueline N., Measure Updates and Specifications Report: Hospital-Wide All-Cause Unplanned Readmission Measure (Version 2.0). Submitted by Yale New Haven Health Services Corporation/Center for Outcomes Research and Evaluation (YNHHSC/CORE). Prepared For: Centers for Medicare & Medicaid Services (CMS), March Hospital-Wide All-Cause Unplanned Readmission Measure Final Technical Report. Submitted by Yale New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (YNHHSC/CORE): Horwitz, Leora et al., Prepared For: Centers for Medicare & Medicaid Services (CMS), July Medicare Hospital Chartbook Performance Report on Outcomes Measures, Centers for Medicare & Medicaid Services (CMS), September Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. Edited by American Psychiatric Association,

20 Behavioral Health Trends in the United States: Results from the 2014 National Survey on Drug Use and Health Report prepared for the Substance Abuse and Mental Health Services Administration (SAMHSA) by RTI International under Contract No. HHSS C with SAMHSA, U.S. Department of Health and Human Services (HHS). ACKNOWLEDGMENTS Thank you to Nghi Ly, Rachel Lu, Brent Bolstrom, Laura Ten Eyck, Wade Bannister, Charlotte Wu, Ronald Ozminkowski, and everyone who have worked with me on SAS and behavioral health outcome effectiveness. CONTACT INFORMATION Allen Hom, PhD Senior Research Analyst Consumer Solutions Group Healthcare Analytics 425 Market Street, San Francisco (415) SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. 20