USING ACS NSQIP TO PROVIDE SURGEON SPECIFIC OUTCOMES Rocco Ricciardi, MD, MPH Chief Scientific Officer Lahey Hospital & Medical Center Burlington, MA 01805
DISCLOSURES None
OBJECTIVES Surgeon specific reporting Data to use Has surgeon reporting helped
SURGEON SPECIFIC OUTCOMES Quality assessment and improvement Reporting in CR Surgery Biannual reports NSQIP outcomes* Institutional Data* SCIP measures *Similar variables *Which data? Variable 5 yr Mortality 2013 Mortality 1.3 (1.1-1.6) 1.3% A 2.2% 1.3% B 1.1% 0.6% C 1.2% 1.6% D 1.3% 2.8% E 0.9% 0% F 1.7% 0.8% G 0.6% 1.9% H X 0.7%
NSQIP American College of Surgeons National Surgical Quality Improvement Program 1 Clinical data Reports 30-day risk-adjusted morbidity and mortality 1 Identifies risk adjusted outcomes at institutional, service level, and procedural level 2 Ideally suited for quality assessment and improvement Improved surgical outcomes in participating hospitals 3 Institute of Medicine named NSQIP Best In The Nation? Value for surgeon specific outcome reporting 1 Fink et al. Ann Surg 236, 344-353; 2002 2 Cima et al. Surgery 150, 943-949, 2011 3 Hall et al. Ann Surg 250, 363-376, 2009.
INSTITUTIONAL DATA Prospective 100% capture data source Clinical data Reports 30-day morbidity and mortality outcomes Not risk-adjusted NSQIP outcomes Other department initiatives Includes inpatient and outpatient data Valuable in QI projects Time consuming and expensive Do we really need 100% capture data?
ASSESSMENT OF SURGEON OUTCOMES -Objective I- Compare NSQIP & Institutional Data Evaluate the comparability of NSQIP and 100% capture institutional quality data to provide individual surgeons with information regarding outcomes
METHODS Compared datasets for departmental & physician reporting 7 surgeons active for entire 5 year study period (1/2008 through 12/2012) NSQIP site-specific data Calculated aggregate departmental & surgeon-specific adverse event rates 1 and 5-year Institutional quality data Quality assessment data on all patients undergoing an operative procedure in CRS (100% capture) Calculated aggregate departmental & surgeon-specific adverse event rates 1 and 5-year Dis Colon Rectum 2015;58:247.
NSQIP DEFINITIONS NSQIP definitions of 30-day adverse events: Mortality Reoperation Urinary tract infection Deep vein thrombosis Pneumonia Superficial site infection Organ space infection Dis Colon Rectum 2015;58:247.
DATA ANALYSIS Aggregate departmental adverse event rates with 95% confidence intervals for each of seven complications for both datasets 1. Assess comparability of datasets Are the departmental/service line rates similar 2. Assess whether the two datasets categorize individual surgeons similarly as outliers for each adverse event Comparing surgeon rates with aggregate point estimates with 95% confidence intervals for each adverse event for each dataset Dis Colon Rectum 2015;58:247.
RESULTS: VOLUME/SURGEON CHARACTERISTICS Institutional Data Total aggregate cases over five years 6459 Inpatient 4173 (65%) Emergent 1297 (20%) Average annual case number per surgeon 184.5 Range of annual case numbers per surgeon 53-378 NSQIP Data Case volume sampled by NSQIP over five years 1786 Average number of annual cases sampled by NSQIP per surgeon 51 Range of annual case number per surgeon sampled by NSQIP 9-115 % of case volume sampled by NSQIP 28% Surgeon A B C D E F G Aggregate % inpatient 62 73 69 57 67 53 64 65 % emergent 25 20 21 22 20 15 13 20 % of surgeon cases sampled by NSQIP 24 28 29 22 34 28 35 28 Dis Colon Rectum 2015;58:247.
COMPARABILITY OF DATASETS Annual Departmental Results Calculated annual aggregate adverse event rates for entire group of surgeons for both datasets Annual 100% capture institutional adverse event rates were compared to the NSQIP 95% confidence intervals Mortality ROR UTI DVT PNM SSI OSI 2008 1.1 4.0 3.6 0.5 1.9 3.5 3.0 2009 1.4 3.0 5.4 1.1 2.1 4.2 2.3 2010 1.7 3.3 4.3 0.6 2.1 4.3 2.5 2011 1.1 2.9 4.0 0.9 1.5 5.0 1.8 2012 1.3 2.3 2.1 0.4 1.2 5.1 1.9 Key Institutional rate is within NSQIP 95% CI Institutional rate is below NSQIP 95% CI Institutional rate is above NSQIP 95% CI Dis Colon Rectum 2015;58:247.
SURGEON OUTLIER STATUS Compared individual surgeon outlier status for each adverse event category within the two datasets Concordance = two datasets classify an adverse event rate similarly Low/low Average/average High/high Discordance = two datasets disagree with one designating average Low/average Average/high Gross discordance Low/high
SURGEON RESULTS (ANNUAL) Compared annual outlier status determined by the two datasets for: 7 Surgeons x 7 Adverse event categories 5 Years 245 Comparisons (35 Comparisons per surgeon) Surgeon A B C D E F G Aggregate group % Concordance 46 46 60 66 51 40 51 51 % Gross Discordance 3 3 9 9 6 9 20 8 Dis Colon Rectum 2015;58:247.
SURGEON RESULTS (5-YEARS) Compared 5-year outlier status determined in each dataset for: x 7 Surgeons 7 Adverse event categories 49 Comparisons 29 agreement (59% concordance) 33% discordance 8% gross discordance Dis Colon Rectum 2015;58:247.
SURGEON RESULTS (5-YEARS) Individual surgeons Complication A B C D E F G 95% CI for Aggregate Mortality Institution 2.2 1.1 1.2 1.3 0.9 1.7 0.6 1.1-1.6 NSQIP 2.8 0.9 0.6 0.4 0.8 2.7 0.9 0.8-1.9 ROR Institution 4.4 4.1 2.2 1.3 2.7 3.4 2.8 2.7-3.5 NSQIP 4.1 4.4 3.2 0.4 2.7 6.8 1.8 2.8-4.5 UTI Institution 3.5 4.1 2.7 3.1 3.6 5.2 6.6 3.4-4.3 NSQIP 5 3.7 4.8 3 3.8 7.7 6.3 3.8-5.7 DVT Institution 0.7 1 0.6 0.6 0.7 0.6 0.3 0.5-0.9 NSQIP 0.9 0.5 1.9 1.3 1.5 1.4 0.9 0.8-1.8 Pneumonia Institution 2.4 1.1 1.6 1.7 2.5 1.9 0.9 1.4-2.1 NSQIP 6 1.6 3.2 3 1.1 4.5 1.8 2.2-3.8 SSI Institution 4.8 3.5 6.2 5.6 4.3 2 5.3 4.0-5.0 NSQIP 10 3.3 7.4 7.2 5.7 3.2 4.5 4.8-7.0 OSI Institution 3.2 2.5 2.1 1.1 2.2 2.2 3.1 1.9-2.6 NSQIP 4.6 3.3 4.2 1.7 2.3 4.1 1.8 2.5-4.2 Dis Colon Rectum 2015;58:247. Low outlier Average High outlier Gross discordance
SUMMARY Each surgeon could improve Good/bad years Agreement between NSQIP and 100% capture institutional data at the institutional/departmental level Overlap of results Poor concordance of data for individual surgeon outcomes Annual results poorest We needed the 100% capture data for surgeon reporting
SURGEON SPECIFIC REPORTING -Objective II- Are We Getting Better Evaluate the effect of surgeon feedback on Process measure adherence Overall surgical outcomes Process measures SCIP measures Removal of foley DVT prophylaxis Antibiotic administration Institutional measures Anastomotic technique Anastomotic leak testing Outcome measures UTI DVT Presented At ASCRS May 2015 Surgical Site Infection Anastomotic leak
PATIENT POPULATION 08-09 10-11 12-13 p Value n 903(N)-2408(I) 669(N) 2694(I) 518(N)-2873(I) Age 56+17 55+17 56+17 NS Male Sex 50% 51% 49% NS Laparoscopy 28.7% 27.7% 29.0% NS Hand-Assist 42% 38% 33% 0.002 Presented At ASCRS May 2015
HAVE WE IMPROVED Variable 08-09 10-11 12-13 p Value DC Urinary Catheter 73% 88% 100% 0.01# UTI 5.5% 4.2% 1.9% 0.01 DVT Prophylaxis 99% 100% 100% NS# DVT/PE 1.1% 1.3% 1.2% 0.5 Timely Abx 71% 96% 100% 0.01# SSI 5.3% 6.7% 4.8% 0.3 #SCIP data
HAVE WE IMPROVED Variable 08-09 10-11 12-13 p Value Hand-Sewn Rights 19% 17% 2% 0.0001* Leak Testing 88% 95% 95% 0.0008* Diversion 19.5% 17% 17% NS Leak 5.2% 4.2% 2.9% 0.05* *Institutional data
SUMMARY Nearly 100% adherence to SCIP process measures of antibiotic administration did not lead to outcome gains for SSI Improvement in process of care for urinary catheter removal and leak testing were associated with improvements in UTI and overall leak Our strategy of biannual surgeon-specific feedback of outcomes with improved adherence to particular processes of care led to an overall improvement in surgical quality
CONCLUSIONS Biannual surgeon-specific feedback was valuable All surgeons were identified as a high outlier in at least one adverse event category Demonstrating the need for continuous quality improvement? Value of Hawthorne effect Providing process measures was helpful Estimates of surgeon-specific outcomes were critical but it is unclear how to best measure them Procedure specific Include larger samples Different outcomes