Intent for these sessions. At this 10 year mark, several general sessions reviewing the evolution and present status of our statistical methods

Transcription

1 Intent for these sessions At this 10 year mark, several general sessions reviewing the evolution and present status of our statistical methods

2 Intent for these sessions Goals: Conceptual understanding of NSQIP statistical methods: Models and profiling with sufficient understanding to explain to others Working knowledge of SAR reports

3 Intent for these sessions NSQIP participant s world Data Benchmarking Reports Cleaning Imputation Eligibility 3 Pillars Patient Expectations Risk Adjustment Shrinkage Modeling Risk adjust Smooth NSQIP statistical world

4 Intent for these sessions Accessible (really?) In ACS NSQIP context Topics will be revisited Mastery is not essential Should be something for everyone

5 Intent for these sessions Polling question: Describe your relationship with ACS NSQIP statistical methods 1. I m confident about statistics and enjoy exploring NSQIP methods 2. I have had some success in understanding NSQIP methods 3. It is often a struggle, but I m slowly building my understanding of NSQIP methods 4. Statistics is the bane of my existence

6 A (Brief) Quantitative History of ACS NSQIP July 25, 2014 Mark E. Cohen, PhD Continuous Quality Improvement American College of Surgeons

7 Quantitative History Why we started The inherent value of knowing your comparative standing on a quality metric (rates, adjusted rates, O/E ratios, odds ratios) Where we started Building off the VA design, 2004 What has changed Bigger: more programs; more hospitals; more cases; more models; more statistical power Better: more stable definitions; more powerful predictors; more informative outcomes; more compelling methodologies; special applications

8 Quantitative History Second Semiannual Progress Report July1, 2004 Jun 30, sites, 8 models, table of O/E ratios This adult SAR, 517 sites, 384 models, site summary tables, bar plots This pediatric SAR, 64 sites, 40 models

9 Quantitative History Big data opportunities Modeling success for tenuous groupings/outcomes More accurately estimated parameters More predictors Novel predictors (CPT linear risk) Target-specific variables The models are better

10 Quantitative History PUF datasets Innovation Highlights Modeling method: Logistic O/E > hierarchical OR & logistic with smoothing Adjustment for patient: ~40 Common variables > surgery-specific variables (Targeted) Adjustment for procedure: RVU > 9-level CPT category > CPT bucket > CPT linear risk (ID-based reports) O/E tables > caterpillars > (site-specific reports) site summaries and bar plots Better control for PATOS Models defined by: Self-declared surgeon specialty > CPT code & surgeon specialty Case occurrence reports (model selection filters) PEDs SAR ISAR Collaboratives reports Risk calculator On-demand (real time): projected odds ratios > rates

11 Quantitative History Does ACS NSQIP Work? Most hospitals in NSQIP improve (clinical data) Numerous case studies with dramatic findings But hospitals need to use their data enrollment can t possibly be sufficient Hospitals can improve without NSQIP QI approaches are ubiquitous In our opinion, ACS NSQIP Represents the most fully-developed QI data program in surgery Increases quality faster than secular trend alone (appropriate research conceptualization and design are critical)

12 Quantitative History Take away NSQIP statistical methods are getting better and contributing to the quality improvement climate Large datasets are helpful

13 Data integrity: Consistent definitions; complete and accurate data; consequences of bad data July 25, 2014 Mark E. Cohen, PhD Continuous Quality Improvement American College of Surgeons

14 Data Integrity Hospital profiling requires statistical models Models are only as good as the data they are built on

15 Data Integrity SCRs are the essential source of quality data Consistent definitions preclude bias idiosyncratic definitions, even if clinically justified, are unacceptable Must guard against over-or under-calling of comorbidities Must guard against over- or under-calling of events No discretion in deciding if events are related to procedure (ROR) Common specifications must trump clinical perfection Common specifications must trump contrary opinion Unbiased case selection - without knowledge of outcomes Must not misplace that troublesome case

16 Data Integrity Conforming data Statistical Inference Hospital (with respect to all hospitals) Redefine co-morbidities Redefine adverse events Lose problem cases Biased Estimate of Hospital Nonconforming Data Statistical Inference Hospital

17 Data Integrity But perfection not required Weighted sampling (arranged with NSQIP) is permitted with risk adjustment for CPT code Some ability to compensate for random errors Redundancy among predictors within patients errors in outcomes balanced between patients Reliable conclusions can be drawn from samples. Large hospitals don t need larger samples.

18 Data Integrity

19 Data Integrity Sample size doesn t affect validity However, larger samples are more reliable but with diminishing returns A simple example SEp= [p(1 -p)/n] N? for p=0.05 and n=100, SE = [0.05(1 0.05)/100] = As n, SE Standard Error of Proportion (P=0.05) Sample Size The standard error (SE) is the standard deviation of the sampling distribution of a statistic.

20 Data Integrity Polling question: Which of the following is acceptableunder NSQIP data integrity requirements? 1. Using a non-nsqip, but clinically better, definition for pneumonia 2. Not reporting a death within 30-days that was clearly unrelated to the surgery (e.g., homicide) 3. Not reporting on a selected and eligible case because of an intra-operative event 4. Submitting a sample of cases that is only a small fraction of all cases done at the hospital

21 Data Integrity Take away Detailed conformance to NSQIP definitions and rules matters If those rules are followed, there is little bias A small amount of random error is inconsequential For practical purposes, samples are sufficiently informative

22 Models and statistical adjustment: SAR variables; logistic models and patient and procedure mix; hierarchical models and shrinkage July 29, 2014 Mark E. Cohen, PhD Continuous Quality Improvement, American College of Surgeons

23 Models What is a model? Models are representations of reality But they are not reality All models are wrong, but some are useful y = x A statistical model provides a mathematical description of the estimated relationship between variables (predictors and outcomes)

24 A very simple model 9 8 Score on Outcome Variable Score on Predictor Variable We observe a relationship between a predictor and an outcome Models will not fit perfectly: Noise Absolute linearity unlikely Lack of control for other variables A sample is not the population But we can still take advantage of the information present We can eyeball the relationship, or use statistical machinery to get a least-squares fit

25 A very simple model Score on Outcome Variable y = x Line is chosen to minimize the sum of squared differences of points from the line hence least squares The model is just a mathematical equation Score on Predictor Variable I This is a simple model When we fit the line, we are modeling the data with an equation This model has 2 parameters -the 2 parameters needed to define a line The intercept The slope For any value on the predictor scale, we have a value on the outcome scale

26 ACS NSQIP Models Our models are more complex Many predictors for risk adjustment Logistic regression (for probabilities) is a bit more complicated than ordinary linear regression The process is the same - just more computationally intensive Fit the data. Estimate the parameters.

27 ACS NSQIP Models Selection Predictor Parameter Step Estimate 0 Intercept Outcome-Specific CPT Risk ASA Class (2 vs. 1) ASA Class (3 vs. 1) ASA Class (4-5 vs. 1) Pre-operative Albumin Inpatient Surgery (Inpatient vs. Outpatient) History of COPD Pre-operative Sepsis(SIRS vs. None) Pre-operative Sepsis(Sepsis vs. None) Pre-operative Sepsis(Septic Shock vs. None) Creatinine > Bmi_class (Class1Obese vs. Normal) Bmi_class (Class2Obese vs. Normal) Bmi_class (Class3Obese vs. Normal) Bmi_class (Overweight vs. Normal) Bmi_class (Underweight vs. Normal) Race_class (American Indian or Alaska Native vs. White) Race_class (Asian vs. White) Race_class (Black or African American vs. White) Race_class (Native Hawaiian or Pacific Islander vs. White) Race_class (Unknown vs. White) Age Current Smoker Surgical Specialty (Cardiac Surgery vs. General Surgery) Surgical Specialty (Gynecology vs. General Surgery) Surgical Specialty (Neurosurgery vs. General Surgery) Surgical Specialty (Orthopedics vs. General Surgery) Surgical Specialty (Otolaryngology vs. General Surgery) Surgical Specialty (Peripheral Vascular vs. General Surgery) Surgical Specialty (Plastics vs. General Surgery) Surgical Specialty (Thoracic vs. General Surgery) Surgical Specialty (Urology vs. General Surgery) Work RVU Functional Status (Partially Dependent vs. Independent) Functional Status (Totally Dependent vs. Independent) Emergency Surgery Bleeding Disorders 0.18 Here is the intercept y = x Intercept = Slope parameter = Here are the parameters (actually 39)

28 Models to benchmarking (risk adjustment) This sequence 1. Build a mathematical model, from a very large data set, that relates risk factors to an outcome 2. Use the model to make risk-adjusted predictions for each patient 3. Sum the predictions for a group patients (at a hospital) 4. Sum the actual events for these patients 5. Compare these sums to yield the benchmarking metric forms the underlying logic of quality benchmarking methods From 600,000 cases we build a model that predicts mortality based on patient factors and procedure performed. For your 1,600 cases, this model predicts 8 deaths (the sum of all 1,600 predicted probabilities = 8); there are actually 14 deaths. There are more deaths than risk-adjusted expectations.

29 Models to benchmarking (risk adjustment) Contemporaneous paradigm- Quarterly SAR/ISARS Very large number of patients, Dozens of predictors Predicted Events Actual Events II Mathematical model (equation) User-based Application Historical Paradigm On-demand or real time Applied to your Patients Profiling depends on comparing model-predicted events to actual events

30 Model Metrics Different benchmarking metrics (numbers are illustrative only) Total sample: 100,000 cases and 4,800 events Global rate = #events/#total = 4,800/100,000 = (4.8%) Odds = # events/# non-events = 4,800/95,200 = Your sample: 1,000 cases and 40 events Observed rate = 40/1000 = (4%) Odds = # events/# non-events = 40/960 = Risk-adjusted: Estimated number = 43; odds = 43/957 = O/E ratio = 40/43 = 0.93 Doing better than expected (1.0) Odds ratio = / = 0.89 Doing better than expected (1.0) Rates: 0.93 X 4.800% = 4.46% Doing better than expected (4.8%)

31 Confidence Intervals Is amount of better meaningful? We need a confidence interval Estimates are based on samples Samples have sampling variability Can this amount of better be explained by chance? We can identify a range of reasonably true values (a 95% CI) And then decide if: The range of reasonable values for your O/E ratio includes 1.0 The range of reasonable values for your odds ratio includes 1.0 The range of reasonable values for your rate includes the grand average rate

32 Confidence Intervals If the CI does not overlap the reference metric (1.0 for O/E and OR, or what the population rate is), then the hospital is a statistical outlier. Low outlier (good) Exemplary High outlier (bad) Needs Improvement Not an outlier As Expected At 1.0 if O/E or odds ratio At global rate if rate W

33 Deciles We also incorporate decile status. Smallest 10% - 1 st first decile Largest 10% - 10 th decile Hospital odds ratios ordered from smallest to largest We declare Exemplary if you are either a low outlier or in the first decile and Needs improvement if you are either a high outlier or in the 10 th decile. The inclusion of decile provides early warning, but there is a downside. For models with small n and low event rate you could have 1 event, be in the 10 th decile and Needs improvement. Must consider context.

34 ACS NSQIP Models So statistical models: Reflect relationships between predictors and outcomes Can be used to risk adjust expectations for the patient and to benchmark hospitals taking into account those expectations More details why risk adjustment is necessary and how it is implemented

35 Why is a statistical model needed? ACS NSQIP needs to provide fair comparisons of surgical quality across hospitals. Since every patient, and every hospital s patient pool and procedure mix are different, we need to compensate for those differences. Hospital A: Smaller operations on healthier patients Hospital B: Bigger operations on sicker patients Hospital B has a higher raw mortality rate is it an inferior hospital?

36 ACS NSQIP Modeling: The patient-level factors Patient attributes Hospitals patients differ in age, general health, comorbidities, laboratory values, etc. Some patients are sicker than others this is the classic focus of risk adjustment. Common important variables (among 40) are: Age, ASA Class, Functional Status, Albumin, Emergent, (pre-operative) Sepsis.

37 ACS NSQIP Modeling: The patient-level factors Procedures Hospitals differ in the complexity/risk profile of surgeries that they perform. Our two surgery-specific variables are: RVU and CPT based risk. RVU: Relative value units From CMS from AMA from expert panel Mostly: [time, effort, expertise] and [practice expense & overhead] Very small part: liability insurance CPT: we use many years of SAR data to assign linear risk, to each outcome, for every primary CPT CPT risk is the most powerful risk-adjusting variable in ACS NSQIP

38 ACS NSQIP Modeling: Hospital factors We don t adjust for hospital factors: Teaching vs not, research vs not, rural vs urban, large vs small, safety-net vs affluent, underfunded vs well-funded By not adjusting for these, we avoid setting different standards with enough hospital-level adjustments there would be no hospital-level quality differences An active area of concern/investigation (CMS, NQF) how to balance health care policy and care-equity interests with fairness in profiling especially in public reporting and P4P arena

39 ACS NSQIP Models We want to risk adjust for patients and procedures, to arrive at fair evaluations of each hospital Modeling options: 1. Logistic modeling 2. Hierarchical (logistic) modeling Accounts for hierarchical data structure Implemented with shrinkage adjustment inherent in modeling 3. Logistic modeling With post-modeling shrinkage (discussed tomorrow)

40 ACS NSQIP Models: The logistic approach Build a model that estimates the probability of an event, for each patient, at every hospital, using all available, useful patient-level information Stepwise selection emphasize predictive value clinical face validity a non-essential luxury (not the best strategy to identify clinical contributors) For each patient we knowthe Observed (O) = 1 or 0 The logistic model allows use to get the Expected (E) which is between 1 and 0

41 ACS NSQIP Models: The logistic approach For each hospital, for each model (case eligibility), sum all the Osand all the Esfor their patients Hospital Patient Observed Expected A A A A A A n SUM O/E = 10 Works reasonably well but there are limitations

42 ACS NSQIP Models: The logistic approach When sample sizes are small or event rates low, the O/E ratio is unstable Six cases submitted in previous example: O/E = 1/0.1 = 10 but, with luck, it could have been O/E = 0/0.1 = 0 neither 10 nor 0 is reasonable More reasonable values might be 1.02 if one event had occurred, and 0.98 if no event had occurred

43 ACS NSQIP Modeling: Shrinkage (hierarchical) When sample sizes are small, we have very little actual information about the hospital. Therefore, we combine information we have for the hospital with what we know about all hospitals. This pooling of information will move the hospital s value towards that of the average hospital. With shrinkage adjustment, fewer hospitals are assigned extreme/unreasonable values. The estimates are stabilized or smoothed. Seems like we re making stuff up, but smoothed estimates are more accurate

44 Effects of Shrinkage Colorectal Morbidity for 61 Hospitals Providing <= 50 cases 2.0 III O/E or OR These estimates will be closer to the truth than O/E= When N >150, The lines approach horizontal Logistic O/E Modeling Method and Metric Hierarchical OR

45 ACS NSQIP Models: The hierarchical approach Hierarchical modeling yields risk-adjusted and shrinkageadjusted(stabilized, smoothed) odds ratios The reported odds ratio is shifted to compensate for low and high risk patients, and low and high risk procedures (risk adjustment) The odds ratio will be shifted towards 1.0 (the expected value if you re like the average hospital) the smaller the sample size, the greater the shift (shrinkage adjustment)

46 ACS NSQIP Modeling: Shrinkage (hierarchical) Polling question: For a particular model, my hospital submitted 10 cases. There were 0 events for these cases. Which is true? 1. We will be Exemplary as, with 0 events, we must be a statistical low outlier 2. We will be Exemplary as, with 0 events, we must be in the in the first decile 3. We might be As expected being neither a statistical low outlier, nor in the first decile

47 ACS NSQIP Modeling: Shrinkage (hierarchical) A shrinkage paradox: I m perfect but I m not Exemplary and in the 4 th decile You have 0 events for 10 cases submitted and are As Expected Sample size too small to be a low statistical outlier Model-based shrinkage moves OR from 0.0 towards 1.0 (to 0.9) Other hospitals also have 0 events but submitted more cases and have less shrinkage toward 1.0 estimates are < 0.9 If there are many of hospitals with 0 events and larger sample sizes, your OR may be in the 4 th decile Strange but explainable

48 Other applications In tomorrow s adult statistics session we will see how similar modeling machinery is used for two important applications Universal Risk Calculator On-demand risk-adjusted and smoothed rate application This is where the 3 rd modeling approach (logistic modeling with post-modeling shrinkage is used)

49 What does The SAR profiling result tell us? Polling question: A hospital s odds ratio for All Cases morbidity in a SAR is below 1.0. Which is most true? 1. The hospital is doing better than the average hospital 2. The hospital is doing worse than the average hospital 3. The hospital is improving over time 4. The hospital is a good choice for a particular patient

50 What does The profiling result tell us? Good versus getting better Each SAR/ISAR models data for a 12-month period Benchmarking is relative to other hospitals in the program during that period SAR/ISAR addresses good -not directly whether getting better Cut morbidity rate in half from 2013 to 2014, but if everyone did, your OR would not change Evaluate raw rates or use On-demand

51 What does The profiling result tell us? Good versus good for a particular patient Given our types of patients and procedure mix, are we doing better or worse than the average ACS NSQIP hospital doing the same procedures on the same patients. This is valuable information; given what we do, how are we doing? But, a patient s perspective might be which hospital offers me the best outcomes for myoperation. This requires an extra-modeling stratification -limiting comparisons of eligible hospitals to those judged to do the required procedure on patients with this risk profile. Also valuable information, but different; given what needs to be done, who should do it?

52 What does The profiling result tell us? OR=0.8 OR=1.2 It really depends on if you re a or a or a

53 What does The profiling result tell us? Public reporting can be inconsistent in differentiating between these perspectives; lay persons may not understand the issue Highly targeted models help to ameliorate the problem As ACS NSQIP models become finer-grained (from ALL CASES to Subspecialty to Targeted), we are comparing more similar procedures across hospitals -so there will be less concern about the for what we do restriction. Focused models move us closer to answering the given what we do, how are we doing and given what needs to be done, who should do it questions simultaneously.

54 ACS NSQIP With that appreciation of model focus, we will move on to the definition and construction of ACS NSQIP models