Critical Appraisal of Risk Adjusted Analysis and Public Reporting of Outcomes in Cardiac Surgery

Transcription

1 Critical Appraisal of Risk Adjusted Analysis and Public Reporting of Outcomes in Cardiac Surgery University of Ottawa Heart Institute Jean Yves Dupuis, MD, FRCPC Cardiac Division of Anesthesiology

2 Disclosure #1 I do not have and none of my immediate family members have a significant interest or other relationship with the manufacturer(s) ) of any of the product(s) ) or provider(s) ) of any of the service(s) presented at this meeting.

3 Disclosure #2 Despite my critical evaluation of the concepts of risk adjusted mortality (RAM) and report cards,, I am a strong supporter of those concepts. Although imperfect, those concepts represent a formal and transparent mean of looking at clinical outcomes. Society and health care providers must learn how to use that knowledge intelligently.

4 QUIZ Pair of patients # 1 Which of these 2 patients died? Patient A 55 yrs old male Class 2 angina Normal LV No other disease Elective CABG x 3 Patient B 83 yrs old male MI < 24 hours Cardiogenic shock on dobu. + dopamine Creatinine:230µmol/L Emergent CABG x 3

5 Pair of patients # 2 Which of these 2 patients died? Patient A 72 yrs old male Unstable angina on I.V. NTG Pulmonary edema MI < 24 hours LVEF = 20% Diabetes mellitus Chronic AF Emergent CABG x 3 Patient B 52 yrs old male Class 2 angina Normal LV Controlled hypertension Elective CABG x 4

6 Pair of patients # 3 Which of these 2 patients died? Patient A 82 yrs old female Class 3 angina Controlled HTN Normal LV 3 vessels disease Aortic stenosis: valve area 0.6 cm 2 Elective CABG x 3 &AVR Patient B 62 years old female Class 3 angina Controlled HTN Normal LV 3 vessels disease Elective CABG x 3

7 Pair of patients # 4 Which of these 2 patients died? Patient A 68 yrs old male Class 4 angina on I.V. heparin Controlled HTN Normal LV No other disease Urgent CABG x 4 Patient B 68 yrs old male Class 4 angina on I.V. heparin Controlled HTN Normal LV Calcified LM coronary: unable to stent Porcelain asc.. aorta Urgent OPCAB x 3 converted to CABG

8

9 Pair of patients # 4 Which of these 2 patients died? Patient A 68 yrs old male Class 4 angina on I.V. heparin Controlled HTN Normal LV No other disease Urgent CABG x 4 Patient B 68 yrs old male Class 4 angina on I.V. heparin Controlled HTN Normal LV Calcified LM coronary: unable to stent Porcelain asc.. aorta Urgent OPCAB x 3 converted to CABG

10 Pair of patients # 5 Which of these 2 patients died? Patient A 75 yrs old male Class 4 angina Multiple PCI + Stents of RCA, LAD and LCX Occl.. LAD & LCX stents LVEF = 50% Urgent CABG x 3 + LAD & Ccx stentectomy Patient B 75 yrs old male Class 4 angina LVEF 50% Controlled HTN Urgent CABG x 3

11 Public Registry & Report Cards Administration Practice analysis Development of guidelines Quality assurance Compare results Risk management Improve Outcome

12 Mortality After CABG Surgery: New York State, Mean: 4.9% Mortality (%) hospitals Hannan EL, et al JAMA 1990;264:

13 Post CABG Mortality: Ontario, Mean: 3.01% Mortality (%) Hospitals Tu JV, Naylor CD. Circulation 1996;94:

14 Doctors Response to Public Dissemination of Cardiac Surgery Results First mortality report comparing hospitals W. Farr, Registrar General Office, London, England Med Times Gazette, February 13, 1864:187 «Only fools would fail to account for differences in patient characteristics»

15 Improving Outcome: Risk Management Theory Risk Analysis Document Improve Outcomes Risk Consumers Management Assess Providers & Modify Process Health Plans Cardiologists/Surgeons Identify Risk Factors Quantify Risk Monitor & Public Report Compare Observed/Predicted Event Rates

16 Risk Adjusted Mortality (RAM) OM: : Observed Mortality EM: : Expected Mortality sum of probabilities of dying for each patient, divided by the total number of patients RAM = OM/EM per hospital (or surgeon) x overall mortality in tested centres

17 Risk Adjusted Mortality (RAM) Report Cards RAM = OM Good RAM < OM Very good RAM > OM Bad

18 New Jersey 2004 Consumer Report for CABG Surgery

19 Pennsylvania s Guide to CABG Surgery

20 Pennsylvania s Guide to CABG Surgery

21 Pennsylvania s Guide to CABG Surgery

22 Cornerstone of the Risk Management Theory: Predicting the Probability of Death e Z /(1 + e Z ) e = natural logarithm = Z = B 0 + B 1 X 1 + +B 19 X 19 B 0 = Intercept or model constant B 1 B 19 = regression coefficient for each X risk factor X = 1 if risk factor is present X = 0 if risk factor is absent

23 The New Jersey 2004 Risk Model for CABG Surgery

24 Discrimination Analysis Receiver Operating Characteristic (ROC) Curve Score Died Survived 100% % Sensitivity 60% 40% Sensitivity = 20% 1 Specificity = 0% 1 Specificity 100%

25 Discrimination Analysis Receiver Operating Characteristic (ROC) Curve Score Died Survived 100% % Sensitivity 60% 40% Sensitivity = 95% 20% 1 Specificity = 70% 0% 1 Specificity 100%

26 Discrimination Analysis Receiver Operating Characteristic (ROC) Curve Score Died Survived 100% % Sensitivity 60% 40% Sensitivity = 95% 20% 1 Specificity = 70% 0% 1 Specificity 100%

27 Discrimination Analysis Receiver Operating Characteristic (ROC) Curve Score Died Survived 100% % Sensitivity 60% 40% Sensitivity = 80% 20% 1 Specificity = 43% 0% 1 Specificity 100%

28 Discrimination Analysis Receiver Operating Characteristic (ROC) Curve Score Died Survived 100% % Sensitivity 60% 40% Sensitivity = 50% 20% 1 Specificity = 19% 0% 1 Specificity 100%

29 Discrimination Analysis Receiver Operating Characteristic (ROC) Curve Score Died Survived 100% % Sensitivity 60% 40% 20% Area under ROC = % 1 Specificity 100%

30 Discrimination Receiver Operating Characteristic Curve Sensitivity (True positive rate) ROC Curve 0.50 useless poor fair good 0.90 excellent 1 Specificity (False positive rate) Swets JA. Science 1988; 240:

31 What does area under ROC mean? % of correct predictions Area under ROC curve = Probability of rightly identifying the patient with the outcome in a pair of randomly selected patients from different risk categories, always one with the outcome and the other without, on successive trials

32 Check your area under ROC Curve Pair #1: Pair #2: Pair #3: Pair #4: Pair #5: B B A B A

33 Pair of patients # 2 Which of these 2 patients died? Patient A 72 yrs old male Unstable angina on I.V. NTG Pulmonary edema MI < 24 hours LVEF = 20% Diabetes mellitus Chronic AF Emergent CABG x 3 Patient B Died 52 yrs old male Class 2 angina Normal LV Controlled hypertension Elective CABG x 4

34 Discrimination Multiple Logistic Regression vs Clinical Judgment Multiple Log. Regression Clinical Judgment Mortality Area under ROC Curve Morbidity Postop.. LOS Pons JMV, et al. Ann Thorac Surg 1999;67: Dupuis JY, et al. Anesthesiology 2001;94:

35 Discrimination Multiple Logistic Regression vs Clinical Judgment Multiple Log. Regression Clinical Judgment Mortality Area under ROC Curve Morbidity Postop.. LOS Pons JMV, et al. Ann Thorac Surg 1999;67: Dupuis JY, et al. Anesthesiology 2001;94:

36 Unaccounted Variables Pair of patients # 4: Same RAM? Patient A 68 yrs old male Class 4 angina on I.V. heparin Controlled HTN Normal LV No other disease Urgent CABG x 4 Patient B 68 yrs old male Class 4 angina on I.V. heparin Controlled HTN Normal LV Calcified LM coronary: unable to stent Porcelain aorta Urgent OPCAB x 3

37 Change in Medical Practice Pair of patients # 5: Same RAM? Patient A 75 yrs old male Class 4 angina Multiple PCI/Stents Occl.. LAD & Ccx stents LVEF = 50% Urgent CABG x 3 + LAD & Ccx stentectomy Patient B 75 yrs old male Class 4 angina LVEF 50% Controlled HTN Urgent CABG x 3

38 RAM and Multiple PCI No previous PCI Single PCI Multiple PCI Adjusted OR: 3.0 (95% CI: ) for mortality 2.3 (95% CI: ) for morbidity Mortality Major morbidity Thielmann M, et al. Circulation 2006;114:I 441 7

39 Do Report Cards Influence Patients Decision (Consumers)? Selecting a Surgeon 784 CABG patients in Pennsylvania (June Dec. 1996) 12 % aware of Consumer Guide < 1% knew correct rating of their hospital and surgeon obstacles for 66% patients: Short time window for decision making Limited awareness of alternatives Schneider EC, Epstein AM. JAMA 1998;279:

40 Do Report Cards Influence Cardiologists Practice? Referring Patients to Surgeons Pennsylvania 612 cardiologists 13%: influence on choice of surgical consultant < 10% discuss RAM data with patients Schneider EC, et al. New Engl J Med 1996;335:251 6 New York 455 cardiologists 32%: influence on choice of surgical consultant < 10% discuss RAM data with patients Hannan EL, et al. Am Heart J 1997;134:1120 8

41 % Surgeons Do Report Cards Influence Cardiac Surgeons Decisions? Accepting High Risk Patients (85 Cardiac Surgeons in Pennsylvania) Much less Less No change More Much more Schneider EC, et al. New Engl J Med 1996;335:251 6

42 Impact on Cardiac Surgeons Career Jha AK, Epstein AM. Health Aff 2006;23:844 55

43 Do Report Cards Influence Outcome? CABG Experience: NYS versus USA 1992 All USA Hospitals NY Sate Hospitals Steinbrook R. N Engl J Med 2006;355:1847 9

44 Report Cards and Mortality Trends Canada Guru V. et al. Am Heart J 2006;152:573 8

45 Report Cards and Mortality Trends Canada Montreal Heart Insitute Tremblay N, et al. Can J Anesth 1993;40: Cartier R, et al. J Thorac Cardiovasc Surg 2000;119:221 9 Guru V. et al. Am Heart J 2006;152:573 8

46 Report Cards and Mortality Trends Canada British Columbia Pate GE, et al. Can J Cardiol 2006;22: Guru V. et al. Am Heart J 2006;152:573 8

47 Can Report Cards Harm Patients? Decision Making in High Risk Patients 40 Parsonnet Europe Japan Ottawa Mortality % > 19 Parsonnet Score points Nashef SAM, et al. Eur J Cardio thorac Surg 2001; 817 (N = 6213 patients) Kawachi Y, et al. Eur J Cardio thorac Surg 2001; 961 (N = 803 patients) Dupuis JY, et al. Anesthesiology 2001; 94:194 (N = 3548 patients)

48 Refusing High Risk Patients on the Basis of Report Cards Wrong Decision for 2 reasons 1. Greatest potential of improving care 2. Larger # of high risk patients favors better RAM

49 Take Home Messages Public Registry and Report Cards 1. Based on complex predictive models not better than clinical judgment 2. Risk of unfair comparisons 3. Impact on mortality trend: questionable 4. Risk of inappropriate decision making 5. Opportunity to share data & follow evolution of clinical practice (Benchmark) 6. Not a replacement for good training, local quality control and clinical trials