Computer Models for Medical Diagnosis and Prognostication

Transcription

1 Computer Models for Medical Diagnosis and Prognostication Lucila Ohno-Machado, MD, PhD Division of Biomedical Informatics Clinical pattern recognition and predictive models Evaluation of binary classifiers (calibration) Ethical implications for clinical practice Source: DOE

2 Computer- Based Models for Medical Diagnosis and Prognostication Lucila Ohno-Machado, MD, PhD Division of Biomedical Informatics University of California San Diego

3 Risk Assessment Popular risk calculators Gail Model (Breast cancer) Framingham Risk Calculator (CVD) APACHE (ICU mortality) Use of individual estimates Prophylaxis for breast cancer Cholesterol management guidelines Continuation of life support

4

5

6

7 22%

8 16%

9 APACHE II Mortality in intensive care units (ICUs) 12 physiologic predictors

10

11 Individualized Genome How many individual genotypes are needed to predict disease?

12 Logistic Regression p i = ( β + β x 1+ e 1 0 i i ) p=1 pi log 1 p i = β 0 + β x i i x x

13 Coronary Angioplasty and Stenting

14 Risk of death in angioplasty National average of deaths after angioplasty is 2%, which is stated in the informed consent. "Informed consent and good clinical practice require a discussion of risks and benefits Alexander et al, 52th ACC meeting Less than 10% of the patients have an estimated risk of death around 2%. Are we lying to the other 90%?

15 Dataset: Attributes Collected History Presentation Angiographic Procedural Operator/Lab age acute MI occluded number lesions annual volume gender primary lesion type multivessel device experience diabetes rescue (A,B1,B2,C) number stents daily volume iddm CHF class graft lesion stent types (8) lab device history CABG angina class vessel treated closure device experience baseline creatinine cardiogenic shock ostial gp 2b3a antagonists unscheduled case CRI failed CABG dissection post ESRD rotablator hyperlipidemia atherectomy angiojet max pre stenosis max post stenosis no reflow Data Source: Medical Record Clinician Derived Other Resnic et al, J Am Col Card 2001; Matheny et al, J Biomed Inf 2005

16 Study Population: Descriptive Statistics Development Set Validation Set Cases 2,804 1,460 Women 909 (32.4%) 433 (29.7%) Age > 74yrs 595 (21.2%) 308 (22.5%) Acute MI 250 (8.9%) 144 (9.9%) Primary 156 (5.6%) 95 (6.5%) Shock 62 (2.2%) 20 (1.4%) Class 3/4 CHF 176 (6.3%) 80 (5.5%) gp IIb/IIIa antagonist 1,005 (35.8%) 777 (53.2%) p=.066 p=.340 p=.311 p=.214 p=.058 p=.298 p<.001 Death 67 (2.4%) 24 (1.6%) Death, MI, CABG (MACE) 177 (6.3%) 96 (6.6%) p=.110 p=.739

17 Multivariate Models Logistic Regression Model Odds p-value Ratio Age > 74yrs B2/C Lesion Acute MI Class 3/4 CHF Left main PCI IIb/IIIa Use Stent Use Cardiogenic Shock Unstable Angina Tachycardic Chronic Renal Insuf Prognostic Risk Score Model beta coefficient Risk Value Artificial Neural Network

18 Logistic Regression, Score, and Neural Networks 1.00 Validation Set: 1460 Cases Sensitivity LR Score ann ROC = 0.50 ROC Area LR: Score: ann: Specificity

19 Risk Score of Death Unadjusted Overall Mortality Rate = 2.1% % Number of Cases % Number of Cases 26% Mortality Risk 21.5% 53.6% 50% 40% 30% 20% 12.4% % 0.4% 1.4% 2.2% 2.9% 1.6% 1.3% 0 to 2 3 to 4 5 to 6 7 to 8 9 to 10 >10 Risk Score Category 10% 0%

20

21 External Validations for CVD Models Model/Cohort AKA Year Published External Validations Framingham Risk Score FRS Dawber et al, Framingham Risk Score FRS Kannel et al, Framingham Risk Score FRS Anderson et al, Glostrup Glostrup Schroll et al, European Society of Cardiology ESC Pyorala et al, Framingham Risk Score FRS Wilson et al, Framingham Risk Score for ATP III FRS ATP III ATP III, Framingham Risk Score FRS D Agostino et al, UK Prospective Diabetes Study UKPDS Stevens et al, Framingham Point System FPS ATP III, Prospective Cardiovascular Munster Study PROCAM Assman et al, Finnish Diabetes Risk Score FINDRISC Lindstrom et al, Systematic Coronary Risk Evaluation SCORE Conroy et al, Diabetes Epidemiology: Collaborative Analysis of Diagnostic Criteria in Europe ASSessing cardiovascular risk using SIGN guidelines DECODE Balkau et al, ASSIGN SIGN, Total 108

22 Predicted / Observed Framingham models tested on European populations

23 Predicted / Observed Framingham models tested on European populations European models tested on North American populations

24 Questions Which model is right? True probability would be the gold-standard What is the true probability? Are the models adequate in discrimination and calibration?

25 Your Risk this program shows the estimated health risks of people with your same age, gender, and risk factor levels x p=1

26 this means that 5 of 100 people with this level of risk will have a heart attack or die

27 Patients like you Input space Output space people with your same age, gender, and risk factor levels me people with this level of risk

28 Patients like you height me gender

29 Patients like you

30 Patients like you height 1 me 0 1 gender

31 Patients like you height risk me gender

32 Evaluation of Predictive Models Error Discrimination Area under ROC Calibration Plot of groups: observed vs expected Hosmer-Lemeshow statistic

33 Discrimination of Binary Outcomes Estimate and Observed outcome ( gold standard, true ) Estimate True Classification into category 0 or 1 is based on thresholded estimates (e.g., if estimate > 0.5 then consider positive )

34 threshold normal Disease True Negative (TN) FN FP True Positive (TP) 0 e.g

35 nl D Sens = TP/TP+FN Spec = TN/TN+FP nl TN FN PPV = TP/TP+FP D FP TP NPV = TN/TN+FN - + Accuracy = TN +TP nl D

36 Sensitivity = 50/50 = 1 Specificity = 40/50 = 0.8 threshold nl nl 40 D 0 40 D nl disease TN TP FP

37 Sensitivity = 40/50 =.8 Specificity = 45/50 =.9 threshold nl nl 45 D D nl disease TN TP FN FP

38 Sensitivity = 30/50 =.6 Specificity = 1 threshold nl nl 50 D D nl disease TN TP FN

39 Threshold 0.7 Threshold 0.4 nl D nl D Threshold 0.6 nl D nl D Sensitivity ROC curve nl D nl D Specificity

40 1 ROC curve Specificity 1 Sensitivity All Thresholds

41 Areas Under the ROC curve concordance index measure adequacy of risk ranking (#concordant pairs + ½ #ties) /all pairs do not measure adequacy of risk estimates (collective nor individual) Discordant Pairs

42 Predicted / Observed Framingham models tested on European populations

43 Calibration D = 0 Measures how close the average estimate is to the observed proportion Goodness-of-fit Hosmer- Lemeshow statistics

44 Calibration D = 0 D = 1 HL = 1 3 D= 0 l= 1 π Dl o Dl π Dl 2

45 Risk Score of Death Unadjusted Overall Mortality Rate = 2.1% % Number of Cases % Number of Cases 26% Mortality Risk 21.5% 53.6% 50% 40% 30% 20% 12.4% % 0.4% 1.4% 2.2% 2.9% 1.6% 1.3% 0 to 2 3 to 4 5 to 6 7 to 8 9 to 10 >10 Risk Score Category 10% 0%

46 Interventional Cardiology Models Validation 5278 patients from BWH ( ) (external validation set) Comparisons use Areas under the ROC curve (AUC) and the Hosmer-Lemeshow goodness-of-fit statistic (deciles)

47

48 Calibration Are predictions obtained from external models good for individual counseling?

49 APACHE II Mortality in intensive care units (ICUs) 12 physiologic predictors

50 Summary of all comparison studies in terms of discrimination (AUC)

51 Summary of HL-GOF H and C statistics. X ² values and degrees of freedom are listed where available, p values are listed otherwise.

52 Standardized mortality ratio in different study comparisons

53 3 1.4 x Cluster 1, Y=0 Cluster 1, Y=1 Cluster 2, Y=0 Cluster 2, Y=1 Cluster 3, Y=0 Cluster 3, Y=1 Cluster 4, Y=0 Cluster 4, Y=1-0.4 x1 Simulated data set Clusters 1 to 4 centered on (0,0), (0,1), (1,0), and (1,1), with Gaussian noise True probability for clusters 1 to 4: 0.01, 0.40, 0.60, and 0.99

54 LR x Cluster 1, Y=0 Cluster 1, Y=1 Cluster 2, Y=0 Cluster 2, Y=1 Cluster 3, Y=0 Cluster 3, Y=1 Cluster 4, Y=0 Cluster 4, Y=1 x x1-0.4 x1 Simulated data set Clusters 1 to 4 centered on (0,0), (0,1), (1,0), and (1,1), with Gaussian noise True probability for clusters 1 to 4: 0.01, 0.40, 0.60, and 0.99

55 x ANN Cluster 1, Y=0 Cluster 1, Y=1 Cluster 2, Y=0 Cluster 2, Y=1 Cluster 3, Y=0 Cluster 3, Y=1 Cluster 4, Y=0 Cluster 4, Y=1-0.4 x1

56 40 Calibration Plot LR Neural Network 35 Sum of squared errors Mean squared error Expected Neural Network Logistic Regression Cross-entropy error Mean cross-entropy error Sum of residuals Mean residual AUC Observed HL-C p Proportion of events Average estimate Neural Net Average estimate Logistic Reg Clusters LR Neural Network Cluster 2 min (GS: 0.4) Cluster 2 max Cluster 3 min (GS: 0.6) Cluster 3 max.85.73

57 Genotype genome transcription RNA transcriptome Source: DOE Protein translation proteome Phenotype physical exam, imaging Physiology tests Metabolites laboratory

58 Will we ever achieve individualized risk assessment? If so, how can we evaluate it? Acknowledgments NIH, Komen Foundation Fred Resnic, Michael Matheny