Panel: Machine Learning in Surgery and Cancer

Transcription

1 Panel: Machine Learning in Surgery and Cancer Professor Dimitris Bertsimas, SM 87, PhD 88, Boeing Leaders for Global Operations Professor of Management; Professor of Operations Research; Co-Director, Operations Research Center Jack Dunn, PhD candidate at the MIT Operations Research Center George Velmahos, MD, PhD, Division Chief of Trauma, Emergency Surgery and Surgical Critical Care at MGH Daisy Zhuo, PhD candidate at the MIT Operations Research Center #MITSloanHSI

2

3

4

5

6

7

8

9

10

11

12

13

14 An Actionable Tool for Cancer Mortality Prediction Ying Daisy Zhuo, Operations Research Center, MIT Joint work with Dimitris Bertsimas, Ph.D, Jack Dunn, Colin Pawlowski, John Silberholz, Ph.D, Alex Weinstein, Ph.D, Eddy Chen, M.D., and Aymen Elfiky, M.D.

15 Cancer Mortality: To Treat or not Treat? Many terminal cancer patients are treated aggressively, despite high risk of short-term mortality Clinicians often over-estimate the prognosis Inaccurate estimates can precipitate decisions that lead to Increased (re)hospitalization Toxicities and lower quality of life Higher health care costs Need accurate prognostic model for better medical decisions

16 Existing Models

17 Existing Models Model Populati on Palliative Prognostic Score Terminal (PaP), exponential cancer multiple regression Palliative Performance Index Terminal (PPI), multiple cancer regression Memorial Sloan Metastati Kettering Cancer c prostate Centre nomograms, cancer accelerated failure after time model castration Cancer Prognostic Score, Cox regression model Intra-hospital Cancer Mortality Risk Model (ICMRM), multivariate logistic regression Terminal cancer Hospitali zed cancer patients Inoperabl Glasgow Prognostic e nonsmall-cell Score (GPS), Cox regression model lung cancer Objective Prognostic Score (OPS), Cox regression model Artificial neural networks Terminal cancer Nonsmall-cell lung cancer Support vector All machine ensemble cancers Bayesian network Artificial neural networks Breast cancer Breast and colorectal cancers Graph based semisupervised learning Breast cancer and others Decision trees and others Breast cancer Training Covariat Sample e Size Size Data Model AUC Reference Demographic s, cancer 36, status and Not reported selected 6 treatment, symptoms, labs Demographic 25, s, cancer Not reported selected 5 status, symptoms total Pirovano 1999 Morita 1999 Demographic s, symptoms, Not reported Smaletz 2002 labs Demographic s, cancer 26, status and Not reported Chuang selected treatment, symptoms Demographic s, cancer 0.82 (intrahospital 14, status and selected 5 treatment, mortality) symptoms, labs Bozcuk total Demographic s, cancer status and Not reported Forrest treatment, 2003 symptoms, labs Demographic s, cancer , status and selected 7 treatment, Not reported Suh 2009 symptoms, labs Demographic Thousand s, cancer 440 s of status and Not reported Chen variables, treatment, 2014 selected 5 gene expression Demographic s, cancer 0.76 (2-year status survival), total (including 0.80 (1-year Gupta receptor status survival), 2014 for breast 0.87 (6-month cancer survival) patients) Demographic clinical, s, cancer (predicting Gevaert 232 genes status, gene good 2006 expression prognosis) 5, Demographic 0.78/0.87 (5- (breast), (breast), s, cancer Burke 5, status and (colorectal ) (colorecta treatment, l) symptoms Demographic s, cancer status and treatment 40, total 202, total year survival: breast / colorectal) 0.81 (5-year survival) 1997 Kim 2013 Demographic s, cancer Not reported Delen status and 2005 treatment Issues Data: Small patient population Large scale registry data, but no detailed patient info Methods: ML method not suitable for the large scale Uninterpretable, no clinical meaning to outputs

18 Our Approach Data EHR of more than 23,000 patients at Dana Farber and Brigham Women from Predict 60-, 90-, 180-day mortality Detailed 401 variables on: Demographics Medical history Treatment history Lab tests Genetic mutations Novel longitudinal modeling Change in weight, etc. Method Data preparations: Optimal Missing Data Imputation * Demonstrated improvements in imputation quality and downstream tasks Predictions: Optimal Classification Trees Highly accurate Interpretable Facilitate discussion with oncologists Comparisons against Logistic regressions Regularized LR CART Gradient boosted trees * Bertsimas, Pawlowski, and Zhuo. From Predictive Methods to Missing Data Imputation: An Optimization Approach. In revision for submission. Bertsimas and Dunn. Optimal Classification Trees. Machine Learning, 2017: 1-44.

19 Decision trees are good fit for this health care application: Interpretable and transparent Treatment guidelines are structured in the format of decision trees Optimal Classification Trees Example treatment pathway for invasive breast cancer. NCCN Guideline Version

20 Optimal Classification Trees Decision trees are good fit for this health care application: Interpretable and transparent Treatment guidelines are structured in the format of decision trees Captures non-linear relationship across variables Figure. Mortality rates by weight change groups.

21 Optimal Classification Trees CART: greedy one-step growing procedure leads to complications: splits are only locally-optimal, overall tree could be far from optimal What if we could solve the entire decision problem at once to find globally optimal trees instead? To date, no globally optimal decision tree method is tractable and scalable to the typical problem sizes Current development by two co-authors provides a solution: Bertsimas and Dunn. Optimal Classification Trees. Machine Learning, 2017: 1-44.

22 Optimal Classification Trees has significant improvement over CART and competitive with random forest / XGBoost in many real world examples: Optimal Classification Trees Figure. Out-of-sample accuracy across 60 real world datasets.

23 Results

24 Tree for 60- day Mortality, Breast Cancer Tool available at:

25 Questionnaire for Physicians Tool available at:

26 Model Comparisons Optimal Classification Trees achieved one the highest performances in mortality predictions compared to other state-of-the-art methods

27 Optimal Classification Trees achieved one the highest performances in mortality predictions compared to other state-of-the-art methods Model Comparisons 60-day mortality 90-day mortality 180-day mortality Accuracy Logistic regression (fewer predictors) 94.6% 93.0% 83.4% Logistic regression 94.3% 92.8% 84.5% Regularized logistic regression 94.9% 93.1% 84.5% CART decision tree 93.6% 92.1% 85.0% Optimal Classification Trees 94.9% 93.3% 86.1% Gradient boosted trees 94.9% 93.6% 87.2% AUC Logistic regression (fewer predictors) Logistic regression Regularized logistic regression CART decision tree Optimal Classification Trees Gradient boosted trees

28 Summary We built an actionable tool for cancer mortality prediction. It makes a significant contributions to clinical oncology, as it is: Personalized and specific Interpretable and clinically meaningful Evidence based and data driven Actionable Validated and accurate Based on state-of-the-art machine learning

29 Machine Learning for Emergency Surgery Jack Dunn PhD student at the MIT Operations Research Center #MITSloanHSI

30 Overview Diverse nature of patients makes it difficult to predict risks arising from emergency surgery Existing methods for risk prediction rely on subjective data, are not interpretable, or have low accuracy Goal: Train highly accurate and interpretable predictive models using EHR data Our approach: Build Optimal Tree models to predict risks of mortality and 18 other surgical complications Significantly higher prediction quality than existing methods Highly interpretable: presented as interactive application for physicians #MITSloanHSI

31 Estimating Risk of Emergency Surgery Over 130 million emergency department visits in the US annually 27 million hospital admissions related to emergency surgery from Emergency surgery carries heightened risk of death and other complications Knowing the risk of post-surgery complications is critical for decision making by physicians, patients and families Understanding risk going into a procedure and comparing to actual outcome permits evaluation of hospitals, departments and physicians #MITSloanHSI

32 The Current State-of-the-art Many scoring mechanisms for estimating risk based on patient s demographic information, lab results, and co-morbidities Mostly based on logistic regression models Most comprehensive approach is the ACS-NSQIP Surgical Risk Calculator: Constructed using the ACS-NSQIP database of surgery patients Input all patient information and get probabilities of each complication #MITSloanHSI

33 ACS-NSQIP Risk Calculator #MITSloanHSI

34 ACS-NSQIP Risk Calculator #MITSloanHSI

35 Limitations of the ACS Calculator Black-box Model: Can t see how it works or understand the estimates One-size-fits-all Every factor is needed to make a prediction Each factor contributes to the risk additively Difficult to use Must specify the exact procedure code (often unknown beforehand) #MITSloanHSI

36 Our Approach Develop a model for predicting the risks of emergency surgery that: Provides better estimates than the ACS calculator Is interpretable and understandable Gives personalized predictions for each patient Is easy for physicians to use Use Optimal Trees, a novel method for constructing decision trees with state-of-the-art performance whilst maintaining interpretability #MITSloanHSI

37 Example Optimal Tree for Mortality #MITSloanHSI

38 Performance of Optimal Trees Risk of mortality AUC of compared to for ACS Calculator Risk of any complication AUC of compared to for ACS Calculator Risks of individual complications AUCs as high as #MITSloanHSI

39 Actionable Tool for Physicians Our models are interpretable and deliver state-of-the-art performance but still need to be communicated to physicians We designed and built an application for use by physicians Interactive questionnaire that calculates risk Questions adapt to previous answers as they go down the tree Integrate with EHR system to answer questions where possible #MITSloanHSI

40 Interactive Application #MITSloanHSI

41 Interactive Application #MITSloanHSI

42 Interactive Application #MITSloanHSI

43 Interactive Application #MITSloanHSI

44 Interactive Application #MITSloanHSI

45 Interactive Application #MITSloanHSI

46 Key Takeaways Optimal Trees is a general-purpose machine learning algorithm that is both accurate and interpretable Applying OCT to predict risks of emergency surgery improves significantly on the state of the art Most importantly, the model and predictions are understandable and interpretable: Allows delivery of risk predictions as an application suitable for everyday use by physicians #MITSloanHSI