Supersparse Linear Integer Models for Interpretable Prediction. Berk Ustun Stefano Tracà Cynthia Rudin INFORMS 2013

Transcription

1 Supersparse Linear Integer Models for Interpretable Prediction Berk Ustun Stefano Tracà Cynthia Rudin INFORMS 2013

2 CHADS 2 Scoring System Condition Points Congestive heart failure 1 Hypertension 1 Age 75 years 1 Diabetes mellitus 1 Prior Stroke ortia orthromboembolism 2 If Total Points 4 è Predict Stroke

3 What Makes CHADS 2 Interpretable? Condition Points Congestive heart failure 1 Hypertension 1 Age 75 years 1 Diabetes mellitus 1 Prior Stroke or TIA or Thromboembolism 2 Few Terms è Sparse Limited Coefficient Values è Meaningful Sign-constrained Relationships è Intuitive Predictions Without Computers è Easy-to-Use

4 State of Interpretable Classification Less Accurate Medical Scoring Systems Logistic Regression LARS Decision Rules Decision Trees More Accurate Support Vector Machines Random Forests More Interpretable Less Interpretable

5 Today Methodology Designing a Model That is Accurate and Interpretable Performance Comparison to State-of-the-Art Classification Models Medical Application mammo Criminology Application violentcrime

6 Today Methodology Designing a Model That is Accurate and Interpretable Performance Comparison to State-of-the-Art Classification Models Medical Application mammo Criminology Application violentcrime

7 Supersparse Linear Integer Models Setup Linear Model Error Occurs When y ŷ apple 0 y sign( T x) apple 0 y T x apple 0

8 Supersparse Linear Integer Models Objective User-Defined Coefficients Meaningful Coefficients & Ease-of-Use 0-1 Loss L 0 Norm L 1 Norm Accuracy Sparsity Meaningful Coefficients & Ease-of-Use

9 MIP Formulation Given N examples and P features N + 3P variables 2N + 4P constraints Solvable for N ~ and P ~ 100 in minutes

10 Today Methodology Designing a Model That is Accurate and Interpretable Performance Comparison to State-of-the-Art Classification Models Medical Application mammo Criminology Application violentcrime

11 State of Interpretable Classification Less Accurate Medical Scoring Systems Logistic Regression LARS Decision Rules Decision Trees More Accurate Support Vector Machines Random Forests More Interpretable Less Interpretable

12 Performance Goal Compare the accuracy and sparsity of SLIM and other classification models on UCI datasets Models SLIM SVM Random Forest LARS Lasso LARS Ridge LARS Elastic Net CART C5.0 Tree C5.0 Rule Datasets breastcancer internetad spambase haberman mammo tictactoe

13 Accuracy vs. Sparsity for breastcancer

14 SLIM vs. LARS Lasso for breastcancer 25% SLIM Lasso 5 Fold CV Test Error 20% 15% 10% 5% 0% Fold Median L 0 Norm

15 SLIM Model for breastcancer Linear Model Scoring System

16 Today Methodology Designing a Model That is Accurate and Interpretable Performance Comparison to State-of-the-Art Classification Models Medical Application mammo Criminology Application violentcrime

17 Overview of mammo Predict whether a mammographic mass lesion is malignant (Class = +1) N = 961 Examples and P = 11 Features Patient Based (i.e. Age) Cell Characteristics (i.e. Shape, Density)

18 Linear Classifiers for mammo SLIM Test Error: 21.5% Lasso Ridge Test Error: 22.9% Test Error: 22.4%

19 Tree Classifiers for mammo C5.0 Tree SLIM no Is the margin circumscribed? yes no Is the shape oval? yes no Is the shape oval? yes benign no Is the margin circumscribed? yes benign no Is the patient over 60 y.o? yes benign malignant benign no Is the shape irregular? yes malignant benign malignant Test Error: 20.0% Test Error: 21.5%

20 Today Methodology Designing a Model That is Accurate and Interpretable Performance Comparison to State-of-the-Art Classification Models Medical Application mammo Criminology Application violentcrime

21 Overview of violentcrime Predict if a young person raised in out-ofhome care will commit a violent crime over the next 3 years (Class = +1) N = 558 Examples and P = 108 Features Imbalanced (Only 19% of Class = +1)

22 SLIM for Imbalanced Datasets Balanced Objective Imbalanced Objective Error Rate for Positive Outcomes Error Rate for Negative Outcomes

23 SLIM Performance on violentcrime Sensitivity = # True Positives = 69% # Positive Outcomes Specificity = # True Negatives = 44% # Negative Outcomes

24 SLIM Model for violentcrime

25 Conclusions Interpretability is important when models are Designed to be used by humans Used to yield insights for data mining SLIM balances accuracy and interpretability

26 Appendix

27 SLIM Generalization Bound

28 SLIM vs. LARS for mammo 50% 45% SLIM Lasso 5 Fold CV Test Error 40% 35% 30% 25% 20% Fold Median L 0 Norm

29 SLIM vs. LARS for haberman 22% SLIM Lasso 5 Fold CV Test Error 20% 18% 16% Fold Median L 0 Norm

30 SLIM vs. LARS for internetad 8% SLIM Lasso 5 Fold CV Test Error 6% 4% 2% 0% Fold Median L 0 Norm

31 SLIM vs. LARS for spambase 30% SLIM Lasso 5 Fold CV Test Error 25% 20% 15% 10% 5% Fold Median L 0 Norm

32 SLIM vs. LARS for tictactoe 25% SLIM Lasso 5 Fold CV Test Error 20% 15% 10% 5% 0% Fold Median L 0 Norm

33 Accuracy vs. Sparsity: haberman

34 Accuracy vs. Sparsity: mammo

35 Accuracy vs. Sparsity: internetad

36 Accuracy vs. Sparsity: spambase

37 Accuracy vs. Sparsity: tictactoe

38 Computational Performance for breastcancer 100 % 6 % 5 80 % 5 % MIP Gap 60 % 40 % L 0 Norm Fold CV Test Error 4 % 3 % 20 % 1 2 % 0 % Runtime (Minutes) Runtime (Minutes) Runtime (Minutes)

39 Computational Performance for breastcancer 100 % 6 % 5 80 % 5 % MIP Gap 60 % 40 % L 0 Norm Fold CV Test Error 4 % 3 % 20 % 1 2 % 0 % Runtime (Minutes) Runtime (Minutes) Runtime (Minutes)

40 Computational Performance for haberman 100 % 30 % 5 80 % MIP Gap 60 % 40 % L 0 Norm Fold CV Test Error 24 % 18 % 20 % 1 12 % 0 % Runtime (Minutes) Runtime (Minutes) Runtime (Minutes)

41 Computational Performance for internetad 100 % 20 MIP Gap 80 % 60 % 40 % 20 % L 0 Norm Fold CV Test Error 6 % 5 % 4 % 3 % 0 % Runtime (Minutes) Runtime (Minutes) Runtime (Minutes)

42 Computational Performance for mammo 100 % 24 % 80 % 9 MIP Gap 60 % 40 % 20 % L 0 Norm Fold CV Test Error 20 % 0 % Runtime (Minutes) Runtime (Minutes) % Runtime (Minutes)

43 Computational Performance for spambase 100 % 29 9 % 80 % MIP Gap 60 % 40 % 20 % L 0 Norm Fold CV Test Error 8 % 7 % 0 % Runtime (Minutes) Runtime (Minutes) 6 % Runtime (Minutes)

44 Computational Performance for tictactoe 100 % % MIP Gap 80 % 60 % 40 % 20 % L 0 Norm Fold CV Train Error 15 % 10 % 5 % 0 % Runtime (Minutes) Runtime (Minutes) 0 % Runtime (Minutes)