Trading off coverage for accuracy in forecasts: Applications to clinical data analysis

Transcription

1 Trading off coverage for accuracy in forecasts: Applications to clinical data analysis Michael J Pazzani, Patrick Murphy, Kamal Ali, and David Schulenburg Department of Information and Computer Science University of California Irvine, CA {pazzani, pmurphy, ali, schulenb}@ics.uci.edu Research supported by Air Force Office of Scientific Research Grant, F J-0430 AIM-94 Thursday, June 30,

2 Inductive Learning of Classification Procedures Given: A set of training examples a. Attribute-value pairs: { (age, 24) (gender, female)... } b. A class label: pregnant Create A classification procedure to infer the class label of an example represented as a set of Attribute-value pairs Decision Tree Weights of neural network Conditional probability of a class given an attribute Rules Rule with confidence factors Typical evaluation of a learning algorithm: Divide available data into a training and test set Infer procedure from data in training set. Estimate accuracy of procedure on data in the test set. AIM-94 Thursday, June 30,

3 Trading off coverage for accuracy Learners usually infer the classification of all test examples Give learner ability to say I don t know on some examples Goal: Learner is more accurate when it makes a classification. Possible applications: Human computer interfaces: Learning Macros Learning rules to translate from Japanese to English Analysis of medical databases - Let learner automatically handle the typical cases - Refer hard cases to a human specialist Evaluation: T- Total number of test examples P- Number of examples for which the learner makes a prediction C- Number of examples whose class is inferred correctly = C P = P T AIM-94 Thursday, June 30,

4 0.94 Trading off coverage for accuracy Lymphography Backprop Activation AIM-94 Thursday, June 30,

5 Goals of this research Modify learning algorithms to trade off coverage for accuracy Learners typically have an internal measure of hypothesis quality Use hypothesis quality measure to determine whether to classify Experimental evaluate trading off coverage for accuracy on databases from UCI Archive of Machine Learning Databases Train on 2/3rds Test on remaining 1/3. Averages over 20 trials. Breast Cancer (699 examples; benign from malignant tumors) Lymphography (148 examples; identify malignant tumors) DNA Promoter (106 examples; Leave-one-out testing) Describe how a sparse clinical database (diabetes data sets) can be analyzed by classification learners. AIM-94 Thursday, June 30,

6 Neural Networks One output unit per class. An output units activation is between 0 and 1. Assign an example to the class with the highest activation. Fever Bloodshot eyes Pregnant Headache Nausea Swollen Glands Gender Cancer Age AIM-94 Thursday, June 30,

7 Trading off coverage for accuracy in Neural Networks 1. Assign an example to the class with the highest activation provided that that activation is above a threshold. 2. Assign an example to the class with the highest activation provided that that the difference between that activation and the next highest is above a threshold. (Didn t make a significant difference in our experiments) AIM-94 Thursday, June 30,

8 0.74 Breast Cancer Backprop Activation 0.9 AIM-94 Thursday, June 30,

9 0 Promoter Backprop Activation AIM-94 Thursday, June 30,

10 Bayesian Classifier An example is assigned to the class that maximizes the probability of that class, given the example. If we assume features are independent P(C i A 1 =V 1j &...A n =V nj ) = P(C i ) Estimate from training data: P(C i ) P C i A k =V kj k P C i A k =V kj P(C i ) Trading off coverage for accuracy: (Like backprop) Only make prediction if P(C i A 1 =V 1j &...A n =V nj ) is above some threshold. AIM-94 Thursday, June 30,

11 4 Breast Cancer Bayesian Classifier ln(probability) AIM-94 Thursday, June 30,

12 A Decision Tree (for determining suitability of contact lenses) No 15n 0h 0s Tears Reduced Normal Age Prescription Astigmatic Prescription Soft Hard < >55 Hyper Myope Yes No Hyper Myope Soft Leaf nodes assign classes (n =no, h =hard, s = soft) Different leaves can be more reliable. No 1n 1h 3s 0n 1h 0s 1h 3s 5n 1h Hard 1n 3h 2s Astigmatic Yes No Soft 0n 0h 3s No 2n 1h 1s AIM-94 Thursday, June 30,

13 Trading off coverage for accuracy in decision trees Estimate the probability that an example belongs to some class given that it classified by a particular leaf Two possibilities: Divide training data *learning set *probability estimation set *unbiased estimate of probability, but not most accurate tree Estimate probability from training data * Use Laplace estimate of probability of class given leaf p(class = i) = k + N i +1 k j N j 3 soft, 1 hard, 0 none P(soft) = 4/7 AIM-94 Thursday, June 30,

14 Breast Cancer ID Maximum Probability 0.2 AIM-94 Thursday, June 30,

15 First Order Combined Learner Learns a set of first order Horn Clauses (Like Quinlan s FOIL) no_payment_due(p) :- enlisted(p, Org) & armed_forces(org). no_payment_due(p) :- longest_absence_from_school(p,a) & 6 > A & enrolled(p,s,u) & U > 5. no_payment_due(p) :- unemployed(p). Negation as failure Selects literal that maximizes information gain p p 1 log 1 p 2 p 1 +n -log p 0 +n 0 Averaging Multiple Models Learn several different rules sets (stocastically select literals) Assign example to the class predicted by the majority of rule sets Trading off coverage for accuracy Only make prediction if at least k of the rules sets agree AIM-94 Thursday, June 30,

16 0.72 Breast Cancer FOCL Number of Voters AIM-94 Thursday, June 30,

17 0 Promotor FOCL Number of Voters AIM-94 Thursday, June 30,

18 Learns a contrasting set of rules HYDRA no_payment_due(p) :- enlisted(p, Org) & armed_forces(org). [LS = 4.0] no_payment_due(p) :- longest_absence_from_school(p,a) & 6 > A & enrolled(p,s,u) & U > 5. [LS = 3.2] no_payment_due(p) :- unemployed(p). [LS = 2.1] payment_due(p) :- longest_absence_from_school(p,a) & A > 36 [LS = 2.7] payment_due(p) :- not (enrolled(p,_,_)) & not (unemployed(p)) [LS = 4.1] Attaches a measure of reliability to clauses (logical sufficiency) ls ij = p(clause ij(t) = true t class i ) p(clause ij (t) = true t class i ) Assigns example to the class of satisfied clause with the highest logical sufficiency Trading off coverage for accuracy: Only make prediction if at ratio of logical sufficiency is greater than a threshold AIM-94 Thursday, June 30,

19 Breast Cancer HYDRA log(ls Ratio) AIM-94 Thursday, June 30,

20 Analysis of the diabetes data sets with classification learners : Pre-breakfast blood glucose : Regular insulin dose : NPH insulin dose : Pre-lunch blood glucose : Regular insulin dose : Pre-supper blood glucose : Regular insulin dose : Unspecified blood glucose Problems with applying machine learning classifiers: 1. There is not a fixed, small number of classes 2. The data isn t divided into a fixed number of attributes 3. We know very little about medicine, diabetes, blood glucose If you have hammer, everything looks like a nail: 1. Predict whether a blood glucose is above mean for the patient 2. Create attributes and values from coded data 3. Come to AIM-94 and be willing to learn AIM-94 Thursday, June 30,

21 Converting the diabetes data set into attribute value format Current glucose measurement: (above or below). CGT: Current glucose time: (in hours) numeric. CGM: Current glucose meal: (unspecified, breakfast, lunch, super, or snack). CGP: Current glucose period: (unspecified, pre, or post). LGV: Last glucose measurement: numeric. ELGV: Elapsed time since last glucose measurement: (in hours) numeric. LGM: Last glucose meal: (unspecified, breakfast, lunch, super, or snack). LGP: Last glucose period: (unspecified, pre, or post). ENPH: Elapsed time since last NPH insulin: (in hours) numeric. NPH: Last NPH dose: numeric. EREG: Elapsed time since last regular insulin: (in hours) numeric. LREG: Last regular dose: numeric. Ran experiments with patients 20 and 27. Trained on 450, tested on PRE BREAKFAST 7.17 PRE LUNCH Below 80 PRE LUNCH 2.83 PRE SUPPER Below 101 UNSPEC UNSPEC 59.0 PRE BREAKFAST Above AIM-94 Thursday, June 30,

22 Backpropagation results 3 (Patient 27) (Patient 27) Activation AIM-94 Thursday, June 30,

23 FOCL results 1 (Patient 27) (Patient 27) Votes FOCL AIM-94 Thursday, June 30,

24 Decision tree results 8 (Patient 27) (Patient 27) Maximum Probability AIM-94 Thursday, June 30,

25 Example of Rule learned by FOCL ABOVE if ENPH 12.0 & LGV 131 & ENPH < 24.0 & CGM SUPPER ABOVE if LGV 132 & LREG < 6.5 & CGT 23.0 ABOVE if ELGV 6.5 & LGV < 130 & LGV 121 ABOVE if ELGV < 56.0 & LGV < 83 & ENPH 24.0 ABOVE if LGV 163 & CGP = UNSPECIFIED & LGV < 181 ABOVE if LGV 131 & LGV < 147 & CGM = LUNCH ABOVE if ENPH 12.0 & LGV 131 & CGT 8.0 & LGV < 142 ABOVE if ELGV 6.5 & LGV 191 & ELGV < 10.5 ABOVE if ELGV 6.5 & CGT 8.0 & LGV < 90 ABOVE if LGV 96 & LGV < 118 & ELGV 4.5 & ENPH 14.5 ABOVE if LGV 128 & ENPH 5.0 & ELGV < 5.5 & LGV < 147 ABOVE if ENPH 5.0 & LGV 189 & ELGV < 4.0 ABOVE if LREG 7.5 & ENPH < 11.5 & LGV < 147 ABOVE if LGV 128 & ELGV 33.5 & CGT < 7.5 Above if at lease 5 hours have elapsed since last NPH insulin Last glucose maesurement was above 189 It s been less than 4 hours since last measurement AIM-94 Thursday, June 30,

26 Conclusions Experimentally evaluated trading off coverage for accuracy in machine learning classifiers Rather than forcing problems to be classification problems, and important issue is to identify new classes of learning problems: Different goals Different example representations We also do research in: Reducing cost of misclassification errors Knowledge-guided induction AIM-94 Thursday, June 30,