Interpretability of Sudden Concept Drift in Medical Informatics Domain

Transcription

1 Interpretability of Sudden Concept Drift in Medical Informatics Domain Gregor Stiglic, Peter Kokol Faculty of Health Sciences University of Maribor Slovenia

2 Presentation Outline Visualization of Concept Drift National Hospital Discharge Data Data Pre-processing Experimental Settings Results Conclusions

3 Visualization of Concept Drift Changes in data distribution or in the concept of the predicted class. (Tsymbal, 2004) Very few research papers dealing with concept drift visualization or visual interpretation Mostly visual data exploration techniques for: Multivariate streams visualization or Univariate time-series anomaly detection K.B. Pratt, and G. Tschapek, Visualizing concept drift, KDD Using brushed parallel histograms

4 National Hospital Discharge Data Hospital discharge records for approximately 1% of US hospitals 10 consecutive years from 2000 to 2009 (approx hospitalizations/year) Altogether 3,106,176 hospitalization events We used 2,509,113 events (after removing nonadults)

5 National Hospital Discharge Data Each NHDS record contains the: Personal characteristics of the patient (age, gender, race, and marital status); Administrative information (length of stay, discharge status, etc.); Medical information Up to 7 diagnoses (optional admitting diagnosis) and Up to 4 surgical and nonsurgical procedures).

6 Data Pre-processing The ICD-9-CM coding of diagnoses 5-digit codes were collapsed to 3-digit codes For example: Flu is assigned to category 487, Influenza 487.0, Influenza with pneumonia 487.1, Influenza with other respiratory manifestations etc. Altogether digit diagnosis codes

7 Data Pre-processing ID Month Sex Age D038 D585 D M 25 A A A A 2 1 F 37 A P A A 21,343 2 F 65 P A A P 21,344 2 M 77 A A A A 21,345 2 M 23 A P A A 2,509, F 81 A A A A Class D585

8 Data Pre-processing Sparse matrix 1187 codes ID Month Sex Age D038 D585 D M 25 A A A A 2 1 F 37 A P A A 21,343 2 F 65 P A A P 21,344 2 M 77 A A A A 21,345 2 M 23 A P A A 2,509, F 81 A A A A Class D585

9 Data Pre-processing Sparse matrix 1187 codes ID Month Sex Age D038 D585 D M 25 A A A A 2 1 F 37 A P A A 21,343 2 F 65 P A A P 21,344 2 M 77 A A A A 21,345 2 M 23 A P A A 2,509, F 81 A A A A Class D585

10 Human Disease Network Metrics We use two co-morbidity measures for visualization: Relative risk RR ij = C ijn M i M j Phi-correlation ij = C ij N M i M j M i M j (N M i )(N M j )

11 Experimental Settings Performance comparison of Static vs. Dynamic Ensemble of Naïve Bayes classifiers Static: trained only on the first 12 months of data, does not change after that Dynamic: updated with each new batch of incoming patients (i.e. each month) 25 updateable NB classifiers are built using Random Spread Subsample (balanced instance sampling) Simple majority voting was used

12 Experimental Settings Performance metrics observed over 119 months AUC (effective for imbalanced classes) F-measure (integration of precision and recall) Own implementation of prequential evaluation in Weka was used Each batch (month) of data is first used for testing, before it can be used for training

13 FEB-00 JUL-00 DEC-00 MAY-01 OCT-01 MAR-02 AUG-02 JAN-03 JUN-03 NOV-03 APR-04 SEP-04 FEB-05 JUL-05 DEC-05 MAY-06 OCT-06 MAR-07 AUG-07 JAN-08 JUN-08 NOV-08 APR-09 SEP-09 Results AUC (D585 Chronic kidney disease) D_585 AUC Static D_585 AUC Dynamic

14 Visualization How to explain the concept drift? Using motion charts: -correlation and relative risk measures w.r.t. the attribute class, Chi-square value or significance (p-value), Morbidity (support) for different diseases, Time. Only disease codes with support over 100 were visualized.

15 Visualization

16 Relative Risk Visualization

17 Visualization Relative Risk Phi-correlation

18 Visualization Relative Risk Phi-correlation Chi-square p-val

19 Visualization Relative Risk Phi-correlation Chi-square p-val Prevalence

20 Visualization Relative Risk Phi-correlation Chi-square p-val Prevalence Time

21 Visualization August 2005

22 Visualization Diagnosis code D401 D403 D404 D428 D583 D584 D585 D588 Description Essential hypertension Hypertensive chronic kidney disease Hypertensive heart and chronic kidney disease Heart failure Nephritis and nephropathy, not specified as acute or chronic Acute renal failure Chronic kidney disease (CKD) Disorders resulting from impaired renal function August 2005

23 Visualization October 2005

24 Visualization October 2005

25 Conclusions D403 and D404 stand out Further examination uncovers change of coding in October 2005 D403: Hypertensive renal disease (Sep 2005) -> Hypertensive chronic kidney disease (Oct 2005) Possible explanation: The new code names brought more attention to D403 and D404 in Oct 2005 and resulted in more accurate coding.

26 Conclusions Visualization can help in interpretation of concept drift events. Possible improvements: Additional variables for visualization of significant changes in variable values ( movement p-values ) Educational use (visualization of all classifiers in an ensemble of classifiers) Integration into MOA instead of Weka + Google Motion Charts implementation

27 Questions Gregor Stiglic Demo visualization available at: