CMP: Data Mining and Statistics Within the Health Services 19/02/2010

Transcription

1 CMP: Data Miig ad Statistics Withi the Health Services 19//1 Data Miig ad Statistics Withi the Health Services Cotet Data Miig Esemble for Predictig Osteoporosis Dr. Wejia Wag School of Computig Scieces Uiversity of East Aglia 1. Itroductio. Esemble: Diversity ad Accuracy 3. Costructig Effective Esembles 4. Osteoporosis Risk Predictio 5. Coclusio Data Pre-processig Data Miig Kowledge Osteoporosis Predictio ( Dr Wejia Wag) Geeric Esemble Cocept Esemble Hypothesis What is esemble? A collectio of differet models {A i } for a give task. Models work i parallel ad the overall output is determied by a decisio fusio strategy, S. Esemble accuracy: A e System Output System Decisio Fusio Versios 1 System iputs Sigle method may be ieffective i dealig with complex problems. Multiple models that work collectively improve the performace. Questio: Ca a esemble really work? Osteoporosis Predictio ( Dr Wejia Wag) 3 Osteoporosis Predictio ( Dr Wejia Wag) 4. Esemble ad Diversity Diversity amog the models is the key Esemble Accuracy is determie by: A e (Φ)f(A i, D,, S) Where, A e Accuracy of Esemble Φ A i Accuracy of idividual model i umber of models i Esemble Φ S the decisio fusio strategy D diversity betwee models Decisio Fusio strategies Decisio strategy should be determied by (1)the type of the task ad () the type of output value of models. Cotiuous (target value or probability) or soft output Discrete Class label Two basic types of fusio strategies: Averagig: for cotiuous( or soft) outputs: Averagig or simple aggregatig fuctios Weighted averagig Decisio template Votig: for discrete or categorical outputs Simple majority votig Weighted votig Osteoporosis Predictio ( Dr Wejia Wag) 5 Osteoporosis Predictio ( Dr Wejia Wag) 6 Dr. Wejia Wag: Data miig esemble for prediictig osteoporosis risk 1

2 CMP: Data Miig ad Statistics Withi the Health Services 19//1.1. Why eed diversity? Esemble A: o diversity Esemble decisio X A e /3 correct Decisio Strategy: Votig /3 right /3 right /3 right V1 X V X V3 X Esemble B: ideal diversity Esemble decisio V1 X Decisio Strategy: Votig /3 right /3 right /3 right V X A e 1% correct X V3 Diversity does ot come easily The studies (Littlewood & Miller, Eckhardt, Hesa, Kohavi, Krogh, Partridge, Wag, etc. ) idicate that models developed idepedetly of each other will statistically fail depedetly. Very difficult to geerate a high level of diversity amog the models. What diversity? Test data Test data Osteoporosis Predictio ( Dr Wejia Wag) 7 Osteoporosis Predictio ( Dr Wejia Wag) 8. Diversity Measures Pair-wised measures Disagreemet Q statistic/kappa statistic Correlatio o-pair-wised Etropy Bias/Variace Coicidet failure (CFD) Esemble Reliability otatios: M : the umber of testig samples, : the umber of models i Esemble A, (ote: should be set to a odd umber whe a votig decisio strategy applies) k : the umber of samples that fail o exactly models i A p(r): probability that r radomly chose models fail o a radom chose test sample. The probability that exactly models fail o a radomly selected test ksample x is defied as: p 1,,..., M Osteoporosis Predictio ( Dr Wejia Wag) 9 Osteoporosis Predictio ( Dr Wejia Wag) 1 Reliability Measure: p(1) (1) The probability that oe radom chose model fails o iput x: p(1) P(oe radomly chose model fails o x) P(exactly models fail o P(chose p P(exactly models fail ) x ad the chose model is oe of the failures model fails exactly models fail ) ) Reliability Measure: p() Similarly, () The probability that TWO radomly chose models fail o x: p() P(two P(exactly models fail o P(chose radomly chose models fail o models both fail exactly P(exactly models fail ) ( 1) * p ( 1) x x) ad the two chose model are both failures ) models fail ) * Osteoporosis Predictio ( Dr Wejia Wag) 11 Osteoporosis Predictio ( Dr Wejia Wag) 1 Dr. Wejia Wag: Data miig esemble for prediictig osteoporosis risk

3 CMP: Data Miig ad Statistics Withi the Health Services 19//1 Reliability Measure: p(r) Majority-correct probability: p(maj) I geeral, (3) Probability that r radomly chose models fail: p( r) P( r radomly chose ( 1) ( r + 1) models fail )... p 1 ( 1) ( r + 1) The probability that majority models i esemble A produce the correct aswer for a iput x: p( maj) p(miority models fail) ( 1)/ p Osteoporosis Predictio ( Dr Wejia Wag) 13 Osteoporosis Predictio ( Dr Wejia Wag) 14 Coicidet Failure Diversity (CFD) If majority models i a esemble do ot fail coicidetally whe votig is applied, this esemble will improve its performace. So we defie CFD: 1 CFD p 1 p 1 1 if p < 1 CFD o diversity, all models are same, if p 1, all models right or wrog. CFD1, maximum diversity esemble will be 1% accurate, < CFD <1: what level of CFD ca be useful? Diversity measuremets o the illustrative example The assessmet of performace ad diversities for the two MVS of 3 models show earlier: The ideal Esemble The worst esemble Idividual model's performace model prob(fail) prob(suc) prob(fail) prob(suc) The model coicidet failures k p k p / / /3.. 3/ Esemble s reliability, diversity ad performace p(1) p()..333 p(1 of correct) p(1 of 3 correct) GD 1.. CFD 1.. p(maj) Osteoporosis Predictio ( Dr Wejia Wag) 15 Osteoporosis Predictio ( Dr Wejia Wag) Costruct Effective Esembles Cadidate Models: ay learig models, e.g. eural etworks Decisio trees Bayesia models Methods Baggig ad Boostig Radom Forests (decisio trees oly) Hybrid: eural ets & decisio trees + feature selectio Baggig (Bootstrap Aggregatig) Baggig starts with the Bootstrappig for times: Bootstrap is a samplig approach with replacemet from the origial traiig set D, gettig a ew dataset with the same umber of examples (M) as the origial traiig set Each example has the same probability of beig sampled, Some examples will ot be sampled, some examples will be sampled two or more times duplicates i the bootstrap sample A learig algorithm is applied to each of the bootstrap samples, producig models. The aswer to a test sample is worked out by a decisio aggregatio method o the outputs of the models. averagig, or votig. Osteoporosis Predictio ( Dr Wejia Wag) 17 Osteoporosis Predictio ( Dr Wejia Wag) 18 Dr. Wejia Wag: Data miig esemble for prediictig osteoporosis risk 3

4 CMP: Data Miig ad Statistics Withi the Health Services 19//1 Baggig algorithm illustratio D Baggig: Pros ad Cos For regressio: Bootstrappig D 1 D D Learig algorithm x m 1 m m ŷ 1 (x) ŷ i (x) ŷ (x) ŷ(x) 1/k.(ŷ 1 (x)+ŷ (x)+ +ŷ (x)) Pros: Reduce the variace, but bias uchaged may be able to improve sigificatly the model s performace if the learig algorithm is ustable. Cos: Degrade the performace of stable procedures ot so effective with stable classificatio algorithms, which are robust to small variatios i the traiig data set. For classificatio: ŷ(x) the majority class i {ŷ 1 (x),,ŷ (x)} Osteoporosis Predictio ( Dr Wejia Wag) 19 Osteoporosis Predictio ( Dr Wejia Wag) AdaBoostig: Basic Cocept Adaboost Algorithm Freud ad Schapire (1997), Breima (1998) Data adaptively re-sampled Previously misclassified observatios weights Previously correctly classified observatios weights All the models are aggregated by weighted votig accordig to their accuracy. Osteoporosis Predictio ( Dr Wejia Wag) 1 Osteoporosis Predictio ( Dr Wejia Wag) AdaBoost: pros ad cos LogitBoost Advatages Very simple to implemet Geeratig i relatively simple classifier Good at reducig bias fairly good geeralizatio LogitBoost (Friedma) Disadvatages Suboptimal solutio Sesitive to oisy data ad outliers Osteoporosis Predictio ( Dr Wejia Wag) 3 Osteoporosis Predictio ( Dr Wejia Wag) 4 Dr. Wejia Wag: Data miig esemble for prediictig osteoporosis risk 4

5 CMP: Data Miig ad Statistics Withi the Health Services 19//1 BagBoostig Itroduced by Dettlig, specifically for DA microarray data. 1 Based o the stadard LogitBoost algorithm. Icorporates baggig at each roud of boostig, with decisio stumps as base learer. The model at each roud of boostig is a set of decisio stumps, whose outputs are combied. BagBoostig was foud to outperform LogitBoost o DA microarray data. LogitBoostR ad BagBoostR We (my PhD studet Geoffrey ad I) developed these algorithms 1 i 7 by icorporatig feature o-replacemet ito LogitBoost ad BagBoostig. Aimed to ehace their performace further. They were tested them o several bechmark DA microarray datasets. compared their performace with that of the umodified algorithms, LogitBoost ad Bagboost. 1 Bioiformatics : (4) 1 G.R.Guile ad W.Wag, IJC 7, Coferece Proceedigs, pp , (7) Osteoporosis Predictio ( Dr Wejia Wag) 5 Osteoporosis Predictio ( Dr Wejia Wag) 6 Test Results of LogitBoostR & BagBoostigR 4. Osteoporosis Predictio LogitBoost LogitBoost-R BagBoostig BagBoostig-R Osteoporosis Osteoporosis is a systemic skeletal disease that reduces boe mass desity, deteriorates microstructure of boe tissue ad cosequetly causes boes fragile ad easy to break. Boe microstructures: Health boe Osteoporotic boe Colo Leukemia Prostate Breast Cacer (ER) Mea fial error rate from 5 radom partitios of data ito traiig ad testig sets, o feature preselectio, 5 boostig iteratios. Osteoporosis Predictio ( Dr Wejia Wag) 7 Osteoporosis Predictio ( Dr Wejia Wag) Osteoporosis Data Data Data: early 3 female cases have bee collected, Iputs: > 4 variables, but 33 risk factors were used, Target: predict if a patiet has osteoporosis Diagostic gold stadard: DEXA screeig. Idetificatio of saliet risk factors ot all 33 risk are equally importat, Idetifyig the saliece of risk factors, Rakig ad choosig risk factors. Demographics of the data sets Four data sets collected from three differet hospitals form 1997 to 5. Data Set DXA QUS PIXI DXA o. of Record s o. of Factor s mea Age s.d ormal Diagosis (%) Osteopeai c osteoporoti c Osteoporosis Predictio ( Dr Wejia Wag) 9 Osteoporosis Predictio ( Dr Wejia Wag) 3 Dr. Wejia Wag: Data miig esemble for prediictig osteoporosis risk 5

6 CMP: Data Miig ad Statistics Withi the Health Services 19//1 4.. Covetioal Risk Models Model SCORE: There are about dozes of classic factorbased models for assessig the risk of osteoporosis, e.g. SCORE OSIRIS ORAT ORAI OST Implemeted ad Tested # Factor Age > 65 Race Rheumatoid Arthritis History of fracture at wrist, hip or rib Eastroge therapy Weight (lbs) The total Score: S Σ s i Cut-off threshold: Risk if S > 6, o risk otherwise Score si 3 x the first digit of the age 5 if ot black, otherwise 4 preset, otherwise 4 for each fracture 1 ever used, otherwise -weight/1 Osteoporosis Predictio ( Dr Wejia Wag) 31 Osteoporosis Predictio ( Dr Wejia Wag) 3 Web applicatio of risk models Web Questioaire Osteoporosis Predictio ( Dr Wejia Wag) 33 Osteoporosis Predictio ( Dr Wejia Wag) 34 Results Test of the risk models Osteoporosis Predictio ( Dr Wejia Wag) 35 Osteoporosis Predictio ( Dr Wejia Wag) 36 Dr. Wejia Wag: Data miig esemble for prediictig osteoporosis risk 6

7 CMP: Data Miig ad Statistics Withi the Health Services 19//1 4.3 Esemble for Osteoporosis Idetifyig risk factors 1 Age Height Weight Fractures H_Loss Firstly, partitio the data set ito two sub sets Oe for traiig, aother for testig The, quatify the saliece of each factor, rak them ad select the most relevat factors Usig the factors selected, we traied some eural ets ad decisio trees as the cadidates of predictors for buildig three types of esemble Buildig esembles of three types eural ets oly called multi-et esemble or system decisio tress oly called multi-tree esemble mixed ets ad trees called hybrid esemble The esembles were tested by the test data Repeat above procedure times to check the cosistece. Rakig of Risk Factors The lower geeralisatio accuracy is, the more saliet a risk factor is. The risk factors with the lowest geeralisatios are most saliet. 4 most saliet factors: kyphosis, FH, Backpai ad Oestrogedeficiecy Factors plotted at top are redudat or less saliet. Clamped performace Traiig epoch Kyphosis BackPai YrsMo FH Cigs alc Cal Iactivity Exercise Steroid Ati-epileptics T4 HRT VitD Bisphos Asthma Hyperparathyroidism Aoresxia UlcerativeColitis PacreaticDisease Coeliac RealDisease RheumatoidArthritis Croh's LiverDisease OesDef Learig Osteoporosis Predictio ( Dr Wejia Wag) 37 Osteoporosis Predictio ( Dr Wejia Wag) 38 Impact directio of the factors Idetifyig the directio of correlatio of factors Weight Height BMI Family History HRT Exercise OesDef Vitami D Steroids Liver Disease Rheumatoid Arthritis Bisphos Kyphosis Fractures Height Loss YrsMo Age DEXA QUS PIXI How may factors to use? Five methods were used for feature rakig Differet umbers of features were selected for traiig the classifiers to see how may features ca produce the best classificatio All L-Corr Clampig D-Tree ARD Radom umber of f act ors selected The test performace of the models that are traied with the factors selected by differet techiques. Osteoporosis Predictio ( Dr Wejia Wag) 39 Osteoporosis Predictio ( Dr Wejia Wag) 4 Esemble Performace Hybrid Esembles A esemble built with 9 eural etworks: best: et8.7, worst: et mea:.65, s.d..4 Esemble performace p(1).34, p().116 p(1/3).863, p(/3).699 CFD.649 Accuracy 74.5% Figure shows the ROC curves of some idividual ets, the esemble ad logistic regressio (LR) for compariso. Sesitivity Specificity et8 et7 et1 MS Osteoporosis Predictio ( Dr Wejia Wag) 41 LR 7% correct predictio for the cases 5% predicted wrog for the oe-cases Hybrid esembles were built with eural ets ad trees. CFD varies as the umber of decisio trees i a esemble chages. Predictio performace: Improved as the diversity ehaced whe majorityvotig strategy applied. Diversity is ot useful if averagig-strategy used. Performace Multi Versio Systems of eural ets ad Trees Average Majority umber of the decisio trees i a MVS Osteoporosis Predictio ( Dr Wejia Wag) 4 CFD diversity: CFD Dr. Wejia Wag: Data miig esemble for prediictig osteoporosis risk 7

8 CMP: Data Miig ad Statistics Withi the Health Services 19//1 Applicatio for gee idetificatio Gee selectio for Colo cacer data We have proposed a boostig based esemble method to idetify the relevace of gees i microarray (MA) data for gee selectio. The method has bee tested o a umber of bechmark MA data sets. The results are compared with other methods ad the best kow results. The gees we selected gave the eve better classificatio results. Error (%) umber of features 1 BFSS Wilcoxo boostig 5 1 All Guile, G. ad Wag, W.: Boostig for feature selectio for microarray data aalysis. Proceedigs of IEEE WCCI-IJC8, pp56-564, Hog Kog, Jue 1-6, 8. Osteoporosis Predictio ( Dr Wejia Wag) 43 Osteoporosis Predictio ( Dr Wejia Wag) Coclusios Esemble is a collectio of models that work together to improve performace. Diversity amog models is essetial. Hybrid esembles higher diversity, More reliable ad more accurate. Best osteoporosis predictio 78% sesitivity, 9% specificity Ackowledgemets This research was fuded (GR/R85914/1 ad GR/R8641/1) by the EPSRC, UK. Collaborators from hospitals: Dr. Sarah Rae & Dr. Adams Youg Researchers: Dr. Graeme Richards (my former Research Assistat) Dr. Geoffrey Guile (my former PhD, RA ow) Mr. Ross Turer (BSc) Osteoporosis Predictio ( Dr Wejia Wag) 45 Osteoporosis Predictio ( Dr Wejia Wag) 46 Dr. Wejia Wag: Data miig esemble for prediictig osteoporosis risk 8