Statistical Methods for Data Mining

Transcription

1 Statistical Methds fr Data Mining Kuangnan Fang Xiamen University

2 Statistics in the news Hw IBM built Watsn, its Jepardy-playing supercmputer by Dawn Kawamt DailyFinance 02/08/2011 Learning frm its mistakes Accrding t David Ferrucci (PI f Watsn DeepQA technlgy fr IBM Research), Watsn s sftware is wired fr mre that handling natural language prcessing. It s machine learning allws the cmputer t becme smarter as it tries t answer questins and t learn as it gets them right r wrng. 2/29

3

4 Enlarge This Image Fr Tday s Graduate, Just One Wrd: Statistics N By STEVE LOHR Published: August 5, 2009 MOUNTAIN VIEW, Calif. At Harvard, Carrie Grimes majred in anthrplgy and archaelgy and ventured t places like Hnduras, where she studied Mayan settlement patterns by mapping where artifacts were fund. But she was drawn t what she calls all the cmputer and math stuff that was part f the jb. Peple think f field archaelgy as Indiana Jnes, but much f what yu really d is data analysis, she said. SIGN IN TO RECOMMEND SIGN IN TO Nw Ms. Grimes des a different kind f digging. She wrks at Ggle, where she uses statistical analysis f munds f data t Thr Swift fr The New Yrk Times cme up with ways t imprve its search engine. Carrie Grimes, senir staff engineer at Ggle, uses statistical analysis f Ms. Grimes is an Internet-age statistician, ne f many data t help imprve the cmpany's search engine. wh are changing the image f the prfessin as a place fr drnish number nerds. They are finding themselves Multimedia increasingly in demand and even cl. PRINT REPRINTS SHARE I keep saying that the sexy jb in the next 10 years will be statisticians, said Hal Varian, chief ecnmist at Ggle. And I m nt kidding. Qute f the Day, New Yrk Times, August 5, 2009 I keep saying that the sexy jb in the next 10 years will be statisticians. And I m nt kidding. HAL VARIAN, chief ecnmist at Ggle. S 3/29

5 ! 4/29

6

7

8 Statistical Learning versus Machine Learning Machine learning arse as a subfield f Artificial Intelligence. Statistical learning arse as a subfield f Statistics. There is much verlap bth fields fcus n supervised and unsupervised prblems: Machine learning has a greater emphasis n large scale applicatins and predictin accuracy. Statistical learning emphasizes mdels and their interpretability, and precisin and uncertainty. But the distinctin has becme mre and mre blurred, and there is a great deal f crss-fertilizatin. Machine learning has the upper hand in Marketing! 28 / 29

9 The Supervised Learning Prblem Starting pint: Outcme measurement Y (als called dependent variable, respnse, target). Vectr f p predictr measurements X (als called inputs, regressrs, cvariates, features, independent variables). In the regressin prblem, Y is quantitative (e.g price, bld pressure). In the classificatin prblem, Y takes values in a finite, unrdered set (survived/died, digit 0-9, cancer class f tissue sample). We have training data (x 1,y 1 ),...,(x N,y N ). These are bservatins (examples, instances) f these measurements. 22 / 29

10 Objectives On the basis f the training data we wuld like t: Accurately predict unseen test cases. Understand which inputs a ect the utcme, and hw. Assess the quality f ur predictins and inferences. 23 / 29

11 Philsphy It is imprtant t understand the ideas behind the varius techniques, in rder t knw hw and when t use them. One has t understand the simpler methds first, in rder t grasp the mre sphisticated nes. It is imprtant t accurately assess the perfrmance f a methd, t knw hw well r hw badly it is wrking [simpler methds ften perfrm as well as fancier nes!] This is an exciting research area, having imprtant applicatins in science, industry and finance. Statistical learning is a fundamental ingredient in the training f a mdern data scientist. 24 / 29

12 Unsupervised learning N utcme variable, just a set f predictrs (features) measured n a set f samples. bjective is mre fuzzy find grups f samples that behave similarly, find features that behave similarly, find linear cmbinatins f features with the mst variatin. di cult t knw hw well yur are ding. di erent frm supervised learning, but can be useful as a pre-prcessing step fr supervised learning. 25 / 29

13 Statistical Learning Prblems Identify the risk factrs fr prstate cancer. Classify a recrded phneme based n a lg-peridgram. Predict whether smene will have a heart attack n the basis f demgraphic, diet and clinical measurements. Custmize an spam detectin system. Identify the numbers in a handwritten zip cde. Classify a tissue sample int ne f several cancer classes, based n a gene expressin prfile. Establish the relatinship between salary and demgraphic variables in ppulatin survey data. Classify the pixels in a LANDSAT image, by usage. 5/29

14 lpsa lcavl lweight age lbph svi lcp gleasn /29

15 lpsa lcavl lweight age lbph svi lcp gleasn pgg45 7/29

16 Statistical Learning Prblems Identify the risk factrs fr prstate cancer. Classify a recrded phneme based n a lg-peridgram. Predict whether smene will have a heart attack n the basis f demgraphic, diet and clinical measurements. Custmize an spam detectin system. Identify the numbers in a handwritten zip cde. Classify a tissue sample int ne f several cancer classes, based n a gene expressin prfile. Establish the relatinship between salary and demgraphic variables in ppulatin survey data. Classify the pixels in a LANDSAT image, by usage. 8/29

17 Phneme Examples Lg-peridgram aa a Frequency Phneme Classificatin: Raw and Restricted Lgistic Regressin Lgistic Regressin Cefficients Frequency 9/29

18 Statistical Learning Prblems Identify the risk factrs fr prstate cancer. Classify a recrded phneme based n a lg-peridgram. Predict whether smene will have a heart attack n the basis f demgraphic, diet and clinical measurements. Custmize an spam detectin system. Identify the numbers in a handwritten zip cde. Classify a tissue sample int ne f several cancer classes, based n a gene expressin prfile. Establish the relatinship between salary and demgraphic variables in ppulatin survey data. Classify the pixels in a LANDSAT image, by usage. 10 / 29

19 sbp tbacc ldl famhist besity alchl age 11 / 29

20 Statistical Learning Prblems Identify the risk factrs fr prstate cancer. Classify a recrded phneme based n a lg-peridgram. Predict whether smene will have a heart attack n the basis f demgraphic, diet and clinical measurements. Custmize an spam detectin system. Identify the numbers in a handwritten zip cde. Classify a tissue sample int ne f several cancer classes, based n a gene expressin prfile. Establish the relatinship between salary and demgraphic variables in ppulatin survey data. Classify the pixels in a LANDSAT image, by usage. 12 / 29

21 Spam Detectin data frm s sent t an individual (named Gerge, at HP labs, befre 2000). Each is labeled as spam r . gal: build a custmized spam filter. input features: relative frequencies f 57 f the mst cmmnly ccurring wrds and punctuatin marks in these messages. gerge yu hp free! edu remve spam Average percentage f wrds r characters in an message equal t the indicated wrd r character. We have chsen the wrds and characters shwing the largest di erence between spam and / 29

22 Statistical Learning Prblems Identify the risk factrs fr prstate cancer. Classify a recrded phneme based n a lg-peridgram. Predict whether smene will have a heart attack n the basis f demgraphic, diet and clinical measurements. Custmize an spam detectin system. Identify the numbers in a handwritten zip cde. Classify a tissue sample int ne f several cancer classes, based n a gene expressin prfile. Establish the relatinship between salary and demgraphic variables in ppulatin survey data. Classify the pixels in a LANDSAT image, by usage. 14 / 29

23 15 / 29

24 Statistical Learning Prblems Identify the risk factrs fr prstate cancer. Classify a recrded phneme based n a lg-peridgram. Predict whether smene will have a heart attack n the basis f demgraphic, diet and clinical measurements. Custmize an spam detectin system. Identify the numbers in a handwritten zip cde. Classify a tissue sample int ne f several cancer classes, based n a gene expressin prfile. Establish the relatinship between salary and demgraphic variables in ppulatin survey data. Classify the pixels in a LANDSAT image, by usage. 16 / 29

25 17 / 29

26 Statistical Learning Prblems Identify the risk factrs fr prstate cancer. Classify a recrded phneme based n a lg-peridgram. Predict whether smene will have a heart attack n the basis f demgraphic, diet and clinical measurements. Custmize an spam detectin system. Identify the numbers in a handwritten zip cde. Classify a tissue sample int ne f several cancer classes, based n a gene expressin prfile. Establish the relatinship between salary and demgraphic variables in ppulatin survey data. Classify the pixels in a LANDSAT image, by usage. 18 / 29

27 Wage Wage Wage Age Year Educatin Level Incme survey data fr males frm the central Atlantic regin f the USA in / 29

28 Statistical Learning Prblems Identify the risk factrs fr prstate cancer. Classify a recrded phneme based n a lg-peridgram. Predict whether smene will have a heart attack n the basis f demgraphic, diet and clinical measurements. Custmize an spam detectin system. Identify the numbers in a handwritten zip cde. Classify a tissue sample int ne f several cancer classes, based n a gene expressin prfile. Establish the relatinship between salary and demgraphic variables in ppulatin survey data. Classify the pixels in a LANDSAT image, by usage. 20 / 29

29 Spectral Band 1 Spectral Band 2 Spectral Band 3 Spectral Band 4 Land Usage Predicted Land Usage Usage 2 {red sil, cttn, vegetatin stubble, mixture, gray sil, damp gray sil} 21 / 29

30 Graphical netwrk 17,214 gene expressins and 22,247 CNVs are available frm TCGA(X.Fan,K.Fang,S.Ma, Q. Zhang. Assisted Graphical Mdel fr Gene Expressin Data Analysis. Statistics in Medicine. 2019). Tp left panel: AGM; Tp right panel: GM; Bttm left panel: di erence between AGM and GM; Bttm middle: di erence between mderate cnnectins ; Bttm right: di erence between strng cnnectins.

31 The FICO scre was first intrduced in 1989 by Fair, Isaac, and Cmpany, whic is used by the vast majrity f banks and credit grantrs, and is based n cnsumer credit files f the three natinal credit bureaus: Experian, Equifax, and TransUnin.

32 Credit scre in China

33 Interesting prblem fr Credit scring Variable selectin reject inference reject ptin imbalance data label nise missing data fraud detectin

34 Data integratin Tw type f Data integratin (Data fushin): Sample integratin and Variable integratin Table 1: 3 datasets n breast cancer frm GEO Breast Gene Sample Case Cntrl GSE GSE GSE Variable integratin: Gene and CNA Integratin f supervised and unsupervised learning

35 CSA Breath Data 1. Tidal vlume breathing fr 5 minutes;

36 CSA Breath Data 1. Tidal vlume breathing fr 5 minutes; 2. Exhaled breath drawn ver the sensr array;

37 CSA Breath Data 1. Tidal vlume breathing fr 5 minutes; 2. Exhaled breath drawn ver the sensr array; 3. Images were cnverted t numerical values in the red, green, blue spectra, and 4 ultravilet spectra.

38 CSA Breath Data 1. Tidal vlume breathing fr 5 minutes; 2. Exhaled breath drawn ver the sensr array; 3. Images were cnverted t numerical values in the red, green, blue spectra, and 4 ultravilet spectra. 4. Ttally 128 (the number f clrants) 7( changes in the red, green, blue, and 4 ultra-clr spectrum f each clrant) = 896 grups.

39 CSA Breath Data

40 Netwrk experience Index

41 The Netflix prize cmpetitin started in Octber Training data is ratings fr 18, 000 mvies by 400, 000 Netflix custmers, each rating between 1 and 5. training data is very sparse abut 98% missing. bjective is t predict the rating fr a set f 1 millin custmer-mvie pairs that are missing in the training data. Netflix s riginal algrithm achieved a rt MSE f The first team t achieve a 10% imprvement wins ne millin dllars. is this a supervised r unsupervised prblem? 26 / 29

42 BellKr s Pragmatic Chas wins, beating The Ensemble by a narrw margin. 27 / 29

43 Kaggle

44 Sftware