Introduction to data mining. MATH 5392 Department of mathematics, UTA

Size: px

Start display at page:

Download "Introduction to data mining. MATH 5392 Department of mathematics, UTA"

Oswald Wilkerson
5 years ago
Views:

1 Intrductin t data mining MATH 5392 Department f mathematics, UTA

2 Curse infrmatin Curse Webpage: math5392-fa17.html Office Hurs: PKH 446, Tu/Thu 1:00-2:00 pm r by appintment Prerequisite: Basic prgramming skills are preferred, but nt required. Required Textbks Hastie, T., Tibshirani, R., and Friedman, J. H. (2008), The Elements f Statistical Learning: Data Mining, Inference, and Predictin, 2nd Editin. Springer. Other Recmmended Textbks and Resurces Bishp, C. M. (2006). Pattern recgnitin and machine learning. Springer. Rencher and Schaalje (2008). Linear Mdels in Statistics, 2nd Editin. Wiley.

3 Assignments Midterm take-hme exam (25%) Tuesday, Oct 17th - the exam will be distributed during the class. Students need t submit the slutin within 48 hurs. Hmewrk assignments (35%) 10 HW assignments (30%) and final HW (5%). The lwest hmewrk scre will be drpped, but yu cannt drp the final hmewrk. Final prjects (40%) The final prject guideline will be annunced later.

4 Assignment submissin frmat All assignments (hmewrk, midterm exam, final prject) must be turned in electrnically, thrugh Blackbard. All assignments must be submitted in R Markdwn and PDF frmat. Wrk submitted in R Markdwn frmat that des nt cmpile, i.e., fails Knit PDF (r HTML), will receive an autmatic grade f 0.

5 What is data mining? Many definitins Nn trivial extractin f implicit, previusly unknwn and useful infrmatin frm data Explratin & analysis, by autmatic r semiautmatic means, f large quantities f data in rder t discver meaningful patterns.

6 What is data mining? Data mining is actually a part f the knwledge discvery prcess (KDD: knwledge discvery frm data).

7 Why data mining? Data cllected and stred at enrmus speeds (GB/hur) Remte sensrs n a satellite Genmic data Web data, e-cmmerce Bank/credit card transactins Cmputers have becme cheaper and mre pwerful. Traditinal techniques infeasible fr raw data.

8 Origins f data mining Overlaps with machine learning, statistics, artificial intelligence, databases, visualizatin but mre stress n Scalability f number f features and instances Stress n algrithms whereas fundatins f methds and frmulatins prvided by statistics and machine learning.

9 Data mining tasks Predictin Methds Use sme variables t predict unknwn r future values f ther variables. Descriptin Methds Find human interpretable patterns that describe the data. Classificatin, clustering, regressin, anmaly detectin, etc.

10 spam data A gal is t predict whether the was spam r nt. The true utcme (ham r spam) is available, alng with the relative frequencies f 57 f the mst cmmnly ccurring wrds and punctuatin marks in the message. gerge yu yur hp free hpl! ur re edu remve spam A learning methd has t decide which features t use and hw: fr example, if (%gerge < 0.6) & (%yu > 1.5) then spam else .

11 Prstate cancer The gal is t predict the lg f prstate specific antigen (lpsa) frm a number f measurements. Frm the scatterplt matrix f the variables, sme crrelatins with lpsa are evident, but a gd predictive mdel is difficult t cnstruct by eye. lpsa lcavl lweight age lbph svi lcp gleasn pgg45 FIGURE 1.1. Scatterplt matrix f the prstate cancer data. The first rw shws the respnse against each f the predictrs in turn. Tw f the predictrs, svi

12 Handwritten digit recgnitin Handwritten ZIP cdes n envelpes frm U.S. pstal mail eight-bit grayscale maps, with each pixel ranging in intensity frm 0 t 255. It is very imprtant t predict, frm the matrix f pixel intensities, the identity f each image (0, 1,..., 9). The errr rate needs t be kept very lw t avid misdirectin f mail

Gene expressin data frm DNA micrarray Clumns - genes; Rws - samples (tumr types) Ranging frm bright green (negative, under expressed) t bright red (psitive ver expressed).

13 Gene expressin data frm DNA micrarray Clumns - genes; Rws - samples (tumr types) Ranging frm bright green (negative, under expressed) t bright red (psitive ver expressed). Hw the genes and samples are assciated? Which samples are mst similar t each ther, in terms f their expressin prfiles acrss genes? Which genes are mst similar t each ther, in terms f their expressin prfiles acrss samples? D certain genes shw very high (r lw) expressin fr certain cancer samples? SIDW SIDW SID73161 GNAL H.sapiensmRN SID RASGTPASE SID ESTs SIDW HumanmRNA SIDW ESTs SID MYBPROTO ESTsChr.1 SID DNAPOLYME SID SIDW31489 SID SIDW SIDW Hmsapiens SIDW Chr MITOCHOND SID47116 ESTsChr.6 SIDW SID SID ESTsChr.3 SID SID PTPRC SIDW SIDW SIDW ESTsCh31 SID SID SID SIDW SIDW SIDW HLACLASSI SIDW SID SIDW SIDW HYPOTHETIC WASWisktt SIDW ESTsChr.15 SIDW SID ESTsChr.5 SIDW SID46536 SIDW ESTsChr.2 SIDW SID ESTsChr.15 SID SID SID ESTs SIDW SMALLNUC ESTs SIDW SIDW SID52979 ESTs SID43609 SIDW ERLUMEN TUPLE1TUP1 SIDW SID SIDW SIDW SIDW ESTsChr.15 SIDW SID SIDW SID SID SIDW ESTsChr.10 SIDW SID SID SIDW SID SID31984 SID42354 BREAST RENAL MELANOMA MELANOMA MCF7D-repr COLON COLON K562B-repr COLON NSCLC LEUKEMIA RENAL MELANOMA BREAST CNS CNS RENAL MCF7A-repr NSCLC K562A-repr COLON CNS NSCLC NSCLC LEUKEMIA CNS OVARIAN BREAST LEUKEMIA MELANOMA MELANOMA OVARIAN OVARIAN NSCLC RENAL BREAST MELANOMA OVARIAN OVARIAN NSCLC RENAL BREAST MELANOMA LEUKEMIA COLON BREAST LEUKEMIA COLON CNS MELANOMA NSCLC PROSTATE NSCLC RENAL RENAL NSCLC RENAL LEUKEMIA OVARIAN PROSTATE COLON BREAST RENAL UNKNOWN

14 Anmaly detectin Detect significant deviatins frm nrmal behavir Applicatins: Credit Card Fraud Detectin Netwrk Intrusin Detectin

15 Characteristics f data in data mining Large n (number f bservatins) and large p (number f predictrs); Predictr variables are f mixed types curse f dimensinality (Bellman, 1957) Mstly cming frm bservatinal studies, instead f designed experiments. Data are cnstantly accumulating and dynamically varying. Data cleaning and preparatin (missing value handling, utliers, categrical variables, etc.) Explratry data analysis is vital; may use 90% f the ttal analysis time.

16 Supervised vs unsupervised Supervised Learning (Learning with a teacher): predictive mdeling, i.e., predicting ne (r mre) utput (r target) variable frm a set f inputs r predictrs; classificatin (als called pattern recgnitin) and regressin. Unsupervised learning (learning withut a teacher): Clustering; Segmentatin; Dimensin reductin; Explring assciatins; etc. Semi-supervised learning: The training sample cntains bth labelled and unlabelled data, typically a small amunt f labelled data with a large amunt f unlabelled data.

17 Supervised learning One respnse (als called dependent variable, target, utcme, utput) Y vs a set f p predictrs (als called inputs, independent variables, cvariates, features). In the regressin prblem, Y is quantitative (e.g. price, bld pressure) In the classificatin prblem, Y takes values in a finite, unrdered set (survived/died, digit 0-9, cancer class f tissue sample). Predictrs culd be f mixed types: nminal, rdinal, interval, and rati.

18 Supervised learning Data typically cnsist f n iid bservatins: {(y i,x i0,x i1,...,x ip ),i=1,...,n} Predictive Mdeling seek a mdel t predict Y The mdel is a learner, which prvides an answer. This answer will be evaluated by cmparing with the crrected answer r the target (i.e., the bserved value prvided by the teacher)

19 Regressin What is a gd mdel f(x) t make predictins f Y at X=x? There can be many Y values at X=4. A gd value is This ideal f(x) is called the regressin functin x y f(4) = E(Y X = 4)

20 Regressin The regressin functin is ptimal predictr f Y with regard t expected predictin errr under squared lss functin. Lss functin (squared lss): Risk functin: By chsing errr is minimized. L(Y,f(x)) = (Y f(x)) 2 E(L(Y,f(X)) = E(Y f(x)) 2 f(x) =E(Y X),, the expected predictin E(Y f(x)) 2 = E X E([Y f(x)] 2 X)

21 Hw t estimate f(x)? Typically we have few data pints with X = 4. S it is hard t btain the regressin functin. Nearest neighbr methd: relax the definitin and let Linear regressin: assume the regressin functin is linear. ˆf(x) = Ave(Y X 2 N(x)) x y E(Y X) = + X

22 Curse f dimensinality Nearest neighbr methds can be lusy when p is large. K bservatins, say 10% f bservatins, that are nearest t a given X=x may be very far away frm x when p is large, leading t a very pr predictin x1 x2 10% Neighbrhd Fractin f Vlume Radius p= 1 p= 2 p= 3 p= 5 p= 10

23 Classificatin The respnse variable Y is qualitative e.g. is ne f (spam,ham) (ham = gd ), digit class is ne f {0,1,,9}. Our gals are t Build a classifier C(X) that assign a class label t a future unlabeled bservatin. Assess the uncertainty in each classificatin. Understand the rles f the predictrs

24 y x Suppse there are tw classes (Y = 0 r 1). Let p k =Pr(Y = kx),k =0, 1 The Bayes ptimal classifier is C(X) =j if p j (X) = max{p 0 (X),p 1 (X)}

25 Unsupervised learning N utcme variable, just a set f predictrs (features) measured n a set f samples. Objective is mre fuzzy find grups f samples that behave similarly, find features that behave similarly, find linear cmbinatins f features with the mst variatin. Difficult t knw hw well yur are ding. Different frm supervised learning, but can be useful as a pre-prcessing step fr supervised learning.

26 Unsupervised learning We bserve nly the features. {(x i0,x i1,...,x ip ),i=1,...,n} We are nt interested in predictin, because we d nt have an assciated respnse variable Y. Principal cmpnents analysis: a tl used fr data visualizatin r data pre-prcessing befre supervised techniques are applied. Clustering: a brad class f methds fr discvering unknwn subgrups in data..

27 Advantage f unsupervised learning Unsupervised learning is mre subjective than supervised learning, as there is n simple gal fr the analysis, such as predictin f a respnse. But techniques fr unsupervised learning are f grwing imprtance in a number f fields: Subgrups f breast cancer patients gruped by their gene expressin measurements. Mvies gruped by the ratings assigned by mvie viewers. It is ften easier t btain unlabeled data frm a lab instrument r a cmputer than labeled data, which can require human interventin.

28 Statistical learning vs machine learning Machine learning arse as a subfield f artificial intelligence. Statistical learning arse as a subfield f statistics. There is much verlap bth fields fcus n supervised and unsupervised prblems. Machine learning has a greater emphasis n large scale applicatins and predictin accuracy. Statistical learning emphasizes mdels and their interpretability, and precisin and uncertainty. But the distinctin has becme mre and mre blurred, and there is a great deal f crss-fertilizatin.

29 Challenges f data mining Scalability Dimensinality Cmplex and hetergeneus data Data quality Data wnership and distributin Streaming data

30 Meaningfulness f answers A big data mining risk is that yu will discver patterns that are meaningless. Rhine Paradx: a great example f hw nt t cnduct scientific research. Jseph Rhine was a parapsychlgist in the 1950 s wh hypthesized that sme peple had extra-sensry perceptin (ESP). He devised (smething like) an experiment where subjects were asked t guess 10 hidden cards (red r blue). He discvered that almst 1 in 1000 had ESP. They were able t get all 10 right! He tld these peple they had ESP and called them in fr anther test f the same type. Alas, he discvered that almst all f them had lst their ESP. He cncluded that yu shuldn t tell peple they have ESP; it causes them t lse it.

Statistical Methods for Data Mining

Statistical Methods for Data Mining Statistical Methds fr Data Mining Kuangnan Fang Xiamen University Email: xmufkn@xmu.edu.cn Statistics in the news Hw IBM built Watsn, its Jepardy-playing supercmputer by Dawn Kawamt DailyFinance 02/08/2011