Machine Learning Approach to Identifying the Dataset Threshold for the Performance Estimators in Supervised Learning

Transcription

1 Machin Larning Approach to Idntifying th Datast Thrshold for th Prformanc Estimators in Suprvisd Larning Zanifa Omary, Frdrick Mtnzi Dublin Institut of Tchnology, Irland Abstract Currntly for small-scal machin larning projcts, thr is no limit which has bn st by its rsarchrs to catgoris datasts for inxprincd usrs such as studnts whil assssing and comparing prformanc of machin larning algorithms. Basd on th lack of such a thrshold, this papr prsnts a stp by stp guid for idntifying th datast thrshold for th prformanc stimators in suprvisd machin larning xprimnts. Th idntification of th datast thrshold involvs prforming xprimnts using four diffrnt datasts having diffrnt sampl sizs from th Univrsity of California Irvin (UCI) machin larning rpository. Th sampl sizs ar catgorisd in rlation to th numbr of attributs and numbr of instancs availabl in th datast. Th idntifid datast thrshold will hlp unfamiliar machin larning xprimntrs to catgoris datasts corrctly and hnc slcting th appropriat prformanc stimation mthod. Kywords: machin larning, machin larning algorithms, datast thrshold, prformanc masurs, suprvisd machin larning. 1. Introduction In rcnt yars, th goal of many rsarchrs in diffrnt filds has bn to build systms that can larn from xprincs and adapt to thir nvironmnts. This volution has rsultd into an stablishmnt of various algorithms such as dcision trs, K-Narst Nighbours (KNN), Support Vctor Machins (SVM) and Random Forsts (RF) that ar transforming problms rising from industrial and scintific filds. Basd on th natur of th datast, ithr balancd or unbalancd, diffrnt prformanc masurs and stimation mthods tnd to prform diffrntly whn applid to diffrnt machin larning algorithms. Th availabl prformanc masurs, such as accuracy, rror rat, prcision, rcall, f1-scor and ROC analysis, ar usd whil assssing and comparing on machin larning algorithm from th othr. In addition to machin larning prformanc masurs, thr ar various statistical tsts, such as McNmar s tst and a tst of th diffrnc of two proportions, also usd to assss and compar classification algorithms. Authors of this papr dscrib thr machin larning prformanc stimation mthods ths ar, hold-out mthod, k-fold cross validation and lav-on out cross validation. Th prformanc of ths stimators dpnds on th numbr of instancs availabl in th datast. From rsarch litratur, th holdout mthod has bn idntifid to work wll on vry larg datasts, but nothing has bn idntifid for th rmaining two stimators. Thrfor, for this papr w will idntify th datast thrshold for th rmaining two stimators. In this papr w prsnt th rsults of th xprimnts prformd using four diffrnt datasts from UCI machin larning rpository togthr with two prformanc stimators. Th accuracy of th datast with all instancs will b rgardd as th thrshold, that is, th minimum valu for th two prformanc stimators. Only on prformanc masur, f1- scor, and on machin larning algorithm, dcision tr, togthr with two prformanc stimators will b usd in th xprimnt for idntifying th datast thrshold. Th rst of this papr is organisd as follows. Sction provids th background of machin larning whr its dfinition, catgoris and th rviw of th machin larning classification tchniqus will b providd. Sction 3 provids th discussion on classification valuations whr prformanc masurs in machin larning and statistical tsts togthr with prformanc stimation mthods will b covrd. Exprimnts for idntifying th datast thrshold will b covrd in sction 4 followd by th rsults of th datast thrshold in sction 5. Conclusion of th papr will b providd in sction 6.. Background In this sction, th background of machin larning will b providd. Th sction is dividd into thr subsctions; machin larning dfinition will b providd in th first sction followd by th discussion on its catgoris in scond subsction. Th rviw of classification tchniqus will b providd on th third and last subsction..1. What is Machin larning? Prior to dlving into formal dfinitions of machin larning it is worthwhil to dfin, in Information and Communication Tchnology (ICT) contxt, th two trms that mak up machin

2 larning; that is, machin or computr and larning. Dfining ths trms will b a guidlin on th slction of appropriat machin larning dfinition for this papr. According to Oxford English Dictionary, computr is a machin for prforming or facilitating calculation; it accpts data, manipulats thm and producs output information basd on a squnc of instructions on how th data has to b procssd. Additionally, larning can b dfind as a procss of acquiring modifications in xisting skills, knowldg and habits through xprinc, xrcis and practic. From th idntifid larning dfinition, Wittn and Frank [1] argu that things larn whn thy chang thir bhaviour in a way that maks thm prform bttr in th futur. From Wittn and Frank s dfinition, larning can b tstd by obsrving th currnt bhaviour and comparing it with past bhaviour. Thrfor, a complt dfinition of machin larning for this papr has to incorporat two important lmnts that ar; computr basd knowldg acquisition procss and has to stat whr skills or knowldg can b obtaind. Mitchll [] dscribs machin larning as a study of computr algorithms that improv automatically through xprinc. This mans computr programs us thir xprinc from past tasks to improv thir prformanc. As w idntifid prviously thr ar two important lmnts that any machin larning dfinition has incorporat in ordr to b rgardd as appropriat for this papr, howvr this dfinition dos not rflct anything rlatd to knowldg acquisition procss for th statd computr programs, thrfor it is considrd insufficint for this papr. Additionally, Alpaydin [3] dfins machin larning as th capability of th computr program to acquir or dvlop nw knowldg or skills from xisting or non xisting xampls for th sak of optimising prformanc critrion. Contrary to th Mitchll s dfinition which lacks knowldg acquisition procss, this dfinition is of mor prfrnc to this papr as it includs two lmnts idntifid prviously that is; knowldg acquisition procss and it indicats whr skills or knowldg can b obtaind. Ovr th past 50 yars, machin larning as any fild of study has grown trmndously [4]. Th growing intrst in machin larning is drivn by two factors as outlind by Alpaydin [3], rmoving tdious human work and rducing cost. As th rsult of automation of procsss, hug amounts of data ar producd in our day-to-day activitis. Doing manual analysis on all of this data is slow, costly and popl who ar abl to do such analysis manually ar rar to b found [5]. Machin larning tchniqus, whn applid to diffrnt filds such as in mdical diagnosis, bio-survillanc, spch and handwriting rcognition, computr vision and dtcting crdit card fraud in financial institutions, hav provd to work with hug amounts of data and provid rsults in a mattr of sconds [3, 4].In th nxt sction a rviw of th two machin larning catgoris is providd... Machin larning catgoris Machin larning can b catgorisd into two main groups that is, suprvisd and unsuprvisd machin larning. Ths two larning catgoris ar associatd with diffrnt machin larning algorithms that rprsnt how th larning mthod works. Suprvisd larning: Suprvisd larning compriss of algorithms that rason from xtrnally supplid instancs to produc gnral hypothsis which thn mak prdictions about futur instancs. Gnrally, with suprvisd larning thr is a prsnc of th outcom variabl to guid th larning procss. Thr ar svral suprvisd machin larning algorithms such as dcision trs, K-Narst Nighbour (KNN), Support Vctor Machins (SVM) and Random Forsts [6]. Ths algorithms will b brifly dscribd in th nxt sction. Unsuprvisd larning: Contrary to suprvisd larning whr thr is a prsnc of th outcom variabl to guid th larning procss, unsuprvisd larning builds modls from data without prdfind classs or xampls [7]. This mans, no suprvisor is availabl and larning must rly on guidanc obtaind huristically by th systm xamining diffrnt sampl data or th nvironmnt [, 8]. Th output stats ar dfind implicitly by th spcific larning algorithm usd and built in constraints [7]..3. Machin Larning Algorithms Although, thr ar various machin larning algorithms dpnding on th application domain; only four tchniqus, that is dcision tr, k-narst nighbour, support vctor machins and random forst, will b discussd. Ths four ar nough to giv radrs an undrstanding of th variations in approachs prsnt in various suprvisd machin larning algorithms takn to classification. Dcision tr: Dcision tr is dfind as as a hirarchical modl basd on nonparamtric thory whr local rgions ar idntifid in a squnc of rcursiv splits in a smallr numbr of stps that implmnts divid-and-conqur stratgy usd in classification and rgrssion tasks [3]. As indicatd in figur 1, th hirarchical structur of th dcision tr is dividd into thr parts that is; root nod,

3 intrnal nods and laf nods. From th prsntd dcision tr of th golf concpt; outlook is th root nod, wind and humidity ar intrnal nods whil ys/no ar laf nods. Th procss starts at th root nod, and is rpatd rcursivly until th laf nod is ncountrd. Th laf nod provids th output of th problm. Outlook Ys Wind Humidity Ys =ovrcast =rain =sunny =fals =tru >77500 <=77500 No Figur 1: Dcision tr for th golf concpt K-Narst Nighbour (KNN): K-Narst Nighbour abbrviatd as KNN is on among th mthods rfrrd to as instanc-basd larning which falls undr th suprvisd larning catgory []. KNN works by simply storing th prsntd training data; whn a nw qury or instanc is fird, a st of similar rlatd instancs or nighbours is rtrivd from mmory and usd to classify th nw instanc [, 8]. Whil classifying, it is oftn usful to tak mor than on nighbour into account and hnc rfrrd to as k-narst nighbour [9]. Th narst nighbours to an instanc ar masurd in trms of th Euclidan distanc, which masurs th dissimilaritis btwn xampls rprsntd as vctor inputs, and som othr rlatd masurs. Howvr, th basis for classifying a nw qury using Euclidan distanc is that, instancs in th sam group ar xpctd to hav a small sparating distanc compard to instancs that fall undr diffrnt groups. Support Vctor Machin (SVM): is a rlativly nw machin larning tchniqu proposd by Vladimir Vapnik and collagus at AT&T Bll laboratoris in 199 and it rprsnts th stat of th art in machin larning tchniqus.th gnral ida of th SVM is to find sparating hyprplans btwn training instancs that maximiz th margin and minimiz th classification rrors [10]. Margin or somtims rfrrd to as gomtric margin is rfrrd to as th distanc btwn th hyprplans sparating two classs and th closst data points to No Ys th hyprplans [11]. Th SVM algorithm is capabl of working with both linarly and nonlinarly sparabl problms in classification and rgrssion tasks. Random Forsts: Briman [1] dfins a random forst as a classifir consisting of a collction of tr-structurd classifirs {h(x), Qk, k=1 } whr th {Qk} ar indpndnt idntically distributd random vctors and ach tr casts a unit vot for most popular class at input x.this tchniqu involvs th gnration of an nsmbl of trs that vot for th most popular class [1]. Although thr ar svral suprvisd machin larning tchniqus, random forst has two distinguishing charactristics; firstly, th gnralisation rror convrgs as th numbr of trs in th forst incrass and th tchniqu dos not suffr from ovrfitting [1]. Accuracy of th individual singl trs that mak up a forst nforcs th convrgnc of th gnralisation rrors and hnc improvmnt in classification accuracy. As th main aim of this papr is to idntify th datast thrshold for prformanc stimators in suprvisd machin larning xprimnts thn th nxt sction provids a rviw of th classification valuation mthods. Som of ths mthods will b rfrrd in xprimnts sction. 3. Classification valuations Whil assssing and comparing prformanc of on larning algorithm ovr th othr, accuracy and rror rat ar among th common mthods that ar widly usd. Othr valuation factors includ spd, intrprtability, as of programmability and risk whn rrors ar gnralisd [3, 13]. This sction dscribs such valuation mthods that ar usd by machin larning rsarchrs whil comparing and assssing th prformanc of th classification tchniqus. Th authors will also intgrat machin larning and statistics by introducing statistical tsts for th purpos of valuating th prformanc of th classifir and for th comparison of th classification algorithms. Th first part of this sction provids th rviw of machin larning prformanc masurs which includ accuracy, rror rat, prcision, rcall, and f1-scor and ROC analysis. Th scond sction will covr statistical tsts Machin Larning Prformanc Masurs In machin larning and data mining, th prfrrd prformanc masurs for th larning algorithms diffr according to th xprimntr s viwpoint [14]. This is much associatd with th background of th xprimntr as ithr in machin

4 larning, statistics or any othr fild as wll as an application domain whr th xprimnt is carrid out. In som application domains, xprimntrs ar intrstd in using accuracy and rror rat whil to othrs prcision, rcall and f1-scor ar of prfrnc. This sction provids th discussion of th prformanc masurs usd in machin larning and data mining projcts. Accuracy: Kostiantis [15] dfins accuracy as th fraction of th numbr of corrct prdictions ovr th total numbr of prdictions. Th numbr of prdictions in classification tchniqus is basd upon th counts of th tst rcords corrctly or incorrctly prdictd by th modl. As indicatd in tabl 1, ths counts ar tabulatd into a confusion matrix (also rfrrd as contingncy) tabl whr tru classs ar prsntd in rows whil prdictd classs ar prsntd in columns. Th confusion matrix shows how th classifir is bhaving for individual classs. TRUE CLASS Tabl 1: Confusion matrix tabl PREDICTED CLASSES YES NO YES TP FN NO FP TN TP= Tru Positivs FP= Fals Positivs TN= Tru Ngativs FN= Fals Ngativs TP Indicats to th numbr of positiv xampls corrctly prdictd as positiv by th modl. TN Indicats th numbr of ngativ xampls corrctly prdictd as ngativ by th modl FP Indicats th numbr of ngativ xampls wrongly prdictd as positiv by th modl. FN Indicats th numbr of positiv xampls wrongly prdictd as ngativ xampls by th modl. Equation 1 As indicatd in quation 1, accuracy only masurs th numbr of corrct prdictions of th classifir and ignors th numbr of incorrct prdictions. With this limitation, rror rat was introducd to masur th numbr of incorrct prdictions rlating to th prformanc of th classifir. Error rat: As dscribd prviously, rror rat masurs th numbr of incorrct prdictions against th numbr of total prdictions. As for som applications it is of intrst to know how th systm rsponds to wrong answrs. This has bn th motiv bhind th introduction of rror rat [16]. Computationally, in rlation to accuracy, rror rat is just 1- Accuracy on th training and tst xampls [8]. It is an appropriat prformanc masur(s) for th comparison of th classification tchniqus givn balancd datasts. Prcision, rcall and f1-scor ar appropriat prformanc masurs for unbalancd datasts. Equation prsnts th formula of calculating rror rat. Equation As most of th datasts usd in our daily livs ar unbalancd, that is, thr is an imbalancd distribution of classs; thr is a nd of having diffrnt classification valuation factors for diffrnt typs of datasts. Prcision, rcall, f1-scor and ROC analysis ar th mtrics which work wll with unbalancd datasts [17]. Prcision: In th ara of information rtrival (IR) whr datasts ar much unbalancd, prcision and rcall ar th two most popular mtrics for valuating classifirs [17, 18]. Howvr, prcision is usd in many application domains whr th dtction of on class sms to b much mor important than th othr such as in mdical diagnosis, pattrn rcognition, crdit risks and statistics. As indicatd in quation 3, it rprsnts th proportion of slctd itms that th systm got right [17] as th positiv xampls to th total numbr of tru positiv and fals positivs xampls. Equation 3 Rcall: It rprsnts th proportion of th numbr of itms that th systm slctd as th positiv xampls to th total numbr of tru positivs and fals ngativs [17]. Contrary to quation 3, whr fals positiv is usd, rcall, as indicatd in quation 4, uss fals ngativs. Rcall is supposd to b high in ordr to rduc th numbr of positiv xampls wrongly prdictd as ngativ xampls. Equation 4

5 Manning and Schutz [17] argu on th advantag of using prcision and rcall ovr accuracy and rror rat. Accuracy rfrs to things got right by th systm whil rror rat rfrs to things got wrong by th systm. As indicatd in quation 1 and rspctivly, accuracy and rror rat ar not snsitiv to any of th TP, FP and FN valus whil prcision and rcall ar. Howvr, for this bhaviour, thr is a possibility of gtting high accuracy whil slcting nothing. Thrfor, as w ar surroundd by unbalancd datast and th biasnss of th accuracy and rror rat ovr TP, FP and TN valus; accuracy and rror rat ar usually rplacd by th us of prcision and rcall unlss th datast is rally balancd. Additionally, in som applications, thr is a trad-off btwn prcision and rcall. Whr as in slcting a documnt in information rtrival for xampl, on can gt low prcision but vry high rcall of up to 100% [17]. Indd, it is difficult to valuat algorithm with high prcision and low rcall or othrwis. Thrfor, f1-scor, which combins prcision and rcall, was introducd. F1-Scor: It combins prcision and rcall with qual importanc into a singl paramtr for optimization and is dfind as Equation 5 and FP rat (fraction of fals positivs) or 1- spcificity plottd in X axis as prsntd in figur. Whn svral instancs ar plottd on a graph thn a curv known as ROC curv is drawn. Th points on th top lft of th ROC curv hav high TP rat and low FP rat and so rprsnt good classifirs [19]. Equation 6 Tru Positiv Rat (TPR) or snsitivity Equation 7 Tru Ngativ Rat (TNR) or spcificity To compar classifirs w may want to rduc th ROC prformanc to a singl scalar valu rprsnting xpctd prformanc. Th common mthod for rducing th ROC prformanc is to masur th ara undr th ROC curv. Aftr drawing th ROC curvs of diffrnt classifirs, th bst classifir is supposd to b narby top lft of th ROC curv. Figur is an xampl of ROC graph for th comparison of thr classifirs; SLN which is a traditional nural ntwork, SVM and C4.5 ruls Rcivr Oprating Charactristic (ROC) graph Fawctt [18] dfins ROC graph as a tchniqu for visualising, organising and slcting classifirs basd on thir prformanc in a D spac. Dspit having svral dfinitions, Fawctt s dfinition has bn adoptd for this book as it shows dirctly whr th tchniqu is usd and in which spac. Originally concivd during World War II to assss th capabilitis of radar systms, ROC graphs which uss ara undr th ROC curvs abbrviatd as AUC-ROC hav bn succssful applid in diffrnt aras such as in signal dtction thory to dpict hit rat and fals alarm rats, mdical dcision making, mdical diagnosis, xprimntal psychology and psychophysics and in pattrn rcognition [18]. Th diffrnc with th prvious prformanc masurs is that, ROC graphs ar much mor usful for domains with skwd class distribution and unqual classification rror costs [18]. With this ability, ROC graphs ar much mor prfrrd than accuracy and rror rat. ROC graphs ar plottd using two paramtrs; TP rat (fraction of tru positivs) or snsitivity which is plottd on th Y axis Figur : ROC curv for th comparison of thr classifirs [19] 3.. Statistical tsts Th classifirs inducd by machin larning algorithms dpnd on th training st for th masurmnt of its prformanc. Statistical tsts com into play whn assssing th xpctd rror

6 rat of th classification algorithm or comparing th xpctd rror rats of two classification algorithms. Though thr ar many statistical tsts, only fiv approximat statistical tsts for dtrmining whthr on larning algorithm outprforms anothr will b considrd. Ths ar McNmar s tst, a tst of th diffrnc of two proportions, rsampld paird t tst, k-fold cross validatd paird t tst and th 5 x cross validatd paird t tst. McNmar s Tst: Is a statistical tst namd aftr Quinn McNmar (1947) for comparing th diffrnc btwn proportions in two matchd sampls and analysing xprimntal studis. It involvs tsting paird dichotomous masurmnts; masurmnts that can b dividd into two sharply distinguishd parts or classifications such as ys/no, prsnc/absnc, bfor/aftr. Th paird rsponss ar fabricatd in a x contingncy tabl and th rsponss ar tallid in appropriat clls. This tst has bn widly applid in a varity of applications to nam a fw; in markting whil obsrving brand switching and brand loyalty pattrns for th customrs, masuring th ffctivnss of advrtising copy or advrtising a campaign stratgy, studying th intnt to purchas vrsus actual purchas pattrns in consumr rsarch, public rlations, oprational managmnt and organisational bhaviour studis and in halth srvics. Considr th application of McNmar s tst in halth institutions for xampl, whr spcific numbr of patints is slctd at random basd on thir visits to a local clinic and assssd for a spcific bhaviour that is classifid as risk factor for lung cancr. Th classification of th risk factor is ithr prsnt or absnt. During thir visits to th clinic thy ar ducatd about th incidnc and associatd risks for lung cancr. Six months latr th patints ar valuatd with rspct to th absnc or prsnc of th sam risk factor. Th risk factor bfor and aftr instructions can b tallid as tabulatd in tabl 3 and valuatd using McNmar s tst. Risk factor aftr Instructio Risk factor bfor Instructions Tabl : Matchd paird data for th risk factors bfor and aftr instructions Whr 00 : Th numbr of patints that shows th prsnc of th risk factor for Rspons 1 and Rspons. 01 : Th numbr of patints shows th absnc of th risk factor for Rspons 1 and th prsnc of th risk factor for Rspons. 10 : Th numbr of patints shows th prsnc of th risk factor for Rspons 1 and th absnc for Rspons. 11 : Th numbr of patints rspondd for th absnc of th risk factor for Rspons 1 and Rspons Rprsnt th total numbr of xampls in th tst st. Undr th null hypothsis th chang in risk factors; from prsnc to absnc and vic vrsa should hav th sam rror rats, which mans [0] For McNmar s, th statistic is as follows Equation 8 x McNmar In a x contingncy tabl with 1 dgr of frdom (1-column x 1-row), that is having on column and on row, th statistic for th McNmar s tst changs to Equation Rspons 1 Rspons Prsnt Absnt 1 x McNmar Th null hypoth sis would idntify that Prsnt Absnt Total Total

7 thr is no significant chang in charactristics btwn th two tims (as in tabl for xampl, bfor and aftr instructions). Thus w will compar our calculatd statistic with a critical x, with 1 dgr of frdom or If th x McNmar 3. 84, th null hypothsis is rjctd and assums a significant chang in th two masurmnts. Evritt [1] commnts on how to apply McNmar tst for th comparison of th classifirs. Having availabl sampl of data S dividd into training st and tsting st, both algorithms A and B ar traind on th training st which rsults in two classifirs P 1 and P. Ths two classifirs ar thn tstd using th tst st. Th contingncy tabl, providd in tabl 1, is usd to rcord how ach xampl has bn classifid. If th null hypothsis is corrct thn, th probability that th valu for th x with 1-dgr of frdom is gratr than 3.84 is lss than 0.05 and th null hypothsis may b rjctd in favour of th hypothsis that th two algorithms hav diffrnt prformanc masurmnts whn traind in a particular training st. Dittrich [0] commnts on th advantag of using this tst compard to othr statistical tst as such Mc Nmar s tst has bn yildd to provid low typ 1 rror. Typ 1 rror mans ability to incorrctly dtct diffrncs whil thr is no diffrnc that xists [0]. Dspit having aformntiond advantags, this tst is associatd with svral problms. Firstly, a singl training st is usd for th comparison of th algorithms and hnc th tst dos not masur th variations du to th choic of th training data. Scondly, Mc Nmar s tst is a simpl holdout tst, whr by having availabl sampl data; tst can b applid aftr th partition of th data into training st and tsting st. For th comparison of th algorithms, th prformanc is masurd using th training data rathr than th whol sampl of data providd. Mc Nmar s tst as a prformanc masur for th comparison of th algorithms from diffrnt application domains is associatd with svral shortcomings. Ths shortcomings hav rsultd into th growth of othr statistical tsts such as a tst for th diffrnc of two proportions, th rsampld t tst, k-fold cross validatd t-tst and 5 x CV paird t tst. A Tst for th Diffrnc of Two Proportions A tst for th diffrnc of two proportions masurs th diffrnc btwn th rror rat of algorithm A and th rror rat of algorithm B [0]. Considr for xampl, P b th proportion of th tst xampls A incorrctly classifid by algorithm A and P B b th proportion of th tst xampls incorrctly classifid by algorithm B, Equation P A, P B Th assumption undrlying this statistical tst is that whn algorithm A classifis an xampl n from tst st th probability of misclassification is P A. Hnc, th numbr of misclassification for n tst xampls is a binomial distribution with man is np A. This statistical tst is associatd with svral problms, firstly as P A and P B ar masurd on th sam tst st, thy ar not indpndnt. Scondly, th tst dos not masur th variations du to th choic of th training st or th intrnal variation of th algorithm. Lastly, this tst suffrs with th sam problm as McNmar tst; dos not masur th prformanc of th algorithm in th whol datast (with all sampl siz) providd; rathr it masurs th prformanc on th smallr training data aftr partition. Th Rsampld Paird t Tst With this statistical tst, usually a sris of 30 trials is conductd, in ach trial, th availabl sampl data is randomly dividd into training st of spcifid siz and tsting st [0]. Larning algorithms ar traind on th training st and th rsulting classifirs ar tstd on th tst st. Considr, P A and PB b th proportion of tst xampls misclassifid by algorithm A and algorithm B rspctivly. For th 30 trials w will rsult into having 30 diffrncs Equation 11 i i) P P P ( ( i) A B [0] Among th potntial drawbacks of this approach is, i th valu of th diffrncs ( P ) ar not indpndnt bcaus th training and tsting sts in th trials ovrlap. Th k-fold cross validatd Paird t tst Th k-fold cross validatd paird t tst was introducd to ovrcom th problm undrlind by th rsampld paird t tst that is, ovrlapping of th trials. This tst works by dividing th sampl siz into k disjoint sts of qual siz T 1 Tk and thn k

8 trials ar conductd. In ach trial, th tst st is T i and th training st is th union of all th othr sts. This approach is advantagous as ach tst st is indpndnt of th othrs. Howvr this tst suffrs from th problm that th training data ovrlap [0]. Considr for xampl, whn k=10, in a 10-fold cross validation, ach pair of th training st shars 80% of th xampls [3]. This ovrlapping bhaviour may prvnt this statistical tst from obtaining a good stimat of th variation that would b obsrvd if ach training st wr compltly indpndnt of th prvious training sts. Th 5 x cross validatd Paird t Tst With this tst, 5 rplications of th twofold cross validation ar prformd [3]. In ach rplication, th availabl data ar partitiond into two qual sizd sts, lt s say S 1 and S. Each larning algorithm is traind on on st and tstd on th othr st and this rsults into four rror stimats as shown in figur 3. Th choic of th numbr of rplications is not th rsponsibility of th xprimntr; this is how th tst rquirs. Th tst allows th applications of only fiv rplications in a twofold cross validation as xploratory studis shows that, th us of mor or lss of fiv rplications incrass th risk of typ I rror which is supposd to b low for th bttrmnt of th tst [0]. This tst has on disadvantag, in ach fold th training st quals th tsting st and hnc rsults into larning algorithms to b traind in training sts half th siz of th whol training sts [0]. For bttr prformanc of th larning algorithm, training st is supposd to b largr than th training st. Figur 3: 5 x cross validation (Adaptd from [3]) 3.3. Prformanc stimation mthods In this subsction w rviw thr prformanc stimation mthods namly, hold-out mthod, k-fold cross validation and lav on out mthod. Ths mthods ar usd to stimat th prformanc of th machin larning algorithms. Hold out mthod: Th holdout or somtims calld tst st stimation [13] works by randomly dividing data into two mutually xclusiv substs; training and tsting or holdout st [, 3]. As shown in figur 4, two-third (/3) of all data is commonly dsignatd for th training and th rmaining onthird, 1/3, for tsting th classifir. Th holdout mthod is rpatd k tims and th accuracy is stimatd by avraging th accuracis obtaind from ach holdout []. Howvr, th mor instancs lft out for tst st, th highr th bias of th stimat []. Additionally, th mthod maks an infficint us of data which inhibits its application to small sampl sizs [14]. Figur 4: Procss of dividing data into training st and tsting st using th holdout mthod (Sourc: Authors) K-Fold Cross Validation: With K-fold cross validation, th availabl data is partitiond into k sparat sts of approximatly qual siz [13]. Th cross validation procdur involvs k itrations in which th larning mthod is givn k-1 as th training data and th rst usd as th tsting data. Itration lavs out a diffrnt subst so that ach is usd as th tst st onc [13]. Th cross-validation is considrd as a computr intnsiv tchniqu, as it uss all availabl xampls in th datast as training and tst sts [14]. It mimics th us of training and tst sts by rpatdly training th algorithm k tims with a function1/k of training xampls lft out for tsting purposs. It is rgardd as th kind of holdout tst stimat. With this stratgy of k-fold cross validation, it is possibl to xploit much largr datast compard to lav-on out mthod. Howvr, sinc th training and tsting is rpatd k tims with diffrnt parts of th original datast, it is possibl to avrag all tst rrors (or any prformanc masur usd) in ordr to obtain a rliabl stimat of th modl prformanc on th nwly tst data [4].

9 Lav on out cross validation: It is also rfrrd to as n fold cross validation whr n is th numbr of instancs [5]. For instanc, givn th datast with n cass, on obsrvation is lft out for tsting and th rst n-1 cass for training [6]. Each instanc is lft out onc and th larning algorithm is traind on all th training instancs. Th judgmnt on th corrctnss of th larning algorithm is basd on th rmaining instancs. Th rsults of all n assssmnts, on for ach instanc, ar avragd and th obtaind avrag rprsnts th final rror stimat of th classification algorithm. Th mthod is attractiv as thr is a gratst possibl amount of data which is usd for training in ach cas, this incrass th possibility of having accurat classifir [5]. Additionally, th mthod tnds to simplify rptition which is prformd in k- fold cross validation (rpatd 10tims for 10-fold cross validation, for xampl) as th sam rsults ar obtaind vry tim. dcid appropriat prformanc stimation mthod for th datast basd on th numbr of instancs. To achiv this, xprimnts will b prformd using on suprvisd machin larning algorithm that is, dcision tr, four datasts with diffrnt sampl sizs (rang from 4177 to 1000 instancs) from UCI machin larning rpository togthr with two prformanc stimation mthods (10 fold cross validation and lav on out). Prformanc of stimation mthods will b masurd using f1-scor. Th xprimnts will b carrid out using an opn sourc machin larning softwar calld RapidMinr. Datasts will b randomly dividd to crat small datasts with diffrnt sampl sizs. Prformanc of stimation mthods will b carrid out for ach randomly cratd datast. Th accuracy of th datast will b obsrvd and will b considrd as th thrshold or th minimum valu for prformanc stimation mthods. Diffrncs in prformanc for th two stimation mthods will thn b analysd and plottd in ordr to idntify which prformanc stimation mthod works bttr than th othr. 4.1 Abalon datast Th first xprimnt involvs th us of th Abalon datast. Th accuracy thrshold btwn th two valus has bn calculatd and is obtaind. Th rsult of this xprimnt is shown in Tabl 3. In figur 6, th lin crossing th two prformanc stimators, with valu , indicats th accuracy thrshold. Analysis from figur 6 howvr, indicats 10 fold CV outwighs lav on out mthod whn datast has 4177 instancs. Figur 5: Procss of randomly slcting a data sampl for us in th tst st with th rmaining data going towards training 4. Exprimnts In this sction xprimnts for idntifying th datast thrshold for prformanc stimators will b prformd and rsults will b prsntd in th nxt sction. Exprimntal stting and mthodology From th rsarch litratur, hold out mthod has bn idntifid to work wll on vry larg datasts, but nothing has bn idntifid for th rmaining two prformanc stimators. As prviously discussd, th main aim of this papr is to dtrmin th datast thrshold for suprvisd machin larning xprimnts. Th stablishd datast thrshold will hlp unfamiliar machin larning xprimntrs to Tabl 3: Rsults for th 10 fold CV and lav on out for th Abalon datast Sampl Siz F1-Scor Diffrnc (f1 scor) 10 fold Lav CV on out

10 Figur 6: Lin graph for th Abalon datast 4.. Contracptiv mthod choic Th scond xprimnt for th stablishmnt of th datast thrshold involvs th datast with 1473 instancs and 10 attributs. Th accuracy thrshold for th prformanc stimation mthods is Rsults hav bn prsntd in tabl 4. From figur 7, it can b concludd that, for th datast with 1473 instancs lav on out mthod is appropriat prformanc stimation mthod compard to 10 fold CV. Tabl 4: Rsults for th 10 fold CV and lav on out stimation for th Contracptiv mthod choic Sampl Siz F1 Scor Diffrnc (f1 scor) 10 Fold Lav CV On Out Figur 7: Lin graph for th contracptiv mthod datast 4.3. Ozon Lvl Dtction Datast Th third datast compris of 536 instancs and 73 attributs. Th accuracy thrshold obtaind for this datast is Rsults hav bn prsntd in tabl 4. From figur 6, it can b concludd that, for datast with 536 instancs, lav on out mthod prforms bttr compars to 10 fold cross validation. Tabl 5: Rsults for th 10 fold CV and lav-on-out stimation for th Ozon layr datast Sampl Siz F1 Scor 10 Fold Lav CV On Out Diffrnc (f1 scor)

11 Figur 8: Lin graph for th Ozon lvl dtction datast 4.4. Intrnt advrtismnt This is th last xprimnt which will dtrmin th datast thrshold for th two prformanc stimators. From th prvious subsctions, xprimnts hav bn prformd for th datast with 4177, 536 and 1473 instancs and th prformanc stimators obtaind ar k-fold cross validation for th first datast whil th othr two, lav-on-out cv has bn idntifid as th appropriat prformanc stimation mthod. This xprimnt involvs th us of th datast with instancs that li btwn th obtaind rsults. This datast contains 379 instancs and 1558 attributs. Figur 9: Lin graph for th Intrnt advrtismnt datast 5. Datast thrshold rsult As prviously discussd, th principal aim of prforming ths xprimnts was to stablish numbr of instancs which can rsult into th classification of th datast as small, mdium or larg. Howvr, from th prvious litratur, hold out mthod has bn idntifid to work wll with larg datasts but nothing has bn don for k-fold cross validation and lav on out mthod. Summary of th rsults from th xprimnts prformd is prsntd in tabl 7. Tabl 6: Rsults for th 10 fold CV and lav-on-out stimation for th Intrnt advrtismnt datast Sampl Siz F1 Scor 10 Fold CV Lav On Out Diffrnc (f1 scor) From rsults indicatd in tabl 6 and figur 9 rspctivly, thr is no any diffrnc btwn th two prformanc stimation mthods. Tabl 7: Numbr of instancs vrsus prformanc stimation mthod Sampl Siz (numbr of instancs) 4177 k-fold CV 379 Nutral 536 Lav on out 1473 Lav on out From tabl 7, with 4177 instancs k-fold cross validation outwighs lav on out mthod and this mans for this numbr of instancs, k-fold is th appropriat mthod. With 536 and 1473 instancs, both ar supportd by lav on out mthod. Th thrshold is obtaind whn th numbr of instancs is 379. Thrfor, for th unfamiliar machin larning xprimntrs, th datast thrshold btwn lav on out and k-fold cross validation is 379. Figur 10 rprsnts datast thrshold with appropriat prformanc stimation mthod.

12 Figur 10: Datast thrshold rsults 6. Conclusions In this papr w hav prsntd rsults from th xprimnts prformd in ordr to stablish th datast thrshold for th prformanc stimation mthods. From th xprimnts prformd, th thrshold has bn idntifid whn th datast has 3179 instancs whr by th diffrnc btwn th two mthods is 0. Th stablishmnt of th datast thrshold will hlp unfamiliar suprvisd machin larning xprimntrs such as studnts studying in th fild to catgoris datasts basd on th numbr of instancs and attributs and thn choos appropriat prformanc stimation mthod. 7. Rfrncs [1] I. H. Wittn and E. Frank, Data Mining: Practical Machin Larning Tools and Tchniqus: Morgan Kauffman, 005. [] T. Mitchll, Machin Larning: MIT Prss, [3] Alpaydin Ethm, Introduction to Machin Larning. Cambridg, Massachustts, London, England: MIT Prss [4] T. Mitchll, "Th Disciplin of Machin Larning," Carngi Mllon Univrsity, Pittsburgh, PA, USA, 006. [5] U. Fayyad, G. Piattsky-Shapiro, and P. Smyth, "Th KDD procss for xtracting usful knowldg from volums of data," Communications of th ACM, vol. 39, pp. 7-34, [6] L. Rokach and O. Maimon. Part C, "Topdown induction of dcision trs classifirs - a survy.," Applications and Rviws, IEEE Transactions on Systms, Man, and Cybrntics, vol. 35, pp , 005. [7] T. Calli and W. F. Bischof, Machin Larning and Imag Intrprtation. York, NY, USA: Plnum Prss, [8] J. Han and M. Kambr, Data Mining: Concpts and Tchniqus: Kauffman Prss., 00. [9] P. Cunningham and S. J. Dlany, "K- Narst Nighbour Classifirs," vol. 008, 007. [10] C. Campbll, An Introduction to Krnl Mthods, 000. [11] M. Brthold and D. J. Hand, Intllignt Data Analysis," 003. [1] L. Briman, "Random Forsts," 001. [13] M. W. Cravn, "Extracting Comprhnsibl Modls from Traind Nural Ntworks." [14] Y. Bngio and Y. Grandvalt, "No Unbiasd Estimator of th Varianc of KFold Cross-Validation," Journal of Machin Larning Rsarch, vol. 5, pp , 004. [15] S. Kostiantis, "Suprvisd Machin Larning: A Rviw of Classification Tchniqus," Informatica, vol. 31, pp , 007. [16] J. Mna, "Data Mining Your Wbsit," [17] C. D. Manning and H. Schütz, Foundations of statistical natural languag procssing: MIT Prss, [18] T. Fawctt, "ROC Graphs: Nots and Practical Considrations for Data Mining Rsarchrs," 004. [19] J. Winklr, M. Niranjan, and N. Lawrnc, Dtrministic and Statistical Mthods in Machin Larning: Birkhausr, 005. [0] T. G. Dittrich, "Approximat Statistical Tsts for Comparing Suprvisd Classification Larning Algorithm," pp , [1] B. Evritt, Th Analysis of Contingncy Tabls: Chapman & Hall/CRC, 199. [] R. Kohavi, "A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Modl Slction," pp , [3] E. Michli-Tzanakou, Suprvisd and Unsuprvisd Pattrn Rcognition: CRC Prss, [4] O. Nlls, "Nonlinar Systm Idntification," 001. [5] I. H. Wittn and E. Frank, Data Mining: Morgan Kauffman, 000. [6] Y.. a. Tang, "Granular Support Vctor Machins for Mdical Binary Classification Problms," pp , 004.