Resampling Methods for the Area Under the ROC Curve

Transcription

1 Resamplng ethods for the Area Under the ROC Curve Andry I. Bandos Howard E. Rockette Department of Bostatstcs, Graduate School of Publc Health, Unversty of Pttsburgh, Pttsburgh, PA, U.S.A Davd Gur Department of Radology, School of edcne, Unversty of Pttsburgh, Pttsburgh, PA, U.S.A Abstract Recever Operatng Characterstc (ROC) analyss s a common tool for assessng the performance of varous classfcaton tools ncludng bologcal markers, dagnostc tests, technologes or practces and statstcal models. ROC analyss ganed popularty n many felds ncludng dagnostc medcne, qualty control, human percepton studes and machne learnng. The area under the ROC curve (AUC) s wdely used for assessng the dscrmnatve ablty of a sngle classfcaton method, for comparng performances of several procedures and as an obectve quantty n the constructon of classfcaton systems. Resamplng methods such as bootstrap, ackknfe and permutatons are often used for statstcal nferences about AUC and related ndces when the alternatve approaches are questonable, dffcult to mplement or smply unavalable. Except for the smple versons of the ackknfe, these methods are often mplemented approxmately,.e. based on the random set of resamples, and, hence, result n an addtonal samplng error whle often remanng computatonally burdensome. As demonstrated n our recent publcatons, n the case of the nonparametrc estmator of the AUC these dffcultes can sometmes be crcumvented by the avalablty of closed-form solutons for the deal (exact) quanttes. Usng these exact solutons we dscuss the relatve merts of the ackknfe, permutaton test and bootstrap n applcaton to a sngle AUC or dfference between two correlated AUCs.. Introducton any dfferent felds are faced wth the practcal problems of detecton of a specfc condton or Key words: ROC, AUC, bootstrap, permutatons, ackknfe, exact varances classfcaton of fndngs the tasks that can be collectvely descrbed as classfcaton of the subects nto categores. The system that defnes the specfc manner of a classfcaton process s termed dfferently dependng on the feld and task at hand (e.g. dagnostc marker, dagnostc system, technology or practce, predctve model, etc.). In ths manuscrpt we wll use the terms classfcaton system or tool to refer to such a system regardless of the feld and the task. Snce the ultmate goal s an applcaton of the classfcaton system to subects from the general target populaton the performance n the target populaton s one of the mportant characterstcs of the classfcaton system. Snce n practce t s usually mpossble to apply the classfcaton system to the whole populaton t s appled to a sample of subects from the target populaton. Based on such a sample the performance of the classfcaton system n the target populaton can be assessed usng statstcal methods. For classfcaton problems, performance s typcally assessed n terms of the multple probabltes of the possble outputs condtonal on the true status of subects (for bnary classfcaton - senstvty or true postve rate and specfcty or false postve rate). ultple probabltes are consdered n order to avod specfcaton of the relatve costs and condtonng on the true class s performed n order to elmnate a dependence on the class dstrbuton wthn the sample. Some classfcaton systems can be supervsed to produce dfferent classfcaton rules. ost commonly such classfcaton systems produce a quanttatve output (e.g. probablty of belongng to a specfc class) and a decson rule s determned by a specfc threshold. Another example s an unlabelled classfcaton tree where a decson rule s determned by a specfc labelng of the termnal nodes (Ferr, Flach, & Hernandez-Orallo 00). For such classfcaton systems an operatng mode (threshold, labelng etc.) s often chosen consderng the class dstrbuton n the target populaton and relatve cost and benefts of the specfc decsons. Because of that, when assessng the performance of the classfcaton system usng a sample from the populaton t s often

2 Resamplng methods for the AUC desrable to have a performance measure that s also ndependent from a specfc operatng mode. For bnary classfcaton tasks (subects are classfed nto the two classes), conventonal ROC analyss provdes a tool to assess the performance of a classfcaton system smultaneously for all operatng thresholds and ndependently of the class dstrbuton n the sample and costs and benefts of varous decsons. The conventonal ROC analyss orgnated n sgnal detecton theory and presently s a wdely used tool for the evaluaton of classfcaton systems (Swets & Pcket, 98; Zhou, Obuchowsk and cclsh, 00; Pepe, 003). The keystone of ROC analyss s the ROC curve whch s defned as a plot of senstvty (true postve rate) versus -specfcty (false postve rate) computed at dfferent possble operatng modes. It llustrates the tradeoff between the two classfcaton rates and enables the assessment of the nherent ablty of a classfcaton system to dscrmnate between subects from dfferent classes (e.g. wth and wthout a specfc dsease or abnormalty). Another benefcal feature of the ROC curve s ts nvarance to monotone transformatons of the data. For example, the ROC curve correspondng to a par of normal dstrbutons representng classfcaton scores (bnormal ROC) s the same as the ROC curve for any par of dstrbuton that s monotoncally transformable to the orgnal par. Because ts constructon requres the probabltes of varous classfcatons condtonal on the true class of the subects, a conventonal Recever Operatng Characterstc (ROC) analyss s only applcable n stuatons where the true class s known for all subects. On the other hand ths feature enables ROC analyss to be used for studes where a fxed number of subects have been selected from each class separately as opposed to takng a sample from the total populaton. Selecton of subects from each class separately elmnates problem resultng from low frequency of a specfc class (e.g. low prevalence of a specfc dsease) and permts more effcent study desgn n regard to statstcal consderatons. Although the ROC curve s qute a comprehensve measure of performance, because t s a whole curve there s often a desre to obtan a smpler summary ndex. Thus, for summarzng the performance of a classfcaton system, more smple ndces such as the area under the ROC curve (AUC), or partal AUC are typcally used. The area under the ROC curve (AUC) s a wdespread measure of the overall dagnostc performance and has a practcally relevant nterpretaton as the probablty of a correct dscrmnaton n a par of randomly selected representatves of each class (Bamber, 975; Hanley & cel, 98). In the presence of a contnuous classfcaton score the AUC s the probablty of stochastc domnance of an abnormal class versus normal class, where abnormal class s expected to have greater scores on average. The AUC s used for assessng the performance of a sngle classfcaton system, comparng several systems and as an obectve quantty for constructng a classfer (Verrelst et al 998; Pepe & Tompson 000; Ferr, Flach, & Hernandez-Orallo 00; Yan et al 003; Pepe, 006). An assessment of the performance of a sngle or a comparson of several classfcaton systems s often ntated by computng the AUCs from the sample selected from the target populaton ( sample AUC ). Snce the performances n the sample mght dffer from that n the target populaton, nferences about the populaton performance should ncorporate assessment of the sample-related uncertanty. A common approach to evaluate the sample-related uncertanty s to estmate the varance of the AUC estmator. The varance estmator can than be used to place confdence ntervals, test hypothess or plan future studes. When comparng two classfcaton systems, an attempt s often made to control for varablty by desgn. amely, the data s collected under a pared desgn where the same set of subects s evaluated under dfferent classfcaton systems, reducng the effect of heterogenety of the samples of subects. On the one hand the pared desgn leads to correlated estmators of the AUCs, requrng specfc analytc methods, but on the other hand, smlar to the pared t-test, because of the completely pared structure the varance for the dfference of the correlated AUCs can be obtaned from the varance of a sngle AUC by drect substtuton. any nonparametrc estmators of the varance of a sngle AUC and the dfference between two correlated AUCs have been proposed. The methods proposed by Bamber n 975 (based on formula from oether 967) and Weand, Gal & Hanley (983) provde unbased estmators of the varance of a sngle AUC and the covarance of two correlated AUCs correspondngly. Hence, these estmators are useful for assessng the magntude of the varablty but may provde no advantages n hypothess testng. The estmator proposed by Hanley & cel (98) explctly depends only upon the AUC and sample sze and thus enables smple estmaton of the sample sze for a planned study. However, ths estmator s known to underestmate or overestmate varance dependng on the underlyng parameters (Obuchowsk 994; Hanley & Haan-Tlak 997) and thus s not optmal for ether varance estmaton or hypothess testng (an mproved estmator of the same knd was proposed by Obuchowsk n 994). Perhaps the most wdely used estmator whch offers both relatvely accurate estmator of the varablty and leads to acceptable hypothess testng s the estmator proposed by DeLong, DeLong and Clarke-Pearson (988). Ths estmator possesses an upward bas whch on the one hand results n an mproved (compared to the unbased estmator) type I error of the statstcal test for equalty of the AUCs when AUCs are small, but on the other hand results n loss of statstcal power when AUCs are large (Bandos 005; Bandos, Rockette & Gur 005).

3 Resamplng methods for the AUC Absence of a unformly superor method, potentally poor small-sample propertes of the asymptotc procedures; complexty or unavalablty of the varance formulas for generalzed ndces (such as for AUC extensons for clustered, repeated and mult-class data) have lead many nvestgators to suggest usng the resamplng methods such as ackknfe, bootstrap and permutatons n applcatons to the AUC and ts extensons (Dorfman, Berbaum & etz, 99; ossman 995; Song, 997; Beden, Wagner, & Campbell, 000; Emr et al, 000; Rutter, 000; Hand & Tll, 00; akas & Yannoutsos 004; Bandos, Rockette, & Gur, 005, 006a,b). Because of the varety of methods for assessng varablty of a sngle AUC estmate or comparng several AUCs t s mportant to know ther relatve advantages and lmtatons. Prevously we developed a permutaton test for comparng AUCs wth pared data, constructed a precse approxmaton based on the closed-form soluton for the exact permutaton varance and nvestgated ts propertes relatve to the conventonal approach (Bandos et al 005). The closed-form solutons for the exact (deal) resamplng varances that we derved n that as well as n our other works permt a better understandng of the relatonshps and relatve advantages of resamplng procedures and other methods for the assessment of AUCs (Bandos 005; Bandos et al. 006b). In ths paper we dscuss the relatve merts of the ackknfe, bootstrap and permutaton procedures appled to a sngle AUC or dfference between two correlated AUCs.. Prelmnares We assume that the true class ( normal or abnormal ) s unquely determned and known for each subect. Hence, accordng to the true status, every subect n the populaton can be classfed as normal or abnormal. We term the ordnal output of the classfcaton as the subect s classfcaton score and denote x and y as scores for normal and abnormal subects correspondngly. Furthermore, wthout loss of generalty, we wll assume that hgher values of the scores are assocated wth hgher probabltes of the presence of abnormalty. Jackknfe s a smple resamplng approach that s often attrbuted to Quenoulle (949) and Tukey (958). any dfferent varetes of the ackknfe can be mplemented n practce. The performance of several of them n hypothess testng about AUC was consdered by Song (997). Although often forgotten, the varance estmators used n the procedure proposed by the DeLong et al. (989) s also a ackknfe varance estmator for the twosample U-statstcs (Arvesen, 969). Ths procedure, whch we wll often term as two-sample ackknfe, s perhaps the most commonly used nonparametrc method for comparng several correlated AUCs. In a more complex mult-reader settng a conventonal one- The general layout of the data we consder conssts of scores assgned to samples of normal and abnormal subects by each of the classfcaton systems. We enumerate subects wth subscrpts, k (for normal);, l (for abnormal). Thus, x denote the classfcaton scores assgned to the th normal and th abnormal subects. When operatng wth more than one classfcaton system we dstngush between them wth m the superscrpt m (e.g. x ). However, when the dscusson concerns prmarly a sngle-system settng we omt the correspondng ndex for the sake of smplcty. Usng the conventons defned above, the nonparametrc estmator of the AUC or sample AUC (equvalent to the Wlxocon-ann-Whtney statstc) can be wrtten as: ψ ( x ) ψ Aˆ ψ () ψ where the order ndcator, ψ, s defned as follows: ψ ψ ( x y ) 0, x < y x y x > y Also, the dot n the place of the ndex n the subscrpt of a quantty denotes summaton over the correspondng ndex; and the bar over the quantty, placed n addton to the dot n the subscrpt, denotes the average over the doted ndex. Under a pared desgn, the dfference n AUCs can be wrtten as: Aˆ Aˆ where w w [ ψ ( x ) ψ ( x )] ( x, y ) ψ ( x ) ψ ( x ) ψ ψ w w Ths representaton llustrates that the dfference n areas under a pared desgn has the same structure as the sngle AUC estmator () and allows one to modfy expressons derved for a sngle AUC to those for the AUC dfference smply by replacng ψ wth w. 3. Resamplng approaches Resamplng approaches such as ackknfe, bootstrap, permutatons and combnaton thereof are wdely used whenever conventonal solutons are questonable, dffcult to derve or unavalable. aor advantages of these methods nclude offerng relable statstcal nferences n small sample problems and crcumventng the dffcultes of dervng the statstcal moments of complex summary statstcs. 3. Jackknfe () (3) (4)

4 Resamplng methods for the AUC sample ackknfe was employed by Dorfman, Berbaum & etz (99) wthn an AOVA framework. The general dea of the ackknfe s to generate multple samples from the sngle orgnal one by elmnatng a fxed number of observatons. The ackknfe samples are then used as a base for calculaton of the pseudo-values of a summary statstc, that are later used for nferental purposes. Snce the nonparametrc estmator of the AUC s an unbased statstc, the one-sample and two-sample ackknfe estmator (averages of the pseudovalues) are equal to the orgnal one. Thus, the dfference n these ackknfe approaches occurs n the varances. A onesample ackknfe computes the varablty of the pseudovalues regardless of the class of the elmnated subect whle the two-sample ackknfe computes a stratfed varance. Both varances can be expressed n a closed-form and thus permt an easy comparson of these (Bandos 005). amely, the two-sample ackknfe varance for the AUC (DeLong et al) can be wrtten as: V J ( A) ( ψ ) ( ) ψ ψ ψ ( ) ( ) A one-sample ackknfe varance has the followng form: V J ( A) ( ψ ψ ) ( ψ ψ ) ( ) ( ) (5) (6) A straghtforward comparson of formulas (5) and (6) reveals that a one-sample ackknfe varance s always larger than the two-sample one. Ths fact lmts the usefulness of a one-sample varance snce the two-sample ackknfe varance s already greater than the Bamber- Weand unbased estmator and thus has an upward bas (Bandos 005). Although the ackknfe approach s straghtforward to mplement and possesses good asymptotc propertes, t s generally consdered to be nferor compared to more advanced resamplng technques such as bootstrap. In applcaton to the dfference between AUCs the bootstrap varance estmator was also found to have lower mean squared error than the ackknfe (Bandos, 005). However, under certan condtons the ackknfe can be consdered as a lnear approxmaton to the bootstrap (Efron & Tbshran, 993) and for some problems the ackknfe mght result n a statstcal procedure that s practcally ndfferent from the bootstrap-based one. 3. Bootstrap A good summary of the general bootstrap methodology can be found n the book by Efron & Tbshran (993). In ROC analyss bootstrap s commonly used for estmaton of varablty or for constructon of confdence ntervals. In recent years t has ganed ncreased popularty n connecton wth ts ablty to obtan nsght nto the components of the varablty of the ndces estmated n mult-reader data (Beden, Wagner & Campbell, 000). The bootstrap was also proposed to be used for estmaton of the varance of the partal AUC (Dodd & Pepe, 003b), varance of the AUC computed from patent-clustered (Rutter, 000) and repeated measures data (Emr et al., 000). The concept of the bootstrap s to buld a model for the populaton sample space from the resamples (wth replacement) of the orgnal data. The nonparametrc bootstrap completes the formaton of the bootstrap sample space by assgnng equal probablty to all bootstrap samples. ext, a value of the summary statstc (called ts bootstrap value) s calculated from every bootstrap sample and the set of all bootstrap values determnes a bootstrap dstrbuton. Such a bootstrap dstrbuton of the summary statstc s a nonparametrc maxmum lkelhood estmator of the dstrbuton of the statstc computed on a sample randomly selected from a target populaton and serves as the bass for the bootstrap estmators of dstrbutonal parameters. Snce, even for a moderately szed problem, t may not be computatonally feasble to draw all possble bootstrap samples, the conventonal approach s to approxmate the bootstrap dstrbuton by computng the bootstrap values correspondng to a random sample of the bootstrap samples. Such a procedure s often called onte Carlo or approxmate bootstrap and the quanttes computed from an approxmate bootstrap dstrbuton are called onte Carlo bootstrap estmators n contrast to the quanttes of the exact bootstrap dstrbuton whch are called deal bootstrap estmators. The onte Carlo bootstrap mght stll be computatonally burdensome and also leads to an addtonal samplng error n the resultng estmators. Some summary statstcs permt crcumventng the drawbacks of the onte Carlo approach by allowng computaton of deal (exact) bootstrap quanttes drectly from the data. Unfortunately, the exact bootstrap varance s rarely obtanable except for smple statstcs such as the sample mean. Some other estmators for whch the exact bootstrap moments have been derved nclude sample medan (artz & Jarret, 978) and L-estmators (Hutson & Ernst, 000). In our recent work (Bandos 005; Bandos, Rockette & Gur, 006b) we have shown that the nonparametrc estmator of the AUC permts the dervaton of the analytcal expresson for the deal bootstrap varance for several commonly used data structures (the bootstrap expectaton of the AUC s equal to the orgnal estmate). These results not only elmnate the need of the onte Carlo approxmaton to the bootstrap of the AUC n exstng methods, but can also be extended to the bootstrap applcatons for the patent-clustered data, repeated measure data, partal areas and potentally to a

5 Resamplng methods for the AUC mult-class AUC extenson (Hand & Tll, 00; akas & Yannoutsos, 004). For the sngle AUC the exact bootstrap varance has the followng form: V B ( A) ( ψ ψ ) ( ψ ψ ) ( ψ ψ ψ ψ ) Unfortunately, there s no unform relatonshp between the bootstrap varance and that of any of the consdered ackknfe varances. The onte Carlo nvestgatons ndcate that the bootstrap varance has unformly smaller mean squared error. It also has a smaller bas except for very large AUC. Thus, the bootstrap often provdes a better estmator of the varablty than the ackknfe. However, the estmator of Bamber (975) and Weand et al. (983), because of ts unbasedness, mght be preferred by some nvestgators. Although the nonparametrc bootstrap s a powerful approach that produces nonparametrc maxmum lkelhood estmators, t s not unformly the best resamplng technque. Davson & Hnkley (997) ndcate that for herarchcal data a combnaton of resamplng wth and wthout replacement may better reflect the correlaton structure n the general populaton. Furthermore, although the bootstrap can be mplemented for a broad range of problems, n stuatons where there s somethng to permute (e.g. sngle ndex hypothess testng, comparson of several ndces) the permutaton approach may be preferable because of the exact nature of the nferences (Efron & Tbshran, 993). 3.3 Permutatons Permutaton procedures are usually assocated wth the early works of Fsher (935). In ROC analyss permutaton tests have been employed for comparson of the dagnostc modaltes (Venkatraman & Begg, 996; Venkatraman 000; Bandos, Rockette & Gur, 005). Permutaton based procedures are resamplng procedures that are specfc to hypothess testng. Smlar to the bootstrap, a permutaton procedure constructs a permutaton sample space, whch conssts of the equally lkely permutaton samples. The permutaton samples are created by nterchangng the unts of the data that are assumed to be exchangeable under the null hypothess. However, unlke the bootstrap sample space, the permutaton sample space s the exact probablty space of the possble arrangements of the data under the null hypothess gven the orgnal sample. The same permutaton scheme can be used wth dfferent summary statstcs resultng n dfferent statstcal tests. The choce of the summary statstc determnes the (7) alternatves that are more lkely to be detected, but may not affect the null hypothess. In ths respect, permutaton tests are smlar to the tests of trend whch, stll assumng overall equalty under the null hypothess, am to detect specfc alternatves n the complementary hypothess, e.g. a specfc trend (lnear, quadratc). For example, when two dagnostc systems are to be compared wth pared data, the natural permutaton scheme conssts of exchangng the pared unts. Several reasonable permutaton tests are possble under such a permutaton scheme. One of these was developed by Venkatraman & Begg (996) for detectng any dfferences between two ROC curves. For ths purpose the authors used a measure specfcally desgned to detect the dfferences at every operatng pont. In our recent work (Bandos, Rockette & Gur, 005) on a test that s especally senstve to the dfference n overall dagnostc performance we used the dfferences n nonparametrc AUCs as a summary measure. Both of these tests assume the same condton of exchangeablty of the dagnostc results under the null hypothess, but dffer wth respect to ther senstvty to specfc alternatves and the avalablty of an asymptotc verson. amely our permutaton test better detects dfferent ROC curves f they dffer wth respect to the AUC, and t has an easy-to-mplement and precse approxmaton whch s unavalable for the test of Venkatraman & Begg. The avalablty of the asymptotc approxmaton to the permutaton test can be an mportant ssue snce the exact permutaton tests are practcally mpossble to mplement wth even moderate sample szes and the onte Carlo approxmaton to the permutaton test s assocated wth a samplng error. Fortunately, n some cases the asymptotc approxmaton can be constructed by appealng to the asymptotc normalty of the summary statstc and usng the estmator of ts varance, f the latter s dervable. For the nonparametrc estmator of the dfference n the AUC we demonstrated (Bandos, Rockette & Gur, 005) that the exact permutaton varance can be calculated drectly wthout actually permutng the data,.e.: where V ( A A ) ( w ) ( w ) Ω (8) w p, q ψ p q 3 p 3 q ( x ) ψ ( x ) denotes the dfference n the order ndcators computed over the scores combned over the two systems. The avalablty of an analytcal expresson for the exact permutaton varance not only permts constructng an easy-to-compute approxmaton, but also makes such an approxmaton very precse even wth small samples. Because of the restrcton to the null hypothess, the permutaton varance s not drectly comparable to

6 Resamplng methods for the AUC prevously menton estmaton methods whch provde estmators of the varance regardless of the magntude of the dfference. However, the propertes of the statstcal tests can be compared drectly wth onte Carlo and the avalablty of the closed-form soluton for the permutaton varance greatly allevates the computatonal burden of ths task. The comparson of the asymptotc permutaton test wth the wdely used procedure of DeLong et al. ndcate the advantages of the former for the range of parameters common n dagnostc magng,.e. AUC greater than 0.8 and correlaton between scores greater than 0.4 (Bandos et al., 005). 4. Dscusson In ths paper we dscussed the relatve merts of basc resamplng approaches and outlne some recent developments n the resamplng-based procedures focused on the area under the ROC curve. The maor drawbacks of the advanced resamplng procedures are computatonal burden and samplng error. Samplng error results from the applcaton of the onte Carlo approxmaton to the resamplng process, and adds to the uncertanty of the obtaned results. Although allevated by the development of faster computers the computatonal burden can stll be substantal especally n the case of teratvely obtaned estmators such as m.l.e. of AUC (Dorfmann & Alf 969; etz, Herman & Shen 998) or when assessng the uncertanty of the resamplng-based estmators (e.g. ackknfe- or bootstrap-after-bootstrap). In our prevous works we showed that for the nonparametrc estmator of the AUC presented here all of the consdered resamplng procedures permt dervaton of the deal varances drectly avodng mplementaton of the resamplng process or ts approxmaton. Such closed-form solutons greatly reduce computatonal burden, elmnate a samplng error assocated wth the onte Carlo approxmaton to the resamplng varances, permt constructon of precse approxmatons to the exact methods and facltate assessment and comparson of the propertes of varous statstcal procedures based on resamplng. In general ackknfe provdes a somewhat smplstc method that, dependng on the problem, may stll offer valuable solutons. In applcaton to estmaton of the nonparametrc AUC, the two-sample ackknfe s preferable over the one-sample due to a smaller upward bas. Bootstrap s a more elaborate resamplng procedure that provdes nonparametrc maxmum lkelhood estmators by offerng an approxmaton to the populaton sample space. Bootstrap s usually preferred over the ackknfe because of cleaner nterpretaton and sometmes better precson. Explotng a formula for the exact bootstrap varance of the AUC we demonstrated that t provdes an estmator of the varance that s more accurate n terms of the mean squared error than the two-sample ackknfe varance and s often more effcent than the unbased estmator. In the case of comparng two AUCs the asymptotc tests based on the bootstrap and ackknfe varances have very smlar characterstcs. However, for more complex problems the bootstrap may perform better than the ackknfe. The permutatons explore the propertes of the populaton sample space assumng the exchangeablty satsfed under the null hypotheses. For the comparson of the performances under a pared desgn the permutaton test can be consdered as preferable over the bootstrap and ackknfe due to the exact nature of the permutaton nferences. The avalablty of the exact permutaton varance permts constructon of an easy-to mplement and precse approxmaton and facltates nvestgaton of the propertes of the permutaton test. Compared to the two-sample ackknfe asymptotc test for comparng two correlated AUCs, the asymptotc permutaton test was shown to have greater statstcal power for the range of parameter common n dagnostc radology. Although ths paper focuses on the most commonly used summary ndex, AUC, the avalablty of the analytcal expresson for the exact varances s not lmted to ths relatvely smple case. Formulas for deal varances may also appear dervable for other AUC related ndces and for dfferent types of data (mult-reader, clustered, repeated measures and mult-class data) as well as under other, more complex, resamplng schemes or study desgns. Acknowledgments Ths work s supported n part by Publc Health Servce grants EB0006 and EB00694 (to the Unversty of Pttsburgh) from the atonal Insttute for Bomedcal Imagng and Boengneerng (IBIB), atonal Health Insttutes, Department of Health and Human Servces. References Arvesen, J.. (969). Jackknfng U-statstcs. Annals of athematcal Statstcs 40(6), Bamber D. (975). The area above the ordnal domnance graph and the area below the recever operatng characterstc graph. Journal of athematcal Psychology, Bandos, A. (005). onparametrc methods n comparng two ROC curves. Doctoral dssertaton, Department of Bostatstcs, Graduate School of Publc Health, Unversty of Pttsburgh. ( /ETD/avalable/etd /) Bandos, A.I., Rockette, H.E., Gur, D. (005). A permutaton test senstve to dfferences n areas for comparng ROC curves from a pared desgn. Statstcs n edcne 4(8), Bandos, A.I., Rockette, H.E., Gur, D. (006a). A permutaton test for comparng ROC curves n multreader studes. Academc Radology 3,

7 Resamplng methods for the AUC Bandos, A.I., Rockette, H.E., Gur, D. (006b). Components of the bootstrap varance of the areas under the ROC curve. IBS EAR 006, Tampa, FL. Beden, S.V., Wagner, R.F., Campbell, G. (000). Components-of-varance models and multple-bootstrap experments: an alternatve method for random-effects recever operatng characterstc analyss. Academc Radology 7, Davson, A.C., Hnkley, D.V. (997). Bootstrap methods and ther applcaton. Ednburgh: Cambrdge Unversty Press. DeLong, E.R., DeLong, D.., Clarke-Pearson, D.L. (988). Comparng the area under two or more correlated recever operatng characterstc curves: a nonparametrc approach. Bometrcs 44(3), Dodd, L.E., Pepe,.S. (003a). Semparametrc regresson for the area under the recever operatng characterstc curve. Journal of the Amercan Statstcal Assocaton 98, Dodd, L.E., Pepe,.S. (003b). Partal AUC estmaton and regresson. Bometrcs 59, Dorfman, D.D., Alf JrE. (969). axmum lkelhood estmaton of parameters of sgnal detecton theory and determnaton of confdence ntervals ratng-method data. Journal of athematcal Psychology 6, Dorfman, D.D., Berbaum, K.S., etz, C.E. (99). Recever operatng characterstc ratng analyss: generalzaton to the populaton of readers and patents wth the ackknfe method. Investgatve Radology 7, Efron, B. (98). The ackknfe, the bootstrap and other resamplng plans. Phladelpha, PA: Socety for Industral and Appled athematcs. Efron, B., Tbshran, R.J. (993). An ntroducton to the bootstrap. ew York: Chapman & Hall. Emr, B., Weand, S., Jung, S.H., Yng, Z. (000). Comparson of dagnostc markers wth repeated measurements: a non-parametrc ROC curve approach. Statstcs n edcne 9, Ferr, C., Flach, P., Hernandez-Orallo, J. (00). Learnng decson trees usng area under the ROC curve. Proceedngs of ICL-00. Fsher, R.A. (935). Desgn of experments. Olver and Boyd, Ednburgh Hand, D.J., Tll, R.J. (00). A smple generalzaton of the area under the ROC curve for multple class classfcaton problems. achne Learnng 45, Hanley, J.A., cel, B.J. (98). The meanng and use of the Area under Recever Operatng Characterstc (ROC) Curve. Radology 43, Hanley, J.A., Haan-Tlak, K.O. (997). Samplng varablty of nonparametrc estmates of the areas under recever operatng characterstc curves: an update. Academc Radology 4, Hutson, A.D., Ernst,.D. (000). The exact bootstrap mean and varance of an L-estmator. Journal of the Royal Statstcal Socety, Seres B (Statstcal ethodology) 6(), artz, J.S., Jarrett, R.G. (978). A note on estmatng the varance of the sample medan. Journal of the Amercan Statstcal Assocaton 73(36), etz, C.E., Herman, B.A., Shen, J. (998). axmum lkelhood estmaton of recever operatng characterstc (ROC) curves from contnuously dstrbuted data. Statstcs n edcne 7, ossman, D. (995). Resamplng technques n the analyss of non-bnormal ROC data. edcal decson makng 5, akas, C.T., Yannoutsos, C.T. (004). Ordered multpleclass ROC analyss wth contnuous measurements. Statstcs n edcne 3, oether GE. Elements on onparametrc Statstcs. Wley & Sons Inc.: ew York 967. Obuchowsk,.A. (994). Computng sample sze for recever operatng characterstcs studes. Investgatve Radology 9, Obuchowsk,.A. (997). onparametrc analyss of clustered ROC curve data. Bometrcs 53, Pepe,.S., Thompson,.L. (000). Combnng dagnostc test results to ncrease accuracy. Bostatstcs, Pepe,.S. (003). The statstcal evaluaton of medcal test for classfcaton and predcton. Oxford: Oxford Unversty Press. Pepe,.S., Ca, T., Longton, G. (006). Combnng predctors for classfcaton usng the area under the recever operatng characterstc curve. Bometrcs 6, -9. Quenoulle,.H (949). Approxmate tests of correlaton n tme seres. Journal of Royal Statstcal Socety, Seres B, Rutter, C.. (000). Bootstrap estmaton of dagnostc accuracy wth patent-clustered data. Academc Radology 7, Song, H.H. (997). Analyss of correlated ROC areas n dagnostc testng. Bometrcs 53(), Swets, J.A., Pcket, R.. (98). Evaluaton of dagnostc systems: methods from sgnal detecton theory. ew York: Academc Press.

8 Tukey, J.W. (958). Bas and confdence n not qute large samples (abstract). Annals of athematcal Statstcs 9, 64. Venkatraman, E.S., Begg, C.B. (996). A dstrbuton-free procedure for comparng recever operatng characterstc curves from a pared experment. Bometrka 83(4), Venkatraman, E.S. (000) A permutaton test to compare recever operatng characterstc curves. Bometrcs 56, Verrelst, H., oreau, Y., Vandewalle, J., Tmmerman, D. (998). Use a mult-layer perceptron to predct malgnancy n ovaran tumors. Advances n eural Informaton Processng Systems, 0. Weand, H.S., Gal,.., Hanley, J.A. (983). A nonparametrc procedure for comparng dagnostc tests wth pared or unpared data. I..S. Bulletn, 3-4. Yan, L., Doder, R., ozer,.c., Wolnewcz, R. (003). Optmzng Classfer performance va an approxmaton to the Wlcoxon-ann-Whtney Statstc. Proceedngs of ICL-003. Zhou, X.H., Obuchowsk,.A., cclsh D.K. (00). Statstcal methods n dagnostc medcne. ew York: Wley & Sons Inc. Resamplng methods for the AUC