A Wild Bootstrap approach for the selection of biomarkers in early diagnostic trials

Transcription

1 Zapf et al. BMC Medcal Research Methodology 25 5:43 DOI.86/s y RESEARCH ARTICLE Open Access A Wld Bootstrap approach for the selecton of bomarkers n early dagnostc trals Antona Zapf *, Edgar Brunner and Frank Konetschke 2 Abstract Background: In early dagnostc trals, partcularly n bomarker studes, the am often to select dagnostc tests among several methods. In case of metrc, dcrete, or even ordered categorcal data, the area under the recever operatng charactertc ROC curve denoted by AUC an approprate overall accuracy measure for the selecton, because the AUC ndependent of cut-off ponts. Methods: For selecton of bomarkers the ndvdual AUC s are compared wth a pre-defned threshold. To keep the overall coverage probablty or the multple type-i error rate, smultaneous confdence ntervals and multple contrast tests are consdered. We propose a purely nonparametrc approach for the estmaton of the AUC s wth the correspondng confdence ntervals and stattcal tests. Th approach uses the correlaton among the stattcs to account for multplcty. For small sample szes, a Wld-Bootstrap approach presented. It shown that the correspondng ntervals and tests are asymptotcally exact. Results: Extensve smulaton studes ndcate that the derved Wld-Bootstrap approach keeps and explots the nomnal type-i error at best, even for hgh accuraces and n case of small samples szes. The strength of the correlaton, the type of covarance structure, a skewed dtrbuton, and also a moderate mbalanced case-control rato do not have any mpact on the behavor of the approach. A real data set llustrates the applcaton of the proposed methods. Concluson: We recommend the new Wld Bootstrap approach for the selecton of bomarkers n early dagnostc trals, especally for hgh accuraces and small samples szes. Keywords: AUC, Dagnostc study, Resamplng, Smultaneous ntervals, Wld bootstrap Background The am of early dagnostc trals, partcularly of bomarker studes, often to select the most promng markers from a canddate set. For convenence, all dfferent knds of dagnostc tests, e.g., magng technques or bomarkers, wll be denoted by dagnostc tests throughout the paper. In these studes, response varables are often not bnary, but measured on a contnuous, dcrete or even ordnal scale and a cut-off value c has not yet been chosen. Therefore, the senstvty.e. true postve proporton and the specfcty true negatve proporton both beng computed based on c cannot be used as selecton crtera. In contrast, the Recever Operatng Charactertc ROC curve llustrates the overall dagnostc *Correspondence: Antona.Zapf@med.un-goettngen.de Department of Medcal Stattcs, Unversty Medcal Center Göttngen, Humboldtallee 32, 3773 Göttngen, Germany Full lt of author nformaton avalable at the end of the artcle performance because t ndependent of the chosen cutoff values see, e.g., DeLong, DeLong and Clark-Pearson []. Because the ROC curve of a dagnostc test nvarant wth respect to any monotone transformaton of the test measurement scale, t an adequate measure for comparng dagnostc tests beng measured even on dfferent scales. The Area Under the ROC-curve AUC represents an accuracy measure whch ndependent from the selected cut-off value c and whch nvarant under any monotone transformaton of the data. Therefore, t an approprate selecton crteron for promng dagnostc tests, and n partcular Xa et al. [2] p. 286 state n ther tutoral about translatonal bomarker dcovery n clncal metabolomcs that the AUC wdely used for performance comparon across dfferent bomarker models. As an example for the evaluaton of dfferent bomarkers we consder the ICM tral by Derchs et al. [3], whch ams 25 Zapf et al.; lcensee BoMed Central. Th an Open Access artcle dtrbuted under the terms of the Creatve Commons Attrbuton Lcense whch permts unrestrcted use, dtrbuton, and reproducton n any medum, provded the orgnal work properly credted. The Creatve Commons Publc Doman Dedcaton waver apples to the data made avalable n th artcle, unless otherwe stated.

2 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 2 of 3 to evaluate the dagnostc accuracy of ntestnal current measurement ICM wth regard to questonable cystc fbros CF. Th study was conducted wth the approval of the local ethcs commttee, MH Hannover, Germany and all patents and/or parents and healthy controls gave ther wrtten nformed consent. In th tral, a total of N = 67 chldren and adults were enrolled. The true dease state of the patents was defned by a composte gold standard, whch consts of typcal CF symptoms plus ether a postve sweat test and/or gene mutatons. By th defnton 26 patents were classfed nto CF referred to as cases and 4 nto CF unlkely referred to as controls. Furthermore, four bomarkers were consdered: I sc,carbachol, I sc,camp/forskoln,and I sc,htamne abbrevated by I carb, I camp,and I hta aswellasthesumof the three measured values, I sum. Boxplots of the data are dplayed n Fgure. In the ROC-curves n Fgure 2 the correspondng estmated AUC s are added. It can be readly seen that the dagnostc accuracy of I carb, I camp,and I hta qute good, and that I sum perfectly dfferentates the cases and the controls. Thus, the remanng queston whch bomarkers have suffcent dagnostc accuracy. There no consensus about the threshold for suffcent dagnostc accuracy. Xa et al. [2] characterze a bomarker wth an AUC <.7 as a qute weak bomarker. In ther study about a bloodbased bomarker panel for stratfyng current rk for colorectal cancer Marshall et al. [4] accept a canddate model wth an AUC >.75 as a predctve model. In contrast, Broadhurst and Kell [5] refer to an AUC >.9 as excellent and to an AUC >.8 as good. Dependng on prevous knowledge or expectatons a threshold for the AUC as ndcator for suffcent dagnostc accuracy should be chosen durng the plannng of the tral. Note that the am of such trals not to test multple hypotheses formulated n terms of AUC dfferences across the bomarkers, but to verfy suffcent dagnostc accuracy for all bomarkers ndvdually. Then comparng the lower lmt of the confdence nterval for the estmated AUC wth th threshold ndcates whether or not the dagnostc test has suffcent dagnostc accuracy. The Gudelne on the choce of the non-nferorty margn of the European Medcnes Agency [6] recommends to demonstrate non-nferorty by use of two-sded 95% or one-sded 97.5% confdence ntervals. Ifseveraldagnostctestsareevaluatednthesametral, t mportant to adjust the confdence ntervals for multplcty. Otherwe there a hgh rk that the accuracy of some dagnostc tests overestmated. Xa et al. [2] p.288 pont out that The probablty of fndng a random assocaton between a gven metabolte and the outcome ncreases wth the total number of comparons. Furthermore they note that the Bonferron correcton a smple but very conservatve method. If the dagnostc tests are repeatedly measured on the same subjects, hence, these Fgure Boxplots of the bomarkers. Boxplot of the four bomarkers n the example, separately for cases and controls.

3 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 3 of 3 Fgure 2 ROC-curves of the bomarkers. ROC-curves of the four bomarkers n the example and correspondng AUC s n the legend. measurements are correlated n general. Therefore t of hghly practcal mportance to take nto account these correlatons n the estmaton of the dagnostc accuracy. The multplcty expert group of the Stattcans n the Pharmaceutcal Industry [7] p.258 states that The partcpants dd, however, agree that for non-nferorty and equvalence trals, compatble smultaneous CIs for the prmary endponts should be presented n all cases.furthermore Strassburger and Bretz [8] recommend the use of sngle-step procedures f the am not to reject as many hypotheses as possble. Therefore we wll confne ourselves to smultaneous confdence ntervals from snglestep procedures whch are compatble wth the results obtaned by hypotheses tests. Among others, Hothorn et al. [9] proposed parametrc smultaneous confdence ntervals, whch correspond to multple contrast tests. However, snce these parametrc approaches are lmted to normally dtrbuted data, Konetschke et al. [] proposed nonparametrc multple contrast tests and compatble asymptotc smultaneous confdence ntervals for relatve treatment effects for ndependent samples based on some theoretcal results developed by Brunner et al. []. In the partcular case of two samples cases and controls the relatve treatment effect equvalent to the AUC see Bamber [2]. In th artcle we wll use th approach n the framework of dagnostc studes, but for pared samples n a multvarate layout. The challenge n early dagnostc trals often that smaller sample szes and hgher AUC s occur. For example nthesystematcrevewofstudesaboutthedagnostc accuracy of pleural flud NT-pro-BNP for pleural effusons of cardac orgn, performed by Janda and Swton [3], the medan total sample sze was 4 mean 2, and the pooled AUC was 98%. Wang et al. [4] reported n another systematc revew about cardac testng for coronary artery dease n potental kdney transplant recpents AUC s between.78 and.92. Kottas et al. [5] found that the Logt tranformaton based confdence nterval for a sngle AUC leads to slghtly conservatve results for small sample szes. Here we suggest Wld Bootstrap based smultaneous confdence ntervals to obtan robust methods for small sample szes and potentally qute large

4 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 4 of 3 AUC s. Hereby, we generalze the method proposed by Arlot et al. [6] for multvarte hgh-dmensonal normal data. In th artcle nonparametrc smultaneous confdence ntervals for multple AUC s n dagnostc studes are presented. Asymptotc ntervals wll be derved as well as ntervals usng the Wld Bootstrap approach. The propertes of these smultaneous ntervals are nvestgated n a smulaton study regardng the type-i error rate and the stattcal power. Furthermore, the results of all ntervals are gven for the example data set presented before n th secton. In the next secton we present the methods, ncludng the stattcal model wth the correspondng hypotheses, and the pont estmators wth ther asymptotc dtrbuton. Furthermore multple contrast tests and correspondng smultaneous confdence ntervals wth or wthout Logt transformaton are derved, and the Wld Bootstrap approach presented n partcular for small sample szes. The results of a smulaton study ncludng robustness evaluatons, and the applcaton of the methods to the example presented above are gven n the Results secton. Fnally, all results are summarzed and dcussed, and a recommendaton gven. Methods Stattcal model and hypotheses We consder a wthn-subject mult-modalty dagnostc tral gven by ndependent and dentcally dtrbuted random vectors X = X,..., X d F, =, control, case; subject s =,..., n, wth margnal dtrbutons X l F l, l =,..., d, 2 where d denotes the number of dagnostc tests. The partton of the data n cases = or controls = based on the gold or reference standard, whch assumed to represent the true dease status of the subjects. In order to allow for contnuous, dcrete or even ordered categorcal data n a unfed way, we use the normalzed verson of the margnal dtrbuton func- tons,.e., F l x = 2 F +,l x + F,l x, where F +,l x = P X l x denotes the rght-contnuous and F,l x = P X l < x denotes the left-contnuous verson of the dtrbuton functon respectvely. In the context of nonparametrc models, the normalzed verson of the dtrbuton functon was frst mentoned by Kruskal [7] and generally dates back to Lévy [8]. Later on, t was used by Ruymgaart [9], Munzel [2], Brunner and Pur [2], Kaufmann et al. [22], among others, to derve asymptotc results for rank stattcs ncludng the case of tes. We note that F l x may be arbtrary dtrbuton functons, wth the excepton of the trval case that both dtrbutons are one-pont dtrbutons see Lange and Brunner [23]. The wthn-subject desgn gven n, whch means that all dagnostc tests are performed n each ndvdual, recommended n the EMA gudelne about dagnostc agents [24] and refers to Desgn n Brunner and Zapf [25]. Foreachofthed dagnostc tests the true AUC gven by AUC l = PX l < Xl +.5 PXl = Xl = F l 3 dfl, l =,..., d. For a convenent dervaton of asymptotc results, the AUC sarecollectednthevectorauc = AUC,..., AUC d. In order to select the most promng dagnostc tests from the canddate set of the d dfferent methods, t our am to test the non-nferorty null hypotheses H : H : d l= d l= { H l : AUC l AUC } { H l : AUC l > AUC } versus wth strong control of the famlywe error rate FWER α smultaneously. The non-nferorty margn AUC assumed to have been fxed durng the plannng phase of the tral. Thus, the set of promng dagnostc tests consts of all markers, whose correspondng AUC l have been declared to be larger than AUC by an adequate multple testng procedure. Pont estmators and asymptotc dtrbuton Unbased and L 2 -constent pont estmators for the AUC s defned n 3 are derved by replacng the unknown dtrbuton functons F l and F l by ther emprcal counterparts F l x = n cx X l, =, ; l =,..., d, n s= where cx denotes the normalzed verson of the count functon,.e. cx {, 2,} correspondng to {x <, x =, x > }, respectvely. The pont estmator ÂUC l = F l l d F = R l. R l. + 5 N 2 can easly be computed usng the means R l. = n of the md- ranks R l, =,. Here, R l n s= Rl 4 denotes the rank of X l among all N = n +

5 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 5 of 3 n observatons X l,..., Xl n, X l l =,..., d. Further let R =,..., Xl n per marker,..., Rd R denote the vectors of the mdranks and let ÂUC = ÂUC,..., ÂUC d denote the vector of the pont estmators. Brunner et al. [] have shown that the vector NÂUC AUC follows, asymptotcally, as N, a multvarate normal dtrbuton wth expectaton and covarance matrx V N = Cov NB, 6 where B = B,..., B d denotes a random vector the components of whch are sums of ndependent random varables B l = n F l n Xl s n s= n s= F l Xl s + 2 AUCl. The covarance matrx V N wth elements v l,m,however, unknown and has to be estmated. Let R l denote the so-called nternal rank of X l among all n observatons X l,..., Xl n for the dagnostc test l n dease status group,andletr = R,..., R d denote the vectors of these nternal ranks. Furthermore, let Z = R R 8 N n denote the vectors of the normed placements F l Xl s = R l s R l s and n F l Xl s = R l s R l s, n respectvely. Then a constent estmator of the covarance matrx gven by V N = N V N, /n + V N, /n,where V N, = n n 7 Z Z. Z Z., =,. 9 s= Here, Z = n n s= Z denotes the vector of means of the normed placements. For more detals we refer to Brunner et al. [] and Kaufmann et al. [22]. Test stattcs and confdence ntervals In order to test the null hypotheses formulated n 4, we frst need to derve an unvarate test stattc for testng the ndvdual null hypothes H l : AUC l AUC. It follows from the asymptotc multvarate normalty of the vector NÂUC AUC that NÂUC l AUC l has, asymptotcally as N, a unvarate normal dtrbuton wth mean and varance v l,l,.e.n, v l,l. Here, v l,l denotes the l-th dagonal element of V N n 6. Hence, by Slutzky s theorem, t follows that T l = ÂUC l N AUC l v l,l D N,, asn, where v l,l denotes the dagonal elements of V N, defned n 9. In partcular, each stattc studentzed wth an ndvdual constent varance estmator and{ thus, the set of hypotheses } and test stattcs = H l, Tl, l =,..., d consttutes a jonttestng famly n the sense of Gabrel [26]. Attenton should be pad to the fact that the estmated varance v l,l equaltozerofâuc l = or.thus,theteststattc T l can not be computed. One possblty to solve th problem to modfy the data slghtly see the analys of the example n the Results secton. A qute conservatve selecton approach can be derved by applyng the Bonferron method denoted as Bonf,.e., the ndvdual null hypothes H l : AUC l AUC wll be rejected at multple level α, ft l z α/d,, where z α/d, denotes the one-sded α/d-quantle of the standard normal dtrbuton. Asymptotc onesded smultaneous confdence ntervals for the treatment effects AUC l are then gven by CI l Bonf = [ÂUC l z α/d, v l,l N ; ]. The global null hypothes H : AUC AUC as defned n 4 wll be rejected, f max{t,..., T d } > z α/d, or, equvalently, f the maxmum of the lower lmts of the confdence ntervals max{ci Bonf,l,..., CId Bonf,l } > AUC.Here =,..., denotes a d-dmensonal vector of s. The Bonferron method, however, a qute conservatve selecton approach see Results secton for more detals. The reason for th that the apparent correlatons among the dfferent pvotal quantttes T,..., T d are not taken nto account by th method. Multple contrast tests and smultaneous confdence ntervals In order to use the correlaton n the selecton approach, t our dea to apply the multple contrast test prncple denoted by MCP, whch uses the correlaton among dfferent test stattcs. The key pont of these procedures to use the jont dtrbuton of a set of stattcs to adjust for multplcty. Thus, the asymptotc multvarate dtrbuton of the vector T = T,..., T d requred. The detals are stated n the next theorem. Theorem. Under the assumpton that N such that N/n N <, =,, the vector T follows, asymptotcally, a multvarate normal dtrbuton

6 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 6 of 3 wth expectaton and correlaton matrx R, wherer = [ r l,m ] l,m=...,d,andr l,m v = l,m. v l,l v m,m The jont dtrbuton of T can be used for the dervaton of a smultaneous test procedure. Let z α, R denote the one-sded α equcoordnate quantle of the multvarate normal dtrbuton wth expectaton and correlaton matrx R,.e., N, R,that d P l= { T l z α, R} = α. For detals see Bretz et al. [27]. Then, the ndvdual null hypothes H l AUCl AUC wll be rejected at multple level α,f T l z α, R. 2 Asymptotc one-sded smultaneous confdence ntervals for AUC l are gven by ] CI l MCP [ÂUC = l v z α, R l,l N ;. 3 The global null hypothes wll be rejected f max{t,..., T d } > z α, R or f max{ci MCP,l,..., CI d MCP,l } > AUC. The correlaton matrx R, however, unknown and must be replaced by a constent estmator R. We propose to replace R by R n the consderatons above, where R =[ r l,m ] l,m=,...,d and r l,m = respectvely. v vl,m l,l v, m,m Smulaton studes ndcate, however, that the speed of convergence of T to a multvarate normal dtrbuton qute slow, partcularly when smaller sample szes and larger numbers of dagnostc tests are consdered. In a varety of applcatons, see e.g. Zou and Yue [28] or Konetschkeetal.[],tturnsoutthattheuseofadequate transformatons e.g., the Logt-transformaton tend to ncrease the speed of convergence. Therefore, smultaneous confdence ntervals wth Logt transformaton wll be derved n the next secton. Multple contrast tests and smultaneous confdence ntervals wth Logt transformaton To derve smultaneous Logt-transformed confdence ntervals let gauc = gauc,..., gauc d :, d R d, denote the vector of Logt-transformed AUC s, where gauc l = log AUC l AUC l Furthermore, let = dag AUC AUC,..., AUC d AUC d. denote the dagonal Jacoban matrx of gauc.underthe addtonal assumpton that N such that N/n f, t follows from Cramer s multvarate δ-theorem see, e.g., Ferguson [29], Theorem 7.4 that N g ÂUC g AUC D N, SN 4 where S N = V N and V N gvenn6.toestmate the asymptotc covarance matrx S N,let = dag ÂUC ÂUC,..., ÂUC d ÂUC d denote the estmated Jacoban matrx of gauc and note that the estmator Ŝ N = V N a constent estmator of S N. Agan there a problem f ÂUC l = or.here, and n turn Ŝ N cannot be calculated. Th problem addressed n the analys of the example n the Results secton. To test the ndvdual hypothes H l : AUC l AUC defne the pvotal quanttes T l = gâuc l N gauc l D N,, ŝ l,l N, l =,..., d, 5 where ŝ l,l denotes the l-th dagonal element of Ŝ N.The jont dtrbuton of the vector T = T,..., T d gvennthenexttheorem. Theorem 2. If N such that N/n f <, then the vector T= T,..., T d follows, asymptotcally, a multvarate normal dtrbuton wth expectaton and correlaton matrx R,where R gvenntheorem. It follows from Theorem 2 that both the vectors T and T have, asymptotcally, as N, the same jont dtrbuton. Both the correlaton matrces of T and T asymptotcally concde due to the dagonal structure of. Now, a smultaneous test procedure, whch takes the correlaton nto account can be derved. The ndvdual null hypothes H l : AUC l AUC wll be rejected at multple level α,f T l z α, R, 6 where z α, R denotes the one-sded equcoordnate quantle of the correspondng multvarate normal dtrbuton where the correlaton matrx R replaced wth the constent estmator R. One-sded smultaneous confdence ntervals for AUC l are then gven by CI l Logt = expt g ÂUC l z α, R ŝ l,l N,, 7

7 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 7 of 3 where expty = expy +expy denotes the nverse Logttransformaton. The global null hypothes H : AUC AUC wll be rejected, f max { T,..., T d} z α, R, orfmax{ci Logt,l,..., CId Logt,l } > AUC.Snce the Logt-functon monotone, the procedure asymptotcally controls the famlywe error rate n the strong sense [26]. Small sample approxmatons wth Wld Bootstrap In the prevous secton approaches for the selecton of dagnostc tests based on the AUC s have been derved. The procedures are based on the asymptotc jont dtrbuton of the vectors T or T, respectvely. The proposed approaches for selecton of dagnostc tests are vald for large sample szes. In order to nvestgate the accuraces of the procedures n terms of controllng the pre-assgned type-i error level under the null hypothes, mantanng the nomnal coverage probablty of the correspondng smultaneous confdence ntervals, and ther powers to detect certan alternatves, extensve smulaton studes were conducted. These smulaton studes ndcate, however, that both the stattcs T n 2 and T n 5 tend to result n lberal or conservatve decons n case of smaller sample szes N and larger AUC AUC.8. The results are n concordance wth the smulaton results proposed for unvarate stattcs by Kottas et al. [5] or Qn and Hotlovac [3]. Therefore, we propose a Wld Bootstrap approach to approxmate ther samplng dtrbutons for small sample szes. Resamplng procedures are wdely known to be qute robust methods, even for small sample szes. However, permutaton methods cannot be used n th setup, snce the dtrbutons of the test stattcs and the resamplng stattcs do not concde, not even asymptotcally Pauly M, Asendorf T, Konetschke F: Permutaton tests and confdence ntervals for the area under the ROC curve, submtted. Smulaton studes ndcate that the use of the conventonal Bootstrap from Efron [3] results n lberal conclusons, partcularly when confronted wth an AUC.7 see Table. Therefore, we dd not further nvestgate the conventonal Bootstrap. In contrast, the Wld Bootstrap approach ensures that the resamplng dtrbuton of the stattcs mmcs the dtrbuton of T Table Emprcal type-i error theoretcal 2.5%of the normal Bootstrap for d = 5 and N = 5 wth varyng case-control-rato and varyng AUC ccr AUC :.68% 2.28% 3.% 5.% 7.8% : 4.9% 2.96% 4.7% 6.4% 2.% and T, asymptotcally. The Wld Bootstrap technque motvated by the resdual bootstrap commonly appled n regresson analys [32-35], and n tme-seres testng problems [36-38]. It also proposed n the context of survval analys [39-42], and wll be explaned n the followng. Let W,..., W n, W,..., W n 8 denote ndependent and dentcally dtrbuted random weghts wth EW = andvarw =, whch are ndependent of the data. We wll nvestgate three dfferent knds of random weghts W n our extensve smulaton study: Rademacher weghts: PW = = PW = = 2. Standard normal weghts: W,..., W n N,. Unform weghts: W,..., W n U Let Z = W Z Z = W Z Z =,, s =,..., n,,..., W Z d [ 2 2, 2 2 Z d, ]. 9 denote N resamplng vectors, where Z gven n 8. Furthermore, let Z = n n k= Z = Z,..., Z d denote ther means and let v l,l = n n s= Z l Z l 2 denote the emprcal varance of Z l,..., Z l n, l =,..., d. In the next theorem t wll be shown that the condtonal resamplng dtrbuton of the vector T = T l = N T,..., T d,where Z l Z l, v l,l /n + v l,l /n 2 mmcs the dtrbuton of both the vectors T and T, asymptotcally. Theorem 3. If N such that N n converges to some fnte constant f, then the condtonal dtrbuton of T gven the data X converges n probablty to the multvarate normal dtrbuton wth expectaton and correlaton matrx R.

8 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 8 of 3 For proof see Addtonal fle. Note that Theorem 3 vald under the null as well as under the alternatve,.e., the resamplng dtrbuton mmcs the dtrbutons of T and T for arbtrary values of AUC = AUC,..., AUC d. Next we wll explan the computaton of the smultaneous confdence ntervals:. Gven the data X, compute the pont estmators ÂUC and V N as gven n 5 and 9, respectvely. 2. Generate N = n + n random weghts W,..., W n as descrbed n 8 3. Compute A j := max{t,..., T d } as gven n Repeat the steps nboot tmes e.g. nboot =, and obtan the values A,..., A nboot. 5a. Compare each A j wth max { T }. Then the ndvdual p-value for H l : AUC l AUC obtaned from nboot nboot j= I{ T l A j },wherei{ } denotes the ndcator functon. 5b. Estmate the quantle z α, R by the one-sded α-quantle z α, of A,..., A nboot to obtan the one-sded α smultaneous confdence ntervals gven by CI l WB = expt g ÂUC l z α, ŝ l,l N,. 2 Results Smulaton results We performed a smulaton study to nvestgate the propertes of the dfferent approaches. All smulatons were conducted wth R envronment, verson R Development Core Team, 2, each wth 5, smulaton runs and 5, bootstrap repettons. The nomnal type- I error was set to 2.5% one-sded and the global null hypothes accordng to 4 was rejected, f at least one of the one-sded p-values was smaller than α = 2.5%. Th means, the famly we error rate n the strong sense FWER controlled, and the one-sded emprcal type- I error should be closed to 2.5%. It also possble to use the correspondng confdence ntervals for decon. Then the global null hypothes rejected f the lower lmt of at least one confdence nterval was above AUC. We generated multvarate normally dtrbuted random vectors wth compound symmetrc correlaton structure and defned the followng scenaro as standard scenaro: a total sample sze N = wth a case-control rato ccr of :, d = 5 dagnostc tests and a correlaton of ρ =.9 between the tests motvated by [2,3,24]; and the example data set. The dfferent parameters and condtons were vared afterwards as follows: The true AUC.5,...,.9 The number of dagnostc tests d 5,, 2 The total sample sze N 5,, 2 The case-control rato ccr :, :2, :4, :9 The true correlaton between the dagnostc tests ρ.3,.6,.9 The covarance structure n the data compound symmetry, unstructured, and dagonal matrx wth heterogeneous varances and postve or negatve parng The dtrbuton of the data normal, skewed = log-normal, ordnal The dfferent parameter constellatons and all smulaton results can be seen n the Addtonal fle 2. Due to computatonal complexty, and ts weak behavor n standard stuatons, we dd not further nvestgate the conventonal Bootstrap n our smulaton study. In a frst step, th standard scenaro was used for the comparon of the three random weghts for the Wld Bootstrap: Rademacher WB-Rade, standard normal WB-Normal and unform WB-Unf weghts. The results are dplayed n the Addtonal fle 3. For an AUC of.5 the three weghts lead to nearly the same emprcal type-i error and are qute conservatve emprcal α.5. For larger AUC s the results are less conservatve and for AUC s above.8 the emprcal type-i error around 2.5%. The Wld Bootstrap approach wth unform weghts, however, more conservatve, whle the standard normal and the Rademacher weghts lead nearly to the same results. Therefore, and to present the smulaton results more clearly, we only consder the standard normal weghts n the followng. The smulaton results for the other weghts are provded n the Addtonal fle 2. In practce often unadjusted wth the local type-i error α equal to the global type-i error α or Bonferron adjusted confdence ntervals for the sngle AUC s are used see for example Shotan et al. [43]. Therefore, n a second step, we compared these approaches agan for the standard scenaro usng the multple contrast test MCP, the smultaneous Logt Logt and the Wld Bootstrap WB-Normal approach. In Fgure 3 t becomes apparent that unadjusted ntervals Unadj lead to hghly lberal conclusons emprcal type-i error 8 9%, whle the Bonferron correcton Bonf too conservatve..5%. Therefore we wll not consder these approaches n the sequel. The MCP approach keeps the type-i error for an AUC of.5, but becomes more and more lberal for larger AUC s up to 4% for AUC =.9. The emprcal type I error of the Logt and the WB- Normal approach comparable and between.5% and 2.9%. In the followng we wll nvestgate the nfluence of

9 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 9 of 3 Fgure 3 Emprcal type-i error for varyng AUC s. Emprcal type-i error of the dfferent approaches for the standard scenaro see text wth varyng AUC s. the dfferent parameter settngs on the type-i error of the Logt and the WB-Normal approach, and also of the MCP approach as the bas of both the approaches despte of ts lberal behavor. The strength of the correlaton, the type of the covarance structure and a skewed dtrbuton do not have any mpact on the behavor of the test see fgures and tables n the Addtonal fles 2, 4, 5, and 6. The mpact of the sample sze N and the number of dagnostc tests d shown n Fgure 4. As expected, for a larger sample sze and a small number of dagnostc tests the type-i error better exploted. As already seen n Fgure 3 the Logt and the WB-Normal approach are comparable f AUC.8 ndependent of N and d. For larger AUC s, the WB-Normal approach leads to a larger emprcal type-i error. On the one hand, th means that α better exploted, on the other hand, th means that the results are lberal. The emprcal type-i error of the Logt approach for AUC =.9 ranges from.3% to 2.%, and of the WB-Normal approach from 2.2% to 2.9%. If the case-control rato ccr not balanced, the emprcal type-i error ncreases wth ncreasng mbalance see Fgure 5. For an AUC of.8 or smaller both approaches are robust to an mbalance up to : 4. For AUC =.9 the lberalty of the WB-Normal approach a dadvantage here, the emprcal type-i error above 2.5%. For a casecontrol rato of : 9, both approaches are far too lberal. Ordnal data was generated usng dcreted normal dtrbutons wth a gven AUC. For th data, representng a 5-pont gradng scale, the emprcal type-i error decreases wth ncreasng AUC AUC =.5: Logt = 2.3%, WB-Normal = 2.2% to AUC =.9: Logt =.7%, WB-Normal =.6%. For detals see Addtonal fle 2. The power was calculated for one example scenaro N = 2, d = 5, ccr = :,ρ =.9, AUC =.7, where the emprcal type-i error of the Logt and of the WB-Normal approach was nearly the same. The true AUC ncreasng from.7 whch equal to AUC to.85, accordng AUC =,...,.5. The power of the two approaches bascally the same. For an AUC of..e. AUC =.8 vs. AUC =.7 the power greater than 8% see Addtonal fle 2. Results for the analys of the example The pont estmators for the AUC s are presented n the Background secton n Fgure 2. The number of 26 cases and 4 controls correspond to a case-control rato of :.6. The Spearman correlaton coeffcents between the bomarkers range from.64 to.95. For I sum the result was AUC=. Because logt =, we modfed the

10 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page of 3 Fgure 4 Emprcal type-i error for varyng N and d. Emprcal type-i error of the MCP, the Logt and the WB-Normal approach for varyng sample sze and number of dagnostc tests. Fgure 5 Emprcal type-i error for varyng ccr. Emprcal type-i error of the MCP, the Logt and the WB-Normal approach for varyng case-control ratos. data for I sum such that we replaced the largest measurement of the controls wth the smallest measurement of the cases. Th mnmal change leads to a pont estmator for the AUC of.9999, and enables us to calculate the confdence ntervals. Th replacement strategy conservatve, snce the effect decreased, and the varance ncreased. The one-sded 97.5% confdence ntervals for all bomarkers usng the MCP, the Logt, and the Wld Bootstrap approach are dplayed n Fgure 6. The results of the Wld Bootstrap wth the three dfferent weghts dffered just n the thrd decmal place. For constency we dplayed the WB-Normal approach here. The pattern of the results the same for all four bomarkers. Accordng to the smulaton results, the MCP ntervals are the shortest, the Logt ntervals are the broadest, and the WB ntervals are n between. In the artcle of Derchs et al. [3] no threshold defned. In Fgure 6 four possble thresholds.8,.85,.9,.95 are marked by sold horzontal lnes. In Table 2 for each of these thresholds the numbers of selected bomarkers, dependng on the ndvdual approach, are lted. Apparently, the Logt approach a more conservatve selecton crteron than the Wld Bootstrap approach. Although the MCP ntervals are clearly shorter than the Wld Bootstrap ntervals, the number of selected bomarkers the same

11 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page of 3 Fgure 6 Confdence ntervals for the bomarkers. One-sded 97.5% confdence ntervals for the four bomarkers usng the MCP, the Logt, and the WB-Normal approach. forthemcpandthewbapproachforthreethresholds. Only for the threshold of.85 the MCP approach would select one bomarker more. Consderng the smulaton results of th secton we would recommend to use the WB-Normal approach. Dcusson It wdely dcussed n the lterature, whether the type- I error should be adjusted for multplcty and whether the Bonferron correcton an approprate approach. Among many others, Wttes [44] states that lack of adjustment can lead to a mnterpretaton of the study results as well as Bonferron adjustment can do. Furthermore Table 2 Number of selected bomarkers of the MCP, the Logt, and the WB-Normal approach for dfferent thresholds based on one-sded 97.5% confdence ntervals Threshold MCP Logt WB-Normal Perneger [45] states that In summary, Bonferron adjustments have, at best, lmted applcatons n bomedcal research, and should not be used when assessng evdence about specfc hypotheses. Nevertheless, n practce often Bonferron adjusted or even unadjusted confdence ntervals for the sngle AUC s are used see for example [43]. Konetschke et al. [] proposed nonparametrc multple contrast tests and smultaneous confdence ntervals for adequate correcton of the type-i error, whch take the dependences wthn the data nto account. Furthermore the authors recommended the transformaton method for example the Logt-transformaton to get less lberal results. However, Qn and Hotlovac [3] notced that the Logt-transformed ntervals are conservatve for hgh accuraces. The reason that the estmator logtâuc qute unstable f ÂUC close to or because of a possbly larger varance. Obuchowsk and Leber [46] compared dfferent confdence ntervals for the AUC and concluded that for small sample szes none of them provdes adequate coverage for hgh accuraces. Concluson InthartclewedervedaWldBootstrapapproach, whch explots the type-i error much better than the Logt-approach, even for hgh accuraces and small

12 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 2 of 3 samples. Nether the strength of correlaton, nor the structure of the covarance matrx, nor a skewed dtrbuton, nor a moderate mbalanced case-control rato has any mpact on th desrable property of the Wld Bootstrap approach. Correspondng to these results we recommend to use the Wld Bootstrap approach wth standard normally dtrbuted weghts for the selecton of bomarkers n early dagnostc trals wth the AUC as selecton crteron. Addtonal fles Addtonal fle : Proof of Theorem 3. Addtonal fle 2: Tables of smulaton results. Addtonal fle 3: Fgure S. Emprcal type-i error of the Wld Bootstrap approach wth the three dfferent weghts for the standard scenaro see artcle, Secton Smulaton results wth varyng AUC s. Addtonal fle 4: Fgure S2. Emprcal type-i error of the MCP, the Logt and the WB-Normal approach for varyng strength of correlaton. Addtonal fle 5: Fgure S3. Emprcal type-i error of the MCP, the Logt and the WB-Normal approach for dfferent covarance structures CS: compound symmetry, UN: unstructured, PP/NP: dagonal matrx wth heterogeneous varances and postve/negatve parng. Addtonal fle 6: Fgure S4. Emprcal type-i error of the MCP, the Logt and the WB-Normal approach for normal and log-normal dtrbuted data. Competng nterests The authors declare that they have no competng nterests. Authors contrbutons AZ, FK, and EB derved the Wld Bootstrap approach. AZ and FK performed the smulaton study and wrote the artcle. EB reved the manuscrpt. All authors read and approved the fnal manuscrpt. Acknowledgements Th work was supported by the Federal Mntry of Educaton and Research [5MMGB]. The authors thank Prof. Ballmann from the DRK-Knderklnk Segen for supportng th work by provdng study data. Author detals Department of Medcal Stattcs, Unversty Medcal Center Göttngen, Humboldtallee 32, 3773 Göttngen, Germany. 2 Department of Mathematcal Scences, The Unversty of Texas at Dallas, 8 W Campbell Road, 758 Rchardson, TX, USA. Receved: 5 October 24 Accepted: 25 March 25 References. DeLong E, DeLong D, Clark-Pearson D. Comparng the areas under two or more correlated recever operatng charactertc curves: a nonparametrc approach. Bometrcs. 988;44: Xa J, Broadhurst D, Wlson M, Whart D. Translatonal bomarker dcovery n clncal metabolomcs: an ntroductory tutoral. Metabolomcs. 23;9: Derchs N, Sanz J, Von Kanel T, Stolpe C, Zapf A, Tümmler B, et al. Intestnal current measurement for dagnostc classfcaton of patents wth quastonable cystc fbros: valdaton and reference data. Thorax. 2;65: Marshall K, Mohr S, Khettab F, Nossova N, Chao S, Bao W, et al. A blood-based bomarker panel for stratfyng current rk for colorectal cancer. Int J Cancer. 2;26: Broadhurst D, Kell D. Stattcal strateges for avodng false dcoveres n metabolomcs and related experments. Metabolomcs. 26;2: EMA. Gudelne on the choce of the non-nferorty margn. Doc. Ref. EMEA/CPMP/EWP/258/ ncludes/document/open_document.jsp?webcontentid=wc53636 date of last access 3/4/5. 7. Phllps A, Fletcher C, Atknson G, Channon E, Dour A, Jak T, et al. Multplcty: dcusson ponts from the stattcans n the pharmaceutcal ndustry multplcty expert group. Pharm Stat. 23;2: Strassburger K, Bretz F. Compatble smultaneous confdence bounds for the Holm procedure and other Bonferron-based closed tests. Stat Med. 28;27: Hothorn T, Bretz F, Westfall P. Smultaneous nference n general parametrc models. Bometrcal J. 28;5: Konetschke F, Hothorn L, Brunner E. Rank-based multple test procedures and smultaneous confdence ntervals. Electron J Stat. 22;6: Brunner E, Munzel U, Pur M. The multvarate nonparametrc Behrens-Fher problem. J Stat Plannng Inference. 22;8: Bamber D. The area above the ordnal domnance graph and the area below recever operatng charactertc graph. J Math Psychol. 975;2: Janda S, Swton J. Dagnostc accuracy of pleural flud NT-pro-BNP for pleural effusons of cardac orgn: a systematc revew and meta-analys. BMC Pulmonary Med. 2;: Wang L, Fahm M, Hayen A, Mtchell R, Banes L, Lord S. Cardac testng for coronary artery dease n potental kdney transplant recpents. Cochrane Database Syst Rev. 2;2:. DOI:.2/ CD869.pub2. 5. Kottas M, Kuss O, Zapf A. A modfed Wald nterval for the area under the ROC curve AUC n dagnostc case-control studes. BMC Med Res Methodology. 24;4: Arlot S, Blanchard G, Roquan E. Some nonasymptotc results on resamplng n hgh dmenson, I: confdence regons. Ann Stat. 2;38: Kruskal W. A nonparametrc test for the several sample problem. Ann Math Stat. 952;23: Lévy P. Calcul des Probabltées. Par: Gauthers-Vllars, Édteurs; Ruymgaart F. A unfed approach to the asymptotc dtrbuton theory of certan mdrank stattcs In: Raoult JP, edtor. Stattque Non Parametrque Asymptotque vol. Lecture Notes on Mathematcs, No. 82. Sprnger, Berln Hedelberg; 98. p Munzel U. Lnear rank score stattcs when tes are present. Stat Probablty Lett. 999;4: Brunner E, Pur M. Nonparametrc methods n factoral desgns. Stat Pap. 2;42: Kaufmann J, Werner C, Brunner E. Nonparametrc methods for analysng the accuracy of dagnostc tests wth multple readers. Stat Methods Med Res. 25;4: Lange K, Brunner E. Senstvty, specfcty and ROC-curves n multple reader dagnostc trals - a unfed, nonparametrc approach. Stat Methodology. 22;9: EMA. Gudelne on clncal evaluaton of dagnostc agents. Doc. Ref. CPMP/EWP/9/98/Rev ncludes/document/open_document.jsp?webcontentid=wc5358 date of last access 3/4/ Brunner E, Zapf A. Nonparametrc ROC analys for dagnostc trals In: Balkrhnan N, edtor. Methods and Applcatons of Stattcs n Clncal Trals vol. Volume 2: Plannng, Analys, and Inferental Methods. Hoboken, New Jersey: John Wley & Sons; 24. p Gabrel K. Smultaneous test procedures - some theory of multple comparons. Ann Math Stat. 969;4: Bretz F, Landgrebe J, Brunner E. Multplcty sues n mcroarray experments. Methods Inf Med. 25;44: Zou G, Yue L. Usng confdence ntervals to compare several correlated areas under the recever operatng charactertc curves. Stat Med. 22;32: Ferguson T. A Course n Large Sample Theory. London: Chapman & Hall; Qn G, Hotlovac L. Comparon of non-parametrc confdence nterval for the area under the ROC curve of a contnuous-scale dagnostc test. Stat Methods Med Res. 28;7: Efron B. Bootstrap methods: Another look at the Jackknfe. Ann Stat. 979;7: 26.

13 Zapf et al. BMC Medcal Research Methodology 25 5:43 Page 3 of Wu C. Jackknfe, Bootstrap and other resamplng methods n regresson analys. Ann Stat. 986;4: Mammen E. When does Bootstrap work? Asymptotc results and smulatons. New York: Sprnger; Beran R. Dagnosng Bootstrap success. Ann Inst Stat Mathematcs. 997;49: Janssen A. Nonparametrc symmetry tests for stattcal functonals. Math Methods Stat. 999;8: Kres J, Paparodt E. Bootstrap for dependent data: a revew, wth dcusson, and a rejonder. J Korean Stat Soc. 2;4: Kres J, Paparodt E. Bootstrappng locally statonary processes. J R Stat Soc - Ser B. 24;77: Konetschke F, Pauly M. Bootstrappng and permutng pared t-test type stattcs. Stat Comput. 24;24: Ln D. Non-parametrc nference for cumulatve ncdence functons n competng rks studes. Stat Med. 997;6:9. 4. Beyersmann J, d Termn S, Pauly M. Weak convergence of the Wld Bootstrap for the Aalen-Johansen estmator of the cumulatve ncdence functon of a competng rk. Scand J Stat. 24;4: Pauly M. Weghted resamplng of martngale dfference arrays wth applcatons. Electron J Stat. 2;5: Dobler D, Pauly M. How to Bootstrap Aalen-Johansen processes for competng rks? Handcaps, solutons, lmtatons. Electron J Stat. 24;8: Shotan A, Murao T, Kmura Y, Matsumoto H, Kamada T, Kusunok H, et al. Identfcaton of serum mrnas as novel non-nvasve bomarkers for detecton of hgh rk for early gastrc cancer. Br J Cancer. 23;9: Wttes J. Clncal trals must cope better wth multplcty. Nat Med. 22;8: Perneger T. What s wrong wth Bonferron adjustments. Br Med J. 998;36: Obuchowsk N, Leber M. Confdence ntervals for the recever operatng charactertc area n studes wth small samples. Academc Radology. 998;5:56 7. Submt your next manuscrpt to BoMed Central and take full advantage of: Convenent onlne submson Thorough peer revew No space constrants or color fgure charges Immedate publcaton on acceptance Incluson n PubMed, CAS, Scopus and Google Scholar Research whch freely avalable for redtrbuton Submt your manuscrpt at