EVALUATION OF DIAGNOSTIC PERFORMANCE USING PARTIAL AREA UNDER THE ROC CURVE. Hua Ma. B.S. Sichuan Normal University, Chengdu, China, 2007

Transcription

1 EVALUATION OF DIAGNOSTIC PERFORMANCE USING PARTIAL AREA UNDER THE ROC CURVE by Hua Ma B.S. Sichuan Normal Univrsity, Chngdu, China, 2007 M.S. Xiamn Univrsity, Xiamn, China, 2010 Submittd to th Graduat Faculty of th Dpartmnt of Biostatistics Graduat School of Public Halth in partial fulfillmnt of th rquirmnts for th dgr of Doctor of Philosophy Univrsity of Pittsburgh 2014

2 UNIVERSITY OF PITTSBURGH GRADUATE SCHOOL OF PUBLIC HEALTH This dissrtation was prsntd by Hua Ma It was dfndd on May 8, 2014 and approvd by Howard E. Rocktt, PhD, Profssor, Dpartmnt of Biostatistics Graduat School of Public Halth, Univrsity of Pittsburgh Jong-Hyon Jong, PhD, Profssor, Dpartmnt of Biostatistics Graduat School of Public Halth, Univrsity of Pittsburgh David Gur, ScD, Profssor, Dpartmnt of Radiology School of Mdicin, Univrsity of Pittsburgh Chung-Chou Chang, PhD, Associat Profssor, Dpartmnt of Biostatistics Graduat School of Public Halth, Univrsity of Pittsburgh Dissrtation Dirctor: Andriy Bandos, PhD, Assistant Profssor, Dpartmnt of Biostatistics, Graduat School of Public Halth, Univrsity of Pittsburgh ii

3 Copyright by Hua Ma 2014 iii

4 Andriy Bandos, PhD EVALUATION OF DIAGNOSTIC PERFORMANCE USING PARTIAL AREA UNDER THE ROC CURVE Hua Ma, PhD Univrsity of Pittsburgh, 2014 ABSTRACT Evaluation of diagnostic prformanc is critical in many filds including but not limitd to diagnostic mdicin. Th Rcivr Oprating Charactristic (ROC) curv is th most widly usd mthodology for dscribing th intrinsic prformanc of diagnostic tsts, with th ara undr th curv (AUC) bing th most commonly usd summary indx of ovrall prformanc. Th partial ara undr th ROC curv (pauc), whn focusd on th rang of practical/clinical rlvanc, is considrd a mor rlvant summary indx than th full AUC. Howvr, svral concptual and analytical difficultis frquntly prvnt th pauc from bing usd. First, in many diagnostic stting th rlvant rang is difficult to dtrmin objctivly. Scond, in thory, du to potntial us of lss information, analysis basd on th pauc could lad to th loss of statistical prcision and thrfor would rquir largr sampl sizs. Through mathmatical drivation, xtnsiv simulation studis and practical xampls, this work invstigats statistical proprtis whn using th pauc. First, this work dmonstrats that in many practical scnarios infrncs basd on pauc could b mor powrful than infrncs basd on th full AUC. Thus, th us of th pauc may lad to not only mor clinically rlvant but also mor conclusiv rsults in analyss of xprimntal data. Scond, this invstigation dmonstrats that th advantags of pauc-basd infrncs dpnd on th shap of ROC curvs. Th convntional binormal modl dos not always adquatly dscrib scnarios whr th pauc is mor iv

5 statistically fficint. Th bi-gamma family of concav ROC curvs is shown to dscrib practically rasonabl scnarios whr ithr pauc or full AUC could b advantagous. Programs for sampl siz stimation basd on bi-gamma modl ar thn dvlopd. Finally, this work invstigats th proprtis of pauc-basd infrncs in scnarios whr diagnostic rsults hav substantial tis (or a mass ) at th lowst diagnostic rsults. For crtain typ of th ROC curvs th xistnc of tis could lad to an incras in statistical fficincy. Forcing a diagnostic systm to rsolv tis could dtrimntally affct rliability and conclusivnss of statistical infrncs. In conclusion, this work provids invstigators with insights into and tools for gnrating practically rlvant conclusions using pauc. Th public halth importanc of this work stms from th rlvanc of th ROC analysis at diffrnt stags of dvlopmnt and rgulatory approval of diagnostic systms in mdicin. Enhancd mthodology for valuation of diagnostic accuracy hlps in th dvlopmnt of improvd diagnostic systms and could acclrat th dlivry and clinical adoption of truly bnficial diagnostic tchnologis and/or clinical practics. v

6 TABLE OF CONTENTS 1.0 INTRODUCTION BACKGROUND OBJECTIVES FACTS RELATED TO THE PRESEARCH FAMILIES OF ROC CURVE, THEIR AUCS AND PAUCS BINORMAL ROC CURVES POWER-LAW ROC CURVES BI-GAMMA ROC CURVES STRAIGHT-LINE ROC CURVES ESTIMATION OF ROC CURVES PARAMETRIC ESTIMATES OF ROC CURVES EMPIRICAL ROC CURVE ESTIMATION OF AUC AND PAUC PARAMETRIC ESTIMATES OF AUC AND PAUC EMPIRICAL ESTIMATES OF AUC AND PAUC ESTIMATION OF VARIANCE OF AUC AND PAUC EVALUATION OF A SINGLE PAUC METHOD STANDARDIZED PARTIAL AUC AND ITS PROPERTIES VARIANCE OF THE PARAMETRIC ESTIMATE OF SPAUC NUMERICAL STUDY EXAMPLES vi

7 3.4 SUMMARY COMPARISON OF TWO CORRELATED PAUCS METHOD NUMERICAL STUDY EXAMPLES SUMMARY PARTIAL AREA UNDER THE ROC CURVE WITH MASS METHOD NUMERICAL STUDY EVALUATION OF A SINGLE PAUC COMPARISON OF CORRELATED PAUC SUMMARY CONCLUSION AND DISCUSSION EVALUATION OF A SINGLE PAUC COMPARISON OF TWO CORRELATED PAUCS PARTIAL AREA UNDER THE ROC CURVE WITH MASS APPENDIX A ON USE OF PARTIAL AREA UNDER THE ROC CURVE FOR EVALUATION OF DIAGNOSTIC PERFORMANCE APPENDIX B ON THE USE OF PARTIAL AREA UNDER THE ROC CURVE FOR COMPARISON OF TWO DIAGNOSTIC TESTS APPENDIX C vii

8 R PROGRAM FOR ESTIMATING SAMPLE SIZES FOR BI-GAMMA ROC CURVES IN EVALUATION OF SINGLE PARTIAL AUC APPENDIX D R PROGRAM FOR ESTIMATING SAMPLE SIZES FOR COMPARISONS OF BI- GAMMA ROC CURVES USING PAUC BIBLIOGRAPHY viii

9 LIST OF TABLES Tabl 3.1 Thortical spauc for binormal ROC curvs with diffrnt b s and full AUCs Tabl 3.2 Varianc of sampling distributions of standardizd pauc for binormal ROC curvs ( 10-3 ) Tabl 3.3 Diffrncs of 2.5% and 97.5% stimatd prcntils of sampling distributions of standardizd pauc for binormal ROC curvs Tabl 3.4 Statistical powr for tsting spauc=0.5 for binormal ROC curvs Tabl 3.5 Sampl siz rquirmnts for two-sidd 95% confidnc intrval for a standardizd pauc to b narrowr than 0.1 whn th ROC curv has a binormal shap Tabl 3.6 Sampl siz rquirmnts for tsting spauc=0.5 whn th ROC curv has a binormal shap Tabl 3.7 Varianc of sampling distributions of standardizd pauc for straight-lin ROC curvs ( 10-3 ) Tabl 3.8 Diffrncs of 2.5% and 97.5% stimatd prcntils of sampling distributions of standardizd pauc for straight-lin ROC curvs Tabl 3.9 Statistical powr for tsting spauc=0.5 whn th ROC curv has a straight-lin shap Tabl 3.10 Sampl siz rquirmnts for two-sidd 95% confidnc intrval for a standardizd pauc to b narrowr than 0.1 whn th ROC curv has a straight-lin shap ix

10 Tabl 3.11 Sampl siz rquirmnts for tsting spauc=0.5 whn th ROC curv has a straightlin shap Tabl 3.12 Thortical valu of spaucs for bi-gamma ROC curvs with diffrnt k s and full AUCs Tabl 3.13 Varianc of sampling distributions of standardizd pauc for bi-gamma ROC curvs ( 10-3 ) Tabl 3.14 Diffrncs of 2.5% and 97.5% stimatd prcntils of sampling distributions of standardizd pauc for bi-gamma ROC curvs Tabl 3.15 Statistical powr for tsting spauc=0.5 whn th ROC curv has a bi-gamma shap Tabl 3.16 Sampl siz rquirmnts for two-sidd 95% confidnc intrval for a standardizd pauc to b narrowr than 0.1 whn th ROC curv has a bi-gamma shap Tabl 3.17 Sampl siz rquirmnts for tsting spauc=0.5 whn th ROC curv has a bigamma shap Tabl 3.18 Exampl: Empirical standardizd partial aras and thir varianc for sampl data from studis of dtction of lung noduls and intrstitial disas Tabl 4.1 Thortical A A for binormal ROC curvs with sam b and a constant diffrnc btwn full AUCs Tabl 4.2 Varianc of mpirical A A for binormal ROC curvs with sam b and a constant diffrnc btwn full AUCs ( 10-4 ) Tabl 4.3 Statistical powr for comparisons of two partial AUCs of bi-normal ROC curvs with diffrncs in full AUCs of x

11 Tabl 4.4 Sampl siz rquirmnts for comparisons of two partial AUCs of bi-normal ROC curvs with diffrncs in full AUCs of 0.05 (btwn-modality corrlation of 0.5) Tabl 4.5 Varianc of diffrnc btwn spaucs of two straight-lin ROC curvs with diffrncs in full AUCs of 0.05 ( 10-4 ) Tabl 4.6 Statistical powr of comparisons of two partial AUCs of straight-lin ROC curvs with diffrncs in full AUCs of Tabl 4.7 Sampl siz rquirmnts of comparisons of two partial AUCs of straight-lin ROC curvs with diffrncs in full AUCs of 0.05 (data consistd of pairs of ratings for 150 normal and 150 abnormal subjcts, with btwn-modality corrlation of 0.5) Tabl 4.8 Thortical A 2 A 1 of two bi-gamma ROC curvs with diffrncs in full AUCs of Tabl 4.9 Varianc of mpirical spauc diffrnc for two non-crossing concav bi-gamma ROC curvs with diffrncs in full AUCs of Tabl 4.10 Statistical powr for comparisons of two partial AUCs of concav non-crossing bigamma ROC typ curvs with diffrncs in full AUCs of Tabl 4.11 Sampl siz rquirmnts for comparisons of two partial AUCs of bi-gamma ROC typ curvs with diffrncs in full AUCs of Tabl 4.12 Sampl siz rquirmnts for infrncs basd on full AUC to achiv th sam powr as comparison of pauc (0, 0.2) shown in tabls Tabl 4.13 Rsults for comparisons of corrlatd ROC curvs prsntd in xampl # Tabl 5.1 Thortical standardizd pauc for concav binormal ROC curvs and corrsponding partial binormal ROC curvs with mass xi

12 Tabl 5.2 Varianc of standardizd pauc for concav binormal and straight-lin ROC curvs and corrsponding partial ROC curvs with mass ( 10-4 ) Tabl 5.3 Statistical powr for concav binormal and straight-lin ROC curvs and corrsponding partial ROC curvs with mass Tabl 5.4 Thortical diffrnc in standardizd paucs for comparisons of two concav binormal ROC curvs and comparisons of corrsponding partial binormal ROC curvs with mass Tabl 5.5 Varianc of diffrnc in standardizd paucs for concav binormal and corrsponding partially concav ROC curvs with mass ( 10-4 ) Tabl 5.6 Statistical powr for comparison of paucs within classs concav binormal ROC curvs, straight-lin ROC curvs, and corrsponding partial ROC curvs with mass Tabl 5.7 Statistical powr for concav binormal ROC curvs and corrsponding partial ROC curvs with mass (fixd AUC diffrnc=0.05) xii

13 LIST OF FIGURES Figur 1.1 ROC curv... 2 Figur 3.1 Valus of th standardizd partial AUC for concav binormal ROC curvs Figur 3.2 Varianc of standardizd pauc(0,) stimats for binormal ROC curvs as a function of th siz of th rang of intrst Figur 3.3 Varianc of standardizd pauc stimats for straight-lin ROC curvs ovr (0,) as a function of th siz of th rang of intrst Figur 3.4 Empirical ROC curvs for th two datasts Figur 4.1 b=1 and lowr AUC= Figur 4.2 Diffrnc in paucs for ROC curvs of intrst (lft) vs. Diffrnc in paucs for straight-lin ROC curvs (right) Figur 4.3 Bi-gamma ROC curvs with AUC= Figur 4.4 Binormal ROC curv (b=1), Bi-gamma ROC curv (κ=1) and a straight-lin ROC curv with AUC= Figur 4.5 Empirical stimats of corrlatd ROC curv from xampl # Figur 5.1 Concav binormal ROC curvs Figur 5.2 Partial concav binormal ROC curvs with mass at FPF qual Figur 5.3 Partial concav binormal ROC curvs with mass at FPF qual xiii

14 1.0 INTRODUCTION 1.1 BACKGROUND A basic problm in valuation of diagnostic prformanc involvs assssmnt of th accuracy of a diagnostic tst in idntifying a patint with a spcific, prdfind condition (abnormal subjct) and a patint without th condition (normal subjct). Th tru status (prsnc or absnc of th abnormality in qustion) of a subjct is assumd to b known for all subjcts usd for accuracy valuation. Th diagnostic tst rsults can b masurd using a binary scal indicating that th subjct is assssd as positiv or ngativ, or an ordinal multi-catgory (.g. 7) scal typically with largr valus rprsnting highr probability of th abnormality bing prsnt, or a continuous scal indicating th liklihood of a pr-spcifid abnormality bing prsnt. For a multi-catgory diagnostic tst, a subjct can b classifid into a positiv or ngativ class according to whthr th tst rsult is gratr than or lss than a pr-spcifid thrshold. Th Rcivr Oprating Charactristic (ROC) analysis is th most widly usd mthodology to invstigat this typ of rsarch objctivs. ROC analysis originatd from signal dtction thory (Grn and Swts, 1966) (Egan, 1975) and has bn wll dvlopd ovr th past 50 yars in particular as rlatd to diagnostic imaging and dcision making (Mtz, 1989) (Hanly, 1989) (McNil t al., 1975) (Zhou t al., 2002). Howvr, many issus rmain and nw mthods ar constantly bing dvlopd. 1

15 Th ROC curv is th plot of snsitivity vrsus 1-spcificity for all possibl dcision thrshold valus of c (Figur 1.1). Lt X and Y dnot th ratings for normal and abnormal subjcts rspctivly. Snsitivity, or tru positiv fraction (TPF), is th probability of tst rsults bing positiv for abnormal subjcts, and can b dfind as follows: ( ) = ( ) = Pr ( > ) snsitivity c TPF c Y c Spcificity, or tru ngativ fraction (TNF), is th probability of tst rsults bing ngativ for normal subjcts, and can b dfind as follows: ( ) = ( ) = Pr ( ) spcificity c TNF c X c Figur 1.1 ROC curv Th most commonly usd summary indx associatd with th ROC curv is th ara undr th ROC curv (AUC). It is dfind as 1 0 ( ) A = ROC f df Th AUC has svral intrprtations. First, it can b intrprtd as th wightd avrag valu of snsitivity of all possibl valus of spcificity (Zhou t al., 2002), or th wightd avrag valu of spcificity of all possibl valus of snsitivity (Mtz, 1989). Scond, it can b 2

16 also intrprtd as th probability that a tst rsult for a randomly slctd abnormal patint indicats a gratr suspicion of disas than th tst rsult for a randomly slctd normal patint (Hanly and McNil, 1982) (Bambr, 1975). If X and Y ar continuous (i.., no tis in rsults ar possibl), thn th AUC can b dfind as Pr ( Y X) >. Th valu of AUC of 1 indicats a prfct systm whras th non-informativ (i.., gussing) diagnostic systm would hav AUC of 0.5. An unbiasd non-paramtric AUC stimat is th ara undr th mpirical ROC curv which is th sam as th Mann-Whitny form of th two-sampl Wilcoxon rank-sum statistic (Shapiro, 1999). A numbr of paramtric and non-paramtric mthods basd on AUC hav bn dvlopd to mak statistical infrncs (Zhou t al., 2002) (Pp, 2003). AUC offrs a singl valu to indicat th accuracy of diagnostic prformanc by considring both snsitivity and spcificity across all possibl thrshold valus; its major limitation is that it summarizs th ntir ROC curv including th rgion which may not b of intrst or practical rlvanc, for xampl, th rgion with vry low spcificity lvls. To rmdy this limitation, partial ara undr th ROC curv (pauc) can b usd to dscrib th intrinsic accuracy of diagnostic tsts in th rang of practical (clinical) intrst. Th pauc is frquntly dfind as 2 1 ( ) A 1, = ROC f df 2 In practic, a rang of ( 0, ) is oftn usd du to th importanc of high-spcificity rang in practic (McClish, 1989) (Jiang t al., 1996). Sinc th pauc ovr an arbitrary intrval (, ) is quivalnt to th diffrnc in pauc(0,2) and pauc(0,1), ( ) 1 2 in focus of my work. A = ROC f df will b 0 3

17 A numbr of statistical mthods and infrncs basd on th pauc using both paramtric and non-paramtric approachs hav bn dvlopd. Ths includ th paramtric stimator of th pauc and its varianc using th bi-normal modl (McClish, 1989) (Jiang t al., 1996). Wiand t al. (1989) proposd a non-paramtric mthod for stimating pauc and its varianc. Basd on Dlong s approach (DLong t al., 1988), Zhang t al. (2002) proposd a simplr mthod to comput th varianc of pauc which was subsquntly improvd by H and Escobar (2008). An altrnativ nonparamtric varianc stimator of th pauc using its xpctd valu was proposd by Liu t al. (2005). Othr non-paramtric mthods hav bn dvlopd such as mpirical liklihood mthods, for comparing two paucs (Huang t al., 2012) (Qin and Zhou, 2006) (Chn and Wong, 2009), and smi-paramtric rgrssion approachs on pauc by Dodd and Pp (2003) and Cai and Dodd (2008). Howvr, svral concptual and analytical difficultis prvnt pauc from bing widly usd. In gnral, paramtric approachs offr improvd fficincy of statistical infrncs, but could introduc substantial bias if th ndd paramtric assumptions ar not satisfid. Undr th corrctly spcifid modl th rlativ fficincy of nonparamtric stimats of partial AUC can b as low as 50% for short rangs of intrst (.g., ), but incras byond 80% fficincy for rangs widr than (Dodd and Pp, 2003). For th full AUC, rsults of paramtric and nonparamtric infrncs ar vry similar (Hajian-Tilaki t al., 1997) (Hajian-Tilaki and Hanly, 2002). Howvr, it is not always asy to vrify appropriatnss of th paramtric assumptions, and for mis-spcifid modls paramtric stimats of pauc could asily hav bias as high as 40% (.g., Dodd and Pp, 2003). For this rason it is oftn rcommndd to us non-paramtric approachs for infrncs about partial AUC (Dodd and Pp, 2003) (Zhang t al., 2002) (H and Escobar, 2008). Non-paramtric analysis of pauc is in primary focus of this work as wll. 4

18 On of ths difficultis is that th scal of valus of pauc incrass with incrasing rang of intrst. To partially ovrcom this limitation, svral partial ara indics hav bn proposd (Zhou t al., 2002) (McClish, 1989). A natural transformation of th partial ara aimd to standardiz th rang of its valus can b writtn as follows (McClish, 1989): 2 2 ( ) ROC f df 1 A A = 1+ = 1+ (1.1) Hr, w trm this indx as th standardizd partial AUC (spauc). For ROC curvs dscribing bttr-than-chanc prformanc, A varis from 0.5 to 1 rgardlss of, and for =1 it rducs to th convntional AUC. Scond, th rlvant rang should b pr-spcifid during study dsign but it is oftn difficult to dtrmin a priori. In addition, it is oftn assumd that bcaus of th us of lss information, analysis basd on th pauc may rsult in th loss of statistical prcision as compard with statistical infrncs basd on th full AUC, and thus its us may rquir largr sampl sizs (Zhou t al., 2002) (Obuchowski and McClish, 1997) (Wiand t al., 1989). Conjcturs about th rlativ stability of th spauc with rspct to th rang of intrst and th dcras in varianc with incrasing rang ar intuitivly appaling and could affct th way statistical analysis is plannd and intrprtd. In analyzing xprimntally ascrtaind datasts from obsrvr prformanc studis w frquntly ncountrd scnarios that contradictd th two conjcturs. Th work prsntd hr primarily focuss on th invstigation of proprtis of statistical infrncs basd on th pauc. In diagnostic radiology, it is natural to obsrv multipl subjcts having th sam diagnostic tst rsults (a ti), in particular at th lowst rang, vn whn th original scal is 5

19 continuous or psudo-continuous (.g confidnc rating scal). A ti at th lowr rating could rflct an important charactristic such as th prvalnc of th obviously normal subjcts (.g., chst imags) in a sampl, or frquncy of th natural absnc of a tstd substanc (Schistrman t al., 2006), or assigning dfault valu to subjcts with biomarkr lvls blow a crtain limit of concntration and/or a limit of dtction (Prkins t al., 2007). Whn ths multipl tis occur at th lowst rating lvl, th ROC curv includs a straight lin sgmnt joining th point corrsponding to th lowst thrshold and th cornr point (1, 1). Sinc this typ of tst rsults has a spik (mass) at th lowr thrshold, for brvity w trm such a curv as an ROC curv with mass. For th ROC curv with mass, a paramtric mixd modl combind with Box-Cox transformation and a non-paramtric approach basd on th Mann- Whitny statistic for th stimation of AUC has bn proposd and discussd (Schistrman t al., 2006). Th paramtric mixd modl approach can b furthr usd to stimat Youdn s Indx and dtrmin th optimal thrshold for tst rsults with mass (Schistrman t al., 2008). Howvr, issus rlatd to th valuation of a singl pauc and th comparisons of two corrlatd paucs associatd with ROC curvs with mass rmain unsolvd to dat. 1.2 OBJECTIVES Th mphasis of this dissrtation will b on invstigations of statistical proprtis whn valuating diagnostic prformanc using pauc. W bliv that in many practical scnarios infrncs basd on pauc could b no lss statistically advantagous than infrncs basd on th full AUC. Thus th us of pauc may actually lad to not only mor rlvant but also mor conclusiv rsults in analyss of xprimntal data and/or rquir smallr sampl sizs in pland 6

20 studis. This should ncourag rsarchrs and practitionrs to mor frquntly apply this highly rlvant, but currntly undrusd summary indx. Th rsults of our invstigation could also provid foundation for dcisions about optimal thrsholds to achiv gratst statistical powr and thrfor smallr sampl sizs whn using pauc. This dissrtation includs th following thr objctivs. Objctiv 1: As rlatd to valuation of a singl diagnostic systm, w invstigat th ffct, if any, of th rang of intrst ( 0, ) on statistical infrncs whn th pauc(0,) is usd as a summary masur of prformanc. W analyz th proprtis of nonparamtric and paramtric stimats of standardizd paucs and thir variancs. Using xtnsiv simulation studis, w invstigat th statistical powr for diffrnt familis of ROC curvs such as binormal ROC curvs, bigamma ROC curvs and straight-lin ROC curvs. Basd on th rsults of this rsarch, w dvlop a program for stimating sampl siz in th valuation of a singl pauc in a rang of practically rlvant scnarios. Objctiv 2: W xtnd th dvlopmnts from objctiv 1 for th task of comparison of accuracy lvls of two diagnostic systms on th basis of pauc computd from th paird data collctd with ach cas ratd undr vry modality. First, w analytically invstigat conditions for th incrasing diffrnc in th standardizd pauc with incrasing siz of th rang of intrst. Basd on xtnsiv simulation studis, w invstigat th statistical powr for comparisons of paucs ovr diffrnt rangs of intrst undr th ROC scnarios (such as binormal ROC curvs, bi-gamma ROC curvs and straight-lin ROC curvs) which lad to diffrnt pattrns of changs in pauc with incrasing rang. Basd on th rsult of this rsarch, w dvlop a program for 7

21 stimating sampl siz for comparison of two corrlatd paucs for a varity of practical scnarios. Objctiv 3: Th task of valuation of diagnostic modalitis is oftn complicatd by prsnc of substantial tis in th data. Using mathmatical considrations and xtnsiv simulations, w invstigat th proprtis of th diffrncs in th paucs and statistical powr ovr diffrnt rangs of intrst. Th xpctation is that th trnds of incrasing/dcrasing varianc with incrasing rang of intrst would bcom lss pronouncd for data with tis at th lowst rating valu (corrsponding to th ROC curv with mass) as compard with data without tis. This could affct th xpctd pattrns in statistical powr. Th rsults of this invstigation will hlp plan th analyss of diagnostic accuracy using data with tis at th lowst rating lvls and mak mor informativ dcisions about th data collction protocols. 8

22 2.0 FACTS RELATED TO THE PRESEARCH 2.1 FAMILIES OF ROC CURVE, THEIR AUCS AND PAUCS BINORMAL ROC CURVES Bi-normal ROC curv is th most widly usd modl in ROC analysis (Zhou t al., 2002). Th nam binormal rflcts th shap of ROC curvs and stms from th fact that binormal ROC curv can rsult from th two (indpndnt) normally distributd random variabls. Howvr, th us of th binormal ROC curv dos not ncssarily imply that th tst rsults ar assumd to follow normal distributions in th subpopulation of normal and abnormal patints. Rathr, th us of a binormal ROC curv implis that th obsrvd diagnostic rsult is rlatd (according to a crtain monotonically incrasing transformation, with possibl grouping for discrt cas) to normally distributd latnt scors. For a pair of latnt scors for normal and abnormal patints which follow two normal 2 2 distributions, i.. X ~ N ( µ x, σ x ) and ~ ( y, y) xprssd as: Y N µ σ rspctivly, th ROC curv can b ( ) 1 ( ) =Φ + Φ ( ) ROC a b x 9

23 whr a ( µ y µ x) = x σ y σ b = and Φ is th cumulativ normal distribution function. This σ y rlationship btwn ( ab, ) and th paramtrs of th distribution of th latnt scors is rarly usd in practic. On of th xcptions is to fit th ROC curv for continuous data using Box- Cox transformation (Zou and Hall, 2000); howvr, this rlationship is vry usful in simulation studis. Th AUC for th binormal ROC can b xprssd as: and th pauc as: 1 1 a A = Φ ( a + bφ ( x) ) dx =Φ b 0 1 ( ( )) A = Φ a + bφ x dx Hillis and Mtz providd an analytic xprssion for pauc in th cas of binormal ROC curvs (Hillis and Mtz, 2012), a b A = FBVN Φ ( ) 1+ b 1+ b 1, ; 2 2 whr (, ; ) F zxρ is th standardizd bivariat normal distribution function with corrlation ρ. BVN POWER-LAW ROC CURVES Anothr wll-known, but simplr and lss flxibl (du to a singl-paramtr typ) family of ROC curvs is dscribd by th powr-law curvs (Egan, 1975) (Hanly, 1988), or Lhman family of th ROC curvs (Gonn and Glnn, 2010). On of th rasons to considr this modl was for invstigating th consquncs of dviation from th binormal assumption (Hanly, 10

24 1988) and nabling simpl infrncs using built-in softwar (Gonn and Hllr, 2010). Powrlaw ROC curv can rsult from two xponntially distributd variabls. Howvr, th us of th powr-law ROC curv dos not ncssarily imply that th tst rsults ar assumd to follow xponntial distributions in th subpopulation of normal and abnormal patints. Rathr, similar to othr paramtric ROC curvs, th us of a powr-low ROC curv implis that th obsrvd diagnostic rsult is rlatd (according to a crtain monotonically incrasing transformation, with possibl grouping for discrt cas) to xponntially distributd latnt scors. For a pair of latnt scors for normal and abnormal patints which follow two xponntial distributions, i.. ( ) ~ x b xprssd as: with th AUC of: X Exp θ and ~ ( y ) Y Exp θ rspctivly, th ROC curv (powr-law) can ROC θ =. θ y ( ) xp x 1 θ θ x y θ x A = xp f df = xp 1 0 θ y θ x θ, y and th pauc of: 1 θ θ x y θ x A = xp x dx = xp 1 0 θ y θ x θ. y BI-GAMMA ROC CURVES Bi-gamma family is anothr of th wll-known familis of th ROC curvs (Egan, 1975) (Dorfman t al., 1996) (Faraggi t al., 2003) (Huang and Pp, 2009). In gnral bi-gamma ROC 11

25 curvs constitut a thr-paramtr family, howvr, in practic a subfamily of concav curvs rprsntd by constant-shap bi-gamma ROC curvs is usd (Dorfman t al., 1996). Similar to th binormal ROC curvs, th constant shap bi-gamma ROC curvs constitut a twoparamtr family, howvr, it offrs mor flxibl shaps of th practically rasonabl concav ROC curvs (a subfamily of concav binormal ROC curv is a on-paramtr family). Th primary disadvantag of bi-gamma ROC curv lis in th rlativ complxity of computations. Howvr, th computational complxity is allviatd with th dvlopmnt of softwar packags and thortical invstigations of th proprtis of bi-gamma ROC curvs (Constantin t al., 1986). A bi-gamma ROC curv can b paramtrizd with paramtrs of th gamma distribution of th latnt (as opposd to actual) ratings for normal and abnormal subjcts. W not that similar to othr ROC modls, th undrlying assumption of a bi-gamma-typ shap of th ROC curv dos not imply an assumption of a bi-gamma distribution of th actual ratings (du to th invarianc of th ROC curv with rspct to monotonically incrasing transformation of th ratings). In othr words, th distributions of latnt ratings ar simply intrmdiat stps for paramtrization of th ROC curv. Th probability dnsity function of th undrlying rating modl of th bi-gamma ROC curv has th following form: 1 1 =, θ τ x k 1 θ f( xk ;, θ ) x k In gnral paramtrs θ and κ could b diffrnt for th latnt normal and abnormal ratings. Th constant-shap bi-gamma ROC curvs ar obtaind by constraining th shap paramtr κ to b th sam for two distributions. Whn κ approachs 0, th bi-gamma ROC curv approachs th shap of a straight-lin and whn κ>1 th shap of th bi-gamma ROC curv rsmbls a binormal ROC curv du to th fact that gamma distribution approachs to ( k ) 12

26 normal distribution whn shap paramtr κ is larg (W not howvr, that this dos not guarant convrgnc of th ROC curvs). Whn κ=1 th bi-gamma ROC curv is quivalnt to th powr-law ROC curv (Egan, 1975) (Hanly, 1989). For a pair of latnt scors for normal and abnormal patints which follow two gamma distributions, i.. ~ (, ) b xprssd as: X Gamma θx κ x and ~ ( y, y) Y Gamma θ κ rspctivly, th ROC curv can ( x ) 1 ( ) ( ) ROC = S S. y Th dnsity of th Gamma distribution is givn by 1 1 = and S dnots θ τ x k 1 θ f( xk ;, θ ) x k ( k ) th survival function of Gamma distribution. Du to th rlationship btwn Gamma and Bta distribution th AUC of th bi-gamma ROC curv can thn b xprssd (Constantin t al., 1986) (Hussain, 2012) as: θ = ( ) ( ) κy, κ = = x θ y 1 κ 1 κ y 1 θx+ θ x y A y x ( 1 x) dx 1 FF ( κ )( );2,2 ;, 0 y κx θx θy κy κx Fbta κx κy B θx + θy whr ( *;2, 2 F y x) F κ κ is th cumulativ distribution function (CDF) of an F random variabl with paramtrs 2κ y and 2 x paramtrs κ x and κ y. κ, and bta (*; x, y ) F κ κ is th CDF of a bta random variabl with As of now thr ar no simplifid xprssions for th pauc, and it is usually computd using numrical intgration according to th original dfinition: 0 y x 1 ( ( )) A = S S dx. 13

27 2.1.4 STRAIGHT-LINE ROC CURVES W dfin a straight-lin ROC curv as th curv consisting of two lin sgmnts th vrtical sgmnt conncting th point (0, 0) and th point (0, 1/a), whr a>1, and a lin sgmnt conncting th point (0, 1/a) and th point (1, 1). Namly: ( ) ROC = a + a (2.1) Such a curv dscribs a thortically important scnario whr diagnostic rsult prfctly sparats th most obvious abnormal patints, whil bing non-informativ for discriminating btwn normal and abnormal patints in th rmaining population. Indd, using a flip of a coin it is possibl to crat a diagnostic tst with oprating charactristics anywhr on th straight lin xtnding to (1, 1) (Wagnr t al., 2010) (Bandos t al., 2010). Thortical importanc of this typ of a ROC curv for th currnt work stms from th ancillary natur of th oprating points with non-zro FPF. In practic th pur straight-lin ROC curvs (i.., with mpirical points alignd around th straight lin) could occur whn a diagnostic systm is forcd to produc continuous (untid) rsults in situations whn thr is littl or no information for distinguishing btwn subjcts (Gur t al., 2006). Straight-lin ROC curv has a constant valu of standardizd partial AUC rgardlss of th rang of intrst (Ma t al., 2013); this offrs an important scnario for invstigating paucbasd infrncs. Th nam straight-lin simply rflcts th shap of th curv. Th ROC curv with a straight-lin shap would rsult from th two (indpndnt) random variabls with uniform distributions. Howvr, du to th ROC invarianc proprty th us of th straight-lin ROC curv dos not ncssarily imply that th tst rsults ar assumd to follow uniform distributions 14

28 in th subpopulation of normal and abnormal patints. In gnral it can b viwd as a curv corrsponding to a diagnostic rsult that prfctly sparats th most obvious abnormal patints, whil is non-informativ for discriminating btwn normal and abnormal patints in th rmaining population. For a pair of latnt scors for normal and abnormal patints which follow two uniform distributions, i.. X ~ U ( 0,1) and ~ ( 0, ) and th AUC can b xprssd as Y U a rspctivly, th ROC curv can b xprssd as: ( ) ROC = a + a A = 1 dx = a a 2 a, whil th pauc can b xprssd as: A = + dx = + a a a 2a ESTIMATION OF ROC CURVES PARAMETRIC ESTIMATES OF ROC CURVES A numbr of approachs xist for paramtric stimation of th ROC curv. Th two gnral classs of paramtric stimation mthods ar distribution-fr and distribution-basd approachs. 15

29 Distribution-fr approachs may plac paramtric assumption on th shap of th ROC ( ) 1 curv,.g., binormal ROC curv, ROC ( ) a b ( ) =Φ + Φ, but not on th distributions of scors for disasd and non-disasd subjcts. For continuous tst rsults, Pp (2003) proposd an stimation mthod involving th mthods of gnralizd stimating quations and gnralizd linar modls which can incorporat covariat information. Zou and Hall (2000) prformd MLE rank-basd stimation of binormal ROC curvs. For discrt tst rsults, a maximum liklihood approach was introducd by Dorman and Alf (1969). Distribution-basd approachs, on th othr hand, stimat conditional distribution of th tst rsults (givn subjcts tru status); th ROC curv is thn stimatd indirctly as th composition quantil and distribution function. For xampl, a naïv distribution-basd approach for stimating th binormal ROC curv (which is rarly usd in practic), assums a normal distribution of th tst rsults. If X and Y ar th tst rsults for th random sampls of m normal and n abnormal subjcts, basd on th invarianc principl, th maximum liklihood stimat (MLE) of th binormal ROC curv can b xprssd as follows (Zhou t al., 2002): ( ) ˆ µ ˆ y µ x whr aˆ =, ˆ σ y ( ) ( ) ˆ 1 =Φ ˆ + Φ ( ) ROC ˆ a b x ˆ ˆ σ x b =, ˆ µ ˆ x, ˆ µ y, ˆ σ x and ˆ σ y ar th ML stimats of th mans and σ y standard dviations, and Φ is th cumulativ normal distribution function. Givn that th binormal distribution assumption is rstrictiv and basd on th invarianc proprty of monotonic transformation of ROC curvs, Faraggi t al. (2003) applid a Box-Cox typ powr transformation to th data, and aftr obtaining th appropriat transformation usd binormal modl. 16

30 2.2.2 EMPIRICAL ROC CURVE Th mpirical ROC curv is a collction of th mpirical oprating points ( whr Th mpirical tru and fals positiv fractions ar computd as follows: ˆ FPF and ˆ TPF ) ( ) TPF ˆ c = n j= 1 I Y n j > c, ( ) FPF ˆ c = m i= 1 [ > c] whr I( x ) = 1 if x is tru and 0 othrwis. Howvr, frquntly th mpirical ROC curv is plottd by conncting th mpirical points with straight lin sgmnts. Som analytical mthods howvr, do not us th points on th straight-lins (Grnhous and Mantl, 1950) (Wiand t al., 1989) (Zhang t al., 2002) (H and Escobar, 2008). Th points on th straight-lin sgmnts btwn th mpirical points dscrib oprating charactristics which might not b attainabl by applying spcific thrsholds to th obsrvabl tst rsults. Howvr, ths could b attaind by random gussing btwn th dcisions at th adjacnt oprating points (Fawctt, 2006) (Wagnr t al., 2010) (Bandos t al., 2010). I X W will us th trm linarly-intrpolatd mpirical ROC curv to distinguish it from th st of discrt ( fpf, tpf ) points. m i. 17

31 2.3 ESTIMATION OF AUC AND PAUC PARAMETRIC ESTIMATES OF AUC AND PAUC Paramtric analyss basd on AUC and partial AUC ar rasonably straightforward. Th prviously mntiond mthods can b usd to stimat smooth ROC curvs. Onc a smooth curv is fittd, th partial ara can b stimatd for any spcifid rang of intrst; its varianc can b valuatd using th dlta mthod basd on th varianc of th modl paramtrs (Zhou t al., 2003). For naïv binormal modl, th stimatd AUC or partial AUC can b computd as follows: 1 ˆ ( ˆ 1 aˆ A = Φ aˆ + bφ ( x) ) dx =Φ 0 ˆ2 1+ b whr ( ˆ µ ˆ y µ x) aˆ =, ˆ σ y 0 ( ˆ 1 ˆ ( )) Aˆ = Φ a + bφ x dx ˆ ˆ σ x b =, ˆ µ ˆ x, ˆ µ y, ˆ σ x and ˆ σ y ar th MLE of th man and standard σ y dviations of th latnt scors (.g., MLE stimats for a and b can b obtaind from th probit rgrssion modl of th discrt tst rsults), and Φ is th cumulativ normal distribution function. Or by using analytic xprssion for pauc in th cas of binormal ROC curvs (Hillis and Mtz, 2012), aˆ bˆ ( ) 1+ bˆ 1+ bˆ ˆ 1 A = FBVN, Φ ; 2 2 whr (, ; ) F zxρ is th standardizd bivariat normal distribution function with corrlation ρ. BVN 18

32 2.3.2 EMPIRICAL ESTIMATES OF AUC AND PAUC m X = and { j} 1 If { i} i 1 n Y = ar th tst rsults for random sampls of m normal and n abnormal j subjcts thn th stimat of th AUC can b xprssd as follows: Aˆ = m n i= 1 j= 1 ψ ( X, Y ) nm i j 1 X < Y ψ XY, = X= Y 0 X > Y whr ( ) 1 2 This non-paramtric AUC stimator is qual to th ara undr th mpirical ROC curv whr th points on th plot ar connctd by straight lins. whr and Th partial ara can b stimatd by (H and Escobar, 2008): F x is th mpirical distribution of m n ˆ 1 A φ, = ( X Y ) i j mn i= 1 j= 1 1, Yj > X i and X i [ r0, ) 1 φ ( X i, Yj) =, Yj = X i and X i [ r0, ) 2 0, Yj < X i and X i [ r0, ) X i. 1 0 x 1 ( ) r = F For any conscutiv ratings r 1 and r 2 whr r2 r1 (, ) 1 2 follows: <, = FPF ( r ) and FPF ( r ) 1 1 =, if 2 2, on can a us linar intrpolation to comput th pauc which can b xprssd as ( 1) ( TPF ( r2) TPF ( r1) ) 2( ) ˆ ˆ A = A + TPF ( r ) + 1 ( )

33 2.4 ESTIMATION OF VARIANCE OF AUC AND PAUC In gnral varianc of th paramtric AUC and pauc stimators can b obtaind from a varianc matrix of th stimatd paramtrs (corrsponding to th ROC fitting approach) using dlta mthod (Zhou t al., 2002). For th naïv fitting of th binormal ROC curv (assuming normally distributd tst rsult) th varianc stimator attains th following closd-form xprssion in trms of a and b paramtrs of th binormal modl (Obuchowski and McClish, 1997): ( ˆ 2 2 ) ( ) ( ˆ) ( ˆ 2, ) Vˆ A = f V aˆ + g V b + fgc aˆ b (2.1) whr: ( ) Vˆ aˆ m a = ( ˆ) Vˆ b = ( ) nb 2 2 2mn ( n+ mb ) 2 2mn f 2 a xp = Φ 2π ( + b ) 2 ( + b ) { ( h) } ( ˆ) Cˆ ab ˆ, ab = 2n and ab h= Φ ( ) + + b 1+ b Whn = 1, this formula will rduc to th varianc stimator for full AUC. 20

34 Du to th clos rlationship to th Mann-Whitny tst statistics (Bambr, 1975), varianc of th AUC stimator for an mpirical ROC curv can b drivd from th formula for th Wilcoxon statistics proposd by Nothr (1967): whr ˆ m 1 n 1 1 Var( A) = ξ10 + ξ01 + ξ11 ik, = 1,..., mjl, = 1,..., n mn mn mn 2 { ψ ( X,Y ), ψ ( X,Y )} = E{ ψ ( X,Y ) ( X,Y )} A, j l ξ10 = Cov i j i l i j ψ i l 2 { i j k j } { i j k j } ξ01 = Cov ψ( X, Y ), ψ( X, Y ) = E ψ( X, Y ), ψ( X, Y ) A, i k ξ 2 2 { ψ ( X,Y )} = E{ ( X,Y ) } = Var ψ 11 i j i j A { ψ (, )} { ˆ i j } A= E X Y = E A For continuous tst rsults which ar oftn ncountrd in many scnarios such as gntic rsarch, H and Escobar (2008) proposd a non-paramtric varianc stimator for th partial ara. Altrnativly, th varianc of mpirical stimators of AUC and pauc can also b stimatd using a nonparamtric bootstrap approachs (Efron and Tibshirani, 1993). Th varianc can b stimatd by rsampling th normal and abnormal subjcts and linarly intrpolating th mpirical ROC curvs. Anothr varianc stimator of th pauc using a nonparamtric approach was proposd by Liu t al. (2005). If { i} i 1 m X = and { j} j 1 n Y = ar th tst rsults for random sampls of m normal and n abnormal subjcts with distribution functions Fx and and mpirical distribution functions F x and stimator A ˆ of pauc can b xprssd as: F y, F y rspctivly, thn th asymptotically unbiasd ˆ 1 1 A = I Y > X = S X n ( ) ( ) j i y i mn j= 1 i Ρ m i Ρ 21

35 y y, i whr S ( z) = 1 F ( z) R is th rank of X among th X s, that is, R = I( X X ) i m i k i k = 1, and { i m R m} Ρ= : (1 ) i. Thy also showd that: 2 ˆ d σ A N A, m + n Whr: σ x σ = σ / λ + σ / λ H W ( ( )) = S F 1 s t dsdt A 2 W 1 1 y x σ, ( ) s t= max st,, ( ( )) H = S F p dp A 1 y x, ( ) λ = 1 λ = m m+ n, ( ) = 1 ( ) and S ( z) 1 F ( z) S z F z x y =. 2 2 Morovr, th consistnt stimators of σ H and σ W, rspctivly can b obtaind: y { ( )} 2 1 ˆ, ˆ 2 2 H = S y X i A m i Ρ σ ˆ σ 1 = S ( X X ) Aˆ. 2 2 W y i k m m i k Ρ ( 1) 22

36 3.0 EVALUATION OF A SINGLE PAUC Th statistical infrnc rgarding diagnostic accuracy of a singl modality (diagnostic systm, classification tool, tc.) is oftn mad on th basis of summary indics such as pauc and AUC. For xampl, diagnostic accuracy for classifying imags as dpicting or not-dpicting lung noduls can b assssd using both point stimation and intrval stimation of pauc and AUC. In th valuation of a singl pauc, w invstigatd th ffct of th siz of th rang of intrst (0, ) on statistical infrncs rgarding th pauc. W analyzd th proprtis of th nonparamtric and paramtric stimats of spaucs and thir variancs. W drivd two important proprtis of th rlationship btwn th spauc and a dfind rang of intrst, which could facilitat a widr and mor appropriat us of this important summary indx. First, w mathmatically provd that th spauc incrass with incrasing rang of intrst for common ROC curvs. Scond, using a comprhnsiv numrical invstigation w dmonstratd that, contrary to common blif, th uncrtainty about th stimatd spauc can ithr dcras or incras with an incrasing rang of intrst. Our rsults indicatd that th pauc could offr advantags in som scnarios in trms of statistical uncrtainty of th stimation. In addition, slction of a widr rang would likly lad to an incrasd stimat vn in th cas of spauc. W dmonstratd that th bi-gamma family of th concav ROC curvs adquatly dscribs a wid rang of scnarios including cass whr pauc is statistically advantagous. This family was usd to dvlop sampl siz 23

37 stimation softwar offring a bttr insight in rlativ mrits of analyzing part of th curv. This portion of th rsarch is publishd in Statistics in Mdicin (Appndix A). 3.1 METHOD STANDARDIZED PARTIAL AUC AND ITS PROPERTIES Basd on th dfinition of standardizd pauc (1.1), it can b shown that th standardizd pauc and th varianc of its stimat ar always largr than convntional pauc and th varianc of its stimat. Indd sinc 1//(2-1), is lss than 1 for all 1: and A 1+ 2 A = A + A ˆ ( ) ( ˆ ) 4 ( ˆ = ) V A V A V A. 2 Unfortunatly, standardization of th partial ara in (1.1) is not idal. Indd, although th rang of A is indpndnt of, th actual valu of A for a givn ROC curv could dpnd on. Morovr, as w dmonstrat in Proposition 3.1 blow, thortically it can ithr incras or dcras with incrasing rang whil rmaining constant only in th cas of a straight-lin ROC curv (Chaptr 2.1.4) composd of two straight-lin sgmnts on vrtical and th othr passing through th point (1,1). Using quation (2.1) it is asy to s that partial AUC for th straight-lin ROC curv passing through th point ( f, t) is: 24

38 2 ( ) = ( ) ( ) + { f t ( ) ( )} A 1 t 21 f 1 1 t 1 f,,, straight and th standardizd partial AUC dos not dpnd on th rang of intrst (indpndnt of ): Proposition 3.1 For any ( 0,1), A t f (3.1) straight ( ) = 1,, ( 1 ) 21 f t ( ) i. ii. iii. Proof: A A A > 0 ROC ( ) > 21 ( A ) + ( 2A 1) = 0 ROC ( ) = 21 ( A ) + ( 2A 1) < 0 ROC ( ) < 21 ( A ) + ( 2A 1) By straightforward diffrntiation of (1.1) w obtain: A 1 = ( ROC ( ) ) A ( 1 ) A = A, th drivativ of standardizd partial AUC can b 2 2 Sinc ( 2 1) writtn as follows: 2 1 A 1 = ROC A 2 2 Th thr claims of this proposition immdiatly follow. {( ( ) ) ( 2 1)( 1 )} Proposition 3.1 implis that givn th ara ovr th rang (0,), w can dtrmin whthr a small incras in th rang would lad to an incras in th standardizd pauc by comparing whthr th point on th ROC curv ROC() is actually abov or blow th fixd 25

39 straight lin, that passs through th point (1,1) and has a slop of 21 ( A ). Altrnativly, this comparison can b conductd by comparing th ngativ diagnostic liklihood ratio (1- ROC())/(1-) with 21 ( A ). Th ngativ diagnostic liklihood ratio, DLR-(), is an important charactristic of binary diagnostic tst (Zhou t al., 2002) (Norman, 1964) (Biggrstaff, 2000) (Bandos t al., 2010). For a point on th ROC curv (,ROC()) it is dfind as (1-ROC())/(1-). Th ROC curv with a dcrasing ngativ diagnostic liklihood ratio is practically important. Such an ROC curv nsurs that starting at any givn oprating point, a thrshold-drivn improvmnt in snsitivity will b bttr than an improvmnt achivd by randomly slctd subjcts that wr tstd ngativ at th givn oprating point (Norman, 1964) (Bandos t al., 2010). Thus, a dcrasing ngativ diagnostic liklihood ratio in th rgion whr xprimntal oprating points ar obsrvd is a natural proprty for many practical diagnostic tsts. Whil rsults of proposition 3.1 ar important for judging th dpndnc of spauc on small changs in th rang of intrst, thy provid littl insight into th mor global bhavior of th spauc, or th gnral form of curvs with always incrasing/dcrasing ar addrssd by th following proposition and its corollaris. Proposition 3.2 A. Ths qustions If th ROC curv has a dcrasing ngativ diagnostic liklihood ratio in (0, 0), namly, ( ) 1 ROC A < 0, thn > 0 1 in th sam rang. Proof: 26

40 Lt us considr from ( 0, 0 ). Sinc for any ` (0,) f 1 ROC 1 f ( f ) f = ` < 0, w can obtain th following inquality : ( ) ( ) 1 ROC 1 ROC ` ROC 1 < or ROC ( `) < 1 ( 1 `) 1 1 ` 1 Hnc ovr th rang (0, ], th partial ara ( ( ) A ) and th standardizd partial ara undr th ROC curv ( A ) ar smallr than th corrsponding aras undr th straight lin ROC curv passing though (, ROC ()). Indd: ( ) ROC 1 A = ROC ( f ) df < 1+ ( f 1 ) df = A,,(, ( )) A < A,(, ( )). straight ROC straight ROC On th othr hand, from (2) w hav: ( ) (,, ( ),(, ( )) ) ( ) straight ROC straight ROC straight, (, ROC ( ) ) ( ) ( ) ROC = 2 1 A + 2A 1 = 2 1 A + 1. Also, sinc A < A, from abov w obtain: straight,(, ROC ( )) ( ) > 21 ( ) + 1= 21 ( ) + ( 2 1) ROC A A A A Finally, applying th rsult (i) of proposition 3.1 w obtain > 0. As w discussd prviously in this sction, a dcrasing ngativ diagnostic liklihood ratio is a natural proprty for many practical diagnostic tsts. W also not that th rsult of proposition 3.2 is dirctly applicabl to concav ROC curvs, as it can b dmonstratd that concavity immdiatly implis a dcrasing diagnostic liklihood ratios. Figur 3.1 illustrats th incras of th standardizd partial AUC with incrasing rang for fiv concav binormal ROC curvs. 27

41 Figur 3.1 Valus of th standardizd partial AUC for concav binormal ROC curvs. W not that proposition 3.2 is dirctly xtndabl to th partial ara indx (McClish, 1989) (Jiang t al., 1996) as wll as to th non-standardizd partial ara. Rsults summarizd in this sction indicat that in practical scnarios, currnt approachs to standardization of th partial AUC do not ncssarily liminat th ffct of th rang of intrst on valus of th standardizd pauc. Morovr, incrasing rang of can frquntly incras th apparnt lvl of diagnostic prformanc. In th nxt two sctions w xamin th statistical uncrtainty in th stimatd standardizd partial AUC VARIANCE OF THE PARAMETRIC ESTIMATE OF SPAUC Th partial AUC and othr ROC rlatd charactristics ar typically stimatd from a sampl of m normal and n abnormal subjcts with obsrvd diagnostic tst rsults of { i} i 1 28 m x = and { y j} 1 n j=

42 corrspondingly. W focus hr on th rlationship btwn th varianc of th spauc and th siz of th rang of intrst. In particular, w xamin th common conjctur that varianc would dcras with incrasing rang, sinc a largr rang incorporats mor availabl information in rgards to th oprating charactristics. W bgin by considring a simpl varianc stimat for th partial ara undr th binormal ROC curv (McClish, 1989). Thn, in sction 5 w prsnt simulation rsults that dmonstrat th gnrality of th drivd conclusions. W can comput th varianc of th stimatd standardizd partial AUC as: V ( Aˆ ), whr 2 ( ˆ ) ˆ ( ) V A = V A is computd according to (2.1). Figur 3.2 dmonstrats th varianc of th stimatd standardizd pauc as a function of th lngth of th rang, for two diffrnt binormal as wll as straight-lin ROC scnarios. Ths scnarios ar basd on a sampl siz of 100, (m=n=50) and dscrib diffrnt shaps of ROC curvs, including concav curvs (b=1) and typical impropr curvs (b=0.5) (Obuchowski and McClish, 1997). Each figur shows varianc functions for fiv ROC curvs with AUCs of 0.55, 0.65, 0.75, 0.85, and W not that hr, as wll as in th invstigations that follow, w considr binormal ROC curvs with b 1 sinc th corrsponding shaps of ths ROC curvs ar mor common in practical applications including, but not limitd to, mdical imaging. Indd, a binormal ROC curv with b>1 implis a wors-than-chanc prformanc in valuations of highly suspicious subjcts (i.., in th rang of high spcificity) which rarly happns in practic. For concav ROC curvs (Figur 3.2a) th varianc of th full AUC can xhibit both pattrns, namly, it can b ithr smallr or largr than varianc of th standardizd partial AUCs on (0, ). Th dcras in varianc with incrasing rang is obsrvd only for ROC curvs with 29

43 AUC valus gratr than In th straight-lin ROC scnarios for which all standardizd partial AUCs ar xactly th sam as th full AUC, th varianc of th standardizd partial AUC incrass. As shown in Figur 3.2b, for an impropr binormal ROC curv, th varianc frquntly incrass with incrasing rang, in particular th varianc of th full AUC (=1) tnds to b largr than th variancs for standardizd partial AUCs ovr most rangs considrd. Th anticipatd dcras in th varianc whn switching to full AUC is vidnt only for th ROC curv with th largst AUC (0.95) considrd hrin. 30

44 a) b=1 b) b=0.5 Figur 3.2 Varianc of standardizd pauc(0,) stimats for binormal ROC curvs as a function of th siz of th rang of intrst. Figur 3.3 Varianc of standardizd pauc stimats for straight-lin ROC curvs ovr (0,) as a function of th siz of th rang of intrst. 31

45 Ths rsults provid an important indication that thr ar a numbr of practical scnarios in which th stimatd partial AUC may b no lss prcis than th stimatd varianc for th full AUC. Varianc is an important charactristic of th statistical uncrtainty, howvr, its usfulnss for non-symmtric distributions is limitd (.g., sampling distribution of stimats of high pauc). Furthrmor, th trnds shown in Figur 3.2 ar basd on th assumption of normality of th tst rsult, and hnc might not b gnralizabl. In ordr to vrify ths trnds w conductd a simulation study as dscribd in th following sction. 3.2 NUMERICAL STUDY In this sction w considrd svral familis of ROC curvs including binormal, bi-gamma and straight-lin ROC curvs. For ach scnario, w computd th standardizd pauc by numrical intgration. W conductd a simulation study to assss th lngth of th qual-tail 95% rang (97.5 th -2.5 th prcntil) and varianc of th sampling distribution of th standardizd pauc. Th statistical powr was stimatd from 1000 rsults of th bootstrap tsts and th sampl siz was computd by stablishd rsults for Wald-typ tsts (Flahault, 2005). In th simulation study th tst rsults for normal and abnormal subjcts wr gnratd from normal distributions with paramtrs slctd to gnrat binormal ROC curvs with spcific valus of AUC (ranging from 0.55 to 0.95) and for thr valus for th shap paramtr b (1, 0.5 and 0.33). Valus for th paramtrs of binormal ROC curvs wr slctd to rflct shaps typically ncountrd in xprimntal prformanc assssmnt studis in diagnostic mdicin. For th bi-gamma ROC curvs th tst rsults wr gnratd from gamma distributions with th sam shap paramtr (Chaptr 2.1.3). For th straight-lin ROC curvs th tst rsults wr gnratd from th 32