Estimation of Area under the ROC Curve Using Exponential and Weibull Distributions

Transcription

1 XI Biennial Conference of the International Biometric Society (Indian Region) on Computational Statistics and Bio-Sciences, March 8-9, Estimation of Area under the ROC Curve Using Exponential and Weibull Distributions R. Vishnu Vardhan, Sudesh Pundir and G. Sameera Abstract--- In recent years the receiver operating characteristic (ROC) curves received much attention in medical diagnosis for classifying the subjects into one of the two groups. Many researchers have provided the mathematical formulation of the curve by assuming some specific distribution. Conventionally, much work has been carried out by assuming normal distribution. In this paper, we focused on estimating the ROC Curve and Area under the Curve (AUC) using Exponential and Weibull distributions. As Exponential and Weibull distributions are important in life testing problems, the performance of ROC forms of these distributions are studied and then results are compared with conventional Binormal ROC form. The entire study was done using real and simulated data sets. In a perspective it is proposed that ROC form of Biweibull is far better than the other two and Biexponential is better than the Binormal model of ROC curve. Keywords--- ROC Curve, Binormal, Biexponential and Biweibull Distributions, Area under the Curve R I. INTRODUCTION OC curves are the most widely used statistical technique used for classification, to estimate the accuracy of a diagnostic test as well as to identify the best threshold. In recent years, it has got much attention in the field of medical diagnosis [3, 5, 7, 9,,, 3, 4]. The main phase for diagnosing any disease is to evaluate its performance and to identify the biomarker which helps in classifying the individuals as healthy or diseased. At the same time it is also important for researchers to understand how best it can evaluate the test s performance. An important technique for evaluating the performance of diagnostic tests is the ROC curves. With appropriate use of ROC curves, the test performance can be further improved for getting the ideal classification. But in most of the situations the above said phenomenon will not be met by the researchers [6]. Statistically speaking, ROC curves are used to characterize the accuracy of a diagnostic test and also used to compare accuracies of two diagnostic tests [4]. So far, many researchers assumed that both (healthy and diseased) groups follow Normal distribution and carried out the estimation and fitting procedure of ROC curve analysis [2, 8, 2]. In most of the cases, the data collected in realistic environment may or may not follow Normal distribution. In this paper, we made an attempt to fit the ROC curve and to estimate the AUC of ROC by assuming that the individuals of both the groups will follow Exponential and Weibull distributions. Hence, we can call these ROC models as Biexponential and Biweibull models. In this paper, first we have discussed the estimation procedure of Binormal and then proposed the fitting and estimation procedure of Biexponential and Biweibull ROC models. In next section, the performance and accuracy measures of these distributions were studied using simulated and real data sets. II. ESTIMATION OF ROC CURVE Here, we will discuss in brief about the Binormal form of ROC, then discussion will be carried out to the other two distributions considered in this work. A. Binormal ROC Curve Let us consider the two populations, i.e., diseased population (D) and healthy population (H) with some classification rule which classifies the individual in any one of these populations. There are two important measures for defining an ROC curve, i.e., true positive rate () and false positive rate (). In probabilistic notation, the can be defined as, the chance of correctly identifying those subjects who are actually suffering from disease ; this is also referred as the sensitivity of the diagnostic test. Similarly the is the chance of wrongly identifying those subjects who are actually not suffering with the disease this is referred as -specificity. Using these two measures the plotting of ROC curve is made. In other words, it is a plot of and with varying cutoffs or thresholds. This gives the meaning that generated ROC will have infinitely many cutoffs and each cutoff will produce a pair of and. The curve will provide more statistical information and properties which are to be studied [6]. Firstly, the curve is of an unknown monotonic transformation from (, ) to (, ) and secondly, the region above the chance diagonal and below the curve is defined as the Area under the ROC curve i.e., AUC. This AUC is another important statistical accuracy measure used to assess the performance of a diagnostic test. R. Vishnu Vardhan, Department of Statistics, Pondicherry University, Puducherry. India. Sudesh Pundir, Department of Statistics, Pondicherry University, Puducherry. India. G. Sameera, Department of Statistics, Pondicherry University, Puducherry. India.

2 XI Biennial Conference of the International Biometric Society (Indian Region) on Computational Statistics and Bio-Sciences, March 8-9, If the curve approaches to the left hand corner and has a larger distance from the chance line then that particular cutoff gives the high percentage of accuracy of a diagnostic test. As it is assumed that the distributions of the test scores (S) follow normal in both diseased (D) and non-diseased (H) population with their respective means µ D, µ H and standard deviations σ D, σ H. Also assume that µ D >µ H, but no constraints are put on standard deviations []. The false positive rate with threshold t is given by and the ROC curve is given as where a= - and b=. It is clear that both a and b are non- negative. B. Area under the ROC Curve (AUC) The accuracy of the test depends on how well the test separates the group being tested into those with and without the disease. It is defined as where y(x) denotes the ROC curve model for a particular distribution. An area of represents a perfect test and an area of.5 represents a worthless test which implies that test have no discrimination power. Now the AUC of ROC for the Binormal distribution is In next sub section, we have proposed the ROC models for Biexponential and Biweibull forms. C. Biexponential Model and AUC Let us assume that the distributions of the scores (S) are exponential in both diseased (D) and non-diseased (H) population with respective means and standard deviations λ D, λ H. The cumulative distribution function is given as where >. The false positive rate with threshold t is given by F(x)=-e (-λx) and the ROC curve is given as

3 XI Biennial Conference of the International Biometric Society (Indian Region) on Computational Statistics and Bio-Sciences, March 8-9, Now the AUC for the Biexponential distribution is Depending on the relationship between λ D and λ H, the shape of the ROC curve gets changed. In next section, the ROC form under Biweibull distribution is discussed D. Biweibull Model and AUC Let us assume that the distributions of the scores (S) are Weibull in both diseased (D) and non-diseased (H) population with respective means and standard deviations. The cumulative distribution function of exponential distribution is given as where c> and x>. Now the false positive rate with threshold t is given by =-P (S t H) and the ROC curve is given as Using value of t from equation, we get =-P(S t D) - Now the AUC for the Biweibull distribution is Depending on the relationship between c D and c H, the shape of the ROC curves gets changed. III. RESULTS AND DISCUSSIONS The entire computations of this paper were carried out using simulation type of study as well as real datasets. Initially, we provide a detailed interpretation about the characteristics of three distributions and the values that are considered. As we mentioned earlier that we have studied the statistical properties of binormal, Biexponential and Biweibull ROC forms, here a simulated environment is developed to observe the variability and the performance of three ROC forms (Figures,2,3). In the case of binormal ROC form, three different typical possibilities are considered by fixing the mean of healthy and varying the diseased mean. Since, the variability in the response is usually observed in diseased subjects rather than the healthy subjects and at the same time it is clear that there are no restrictions about the standard deviations. Intuitively, one can imagine that as the distance between two means of D and H is widened a better discrimination can be met and vice versa. If we consider µ D =3.4 and µ H =2.5, the AUC so obtained is.786. Here the distance between two means is moderate and 78.6% of typical classification can be made. Using the proposed functional forms of AUC of Biexponential and Biweibull ROC models, similar kind of simulation studies have been conducted. In Biexponential model the parameter of healthy population will be larger than that of diseased population, since the mean under exponential is /λ. Even though this kind of situation is observed the practical implication in computing AUC will not be affected. Suppose if we consider λ H = and λ D =94, the AUC expression produces a value.576.

4 XI Biennial Conference of the International Biometric Society (Indian Region) on Computational Statistics and Bio-Sciences, March 8-9, Table : ROC Curve Parameters and AUC Measure Binormal (σ D = σ H =.8) Biexponential Biweibull µ D µ H AUC λ D λ H AUC c D c H AUC In case of Biweibull distribution again three situations are considered. As the values of c D and c H parameters vary, typical shapes of ROC forms under Biweibull distribution can be viewed. An interesting point a researcher has to focus under Biweibull model is, for any values of c D and c H the ROC curve attains an S shape. After crossing the diagonal line, the curve will have a concave shape from that point to the right upper corner point (, ). In literature the curves of this kind are referred to as Notproper ROC curves [5]. With larger distance between parameters, the ROC curve under Biweibull model attains a zig zagged S shaped curve, which means the vertical line of zig zag meets the diagonal line at value.5..8 BINORMAL ROC Curve.6 D=3.4, H=2.5 D=2.8, H=2.5 D=4.6, H= Figure : Typical Forms of Binormal ROC Curve.8.6 BIEXPONENTIAL ROC Curve D=948, H= D=.35743, H= D=739, H=.6.8 Figure 2: Different Forms of Biexponential ROC Curves

5 XI Biennial Conference of the International Biometric Society (Indian Region) on Computational Statistics and Bio-Sciences, March 8-9, BIWEIBULL ROC Curve.6 D=, H=2.5 D=5, H=2.5 D=6, H= Figure 3: Different Forms of Biweibull ROC Curves A. Breast Cancer Data The data was collected from [6]. This dataset contains 27 samples of which censored subjects are 35 and died cases are 72. Totally there are 9 variables, of which pathological tumor size is the influential variable to diagnose. The range of this variable is (., 7.). P-P plots have been plotted and it is observed that the pathological tumor size is following all the three distributions. In the table 2 we have reported the values of statistical parameters for the three distributions considered. As in the case of simulation study, similar type of phenomenon has been perceived in this dataset too. Focusing on the performance of three statistical distributions with their respective ROC models (Figure 4), binormal model seems to perform better than the other two. Even though a slight margin of difference exists between Biexponential and Biweibull, they provide equal percentage of classification. B. Tuberculosis Data The data was collected from Sri Venkateswara University of Medical Sciences, a tertiary hospital in Tirupati. Data consists of samples with 4 variables. Out of these Adenosine Deaminase (ADA) is the influential factor to diagnose. Here also P-P plots were plotted and outliers were identified. Using 5% trimmed mean, the outliers were removed from the dataset and the entire computations were carried out for the remaining samples (N=8). In table 2, the parameter values of this data were reported along with the accuracy measure AUC. Even though the variable follows all the three distributions, the binormal model provides a better accuracy when compared to the others. The accuracy obtained in the binormal model is 99.8% where as it is just 58.9% in case of exponential and 69.2% in case of Weibull. Thus, we claim that binormal model is far better than the other two forms and can be preferred for better classification. The ROC curves of the three statistical distributions were plotted and the same can be seen in figure 5. Distributions and their Parameters Binormal Table 2: ROC Curve Parameters and AUC Measure Breast Cancer Data Tuberculosis Data D H AUC D H AUC µ σ Biexponential λ Biweibull c

6 XI Biennial Conference of the International Biometric Society (Indian Region) on Computational Statistics and Bio-Sciences, March 8-9, ROC Curves for Breast Cancer Data.6 Binormal Biexponential Biweibull Diag.6.8 Figure 4: ROC Curves for Three Statistical Distributions using Breast Cancer Data.8 ROC Curves for Tuberculosis Data.6 Binormal Biexponential Biweibull.6.8 Figure 5: ROC Curves for Three Statistical Distributions using Tuberculosis Data IV. CONCLUSIONS In the classification theory, Binormal ROC model has made its landmark. Many researchers have proposed various mathematical procedures for handling ROC curve model. In this paper, the authors made an attempt to observe the statistical properties of ROC curves underlying Normal, Exponential and Weibull distributions. Using the obtained results, it is clear that the binormal model performs in a better way when compared to the other two and at the same time it is the accuracy measure of binormal varies much. The ROC curve of Biweibull has attained S shaped pattern, indicating a case of not proper ROC curves. Both in simulations and realistic datasets the accuracy measure AUC under Biexponential is not attaining a value beyond.7. The authors claim that for giving a better classification it is suggestible to consider the binormal ROC form. Even though datasets follow life distributions, exponential and Weibull, the binormal model provides a better AUC. REFERENCES [] Bamber, D. (975), the area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 2: [2] Dorfman and Alf (969), Maximum Likelihood Estimation of parameters of signal detection theory and determination of confidence interval rating method data, Journal of Mathematical Psychology; 6; [3] James A Hanley, Barbara J Mc Neil (982), A Meaning and Use of the area under a Receiver Operating Characteristics (ROC) Curves, Radiology; 43; [4] James A Hanley, Barbara J Mc Neil, (983), A method of Comparing the Areas Under Receiver Operating Characteristics Analysis derived from the same cases, Radiology; 48;

7 XI Biennial Conference of the International Biometric Society (Indian Region) on Computational Statistics and Bio-Sciences, March 8-9, [5] John. A. Swets et. al (979), Assessment of Diagnostic Technologies, Science; 25; [6] Krzanowski, WJ and Hand, DJ (29), ROC curves for continuous data, Monographs on Statistics and Applied Probability, CRC Press, Taylor and Francis Group, NY [7] Metz CE (978), Basic Principles of ROC analysis, Seminars in Nuclear Medicine, 8: [8] Ogilive and Creelman (968), Maximum Likelihood Estimation of Receiver Operating Characteristic Curve Parameters, Journal of Mathematical Psychology; 5; [9] Pepe MS (997), A regression modeling framework for receiver operating characteristic curves in medical diagnostic testing, Biometrika, 84(3): [] Pepe MS (998), Three approaches to regression analysis of receiver operating characteristic for continuous test results, Biometrics 54:24-35 [] Pepe MS (2), An interpretation for ROC curve and inference using GLM procedure, Biometrics, 56: [2] Pepe, MS, (23), the statistical evaluation of medical tests for classification and prediction, Oxford Statistical Science Series, Oxford University Press. [3] Qin J and Zhang B (997), A goodness-of-fit test for logistic regression models based on case-control data, Biometrika, 84:69-68 [4] R Vishnu Vardhan and K.V.S Sarma (2), On the Relationship between the Odds Ratio and the Area under the ROC Curve in the context of Logistic Regression for Comparing Several Biomarkers, International Journal of Statistics and Systems; 5; [5] Stefano Parodi, Vito Pistoia, Marco Muselli (28), Not proper ROC curves as new tool for the analysis of differentially expressed genes in microarray experiments; BMC Bioinformatics 28, 9:4 [6] ACKNOWLEDGEMENTS The authors would like to acknowledge Dr. Alladi Mohan, Professor, Department of General Medicine, Sri Venkateswara Institute of Medical Sciences (SVIMS, Tirupathi) for providing the Tuberculosis data to carry out this research work.