Receiver operating characteristic

Transcription

1 Receiver operating characteristic From Wikipedia, the free encyclopedia In signal detection theory, a receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot of the sensitivity, or true positives, vs. (1! specificity), or false positives, for a binary classifier system as its discrimination threshold is varied. The ROC can also be represented equivalently by plotting the fraction of true positives (TPR = true positive rate) vs. the fraction of false positives (FPR = false positive rate). Also known as a Relative Operating Characteristic curve, because it is a comparison of two operating characteristics (TPR & FPR) as the criterion changes. [1] ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. ROC analysis is related in a direct and natural way to cost/benefit analysis of diagnostic decision making. The ROC curve was first developed by electrical engineers and radar engineers during World War II for detecting enemy objects in battle fields, also known as the signal detection theory, and was soon introduced in psychology ROC curve of three epitope predictors. to account for perceptual detection of signals. ROC analysis since then has been used in medicine, radiology, and other areas for many decades, and it has been introduced relatively recently in other areas like machine learning and data mining. Contents 1 Basic concept 2 ROC space 3 Curves in ROC space 4 Further interpretations 5 History 6 See also 7 References 7.1 General references 8 Further reading 9 External links Basic concept 1 sur 6 05/04/10 12:13

2 See also: Type I and type II errors A classification model (classifier or diagnosis) is a mapping of instances into a certain class/group. The classifier or diagnosis result can be in a real value (continuous output) in which the classifier boundary between classes must be determined by a threshold value, for instance to determine whether a person has hypertension based on blood pressure measure, or it can be in a discrete class label indicating one of the classes. Let us consider a two-class prediction problem (binary classification), in which the outcomes are labeled either as positive (p) or negative (n) class. There are four possible outcomes from a binary classifier. If the outcome from a prediction is p and the actual value is also p, then it is called a true positive (TP); however if the actual value is n then it is said to be a false positive (FP). Conversely, a true negative has occurred when both the prediction outcome and the actual value are n, and false negative is when the prediction outcome is n while the actual value is p. To get an appropriate example in a real-world problem, consider a diagnostic test that seeks to determine whether a person has a certain disease. A false positive in this case occurs when the person tests positive, but actually does not have the disease. A false negative, on the other hand, occurs when the person tests negative, suggesting they are healthy, when they actually do have the disease. Let us define an experiment from P positive instances and N negative instances. The four outcomes can be formulated in a 2"2 contingency table or confusion matrix, as follows: actual value p n total Terminology and derivations from a confusion matrix true positive (TP) eqv. with hit true negative (TN) eqv. with correct rejection false positive (FP) eqv. with false alarm, Type I error false negative (FN) eqv. with miss, Type II error sensitivity or true positive rate (TPR) eqv. with hit rate, recall TPR = TP / P = TP / (TP + FN) false positive rate (FPR) eqv. with false alarm rate, fall-out FPR = FP / N = FP / (FP + TN) accuracy (ACC) ACC = (TP + TN) / (P + N) specificity (SPC) or True Negative Rate SPC = TN / N = TN / (FP + TN) = 1! FPR positive predictive value (PPV) eqv. with precision PPV = TP / (TP + FP) negative predictive value (NPV) NPV = TN / (TN + FN) false discovery rate (FDR) FDR = FP / (FP + TP) Matthews correlation coefficient (MCC) F1 score F1 = 2TP / (P + P') Source: Fawcett (2006). prediction outcome p' n' True Positive False Negative False Positive True Negative P' N' ROC space total P N 2 sur 6 05/04/10 12:13

3 The contingency table can derive several evaluation "metrics" (see infobox). To draw an ROC curve, only the true positive rate (TPR) and false positive rate (FPR) are needed. TPR determines a classifier or a diagnostic test performance on classifying positive instances correctly among all positive samples available during the test. FPR, on the other hand, defines how many incorrect positive results occur among all negative samples available during the test. An ROC space is defined by FPR and TPR as x and y axes respectively, which depicts relative trade-offs between true positive (benefits) and false positive (costs). Since TPR is equivalent with sensitivity and FPR is equal to 1 - specificity, the ROC graph is sometimes called the sensitivity vs (1 - specificity) plot. Each prediction result or one instance of a confusion matrix represents one point in the ROC space. The best possible prediction method would yield a point in the upper left corner or coordinate (0,1) of the ROC space, representing 100% sensitivity (no false negatives) and 100% specificity (no false positives). The (0,1) point is also called a The ROC space and plots of the four prediction examples. perfect classification. A completely random guess would give a point along a diagonal line (the so-called line of no-discrimination) from the left bottom to the top right corners. An intuitive example of random guessing is a decision by flipping coins (head or tail). The diagonal line divides the ROC space in areas of good or bad classification/diagnostic. Points above the diagonal line indicate good classification results, while points below the line indicate wrong results (although the prediction method can be simply inverted to get points above the line). Let us look into four prediction results from 100 positive and 100 negative instances: A B C C' TP=63 FP=28 91 TP=77 FP= TP=24 FP= TP=76 FP=12 88 FN=37 TN= FN=23 TN=23 46 FN=76 TN=12 88 FN=24 TN= TPR = 0.63 TPR = 0.77 TPR = 0.24 TPR = 0.76 FPR = 0.28 FPR = 0.77 FPR = 0.88 FPR = 0.12 ACC = 0.68 ACC = 0.50 ACC = 0.18 ACC = 0.82 Plots of the four results above in the ROC space are given in the figure. The result A clearly shows the best among A, B, and C. The result B lies on the random guess line (the diagonal line), and it can be seen in the table that the accuracy of B is 50%. However, when C is mirrored onto the diagonal line, as seen in C', the result is even better than A. [C' should not be mirrored across the diagonal line, but rather through the center point. The calculations, above, are correct, and if you plot C' by hand, you will see that it should appear farther to the left and lower down. This proper location in the diagram would still be better than A.] Since this mirrored C method or test simply reverses the predictions of whatever method or test produced the C contingency table, the C method has positive predictive power simply by reversing all of its decisions. When the C method predicts p or n, the C' method would predict n or p, respectively. In this manner, the C' test would perform the best. While the closer a result from a contingency table is to the upper left corner the better it predicts, the distance from the random guess line in either direction is the best indicator of how much predictive power a method has, albeit, if it is below the line, all of its predictions including its more often wrong predictions must be reversed in order to utilize the method's power. Curves in ROC space Discrete classifiers, such as decision tree or rule set, yield numerical values or binary label. Applied to a set of instances, such classifiers yield a single point in the ROC space. Other classifiers, such as naive Bayesian and neural network, produce probability values representing the degree to which class the instance belongs to. For these methods, setting a threshold value will determine a point in the ROC space. For instance, if probability values below or equal to a threshold value of 0.8 are sent to 3 sur 6 05/04/10 12:13

4 the positive class, and other values are assigned to the negative class, then a confusion matrix can be calculated. Plotting the ROC point for each possible threshold value results in a curve. Further interpretations How an ROC curve can be interpreted Sometimes, the ROC is used to generate a summary statistic. Three common versions are: the intercept of the ROC curve with the line at 90 degrees to the no-discrimination line [citation needed] the area between the ROC curve and the no-discrimination line the area under the ROC curve, or "AUC", or A' (pronounced "a-prime") [2] d' (pronounced "d-prime"), the distance between the mean of the distribution of activity in the system under noise-alone conditions and its distribution under signal-alone conditions, divided by their standard deviation, under the assumption that both these distributions are normal with the same standard deviation. Under these assumptions, it can be proved that the shape of the ROC depends only on d'. The AUC is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one. [3] It can be shown that the area under the ROC curve is closely related to the Mann Whitney U, which tests whether positives are ranked higher than negatives. It is also equivalent to the Wilcoxon test of ranks. The AUC is related to the Gini coefficient (G 1 ) by the following formula [4] G = 2AUC, where: In this way, it is possible to calculate the AUC by using an average of a number of trapezoidal approximations. However, any attempt to summarize the ROC curve into a single number loses information about the pattern of tradeoffs of the particular discriminator algorithm. The machine learning community most often uses the ROC AUC statistic for model comparison. [5] This measure can be interpreted as the probability that when we randomly pick one positive and one negative example, the classifier will assign a higher score to the positive example than to the negative. In engineering, the area between the ROC curve and the no-discrimination line is often preferred, because of its useful mathematical properties as a non-parametric statistic [citation needed]. This area is often simply known as the discrimination. In psychophysics, d' is the most commonly used measure. The illustration at the top right of the page shows the use of ROC graphs for the discrimination between the quality of different epitope predicting algorithms. If you wish to discover at least 60% of the epitopes in a virus protein, you can read out of the graph that about 1/3 of the output would be falsely marked as an epitope. The information that is not visible in this graph is that the person that uses the algorithms knows what threshold settings give a certain point in the ROC graph. Sometimes it can be more useful to look at a specific region of the ROC Curve rather than at the whole curve. It is possible to compute partial AUC. [6] For example, one could focus on the region of the curve with low false positive rate, which is often of prime interest for population screening tests. [7] Another common approach for classification problems in which P << N (common in bioinformatics applications) is to use a logarithmic scale for the x-axis. [8] History The ROC curve was first used during World War II for the analysis of radar signals before it was employed in signal detection theory. [9] Following the attack on Pearl Harbor in 1941, the United States army began new research to increase the prediction of 4 sur 6 05/04/10 12:13

5 correctly detected Japanese aircraft from their radar signals. In the 1950s, ROC curves were employed in psychophysics to assess human (and occasionally non-human animal) detection of weak signals. [9] In medicine, ROC analysis has been extensively used in the evaluation of diagnostic tests. [10][11] ROC curves are also used extensively in epidemiology and medical research and are frequently mentioned in conjunction with evidence-based medicine. In radiology, ROC analysis is a common technique to evaluate new radiology techniques. [12]. In the social sciences, ROC analysis is often called the ROC Accuracy Ratio, a common technique for judging the accuracy of default probability models. ROC curves also proved useful for the evaluation of machine learning techniques. The first application of ROC in machine learning was by Spackman who demonstrated the value of ROC curves in comparing and evaluating different classification algorithms. [13] See also Constant false alarm rate Detection theory False alarm Gain (information retrieval) References 1. ^ Signal detection theory and ROC analysis in psychology and diagnostics : collected papers; Swets, ^ J. Fogarty, R. Baker, S. Hudson (2005). "Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction" ( ACM International Conference Proceeding Series, Proceedings of Graphics Interface Waterloo, Ontario, Canada: Canadian Human-Computer Communications Society. /citation.cfm?id= ^ Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27, ^ Hand, D.J., & Till, R.J. (2001). A simple generalization of the area under the ROC curve to multiple class classification problems. Machine Learning, 45, ^ Hanley, JA; BJ McNeil ( ). "A method of comparing the areas under receiver operating characteristic curves derived from the same cases" ( Radiology 148 (3): PMID ( Retrieved ^ McClish, Donna Katzman ( ). "Analyzing a Portion of the ROC Curve" ( /9/3/190). Med Decis Making 9 (3): doi: / x ( / %2F X ). PMID ( /cgi/content/abstract/9/3/190. Retrieved ^ Dodd, Lori E.; Margaret S. Pepe (2003). "Partial AUC Estimation and Regression" ( / / ). Biometrics 59 (3): doi: / ( / %2F ). PMID ( Retrieved ^ [1] ( 9. ^ a b D.M. Green and J.M. Swets (1966). Signal detection theory and psychophysics. New York: John Wiley and Sons Inc.. ISBN ^ M.H. Zweig and G. Campbell (1993). "Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine". Clinical chemistry 39 (8): PMID ( 11. ^ M.S. Pepe (2003). The statistical evaluation of medical tests for classification and prediction. New York: Oxford. 12. ^ N.A. Obuchowski (2003). "Receiver operating characteristic curves and their use in radiology". Radiology 229 (1): 3 8. doi: /radiol ( PMID ( /pubmed/ ). 13. ^ Spackman, K. A. (1989). "Signal detection theory: Valuable tools for evaluating inductive learning". Proceedings of the Sixth International Workshop on Machine Learning. San Mateo, CA: Morgan Kaufmann. pp General references X. H. Zhou, N. A. Obuchowski, and D. M. McClish (2002). New York, USA: Wiley & Sons. ISBN Further reading 5 sur 6 05/04/10 12:13

6 Zou, K.H., O'Malley, A.J., Mauri, L. (2007). Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation, 6;115(5): Lasko, T.A., J.G. Bhagwat, K.H. Zou and Ohno-Machado, L. (2005). The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics, 38(5): Balakrishnan, N., (1991) Handbook of the Logistic Distribution, Marcel Dekker, Inc., ISBN Gonen M., (2007) Analyzing Receiver Operating Characteristic Curves Using SAS, SAS Press, ISBN Green, W.H., (2003) Econometric Analysis, fifth edition, Prentice Hall, ISBN Hosmer, D.W. and Lemeshow, S., (2000) Applied Logistic Regression, 2nd ed., New York; Chichester, Wiley, ISBN Brown, C.D., and Davis, H.T. (2006) Receiver operating characteristic curves and related decision measures: a tutorial, Chemometrics and Intelligent Laboratory Systems, 80:24 38 Mason, S.J. and Graham, N.E. (2002) Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Q.J.R. Meteorol. Soc., 128: Pepe, M.S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford. ISBN Carsten, S. Wesseling, S., Schink, T., and Jung, K. (2003) Comparison of Eight Computer Programs for Receiver- Operating Characteristic Analysis. Clinical Chemistry, 49: Swets, J.A. (1995). Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers. Lawrence Erlbaum Associates. Swets, J.A., Dawes, R., and Monahan, J. (2000) Better Decisions through Science. Scientific American, October, pages External links Kelly H. Zou's Bibliography of ROC Literature and Articles ( /ppl/zou/roc.html) An introduction to ROC analysis ( A more thorough treatment of ROC curves and signal detection theory ( /notes/signal/) Diagnostic test evaluation - online calculator ( Tom Fawcett's ROC Convex Hull: tutorial, program and papers ( /ROCCH/index.html) Peter Flach's tutorial on ROC analysis in machine learning ( The magnificent ROC ( An explanation and interactive demonstration of the connection of ROCs to archetypal bi-normal test result plots Retrieved from "" Categories: Detection theory Data mining Socioeconomics Biostatistics Statistical classification This page was last modified on 30 March 2010 at 13:47. Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. See Terms of Use for details. Wikipedia is a registered trademark of the Wikimedia Foundation, Inc., a non-profit organization. 6 sur 6 05/04/10 12:13