Evaluation of diagnostic tests

Transcription

1 Evaluation of diagnostic tests Biostatistics and informatics Miklós Kellermayer Overlapping distributions Assumption: A classifier value (e.g., diagnostic parameter, a measurable quantity, e.g., serum concentration) changes (e.g., increases) in disease. Diagnostic objective: Predict the outcome (healthy versus ill) based on the classifier value. ΔN h Novel representation: healthy combined healthy ill ill ΔN d Classifier 2

2 ΔN h healthy ill Full overlap Based on overlap magnitude: Useless method ΔN d ΔN h healthy ill Partial overlap Real-life situation ΔN d ΔN h healthy ill Complete separation Perfect method ΔN d 3 Real-life situation: Partial overlap ΔN h healthy ill ΔN d Discrimination threshold Actual value of classifier (diagnostic method) Negative Positive Prediction Healthy Diseased (sick) True Negative () ( correct rejection ) False Negative() ( miss, Type II error) False Positive () ( false alarm, Type I error) True Positive () ( hit ) 4 2

3 Diagnosed negative Discrimination threshold Diagnosed positive Frequency Healthy Classifier (test parameter) Diseased (sick) Classifier (test parameter) Disease present Diagnosed negative Discrimination threshold Diagnosed positive 5 Contingency table: Confusion matrix (binary classification) Δn h True negative negative positive False postive healthy diseased Real condition: healthy or diseased Δn d False negative Test result (prediction): negative or positive True positive 6 3

4 Prevalence -Measure of how common the disease is -Probability prior to test (a priori probability) Frequency of diseased in examined population = w = diseased total = + ACC SPC = R SPC ACC = accuracy SPC = specificity R = true positive rate (sensitivity) 7 Shape of combined distributions w = 25% w = 50% w = 75% 8 4

5 Important parameters of diagnostic goodness The goodness of a test can be described in terms of the following diagnostic parameters:. True Positive Rate, R (sensitivity) 2. True Negative Rate, R (specificity, SPC) 3. Positive Predictive Value, PPV (precision, diagnostic relevance) 4. Negative Predictive Value (diagnostic segregation) Only three of them are independent! Every method must be compared with a reference ( Gold standard ) Gold standard: method known to work; often autopsy 9 A priori (before test) probabilities are independent of prevalence -True Positive Rate (R) -Hit Rate -Recall Diagnostic sensitivity Probability that the test finds the diseased positive. Positive within diseased. P(positive diseased) = R = diseased = + Large-sensitivity tests (00%) are required in early diagnosis (screening) so that few patients remain unrecognized. 0 5

6 Discrimination threshold sensitivity R = 50% R = + R = 70% R = 90% R = 00% Diagnostic specificity (SPC) -True Negative Rate (R) Probability that the test finds a healthy negative. Negative among healthy P(negative healthy) = SPC = healthy = + High-specificity tests (near 00 %) are important when the false positive values have severe consequences (e.g., surgery). 2 6

7 Discrimination threshold specificity SPC = 50% SPC = 70% SPC = 90% SPC = + SPC = 00% 3 False Negative Rate (R) -Rate of Type-II error Probability that the test finds the diseased negative. Negative among diseased. P(negative diseased) = R = diseased = + R = True Positive Rate, sensitivity 4 7

8 False Positive Rate (R) -Rate of Type-I error Probability that the test finds a healthy positive. Positive among healthy. P(positive healthy) = SPC = healthy = + 5 A posteriori (after test) probabilities depend strongly on prevalence Diagnostic precision -Positive Predictive Value (PPV) -Relevance Probability of diseas if test is positive. Diseased among positive. P(diseased positive) = PPV = total positive = + = R w R w + ( SPC) ( w) SPC = specificity R = True Positive Rate, sensitivity 6 8

9 Negative Predictive Value (NPV) -Correct negativity -Segregation Probability of healthiness if test is negative. Healthy among negative P(healthy negative) = NPV = total negative = + = SPC ( w) SPC ( w) + ( R) w SPC = specificity R = True Positive Rate, sensitivity 7 False Discovery Rate (FDR) -False alarm rate Probability of healthiness if test is positive. Healthy among positive. P(healthy positive) = PPV = total positive = + PPV = Positive Predictive Value (precision) 8 9

10 False Reassurance Rate (FRR) -False reassurance rate Probability of disease if test is negative. Diseased among negative. P(diseased negative) = NPV = total negative = + NPV = Negative Predictive Value 9 Diagnostic efficiency -Accuracy (ACC) Ratio of correct classification = ACC = + total = + = R w + SPC ( w) SPC = specificity R = True Positive Rate, sensitivity Discrimination threshold is chosen so that accuracy is maximized. 20 0

11 Effect of prevalence Case : w = 50% SPC = 90% SPC = 90% Gold standard (ACC, de = 90%) Case 2: w = 0% Gold standard (ACC, de = 90%) NPV = 90% test negative positive healthy 90 0 diseased 0 90 Precision, PPV = 90% NPV = 99% test negative positive healthy diseased 0 90 PPV = 50% Sensitivity (R) = 90% Sensitivity (R) = 90% 2 In case of very small prevalence a highly sensitive and specific test could be of low precision (Positive Predictive Value, PPV, relevance). prevalence = 0. % sensitivity = 98 % specificity = 98 % precision = 4 % 22

12 Comparison of diagnostic tests: the ROC space ROC: Receiver Operating Characteristic ROC curve is a graphical plot of the sensitivity (R) versus false positive rate (-specificity) for a binary classifier system as its discrimination threshold is varied. First ROC curve used in World War II for analysis of radar signals. In the 950s, ROC curves were employed in psychophysics to assess detection of weak signals, then later in medicine in the evaluation of diagnostic tests. specificity 0 sensitivity sensitivity 0 0 -specificity 0 23 Example of a ROC curve sensitivity ROC curves of three epitope predictors -specificity 24 2

13 Application of the ROC space 25 Dependence of ROC curve on diagnostic parameters I: accuracy Equation of ROC curve: R = w w ( SPC) + w ACC + w w Dependent variable Slope R = Tre Positive Rate, senstivity w = prevalence SPC = specificity Independent variable y-intercept ACC = Accuracy (diagnostic efficiency) Sensitivity (R) Prevalence, w = 0.5 (therefore slope = ) ACC = ACC = specificity Increasing accuracy increases y-intercept, hence improves classification. 26 3

14 Case : w = 0., slope= 9 Dependence of ROC curve on diagnostic parameters II: prevalence If w < 0.5, at identical accuracies the slope is greater than. R = w w ( SPC) + w ACC + w w Dependent variable Slope Independent variable y-intercept If w > 0.5, then at identical accuracies the slope is smaller than. Case 2: w = 0.6, slope= 0.66 w = 0. w = 0.6 de = de = sensitivity de = 0.5 sensitivity de = specificity 0 -specificity 27 Identical accuracies define isoefficiency curves in the ROC space 28 4

15 Comparison of diagostic methods in ascites Carcinoembryonic antigen (CEA) and cholesterol are both increased in the ascites in carcinosis, raising the possibility of using either of them as diagnostic tools. Which one is better? What discrimination threshold should be used? CEA cholesterol Gulyás M, Kaposi AD, Elek G, Szollár LG, Hjerpe A, Value of carcinoembryonic antigen (CEA) and cholesterol assays of ascitic fluid in cases of inconclusive cytology, J Clinical Pathology 200 (54) Cholesterol level in ascites provides better test across all prevalences 30 5

16 Twist in the ROC space... Alcoholism diagnostics with CDT (carbohydrate deficient transferrin) and γ-gt (gamma-glutamyltransferase) CDT is better at low prevalences. In case of high prevalences, the GGT test excels. sensitivity sensitivity specificity specificity 3 6