2011 ASCP Annual Meeting

Size: px

Start display at page:

Download "2011 ASCP Annual Meeting"

Loraine Clarke
5 years ago
Views:

1 Diagnostic Accuracy Martin Kroll, MD Professor of Pathology and Laboratory Medicine Boston University School of Medicine Chief, Laboratory Medicine Boston Medical Center

2 Disclosure Roche Abbott

3 Course Objectives List reasons why diagnostic accuracy is important. Interpret Predictive value, ROC curves, odds ratio and relative risk. Describe the proper use of these tools. Evaluate the plan and quality of study designs

4 Diagnostic Accuracy The REASONS Why

5 The Challenge of the Complex Patient Complex patients present with numerous risk factors (CHD, renal, metabolic) One-fourth US population has metabolic syndrome 80 million adults have cardiovascular disease Patients do better when Dx and Rx immediately and effectively for complications C Moriates, A Maisel. The Utility of Biomarkers in sorting out the complex patient. Amer J Med. 2010; 123:

6 Clinical Significance of Biomarkers for the Complex Patient Inexpensive tools that may help differentiate diseases in complex patients Both sensitive and specific: disease state Novel biomarkers Highly sensitive troponin Procalcitonin Neutralphil gelatinase lipocalin (N-GAL) C Moriates, A Maisel. The Utility of Biomarkers in sorting out the complex patient. Amer J Med. 2010; 123:

7 Hypothetical Combination of BNP, N-GAL and Creatinine Concentratio on BNP N-GAL Determine the euvolemic state in Decompensated heart failure. creatinine Time C Moriates, A Maisel. The Utility of Biomarkers in sorting out the complex patient. Amer J Med. 2010; 123:

Important Biomarkers for CAD MI Risk LDL/HDL HgA1c MPO CRP HCY Unstable Angina MPO D-dimer IMA FABP Inflammation CRP CD40L OxLDL MCP-1 ST2 TNFα Myocard Ischemia MDA-LDL IMA FFA BNP Choline PIGF

8 Important Biomarkers for CAD MI Risk LDL/HDL HgA1c MPO CRP HCY Unstable Angina MPO D-dimer IMA FABP Inflammation CRP CD40L OxLDL MCP-1 ST2 TNFα Myocard Ischemia MDA-LDL IMA FFA BNP Choline PIGF Plaque Rupture MMPs CD40L MPO CRP HCY IL-6 Myocardial Necrosis Myo CK-MB TnI/TnT BNP C Moriates, A Maisel. The Utility of Biomarkers in sorting out the complex patient. Amer J Med. 2010; 123:

9 Trend in Patient Care Patients are more complex: Obesity, CVD, DM, Metabolic Syndrome, Chronic renal disease Complex patients difficult to diagnose: need inexpensive tools Novel biomarkers further elucidate underlying pathologies with earlier detection C Moriates, A Maisel. The Utility of Biomarkers in sorting out the complex patient. Amer J Med. 2010; 123:

10 Insulin, Proinsulin Risk Type 2 Diabetes Mellitus in Women Parameter Insulin, pmol/l, Q3 Insulin, pmol/l, Q4 Proinsulin, pmol/l Proinsulin, pmol/l Range Odds Ratio (95% C.I.) ( ) > (1.8-18) 18) (0.8-36) >= (5.8-47) Pradhan. Am J Med. 2003:114:

11 Reference Range Concept F R A C T I O N Healthy 2 SD 2 SD 10 Diseased TEST RESULT Clear separation

12 Reference Range Reality F R A C T I O N Non-Diseased FN 2 SD 2 SD 10 Indeterminant Diseased TEST RESULT FP

13 Definitions Sensitivity: Ability of a test to yield positive results for patients with the disease or condition Relates to true positives Specificity Ability of a test to yield negative results in the absence of the disease or condition Relates to true negatives

14 Predictive Value of a Test Bayes Theorem P( D / T+ ) = + / D) P( D) P( T+ ) PVP: likelihood that someone with a positive test will have the characteristic PVN: likelihood that someone with a negative test will NOT have the characteristic What Physicians use Prevalence dependent P( T

15 Contingency Table Cond + Cond - Test PVP 0.62 Test neg Sensitivity Specificity PVN 0.63

16 The Problem of Prevalence: S&S = 0.9 Prevalence P(H)= 0.2 : PVP = 0.7; PVN = 0.97 P(H)= 0.05 : PVP = 0.3; PVN = 0.99 Usefulness depends on Prevalence. Cutoff crucial. F R A C T I O N Non-diseased Diseased cutoff TEST RESULT

17 Designing the Basic Evaluation Define the Clinical Question Identify the role of the test(s) in making the decision. Select a Statistically Valid, Representative Study Sample Select, prospectively, a statistically valid sample that consists of representative subjects Establish the True Clinical State or Outcome of each Subject Test the Study Subjects Perform the test(s) without knowing the clinical classification of the subjects. Assess the Clinical Accuracy of the Test(s) Construct and analyze receiver operating characteristic (ROC) plots, calculate OR, RR, HR

18 Define the Clinical Question Can the test discriminate between the condition and other disease states Example: troponin distinguish between MI and other chest pain causes Which test is best? Assume population of similar subjects Share disease characteristics, e.g., HF Age, gender, stage of disease

19 Select Representative Study Sample Choose a relevant sample of subjects; need a similar background Example: pancreatitis; both positive and negative groups should present with similar history and symptoms

20 Disease Spectrum Stage of disease or condition Severity of disease Genetic background Body composition Multiple diseases (co-morbidity) Lifestyle Demographics

21 Spectrum Effect Different condition substrata affects test performance estimators Condition of interest presented as binary More than binary manifestation Varying substrata alters ability of plot to adequately differentiate test abilities Alters Sensitivity and Specificity

22 Sample Size Dependent Prevalence Sensitivity Specificity Often 100 Equal positive and negative

23 Establish True Clinical State Gold Standard Biopsy Surgical or autopsy findings Imaging data Long-term clinical outcome Definitive test

24 Independent Classification Determine clinical state independent of the test under investigation Avoid the circular argument Avoid similar tests Frequently violated

25 Running the Test Identical specimens For comparing two tests Same subject Obtained at the same time Assay when specimens are stable Analytical test conditions Accurate (trueness) Precision (reproducibility)

26 Why Should One Learn About ROC Curves? Determine whether test is better than chance Find optimum cutoff Evaluate asymmetry between sensitivity and specificity Assessment of diagnostic accuracy Evaluate whether two tests for the same process have the same diagnostic accuracy

27 Normals = Abnormals Normal Abnormal 1 ROC Curve Numb ber PSA Value Sensitivity Specificity

28 Abnormals >> Normals Num mber Sensit tivity Values Normal Cancer 1-Specificity

29 Abnormals > Normals PSA Distributions PSA C o u n t PSA Normal Cancer itivity Sens Specificity

30 Table 1. Determinations of Assay X for Eight Patients. Assay X Patient D x

31 Calculation for Values of X Assay Values Dis X > 1 X > 2 X > 3 X > 4 X > 5 X > TN 2 0 FP 3 1 TP 3 0 FP 4 1 TP 5 1 TP 5 0 FP 6 1 TP TN TN TP FP TP TP FP TP TN TN FN TN TP TP FP TP TN TN FN TN FN TP FP TP TN TN FN TN FN FN TN TP TN TN FN TN FN FN TN FN TP TN FP FN

32 ROC Plot for Assay X Sensitivity (true posit tives) No discrimination assay x Specificity (false positives)

33 Dotted line intersects ROC curve where Sensitivity=Specificity, 0.75 ROC Plot for Assay X Assay X No discrimination Choosing the optimal CUTOFF. ) True positive rate (Sensitivity) False positive rate (1-Specificity)

34 Dotted line intersects ROC curve where Sensitivity=Specificity, 0.75 ROC Plot for Assay X Assay X No discrimination Choosing the optimal CUTOFF. ) True positive rate (Sensitivity) False positive rate (1-Specificity)

35 Choice of Sensitivity ROC Plot for Assay X Assay X No discrimination True positive rate (Sensitivity y) False positive rate (1-Specificity)

36 Choice of Specificity ROC Plot for Assay X Assay X No discrimination True positive rate (Se ensitivity) False positive rate (1-Specificity)

37 Comparing Two Markers ROC curve for tw o diagnostic m arkers Sensitivit ty Marker A Marker B 1-Specificity

38 AUC ROC Plot for Assay X Assay X No discrimination AUC = 0.81 True positive rate (Se ensitivity) B C A Is test useful if SE = 0.20? SE = 0.15? AUC z 50 1 = 0. SE False positive rate (1-Specificity)

39 Comparing Two Markers AUC: A: Hypothetical Marker Comparison B: 0.89 Sensit tivity Are they significantly Different, if SE = 0.05 SE = z = AUC AUC 2 1 SE 1-Specificity Equivocal BNP NT-proBNP Test same samples from same patients Variance estimate includes correlation between samples

40 Uses of ROC Plots Determine whether test is better than chance Find optimum cutoff Evaluate asymmetry between sensitivity and specificity Assess diagnostic accuracy: AUC, C- statistic (additive effect) Compare two tests for same process: same or dissimilar diagnostic accuracies

41 Clinical Evaluations Discrete Testing Two possibilities Tests Relative Risk Odds Ratio Comparison of means or distributions inadequate to assess diagnostic accuracy (t- test, non-parametric comparisons)

42 Relative Risk of Disease (condition) Ratio of incidence of those with the positive test (T+) divided by those a negative test (T-) Calculated from a cohort where T+ and T- are identified first, and disease later Clinical Trials Originally used medication or toxic material

43 Relative Risk Many tests assign risk instead of discrimination between diseases Large studies Compare test positivity with risk of developing disease RR: prospective study Ratio disease positive test vs negative test

44 Contingency Table: Relative Risk Cond + Cond - Risk Test T Test neg T Sensitivity Specificity Ratio

45 Contingency Table: Relative Risk Cond + Cond - Risk Test + A B A+B T+ A/(A+B) Test neg C D C+D T- C/(C+D) A+C B+D A Sensitivity Specificity Ratio ( A+ B) RR= C A/(A+C) D/(B+D) ( C+ D)

46 Relative Risk: HOPE Study Microalbuminuria, >55 YO, h/o CVD or DM w 1 risk factor 5 year interval: Albumin/creatinine ratio (urine) Mortality, + Mortality, - Risk Albuminuria, T Albuminuria, T Sensitivity Specificity Ratio CI Gerstein. JAMA 2001; 286:421-6.

47 Relative Risk of highest to lowest quartile CRP: 4.4 Interleukin 6: 2.2 LDL cholesterol: 2.4 Apoliprotein B-100: Q1 Q2 Q3 Q4 CRP and other inflammation markers in prediction of CDV in women. Ridker et al. N Engl J Med 2000;342:

48 Novel Risk Factors for Systemic Atherosclerosis (PAD) 95% C.I. Variable RR: Q3 RR: Q4 P value* LDL-C 1.8 ( ) 2.3 ( ) HDL-C 0.6 ( ) 0.5 ( ) 0.03 Apo-B 2.0 ( ) 2.9 ( ) <.001 Lipoprotein(a) 0.9 ( ) 1.1 ( ) 0.6 Fibrinogen 1.4 ( ) 2.2 ( ) 0.02 CRP 1.7 ( ) 2.8 ( ) 0.01 *Trend across quartiles. Ridker. JAMA 2001:285:

49 Relative Risk Importance Large prospective studies Set policy and major practice patterns RR for cholesterol of 240 mg/dl is twice that for cholesterol at 200 mg/dl Expect values greater than 1.0, usually greater than 1.5

50 Odds Ratio Valuable in characterizing populations at risk Can identify when marker is not beneficial Sensitive to study stress on death or survival (RR insensitive to survival) Measures strength of association between marker and disease Judge using confidence limits

51 Odds Ratio Equals RR only in rare diseases Not as reliable as relative risk Many more studies than with RR Positive over 1.0, expect > 3.0 Very useful, not as powerful as RR Uses much lower number of cases Case-control studies (less expensive than prospective studies)

52 Odds Ratio Interpretation When equal 1.0, no association When < 1.0, inverse association, less likely When > 1.0, direct association, more likely Moderate association OR > 3.0 Pepe MS. Am J Epidemiol 2004;159:

53 Reasons for Odds Ratio Retrospective studies more common Use Odds ratio Fraction test positivity given disease divided by fraction test negativity in absence of disease

54 Calculation of the Odds Ratio Cond + Cond - ODDS RATIO Test + A B Test neg C D A+C B+D Ratio A*D/B*C Sensitivity Specificity A D A/(A+C) D/(B+D) OR = B C Also by Logistic regression analysis

55 Odds Ratio: HOPE Study Microalbuminuria, >55 YO, h/o CVD or DM w 1 risk factor 5 year interval: Albumin/creatinine ratio (urine) Mortality, + Mortality, - ODDS RATIO Albuminuria, A*D Albuminuria, Ratio B*C Sensitivity Specificity Ratio RR 1.8 ( ) Gerstein. JAMA 2001; 286:421-6.

56 Insulin, Proinsulin Risk Type 2 Diabetes Mellitus in Women Parameter Insulin, pmol/l, Q3 Insulin, pmol/l, Q4 Proinsulin, pmol/l Proinsulin, pmol/l Range Odds Ratio (95% C.I.) ( ) > (1.8-18) 18) (0.8-36) >= (5.8-47) Pradhan. Am J Med. 2003:114:

57 Assessment 54 Biomarkers for Biopsy- detectable Prostate Cancer Marker (serum) OR (P) AUC, % (P) PSA 2.74 (<0.001) 80.4 (<0.001) AFP 1.55 (0.04) 57.5 (0.02) Kallikrein (0.02) 57.3 (0.02) MMP (0.65) 53.9 (0.15) EGFR 0.65 (0.34) 54.9 (0.09) CEA 0.90 (0.12) 54.9 (0.09) Cancer Epid Biomarkers Prev 2007;16:

58 Use of RR and OR For tests used to define risk: cholesterol, hs-crp, cancer, stroke, genomics Expect manufacturer to present information in package inserts and literature Should find published studies

59 Conclusions Predictive value applies to patient Sensitivity and specificity are basic measures Remember for studies Same sample of subjects Same specimens to test Gold standard is critical Biomarkers must show independence

60 ROC Curve Conclusions Prevalence independent Spectrum dependent Choose Cut-off AUC Test signficance Compare tests Evaluate improved information from test

61 Relative Risk and Odds Ratio Conclusions Define test usefulness Use Confidence interval to evaluate Prospective and case-controlled controlled studies Superior to t-testtest

62 Perspective on Diagnostic Accuracy RR, OR and HR depicts association, not discrimination Need high values, best if > 3.0 Useful in eliminating potential markers Confidence interval straddles 1.0 P >.05

63 Perspective on Diagnostic Accuracy RR and OR and HR can be statistically significant even with large overlap AUC (ROC) or c-statistic used for discrimination Affected by disease spectrum effect (bias) Other criteria Improvement shown change in c-statistic Reclassification

64 Change in c-statistic Use logistic regression Combine models Usual risk factors Typical biomarkers Varying combinations of the new markers Look for significant improvement in the c- statistic

65 Examples change in c-statistic Hope study, addition of NT-proBNP raised AUC from 0.73 to 0.80 (signficant) Addition of genetic variant SNP 9p21 did not raise the AUC Studies using multiple genetic markers Raise the RR only about 1.1 As number genetic markers increase, incremental gain decreases TJ Wang. Circulation. 2001;123:

66 THANK YOU!

67 Contact Information Martin Kroll, MD Chief, Department of Laboratory Medicine Boston Medical Center 670 Albany Street, Room 735 Boston, MA

68 Additional Slides

69 Number of Samples TPF or FPF One half CI Width N

70 References Zweig M, Campbell G. Clin Chem 1993;39:584. Mulherin SA, Miller WC. Spectrum bias. Ann Intern Med. 2002;137: Hanley JA, McNeil BJ. The meaning and use of the area under a ROC curve. Radiology. 1982;143: Hanley JA, McNeil BJ. A method of comparing the areas under ROC curves derived from the same cases. Radiology. 1983;143: Zhou XH, Obuchowski NA, McClish DK. Statistical Methods in Diagnostic Medicine, Wiley, NY, 2002.

70 Diagnostic Accuracy. Martin Kroll MD

70 Diagnostic Accuracy. Martin Kroll MD 7 Diagnostic Accuracy Martin Kroll MD 2 Annual Meeting Las Vegas, NV AMERICAN SOCIETY FOR CLINICAL PATHOLOGY 33 W. Monroe, Ste. 6 Chicago, IL 663 7 Diagnostic Accuracy Pathologists and laboratory professionals