Reducing Decision Errors in the Paired Comparison of the Diagnostic Accuracy of Continuous Screening Tests

Transcription

1 Reducing Decision Errors in the Paired Comparison of the Diagnostic Accuracy of Continuous Screening Tests Brandy M. Ringham, 1 Todd A. Alonzo, 2 John T. Brinton, 1 Aarti Munjal, 1 Keith E. Muller, 3 Deborah H. Glueck 1 1 Department of Biostatistics and Informatics, University of Colorado Denver 2 Department of Preventive Medicine, University of Southern California 3 Department of Health Outcomes and Policy, University of Florida

2 Acknowledgements The project described was supported by Award Number NCI 1R03CA A1 from the National Cancer Institute, and by Award Number NIDCR 3R01DE A1S1 from the National Institute of Dental and Craniofacial Research. The content is solely the responsibility of the authors, and does not necessarily represent the official views of the National Cancer Institute, the National Institute of Dental and Craniofacial Research nor the National Institutes of Health. 2

3 Outline Science Case Study Cancer Screening Trial Design Cancer Screening Analysis Statistics Bias Correction Algorithm Evaluation Studies Oral Cancer Screening Demonstration 3

4 Oral Cancer Screening Case Study VISIBLE LIGHT AUTOFLUORESCENCE No visible lesion Dark region confirmed to be carcinoma in situ 4

5 Paired Cancer Screening Trial Yes Observed disease Gold standard test Confirmed disease? Yes Participants screened by both tests Screen positive on either test? Yes No No Follow-up Signs and symptoms? No No observed disease 5

6 Paired Cancer Screening Trial Yes Observed disease Screening Test 1 Gold standard test Yes Confirmed disease? Participants screened by both tests Screen positive on either test? Yes No No Screening Test 2 Follow-up Signs and symptoms? No No observed disease 6

7 Paired Cancer Screening Trial Yes Observed disease Screening Test 1 Gold standard test Yes Confirmed disease? Participants screened by both tests Screen positive on either test? Yes No No Screening Test 2 Follow-up Signs and symptoms? No No observed disease 7

8 Paired Cancer Screening Trial Yes Observed disease Screening Test 1 Gold standard test Yes Confirmed disease? Participants screened by both tests Screen positive on either test? Yes No No Screening Test 2 Follow-up Signs and symptoms? No No observed disease 8

9 Paired Cancer Screening Trial Yes Observed disease Screening Test 1 Gold standard test Yes Confirmed disease? Participants screened by both tests Screen positive on either test? Yes No No Screening Test 2 Follow-up Signs and symptoms? No No observed disease 9

10 Paired Cancer Screening Trial Yes Observed disease Screening Test 1 Gold standard test Yes Confirmed disease? Participants screened by both tests Screen positive on either test? Yes No No Screening Test 2 Follow-up Signs and symptoms? No No observed disease 10

11 Hypothetical Cancer Screening Data Omniscient Viewpoint 3 1 True Non-Cases Screening Test 1 Score True Cases 4 Screening Test 2 Score 2 11

12 Hypothetical Cancer Screening Data Study Investigator s Viewpoint Screening Test 1 Score 3 1 Observed Non-Cases Observed Screen Positive Cases Observed Interval Cases 4 2 Screening Test 2 Score 12

13 Hypothetical Cancer Screening Data Omniscient Viewpoint Screening Test 1 Score 3 1 Observed Non-Cases Observed Screen Positive Cases Observed Interval Cases 4 2 Missed Cases Screening Test 2 Score 13

14 Analysis of Cancer Screening Studies Disease Status + - Screening Test + - a c b d 14

15 Analysis of Cancer Screening Studies Screening Test + - Disease Status + - a b c d Sensitivity = a a+c 15

16 Analysis of Cancer Screening Studies Screening Test + - Disease Status + - a b c d Sensitivity = a a+c Specificity = d b+d 16

17 Analysis of Cancer Screening Studies Screening Test + - Disease Status + - a b c d Sensitivity Area Under the Curve 1 - Specificity 17

18 Comparing Two Continuous Screening Tests True Screening Test 1 Screening Test 2 Sensitivity True Observed Corrected 1 - Specificity (Glueck et al., 2009) 18

19 Comparing Two Continuous Screening Tests Observed Screening Test 1 Screening Test 2 Sensitivity True Observed Corrected 1 - Specificity (Glueck et al., 2009) 19

20 Comparing Two Continuous Screening Tests Observed Screening Test 1 Screening Test 2 Sensitivity True Observed Corrected 1 - Specificity (Glueck et al., 2009) 20

21 Comparing Two Continuous Screening Tests Corrected Screening Test 1 Screening Test 2 Sensitivity True Observed Corrected 1 - Specificity (Glueck et al., 2009) 21

22 Types of Bias (Rutjes et al., 2007) 22

23 Bias Correction Algorithm 1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases. 2. Use the maximum likelihood estimates and the sampling fractions in each partition to calculate weighted estimates. 23

24 Bias Correction Algorithm 1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases. Screening Test 1 Score (Nath, 1971) Screening Test 2 Score 24

25 Bias Correction Algorithm 1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases. Screening Test 1 Score (Nath, 1971) Screening Test 2 Score 25

26 Bias Correction Algorithm 1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases. Screening Test 1 Score μ 1 μ 2 σ 2 1 σ 2 2 ρ (Nath, 1971) Screening Test 2 Score 26

27 Bias Correction Algorithm 1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases. Screening Test 1 Score μ 1 μ 2 σ 2 1 σ 2 2 ρ (Nath, 1971) Screening Test 2 Score 27

28 Bias Correction Algorithm 1. Find the maximum likelihood estimates of the parameters of the bivariate Gaussian distribution of test scores for the cases. Screening Test 1 Score μ 1 μ 2 σ 2 1 σ 2 2 ρ (Nath, 1971) Screening Test 2 Score 28

29 Bias Correction Algorithm 2. Use the maximum likelihood estimates and the sampling fractions in each partition to calculate weighted estimates. 29

30 Bias Correction Algorithm 2. Use the maximum likelihood estimates and the sampling fractions in each partition to calculate weighted estimates. Screening Test 1 Score Screening Test 2 Score 30

31 Bias Correction Algorithm 2. Use the maximum likelihood estimates and the sampling fractions in each partition to calculate weighted estimates. Screening Test 1 Score Screening Test 2 Score 31

32 Weighted Estimates 32

33 Weighted Estimates 33

34 Weighted Estimates 34

35 Weighted Estimates 35

36 Weighted Estimates (Equation 3.3.1, p. 81, Kish, 1995; Proposition 5.2, p. 348, Ross, 2009) 36

37 Weighted Estimates (Equation 3.3.1, p. 81, Kish, 1995; Proposition 5.2, p. 348, Ross, 2009) 37

38 Weighted Estimates (Equation 3.3.1, p. 81, Kish, 1995; Proposition 5.2, p. 348, Ross, 2009) 38

39 Weighted Estimates (Equation 3.3.1, p. 81, Kish, 1995; Proposition 5.2, p. 348, Ross, 2009) 39

40 Design for Simulation Studies Define 14 parameters Generate bivariate Gaussian data Deduce observed disease status Bias correction algorithm Complete analysis Observed analysis Corrected Analysis 40

41 Design for Simulation Studies Define 14 parameters Generate bivariate Gaussian data Deduce observed disease status Bias correction algorithm Complete analysis Observed analysis Corrected Analysis 41

42 Design for Simulation Studies Define 14 parameters Generate bivariate Gaussian data Deduce observed disease status Bias correction algorithm Complete analysis Observed analysis Corrected Analysis 42

43 Design for Simulation Studies Define 14 parameters Generate bivariate Gaussian data Deduce observed disease status Bias correction algorithm Complete analysis Observed analysis Corrected Analysis 43

44 Design for Simulation Studies Define 14 parameters Generate bivariate Gaussian data Deduce observed disease status Bias correction algorithm Complete analysis Observed analysis Corrected Analysis 44

45 Design for Simulation Studies Define 14 parameters Generate bivariate Gaussian data Deduce observed disease status Bias correction algorithm Complete analysis Observed analysis Corrected Analysis 45

46 Evaluation of the Method Sensitivity 1 - Specificity 46

47 Evaluation of the Method Sensitivity Type I Error 1 - Specificity 47

48 Evaluation of the Method Sensitivity 1 - Specificity 48

49 Evaluation of the Method Sensitivity Power 1 - Specificity Wrong Rejection Fraction Correct Rejection Fraction 49

50 Disease Prevalence Simulation Study 50

51 Percent Identified Simulation Study Bias? Percent Identified ( Test 1 / Test 2) Complete Type I Error Observed Type I Error Corrected Type I Error Yes 15/ / / No 15/ / /

52 Percent Identified Simulation Study Bias? Percent Identified ( Test 1 / Test 2) Complete Type I Error Observed Type I Error Corrected Type I Error Yes 15/ / / No 15/ / /

53 Percent Identified Simulation Study Bias? Percent Identified ( Test 1 / Test 2) Complete Type I Error Observed Type I Error Corrected Type I Error Yes 15/ / / No 15/ / /

54 Recommendation Study investigators should conduct a simulation of their study using both the standard analysis and the bias correction method. Study investigators should choose the analysis plan that has a nominal Type I error rate and the highest power for the correct decision. 54

55 Oral Cancer Screening Demonstration VISIBLE LIGHT AUTOFLUORESCENCE No visible lesion Dark region confirmed to be carcinoma in situ 55

56 Oral Cancer Screening Analysis Sensitivity Standard Δ = 0.06 (p = 0.005) Specificity Autofluorescence Visible Light 56

57 Decision Errors Simulation Type I Error Wrong Rejection Correct Rejection Corrected Standard Corrected Standard Corrected Standard 57

58 Oral Cancer Screening Analysis Sensitivity Standard Δ = 0.06 (p = 0.005) Specificity Autofluorescence Visible Light 58

59 Oral Cancer Screening Analysis Standard Corrected Sensitivity Δ = 0.06 (p = 0.005) Δ = (p = 0.001) Specificity Autofluorescence Visible Light 59

60 Oral Cancer Screening Analysis Sensitivity True Standard Corrected Δ = (p = 0.001) Δ = 0.06 (p = 0.005) Δ = (p = 0.001) Specificity Autofluorescence Visible Light 60

61 References Glueck, D. H., Lamb, M. M., O'Donnell, C. I., Ringham, B. M., Brinton, J. T., Muller, K. E., Lewin, J. M., et al. (2009). Bias in trials comparing paired continuous tests can cause researchers to choose the wrong screening modality. BMC Medical Research Methodology, 9, 4. doi: / kish, L. (1995). Survey Sampling. Wiley-Interscience. Kish, L. (1995). Survey Sampling. Wiley-Interscience. Lewin, J. M., D'Orsi, C. J., Hendrick, R. E., Moss, L. J., Isaacs, P. K., Karellas, A., & Cutter, G. R. (2002). Clinical comparison of full-field digital mammography and screen-film mammography for detection of breast cancer. AJR. American Journal of Roentgenology, 179(3), Nath, G. B. (1971). Estimation in Truncated Bivariate Normal Distributions. Journal of the Royal Statistical Society. Series C (Applied Statistics), 20(3), doi: / Obuchowski, N. A., & McClish, D. K. (1997). Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Statistics in Medicine, 16(13), Ross, S. (2009). First Course in Probability, A (8th ed.). Prentice Hall. Rutjes A.W.S., Reitsma J.B., Coomarasamy A., Khan K.S., Bossuyt P.M.M. (2007). Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol. Assess., 11(50). Zhou, X.-H., McClish, D. K., & Obuchowski, N. A. (2002). Statistical Methods in Diagnostic Medicine (1st ed.). Wiley-Interscience. 61

62 Literature Review Re-weighting, imputation, and Bayesian approaches have been proposed to reduce the effect of partial verification bias Maximum likelihood methods and latent class models have been proposed to estimate diagnostic accuracy in the presence of imperfect reference standard bias A method using general estimating equations can correct for missing disease status, but does not account for misclassification of disease status. We have not found any methods that reduce the effect of paired screening trial bias. 62

63 Nath Algorithm (Nath, 1971) 63

64 Nath Algorithm 1 1 (Nath, 1971) 64

65 Nath Algorithm 2 2 (Nath, 1971) 65

66 Nath Algorithm 3 (Nath, 1971) 66

67 67