SYSTEMATIC REVIEWS OF TEST ACCURACY STUDIES

Transcription

1 Biomarker & Test Evaluation Program SYSTEMATIC REVIEWS OF TEST ACCURACY STUDIES Patrick MM Bossuyt

2

3

4

5 Structure 1. Clinical Scenarios 2. Test Accuracy Studies 3. Systematic Reviews 4. Meta-Analysis 5. Interpretation 6. Discussion

6 1. Clinical Scenario - One D-dimer in Suspected Pulmonary Embolism

7 Pulmonary Embolism

8 Jaapa.com

9 Pulmonary Angiography

10 Ventilation/perfusion scan

11 Computed Tomography

12

13 1. Clinical Scenario - Two Urinary bladder tumor markers In Bladder Cancer Recurrence

14 Bladder Cancer

15 Predisposition Risk Stratification Early Detection Surveillance Disease Diagnosis Monitoring (Disease) Staging Treatment Selection Prognosis Monitoring (Treatment )

16 Urine Cytology

17 Cystoscopy

18 Urinary bladder tumor markers

19 2. Test Accuracy Studies

20 Test Accuracy How good is the test in correctly classifying patients as being diseased?

21 Test Accuracy How good is the test in correctly classifying patients as having the target condition?

22

23

24 Purpose of Testing: Target Condition Link between test and Management Downstream Consequences Patient Outcomes Central in every accuracy study Central in every systematic review

25 Diagnostic Accuracy Study Index Test Gold Standard

26 Diagnostic Accuracy Study Series of Patients Index Test Reference standard Cross-classification

27 The Results Reference Standard Target Condition Other Condition Positive Negative

28 The Results Reference Standard Target Condition Other Condition Positive TP FP Negative FN TN

29 Urinary bladder tumor markers

30

31 BTA stat

32 Diagnostic Accuracy Study 501 patients BTA stat Cystoscopy Cross-classification

33 The Results Cystoscopy BCR No BCR Positive Negative BCR: bladder cancer recurrence

34 Measures of Test Accuracy Sensitivity & Specificity 54 % 74 % Pos. & Neg. Pred. Value 43 % 81 % Likelihood Ratios Diagnostic Odds Ratio 3.2 BCR no BCR Positive Negative

35

36

37 Diagnostic Accuracy Study 2302 ED Patients Simplify D-dimer Reference standard Cross-classification

38 The Results Reference Standard PE No PE Positive Negative 21 1,591 PE: pulmonary embolism

39 Measures of Test Accuracy Sensitivity & Specificity 81 % 73 % P & N Predictive Value 13 % 99 % Likelihood Ratios Diagnostic Odds Ratio 11 PE No PE Positive Negative 21 1,591

40 Diagnostic Accuracy Study Series of Patients Index Test Reference standard Cross-classification

41 Christopher Study Christopher JAMA 2006

42 ROC D-Dimer 1 marker: ddimer TPR marker ddimer FPR

43 Question Are sensitivity and specificity stable test properties?

44 D-Dimer Non PE PE

45 D-Dimer D-dimer result Clinical Reference Standard

46 D-Dimer (Non PE only) D-dimer result Age Group

47 D-Dimer (PE only) D-dimer result Age Group

48 outpatients inpatients JE Schrecengost; Clin Chem 2003

49 Jaapa.com

50

51 3. Systematic Reviews 3.1 General

52 Why? More precise estimates More valid estimates (risk of bias) Identify sources of heterogeneity Address questions not directly considered in the primary studies

53 Step 1 Define Objectives Review question: Patients Index Test (comparator test) Outcome Target condition

54 Step 2 Identification Of Studies

55

56

57

58 Failure to Publish ClinicalTrials.gov 418 studies registered (125 on imaging) Excluding 94 registered after completion N=324 (Daniel Korevaar et al.) (months)

59 Publication bias Limited evidence of publication bias in test accuracy research Typical methods fail Best method: estimates against ESS (Deeks) Typically low power

60 Step 3 Selection Of Studies

61 3. Systematic Reviews 3.2 Appraisal

62 Diagnostic Accuracy Study ED Patients Simplify D-dimer Reference standard Cross-classification

63 Healthy controls PE Patients Healthy controls Simplify D-dimer Cross-classification

64 Diagnostic Accuracy Study ED Patients Simplify D-dimer Reference standard Cross-classification

65 The results Reference Standard Target Condition Other Condition Positive Negative

66 Verification - partial ED Patients Simplify D-dimer Multislice CT Cross-classification

67 Partial verification Reference Standard Target Condition Other Condition Positive Negative

68 Partial verification: random Reference Standard Target Condition Other Condition Positive Negative

69 Partial verification: typical Reference Standard Target Condition Other Condition Positive Negative

70 Partial verification: extreme Reference Standard Target Condition Other Condition Positive Negative

71 Multiple Reference Standards ED Patients Simplify D-dimer Multislice CT Cross-classification

72 Multiple Reference Standards ED Patients Simplify D-dimer Multislice CT Follow-up Cross-classification

73 Multiple Reference Standards Reference Standard Target Condition Other Condition Positive Negative

74 Multiple Reference Standards Reference Standard Target Condition Other Condition Positive Negative

75 Multiple Reference Standards ED Patients Simplify D-dimer Multislice CT Follow-up Cross-classification

76 Multiple Reference Standards Multislice CT Follow-up

77 Multiple Reference Standards Reference Standard Target Condition Other Condition Positive Negative

78 Multiple Reference Standards Reference Standard Target Condition Other Condition Positive Negative

79 Multiple Reference Standards Reference Standard Target Condition Other Condition Positive Negative

80 Multiple Reference Standards Typically: Sensitivity Specificity higher higher Direction and magnitude may vary

81

82 Differential Verification Bias

83 Ann Intern Med 2004;140:

84 JAMA 15 SEP 1999; 282; Times Cited:: >750

85 healthy control 3.0 different reference tests 2.2 partial verification 1.0 not blinded 1.3 non-consecutive 0.9 retrospective 1.0 no description test 1.7 no description population 1.4 no description reference relative Diagnostic Odds Ratio

86

87

88

89

90

91 QUADAS-2 Two sets of issues Risk of Bias Concerns of Applicability Four domains Patient Selection Index Tests Reference Standard Flow & Timing

92

93

94 3. Systematic Reviews 3.3 Graphical Summary

95

96 4. Meta-Analysis

97 Meta-Analysis: Overview 0. Simple Methods 1. Improper Methods 2. Discarded Method Summary ROC (SROC) 3. Hierarchical Methods Hierachical ROC (HROC) Bivariate Trivariate 4. Alternative Methods

98 0. Simple Methods Meta-analysis of DOR Mantel-Haenszel Dersimonian-Laird Lehmann model

99 1. Improper Methods Meta-Analysis of Sensitivity Meta-Analysis of Specificity Meta-Analysis of Likelihood Ratio Positive Meta-Analysis of Likelihood Ratio Negative

100

101 2. Summary ROC method regression weights: (1) equal or (2) var lndor

102 2. Summary ROC method

103 2. Summary ROC method

104 3. Hierarchical SROC Within study i

105 3. Hierarchical SROC Between studies dis ij true disease status (-0.5, 0.5) q i positivity threshold (vide S i /2 in sroc) N(Q,s b ) a i lndor in study i N(L,s a ) b scale parameter fixed

106 3. Hierarchical SROC

107

108 3. Bivariate Method Within study i A i diseased B i non-diseased

109 3. Bivariate Method Between studies

110

111 Interchangeable

112 sensitivity Hierarchical SROC model accuracy shape threshold specificity

113 sensitivity Bivariate model sensitivity correlation specificity specificity

114 4. Alternative: MA Predictive Values Within study i A i positives B i negatives

115 6. Alternative: MA Predictive Values Between studies Linear mixed model

116 6. Alternative: MA Predictive Values ED Patients Simplify D-dimer + - Multislice CT Follow-up Cross-classification

117 4. Meta-Analysis Heterogeneity

118 CT for acute appendicitis specificity Terasawa et al 2004

119 Anti-CCP for rheumatoid arthritis specificity Nishimura et al 2007

120 Rheumatoid factor for rheumatoid arthritis Cutoffs from 3 to 100 U/ml specificity Nishimura et al 2007

121 Exploring Heterogeneity: Approaches Subgroup Analysis Meta-regression Sensitivity Analysis

122 Meta-regression: HROC

123

124 Meta-regression: bivariate

125

126 Beware Heterogeneity always expected in sens / spec

127 4. Systematic Reviews Comparisons

128 Diagnostic Accuracy Study Series of Patients Simplify D-dimer Reference standard Cross-classification

129 Direct comparison: paired study 143 ED Patients Simplify D-dimer SimpliRed Reference standard Cross-classification

130 Direct comparison: unpaired 144 ED Patients Simplify D-dimer SimpliRed Reference standard Reference standard Cross-classification Cross-classification

131 Direct comparison: randomized 145 ED Patients Simplify D-dimer R SimpliRed Reference standard Reference standard Cross-classification Cross-classification

132 Indirect comparison 146 ED Patients ED Patients Simplify D-dimer SimpliRed Reference standard Reference standard Cross-classification Cross-classification

133

134

135 5. Interpretation

136 Average sensitivity and specificity Value always lies on the summary ROC curve Interpretation presumes that the studies have used a common threshold Best estimate of the sensitivity and specificity of the test used at the common threshold

137 Confidence regions Depicts the uncertainty in the average sensitivity and specificity value Usually a 95% region is plotted, analogous to a 95% confidence interval 95% certain that the average value lies within the region

138 Prediction regions Depicts heterogeneity or unexplained variability between studies, showing how accuracy varies across the studies 50% or 95% regions 50% chance that the performance of the test in similar settings lies in this region

139 Interpretation Fit for purpose Review question?

140 5. Clinical Scenario - One D-dimer in Suspected Pulmonary Embolism

141

142 Sensitivity ROC space Specificity

143 Sensitivity ROC space Target Region Specificity

144 Sensitivity ROC space Target Region Specificity

145

146 Diagnostic Accuracy Study 2302 ED Patients Simplify D-dimer Reference standard Cross-classification

147 Measures of Test Accuracy Sensitivity & Specificity 81 % 73 % P & N Predictive Value 13 % 99 % Likelihood Ratios Diagnostic Odds Ratio 11 PE No PE Positive Negative 21 1,591

148 Sensitivity ROC space Target Region Specificity

149

150

151 D-dimer: Systematic Review

152 5. Clinical Scenario - Two Urinary bladder tumor markers In Bladder Cancer Recurrence

153 Bladder Cancer

154 Urine Cytology

155 Cystoscopy

156 Urinary bladder tumor markers

157 Sensitivity ROC space Target Region Cytology Specificity

158

159 Diagnostic Accuracy Study 501 patients BTA stat Cystoscopy Cross-classification

160 Measures of Test Accuracy Sensitivity & Specificity 54 % 74 % Pos. & Neg. Pred. Value 43 % 81 % Likelihood Ratios Diagnostic Odds Ratio 3.2 BCR no BCR Positive Negative

161 Sensitivity ROC space Target Region Cytology Specificity

162 The authors

163

164 Systematic Review

165 Sensitivity ROC space Specificity

166 Sensitivity ROC space Specificity

167 Sensitivity ROC space Specificity

168

169

170 6. Discussion

171

172

173

174

175 Research Topics Qualify the Evidence? IPD Meta-Analysis Network Meta-Analysis

176 Single Study Test

177 Single study

178 Sample Size

179 Single Study