Systematic Reviews and meta-analyses of Diagnostic Test Accuracy. Mariska Leeflang

Transcription

1 Systematic Reviews and meta-analyses of Diagnostic Test Accuracy Mariska Leeflang

2

3 This presentation 1. Introduction: accuracy? 2. QUADAS-2 exercise 3. Meta-analysis of diagnostic accuracy 4. Interpretation of the results 5. Discussion

4 Diagnostic Test Accuracy - Outcome measures: sensitivity, specificity - Also predictive values, odds ratios, likelihood ratios - Study design: typically cross-sectional - Index test results and reference standard result are measured at the same time - Diagnosis is about the current status of the patient

5 Ideal accuracy design Consecutive sample of all patients suspected of the disease gold standard Index test Reference standard Patients who in practice will receive the index test and/or its comparator TP FP FN TN

6 Limitations of test accuracy? How well the test identify the target disorder? Does not directly assess effect of test on outcomes Does not directly answer the question of whether using a test does more good than harm Only possible when there is an adequate reference standard

7

8

9 Systematic reviews of diagnostic test accuracy - PICO (in PRISMA-DTA) - Patients: patients who will be tested in practice with the test you are interested in PIT: - Index test: the test or tests you are interested in Patients - Comparison: an alternative test (NOT necessarily the reference standard) Index tests - Outcome: target condition yes/no, as defined by the reference standard Target Condition

10 PICO is not enough Example schistosomiasis in endemic countries

11 Quality assessment: QUADAS-2 Patient Selection Index Test(s) Reference Standard Flow and Timing Risk of Bias Consecutive sample No case control All inclusive Blinding Pre-specified cut-off Likely to correctly classify the disease Blinding Appropriate time interval Partial Verification Differential verification Missings Concerns regarding applicability Representative sample? Index test same as in practice? Appropriate target condition? N/A

12 Patient Selection Consecutive sample No case control All inclusive Representative sample?

13 The reference standard Reference Standard Likely to correctly classify the disease Blinding Appropriate target condition?

14 Example malaria

15 The reference standard Reference Standard Likely to correctly classify the disease Blinding Appropriate target condition? Examples?

16 Next step: meta-analysis

17 Ultimate goal of meta-analysis Robust conclusions with respect to the research question(s) 17

18 Summary of which values? Sensitivity Specificity Positive Predictive Value Negative Predictive Value Positive Likelihood Ratio Negative Likelihood Ratio Diagnostic Odds ratio ROC curves AUROCs Index- Test Disease (Ref. test) Pres. Abs. + TP FP - FN TN 18

19 Focus on sensitivity and specificity Predictive values are directly depending on prevalence Pooling likelihood ratios may lead to misleading / impossible results Pooling odds ratios may be okay, but are difficult to interpret. From the summary sensitivity and specificity, it is still possible to calculate LRs and PVs. 19

20 Sensitivity Sensitivity ECG in Coronary Heart Disease F/T PSA for Prostate cancer specificity specificity

21 Sensitivity Sensitivity GLAL in Gram Negative Sepsis Dipstick for Urinary Tract Infection specificity specificity

22 0 sensitivity Threshold effects Fetal fibronectin for predicting spontaneous birth Decreasing threshold increases sensitivity but decreases specificity Increasing threshold increases specificity but decreases sensitivity specificity

23 Implicit and explicit threshold effects Explicit threshold: different thresholds are used for test positivity Implicit threshold: there is no or only one threshold, but in some cases tests are earlier regarded as positive than in other cases 23

24 Explicit threshold: (ROC) curve Deeks, J. J BMJ 2001;323:

25 Implicit threshold CT scan for cervical cancer. 25

26 How to summarize? Sensitivity and specificity are correlated So you have to take this correlation into account Sensitivity and specificity depend on threshold So you have to take that into account too Diagnostic Odds Ratio is built up from sensitivity and specificity DOR has also specific characteristics So this may be central in the analyses

27 Symmetrical ROC curves and DORs uninformative test Ideally near the upper-left corner. line of symmetry ROC curve is asymmetric when test accuracy varies with threshold Specificity.2 0 DOR = 90 DOR = 6 DOR = 15 DOR = 3 27

28 Meta-analysis: different approaches Pooling separate estimates Not recommended Summary ROC model Traditional approach, relative simple (not recommended) More complex models Bivariate random approach Hierarchical summary ROC approach 28

29 Two types of analysis Meta-regression of sensitivity and specificity Meta-regression of DOR

30 Hierarchical SROC model 1 accuracy shape threshold Specificity 0 30

31 Bivariate model 1.5 sensitivity specificity correlation Specificity 0

32 (advanced) HSROC and Bivariate methods Hierarchical / multi-level allows for both within and between study variability, and within study correlations between diseased and non-diseased groups Logistic correctly models sampling uncertainty in the true positive proportion and the false positive proportion no zero cell adjustments needed Random effects allows for heterogeneity between studies Regression models used to investigate sources of heterogeneity 32

33 Parameterizations HSROC Mean lndor Variance lndor Bivariate Mean logit sens Variance logit sens Mean threshold Variance threshold Mean logit spec Variance logit spec Shape of ROC Correlation between sensitivity and specificity Other than the parameterization, the models are mathematically equivalent. See Harbord et al. Biostatistics 2006;1:

34 Codes in SAS HSROC: logitp = (theta + ut + (alpha + ua)*dis) * exp(-(beta)*dis); Bivariate: logitp = (_sens + usens)*_dis + (_spec + uspec)*_nondis;

35 Fitting hierarchical models Bivariate model is a linear mixed model HSROC model is a non linear mixed model Wider choice of software available for fitting the bivariate model

36 Software options BIVARIATE MODEL Can be fitted using SAS, Stata, WinBUGS, MLWin and R There are some easy to use programs / codes; but these lack flexibility HSROC MODEL Can be fitted using SAS, WinBUGS, MLWin and R As far as I know, no easy to use codes; but some programs provide HSROC parameters derived from bivariate model

37 Meta-analysis: SROC plots HSROC model (curve) Bivariate model (point estimate + confidence region)

38 But In fact all questions are comparative Bossuyt, BMJ, 2006

39 Comparing two tests Takwoingi, Ann Int Med, 2013

40 Comparative sroc plots

41 Some new developments - Copula - Networks - Bayesian analyses - Latent class meta-analysis

42 Heterogeneity Quality Interpreting the results of a Diagnostic Test Accuracy review Clinical pathway Clinical problem Implications for practice, policy and research sens/spec Review question Review proces Bivariate Meta-analysis HSROC What do the results mean? Take home message sroc curves

43 Interpretation Using absolute numbers help to understand the meaning of sensitivity and specificity The clinical pathway is crucial to understand the consequences of testing

44 From findings to conclusions 1. Sensitivity and specificity alone are not very informative 2. Using natural frequencies may help interpretation 3. The consequences of testing are important

45 Some new developments 1. In searching and selection 2. In data-extraction (including quality ass.) 3. Meta-analysis 4. Interpretation

46 Automated review process - Automatic searches, text mining - Automatic data-extraction - Electronic data-extraction forms - Electronic data management systems

47 Electronic data management EPPI Reviewer: Distiller SR: Rayyan: Covidence: Swift review:

48

49 New developments interpretation

50 Evidence Profile (GRADE) Downgrading: Risk of bias Indirectness Inconsistency Imprecision Publication bias Upgrading: Magnitude of effect Dose-response relation Plausible direction of residual confounding Summary of Findings Table (GRADE) showing final quality and importance of outcome

51

52 Questions? Mariska Leeflang

53 Some extra slides. These slides were not shown in the presentation, but may still be informative for some of you. It is about translating and interpreting the results from an example review.

54 Invasive Aspergillosis Aspergillus spp. invade from respiratory tract to other organs and bloodstream. In immunocompromized patients Diagnosis only definite postmortem Antemortem diagnosis is probability-diagnosis: possible / probable / proven AI Prevalence = 4.5 to 8% (in hematology departments) Mortality = 10 % (1990) to 30% (2002) one year s survival (Upton, 2007)

55 Decision Tree Clinical signs Platelia + + HRCT + + Radiology + Antifungal therapy

56 What do these results mean? Studies reported a median prevalence of 11% So in every 200 patients, 22 will have AI Of these, 16 will be positive on GM test Of the 178 pts without AI, 158 will have a negative GM test

57 In every 200 patients, 22 will have AI Of these, 16 will be positive on GM test (and 6 negative) Of the 178 pts without AI, 158 will have a negative GM test (and 20 a positive test) In total there will be 36 positive test results (20 FP) What will happen to these patients? In total there will be 164 negative test results (6 FN) What will happen to these patients?

58 Decision Tree Clinical signs Platelia + + HRCT + + Radiology + Antifungal therapy

59 Clinical pathway All immunocompromized patients OR All immunocompromized patients with fever Clinical signs and symptoms Platelia X-ray X-ray HRCT Antifungal treatment

60 Will it be positive before signs and symptoms become suggestive? If GM test positive, will HRCT be positive as well (or will HRCT become positive at a later stage)? Will GM positive patients profit from treatment? Leeflang, Cochrane Library, 2008