Empirical assessment of univariate and bivariate meta-analyses for comparing the accuracy of diagnostic tests

Transcription

1 Empirical assessment of univariate and bivariate meta-analyses for comparing the accuracy of diagnostic tests Yemisi Takwoingi, Richard Riley and Jon Deeks

2 Outline Rationale Methods Findings Summary

3 Motivating example Gurusamy et al. Endoscopic retrograde cholangiopancreatography versus intraoperative cholangiography for diagnosis of common bile duct stones. Cochrane Database of Syst Rev 2013.

4 Motivating example Sensitivity Specificity Endoscopic retrograde cholangiopancreatography Intraoperative cholangiography

5 Bivariate model sensitivity sensitivity correlation specificity Models both logit(sensitivity) and logit(specificity) and the correlation between them logit(sensitivity) and logit(specificity) are specified as random study effects Combines two random effects meta-analysis of sensitivity and specificity in a single model specificity

6 Bivariate model specification Level 1 (allows for within-study variability) tp ~ binomial( tp + i i tn ~ binomial( tn + i i fn, Se i i fp, Sp ) i i ) Number diseased Number not diseased log it ( Se ) µ, i Level 2 (allows for between-study variability) µ µ B A, i, i = A i and B i µ A ~ N µ B log it ( Sp ) = µ, i, with 2 σ A = σ AB σ σ AB 2 B

7 Bivariate model for comparative meta-analysis Assuming a test type covariate Z that may affect both sensitivity and specificity, the model can be extended as = ,, with, ~ B AB AB A i B B i A A i B A i Z v Z v N σ σ σ σ µ µ µ µ Effect of test type on variance parameters can also be investigated

8

9 Applies to meta-analysis of a single test results indicate that simpler hierarchical models are valid in situations with few studies or sparse data. For synthesis of sensitivity and specificity, univariate random effects logistic regression models are appropriate when a bivariate model cannot be fitted

10 Univariate model for comparative meta-analysis Assuming a test type covariate Z that may affect both sensitivity and specificity, the model can be expressed as = ,, with, ~ B AB AB A i B B i A A i B A i Z v Z v N σ σ σ σ µ µ µ µ 0 0 Bivariate model simplifies to 2 univariate random effects logistic regression models for sensitivity and specificity

11 Estimates from bivariate and univariate models comparing ERCP and IOC Test Logit sensitivity (SE) Logit specificity (SE) Variance of random effects for logit sensitivity (SE) Variance of random effects for logit specificity (SE) Correlation of the logits (SE) Sensitivity (95% CI) Specificity (95% CI) Bivariate model ERCP 1.55 (0.30) 5.35 (2.25) 0.22 (0.27) 2.95 (6.34) 0.41 (1.05) 82.5 ( ) 99.5 ( ) IOC 7.06 (4.53) 4.15 (0.52) 16.7 (26.9) 0.25 (0.54) (0.98) 99.9 ( ) 98.5 ( ) Univariate model ERCP 1.56 (0.30) 5.32 (2.19) 0.22 (0.26) 2.83 (5.87) ( ) 99.5 ( ) IOC 6.12 (3.28) 4.19 (0.57) 9.74 (12.7) 0.34 (0.67) ( ) 98.5 ( )

12 Test Estimates from bivariate and univariate models comparing ERCP and IOC Variance of random effects for logit sensitivity (SE) Variance of random effects for logit specificity (SE) Correlation of the logits (SE) Sensitivity (95% CI) Specificity (95% CI) Bivariate model ERCP 0.22 (0.27) 2.95 (6.34) 0.41 (1.05) 82.5 ( ) 99.5 ( ) IOC 16.7 (26.9) 0.25 (0.54) (0.98) 99.9 ( ) 98.5 ( ) Univariate model ERCP 0.22 (0.26) 2.83 (5.87) ( ) 99.5 ( ) IOC 9.74 (12.7) 0.34 (0.67) ( ) 98.5 ( )

13 Univariate or bivariate comparative meta-analyses: does it matter?

14 Aim To investigate validity of assumption of equal variances when comparing test accuracy in bivariate metaregression models. Are there important differences between findings from bivariate meta-regression models that assume common variances across tests and those which allow variances to differ by test? To examine the impact of using univariate random effects logistic regression models. Are findings from univariate meta-regression models similar to those from bivariate meta-regression models?

15 Data source Reviews identified in DARE from

16 Eligibility criteria Reviews identified in DARE from Included if 1. Diagnostic accuracy of 2 tests compared 2. Meta-analyses were performed 3. Possible to derive 2x2 tables for included studies

17 Data analysis A. Preliminary meta-analysis of each test in a test comparison performed Bivariate model fitted to assess model stability and estimation of correlation parameter B. Comparative meta-analyses of each test comparison Bivariate model with and without equal variances Univariate model with and without equal variances

18 Criteria for assessment of performance 1. Difference in magnitude of relative test performance expressed as ratio of relative sensitivities and ratio of relative specificities 2. Difference in precision of measures of relative test performance expressed as a ratio of standard errors 3. Change in statistical significance at the 5% level: do confidence intervals include 1? 4. Change in direction of effect (qualitative change): is the ranking of a pair of tests in terms of superior sensitivity or specificity consistent between the two models?

19 Findings cohort of reviews 57 reviews and test comparisons included Total number of studies in a test comparison ranged between 6 and 103 At least one study had a zero cell in 53 test comparisons

20 Preliminary bivariate meta-analyses: Is correlation reliably estimated? Index test - correlation parameter Comparator test - correlation parameter Rarely similar for a pair of tests Correlation = +1 for 12/114 (11%) Correlation = 1 for 22/114 (19%) Estimated within boundary of parameter space (> 1 < +1) for remaining 80 (70%)

21 Comparative bivariate meta-analyses: Is it important for variances to differ by test? Ratio of point estimates Differences in magnitude 11 (22%) test comparisons had more than a 10% difference in relative sensitivity and/or relative specificity. Across 49 test comparisons, median (IQR) ratios of relative sensitivities and relative specificities were 1.00 (0.99 to 1.01) and 1.00 (0.98 to 1.01). 0.6 Relative sensitivities Relative specificities

22 Comparative bivariate meta-analyses: Is it important for variances to differ by test? Differences in magnitude Differences in precision Ratio of point estimates Ratio of standard errors Relative sensitivities Relative specificities Relative sensitivities Relative specificities

23 Comparative bivariate meta-analyses: Is it important for variances to differ by test? Standard errors were on average higher for models with unequal variances compared to models with equal variances. Median (IQR) = 1.37 (1.09 to 1.77) for ratios of standard errors of log relative sensitivities and 1.39 (1.15 to 2.05) for those of log relative specificities. Ratio of standard errors Differences in precision Relative sensitivities Relative specificities

24 Test comparison ID Comparative bivariate meta-analyses: Is it important for variances to differ by test? A Relative sensitivity with 95% confidence interval Relative specificity with 95% confidence interval For 21 (43%) Equal variances of the Unequal 49 test variances comparisons, likelihood Equal variances ratio tests Unequal variances Test comparison ID B indicated statistically significant differences in model fit.

25 Test comparison ID Comparative bivariate meta-analyses: Is it important for variances to differ by test? A 15 (31%) test comparisons had a change in the statistical significance of relative sensitivity or relative specificity while 4 (8%) had a change in both measures. Relative sensitivity with 95% confidence interval Relative specificity with 95% confidence interval Qualitative Equal variances differences Unequal were variances observed for 11 Equal (22%) variances test comparisons. Unequal variances Test comparison ID B

26 Univariate vs bivariate comparative metaanalyses: Are findings similar? Differences in magnitude Differences between both models were negligible. Ratio of point estimates Across 48 test comparisons, median (IQR) ratios of relative sensitivities and relative specificities were 1.00 (1.00 to 1.01) and 1.00 (1.00 to 1.00) Relative sensitivities Relative specificities

27 Univariate vs bivariate comparative metaanalyses: Are findings similar? Differences in magnitude Differences in precision Ratio of point estimates Ratio of standard errors Relative sensitivities Relative specificities Relative sensitivities Relative specificities

28 Univariate vs bivariate comparative metaanalyses: Are findings similar? Standard errors tended to be higher for estimates from bivariate models relative to those from univariate models. Median (IQR) ratios of standard errors for log relative sensitivities and log relative specificities were 1.00 (1.00 to 1.05) and 1.00 (1.00 to 1.01). Ratio of standard errors Differences in precision Relative sensitivities Relative specificities

29 Univariate vs bivariate comparative metaanalyses: Are findings similar? A B Test comparison ID Test comparison ID Relative sensitivity with 95% confidence interval Relative specificity with 95% confidence interval Univariate model Bivariate model Univariate model Bivariate model

30 Univariate vs bivariate comparative metaanalyses: Are findings similar? 8 test comparisons where likelihood ratio tests indicated statistically significant difference in model fit between univariate and bivariate models.

31 Univariate vs bivariate comparative metaanalyses: Are findings similar? Change in statistical significance

32 Univariate vs bivariate comparative metaanalyses: Are findings similar? Qualitative change

33 Summary and conclusions Assumption of equal variances in comparative metaanalyses is not always justified Validity of assumptions should be investigated if data permits Minimal impact of using a bivariate structure for comparative meta-analyses Univariate meta-regression is an alternative when bivariate meta-regression is not feasible Provides a solution to non-convergence when data are sparse