Meta-analysis of diagnostic test accuracy studies with multiple & missing thresholds

Transcription

1 Meta-analysis of diagnostic test accuracy studies with multiple & missing thresholds Richard D. Riley School of Health and Population Sciences, & School of Mathematics, University of Birmingham Collaborators: Apratim Guha, Atanu Biswas, Yemisi Takwoingi Joie Ensor, R. Katie Morris, Jonathan J. Deeks

2 Rationale

3 SENSITIVITY vs. SPECIFICITY pdf Test - Test + Group 1 Diseased Group 0 Healthy Group 0 (Healthy) Test + TP FP Test - FN TN TN TP Group 1 (Diseased) D T Diagnostic variable, D Threshold Sensitivity = number of true positives / total with disease Specificity = number of true negatives / total without disease 3 3

4 SENSITIVITY vs. SPECIFICITY pdf Test - Test + Group 1 Diseased Group 0 Healthy Group 0 (Healthy) Test + TP FP Test - FN TN TN TP Group 1 (Diseased) D T Diagnostic variable, D Threshold Each study may report more than one 2x2 table One for each different threshold value considered 4 4

5 TRACING OUT THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE 1 Lower threshold Sensitivity Higher threshold 0 (1-Specificity) 1 5 5

6 Clinical question Pre-eclampsia is a major cause of maternal and perinatal morbidity and mortality, and occurs in 2-8% of all pregnancies The diagnosis of pre-eclampsia is determined by the presence of elevated blood pressure combined with significant proteinuria Gold-standard for detection of significant proteinuria is the 24 hour urine collection... but this is time consuming and inconvenient Alternative, quicker and yet accurate diagnostic tests required

7 Data Systematic review identified 13 studies evaluating spot protein:creatinine ratio (PCR) for detecting significant proteinuria in pregnancy 23 different thresholds reported, with a minimum of one and a maximum of seven studies per threshold Threshold values ranged from 0.13 to Five studies provided 2x2 table for just one threshold The other eight studies reported results for each of multiple thresholds, up to a maximum of nine thresholds in any study

8 How to do meta-analysis?

9 But summary meta-analysis result is not for a particular threshold and can t identify threshold results from the SROC curve (1) Current standard approaches: Choose one threshold per study Standard approach for meta-analysis Still allows summary ROC curves to be produced

10 (1) Current standard approaches: Separate meta-analysis at each threshold Use ALL thresholds in each study Take each threshold separately Apply standard meta-analysis to each threshold, e.g. bivariate random effects meta-analysis model - models binomial nature of the data - obtain average sensitivity & specificity for each threshold - allows for between-study heterogeneity - accounts for any between-study correlation

11 Separate meta-analysis at each threshold sensitivity specificity.7.6.5

12 However... throwing away information? A separate meta-analysis for each threshold omits any studies that do not report the threshold of interest But such omitted studies may contain related information from other thresholds that are available. In particular... recall the ROC curve Test accuracy results from neighbouring thresholds are similar Results for a missing threshold are bounded between any pair of higher and lower thresholds that are available. Statistically like to utilise such related information (correlation)

13 (2) Sophisticated approaches: (i) Multinomial regression with random-effects Hamza et al. method

14 (2) Sophisticated approaches: (ii) Poisson survival model with random-effects Compared to Hamza: - the multinomial distribution of the counts is replaced by a Poisson distribution of the number of events, - the random normal distribution to describe betweenstudy variation is replaced by a multivariate gamma distribution.

15 (2) Sophisticated approaches: (iii) Many others. Tosteson AN, Begg CB: A general regression methodology for ROC curve estimation. Medical Decision Making 1988, 8: Kester ADM, Buntinx F: Meta-analysis of ROC curves. Medical Decision Making 2000, 20: Poon WY: A latent normal distribution model for analysing ordinal responses with applications in meta-analysis. Statistics in Medicine 2004, 23: Bipat S, Zwinderman AH, Bossuyt PMM, Stoker J: Multivariate Random-Effects Approach: For Meta-Analysis of Cancer Staging Studies. Acad Radiol 2007, 14: Dukic V, Gatsonis C: Meta-analysis of Diagnostic test accuracy assessment studies with varying number of thresholds. Biometrics 2003, 59: Typically not used / difficult to fit with missing data / don t utilise established multivariate methods / don t handle studies with 1 threshold

16 (3) A practical sensitivity analysis: imputation approach Aim: Evaluate impact of missing thresholds but enable established Cochrane methods We have missing data that is bounded Results for a missing threshold are bounded between any pair of higher and lower thresholds that are available Step 1: Impute missing threshold results On the logit-scale, assume: - a linear increase in specificity as the threshold increases - a linear decrease in sensitivity as the threshold increases Convert back to a 2x2 table for the missing threshold Step 2: Apply a standard meta-analysis at each threshold separately

17 Example: Al Ragib study logit-sensitivity logit-specificity.75.5 imputed observed

18 Example: Al Ragib study threshold threshold High Normal ID, t value, x Imputed? TP FP FN TN Total proteinuria proteinuria No Yes Yes Yes Yes No

19 Example: Al Ragib study threshold threshold High Normal ID, t value, x Imputed? TP FP FN TN Total proteinuria proteinuria No Yes Yes Yes Yes No Using this method, an additional 50 threshold results were imputed Vast more information available for meta-analysis at each threshold Key question: are the original conclusions (before imputation) robust?

20 Separate versus imputation approach Sensitivity Specificity no imputation with imputation

21 Imputation reveals weaker clinical conclusion The imputation approach generally Imputation: - reveals LOWER summary test accuracy - increased heterogeneity Original: Suggest PCR test is not suitable (on its own) for diagnosing proteinuria

22 Imputation approach: pros and cons Advantages Utilises more information from neighbouring thresholds Allows standard bivariate meta-analysis at each threshold Incorporates studies with just one threshold Reveals potential over-optimistic in published results (due to selective reporting of thresholds: see Rifai et al.) Disadvantages Single imputation used; multiple imputation now essential Uncertainty of the imputed value needs to be acknowledged (view as a sensitivity analysis, focus on impact on summary estimates)

23 (4) multivariate-normal approach Rather than imputation, consider modelling correlation Extend bivariate-normal model of Reitsma et al. (2002) Step 1: Obtain for each study logit-sensitivity and logit-specificity estimates at each threshold their variance-covariance matrix (variances and correlation) Step 2: Apply a multivariate-normal meta-analysis model Assume logit estimates follow an approx multivariate normal dist Jointly synthesises all threshold simultaneously Correlation borrows strength across thresholds

24 Univariate versus multivariate approach Sensitivity Specificity multivariate univariate

25 Multivariate-normal approach: pros & cons Advantages Can lead to improved precision of estimates Reduces selective reporting of thresholds Disadvantages Laborious to fit Between-study covariance matrix poorly estimated (correlations often +1 and -1) Uses approximate normal sampling distribution Requires continuity correction if there are zero cells

26 Final thing Ordering Sensitivity and specificity should be ordered across thresholds As threshold increases sensitivity decreases, specificity increases Sophisticated methods in part (2) ensure this but perhaps also causes convergence problems given missing data Simpler approaches do not ensure ordering across thresholds e.g. the summary sensitivity estimate increases from 0.82 to 0.94 when the threshold increases from 0.28 to 0.3 This is due to the imbalance in number of studies per threshold Order by meta-regression of meta-analysis results VS threshold

27 Constrain after multi-normal meta-analysis Sensitivity Specificity unconstrained summary results constrained summary results

28 Constrain after multi-normal meta-analysis Sensitivity For constrained results they are ordered; threshold value increases from 0.13 to Specificity unconstrained summary results constrained summary results

29 So main message is. For continuous tests, ideally want to produce meta-analysis results for each threshold & an interpretable SROC curve If possible, move away from ROC curves where particular thresholds are not identifiable Difficult given missing thresholds across studies This is a non trivial problem Therefore usually ignored encourage researchers to consider it

30 Conclusions: dealing with multiple thresholds Sophisticated methods available but may not be possible Propose multivariate-normal or imputation approaches Enable more familiar methods Assess if summary results are robust to the missing thresholds Potentially reduce selective reporting Currently extending to multiple imputation Future work: model parametric distribution of raw data

31 THANK YOU! Selected references Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005; 58: Chu H, Cole SR. Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach. J Clin Epidemiol 2006; 59: ; author reply Hamza TH, Arends LR, van Houwelingen HC, Stijnen T. Multivariate random effects meta-analysis of diagnostic tests with multiple thresholds. BMC Medical Research Methodology 2009; 9: 73. Riley RD et al. Meta-analysis of diagnostic test studies with multiple and missing thresholds: multivariate-normal & imputation approaches (to be submitted) Rifai, N., Altman, D. G., Bossuyt, P. M. (2008). Reporting Bias in Diagnostic and Prognostic Studies: Time for Action. Clin. Chem. 54: