MIDAS RETOUCH REGARDING DIAGNOSTIC ACCURACY META-ANALYSIS

MIDAS RETOUCH REGARDING DIAGNOSTIC ACCURACY META-ANALYSIS Ben A. Dwamena, MD University of Michigan Radiology and VA Nuclear Medicine, Ann Arbor 2014 Stata Conference, Boston, MA - July 31, 2014 Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 1 / 40

Medical Diagnostic Test Any measurement aiming to identify individuals who could potentially benefit from preventative or therapeutic intervention This includes: 1 Elements of medical history 2 Physical examination 3 Imaging procedures 4 Laboratory investigations 5 Clinical prediction rules Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 2 / 40

Diagnostic Accuracy Studies Figure: Basic Study Design SERIES OF PATIENTS INDEX TEST REFERENCE TEST CROSS-CLASSIFICATION Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 3 / 40

Diagnostic Accuracy Studies Figure: Distributions of test result for diseased and non-diseased populations defined by threshold (DT) Test - Test + Group 0 (Healthy) TN TP Group 1 (Diseased) D T Diagnostic variable, D Threshold Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 4 / 40

Diagnostic Test Performance 1 The performance of a diagnostic test assessed by comparison of index and reference test results on a group of subjects 2 Ideally these should be patients suspected of the target condition that the test is designed to detect. Binary test data often reported as 2 2 matrix Reference Test Reference Test Positive Negative Test Positive True Positive False Positive Test Negative False Negative True Negative Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 5 / 40

Measures of Diagnostic Performance Sensitivity (true positive rate) The proportion of subjects with disease who are correctly identified as such by test Specificity (true negative rate) The proportion of subjects without disease who are correctly identified as such by test Positive predictive value The proportion of test positive subjects who truly have disease Negative predictive value The proportion of test negative subjects who truly do not have disease Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 6 / 40

Measures of Diagnostic Performance Likelihood ratios (LR) The ratio of the probability of a positive (or negative) test result in the patients with disease to the probability of the same test result in the patients without the disease Diagnostic odds ratio The ratio of the odds of a positive test result in patients with disease compared to the odds of the same test result in patients without disease. ROC Curve Plot of all pairs of (1-specificity, sensitivity) as positivity threshold varies Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 7 / 40

Meta-analysis 1 Glass(1976) Meta-analysis refers to the statistical analysis that combines the results of some collection of related studies to arrive at a single conclusion to the question at hand 2 Meta-analysis may be based on aggregate patient data (APD meta-analysis) or individual patient data (IPD meta-analysis) Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 8 / 40

Meta-analytical Methods 1 Meta-analysis of sensitivity and specificity separately by direct pooling or modeling using fixed-effects or random-efffects approaches 2 Meta-analysis of postive and negative likelihood ratios separately using fixed-effects or random-efffects approaches as applied to risk ratios in meta-analysis of therapeutic trials 3 Meta-analysis of diagnostic odds ratios using fixed-effects or random-efffects approaches as applied to meta-analysis of odds ratios in clinical treatment trials 4 Summary ROC Meta-analysis using fixed-effects or random-efffects approaches Summary ROC methods provide the most general approach Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 9 / 40

Summary ROC Meta-analysis of Diagnostic Test Accuracy The most commonly used and easy to implement method 1 Linear regression analysis of the relationship D = a + bs where : D = (logit TPR) - (logit FPR) = ln DOR S = (logit TPR) + (logit FPR) = proxy for the threshold 2 a and b may be estimated by weighted or unweighted least squares or robust regression, back-transformed and plotted in ROC space 3 Differences between tests or subgroups may examined by adding covariates to model Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 10 / 40

Summary ROC Meta-analysis of Diagnostic Test Accuracy 1 Assumes variability in test performance due only to threshold effect and within-study variability 2 Does not provide average estimates of sensitivity and specificity 3 Continuity correction may introduce non-negligible downward bias to the estimated SROC curve 4 Does not account for measurement error in S 5 Ignores potential correlation between D and S 6 Confidence intervals and p-values are likely to be inaccurate Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 11 / 40

Threats to Validity of Diagnostic Meta-analysis Results Valid meta-analyses of test accuracy must account for the following : 1 Threshold Effects 2 Unobserved heterogeneity 3 Methodological quality bias 4 Explainable Heterogeneity 5 Publication and other sample size-related bias Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 12 / 40

Extent of Heterogeneity 1 Assessed statistically using the quantity I 2 described by Higgins and colleagues 2 Defined as percentage of total variation across studies attributable to heterogeneity rather than chance 3 I 2 is alculated as: I 2 = ((Q df )/Q) 100. Q is Cochran s heterogeneity statistic; df equals degrees of freedom. Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 13 / 40

Extent of Heterogeneity 1 I 2 lies between 0% and 100% 2 0% indicates no observed heterogeneity 3 Greater than 50% considered substantial heterogeneity 4 Advantage of I 2 : does not inherently depend on the number of the studies Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 14 / 40

Explaining Heterogeneity: Meta-regression Formal investigation of sources of heterogeneity is performed by meta-regression: a collection of statistical procedures (weighted/unweighted linear, logistic regression) in which the study effect size is regressed on one or several covariates Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 15 / 40

Bivariate Random-Effects Model Recommended hierarchical model for meta-analysis of binary test data (a generalized linear mixed model with binomial family and logit link function (commonly) but may use probit or complementary log-log) 1 Joint modeling of sensitivity(se) and specificity (Sp) Preserves bivariate data structure. Estimates between-study heterogeneity and any existing correlation between these two measures (often due to threshold effects) via random effects. 2 Provides informative clinical results. summary sensitivity, specificity, diagnostic odds ratio and likelihood ratios. summary receiver operating curve (SROC) Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 16 / 40

Meta-analysis of sensitivity and specificity Specification of Bivariate Model TN i µ i Bin(TN i + FP i, Sp i ) logit(sp i ) = X i α + µ i TP i ν i Bin(TP i + FN i, Se i ) ( µi ν i ) N logit(se i ) = Z i β + ν i [( ) 0, 0 ( )] σµ ρσ µ σ ν ρσ µ σ ν σ ν Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 17 / 40

Meta-analysis of sensitivity and specificity Bivariate Model The index i represents study i in the meta-analysis. TN, FP, TP and FN represent the number of true negatives, false positives, true positives, and false negatives. Sp = TN TN+FP and Se = TP TP+FN. X i, Z i represent possibly overlapping vectors of covariates related to Sp and Se The covariance matrix of the random effects µ and ν is parameterized in terms of the between-study variances σ 2 µ and σ 2 ν and the correlation ρ Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 18 / 40

Estimation 1 Maximizing an approximation to the likelihood integrated over the random effects. 2 Different integral approximations are available, with adaptive Gaussian quadrature as method of choice 3 Requires a number of quadrature points to be specified 4 Estimation accuracy increases as the number of points increases, but at the expense of an increased computational time. Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 19 / 40

Stata-native commands 1 Before Stata 10: gllamm gllamm ttruth disgrp1 disgrp2, nocons i(study) nrf(2) eqs(disgrp1 disgrp2) /// f(bin) l(logit) denom(num) adapt ip(m) nip(nip) 2 Stata 10: xtmelogit xtmelogit (ttruth disgrp1 disgrp2, noc)(study: disgrp1 disgrp2, noc cov(unstr)), /// bin(num) laplace var nofet noret nohead refineopts(iterate(4)) 3 Stata 13: meglm and company meglm (ttruth disgrp1 disgrp2, noconstant)(study: disgrp1 disgrp2, noconstant /// cov(exch)), family(binomial _num) notab nohead nolr nogr dnumerical meglm (ttruth disgrp1 disgrp2, noconstant)(study: disgrp1 disgrp2, noconstant /// cov(exch)), family(binomial _num) link(probit) notab nohead nolr nogr Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 20 / 40

User-written dedicated commands 1 Before Stata 10: midas (v. 1.0, August 2007) 2 Stata 10: midas (v.2.0, December 2008) and metandi (March2008) 3 Stata 13: midas v.3.0 Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 21 / 40

metandi An estimation commmand by Roger Harbord, University of Bristol 1 Performs meta-analysis of diagnostic test accuracy studies in which both the index test under study and the reference test (gold standard) are dichotomous. 2 Fits two-level mixed logistic regression model, with independent binomial distributions for the true positives and true negatives within each study, and a bivariate normal model for the logit transforms of sensitivity and specificity between studies. Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 22 / 40

metandi 1 Estimates are displayed for the parameters of both the bivariate model and the Hierarchical Summary Receiver Operating Characteristic (HSROC) model 2 In Stata 8 or 9, makes use of the user-written command gllamm. 3 In Stata 10 metandi uses xtmelogit by default. 4 Limited analytic and graphic options Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 23 / 40

midas v.1.0, v2.0 A comprehensive and medically popular program for diagnostic test accuracy meta-analysis. 1 Implementation of some of the contemporary statistical methods for meta-analysis of binary diagnostic test accuracy. 2 Primary data synthesis is performed within the bivariate mixed-effects logistic regression modeling framework. 3 Likelihood-based estimation is by adaptive gaussian quadrature using gllamm(version 1.0) or xtmelogit (version 2.0) Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 24 / 40

midas v.1.0, v2.0 1 Average sensitivity and specificity (optionally depicted in SROC space with or without confidence and prediction regions), and their derivative likelihood and odds ratios are calculated from the maximum likelihood estimates. 2 facilitates exploratory analysis of heterogeneity, threshold-related variability, methodological quality bias, publication and other precision-related biases. 3 Bayes nomograms, likelihood-ratio matrices, and probability modifying plots may be derived and used to guide patient-based diagnostic decision making. Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 25 / 40

midas v.1.0, v2.0 Relevant sources/documentation 1 midas from http://fmwww.bc.edu/repec/bocode/m 2 help file in pdf form. http://fmwww.bc.edu/repec/bocode/m/midas.pdf 3 Meta-analytical integration of diagnostic accuracy studies in Stata http://repec.org/nasug2007/bd-nasug2007.ppt 4 Meta-analytical integration of diagnostic accuracy studies in Stata http://repec.org/wcsug2007/dwamena-wsug2007.pdf Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 26 / 40

midas v3.0 Updated for Stata 13 1 Estimation command and a wrapper for meglm in Stata 13. 2 Flexibility for specifying covariance structures. 3 Link functions other than logit (e.g. probit, cloglog). 4 Extensive post-estimation options and specification of starting values (especially with sparse data). 5 Univariate (independent) versus bivariate (correlated) modeling of sensitivity and specificity. Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 27 / 40

Data transformation 1 Binomial or Bernoulli gen study = _n gen ttruth1 = tn gen ttruth2 = tp gen num1 = tn+fp gen num2 = tp+fn reshape long num ttruth, i(study) j(dtruth) string tabulate dtruth, generate(disgrp) 2 Bernoulli gen freq=1 bin2bern ttruth, fw(freq) binomial(num) Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 28 / 40

midas 3.0 Flexible Covariance structures 1 Independent: one unique variance parameter per random effect, all covariances 0 2 Exchangeable: equal variances for random effects, and one common pairwise covariance 3 Identity: equal variances for random effects, all covariances 0 4 Unstructured: all variances and covariances to be distinctly estimated Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 29 / 40

midas 3.0 Alternative Link Functions Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 30 / 40

midas 3.0 Univariate (independent) versus bivariate (correlated) modeling Disparate Variances Equal Variances Bivariate Unstructured Exchangeable Univariate Independent Identity Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 31 / 40

midas 3.0 Estimation Syntax midas varlist(min=4 max=4) [ if ] [ in ], [ ID(varname) Link(string) NIP(integer 20) VARiance(string) nonumerical SORTby(varlist min=1) LEVEL(integer 95) noestimates FITstats noheader ] Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 32 / 40

midas 3.0 Replay/Post-estimation Syntax midas [ if ] [ in ], [ Level(cilevel) FITstats noheader noestimates UPVstats(numlist min=2 max=2) FORest SROC(string) FAGAN(numlist min=1 max=3) CONDIProb(string) LRMatrix(string) LINPred FITted MODdiag XSIZE(passthru) YSIZE(passthru) TITLE(passthru) cc(real 0.5) MScale(real 0.90) TEXTScale(real 0.90) CSIZE(real 36) SCHEME(passthru) GRSave(string) XTITLE(passthru) YTITLE(passthru) ] Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 33 / 40

------------------------------------------------------------------------------ parameter Coef. Std. Err. z P> z [95% Conf. Interval] -------------+---------------------------------------------------------------- thetaspe 2.06 0.40 5.15 0.00 1.28 2.85 thetasen 1.23 0.37 3.36 0.00 0.51 1.94 tausqspe 1.16 0.50 2.32 0.02 0.18 2.14 tausqsen 1.16 0.50 2.32 0.02 0.18 2.14 covtausq -0.54 0.46-1.18 0.24-1.44 0.36 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ parameter Coef. Std. Err. z P> z [95% Conf. Interval] -------------+---------------------------------------------------------------- Sens 0.77 0.06 12.07 0.00 0.65 0.90 Spec 0.89 0.04 22.13 0.00 0.81 0.97 DOR 3.29 0.43 7.59 0.00 2.44 4.14 LRP 6.85 2.29 2.99 0.00 2.36 11.34 LRN 0.26 0.07 3.71 0.00 0.12 0.39 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ parameter Coef. Std. Err. z P> z [95% Conf. Interval] -------------+---------------------------------------------------------------- Isqspe 0.65 0.08 7.67 0.00 0.48 0.82 Isqsen 0.82 0.05 16.45 0.00 0.72 0.91 Isqbiv 0.72 0.06 12.41 0.00 0.61 0.83 ------------------------------------------------------------------------------ Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 34 / 40

0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Empirical Bayes predicted versus observed test outcomes midas, ebpred(forest) study 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 Sensitivity Specificity MLE of mean sensitivity and specificity (solid vertical lines) Empirical Bayes (solid lines and markers) Observed data (dashed lines and markers) Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 35 / 40

Summary ROC curve midas, sroc(pred conf mean curve) 1.0 15 11 4 12 2 13 35 10 33 7 8 9 18 37 14 5 26 19 6 16 3 21 29 Sensitivity 0.5 38 30 1 17 20 24 36 22 39 31 25 32 28 23 27 34 Observed Data Summary Operating Point SENS = 0.73 [0.63 0.80] SPEC = 0.96 [0.92 0.97] SROC Curve AUC = 0.95 [0.92 0.96] 95% Confidence Contour 0.0 1.0 0.5 Specificity 95% Prediction Contour 0.0 Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 36 / 40

Funnel plot with superimposed regression line midas, pubbias.05 Deeks Funnel Plot Asymmetry Test pvalue = 0.46.1 32 39 19 18 33 8 35 Study Regression Line 27 36 31 30 13 1/root(ESS).15.2 34 21 25 24 28 22 20 38 9 5 17 26 16 14 4 2 7 10 11 12 15.25 23 3 6 37.3 29 1 1 10 100 1000 Diagnostic Odds Ratio Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 37 / 40

Fagan plot (Bayes Nomogram) midas, fagan(0.25) Pre test Probability (%) 0.1 0.2 0.3 0.5 0.7 1 2 3 5 7 10 20 30 40 50 60 70 80 90 93 95 97 98 99 99.3 99.5 99.7 99.8 99.9 Likelihood Ratio 1000 500 200 100 50 20 10 5 2 1 0.5 0.2 0.1 0.05 0.02 0.01 0.005 0.002 0.001 Prior Prob (%) = 25 LR_Positive = 16 Post_Prob_Pos (%) = 84 LR_Negative = 0.29 Post_Prob_Neg (%) = 9 99.9 99.8 99.7 99.5 99.3 99 98 97 95 93 90 80 70 60 50 40 30 20 10 7 5 3 2 1 0.7 0.5 0.3 0.2 0.1 Post test Probability (%) Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 38 / 40

Residual-based goodness-of-fit, bivariate normality, influence and outlier detection analyses midas, moddiag Deviance Residual 1.00 0.75 0.50 0.25 0.00 (a) Goodness Of Fit 0.00 0.25 0.50 0.75 1.00 Normal Quantile Mahalanobis D squared 1.00 0.75 0.50 0.25 0.00 (b) Bivariate Normality 0.00 0.25 0.50 0.75 1.00 Chi squared Quantile Cook s Distance 2.00 1.50 1.00 0.50 0.00 (c) Influence Analysis 8 9 0 10 20 30 40 study 29 32 Standardized Residual(Diseased) 3.0 2.0 1.0 0.0 1.0 2.0 3.0 9 29 (d) Outlier Detection 8 18 7 15 12 11 42 13 33 35 19 26 5 26 16 19 6337 5130 637 10 14 21 28 21 34 32 29 18 27 31 23 39 35 25 10 14 20 38 24 12 17 22 11 36 15 89 72 4 16 3 32 30 25 1 20 38 24 17 22 36 39 31 34 23 27 28 3.0 2.0 1.0 0.0 1.0 2.0 3.0 Standardized Residual(Healthy) Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 39 / 40

CONCLUDING REMARKS 1 Ado-file will be available shortly after conference on SSC. 2 To include bayesian estimation in future version 3 Thanks for your rapt attention Dwamena B.A. (UM-VA) Midas Update Stata Boston 2014 40 / 40