Methods for assessing consistency in Mixed Treatment Comparison Meta-analysis Sofia Dias, NJ Welton, AE Ades RSS 2009, Edinburgh Department of Community Based Medicine
Overview Mixed Treatment Comparison Metaanalysis (MTC) Consistency assumptions How many inconsistencies? Different methods to assess consistency Model fit Direct v indirect evidence Conclusion 2
Example: Thrombolytic Treatment for acute myocardial infarction 9 treatments, 50 trials Two 3-arm trials 16 direct comparisons (out of 36) Data is number of 30-day mortalities out of total number of patients 3
Thrombo: Treatment Network r-pa (5) ASPAC (9) PTCA (7) SK + t-pa (4) SK (1) t-pa (2) Acc t-pa (3) SK (1) TNK (6) Acc t-pa (3) t-pa (2) 8 Acc t-pa (3) 1 SK+t-PA (4) 2 1 r-pa (5) 1 2 TNK (6) 1 PTCA (7) 8 3 11 t-pa (2) UK (8) UK (8) 1 3 2 ASPAC (9) 4 3 2 4
Introduction to MTC 1. Four treatments A, B, C, D 2. Take treatment A as the reference treatment 3. Then the treatment effects (eg. log odds ratios) of B, C, D relative to A are the basic parameters 4. Given them priors: d AB, d AC, d AD ~ N(0,100 2 ) 5
Functional parameters in MTC The remaining contrasts are functional parameters d BC = d AC d AB d BD = d AD d AB d CD = d AD d AC CONSISTENCY assumption Any information on functional parameters tells us indirectly about basic parameters Either FE or RE model satisfying these conditions 6
MTC Random effects model The underlying model is I ik i ik k 1 continuous measure of the treatment effect Priors ~ U (0,10) d N k 1, k i effect of baseline treatment in trial i 2 ~ (0,100 ) 2,...,9 2 ~ N(0,100 ) i 1,...,50 trial-specific treatment effect of k-th treatment relative to baseline treatment ~ N d d, ik 1, t 1, t ik i1 MTC consistency equations 2 7
Consistency We assume that the treatment effect d BC estimated by BC trials, would be the same as the treatment effect estimated by the AC and AB trials if they had included B and C arms Assume that trial arms are missing at random 8
Inconsistencies In a pre-specified population The true treatment effects must be consistent But there may be inconsistencies in the EVIDENCE How to check for this? 9
Main ideas for checking consistency 1. Compare posterior distributions obtained from direct and indirect evidence for each comparison (triangle structures) P-values, plots 2. Model fit/comparison problem Fit models with and without consistency assumptions Compare model fit (residual deviance, DIC) 3. A mixture of both 10
Comparison of direct and indirect estimates Bucher s method for triangles (JCE, 1997) Separate Pairwise meta-analyses on all contrasts Calculate indirect estimate (using consistency equations) Ignores network Can be extended to whole networks* Problems when three-arm trials included or when random effects models used. * Dias et al. Submitted to Statistics in Medicine (2009) 11
Bucher: FE pairwise MA Indirect Direct inconsistency Triangle Node mean sd mean sd mean sd z P-value 1,2,7 2,7-0.66 0.19-0.54 0.42-0.12 0.46 0.26 0.791 1,2,8 2,8-0.37 0.52-0.30 0.35-0.07 0.62 0.11 0.912 1,2,9 2,9 0.00 0.05 0.02 0.04-0.02 0.06 0.31 0.757 1,3,5 3,5 0.10 0.10 0.02 0.07 0.08 0.12 0.65 0.515 1,3,7 3,7-0.51 0.19-0.22 0.12-0.29 0.23 1.30 0.194 1,3,8 3,8-0.21 0.52 0.14 0.36-0.36 0.63 0.56 0.573 1,3,9 3,9 0.15 0.06 1.41 0.42-1.26 0.42 2.99 0.003 12
How many inconsistencies? Inconsistencies are properties of loops Inconsistency degrees of freedom (ICDF) is the maximum number of possible inconsistencies* Informally described as the number of independent 3-way loops in the evidence structure In this example the ICDF is seven Count independent 3-way loops discounting the loop formed only by the 3-arm trial (1,3,4), in blue. * Lu and Ades, JASA 2006 13
Thrombo: Treatment Network r-pa (5) ASPAC (9) PTCA (7) SK + t-pa (4) TNK (6) SK (1) Acc t-pa (3) t-pa (2) UK (8) 14
Inconsistency models* Consistency model 9 treatments, 8 basic parameters Inconsistency model Add 7 parameters Compare model fit 2 1, x, y ~ N(0, Inconsistency ) d d d N 2 1,2, 1,3, 1,4, ~ (0,100 ) d d d 2,3 1,3 1,2 d d d 8,9 1,9 1,8 d d d 2,7 1,7 1,2 d d d 2,8 1,8 1,2 d d d 3,9 1,9 1,3 1,2,7 1,2, 8 1,3,9 * Lu and Ades, JASA 2006 15
Box plot of inconsistency factors ω 2.0 {1,3,9} 1.0 {1,2,7} {1,2,8} {1,2,9} {1,3,5} {1,3,7} {1,3,8} 0.0-1.0 16
Node-splitting* Node splits on each contrast (node, eg. BC) Studies which compare BC directly: inform direct estimate Rest of data with BC removed: inform indirect estimate Draw plots of posterior distributions based on direct and indirect evidence Bayesian p-value to check for consistency Compare model fit Check between-trial heterogeneity parameters residual deviance, DIC statistics *Ohlssen & Spiegelhalter, Workshop (2006) Marshall & Spiegelhalter, Bayesian Analysis (2007) Dias et al. Submitted to Statistics in Medicine (2009) 17
Compare direct and indirect evidence Density 0.0 0.5 1.0 1.5 full MTC MTC excl. direct indirect direct Density 0 2 4 6 8 10 direct full MTC MTC indirect excl. direct -2-1 0 1 2-0.5 0.0 0.5 1.0 log-odds ratio Consistent log-odds ratio Possibly inconsistent? 18
Inconsistent! Density 0 1 2 3 4 5 6 7 MTC excl. direct indirect full MTC direct Node (3,9) is split Direct evidence on (3,9) conflicts with indirect evidence Bayesian p-value < 0.005 0.0 0.5 1.0 1.5 2.0 2.5 log-odds ratio 19
Independent mean effects model Consistency model 9 treatments, 8 basic parameters d d d d N 2 1,2, 1,3, 1,4,, 1,9 ~ (0,100 ) d d d 2,3 1,3 1,2 d d d 8,9 1,9 1,8 Independent mean effects model 15 parameters (one for each pairwise contrast) d d d d d d N 2 1,2, 1,3, 1,4,, 3,7, 3,8, 3,9 ~ (0,100 ) No consistency assumptions 20
Compare residual deviance for each data point dev split basic nodes 0.5 1.0 1.5 2.0 2.5 45 44 44 45 Trials 44,45 compare treatments 3 and 9 0.5 1.0 1.5 2.0 2.5 3.0 dev MTC 21
Compare model fit (RE model) Model Residual deviance* pd DIC Between-trial heterogeneity σ MTC 102.7 61.6 164.3 0.97 Independent mean effects 97.4 67.8 165.2 0.11 Inconsistency (ω-factors) 98.4 64.6 163.0 0.09 inconsistency variance = 0.40 Node (3,9) split 96.9 58.7 155.6 0.07 * Compare to 102 data points 22
Conclusions Explore heterogeneity WITHIN contrasts before looking at inconsistency BETWEEN contrasts Different methods allow comparison of data in different ways Use global goodness of fit to justify choice of consistency model Node-splitting to explore how evidence is being combined But splits each pairwise comparison How many inconsistencies? Independent mean effects model solves multiplicity issue Extra parameters = max number of inconsistencies If inconsistency detected, reconsider entire dataset! 23
References Our website: https://www.bris.ac.uk/cobm/research/mpes Bucher HC, Guyatt GH, Griffith LE, Walter SD. The Results of Direct and Indirect Treatment Comparisons in Meta-Analysis of Randomized Controlled Trials. Journal of Clinical Epidemiology 1997; 50: 683-691. Dias S, Welton NJ, Caldwell DM, Ades AE. Checking consistency in mixed treatment comparison metaanalysis. Submitted to Statistics in Medicine. Lu G, Ades AE. Assessing evidence consistency in mixed treatment comparisons. Journal of the American Statistical Association 2006; 101: 447-459. Marshall EC, Spiegelhalter DJ. Identifying outliers in Bayesian hierarchical models: a simulation-based approach. Bayesian Analysis 2007; 2: 409-444. 24