Hypothetical Scenario

Using Indirect and Multiple Treatment Comparisons Methods To Estimate Comparative Effectiveness of Alternative Treatments Workshop September 27, 2010 Beth Devine, PharmD, MBA, PhD Rafael Alfonso C., MD, MSc UW Pharmaceutical Outcomes Research & Policy Program Hypothetical Scenario Your institution is deciding on treatments for Rheumatoid Arthritis and YOU have been selected to review the comparative effectiveness of biologic therapies in patients who have inadequate response to oral DMARDs, especially Methotrexate. Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. Evidence Based Medicine Framework Question PICO Literature search Validity/Strength of Infer rence Time Spent in Critical Appraisa al Appraisal Levels of evidence Integrate evidence 1

Literature Search We performed a literature search to identify the best available evidence of biological therapies for RA in patients with MTX IR Only RCT comparing biologic therapies to placebo or MTX, no head to head comparison across all the available treatments Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. Network of studies available Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. Meta analysis Quantitatively combined results of comparable studies of the same agent to obtain overall estimate of effect. Indirect Treatment Comparison Statistical comparison of two or more agents that have not been directly compared to each other, but that have one comparator in common, thus creating a network Mixed Treatment Comparison Extension of ITC where both direct and indirect evidence is included Validity of evidence synthesis relies on methods that: appreciate within trial randomization and Limit bias due to lack of randomization across trials Jansen. Value in Health 2008;11(5):956-64; Lumley. Stat Med 2002;21:2313-24; Lu & Ades. Stat Med 2004;23:3105-24; Sutton & Ades. Pharmacoeconomics 2008;26(9);753-67 2

ITC/MTC Assumptions Song. BMJ 2009;338:b1174 Statistical Methods Fixed effects models We believe all trials estimate the same treatment effect Mantel Haenszel method Peto method Random effects models We believe there is variability between studies; that there is a true mean effect and the effect from each study is distributed around that mean DerSimonian Laird method Statistical Methods Frequentist Probability is the limit of relative frequency (Pr(A B) Assumes experiment can be repeated under identical conditions Only applies to events that have not yet occurred Once data collected, probability no longer applies! After data collected can only describe uncertainty Applies to repetitititititive phenomena Flipping a coin Hollenbeak.ISPOR. May 15, 2010 Thompson. The Stata Journal 2006;6(4):530-49 Bayesian Based on subjective probability Probability is my degree of belief in a statement Probability y describes uncertainty Uncertainty has a probability distribution Obtain a distribution of your model parameters Parameters are random and data are fixed Calculate 95% credible intervals Probability the event lies within the interval P(B A) = [P(A B)p(B)] / P(A) 3

Statistical Methods Monte Carlo simulation estimates random sequence of chains, where next chain relies only on its immediate predecessor Markov chain Markov chain Monte Carlo simulation (MCMC) set up a Markov chain whose distribution is the posterior distribution Chain must run to convergence before estimating posterior probabilities burn ins A special type of algorithm cycles through each model parameter one at a time is called Gibbs sampling WinBUGS = Bayesian inference Using Gibbs Sampling, for Windows http://www.mrc-bsu.cam.ac.uk/bugs/ or Lunn, Thomas, Best, Speigelhalter. Stat Comput 2000;10:325-37 Thompson. The Stata Journal 2006;6(4):530-49 Methods: Model Development Two random effects logistic regression models (6 and 12 months), using the ACR 50 as the primary outcome. Randomization was preserved MTX and/or placebo, as common comparators to biologics. Mean disease duration and Mean baseline HAQ DI score were included as meta regression covariates, to account for heterogeneity between trials These covariates have prognostic value in determining the effect of RA treatment Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. Methods: Model Compilation and Running the Models Our statistical model is represented by the following equations: For the control arms: logit (p i 0) = α i + β i m i 0 For the treatment arms: logit (pij) = αi + βimij + θi + γ1x1i + γ2x2i and j=1, Ji We ran three chains and 20,000 iterations after a burn in of 10,000 iterations. We assessed the accuracy of the posterior assessments by calculating the Monte Carlo error for each parameter. For nomenclature see notes below Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. 4

Results: 6 months In the 6 month model, when compared to placebo, all biologic DMARDS and MTX were significantly more efficacious and ranked in the following order: Certolizumab (median log odds ratio=2.6), tocilizumab (1.7), rituximab (1.6), infliximab (1.6), etanercept (1.4), adulimumab (1.4), golimumab (1.4), abatacept (1.2), anakinra (1.0), MTX (0.8). There were 45 pairwise comparisons. Certolizumab was significantly more efficacious than MTX; none others were significant. Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. 6-month Model - Pairwise comparisons between biologic DMARDS and MTX (99.9% CrI) 6 month model - Odds ratios by treatment and disease duration compared to comparator (95% CrI) 5

Results: 12 months The rank order in the 12 month analysis was: Certolizumab (2.0), rituximab (2.0), adalimumab (1.4), infliximab (1.4), etanercept (0.9), abatacept (0.6), and MTX (0.8). There were 21 pairwisecomparisons; nonewere significant. The results of the model parameterized by therapeutic class revealed that each class was more efficacious than placebo. In pairwise comparisons, each class was more efficacious than MTX, but none more efficacious than another. Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. Limitations We assumed treatment effects are exchangeable between studies for both biologic and MTX treatment We assumed treatment effect similar between study arms withineachstudy didnotdifferentiate differentiate amongdoses We evaluated only one outcome ACR50(not safety) Meta regression controlled for some, but not all, between study heterogeneity Our results are dependent on the studies included We did not address publication bias Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. Conclusions Using emerging ITC methods enabled us to compare the efficacy of biologic DMARDS for treatment of RA in the absence of head to head trials. Our methods enabled us to rank order these treatments. Further analysis by drug and by therapeutic class suggests that biologic DMARDS are similarly efficacious. Based on the manuscript to be published in Pharmacotherpy by: Devine BE, Alfonso Cristancho R, Sullivan S. 6

Now Let s Practice using WinBUGS! Advantage of Bayesian analysis in ITC/ MTC is that it allows calculation of the probability of which treatment is best http://www.mrc-bsu.cam.ac.uk/bugs/ or Lunn, Thomas, Best, Speigelhalter. Stat Comput 2000;10:325-37 Outcome Measures How is your outcome of interest measured? Binary (e.g. dead or alive) Continuous (e.g blood pressure) Categorical/ordinal (e.g. severity scale) Bi Binary outcomes most common We will consider here Continuous Similar approach to binary Ordinal More complex and more rare 7

Binary Outcome Measures Binary outcome data from a comparative study can be expressed in a 2 x 2 table RCT Failure/Dead New Treatment A B Control C d Success/Alive Three common outcome measures: Odds ratios, risk ratios, risk differences Binary Outcome Measures Odds ratios OR = ad/bc Var ln(or) = 1/a + 1/b + 1/c + 1/d Work on log (ln) scale due to better statistical properties Relative Risk RR (a/(a+b))/ (c/(c+d)) Var ln(or) = 1/a 1/(a+b) + 1/c 1/(c+d) Work on log (ln) scale due to better statistical properties Binary Outcome Measures Risk Difference RD = (a/(a+b)) (c/(c+d)) Var = [p 1 / (1 p 1 ]/n 1 + [p 2 / (1 p 2 ]/n 2 ] Where: p 1 = a/(a+b) p 2 = c/(c+d) n 1 = a+b and n 2 = c+d 8

Fixed Effects Model Statistical homogeneity Formally assume: Y i = Normal(d,V i ) Random error We estimate the common true effect, d Point estimate True effect Generic Fixed Effect Y i ~ Normal(d,V i ) where i= 1.N studies Y i is the observed effect in study i with Variance V i All studies assumed to be measuring the same underlying effect size, d For a Bayesian analysis, a prior distribution must be specified for d Choice of Prior for d Often, amount of information in studies is large enough to render any prior of little importance therefore choice not critical Often specified as vague or flat Often specified as vague or flat E.g. If meta analysis is on ln(or) scale, could specify d~normal (0, 10 5 ) This states a priori we would be 95% certain that true value of d is between [0±1.96( 10 5 )] On OR scale: 10 269 to 10 269 very flat! 9

Fixed Effect with Prior Y i ~ Normal(d,V i ) where i= 1.N studies d ~ Normal(0, 10 5 ) Models are specified in WinBUGS using formulas similar to this algebra Note: Normal distributions are specified by mean and precision where precision = 1/variance Estimate model paramter using MCMC, rather than inverse weighting of variance Example: Meta analysis, RCTs of effect of aspirin preventing death after acute MIs Study Aspirin Group Placebo Group Deaths Total Deaths Total MRC 1 49 615 67 624 CDP 44 758 64 771 MRC 2 102 832 126 850 GASP 32 317 38 309 PARIS 85 810 52 406 AMIS 246 2267 219 2257 ISIS 2 1570 8587 1720 8600 Fleiss 1993 Example: Calculation: Log(OR) & Variance For MCR 1 Survive Die Aspirin 566 49 Placebo 557 67 OR=(566*67)/ 67)/ (557*49) = 1.389 Log(ln)OR = 0.3289 Variance lnor = 1/566 + 1/49 + 1/557 + 1/67 = 0.0389 Note this is OR for Survival If 2x2 table contains any zeros, common to add 0.5 to those cells before calculations 10

Example:Aspirin Data to be Combined Study OR LnOR (Y i ) Var(lnOR) (V i ) Weight (1/V i ) MRC 1 1.39 0.33 0.04 25.71 CDP 1.47 0.39 0.04 24.29 MRC 2 1.25 0.22 0.02 48.77 GASP 1.25 0.22 0.06 15.44 PARIS 1.25 0.23 0.02 28.41 AMIS 0.88 0.12 0.01 103.92 ISIS 2 1.12 0.11 0.002 664.26 Note: ISIS-2 with small variance and large weight (1/0.002) Launch WinBUGS Click on WinBUGS14.exe Click File Open Load aspirin FE.odc Components of WinBUGS.odc file Model { < Likelihood> <Prior distributions> } #Data <List or column format> #Starting Values <List or mixture of list and column format> 11

Steps for Running a Model in WinBUGS 1. Make model active. Doodles: If in own window, click title bar. If in compound document, double click the doodle (should have hairy border). Text: Simply highlight the word model at the beginning of your model. 2. Bring up Model Specification Tool (menu: Model > Specification) 3. Click check model Should see model is syntactically correct in lower left corner of window. 4. Highlight first row of data containing variable labels (if in rectangular format) 5. Click load data Should see data is loaded in lower left corner of window. 6. If using multiple chains, enter number in num of chains box. Otherwise, proceed. 7. Click compile Should see model is compiled in lower left corner of window. 8. Highlight line containing initial values: list( ) 9. Click load inits If using multiple chains, you will need to repeat steps 8 9 for each chain. Should see model is initialized. 10. Bring up Sample Monitor Tool (menu: Inference > Samples) Enter name of each node you wish to monitor and click set 11. Bring up Update Tool (menu: Model > Update) 12. Enter a number of samples to take and click update. Should see model is updating. Load and Check Model 12

Load and Check Data Compile Compile Model Model Load Initials 13

Pooled OR: median 1.12 (1.05 to 1.19) 14

Random Effects Model Model Within studies Y i ~Normal( i,vi) Across studies ~Normal(d 2 I Normal(d, ) d=solid line =dotted lines 2 = variability between studies (heterogeneity) Y 5 Trial-specific effects=dotted lines True Mean Effect = solid line 5 Generic Random Effect Y i ~ Normal(,V i ) where i= 1.N studies i ~ Normal(d, 2 ) As for fixed effect, Y i is observed effect in study i with variance V i Now study specific effects, I are allowed to be different from each other and are assumed to be sampled from a Normal distribution with mean d and variance 2 For a Bayesian analysis, a prior distribution is required for 2 as well as for d Choice of Prior for 2 This is a little trickier than for d Variances cannot be negative so Normal distribution is not a good choice Examples in WinBUGS Manual use Uniform distribution. E.g. ~ Uniform (0,10) of 10 is massive, because we are working with ORs; even of 1 or 2 is large Specification of vague priors on variance components is complex and is an active area of research 15

Generic Random Effects Model Load aspirin RE.odc 16

Results of Aspirin RE model Pooled OR: median 1.149 (0.976 1.434) OR now contains 1 Heterogeneity (log scale)=0.02 (95% CrI: 0 0.4) Compare with FE model: Pooled OR: median 1.12 (1.05 to 1.19) Bayesian CrI wider than classical CI 2 is random variable and uncertainty is included in pooled result Random Effects Binomial Model Load aspirin RE bin.odc If ORs are being pooled it is possible to specify a model that does not require assumption of normality of study effect sizes Useful when events are rare Avoids need to add continuity correction factors when cells with 0 events exist Considered more exact Based on logistic regression Random Effects Binomial Model r Ai ~Bin(p Ai, n Ai ) r Bi ~Bin(p Bi, n Bi ) logit(p Ai )=µ i logit(p Bi )=µ i + i i ~N(d, 2 ) where i=1...n studies n and n = total number of patientsin ith study n Ai and n Bi = total number of patients in ith study r Ai and r Bi = total number of events in each group p Ai and p Bi = underlying probabilities of event in each group µ i = estimated log odds of event in group A (requires prior) I = log (OR) between groups d=true mean effect; =heterogeneity between studies Logit(p)=log((p)/(1 p)) unconstrained scale used logistic regression 17

Compare our Three Odds Ratios and CrIs Fixed effects Normal Distribution OR=1.12 (95% CrI: 1.05, 1.19) Random Effects Normal Distribution OR=1.15 (95% CrI: 0.97, 1.44) Random Effects Binary Distribution OR=1.14 (95% CrI: 0.98, 1.44) MCMC Basics Now that we ve run a few models consider sensitivity analyses Sensitivity to prior distributions esp. important for distributions of variance/precision parameters Sensitivityto initialvalues Sensitivity to initial values Multiple chains using very different starting values & comparing using Brooks Gelman Rubin Statistics Length of burn in & ample: examine history/trace plots Auto correlation high auto correlation suggests samples are not independent & poor/slow mixing of chains: Run model longer or reparameterize (center variables) Convergence diagnostics & CODA & BOA 18

Interpreting Random Effects A single parameter cannot adequately summarize heterogeneous effects Therefore estimation and reporting of 2 is important This tells us how much variability there is between estimates from the population of studies In some instances studies contain both beneficial and harmful effects very important! Looking to the Future (The Future is Here!) Data Sources Evidence Synthesis RCT1 RCT2 Obs 1 Routine Care Meta- Analysis General Synthesis Bayes Theorem Combination Model Clinical Adverse Inputs (w/ Effects Effects undertainty) Decision Model Tx A Fib Warfarin NO Warfarin Stroke No Strk Stroke No Strk Bleed NoBld Bleed NoBld Bleed NoBld Bleed NoBld Utility Utility Utility Utility Utility Utility Utility Utility Cost Cost Cost Cost Cost Cost Cost Costs Questions? 19