Agenda. Frequent Scenario WS I: Treatment Comparisons and Network Meta-Analysis: Learning the Basics

Transcription

1 WS I: onducting & Interpreting Indirect Treatment omparisons and Network Meta-nalysis: Learning the asics onducted by members of the ISPOR Indirect Treatment omparisons Task Force Joe appelleri PhD MPH Jeroen P Jansen PhD Neil Hawkins PhD Rachael Fleurence PhD genda ackground oncepts Example Reporting and interpreting results Q & Indirect Treatment omparisons and Network Meta-nalysis: ackground Joe appelleri Frequent Scenario New drugs are often compared with placebo or standard care, but not against each other, in trials aimed to contribute (as expeditiously as possible) toward obtaining approval for drug licensing ommercial incentive to compare the new treatment with an active control may be wanting Such commercial impetus may be lacking because it is not practical -- drugs are still investigational and there are constraints due to small sample sizes and short durations of follow-up cknowledged Value of Indirect Treatment omparisons vailable treatments tend to increase over time For example, a large number of different (old and new) drugs are available for the treatment of depressive and other mental disorders How to makes sense of it all? linical, patients, and health-policy makers often need to decide which treatment is best Make sense of a complex set of treatments Primary: comparison between active treatments Secondary: pure effect of drug (vs. placebo) Select the best choice(s) of treatment 1

2 Value of Indirect Treatment omparisons Regimes for the Treatment of hildren with cute Pyelonephritis When there is no or insufficient evidence from direct comparison trials, it may be possible, in theory, to use results of different trials to estimate the relative efficacy of different treatments Many interventions have not been directly compared in randomized controlled trials (RTs) Over the period 1971-, 54 trials included over 1 different chemotherapy regimens for advanced non-small-cell lung cancer but direct evidence from at least 1 individuals is available on only two regimens (Lancet 6; 368: ) Despite mounting evidence from 18 trials spanning and evaluating 4 regimens, evidence is available only on a few direct comparisons. Source: Ioannidis. Lancet 6; 368: Value of Indirect Treatment omparisons Offers the opportunity to compare what may not have been otherwise compared Different therapeutic classes Different drugs within the same class Different doses of the same drug General Principles: From Theory to Practice Guidelines are useful to support the appropriate use of certain methods like network meta-analysis pplication of indirect comparisons should be based on the scientific merits of the network meta-analysis -- not on their results per se General Principles Well-conducted RTs provide the most valid estimates of the relative efficacy of competing health-care interventions dvocate their development and use whenever possible, with replication Well-conducted observational studies also have merit General Principles Ensure that the use of network metaanalysis adheres to certain standards and assumptions that can be tested or evaluated Doing so will make this method more credible

3 Indirect Treatment omparisons and Network Meta-nalysis: The oncepts Jeroen P Jansen Networks of evidence efinitions De nalysis & synthesis methods losed loops in network: combination of direct and indirect evidence nchored IT (or adusted IT) Mixed Treatment omparison Network meta-analysis Step wise approach (multiple sequential meta-analyses, followed by indirect comparison of pooled pairwise results) Simultaneous evidence synthesis of all pairwise comparisons across the range of interventions Models without covariates Treatment*covariate interactions in meta-regression models to improve similarity / consistency assumptions and explain heterogeneity Frequentist framework ayesian framework (probabilistic interpretation of uncertainty and ranking of interventions) Indirect comparison Indirect comparison Observed trial data omponents of treatment effect Drug Drug Trial 1 Trial Standardize placebo treatment Drug Drug Intervention arm Trial 1 Intervention arm Trial -subtracted effect d d d d d d Drug Drug Drug Drug Source: Jansen et al., 8 RT 1 Is an indirect comparison biased? Tx 61% 4% 4 weeks RR=.5 RT 1 Is an indirect comparison biased? Tx 61% 4% 4 weeks RR=.5 RT Tx 63% RR=.1 3% 4 weeks RT Tx 63% RR=.1 3% 4 weeks RT 3 nchored Indirect comparison Tx 4% RR=.1 RR=.5 /.1=1. % 3 weeks No bias, because differences in study characteristics across trials do not influence the RR RT 3 Tx 5% RR=.5 % 36 weeks ias in anchored indirect comparison of trial 1 with trail 3, because there are differences in study follow-up across trials that do influence the RR 3

4 Mixed Treatment omparisons d d Similarity and onsistency assumptions With an indirect comparison and MT the comparability of the different types of studies must be considered because the value of randomization does not hold across trials. D d dd Heterogeneity variation in the same relative treatment effect between studies Similarity assumption If there is an imbalance in the distribution of effect modifiers (treatment x covariate interactions) between trials then an indirect comparison or MT is biased. Direct estimate D Indirect estimate Evidence inconsistency discrepancy between different types of comparisons omparisons -, - & - are consistent onsistency relates to the loops of evidence Source: Jansen et al., 8 Source: ooper et al., 9, Jansen et al., 8 Indirect comparison: -step approach Simultaneous evidence synthesis & indirect comparison with regression model 1a) Meta-analysis of trials d d ) Obtain indirect estimate of the efficacy of Tx vs given the pooled results of the meta-analyses d d d D d d d dd k-1 basic parameters: d, d, d D (Priors needed for these in ayesian model) Remaining contrasts are functional parameters, e.g : d =d -d, d D =d D -d, d D =d D -d d D,, total of k(k-1)/ contrasts 1b) Meta-analysis of trials Direct estimate Indirect estimate Source: Mapi Jansen Values et al., 8 Source: Jansen et al., 8 1 study Fixed effects meta-analysis η k d k k Noise / random error Observed effect in individual studies Underlying effect η k d k k 4

5 Fixed effects network metaanalysis model Random effects metaanalysis η k b b d bk d b k d b b,,, k,, D, if k b if k 'after' b d Underlying effects with heterogeneity η ~ N d, k k η d k bk b b ~ N Random effects network meta-analysis bk b,,, k,, D, d, ~ Nd d, bk k b if k b if k 'after' b ayes Theorem Posterior Prior Likelihood ayes Theorem Difference between frequentist and ayesian output μ Tx Tx Frequentist output: Point estimate (with 95%I) No significant difference between Tx and Tx Probability that Tx is better Probability that Tx is better 75% Probability that Tx is better than Tx ayesian output: Probability distribution μ Tx Tx dapted from Hollenbeak,ISPOR 4 5

6 Drug Eluting Stents for oronary rtery Disease Indirect Treatment omparisons and Network Meta-nalysis: Example Neil Hawkins omparators are Metal Stent () Paclitaxel Eluting Stent () Sirolimus Eluting Stent () inary Endpoint Target lesion revasculariation 17,69 subects in 37 studies Study 16 Study 31 Study 3 Study 33 Study 34 Study 35 Study 8 Study 1 Study 14 Study 15 Study 18 Study Study Study 3 Study 4 Study 5 Study 6 Study 7 Study 36 Study 4 Study 5 Study 6 Study 7 Study 1 Study 11 Study 1 Study 13 Study 17 Study 19 Study Study 1 Study 8 Study 9 Study 3 Study 3 Study 37 Study 9 Network of trial evidence Odds Ratio of versus (95% I) Direct comparisons form a network of evidence OR =.5 (.18 to.35) Odds Ratio of versus OR =.7 (.61 to.87) Odds Ratio of versus OR =.43 (.31 to.58) Network meta-analysis Treatment effects (log hazard ratios) are estimated for each treatment ( and ) compared to : β, β Values are estimated to best fit the observed trial data: Log transf. Odds Ratio of vs : Ln OR = β β Ln OR = β Ln OR = β Results of pairwise comparisons Odds Ratio (95% ri) compared to stent: Stent: -.66 (.3 to 3.46 ) 4.1 ( 3 3. to ).38 (.9 to.49 ) ( 1. to 1.96 ) May be estimated using ayesian or Maximum Likelihood techniques.5 (.19 to.31 ).65 (.51 to.8 )

7 L bbe Plots Odds Ratio dusted comparison: ucher et al. method OR and OR are available from pairwise meta-analyses response Response Response response Response Odds Ratio Odds Ratio response Response response Response Response Estimate odds ratios of treatment vs. (OR ) Taking logs: lnor OR OR ucher H, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons 38 in meta-analysis of randomized controlled trials. Journal of linical Epidemiology 1997;5(6):683-9 OR llows the Standard Error (SE) to be alculated on the Log scale: SE lnor ln OR ln OR SElnOR SElnOR vs adusted indirect comparison Treatment omparator Odds Ratio (95% I) Log Odds Ratio (SE).5 (.18 to.35) (.17) onsistency vs : Odds Ratio (95% I) Direct:.7 (.61 to.87) Indirect:.58 (.37 to.9) ombined:.65 (.51 to.8 ).43 (.31 to.58) -.84 (.16).7 (.61 to.87).58 (.37 to.9) -.54 (.3).5 (.18 to.35).43 (.31 to.58) 39 Outline Indirect Treatment omparisons and Network Meta-nalysis: Reporting and interpreting results ssessing the internal and external validity of IT/MT Reporting IT/MT results Interpreting IT/MT results Decision-making in the absence of RTs onclusion Rachael Fleurence 7

8 ssessing the internal and external validity of an IT & network metaanalysis Internal Validity ppropriate identification of studies Quality of included studies onfounding bias violating similarity and consistency assumptions External Validity External validity of included trials 43 Report section Introduction Reporting IT/MT results hecklist Item re the rationale for the study and the study obectives stated clearly? Does the methods section include the following? Description of eligibility criteria Information sources Search strategy Study selection process Data extraction (validity assessment of individual studies) What to look for in the paper clear rationale for the review clear obective or research question that pertains to the network meta-analysis Methods Does the methods section include systematic review of the literature in accordance with entre for Reviews and Dissemination (RD) guidelines and PRISM 44 Reporting IT/MT results Reporting IT/MT results Report section Methods hecklist Item re the outcome measures described? Is there a description of methods for analysis/synthesis of evidence? Do the methods described include the following? Description of analyses methods/models Handling of potential bias/inconsistency nalysis framework re sensitivity analyses presented? What to look for in the paper Justification of outcome measures selected for analysis Description and ustification of statistical model(s) used: Multiple meta-analysis of pairwise comparisons vs. network meta-analysis models; Fixed vs. random effects models; Models without or with covariate (interactions) Description of whether analyses were performed in frequentist vs. ayesian approach. Description of how possible bias / inconsistency was evaluated (either qualitative or quantitative, e.g. comparison of direct evidence with the indirect evidence). If meta-regression models are used rational for selection of covariates in models. Description of comparative effectiveness estimates used for presentation of findings (e.g. odds ratio, relative risk, hazard ratio, difference in change from baseline). Description of whether relative effect measures were transformed into expected (absolute) outcomes (e.g. proportion of responders). Rationale for and description of sensitivity analyses Studies included Prior distributions for model parameters in ayesian framework 45 Report section Results hecklist Item Do the results include a summary of the studies included in the network of evidence? Individual study data? Network of studies? Does the study describe an assessment of model fit? re competing models being compared? What to look for in the paper Description of results of study identification and selection process Table/list of studies with information regarding study design and patient characteristics (that might act as effect modifiers). These are important to udge potential similarity/ consistency issues. Figure of network of studies Table with raw data by study and treatment as used for the analysis/model. (Optionally present relative effects of available direct comparisons of each study) Justification of model results 46 Reporting IT/MT results Reporting IT/MT results Report section Results hecklist Item re the results of the evidence synthesis (IT/MT) presented clearly? Sensitivity/scenario analyses What to look for in the paper Table/ Figure with results for the pairwise comparisons as obtained with analyses; Point estimates and measure of uncertainty (95% intervals) In ayesian framework probability to reflect decision uncertainty (i.e. probability which treatment is best if multiple treatments are being compared, and probability that one treatment is better than the comparator) Description of (different) findings with sensitivity/scenario analysis. Report section hecklist Item What to look for in the paper Discussion Does the discussion include the Summary of findings following? Description/summary of main findings Internal I t l validity of analysis Internal validity (individual trials; publication bias; differences across trials that might violate similarity and consistency it assumptions) External validity Implications of results for target audience Discussion regarding generalisability of findings (given patient population within and across trials in network.) Interpretation of results from a biologic and clinical perspective

9 Interpreting IT/MT results Do the findings apply to the decision problem at hand? Which treatment is most effective in the population of interest? What should we do with multiple endpoints? What about safety, patient preferences and costs? Decision Making in the absence of RTs IT/MT are a good option in the absence of head to head pragmatic randomized trials IT/MT remain a form of observational evidence Should we use comparative observational data? Decisions will benefit from explicit and transparent assessments 49 5 onclusion Indirect Treatment omparisons and Network Meta-nalysis: onclusions Network meta-analysis (IT/MT) are an extension of traditional pairwise meta-analysis IT can provide complementary evidence to direct evidence oth direct and indirect evidence contribute to total body of evidence. The results from indirect evidence in combination with the direct evidence in a MT, may strengthen the assessment between treatments directly evaluated 5 onclusion Similarity and consistency assumption need to hold; Risk of residual confounding bias remains Researchers must continue to conduct RTs with head- to-head comparisons of different treatments Indirect Treatment omparisons and Network Meta-nalysis: Questions & nswers pplications of this method are likely to grow particularly in light of ER 53 9

10 Informal indirect comparison: Odds ratio of haloperidol vs. placebo suggested a greater treatment effect than the odds ratio of risperidone vs. placebo, despite the overlapping confidence intervals WHT N GO WRONG? Risperidone and Haloperidol for Schizophrenia (Outcome: Not linically Improved) Risperidone and Haloperidol for Schizophrenia: What Happened? Similarity assumption violated Patient characteristics, dose of drug, and treatment duration were similar between the two sets of placebo- controlled trials dusted indirect comparison: Favors haloperidol over risperidone Direct comparison: Favors risperidone over haloperidol ombination of direct and indirect: Validity doubtful given their inconsistent evidence Source: Song 9 Source: Song. What is indirect comparison. Part of the What is? series February 9. ut clinical improvement was defined differently -controlled trials of risperidone: % or more greater reduction in total score on the Positive and Negative Syndrome Scale or rief Psychiatric Rating Scale -controlled trials of haloperidol: rated by clinicians using the linical Global Impression or other scales Risperidone and Haloperidol for Schizophrenia: What Happened? onsistency assumption violated for combining direct and indirect estimates Inconsistent results between them may give invalid and misleading results Example: risperidone versus haloperidol for schizophrenia (large I-square = 85%) When results are inconsistent, it is important to investigate possible causes of discrepancy hance, invalid indirect comparison, invalid headto-head comparison, clinically meaningfully heterogeneity across trials 1