5/10/2010. ISPOR Good Research Practices for Comparative Effectiveness Research: Indirect Treatment Comparisons Task Force

Transcription

1 ISPOR Good Research Practices for Comparative Effectiveness Research: Indirect Treatment Comparisons Task Force Presentation of Two Draft Reports 17 May 2010 ITC Task Force: Leadership Team Lieven Annemans PhD MSc Ghent University and Brussels University Task Force Chair Annabel Barrett BSc Eli Lilly and Co Ltd, Surrey, UK Cornelis Boersma PhD, MSc University of Groningen, the Netherlands Joseph C Cappelleri PhD, MPH Pfizer Inc, US Jon Clouse RPh MS United HealthCare, US Beth Devine PharmD, MBA, PhD University of Washington, US Mireya Diaz PhD Henry Ford Health Systems, US Rachael Fleurence, PhD United BioSource Corp, US Neal Hawkins PhD Oxford Outcomes Ltd, UK David C Hoaglin PhD United BioSource Corp, US Robbin Itzler PhD Merck Research Laboratories, US ITC Task Force: Leadership Team (cont) Jeroen Jansen PhD Mapi Values, US Kay M Larholt ScD Karen Lee MA Canadian Agency for Drugs and Technologies (CADTH), Canada David Scott MA Oxford Outcomes Ltd, UK David Thompson PhD I3 Innovus, US ITC Task Force: Objectives Develop recommendations for good research practices for indirect treatment comparisons (ITC) & mixed treatment comparisons (MTC) Prepare two manuscripts Guidance for consumers of ITC research (i.e., those who use ITC data for decision making) Guidance for producers of ITC research (i.e., those who conduct analyses to generate ITC data) Guidelines for Consumers of ITC Broad Objectives Provide background on ITC/MTC research Why are ITC/MTC studies conducted? What are evidence networks? Which methods apply to which circumstances? Provide guidance on how to be an informed consumer of ITC/MTC research What main features to look for in an ITC/MTC report? What are strengths & limitations of ITC/MTC methods? Guidelines for Producers of ITC Broad Objectives Provide an overview of evidence gathering strategies for ITC/MTC How to search the literature for evidence networks Provide an overview of current statistical methods for ITC/MTC Common vs. different baseline treatments Fixed- vs. random-effects models Frequentist vs. Bayesian approaches Meta-regression Provide guidance on good practices for conduct of ITC/MTC research 1

2 Authors (lead + alphabetical listing) Paper 1 Guidelines for Review of ITC Research Rachael Fleurence PhD (Lead Author) Annabel Barrett BSc Cornelis Boersma PhD, MSc Joseph Cappelleri PhD, MPH Beth Devine PharmD, MBA, PhD Mireya Diaz PhD Neal Hawkins PhD Robbin Itzler PhD Jeroen Jansen PhD Karen Lee MA Introduction Evidence networks Analyzing the evidence Underlying assumptions Statistical models Analytic frameworks Critically reviewing an ITC study report Internal validity External validity Reporting Interpreting ITC results Decision-making in the absence of RCTs Conclusion Appendix: Terminology section Introduction Given the evidence available, network meta-analysis (ITC/MTC) are often needed Head-to-head (direct) comparisons are unavailable or do not provide all the evidence available Network meta-analysis (ITC/MTC) are an extension of standard pairwise metaanalysis by including multiple pairwise comparisons across a range of interventions The goal of the Task Force is to provide guidance Networks of evidence Anchored ITC (or adjusted ITC) Network meta-analysis Closed loops in network: combination of direct and indirect evidence Mixed Treatment Comparison Validity of evidence synthesis relies on methods that appreciate within trial randomization; and limit bias due to lack of randomization across trials 2

3 Analyzing the evidence Heterogeneity variation in the same relative treatment effect between studies Similarity assumption If there is an imbalance in the distribution of effect modifiers (treatment x covariate interactions) between trials comparing different interventions then an ITC/MTC is biased. Evidence inconsistency discrepancy between different types of comparisons Comparisons A-B, B-C & A-C are consistent Consistency relates to the loops of evidence Analyzing the evidence Similarity assumption Consistency assumption A B C Direct comparison Indirect comparison Direct comparison A B C Indirect comparison Analyzing the evidence Statistical model choice Justification of models given evidence base endpoints, appreciation of randomization, heterogeneity, similarity/consistency Model critique / comparison of different models Analysis frameworks (Frequentist / Bayesian) The field is still in its infancy and methods are developing rapidly. Critically reviewing an ITC study - validity Internal validity Evidence base considered Bias in single trials Poor quality of individual trials Publication bias Are the outcomes, and statistical models appropriate given network of evidence? Bias due to similarity/consistency issues? External validity Source data RCTs designed for registration purposes with very homogenous population Critically reviewing an ITC study- Reporting Critically reviewing an ITC study- Reporting Introduction Clear statement of objective and rationale for comparisons and comparators Methods Literature review methods (PRISMA) Evidence network development Choice of models and exploration of model fit Results Source data from individual trials Model fit Base-case scenario (evidence base) Alternative scenarios Sensitivity analyses Discussion Critical assessment of the results with respect to internal validity (e.g. similarity/consistency) of ITC/MTC study Clinical and policy relevance Checklist in manuscript summarizes key issues regarding validity and reporting of ITC/MTC studies to assist decision-makers in evaluating the quality and relevance of a ITC/MTC (Note: checklist is not a validated instrument) 3

4 Interpreting an ITC study Are differences between treatments most likely true, or can they be explained away due to bias in the ITC/MTC To what extent are ITC/MTC results in line with expectations based on results of individual studies (and other evidence.) Do the results make sense from a biological / mode of action / clinical perspective? What do the results mean and what are implications for decision-making? Decision-making in the absence of ITC/MTC of RCTs Can we use evidence other than ITC/MTCs of RCTs for decision making? An ITC/MTC of RCTs is observational evidence, albeit with a greater internal validity than non-randomized comparative studies. Alternatively: What level of observational evidence can be considered to have sufficient internal validity to inform decision-making? Decision makers face trade off Risk of making the wrong decision by relying on lower quality evidence than (ITCs/MTCs of) RCTs, Postponing the decision by waiting for (ITC/MTCs of) RCTs, possibly forgoing health benefits Paper 2 Guidelines for Conduct of ITC Research Q & A Authors (lead + alphabetical listing) David C Hoaglin PhD (Lead Author) Annabel Barrett BSc Cornelis Boersma PhD, MSc Joseph C Cappelleri PhD, MPH Mireya Diaz PhD Neal Hawkins PhD Robbin Itzler PhD Jeroen Jansen PhD Kay M Larholt ScD David Scott MA David Thompson PhD Key Topics Evidence networks Statistical methods Assumptions Models Frequentist vs Bayesian approaches Checklist for good research practices Implementation Future research 4

5 Partial Overview Evidence Networks (1) All relevant treatments & trials comparing them directly Basic search standards from PRISMA Statement Assemble indirect comparators; shorter paths have more influence Some comparisons may have both indirect and direct evidence Evidence Networks (2) Totality of evidence? Can treatments not viewed as comparators contribute? Are higher-order indirect comparisons worth the effort? Start with comparators of interest. Iterate Capture relevant meta-analyses Statistical Methods, Assumptions (1) What are we trying to estimate? True effect of C relative to B (via A) d BC = μ C μ B = (μ C μ A ) (μ B μ A ) = d AC d AB Actual network arises because links not present are missing at random from network of all pairwise comparisons Statistical Methods, Assumptions (2) Variation in relative treatment effects Common effect: fixed-effects Distribution of effects: random-effects Similarity: distribution of treatment-effect modifiers must be similar across pairwise comparisons Consistency of indirect evidence with direct evidence (MTC) Checking may require statistical and clinical judgment Statistical Methods, Models (1) Measures of relative effect include Odds ratio ( responders ) Rate ratio ( events per person-year ) Risk ratio for having an event Risk ratio for not having an event Mean difference Hazard ratio ( survival ) Risk difference (often more difficult) Can present results in another scale 5

6 Statistical Methods, Models (2) Models for measures of relative effect Meta-analysis of AB trials Fixed-effects Random-effects Fixed-effects network meta-analysis (ITC or MTC) Random-effects network meta-analysis (ITC or MTC) Statistical Methods, Models (3) Meta-regression models may account for heterogeneity, minimize bias from violations of similarity/consistency, and improve prediction Approaches to capture impact of differences in study-level covariates Add constant treatment*covariate interactions Add treatment-specific treatment*covariate interactions Exchangeable treatment*covariate interactions Frequentist Statistical Methods Basis for traditional procedures Weighted means with confidence intervals (based on assumed normal distribution) Mantel-Haenszel estimate (a fixed-effects procedure) Maximum likelihood Bayesian Statistical Methods Likelihood relates data to parameters Prior distributions reflect initial uncertainty about the parameters ( integrate over) Vague (non-informative) Informative (e.g., from previous studies) Subject to sensitivity analysis Posterior distributions reflect uncertainty about parameters and other quantities, given the data (and the priors) Summarize posterior (e.g., mean, CrI) Can estimate ranking of treatments Good Research Practices Checklist Follow applicable parts of PRISMA Key components for ITC and MTC studies Objectives Search strategies Data collection Statistical analysis plan Data analysis Reporting Details in paper (also consumers checklist) Implementation Illustrative example of ITC and MTC Three treatments Several direct comparisons of each pair Fixed-effects meta-analysis (of OR) Random-effects meta-analysis Bayesian direct meta-analysis Bayesian ITC Bayesian MTC Details in appendix (including data and WinBUGS code) Software 6

7 Future Research (1) Methods for verifying similarity and consistency Empirical support for frequentist methods Methods to improve similarity of trials and empirical assessment of their benefit Individual patient data in ITC and MTC How to handle heterogeneity with small numbers of studies per intervention? Future Research (2) Multiple outcomes Place of ITC and MTC in hierarchy of study designs for strength of evidence Structure and properties of networks Size of network; evidence space versus decision space Q & A 7