Live WebEx meeting agenda

Transcription

1 10:00am 10:30am Using OpenMeta[Analyst] to extract quantitative data from published literature Live WebEx meeting agenda August 25, 10:00am-12:00pm ET 10:30am 11:20am Lecture (this will be recorded) 11:20am 12:00pm Using OpenMeta[Analyst] to conduct subgroup meta-analysis and meta-regression

2 Binary Outcomes Outcomes that have two states (e.g., dead or alive, success or failure) The most common type of outcome reported in clinical trials 2x2 tables commonly used to report binary outcomes

3 A Sample 2x2 Table Binary outcomes data to be extracted from studies Treatment Placebo Vascular deaths (event) 791 1,029 Survival (no event) Total 7,801 8,592 7,566 8,595

4 Treatment Effect Metrics That Can Be Calculated From a 2x2 Table Events No Events Group Rates Treatment a b TR = a a + b Control c d CR = c c + d Treatment Effects Metrics Risk Difference Odds Ratio Risk Ratio TR / (1 - TR) RD = TR - CR OR = RR = CR / (1 - CR) OR = (a / b) / (c / d) TR CR

5 Some Characteristics and Uses of the Risk Difference Value ranges from -1 to +1 Magnitude of effect is directly interpretable Has the same meaning for the complementary outcome (e.g., 5% more people dying is 5% fewer living) Across studies in many settings, tends to be more heterogeneous than relative measures Inverse is the number needed to treat (NNT) and may be clinically useful If heterogeneity is present, a single NNT derived from the overall risk difference could be misleading

6 Some Characteristics and Uses of the Odds Ratio Value ranges from 1/oo to + Has desirable statistical properties; better normality approximation in log scale than risk ratio Symmetrical meaning for complementary outcome (the odds ratio of dying is equal to the opposite [inverse] of the odds ratio of living) Ratio of two odds is not intuitive to interpret Often used to approximate risk ratio (but gives inflated values at high event rates)

7 Some Characteristics and Uses of the Risk Ratio Value ranges from 0 to Like its derivative, relative risk reduction, is easy to understand and is preferred by clinicians Example: a risk ratio of 0.75 is a 25% relative reduction of the risk Requires a baseline rate for proper interpretation Example: an identical risk ratio for a study with a low event rate and another study with higher event rate may have very different clinical and public health implications Asymmetric meaning for the complementary outcome Example: the risk ratio of dying is not the same as the inverse of the risk ratio of living

8 Let s extract data! 8 Sometimes you need to look for data in multiple places

9 Introduction to Meta-analysis: Focusing on Heterogeneity LARGELY ADAPTED FROM THE AGENCY FOR HEALTHCARE RESEARCH AND QUALITY (AHRQ) TRAINING MODULES FOR SYSTEMATIC REVIEWS METHODS GUIDE PRESENTED BY MEI CHUNG, PHD, MPH ASSISTANT PROFESSOR DEPARTMENT OF PUBLIC HEALTH AND COMMUNITY MEDICINE, SCHOOL OF MEDICINE, TUFTS UNIVERSITY

10 Systematic Review Process Overview Ask FORMULATE STUDY QUESTION ESTABLISH PROTOCOL 10 Identify LITERATURE SEARCH / RETRIEVAL Acquire CRITICAL APPRAISAL PAPER SELECTION per PROTOCOL Appraise Synthesize DATA EXTRACTION and QUALITY ASSESSMENT ANALYSIS and INTERPRETATION WEIGHTED AVERAGE REGRESSION SENSITIVITY ANALYSIS Qualitative & quantitative synthesis

11 Introduction to Meta-analysis 11 Many readily available (and user friendly) software to allow people without much statistical background to perform meta-analysis But.. Most developed to combined randomized trial results Doing a meta-analysis is (thus) very easy but doing one well is hard Many pre-analysis steps (e.g., extracting quantitative data from the literature) require some biostatistics knowledge Many missing data (in the literature) may be required to be imputed/calculated Interpretations of (study-level) meta-analysis results are different from analysis of people-level data

12 Typical Caveats of Meta-analyses 12 Many assumptions are made in meta-analyses, so care is needed in the conduct and interpretation. Most meta-analyses are retrospective exercises, suffering from all the problems of being an observational design. Researchers cannot make up missing information or fix poorly collected, analyzed, or reported data.

13 Reasons To Conduct Meta-Analyses 13 Improve the power to detect a small difference if the individual studies are small Improve the precision of the effect measure Compare the efficacy of alternative interventions and assess consistency of effects across study and patient characteristics Gain insights into statistical heterogeneity Help to understand controversy arising from conflicting studies or generate new hypotheses to explain these conflicts Force rigorous assessment of the data

14 Principles of Combining Data for Basic Meta-Analyses 14 For each analysis, one study should contribute only one treatment effect. The effect estimate may be for a single outcome or a composite. The intervention/exposure, comparator & outcome being combined should be the same or similar, based on clinical plausibility across studies. Know the research question. The question drives study selection, data synthesis, and interpretation of the results.

15 Heterogeneity (Diversity) 15 Is it reasonable? Are the characteristics and effects of studies sufficiently similar to estimate an average effect? Types of heterogeneity: Clinical diversity: Are the studies of similar treatments, populations, settings, design, et cetera, such that an average effect would be clinically meaningful? Methodological diversity: Are the studies of similar design and conduct such that an average effect would be clinically meaningful? Statistical heterogeneity: Is the observed variability of effects greater than that expected by chance alone?

16 Example of a heterogeneity in a set of studies 16 Methodological diversity Clinical diversity Table source: Kehle et al. Interventions to Improve Veterans Access to Care: A Systeamtic Review of the Literature. J Gen Intern Med 26(Suppl 2)589-96

17 Statistical Heterogeneity 17 Statistical heterogeneity exists when the results of individual studies are not consistent among themselves. Clinical diversity Methodological diversity Biases Chance Statistical heterogeneity

18 Statistical Heterogeneity Testing 18 Is the observed variability of effects greater than that expected by chance alone? Like any statistical testing. The statistical power is depending on the sample size (= number of studies in meta-analysis) Two statistical measures are commonly used to assess statistical heterogeneity: Cochran s Q-statistics (usually shown as a p-value for heterogeneity testing) I 2 index: ranging from 0 to 100%

19 From: Cannabinoids for Medical Use: A Systematic Review and Meta-analysis JAMA. 2015;313(24): doi: /jama SYNTHESIS METHODS: Clinical heterogeneity was assessed by grouping studies by indication, cannabinoid, and outcome. If there were 2 or more trials within a single grouping, data were pooled using random-effects meta-analysis. Figure Legend: Improvement in Pain Odds indicate 30% or greater improvement in pain with cannabinoid compared with placebo, stratified according to cannabinoid. The square data markers indicate odds ratios (ORs) from primary studies, with sizes reflecting the statistical weight of the study using random-effects meta-analysis. The horizontal lines indicate 95% CIs. The blue diamond data markers represent the subtotal and overall OR and 95% CI. The vertical dashed line shows the summary effect estimate, the dotted shows the line of no effect (OR = 1). Copyright 2015 American Medical Association. All rights reserved.

20 Assumptions of the Fixed Effect and Random Effects Models 20 Fixed effect model: True effect is the same in all studies and observed variations in the effect sizes across studies are due to individual study s sampling variations Therefore, more weight toward larger studies Random effects model Different studies (although similar enough but not totally identical) have different effect sizes for two reasons: Sampling variability (same assumption as the fixed effect model) Random variation (because the effect sizes themselves are sampled from a population of effect sizes

21 Weights of the Fixed Effect and Random Effects Models Fixed Effect Weight Random Effects Weight 21 w i = 1 v i w * i = v i 1 + v * where: v i = within study variance v * = between study variance Difference between fixed and random effects models is the between study variance (i.e., heterogeneity) When (statistical) heterogeneity is small/none, the two models should produce the same results

22 Random-effects (DL) model vs. fixed effect (MH) model 22

23 23 Reference: Cornell JE et al. Random-Effects Meta-analysis of Inconsistent Effects: A Time for Change. Ann Intern Med. 2014; 160:

24 Dealing With Heterogeneity HETEROGENEOUS TREATMENT EFFECTS IGNORE ESTIMATE (insensitive) INCORPORATE EXPLAIN FIXED EFFECTS MODEL DO NOT COMBINE WHEN HETEROGENEITY IS PRESENT RANDOM EFFECTS MODEL SUBGROUP ANALYSES META- REGRESSION (control rate, covariates)

25 Promises of Subgroup Analyses 25 Subgroup analyses can help: identify modifiers of the treatment effect, recognize biologically interesting phenomena, or formulate hypotheses.

26 Illustrative Example for Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality 26 Miller ER 3rd, et al. Ann Intern Med. 2005;142: Reprinted with permission from the American College of Physicians.

27 Hazards of Subgroup Analyses: Multiple Testing (I) Subgroup analyses are a form of multiple testing. When uncontrolled, multiple testing can yield spurious findings. Most meta-analyses do not perform statistical adjustments for multiple testing. 27

28 Can You Avoid the Hazards of Subgroup Analyses? (I) 28 When analyzing data, it is important to distinguish subgroup analyses that are specified a priori (without knowing what the data are) versus those that are specified post hoc (after the researcher has been exposed to the data). This distinction is very clear when analyzing a prospective study. In most meta-analyses, the distinction is not as clear.

29 Can You Avoid the Hazards of Subgroup Analyses? (II) Most meta-analyses use data that are published (and potentially known). When researchers adequately prepare before embarking on a meta-analysis, they inevitably become acquainted with the data they will analyze. This makes it difficult for researchers to claim that they specified subgroups without knowing anything about their data. 29

30 Can You Avoid the Hazards of Subgroup Analyses? (III) Meta-analysts should do their best to define subgroups that make methodological and biological sense. Treat the results of subgroup analyses with a healthy dose of skepticism, especially when adjustments for multiple testing are not performed. 30

31 Beyond Subgroup Analyses: Meta-Regression Meta-regression can help examine how the treatment effect changes across the levels of a variable. All subgroup analyses can be formulated in a metaregression framework, but meta-regression goes well beyond subgroup analyses. 31

32 Illustrative Example for Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality 32 Miller ER 3rd, et al. Ann Intern Med. 2005;142: Reprinted with permission from the American College of Physicians.

33 Corresponding Univariate Meta-Regression: A Meta-analysis of Vitamin E Doses and Mortality 33 Miller ER 3rd, et al. Ann Intern Med. 2005;142: Reprinted with permission from the American College of Physicians.

34 Two Types of Covariates in Meta-Regressions 34 Study level Examples: presence/absence of blinding, intervention dose (in experimental studies) Participant level Examples: mean age, proportion of diabetic patients, mean intake of vitamin E (in observational studies)

35 Spurious Associations in Meta-Regressions or Subgroup Analyses (I) Aggregate-data meta-regressions on participantlevel covariates can mislead, because they are susceptible to ecological fallacy. The observed relationship between the study-level treatment effect and the mean of a patient-level factor does not necessarily reflect the corresponding true relationship at the individual patient level. 35

36 Spurious Associations in Meta-Regressions or Subgroup Analyses (II) Therefore, associations of treatment effect and participant-level covariates should be interpreted with caution. Such associations can be biologically plausible and informative, or can mislead. Unfortunately, there is no universal way to distinguish true from fallacious findings. 36

37 37 Key Messages READ ALL META-ANALYSES WITH SUSPICIONS IN ADDITION TO THE TYPICAL LIMITATIONS, ASSUME MOST META-ANALYSES CONTAIN SOME DATA COLLECTION AND/OR MANIPULATION ERRORS ESPECIALLY IF DATA ARE NOT REPORTED AND THE ANALYSES ARE COMPLEX

38 Key Messages (I) 38 Basic meta-analyses can be easily carried out with readily available statistical software. Relative measures are more likely to be homogeneous across studies and are generally preferred. The random effects model is the appropriate statistical model in most instances. The decision to conduct a meta-analysis should be based on: a well-formulated question, appreciation of the heterogeneity of the data, and understanding of how the results will be used.

39 Key Messages (II) 39 Subgroup analyses, meta-regressions, and controlrate meta-regressions are tools to explore betweenstudy heterogeneity. Use them to understand data. They are mostly hypothesis-forming tools. Especially for meta-regressions on patient-level covariates, ecological fallacy may mislead. Beware when interpreting their results.