Disclosures. An Introduction to Meta Analysis. Biography, SL Norris 2/20/2012

Similar documents
The Meta on Meta-Analysis. Presented by Endia J. Lindo, Ph.D. University of North Texas

How to do a meta-analysis. Orestis Efthimiou Dpt. Of Hygiene and Epidemiology, School of Medicine University of Ioannina, Greece

Performance of the Trim and Fill Method in Adjusting for the Publication Bias in Meta-Analysis of Continuous Data

18/11/2013. An Introduction to Meta-analysis. In this session: What is meta-analysis? Some Background Clinical Trials. What questions are addressed?

Meta-Analysis. Zifei Liu. Biological and Agricultural Engineering

Choice of axis, tests for funnel plot asymmetry, and methods to adjust for publication bias

Live WebEx meeting agenda

Comparison of Different Methods of Detecting Publication Bias

Traumatic brain injury

Results. NeuRA Mindfulness and acceptance therapies August 2018

Meta Analysis. David R Urbach MD MSc Outcomes Research Course December 4, 2014

Problem solving therapy

Explaining heterogeneity

C2 Training: August 2010

Introduction to Meta-Analysis

Results. NeuRA Treatments for internalised stigma December 2017

Results. NeuRA Hypnosis June 2016

Results. NeuRA Forensic settings April 2016

Research Synthesis and meta-analysis: themes. Graham A. Colditz, MD, DrPH Method Tuuli, MD, MPH

Results. NeuRA Worldwide incidence April 2016

Meta-analysis: Methodology

Animal-assisted therapy

Introduction to meta-analysis

Results. NeuRA Family relationships May 2017

Evaluating the results of a Systematic Review/Meta- Analysis

Distraction techniques

NeuRA Sleep disturbance April 2016

Cochrane Breast Cancer Group

The QUOROM Statement: revised recommendations for improving the quality of reports of systematic reviews

Critical Appraisal of a Meta-Analysis: Rosiglitazone and CV Death. Debra Moy Faculty of Pharmacy University of Toronto

Fixed Effect Combining

Contour enhanced funnel plots for meta-analysis

Systematic Reviews. Simon Gates 8 March 2007

Cochrane Pregnancy and Childbirth Group Methodological Guidelines

Results. NeuRA Motor dysfunction April 2016

Transcranial Direct-Current Stimulation

Results. NeuRA Treatments for dual diagnosis August 2016

Controlled Trials. Spyros Kitsiou, PhD

In many healthcare situations, it is common to find

Learning objectives. Examining the reliability of published research findings

Robert M. Jacobson, M.D. Department of Pediatric and Adolescent Medicine Mayo Clinic, Rochester, Minnesota

The moderating impact of temporal separation on the association between intention and physical activity: a meta-analysis

Learning from Systematic Review and Meta analysis

Models for potentially biased evidence in meta-analysis using empirically based priors

Introduction to systematic reviews/metaanalysis

Meta-analyses: analyses:

Systematic reviews and meta-analyses of observational studies (MOOSE): Checklist.

Current Issues with Meta-analysis in Medical Research

Assessing publication bias in genetic association studies: evidence from a recent meta-analysis

NeuRA Obsessive-compulsive disorders October 2017

Critical Thinking A tour through the science of neuroscience

Method. NeuRA Biofeedback May 2016

Meta-analysis: Advanced methods using the STATA software

NeuRA Decision making April 2016

False statistically significant findings in cumulative metaanalyses and the ability of Trial Sequential Analysis (TSA) to identify them.

Publication Bias in Meta-analysis. Hannah R. Rothstein Department of Management, Zicklin School of Business, Baruch College, New York, USA

PROTOCOL. Francesco Brigo, Luigi Giuseppe Bongiovanni

Systematic Review & Course outline. Lecture (20%) Class discussion & tutorial (30%)

Results. NeuRA Herbal medicines August 2016

Types of Data. Systematic Reviews: Data Synthesis Professor Jodie Dodd 4/12/2014. Acknowledgements: Emily Bain Australasian Cochrane Centre

Meta-Analyses: Considerations for Probiotics & Prebiotics Studies

Capture-recapture method for assessing publication bias

Meta-Analysis De-Mystified: A Step-by-Step Workshop

Coping with Publication and Reporting Biases in Research Reviews

Meta-Analysis and Subgroups

Supplementary Online Content

Systematic Reviews and Meta- Analysis in Kidney Transplantation

Meta-analysis: Basic concepts and analysis

Applying Evidence-Based Practice with Meta-Analysis

Treatment effect estimates adjusted for small-study effects via a limit meta-analysis

UNCORRECTED PROOFS. Software for Publication Bias. Michael Borenstein Biostat, Inc., USA CHAPTER 11

Evidence-Based Medicine and Publication Bias Desmond Thompson Merck & Co.

Essential Skills for Evidence-based Practice Understanding and Using Systematic Reviews

Cochrane Bone, Joint & Muscle Trauma Group How To Write A Protocol

How to Conduct a Meta-Analysis

Principles of meta-analysis

Systematic reviews and meta-analyses

Workshop: Cochrane Rehabilitation 05th May Trusted evidence. Informed decisions. Better health.

Determinants of quality: Factors that lower or increase the quality of evidence

School of Dentistry. What is a systematic review?

Meta-analysis of safety thoughts from CIOMS X

Systematic Reviews and Meta-analysis

Standards for the reporting of new Cochrane Intervention Reviews

Introduction to diagnostic accuracy meta-analysis. Yemisi Takwoingi October 2015

Meta-analysis of diagnostic research. Karen R Steingart, MD, MPH Chennai, 15 December Overview

Misleading funnel plot for detection of bias in meta-analysis

Chapter 9: Analysing data and undertaking meta-analyses

Trials and Tribulations of Systematic Reviews and Meta-Analyses

A protocol for a systematic review on the impact of unpublished studies and studies published in the gray literature in meta-analyses

Results. NeuRA Essential fatty acids August 2016

What is indirect comparison?

How to interpret results of metaanalysis

NeuRA Schizophrenia diagnosis May 2017

Critical appraisal: Systematic Review & Meta-analysis

Fixed-Effect Versus Random-Effects Models

What is meta-analysis?

GUIDELINE COMPARATORS & COMPARISONS:

Results. NeuRA Maternal infections April 2016

Table S1- PRISMA 2009 Checklist

Meta-Analysis David Wilson, Ph.D. Upcoming Seminar: October 20-21, 2017, Philadelphia, Pennsylvania

Transcription:

An Introduction to Meta Analysis Susan L. Norris, MD, MPH, MS Associate Professor Oregon Health & Science University Portland, OR norriss@ohsu.edu Biography, SL Norris MD, MS, University of Alberta MPH, University of Washington Board Certified: general surgery (Canada), family medicine (US) Clinical practice: Group Health Cooperative, Washington state, 1990 1999 CDC: directed systematic review group with focus on diabetes, 1999 2004 Oregon Health & Science University (current) clinical practice guideline development systematic review methodology (nonrandomized studies, reporting bias), comparative effectiveness reviews effects of physician industry relationships on practice guidelines and systematic reviews 2 Disclosures Financial: none Intellectual Member, GRADE Working Group Methods work: nonrandomized studies, sources of bias in reviews Impact of physician industry relationships on primary research and evidence synthesis Professional Investigator, Evidence based Practice Center (AHRQ) Funders: CDC, NIH, AHRQ, American College of Chest Physicians, American Urological Association 3 1

Learning Objectives 1. To understand the difference between a metaanalysis and a systematic review 2. To understand when a meta analysis might be useful and when it may be misleading 3. To define heterogeneity and understand how it affects a meta analysis 4. To understand the difference between a fixed effects and a random effects model 5. To be able to interpret a forest plot 4 Outline 1. Introduction 2. Terminology 3. Basic considerations 4. Modelsfor combining data 5. Assessing and exploring heterogeneity 6. Publication bias 7. Reading and interpreting meta analyses 8. Resources 5 Part 1. Introduction 2

Example of a Meta-analysis 7 Meta analysis: Enoxaparin versus Unfractionated Heparin Gould et al. Ann Intern Med 1999 May 18;130(10):800 9 8 Working Example of a Systematic Review and Meta analysis 9 3

Proliferation of Meta Analyses 1976 6 papers on meta analysis 1991 approximately 600 meta analyses existed in the social and behavioral sciences 1993 678 meta analyses were found in the medical literature alone 2005 over 2000 reviews in the Cochrane library 2011 over 4500 reviews in the Cochrane library 10 Why the Proliferation of Meta analyses? We need more scholarly effort concentrated on the problem of finding knowledge that lies untapped in completed research studies We need methods for the orderly summarization of studies so knowledge can be extracted from the myriad individual researches This endeavor deserves higher priority now than adding a new experiment or survey to the pile. (Glass, 1976) 11 Part 2. Terminology 12 4

Typology Integrative publications Systematic review Narrative review Nonsystematic review Review of comparative effectiveness Qualitative synthesis Quantitative synthesis Meta analysis Practice guidelines Economic evaluation Decision analysis 13 Typology Integrative publications Systematic review Narrative review Nonsystematic review Review of comparative effectiveness Qualitative synthesis Quantitative synthesis Meta analysisanalysis Practice guidelines Economic evaluation Decision analysis 14 Systematic Review A concise summary of the best available evidence that addresses a sharply defined clinical question (Mulrow 1998) Qualitative synthesis = narrative summary Quantitative synthesis = meta analysis 15 5

Meta Analysis IOM 2011, Finding What Works in Health Care the statistical combination of results from multiple individual studies MA is a broad term that encompasses a wide variety of methodological approaches whose goal is to quantitatively synthesize and summarize data across a set of studies Other terms: quantitative synthesis, pooling, pooled analysis 16 Steps in a Systematic Review Develop the review question Develop inclusion/exclusion criteria Search for literature Quality assess individual studies Data abstraction and analysis Synthesis of findings Grading the strength of evidence 17 Synthesis in Systematic Reviews Involves 4 questions: 1. What is the direction of effect? 2. What is the effect size? 3. Is the effect consistent across studies? 4. What is the strength of evidence for the effect? Meta analysis provides statistical methods for addressing Q 1 3 Narrative synthesis uses subjective methods to examine Q 1 4 18 6

Part 3. Basic Considerations 19 Why Perform a Meta analysis? To increase power chance of detec ng a real effect if it exists To improve precision To answer questions not posed by individual studies Eg, examine effects across different populations, interventions, settings, outcomes To explore heterogeneity 20 Does it make sense to perform a meta analysis? given that the studies differ in various ways and the analysis amounts to combining apples and oranges? combining applies and oranges makes sense if your goal is to produce a fruit salad Robert Rosenthal, from 21 Borenstein et al. An Introduction Meta-Analysis, 2009 7

Does it make sense to perform a meta analysis? 1. Does is make sense to combine studies? 2. What studies should be included? Consider these questions in the context of the goal of the meta analysis: Estimate a common effect across similar studies Assess dispersion (heterogeneity) Both 22 Does it make sense to combine data across studies? Conceptual considerations Clinical diversity Methodological diversity Statistical considerations Statistical heterogeneity 23 When Not to Combine Studies in a Meta analysis Studies are too diverse with respect to PICO to provide a meaningful answer apples and oranges Included studies are biased In the presence of significant publication and/or reporting biases: summary measure is not valid 24 8

Steps in a Meta analysis 1. Calculate a summary statistic for each study 2. Calculate a summary (pooled) intervention effect: A weighted average of the intervention effects estimated in individual studies 3. Estimate heterogeneity 4. Assess potential for bias 25 Choice of Summary Statistic at the Individual Study Level Depends on: Which one makes most sense Which is most consistently reported Which is mathematically appealing 26 Choice of Summary Statistic at the Individual Study Level Dichotomous (binary) data Relative effect: risk ratio, odds ratio Absolute effect: Risk difference NNT: not applicable to meta analysis Continuous data Difference in means: all outcomes are on the same scale Standardized mean difference When outcome is measured on a different scale Can be difficult to interpret Time to event (survival outcomes): hazard ratios 27 9

4 4 2/20/2012 Part 4. Models for Combining Data Simple Average (-6.2) + (- 7.7) + (-0.1) 3 = -4.7 mmhg Study Sample 95% Confidence Size mmhg Interval ANBP 554-6.2-6.9 to -5.5 EWPHE 304-7.7-10.2 to -5.2 Kuramoto 39-0.1-6.5 to +6.3 k x i i 1 X n 29 Weighted Average (554 x -6.2) + (304 x -7.7) + (39 x -0.1) 554 + 304 + 39 = -6.4 mmhg Study Sample Size 95% Confidence mmhg Interval ANBP 554-6.2-6.9 to -5.5 EWPHE 304-7.7-10.2 to -5.2 k w i x i i 1 X k w i i 1 Kuramoto 39-0.1-6.5 to +6.3 30 10

General Formula - Weighted Average Effect Size (d + ) k w i d i d i 1 k w i i 1 where: d i = effect size of the i th study w i = weight of the i th study k = number of studies 31 Weighted average over i studies Generic Inverse Variance Weighted Average With weight defined as inverse of the SE squared k i WY 1 i i k i W 1 i k WY i 1 i i1 kw i 2 W i 1 i s i 32 Two Meta analytic Models Fixed effect meta analysis Assumes that the true effect size each study is trying to estimate is the same (i.e. fixed) across all the studies. There will be differences in the estimates each study arrives at, but tthese are just tbecause of chance. Random effects meta analysis Incorporates an estimate of the between study variation (heterogeneity) by assuming that there is more than one true effect 33 11

FIXED EFFECTS MODEL POOLED RESULT ESTIMATED TREATMENT EFFECT SINGLE TRUE TREATMENT EFFECT RESULTS OF MULTIPLE CLINICAL TRIALS RANDOMLY DISTRIBUTED AROUND THE TRUE TREATMENT EFFECT TREATMENT EFFECTS (RD, OR, RR) 34 RANDOM EFFECTS MODEL MULTIPLE TRUE TREATMENT EFFECTS (distribution of treatment effects) TREATMENT EFFECTS (RD, OR, RR) 35 RANDOM EFFECTS MODEL POOLED RESULT SINGLE ESTIMATED TREATMENT EFFECT MULTIPLE TRUE TREATMENT EFFECTS (distribution of treatment effects) RESULTS OF MULTIPLE CLINICAL TRIALS RANDOMLY DISTRIBUTED AROUND EACH OF THE TRUE TREATMENT EFFECT TREATMENT EFFECTS (RD, OR, RR) 36 12

Fixed Effect and Random Effects Models Fixed Effect Weight w i 1 v i Random Effects Weight * 1 i i w v v * where: v i = within study variance v * = between study variance 37 Fixed effect Random effects k k i WY 1 i i W ( ) 1 i i Yi k k W i 1 i W ( ) i 1 i with W i 1 2 s i with W ( ) i 1 ˆ 2 2 s i 38 Which Model is Appropriate? Fixed effect: All studies in the analysis are identical Goal is to compute the common effect size ES for the identified population and not to generalize to other populations relatively rare Random effects: When expect variation in effects across studies are not all the same Want to generalize to a range of scenarios Choice between FE and RE should not be based on a statistical test, but rather an understanding of whether or not all studies share a common ES Generally the random effects model is more plausible 13

Comparison between fixed and random effects models Fixed Effect Random Effects 40 Part 5. Assessing and Exploring Heterogeneity 41 Why do Empirical Results Vary? Different study populations Different treatments or protocols Quality of technical design or execution Random variation 42 14

Heterogeneity Homogeneity: degree to which studies are sufficiently similar that a single average effect would be meaningful Heterogeneity: any kind of variability among studies in a systematic review Clinical diversity: variability in participants, interventions, outcomes Methodological diversity: variation in study design and risk of bias Statistical heterogeneity: a consequence of clinical or methodological diversity Only pool studies that will provide a meaningful summary 43 Presenting, Assessing, and Exploring Heterogeneity Present Forest plot Assess Cochran s Q (Chi square test) I 2 (Higgins) Explore Subgroup analysis Meta regression Sensitivity analysis 44 Example of a Forest Plot Showing Heterogeneity Association between the metabolic syndrome and cardiovascular disease ES Ford Diabetes Care, 2005. 28:1769 1778 15

Heterogeneity has Two Components Variation in effect size includes both true heterogeneity and random error Want to isolate the true variance: Compare the observed dispersion with the amount expected if all studies shared a common effect size The excess is assumed to reflect real differences among studies This portion of the variance is then used to create several measures of heterogeneity Chi Square Homogeneity Test (Mantel Haenszel) k 2 2 ( k 1 ) df i i i 1 Q w d d NOTE: d = ln(ori d+ = ln(ormh) wi = 1/variance (ORi) Variance (ORi) = 1/ai + 1/bi + 1/ci + 1/di 47 Heterogeneity: Higgins I 2 I 2 quantifies the proportion of the observed dispersion that is real rather than spurious (due to random error) 2 Q ( df) I Q Q is Cochran s Q chi-square statistic I 2 is expressed as a percentage [0-100%] Negative values set to 0 25%, 50%, and 75% represent low, moderate, and high heterogeneity I 2 can be directly compared between meta-analyses with different numbers of studies and different types of outcome data 48 16

Strategies for Dealing with Heterogeneity Check that the data are correct Do not do a meta analysis Ignore heterogeneity: fixed effects model Use a random effects model dl Explore heterogeneity: subgroup analysis, meta regression Change the effects measure Exclude studies 49 Dealing with Heterogeneity HETEROGENEOUS TREATMENT EFFECTS IGNORE ESTIMATE (insensitive) INCORPORATE EXPLAIN FIXED EFFECTS MODEL DO NOT COMBINE WHEN HETEROGENEITY IS PRESENT RANDOM EFFECTS MODEL SUBGROUP ANALYSES META- REGRESSION (control rate, covariates) Lau 1997 Subgroup Analysis Explore heterogeneity with subgroup analyses Can be used to answer specific questions about patient groups Can be used to investigate heterogeneousresults results Data are often insufficient in the primary studies Subgroup analyses are observational: no randomized comparisons Risk of confounding by bias from other study level characteristics Risk of false (+) if excessive number of subgroup 51 17

Meta Regression An observational study: confounding is a possibility Extends random effects meta analysis to estimate the extent to which study level covariates explain heterogeneity in the results Use even if the test for heterogeneity is non significant Use study level covariates Avoid ecologic fallacy Specify the study covariates before you begin Avoid data dredging 52 Sensitivity analyses Vary inclusion criteria Change your assumptions Fixed versus random effects model Different metrics Cumulative meta-analysis 53 Part 6. Publication Bias 54 18

Publication Bias Definition: The publication of studies based on the magnitude and direction of the findings If missing studies are systematically different from identified studies, then bias is introduced into the systematic review Published studies are more likely to report positive results Literature based meta analyses tend to bias results towards statistically significance and appear convincing due to increased precision 55 Types of Publication Bias English language bias Availability bias Cost bias Familiarity bias Duplicate publication bias Citation bias Time lag bias 56 Impact of publication bias Combination chemotherapy vs. monotherapy in ovarian cancer Meta analysis based on published and registered trials Survival ratio (95% confidence interval) Published trials (p=0.004) Registered trials (p=0.17) 0.7 1.0 1.3 57 19

Publication Bias 1. How minimize PB? 2. How assess if PB has any impact on the observed effect? 3. How assessifpbmight be entirelyresponsibleresponsible for the observed effect? 4. How much effect does PB have on the observed effect size? 58 Publication Bias 1. How minimize PB? Exhaustive searching Include grey literature 2. How assess if PB has any impact on the observed effect? Graphical/visual assessment: Funnel plot Statistical testing: Egger s 1997, Begga nd Mazumdar 2004test 3. How assess if PB might htb be entirely responsible for the observed effect? Statistical testing: Fail safe N How many studies with would be needed to add to the MA before the p value became nonsignificant? 4. How much effect does PB have on the observed effect size? Statistical testing: Trim and Fill (Duval and Tweedie) Restrict analysis to larger studies 59 Assumptions of Models Examining Publication Bias Large studies are likely to be published regardless of statistical significance Small studies are at greatest risk for being lost: only the largest effects are likely to be published Effect of these assumptions: Expect bias to increase in sample size does down Various methods for assessing PB are based on this model 60 20

10 9 8 7 1/standard error 6 5 4 3 2 1 0-1.5-1 -0.5 0 0.5 1 1.5 Effect size A simulated funnel plot [n=35] Overall effect: 0.075 [-0.011, 0.161] 61 10 9 8 7 1/standard error 6 5 4 3 2 1 0-1.5-1 -0.5 0 0.5 1 1.5 Effect size Removal of the 5 leftmost points is consistent with a model for publication bias [n=30] Overall effect: 0.097 [0.009, 0.185] 62 Funnel Plots One mechanisms for displaying the relationship between sample size and effect size Plots effect size vs SS or a measure of variance: large studies cluster around mean effect; smaller studies tend to be spread across a broad range of values; creates a funnel Now usually plotted as effect size (X axis) versus 1/SE (Y axis) In the absence of publication bias, studies are distributed symmetrically about the mean effect size, as sampling error is random In the presence of publication bias, there will be missing studies with nonsignificant effect size and smaller sample size or large SE Interpretation of funnel plot is subjective, so various tests have been devised 63 21

Possible Causes of Funnel Plot Asymmetry Publication bias Funnel plot assumes that if effect size is larger in smaller studies, it is due to publication bias True heterogeneity There may be real, small study effects Choice of precision measure Choice of effect measure Trial quality Chance 64 Part 7. Reading and Interpreting Meta-Analyses 66 22

Working Example of a Systematic Review and Meta analysis 67 Mortality with Tiotropium vs Placebo (Singh et al. BMJ 2011) 68 Mortality with Tiotropium vs Placebo: Subgroups (Singh et al. BMJ 2011) 69 23

Mortality with Tiotropium vs Placebo: Sensitivity Analysis (Singh et al. BMJ 2011) 70 Part 8. Resources on Meta-Analyses Resources for Performing Meta Analysis Organizations Cochrane Collaboration: www.cochrane.org NICE, SIGN AHRQ: Methods Guide for Effectiveness and Comparative Effectiveness Reviews 2009, http://effectivehealthcare.ahrq.gov 72 24

Books on Meta Analyses Systematic Reviews in Health Care: Metaanalysis in context. Egger M, Davey Smith G, Altman DG. (eds). BMJ Books, 2001 Introduction to Meta-Analysis. Borenstein M. Hedges LV, Higgins JPT, Rothstein H. Wiley, 2009. 73 Books on Meta Analyses, Cont d Finding What Works in Health care ; Standards for Systematic Reviews. Institutes of Medicine of the National Academies, 2011. Systematic Reviews of Interventions. Higgins JPT, Green S. Wiley & Sons, 2008. 74 Meta analysis Software Customized software Commercial packages Comprehensive Meta Analysis [www.meta Analysis.com] Meta Win [www.metawinsoft.com] Freely available RevMan (www.cochrane.org) org) Meta Analyst MIX (Meta analysis with Interactive explanations) Statistical packages with meta analysis capability via custom code or macros Stata, SAS, SPSS, R 75 25

Contemporary guidelines for reporting meta analyses: PRISMA 2009 76 77 78 26

Suggested Readings 1. Schriger DL, Altman DG, Vetter JA, Heafner T, Moher D. Forest plots in reports of systematic reviews: a cross sectional study reviewing current practice. Int J Epidemiol 2010;39:421 429 2. Singh S, Loke YK, Enright PL, Furberg CD. Mortality associated with tiotropium mist inhaler in patients with chronic obstructive pulmonary disease: systematic review and meta analysis of randomised controlled trials. BMJ 2011; 342:d3215 3. Lau J, Ioannidis JPA, Schmid CH. Quantitative Synthesis in Systematic Reviews. Ann Intern Med. 1997;127:820 826 4. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta analysis detected by a simple, graphical test. BMJ 315:629 79 Conclusions Meta analysis is the statistical combination of results from two or more studies Advantages: increase in power, improved precision, ability to answer questions not posed by individual studies, examine conflicting evidence Caveats Relies on an underlying systematic review Limited by available data Beware of spurious precision Most MA involve a weighted average of effects estimates from different studies Heterogeneity across studies must be considered and explained Use sensitivity analyses to examine robustness of findings 80 References & Resources 1. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994;50:1088 1101. 2. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta analysis. Chichester, UK: Wiley. 3. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]. The Cochrane Collaboration, 2009. Available from www.cochranehandbook.org. 4. Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel plot based method of testing and adjusting for publication bias in meta analysis. Biometrics, 56(2), 455 463. 5. Egger, M., Smith, George Davey, Altman, Douglas. (2001). Systematic Reviews in Health Care: Meta Analysis in Context (2 ed.): BMJ Books. 6. Ford, E. S. (2005). Risks for all cause mortality, cardiovascular disease, and diabetes associated with the metabolic syndrome: a summary of the evidence. [Review]. Diabetes Care, 28(7), 1769 1778. 7. Glass, G. V. (1976). Primary, Secondary, and Meta Analysis of Research. Educational Researcher, 5(10), 3 8. 8. Gould, M. K., Dembitzer, A. D., Doyle, R. L., Hastie, T. J., & Garber, A. M. (1999). Low molecular weight heparins compared with unfractionated heparin for treatment of acute deep venous thrombosis. A metaanalysis of randomized, controlled trials. [Comparative Study Meta Analysis Research Support, U.S. Gov't, P.H.S.]. Ann Intern Med, 130(10), 800 809. 9. Lau, J., Ioannidis, J. P., & Schmid, C. H. (1997). Quantitative synthesis in systematic reviews. [Research Support, U.S. Gov't, P.H.S.]. Ann Intern Med, 127(9), 820 826. 10. Michael Borenstein, L. V. H., Julian P. T. Higgins, Hannah R. Rothstein. (2009). Introduction to Meta Analysis: John Wiley & Sons, Ltd. 27

References & Resources 11. Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta analyses: the PRISMA statement. [Guideline 12. Research Support, Non U.S. Gov't]. PLoS Med, 6(7), e1000097. doi: 10.1371/journal.pmed.1000097 13. Mulrow, C. D., Cook, D. J., & Davidoff, F. (1997). Systematic reviews: critical links in the great chain of evidence. [Comment Editorial]. Ann Intern Med, 126(5), 389 391. 14. Nissen, S. E., & Wolski, K. (2007). Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. [Meta Analysis]. N Engl J Med, 356(24), 2457 2471. doi: 10.1056/NEJMoa072761 15. Research, C. o. S. f. S. R. o. C. E., & Medicine, I. o. (2011). Finding What Works in Health Care: Standards for Systematic Reviews: The National Academies Press. 16. Schriger, D. L., Altman, D. G., Vetter, J. A., Heafner, T., & Moher, D. (2010). Forest plots in reports of systematic reviews: a cross sectional study reviewing current practice. International Journal of Epidemiology, 39(2), 421 429. doi: 10.1093/ije/dyp370 17. Singh, S., Loke, Y. K., Enright, P. L., & Furberg, C. D. (2011). Mortality associated with tiotropium mist inhaler in patients with chronic obstructive pulmonary disease: systematic review and meta analysis of randomised controlled trials. [Meta Analysis Research Support, N.I.H., Extramural Review]. BMJ, 342, d3215. doi: 10.1136/bmj.d3215 28