Meta-Analyses: Considerations for Probiotics & Prebiotics Studies Daniel J. Tancredi, PhD Associate Professor in Residence Department of Pediatrics UC Davis School of Medicine
Objectives Provide an overview of systematic reviews & meta-analyses Discuss with you key considerations & controversies for sponsors of probiotic/prebiotic studies
Key sources of material for talk Julie Glanville, Sarah King, Francisco Guarner, Colin Hill, Mary Ellen Sanders. Nutrition Journal (2015) 14:16
Key sources of material for talk Systematic Reviews in Health Care: Meta-Analysis in Context, 2nd Edition Matthias Egger (Editor), George Davey-Smith (Editor), Douglas Altman (Editor) March 2001, BMJ Books
Definitions: Systematic Review: A review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies. Source: http://community.cochrane.org/glossary/
Systematic Reviews: Scientific Virtues Protocol-driven Transparent (understandable) Reproducible Objective Rigorous (aims to reduce biases and imprecision)
Narrative vs. Systematic Reviews Feature Narrative Systematic Question Often broad Focused Sources & search Not usually specified, potentially biased Selection Comprehensive sources, explicit search strategy Criterion-based, uniformly applied Appraisal Variable Rigorous Synthesis Often qualitative Quantitative Inferences Sometimes evidence-based Source: Cook DJ et al (1998) Synthesis of best evidence for clinical decisions Ch. 1 of Systematic Reviews: Synthesis of Best Evidence for Health Care Decisions (ed. Mulrow C & Cook D). ACP, Philadelphia. Usually evidencebased
When to do a systematic review Beginning and end of every RCT!? Clarke et al. Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report. J R Soc Med. 2007 Apr; 100(4): 187 190. CONSORT 2010 recommends (does not require) post-rct SR: as an internal validity check (item 20) to situate results in context (item 22) CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869
Example: Before/After Meta-analysis in RCT Meta-analysis of RCTs of probiotics for management of infant colic (crying time at 21 days post-intervention) Previous Updated Valerie Sung et al. BMJ 2014;348:bmj.g2107 2014 by British Medical Journal Publishing Group
When to do a systematic review (cont.) According to EFSA food/feed safety guidance, a systematic review may be useful when research question could be answered by an envisionable primary research study design and practical and worthwhile when SR on available data could improve precision of estimated values for parameters of interest Source: Application of SR methodology to food and feed safety assessments EFSA Guidance for those carrying out systematic reviews. EFSA Journal 2010; 8(6):1637
also from EFSA Q. What are the requirements for meaningful SR results for health benefits of foods or constituents? A. SR s (with or without meta-analyses) can be useful if: 1. They ask a clear and specific question 2. They are carried out rigorously so as to minimise bias and random error 3. Reported well enough to allow assessment of the level of bias in the underlying evidence & in the review process --Lee Hooper, U. East Anglia In presentation (http://www.efsa.europa.eu/en/events/documents/131120- p07.pdf ) at 2013 EFSA Technical meeting on the reporting of human studies submitted for the scientific substantiation of health claims http://www.efsa.europa.eu/en/supporting/doc/569e.pdf)
Don t forget: RCTs are still primary Sponsors must strive for high quality RCTs Rationale: The assessment of individual RCTs is the key stage in weighing the totality of evidence in favor of a claim. Sources: Appendix H (Study Quality) of EFSA's "Opinion of the Panel on dietetic products, nutrition and allergies (NDA) on a request from the Commission related to scientific and technical guidance for the preparation and presentation of the application for authorisation of a health claim" and Table 13a of Health Canada s Guidance Document for Preparing a Submission for Food Health Claims (March, 2009)
Another Example Probiotics for the prevention of Clostridium difficile associated diarrhea in adults and children Cochrane Database of Systematic Reviews 31 MAY 2013 DOI: 10.1002/14651858.CD006095.pub3 http://onlinelibrary.wiley.com/doi/10.1002/14651858.cd006095.pub3/full#cd006095-fig-0003
How to do a SR: Seven Steps 1. Frame question (& preparing protocol) 2. Systematically locate relevant studies 3. Study selection 4. Data extraction (or assembly for IPDMA) 5. Assess quality of studies 6. Synthesize evidence (via meta-analyses) 7. Interpret findings Source: Glanville J, King S, Guarner F, Hill C, Sanders ME. A review of the systematic review process and its applicability for evaluating evidence for health claims on probiotic foods in the EU. Nutrition Journal (2015) 14:16
Register and publish protocol early! Pre-specify research question, eligibility criteria & methods to be used for all steps in protocol Register SR at PROSPERO http://www.crd.york.ac.uk/pros PERO/ Publish protocol
Frame question Explicit specification of PICOS Population, including settings/locations Intervention(s), including vehicles/matrices Comparison Outcome(s), including timing & how measured Study types (designs & methodological quality)
Systematically locate relevant studies Comprehensive (published & unpublished) Explicit description of Sources (trial registries, MEDLINE, EMBASE, Cochrane Controlled Trials Register) Queries (search terms & logic operators) Methods for tracing citation linkages [handsearching key journals & personal communication with experts]
Study Selection 2+ observers check eligibility Develop strategy to resolve disagreements 2-stages: Titles & Abstracts Full study Keep track of inclusion/exclusion decisions for each stage (for PRISMA Flow Chart)
From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097
Data Extraction/Assembly Develop, pilot test & refine a standard data extraction form 2+ observers [Replicate published results] Frank D Amico, PhD Professor of Mathematics Duquesne University
Assess Quality of Studies GRADE approach (quality is degree of confidence that effect size estimate is close to true parameter) For RCTs, assess random sequence generation/concealment, Blinding, Incomplete outcome data, Selective reporting and other biases [validation of intervention/comparator composition] See http://www.gradeworkinggroup.org/index.htm for guidelines from the Grading of Recommendations Assessment, Development and Evaluation Working Group Also see: http://handbook.cochrane.org/chapter_8/table_8_5_a_the_cochrane_colla
Probiotics for the prevention of Clostridium difficile associated diarrhea in adults and children Cochrane Database of Systematic Reviews 31 MAY 2013 DOI: 10.1002/14651858.CD006095.pub3 http://onlinelibrary.wiley.com/doi/10.1002/14651858.cd006095.pub3/full#cd006095-fig-0002
Synthesize Evidence Tabulate results (study-specific) Examine forest plots Assess heterogeneity Statistical distinction: sample v. population Consider MA of all trials or subgroups Perform sensitivity analysis Examine funnel plots (publication bias) Make available list of excluded studies Sackett DL, Glasziou P, Chalmers I. Meta-analysis may reduce imprecision, but it can t reduce bias. Unpublished commentary commissioned by the New England Journal of Medicine, 1997. (see SR in Health Care, p. xiv)
Funnel Plots Example Symmetrical plot in the absence of reporting bias Asymmetrical plot in the presence of reporting bias http://handbook.cochrane.org/chapter_10/figure_10 _4_a_hypothetical_funnel_plots.htm
Meta-Analysis Meta-analysis may reduce imprecision, but it can t reduce biases Heuristically, a pooled effect size is estimated as a weighted averages of sample effect sizes, resulting in a more precise estimate (with a smaller uncertainty interval) Assess explainable/unexplainable heterogeneity in effect sizes, including subgroups Sensitivity analyses to assess robustness EFSA requires rationale & justification for MA Sackett DL, Glasziou P, Chalmers I. Meta-analysis may reduce imprecision, but it can t reduce bias. Unpublished commentary commissioned by the New England Journal of Medicine, 1997. (see SR in Health Care, p. xiv)
Copyright 2005 Elsevier Ltd Figure Example from Jack Cuzick s. Forest plots and the interpretation of subgroups. The Lancet 2005 365,Issue 9467
Meta-analysis of RCTs of probiotics for management of infant colic Outcome is crying time (min/day) at 21 days postintervention) Valerie Sung et al. BMJ 2014;348:bmj.g2107 2014 by British Medical Journal Publishing Group
Boxplots of Infant Crying Time in 3 RCTs for Managing Colic By Study Arm and Day of Follow-up Placebo L. reuteri DSM 17938
Present & Interpret Findings Follow PRISMA statement (www.prismastatement.org) for reporting (Indeed, follow PRISMA after conceiving SR, to make sure you re accounting for everything you ll need!) Explain departures from PRISMA (per EFSA)
Heterogeneity of causal effects
Three classes of evidence for causal claims in Howick et al s The evolution of evidence hierarchies: what can Bradford Hill s guidelines for causation contribute? Source: J R Soc Med. 2009 May 1; 102(5): 186 194 Copyright 2009, The Royal Society of Medicine
RCTs can provide direct (probabilistic) evidence of causality Intervention Black box Outcome Control Black box Outcome Randomization could yield sufficient evidence to outweigh plausible sources of confounding, even without knowing mechanism.
Howick et al s Two Levels of Mechanistic Evidence 1. Internal substudies of causal links in Black box 2. Purported Mechanism of Action demonstrated in external studies Source: J R Soc Med. 2009 May 1; 102(5): 186 194 2009, The Royal Society of Medicine
Challenge to Combinability: Black Boxes Depend on Study PICOS Study A Intervention Black box(a) Outcome Control Black box(a) Outcome Study B Intervention Control Black box(b) Black box(b) Outcome Outcome
The blank wheel of the pragmatic explanatory continuum indicator summary (PRECIS) tool. E represents the explanatory end of the pragmatic explanatory continuum. Kevin E. Thorpe et al. CMAJ 2009;180:E47-E57 2009 by Canadian Medical Association
Figure A: PRECIS summary of four unrelated RCTs, for illustration 2009 by Canadian Medical Association Kevin E. Thorpe et al. CMAJ 2009;180:E47-E57 NB: Potential use for modified PRECIS tools for SR has begun (see PMID: 21474282).
Possible distribution of mechanisms among probiotics Hill, C. et al. (2014) The International Scientific Association for Probiotics and Prebiotics consensus statement on the scope and appropriate use of the term probiotic Nat. Rev. Gastroenterol. Hepatol. doi:10.1038/nrgastro.2014.66
Some rationales for combining strains Rationale: Common structural or secreted product Example: All L. bulgaricus & S. thermophilus Rationale: Common MoA known to be n.&s. for effect (e.g., production of a specific bacteriocin or range of bacteriocins known to be active against a specific pathogen, or induction of immune mechanisms needed for the effect) Example: L. salivarius strains A & B Other rationales: common identity, taxonomy, physiological effect (in humans) Glanville et al. Nutrition Journal (2015) 14:16
Remember High quality RCTs are key Well-done SR s can provide more precise estimates of causal effects (narrower uncertainty intervals) and assess (some) heterogeneity of treatment effects Pooling different studies requires scientific basis for the common black box assumption
Questions and Comments djtancredi@ucdavis.edu