Striving for the Gold Standard in the Systematic Review Process

Transcription

1 Striving for the Gold Standard in the Systematic Review Process ESTHER F MYERS, PHD, RDN, FAND EFMYERS@EFMYERSCONSULTING.COM APRIL 22, /24/2017 1

2 Disclosures Past Chief Science Officer for Academy of Nutrition and Dietetics ( ) Evidence Analysis Library oinform decision making of dietetics professionals Nutrition Today contributor on evidence analysis and policy topics (2012 to present) Member of the ILSI North America Caffeine Systematic Review Team Expertise in systematic review methodology ILSI North America provided an honorarium and travel funds to attend this meeting. 5/24/2017 2

3 Meeting The Standards 5/24/2017 3

4 Goal: SYNTHESIZE BEST AVAILABLE RESEARCH TO INFORM DECISION MAKING 5/24/2017 4

5 Common purposes of systematic reviews Create treatment recommendations/ guidelines Identify gaps in research Input to policy decisions 5/24/2017 5

6 Use of systematic review methodology Medicine Emerged in 1980 s JAMA series on evidence based medicine starting 1995 Nutrition 2000 evidence based guidelines for CMS 2001 EBPC Nutrition Topics (AHRQ) 2004 AND Evidence Analysis Library (EAL) 2009 Nutrition Evidence Library (NEL) Toxicology 2015 OHAT Handbook 5/24/2017 6

7 Evolving Guidance Documents Publishing systematic reviews 1999-Quality of Reporting of Meta-analyses (QUOROM) 2009-Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) Conducting Systematic Reviews 1993 Overview Quality Assessment Questionnaire (OQAQ) 2001, AHRQ Other Organizational Methodology Manuals, Grades of Recommendation, Assessment, Development and Evaluation (GRADE) 2007 AMeaSurement Tool to Assess systematic Reviews (AMSTAR) Cochrane Methods Manuals Finding What Works in Health Care: Standards for Systematic Reviews Office of Health Assessment and Translation (OHAT) 5/24/2017 7

8 Caffeine Systematic Review Guidance Finding What Works in Health Care: Standards for Systematic Reviews (IOM) Office of Health Assessment and Translation (OHAT) Grades of Recommendation, Assessment, Development and Evaluation (GRADE) Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) 5/24/2017 8

9 Finding Provides standards for: Initiating a review (Chapter 2) Finding and assessing individual studies (Chapter 3) Synthesizing the body of evidence (Chapter 4) Reporting systematic reviews (Chapter 5) 5/24/2017 9

10 Standards for Initiating a Systematic Review 2.1 Establish team - expertise and experience 2.2 Manage bias and conflict of interest of team 2.3 Ensure user and stakeholder input 2.4 Manage Bias and COI 2.5 Formulate topic for systematic review 2.6 Develop systematic review protocol 2.7 Submit protocol for peer review 5/24/

11 2.1 Establish multidisciplinary team with appropriate expertise and experience Required expertise: Pertinent clinical content areas Systematic review methods Searching for relevant evidence Quantitative methods Other expertise as appropriate

12 2.1 Establish team with appropriate expertise and experience Entity Description Roles Scientific Review Team: ToxStrategies Scientists with a range of expertise (caffeine, toxicology, epidemiology, systematic review, literature searching, etc.) A, D, F, I, P Develop and perform the SR (consistency in application of SR process, independent assessment, documentation) Scientific Advisory Board Multidisciplinary experts (systematic review, behavior, cardiovascular, bone & calcium, acute, pharmacokinetics PhD s and MDs from academic, private, and clinical practices) I, A, D Provide input, review, and approval. Develop protocol and conclusions. Sponsor: ILSI North America ILSI North America Caffeine Working Group *Budgetary A = approve; D = develop; F = facilitate; I = input; P = perform 5/24/

13 Name Team Role Clinical Content Systemati c Review Methods Tox Strategies Team Searching expertise Quantitative methods expertise D Wikoff, PhD Tox St Food Safety X X Other C Doepker, PhD Tox St Food Safety GRAS expert B Welsh, PhD Tox St Neuro Toxicology R Henderson Tox St Dev and Repro G. Brorby,PhD Tox St Toxicology X J Britt Tox St Epi & Toxicology S Harvey Tox St X J Urban, PhD Tox St Epi & Toxicology Additional Consultants Tox Strategies Biostatistician L Graham X Systematic Review Librarian X X X 5/24/

14 Name Scientific Advisory Board Team Role Clinical Content J Goldberger, MD SAB Cardiovascular & Toxicity H Lieberman, PhD SAB Physical performance Behavior Systematic Review Searching experti se Quantitative methods expertise Other Caffeine review, expert witness 5/24/ research papers & 4 book chapters Government liaison to ILSI North America E Myers, PhD SAB Nutrition x IOM Dietary Supplement Cmte, AND EAL C O Brien, MD SAB Behavior J Peck, PhD SAB Reproductive ad developmental toxicity Scientific Advisor to ILSI North America M Tenebein, PhD SAB Pharmacology Public statement on energy drinks C Weaver, PhD SAB Bone/Calcium Board Member ILSI Global

15 ILSI North America Sponsor Role Establish scope of project ILSI North America Caffeine Working Group* Identified need for use of IOM Standards Select vendor (Tox Strategies) Provide Resources Agreements for SAB and Tox Strategies Decisions about additional resources* Coordinate Teleconferences Facilitate Stakeholder input FDA, Health Canada* No input on systematic review content 5/24/

16 2.2 Manage bias and conflict of interest (COI) of team 2.4 Manage bias and COI for individuals providing input into systematic review Scientific Advisory Board and Tox Strategies Team COI Plan established COI declaration Two times at beginning and again when manuscript was prepared Disclosure To group in teleconference Included in Prospero protocol Managed on calls / meetings Self declare potential COI if topics came up 5/24/

17 2.3 Ensure user and stakeholder input as the review is designed and conducted ILSI North America Caffeine Working Group Initial scope of project, Federal Agency input (FDA and Health Canada) Beginning of project and Phase 1 (Initiate) Conclusion Phase 2 (Find and Assess) Conclusion Phase 3 (Synthesize) Scientific Board Input Four SAB Meeting Calls (Phase 1, Phase 2, Phase 3 and Phase 4 2 Endpoint SAB member calls (Phase 3 and Phase 4) Multiple s soliciting input Phase 1, Phase 2, and Phase 4 5/24/

18 2.5 Formulate topic for systematic review PECO format (OHAT) Population Exposure Comparator (Nawrot, et al) Outcome Levels of questions Overarching Question Sub questions Specific Populations Specific Health Outcomes Specific Endpoints 5/24/

19 Topics Divided Into Two Categories TOPICS FOR SYSTEMATIC REVIEW Five Health Outcomes Acute toxicity Cardiovascular Bone and Calcium Human behavior Reproduction/development 27 specific Endpoints and 4 sub-populations TOPICS FOR BACKGROUND LITERATURE REVIEW Pharmokinetics (PK) is contextual topic Characterize the current understanding of caffeine PK Critically review anything that advances the science, particularly with respect differences/similarities between our populations of interest, characterization of PK in non-adult populations of interest, characterization of PK (particularly fast/slow phenotypes) in context of our endpoints of interest. 5/24/

20 Analytic Framework P C E O o For [population], is caffeine intake above [dose], compared to intakes [dose] or less, associated with adverse effects on [endpoint]? 5/24/

21 2.6 Develop systematic review protocol 1. Context and rational for review 2. Study Screening and Selection Criteria Outcome measures, time points, interventions, and comparison groups 3. Search strategy 4. Procedures for study selection 5. Data extraction strategy 6. Process for identifying/resolving disagreements 7. Approach to critically appraising individual studies 8. Method for evaluating body of evidence 9. Planned analysis of patient subgroups 10. Timeline 5/24/

22 2.6.1 Context and Rationale 5/24/

23 2.6.2 Study Screening and Selection Criteria Example: Prospero protocol-reproductive and development Inclusion/Exclusion Criteria Types of studies to be included All study types (except case studies without controls) with quantitative exposure and outcome Only reviews with original data are included (others SR s consulted) Exclude studies only assessing beneficial or therapeutic endpoints/outcomes Condition or domain being studied Reproduction and development 5/24/

24 Study Screening and Selection Criteria (cont.) Participants/Population Healthy adults, healthy pregnant women Exposure: 400 mg/day, 300 mg/day (pregnant women) Comparator: <400 mg/day, <300 mg/day 5/24/

25 Study Screening and Selection Criteria (cont.) INCLUSION Healthy population, athletes, military and pregnant women subjects not specifically described as hospitalized, diagnosed with disease, or receiving medical treatment at time of study Studies evaluating effect of caffeine exposure in humans Healthy subjects if used as control group EXCLUSION Unhealthy population, asthmatics, smoking, high risk conditions, e.g. multiple gestations, pre-existing gestational diabetes, hypertension/preeclampsia, history of preterm deliver, placental abnormalities, etc IVF or fertility treatments Studies in animal species or in vitro studies Case studies with no comparison group

26 Study Screening and Selection Criteria Exposure Exposure to caffeine = 400 mg/day for healthy adults, 300 mg/day for pregnant women INCLUSION Studies with quantitative exposure to caffeine associated in an adverse effect Caffeine exposure from coffee, tea, chocolate, cola, energy drinks, supplements, medicines, energy shots, chewing gum sport gel and sports bars Studies evaluating effects of caffeine alone from above sources) or combination with compounds EXCLUSION Studies without quantitative exposure to acceptable caffeine source Studies with only decaffeinated exposures, placebo, Yerba mate, guarana, damiana, and/or contaminants of caffeine or caffeine metabolites Studies that evaluate caffeine in combination with other OTC with pharmacologically active compounds Studies assessing caffeine exposure after the end of pregnancy or birth 5/24/

27 Study Screening and Selection Criteria (cont.) Outcome measurement Primary Outcome- Adverse reproductive and developmental effects INCLUSION Effects on birth weight, fertility, fetal growth, miscarriage, sperm, etc Studies of maternal outcomes (e.g. preeclampsia, endometriosis, gestational diabetes, post-partum depression, etc) EXCLUSION Studies of non adverse effects, fetal heart rate, fetal movement, and fetal reactivity Secondary Outcome Pharmocokinetic data that could be used to interpret the primary outcome (e.g. contextual information) 5/24/

28 2.6.3 Outcome measures, time points, interventions, and comparison groups Reproductive & Developmental Endpoints Fertility, fecundability and male repro measures Prenatal exposure related to childhood behavior Time Points Exposure During Pregnancy Outcome During Pregnancy, At Birth, or After Birth Spontaneous abortion Recurrent miscarriage Stillbirth Birth defects Preterm birth and gestational age Fetal growth Childhood cancers other Intervention/EXPOSURE Total Daily Caffeine Intake Comparison Group Pregnant women or children of women with daily intake of 300 mg caffeine during pregnancy 5/24/

29 2.6.3 Search Strategy Input from SAB Designed by Systematic Review Trained Librarian Common elements Databases: PubMed, EMBASE, Cochrane Database of Systematic Review Dates: 2001 and June Published search strategy on Prospero Types of studies (no reviews without original data) Exclude therapeutic endpoints or outcomes Quantitative exposure characterized in humans Specific elements to clinical outcome and population 5/24/

30 2.6.4 Procedures for study selection Titles, Abstracts, compared to criteria by 2 people Classified as Included, Excluded or Unclear Reviewed by SAB clinical content expert Full Articles reviewed by Tox Strategies Team Unclear, discuss by 2 other Tox Strategy team members Inclusion and Exclusion Lists provided to SAB Dual review was accomplished by Tox Strategies and SAB 5/24/

31 2.6.5 Data extraction strategy Data extracted into Distiller SR Excel spreadsheets were provided to SAB Uploaded to AHRQ Systematic Review Database Repository (SRDR) Common data elements and Endpoint specific elements Data extraction forms pilot tested 5/24/

32 2.6.6 Process for identifying and resolving disagreements (Tox Strategies, EFM, SAB Member(s)) Teleconferences with individual SAB members, Tox Strategies, and EFM Always potential for minority opinion Was not necessary 5/24/

33 2.6.7 Approach to critically appraising individual studies OHAT risk of bias rating tool (2015) 11 criteria: Randomization Allocation concealment Identical experimental conditions Confounding (design/analysis) Blinding during study Missing outcome data Confident of exposure Outcome assessment Outcomes reporting Other threats to internal validity Overall risk of bias score U.S. National Toxicology Program: Office of Health Assessment and Translation Risk of Bias Rating Tool for Human and Animal studies (2015) 5/24/

34 2.6.8 Method for evaluating body of evidence GRADE METHODOLOGY Assessment is based on the body of evidence as a whole Considerations for positive and negative findings Quality of data (as reflected by Risk of Bias) Level of adversity Consistency Magnitude of effect 5/24/

35 2.6.8 Method for evaluating body of evidence Body of Evidence classified as: High (++++) Moderate (+++) Low (++) Very Low (+)

36 2.6.8 Method for evaluating body of evidence Tables of evidence 5 Tables one for each Health Outcome Included individual Endpoints Narrative Chapters for each Health Outcome Conclusion Statements

37 Narrative Conclusion Statements Two Levels: Individual Endpoint conclusion statement Health Outcome conclusion statement EXAMPLE 10 individual Endpoint conclusion statements One overall Reproduction and Development Health Outcome conclusion statement 5/24/

38 2.6.9 Planned analysis of differential treatment effects in subgroups PECO statement and Analytical Framework included four populations: Healthy Adults Healthy Pregnant Women Healthy Adolescents (12-19 years old) Healthy Children (3 11 years old 5/24/

39 Timeline Originally planned as a 2 year project 5/24/

40 2.7 Submit protocol for peer review Five Protocols submitted to PROSPERO Systematic review of the effects of caffeine consumption in healthy adults and pregnant women acute adverse reproductive and developmental effects cardiovascular effects bone and calcium balance effects behavioral effects 5/24/

41 Standards for Finding and Assessing Individual Studies 3.1 Conduct a comprehensive systematic search for evidence 3.2 Take action to address potentially biased reporting 3.3 Screen and Select studies 3.4 Document the search 3.5 Manage data collection 3.6 Critically appraise each study 5/24/

42 3.1 Conduct a comprehensive systematic search for evidence Each of the search plans identified in the Prospero protocols were conducted. Reviewed 5/24/

43 3.2 Take action to address potentially biased reporting IOM recommends: Grey Literature Ask researchers to clarify data Invite submission of unpublished data Handsearch Web search Non English Language 5/24/

44 3.3 Screen and select studies Screening protocol pilot tested and used protocol criteria 2 people Response to criteria uploaded into Distiller SR Title/Abstract and Full Text review Included Excluded Further Discussion Did include observational research in addition to just RCT 5/24/

45 3.4 Document the search SEARCH SYNTAX Attachment to Prospero Protocols RESULTS UPLOADED INTO SR DISTILLER 5/24/

46 Manuscript include Summary Diagram LINE BY LINE DISPOSITION OF ARTICLES AHRQ Systematic Review Data Repository (SRDR) 5/24/

47 Systematic review of the adverse bone and calcium balance effects of caffeine consumption in healthy adults, pregnant women, adolescents, and children Link: Systematic review of the adverse cardiovascular effects of caffeine consumption in healthy adults, pregnant women, adolescents, and children Link: Systematic review of acute adverse effects of caffeine consumption in healthy adults, pregnant women, adolescents, and children Link: Systematic review of the adverse behavioral effects of caffeine consumption in healthy adults, pregnant women, adolescents, and children Link: Systematic review of the adverse reproductive and developmental effects of caffeine consumption in healthy adults and pregnant women Link:

48 3.5 Manage data collection Trained and pilot tested forms prior to starting Distiller SR software used for data collection/ management Reviewers working independently data extraction Standard data collection forms for each Health Outcome, Population, and Endpoint Authors data and extractor compiled data Second level of review of data extracted by SAB ed questions Teleconference if needed to resolve questions/concerns 5/24/

49 3.5 Manage data collection (cont.) Data extracted into SR Distiller included: (1) basic information as reported by the author: (2) Customized fields or project-specific fields included the following: outcome(s) and endpoint(s) objective methods study design categorization source of caffeine exposure exposure metric (measured, self-report) population reference/comparison confounders results dose-response evaluation conclusion pharmacokinetic information funding sources COIs. standardization of exposure/dose to SR, selection of endpoints and exposure/dose for comparisons, and information related to assessing trends, consistency 5/24/

50 3.6 Critically appraise each study Risk of Bias Heat Maps U.S. National Toxicology Program: Office of Health Assessment and Translation Risk of Bias Rating Tool for Human and Animal studies (2015) RoB for each domain is indicated by color: Dark Green: definitely low risk of bias (+2) Light Green: probably low risk of bias (+1) Light Red: probably high risk of bias (-1) Dark Red: definitely high risk of bias (+2) White: does not apply 5/24/

51 Standards for Synthesizing the Body of Evidence Qualitative synthesis 4.1 For each outcome use a prespecified method to evaluate the body of evidence 4.2 Conduct a qualitative synthesis 4.3 Decide if review will include quantitative analysis (meta analysis) NO (4.4 only applies to meta analysis) 5/24/

52 4.1 For each outcome use a prespecified method to evaluate the body of evidence Forest Plots Narrative Conclusion Statement Table Showing Confidence in Body of Evidence Final Summary for all endpoint/health Outcomes 5/24/

53 Plots 5/24/

54 Confidence in Body of Evidence Table Initial Confidence Rating Impact of Risk of Bias Impact of other Grade Criteria Final Confidence Rating 5/24/

55 Overall Summary Table 5/24/

56 Overall Summary Table OUTCOME/ENDPOINT COMPARATOR POPULATION NUMBER OF STUDIES RANGE OF NOELS (LOELS) 5/24/

57 Overall Summary Table COMPARATOR IS TOO HIGH ACCEPTABLE TOO LOW 5/24/

58 Overall Summary Table LEVEL OF ADVERSITY TYPE OF EFFECT IMPORTANCE OF EFFECT IN DECISION MAKNG LEVEL OF CONFIDENCE 5/24/

59 Overall Summary Table NARRATIVE OVERALL CONCLUSION and OVERALL COMPARATOR CONCLUSION 5/24/

60 4.2 Conduct a qualitative synthesis CONCLUSION STATEMENT CONTENT Population Number of Studies Spectrum of Endpoint Conclusions Comparator (Nawrot level) is: Too high, Acceptable or Too Low Level of Adversity Type of effect and Importance in Decision Making Overall level of confidence and reasons is stated Very low/low/moderate/high 5/24/

61 Example of one individual Endpoint narrative conclusion statement Reproduction and Development Recurrent Miscarriage Outcome (1 of 9 outcomes) The evidence from these four studies provides a moderate level of confidence that 300 mg/day is an acceptable intake that would not be associated with significant concerns for endpoints related to recurrent miscarriage in healthy pregnant women. Confidence is increased by the overall low level of indirectness (Table 1), low risk of bias (Figure 10), and reasonable consistency in the lack of findings. 5/24/

62 Example of one Health Outcome conclusion statement In summary, the SR of 58 studies (55 observational and 3 RCTs) provided evidence to evaluate (describe health outcomes, endpoints and populations) When the weight of the evidence was considered, an intake of 300 mg caffeine/day in pregnant women and 400 mg /day in healthy adults are generally without significant concern regarding overt adverse effects on.. (describe outcomes evaluated) There is a moderate level of confidence in the evidence base which supports this conclusion 5/24/

63 4.3 Decide if review will include quantitative analysis (meta analysis) 4.4 NA (only applies to meta analysis) Not needed to answer question No meta analyses completed (~27 meta analyses) 5/24/

64 Standards for Reporting Systematic Reviews 5.1 Prepare final report using structured format 5.2 Peer review draft report 5.3 Publish in manner that ensures free access 5/24/

65 5.1 Prepare a final report using a structured format Title Abstract Executive Summary (in future) Summary for lay public (in future) Introduction Methods Results Discussion Funding sources and COI 5/24/

66 5.2 Peer review draft report Tox Strategies prepared drafts Scientific Advisory Board reviewed manuscript two times prior to submission Peer Review by Journal reviewers 5/24/

67 5.3 Publish in manner that ensures free access Food and Chemical Toxicology Journal Article Open Access AHRQ SRSD will include data Protocols on Prospero 5/24/

68 Now that you have the background let s get to the details 5/24/