Quality and Reporting Characteristics of Network Meta-analyses: A Scoping Review Andrea C. Tricco MSc, PhD Scientist, Li Ka Shing Knowledge Institute of St. Michael s Hospital Assistant Professor, Dalla Lana School of Public Health, University of Toronto CADTH Symposium 2016
Conflict of interest statement: I have no actual or potential conflict of interest in relation to this presentation. Funding: This research was funded by the Canadian Agency for Drugs and Technologies in Health.
Definition of Network Meta-analysis (NMA) NMA is an extension of indirect comparisons that allows the combination of direct with indirect comparisons, and also the simultaneous analysis of the comparative effects of many interventions Cochrane Handbook, http://handbook.cochrane.org/ 3
Network meta-analysis (NMA) 4
Steps for a systematic review in a NMA Start with a systematic review and NMA protocol Conduct a comprehensive literature search Use pre-defined eligibility criteria, same types of data abstracted Use the same risk of bias appraisal as recommended for reviews Synthesize totality of evidence and report using PRISMA-NMA Discuss strengths and limitations Conduct each step by 2 reviewers, independently Hutton et al., Ann Int Med, 2015 5
Methodological considerations Broader PICOS than systematic reviews with a metaanalysis Literature search must include all interventions/comparators = Expect to include many more studies! Decisions on how to lump and split treatments into nodes (e.g., dosing, administration, drug class versus specific drug analysis) = A lot of clinical guidance! 6
Statistical considerations Network geometry Key assumptions: 1. Homogeneity similar considerations as metaanalysis 2. Similarity (transitivity) effect modifiers 3. Consistency agreement between direct and indirect evidence 7
Assumptions underlying indirect comparisons and NMA Assumption for indirect and mixed comparison Conceptual definition (Transitivity) Property of parameters and data (Consistency) Clinical Methodological Statistical Cipriani et al Ann of Int Medicine 2013
Transitivity Interpretation of transitivity: 1. Treatment A is similar when it appears in AB and AC trials 2. The two sets of trials AB and AC do not differ with respect to the distribution of effect modifiers 3. Participants included in the network could in principle be randomized to any of the 3 treatments A, B, C 4. Missing treatment in each trial is missing at random 5. There are no differences between observed and unobserved relative effects of AB and AC beyond what can be explained by heterogeneity A B C Salanti Res Synth Methods 2012
Consistency Direct and indirect evidence are in agreement B B A Consistency is a property of a closed loop (path that starts and ends at the same node) C C 10
Heterogeneity The heterogeneity variance can be estimated using several approaches E.g., DerSimonian and Laird (DL), Maximum Likelihood (ML), Restricted Maximum Likelihood (REML) In a Bayesian framework several priors can be assigned E.g., Informative, Minimally informative, Vague Several heterogeneity assumptions can be considered 1. Common within-network heterogeneity: all treatment comparisons in the network are associated with the same magnitude of heterogeneity 2. Comparison-specific heterogeneity: each treatment comparison in the network has its own amount of heterogeneity 3. Common within-loop heterogeneity: treatment comparisons included in each closed loop in the network share the same amount of heterogeneity 11
Objective To explore the quality and reporting characteristics of all published NMAs through a scoping review 12
Methods A priori protocol compiled using PRISMA-P Methods guided by Joanna Briggs Institute Methods Manual MEDLINE, EMBASE, PubMed and Cochrane searched from inception to April 14, 2015 without any language or publication status restrictions Included NMAs with 4 treatments, randomized trials, and using a valid statistical analysis Pairs of reviewers independently performed screening of citations and full-text articles; abstracted data; and assessed quality of included studies (AMSTAR and ISPOR) 13
Included Eligibility Screening Identification Results study flow Additional records identified through other sources (n = 253) Citations identified through database searching (n = 3727) Citations after duplicates removed (n = 3538) Citations screened (n = 3538) Full-text articles assessed for eligibility (n = 878) 625) Studies included in qualitative synthesis (n = 496) Records excluded (n = 2913) Not an NMA (n=2601) NMA includes observational studies (n=61) Includes <4 treatments (n=216) Commentary/Protocol (n=35) Full-text excluded (n = 382) Unable to locate (n = 3) Duplicate (n=16) (Commentary/Protocol (n = 91) Invalid NMA (n = 98) NMA includes observational studies (n = 21) Includes <4 treatments (n = 70) No. of treatments > no. of studies (n = 76) Companion report (n = 7)
Results geographic trends North America 31% Africa 0.6% Europe 52% Asia 13% Central & South America 1.2% Australia 1.4% Multiple regions: 0.4% Geographic distribution of publications by region (n=496)
Results temporal distribution 25.0% 23.8% 20.0% 18.3% 15.0% 10.0% 11.7% 12.5% 10.5% 6.3% 6.7% 5.0% 0.0% 2.8% 2.4% 2.4% 1.0% 0.2% 0.2% 0.4% 0.2% 0.2% 0.4% 1997 1999 2000 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015* *2015 sample is not complete Temporal distribution of publications (n=496)
Results frequency of terms NMA terminology (n=496) Count (%) Network meta-analysis 232 (38.9%) Mixed treatment comparisons 122 (20.5%) Indirect comparisons 57 (9.6%) Word Cloud of NMA terminology based on frequency (n=496)
Results cited methodology Most Commonly Cited Papers Count (%) Lu & Ades 2004 149 (30%) Caldwell 2005 82 (17%) Bucher 1997 71 (14%) Word cloud of cited methodology paper (n=496)
Results journal disciplines Journal Discipline (n= 496) Count (%) Medicine, General & Internal 121 (23.8%) Health Care Sciences & Services 56 (11.0%) Pharmacology & Pharmacy 35 (6.9%) Cardiac & Cardiovascular Systems 29 (5.7%) Endocrinology & Metabolism 25 (4.9%) Oncology 21 (4.1%) Multidisciplinary Sciences 19 (3.7%) Rheumatology 17 (3.3%) Clinical Neurology 15 (2.9%) Gastroenterology & Hepatology 15 (2.9%)
Results NMA characteristics NMA Characteristics (n=496) Count (%) Publicly-sponsored 186 (38%) Industry-sponsored 103 (21%) Funding Non-sponsored 106 (21%) Industry and publicly sponsored 9 (2%) Funding source not reported 92 (19%) Article Type Application paper 438 (88%) Application paper (no review method) 18 (4%) Development paper 40 (8%) Review Size (n=438)* median (Q1, Q3) 25 (14, 48) *Review size reported for application papers with full review methods only
A priori protocol and review design Review Type Results review design Method Characteristics (n=438) Count (%) Systematic review 356 (81.3%) Knowledge Synthesis Overview of reviews 7 (1.6%) Name Not reported 75 (17.1%) <6 months 55 (12.6%) Review Duration 6-12 months 132 (30.1%) >12-24 months 106 (24.2%) >24 months 81 (18.5%) Not reported 64 (14.6%) Use of a protocol mentioned 66 (15.1%) A Priori Protocol Published 40 (9.1%) Registered 25 (5.7%) Available upon request 6 (1.4%) Not reported 301 (68.7%) Research Question Clearly Reported 437 (99.8%) Unclear/inferred 1 (0.2%) Eligibility Criteria Clearly reported 430 (98.2%) Unclear/inferred 1 (0.2%) Not reported 7 (1.6%)
Identifying relevant studies Results literature search Method Characteristics (n=438) Count (%) Search String Complete literature search 207 (47.3%) Medical subject headings only 173 (39.5%) Not reported 58 (13.2%) Databases Searched Searched >1 database 407 (92.9%) Searched only 1 database 29 (6.6%) Not reported 2 (0.5%) Additional Search Strategy Limits Applied Scanned references 309 (70.5%) Grey literature searched 270 (61.6%) Consulted topic experts 80 (18.3%) Consulted librarian 67 (15.3%) Performed updated search 62 (14.2%) Manually searched select journals 37 (8.4%) Limited by date 135 (30.8%) Limited by language 147 (33.6%) Limited by study design 291 (66.4%) Other limits (e.g., age, humans) 129 (29.5%)
Data Abstraction & Quality Assessment Study Selection Results screening, abstraction, quality Method Characteristics (n=438) Count (%) Title & Abstract Screening Reported 402 (91.8%) Not reported 36 (8.2%) Full-text Screening Reported 405 (92.5%) Not reported 33 (7.5%) Completely in PRISMA-like flow diagram 374 (85.4%) Study flow Completely in text/table only 20 (4.6%) Partially reported 15 (3.4%) Not reported 29 (6.6%) Data Abstraction Reported 414 (94.5%) Not reported 24 (5.5%) Quality Appraisal Reported 346 (79.0%) Not reported 92 (21.0%)
Quality Assessment Data Abstraction Full-text Screening Title & Abstract Screening Results conduct of review process 2 independent reviewers 1 reviewer & 1 verifier 1 reviewer only Not done Done but unclear # of reviewers Not reported 2 independent reviewers 1 reviewer & 1 verifier 1 reviewer only Not done Done but unclear # of reviewers Not reported 2 independent reviewers 1 reviewer & 1 verifier 1 reviewer only Not done Done but unclear # of reviewers Not reported 2 independent reviewers 1 reviewer & 1 verifier 1 reviewer only Not done Done but unclear # of reviewers Not reported 2% 4% 0.5% 3% 2% 0.2% 2% 0.0% 2% 0.5% 5% 5% 8% 7% 17% Short-cuts taken (6%) 21% Inadequate reporting (29%) Short-cuts taken (4%) 24% Inadequate reporting (31%) 54% 21% Short-cuts taken (23%) Inadequate reporting (22%) 41% Short-cuts taken (7%) 30% Inadequate reporting (51%) 21% 65% 64% Method characteristics conduct of review process (n=438) 0% 10% 20% 30% 40% 50% 60% 70%
Results streamlined methods NMAs that leveraged previous systematic review(s) Count (%) Leveraged previous systematic reviews (n=456)* Yes 78 (17%) No 378 (83%) Leverage Process (n= 78) Updated literature search of previous systematic review(s) 37 (47%) Used literature database of previous systematic review(s) 20 (26%) Updated and expanded literature search of previous systematic review(s) 9 (12%) Cochrane update 6 (8%) Used database of previous systematic review(s) 3 (4%) Expanded literature search of previous systematic review(s) 2 (3%) Updated literature search of previous systematic review(s) and used data from previous review *Includes all application papers 1 (1%)
Results AMSTAR components A priori design 31% 69% Duplicate screening and data extraction 58% 39% 3% Comprehensive literature search 84% 0.5% 15% Grey Literature 62% 33% 5% List of includes / excludes 18% 1% 82% Study characteristics 83% 13% 5% Quality Appraisal 62% 21% 17% Quality of evidence incorporated in conclusions 48% 1% 29% 21% Appropriate pooling methods 86% 12% 2% Publication bias assessed 36% 0.5% 63% Conflict of interest declared 16% 84% 0% 20% 40% 60% 80% 100% YES UNCLEAR NO NOT APPLICABLE Review methodological quality AMSTAR (n=438)
Score Distribution (%) Results AMSTAR scores overall 20% Moderate: 57% 16% 17% 15% median (IQR)= 6 (4,7) 12% Low: 18% 13% 12% 11% High: 25% 8% 8% 7% 7% 4% 4% 3% 1% 1% 0% 0 1 2 3 4 5 6 7 8 9 10 11 Low Quality Moderate Quality High Quality AMSTAR Score Review methodological quality AMSTAR overall quality (n=438)
Results AMSTAR scores over time 100% 25% 14% 10% 19% 7% 13% 19% 10% 11% 21% 80% 38% 50% 60% 40% 100% 100% 100% 25% 64% 50% 90% 50% 64% 62% 59% 71% 79% 74% 20% 0% 50% 50% 31% 29% 21% 25% 22% 19% 13% 10% 4% 1999 2000 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Year of Publication Low (0-3) Moderate (4-7) High (8-11) Review methodological quality vs. publication year (n=438)
Results ISPOR components Consistency Assessed 54% 20% 27% Direct & indirect evidence combined 48% 21% 31% Accounted for inconsistency 31% 41% 1% 27% Valid rationale for FE or RE model 81% 19% Heterogeneity assumption explored 12% 73% 14% Accounted for heterogeneity 46% 20% 24% 10% 0% 20% 40% 60% 80% 100% YES UNCLEAR NO Not Applicable NMA methodological quality ISPOR (n=456)
Summary Sub-optimal quality of reporting was observed PRISMA-NMA may lead to improvements in reporting over time Less than a quarter of the published NMAs were considered high quality (AMSTAR 8), however, this improved over time Authors of published NMAs used streamlined methods: 23% streamlined data abstraction, 17% leveraged previous reviews About 20% did not assess consistency, provide rationale for FE/RE model or account for heterogeneity ISPOR may lead to improvements in review/nma conduct
Acknowledgements Co-investigators on funding proposal: Dr. Sharon Straus, Dr. Areti Angeliki Veroniki Research team: Wasifa Zarin Vera Nincic Afshin Vafaei Dr. Shannon Sullivan Dr. Emily Reynen Sanober Motiwala Maria Petropoulou Myrsini Gianiasti Jesmin Antony Caitlin Daly Joycelyne Ewusie Dr. Adriani Nikolakopoulou Dr. Anna Chaimani Dr. Georgia Salanti CIHR (new investigator award, MAGIC team grant) 31
Questions? Email: triccoa@smh.ca Office Location: 209 Victoria Street, 7th floor, Room 721, East building, Toronto, ON Telephone: 416.864.6060 x77521 32