Meta-analysis: Methodology - PDF Free Download

Meta-analysis: Methodology Example: Assessment of cardiovascular safety profile of new generation BCR-ABL TKIs in patients with CML Haguet Hélène 04/03/2016

Meta-analysis = statistical combination of results from two or more separate studies to answer a common question Compute effect size + variance for each study Assign weights based on study variance Compute the weighted mean Assign weight depending of the study precision Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 2

Design the protocol Rationale potential interest PRISMA protocols Objectives: define the question: PICO Participants (diseases, conditions) Interventions (treated arm) Comparators (control arm) Outcomes of interest Eligibility criteria: PICO + study design Search strategy Data collection Statistical analysis (model) Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Shamseer L, et al. Bmj. 2015;349:g7647. 3

Random or fixed-effect model? Fixed-effect model Estimates a single effect that is assumed to be common to every study Random-effects model Allow that the true effect size may vary from study to study Observed variance = within-studies + between-studies variance both used in weights assignment Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 4

Random or fixed-effect model? Fixed-effect model Random-effects model Borenstein M, et al. 2010;1(2):97-111. Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 5

Databases Bibliographic databases Search strategy E.g. PubMed, Scopus, Cochrane Library, EMBASE Abstracts from international congresses Clinical trial registers E.g. www.clinicaltrials.gov, WHO international clinical trial register, EudraCT Keywords and Boolean operators Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 6

Search strategy Example #1: ponatinib [Title] (69) #2: AP24534 [Title] (8) #3: ((#1) OR #2) (71) #4: (#3) and "randomized controlled trial" (1) #5: (#3) and "randomized trial" (0) #6: (#3) and "randomized clinical trial" (0) #7: (#3) and "randomised controlled trial" (0) #8: (#3) and "randomised trial" (0) #9: (#3) and "randomised clinical trial" (0) #10: (((((#4) OR #5) OR #6) OR #7) OR #8) OR #9 (1) Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 7

Study selection Removal of duplicates In 2 stages Screening of abstracts and titles independently by 2 reviewers (+ discussion with 3 rd reviewers) Selection of included studies based on the entirety of the paper Process described in flow diagram PRISMA statement Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 8

PRISMA flow chart Moher D, et al. Bmj. 2009;339:b2535. Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 9

Quality assessment Use quality scoring system By 2 reviewers independently Exclude low quality studies E.g JADAD score, Chalmers scale Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 10

Data collection Extracted by 2 independent reviewers Standardized data extraction form E.g. study characteristics, study design, population, outcomes Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 11

Data collection Selection of data by the 2 reviewers Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 12

Statistical analysis Direction and size of the effect? Effect size measure + 95%CI Is the effect consistent across studies? Heterogeneity assessment What is the strength of evidence for the effect? Quality assessment Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 13

Effect size measures = summarises the observed intervention effect Depend of the type of data Means (e.g. improvement of blood pressure) Binary data (e.g. survival) Correlational data Binary data Risk ratio Odds ratio (often the best choice) Risk difference Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 14

Statistical analysis Comprehensive Meta-Analysis software version 2.2.046 Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 15

Heterogeneity assessment Cochran s Q statistics Reported with p value Low power to detect heterogeneity Common use of 0.10 as cut-off value for significance I 2 statistic I 2 : % of observed total variation across studies that is due to real heterogeneity rather than chance <25%: low heterogeneity 25-50%: moderate heterogeneity 50-75%: high heterogeneity Less dependent of the number of studies Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 16

Heterogeneity assessment No heterogeneity: similar to fixed-effect model In case of heterogeneity: Explore the causes: Subgroup analysis (!! False negative and positive) E.g. Treatments, population characteristics Meta-regression Check data and effect size measure Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 17

Assessing risk of bias Publication bias Funnel plots Assessment of the funnel plot asymmetry Egger s linear regression test Begg and Mazumbar rank correlation test Fail-safe number = how many new studies averaging a null result are required to bring the overall treatment effect to nonsignificant Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 18

One-way sensitivity analysis Aim: explore the robustness of the findings Remove one single study at a time Vascular occlusive events Study name Treatment Statistics with study removed Peto odds ratio (95% Lower Upper CI) with study removed Point limit limit p-value NCT01650805 EPIC Ponatinib 3,410 2,317 5,020 0,000 NCT00574873 BELA Bosutinib 3,443 2,382 4,977 0,000 NCT00471497 ENESTnd Nilotinib 3,518 2,140 5,785 0,000 NCT00760877 ENESTcmr Nilotinib 3,335 2,293 4,852 0,000 NCT01275196 ENESTchina Nilotinib 3,395 2,360 4,885 0,000 NCT00802841 LASOR Nilotinib 3,303 2,281 4,782 0,000 NCT00852566 NordCML006 Dasatinib 3,392 2,358 4,881 0,000 NCT00070499 Dasatinib 3,395 2,360 4,884 0,000 NCT00481247 DASISION Dasatinib 3,321 2,278 4,840 0,000 NCT00103844 START-R Dasatinib 3,404 2,363 4,903 0,000 NCT00320190 Dasatinib 3,524 2,449 5,070 0,000 NCT01460693 SPIRIT2 Dasatinib 3,645 2,465 5,390 0,000 3,418 2,379 4,909 0,000 0,01 0,1 1 10 100 Imatinib New generation TKI Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Douxfils J, et al. JAMA Oncol. 2016. 19

Forest plot Douxfils J, et al. Journal of the American Heart Association. 2014;3(3):e000515. Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 20

Guidelines: PRISMA checklist Moher D, et al. Bmj. 2009;339:b2535. 21

Guidelines: PRISMA checklist Moher D, et al. Bmj. 2009;339:b2535. 22

Criticisms 1 number cannot summarize a research field Assessment of heterogeneity and dispersion between studies Publication bias overestimation of the true effect size Assessed by funnel plot + asymmetry tests Mixing apples and oranges If heterogeneity between studies, it should be investigated Be clear and transparent 23

Resources Borenstein M, et al. Introduction to Meta-Analysis 2009. Cooper H, et al. The Handbook of Research Synthesis and Meta-Analysis, 2nd Edition 2009. Higgins J, et al, Cochrane Handbook for Systematic Reviews of Interventions 24

Vascular occlusive events Ponatinib: arterial and venous occlusive events identified 1 Temporal suspension of ponatinib marketing (FDA decision) Early abortion of the phase III clinical trial (EPIC trial) Risk minimization measures 2 1 Giles, F. J. et al. Leukemia 27(6): 1310-1315. 2 EMA. European Medicines Agency recommends changes in use of leukaemia medicine Iclusig (ponatinib) in order to minimise risk of blood clots 1

Vascular occlusive events Dasatinib: no vascular occlusive events 3 Bosutinib: no vascular occlusive events 4 Nilotinib: serious cases of PAOD 5,6 Cardiac and arterial occlusive events included in labeling information 3 Food and Drug Administration. Label information SPRYCEL. 4 Food and Drug Administration. Label information BOSULIF. 5 Food and Drug Administration. "Label information - TASIGNA." 6 Quintas-Cardama A., et al. Clin Lymphoma Myeloma Leuk 12(5): 337-340. 2

Rationales TKIs may alter other tyrosine kinases class effect? 7 2nd generation TKIs have demonstrated higher efficacy on surrogate outcomes in the treatment of CML 7 Giles, F. J. et al. (2013). Leukemia 27(6): 1310-1315. 3

Question Risk of CV occlusive events associated to new generation BCR-ABL TKIs in CML compared with imatinib? Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 4

Systematic review and Metaanalysis Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Protocol: online registration: PROSPERO 2014: CDR42014014147 Literature search: Scientific articles: Pubmed, scopus and Cochrane library Congress abstracts: ASH, ASCO, ESMO Clinical trial register: www.clinicaltrials.gov Study selection: Randomized clinical trials Comparing new generation TKIs vs imatinib Patients with CML 5

Systematic review and Metaanalysis Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Data collection: Vascular occlusive events (+ OS and MMR) Arterial occlusive events, venous occlusive events Study characteristics, population characteristics, JADAD scale Statistical analysis: Random-effects model Exception: fixed-effect model for venous occlusive events Effect size measure: Odds ratio using Peto method 6

Peto odds ratio Odds Ratio Peto Odds Ratio OR = AD BC Brockhaus AC, et al. Statistics in medicine. 2014;33(28):4861-74. 7

Systematic review and Metaanalysis Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Statistical analysis: Stratification By treatment Heterogeneity Q statistic I 2 value Publication bias: funnel plots Robustness: 1-way sensitivity analysis 8

Vascular occlusive events 8 Vascular occlusive events atment Group by Comparison Study name Statistics for each study Peto odds ratio and 95% CI Peto Lower Upper Relative odds ratio limit limit p-value weight sutinib Bosutinib NCT00574873 BELA 2,768 0,388 19,769 0,310 100,00 Bosutinib 2,768 0,388 19,769 0,310 atinib Dasatinib NCT00852566 NordCML006 8,092 0,160 409,341 0,296 3,30 atinib Dasatinib NCT00070499 7,389 0,147 372,385 0,317 3,31 atinib Dasatinib NCT00481247 DASISION 4,855 1,301 18,120 0,019 29,30 atinib Dasatinib NCT00103844 START-R 4,460 0,230 86,505 0,323 5,78 atinib Dasatinib NCT00320190 0,085 0,002 4,614 0,227 3,19 atinib Dasatinib NCT01460693 SPIRIT2 2,317 0,887 6,052 0,086 55,12 Dasatinib 2,913 1,428 5,942 0,003 otinib Nilotinib NCT00471497 ENESTnd 3,307 1,949 5,611 0,000 80,15 otinib Nilotinib NCT00760877 ENESTcmr 4,826 1,178 19,777 0,029 11,26 otinib Nilotinib NCT01275196 ENESTchina 7,334 0,146 369,606 0,319 1,46 otinib Nilotinib NCT00802841 LASOR 7,475 1,270 43,982 0,026 7,13 Nilotinib 3,700 2,305 5,940 0,000 atinib Ponatinib NCT01650805 EPIC 3,470 1,231 9,779 0,019 100,00 Ponatinib 3,470 1,231 9,779 0,019 Overall 3,418 2,379 4,909 0,000 0,01 0,1 1 10 100 Imatinib New generation TKI 8 Douxfils J, et al. JAMA Oncol. 2016. 10

Vascular occlusive events 8 Vascular occlusive events atment Group by Comparison Study name Statistics for each study Peto odds ratio and 95% CI Peto Lower Upper Relative odds ratio limit limit p-value weight sutinib Bosutinib NCT00574873 BELA 2,768 0,388 19,769 0,310 100,00 Bosutinib 2,768 0,388 19,769 0,310 atinib Dasatinib NCT00852566 NordCML006 8,092 0,160 409,341 0,296 3,30 atinib Dasatinib NCT00070499 7,389 0,147 372,385 0,317 3,31 Bosutinib: 3 events/248 pts atinib Dasatinib NCT00481247 DASISION 4,855 1,301 18,120 0,019 29,30 atinib Dasatinib NCT00103844 START-R 4,460 0,230 86,505 0,323 Imatinib: 1 event/251 pts 5,78 atinib Dasatinib NCT00320190 0,085 0,002 4,614 0,227 3,19 atinib Dasatinib NCT01460693 SPIRIT2 2,317 0,887 6,052 0,086 55,12 Dasatinib 2,913 1,428 5,942 0,003 otinib Nilotinib NCT00471497 ENESTnd 3,307 1,949 5,611 0,000 80,15 otinib Nilotinib NCT00760877 ENESTcmr 4,826 1,178 19,777 0,029 11,26 otinib Nilotinib NCT01275196 ENESTchina 7,334 0,146 369,606 0,319 1,46 otinib Nilotinib NCT00802841 LASOR 7,475 1,270 43,982 0,026 7,13 Nilotinib 3,700 2,305 5,940 0,000 atinib Ponatinib NCT01650805 EPIC 3,470 1,231 9,779 0,019 100,00 Ponatinib 3,470 1,231 9,779 0,019 Overall 3,418 2,379 4,909 0,000 0,01 0,1 1 10 100 Imatinib New generation TKI 8 Douxfils J, et al. JAMA Oncol. 2016. 10

Vascular occlusive events 8 Vascular occlusive events atment Group by Comparison Study name Statistics for each study Peto odds ratio and 95% CI Peto Lower Upper Relative odds ratio limit limit p-value weight sutinib Bosutinib NCT00574873 BELA 2,768 0,388 19,769 0,310 100,00 Bosutinib 2,768 0,388 19,769 0,310 atinib Dasatinib NCT00852566 NordCML006 8,092 0,160 409,341 0,296 3,30 atinib Dasatinib NCT00070499 7,389 0,147 372,385 0,317 3,31 atinib Dasatinib NCT00481247 DASISION 4,855 1,301 18,120 0,019 29,30 atinib Dasatinib NCT00103844 START-R 4,460 0,230 86,505 0,323 5,78 atinib Dasatinib NCT00320190 0,085 0,002 4,614 0,227 3,19 atinib Dasatinib NCT01460693 SPIRIT2 2,317 0,887 6,052 0,086 55,12 Dasatinib (overall) 2,913 1,428 5,942 0,003 otinib Nilotinib NCT00471497 ENESTnd 3,307 1,949 5,611 0,000 80,15 otinib Nilotinib NCT00760877 ENESTcmr 4,826 1,178 19,777 0,029 11,26 otinib Nilotinib NCT01275196 ENESTchina 7,334 0,146 369,606 0,319 1,46 otinib Nilotinib NCT00802841 LASOR 7,475 1,270 43,982 0,026 7,13 Nilotinib (overall) 3,700 2,305 5,940 0,000 atinib Ponatinib NCT01650805 EPIC 3,470 1,231 9,779 0,019 100,00 Ponatinib (overall) 3,470 1,231 9,779 0,019 Overall 3,418 2,379 4,909 0,000 0,01 0,1 1 10 100 Imatinib New generation TKI 8 Douxfils J, et al. JAMA Oncol. 2016. 10

Limitations Lack of individual data Time to event Number of deaths due to VOE Lack of homogeneity in evaluation criteria (VOE) between studies 8,9 8 Yang EH, et al. Future Oncol. 11(14):1995-8. 9 Groarke JD, et al. The New England journal of medicine. 369(19):1779-81. 13

Strengths Consistent with the signal Robustness confirmed by the sensitivity analysis No heterogeneity between studies No evidence of publication bias Published & unpublished studies included 14

Standard Error Funnel plot Funnel Plot of Standard Error by Log Peto odds ratio 0 1 2 3-3 -2-1 0 1 2 3 Log Peto odds ratio 15

6th NTHC symposium 18

Acknowledgments Douxfils Jonathan Mullier François Chatelain Christian Graux Carlos Dogné Jean-Michel 42

Thank you for your attention!

PRISMA protocol checklist Shamseer L, et al. Bmj. 2015;349:g7647.

FEM vs REM Both: weights depend of the precision of the estimation (= of the overall study error variance) Difference REM vs FEM: definition of this variance Fixed-effect model Only 1 source of variation: the estimation error ε i = difference between common true mean and observed mean Within-study variance depends of: - The variance of individual observation - The size of the sample Random-effects model 2 sources of variation - The estimation error within study ε i - The estimation error between study ξ i Overall study error variance: combination of the variance of these 2 parameters Borenstein M, et al. 2010;1(2):97-111.

FEM vs REM Fixed-effect model Random-effect model ε 1 ε 1 ξ 1 Adapted from Borenstein M, et al. 2010;1(2):97-111 Adapted from Borenstein M, et al. 2010;1(2):97-111 Circle: true mean Square: observed mean (differs from the true mean because of estimation error)

Raw mean difference Effect size: means Only if studies used the same scale Standardized mean difference In case of different evaluation of the outcome (the scale of measurement differed) Division of the mean difference in each study by study s standard deviation Response ratios When the measure is unlikely to be 0, but has a natural 0 point (e.g. length)

Odds ratio vs risk ratio Divergence large when the outcome is common Avoid quantitative statements about OR Holcomb WL, Jr., et al. Obstetrics and gynecology. 2001;98(4):685-8.

Meta-analysis of observational studies Controversial: potential biases Diversity of study designs and populations Publication bias (could have particular impact) Recommendations: Use broad inclusion criteria Perform analysis relating suspected source of bias and variability Investigate heterogeneity Stroup DF, et al. JAMA. 2000;283(15):2008-12. 50

Meta-analysis of single-arm clinical trials Controversial: lack of control: effect of site-specific variables Interrupted-time series (ITS) study For studies with multiple time-points before and after an intervention (at least 3 data points before and 3 after the intervention) Repeated measures study: if the measures are repeated in the same individuals Effective Practice and Organisation of Care (EPOC). 2013. Available at: http://epoc.cochrane.org/epoc-specific-resources-review-authors 51

Effective Practice and Organisation of Care (EPOC). 2013. Available at: http://epoc.cochrane.org/epoc-specific-resources-review-authors 52