Meta-analysis: Methodology

Transcription

1 Meta-analysis: Methodology Example: Assessment of cardiovascular safety profile of new generation BCR-ABL TKIs in patients with CML Haguet Hélène 04/03/2016

2 Meta-analysis = statistical combination of results from two or more separate studies to answer a common question Compute effect size + variance for each study Assign weights based on study variance Compute the weighted mean Assign weight depending of the study precision Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 2

3 Design the protocol Rationale potential interest PRISMA protocols Objectives: define the question: PICO Participants (diseases, conditions) Interventions (treated arm) Comparators (control arm) Outcomes of interest Eligibility criteria: PICO + study design Search strategy Data collection Statistical analysis (model) Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Shamseer L, et al. Bmj. 2015;349:g

4 Random or fixed-effect model? Fixed-effect model Estimates a single effect that is assumed to be common to every study Random-effects model Allow that the true effect size may vary from study to study Observed variance = within-studies + between-studies variance both used in weights assignment Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 4

5 Random or fixed-effect model? Fixed-effect model Random-effects model Borenstein M, et al. 2010;1(2): Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 5

6 Databases Bibliographic databases Search strategy E.g. PubMed, Scopus, Cochrane Library, EMBASE Abstracts from international congresses Clinical trial registers E.g. WHO international clinical trial register, EudraCT Keywords and Boolean operators Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 6

7 Search strategy Example #1: ponatinib [Title] (69) #2: AP24534 [Title] (8) #3: ((#1) OR #2) (71) #4: (#3) and "randomized controlled trial" (1) #5: (#3) and "randomized trial" (0) #6: (#3) and "randomized clinical trial" (0) #7: (#3) and "randomised controlled trial" (0) #8: (#3) and "randomised trial" (0) #9: (#3) and "randomised clinical trial" (0) #10: (((((#4) OR #5) OR #6) OR #7) OR #8) OR #9 (1) Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 7

8 Study selection Removal of duplicates In 2 stages Screening of abstracts and titles independently by 2 reviewers (+ discussion with 3 rd reviewers) Selection of included studies based on the entirety of the paper Process described in flow diagram PRISMA statement Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 8

9 PRISMA flow chart Moher D, et al. Bmj. 2009;339:b2535. Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 9

10 Quality assessment Use quality scoring system By 2 reviewers independently Exclude low quality studies E.g JADAD score, Chalmers scale Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 10

11 Data collection Extracted by 2 independent reviewers Standardized data extraction form E.g. study characteristics, study design, population, outcomes Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 11

12 Data collection Selection of data by the 2 reviewers Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 12

13 Statistical analysis Direction and size of the effect? Effect size measure + 95%CI Is the effect consistent across studies? Heterogeneity assessment What is the strength of evidence for the effect? Quality assessment Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 13

14 Effect size measures = summarises the observed intervention effect Depend of the type of data Means (e.g. improvement of blood pressure) Binary data (e.g. survival) Correlational data Binary data Risk ratio Odds ratio (often the best choice) Risk difference Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 14

15 Statistical analysis Comprehensive Meta-Analysis software version Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 15

16 Heterogeneity assessment Cochran s Q statistics Reported with p value Low power to detect heterogeneity Common use of 0.10 as cut-off value for significance I 2 statistic I 2 : % of observed total variation across studies that is due to real heterogeneity rather than chance <25%: low heterogeneity 25-50%: moderate heterogeneity 50-75%: high heterogeneity Less dependent of the number of studies Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 16

17 Heterogeneity assessment No heterogeneity: similar to fixed-effect model In case of heterogeneity: Explore the causes: Subgroup analysis (!! False negative and positive) E.g. Treatments, population characteristics Meta-regression Check data and effect size measure Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 17

18 Assessing risk of bias Publication bias Funnel plots Assessment of the funnel plot asymmetry Egger s linear regression test Begg and Mazumbar rank correlation test Fail-safe number = how many new studies averaging a null result are required to bring the overall treatment effect to nonsignificant Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 18

19 One-way sensitivity analysis Aim: explore the robustness of the findings Remove one single study at a time Vascular occlusive events Study name Treatment Statistics with study removed Peto odds ratio (95% Lower Upper CI) with study removed Point limit limit p-value NCT EPIC Ponatinib 3,410 2,317 5,020 0,000 NCT BELA Bosutinib 3,443 2,382 4,977 0,000 NCT ENESTnd Nilotinib 3,518 2,140 5,785 0,000 NCT ENESTcmr Nilotinib 3,335 2,293 4,852 0,000 NCT ENESTchina Nilotinib 3,395 2,360 4,885 0,000 NCT LASOR Nilotinib 3,303 2,281 4,782 0,000 NCT NordCML006 Dasatinib 3,392 2,358 4,881 0,000 NCT Dasatinib 3,395 2,360 4,884 0,000 NCT DASISION Dasatinib 3,321 2,278 4,840 0,000 NCT START-R Dasatinib 3,404 2,363 4,903 0,000 NCT Dasatinib 3,524 2,449 5,070 0,000 NCT SPIRIT2 Dasatinib 3,645 2,465 5,390 0,000 3,418 2,379 4,909 0,000 0,01 0, Imatinib New generation TKI Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Douxfils J, et al. JAMA Oncol

20 Forest plot Douxfils J, et al. Journal of the American Heart Association. 2014;3(3):e Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 20

21 Guidelines: PRISMA checklist Moher D, et al. Bmj. 2009;339:b

22 Guidelines: PRISMA checklist Moher D, et al. Bmj. 2009;339:b

23 Criticisms 1 number cannot summarize a research field Assessment of heterogeneity and dispersion between studies Publication bias overestimation of the true effect size Assessed by funnel plot + asymmetry tests Mixing apples and oranges If heterogeneity between studies, it should be investigated Be clear and transparent 23

24 Resources Borenstein M, et al. Introduction to Meta-Analysis Cooper H, et al. The Handbook of Research Synthesis and Meta-Analysis, 2nd Edition Higgins J, et al, Cochrane Handbook for Systematic Reviews of Interventions 24

25

26 Vascular occlusive events Ponatinib: arterial and venous occlusive events identified 1 Temporal suspension of ponatinib marketing (FDA decision) Early abortion of the phase III clinical trial (EPIC trial) Risk minimization measures 2 1 Giles, F. J. et al. Leukemia 27(6): EMA. European Medicines Agency recommends changes in use of leukaemia medicine Iclusig (ponatinib) in order to minimise risk of blood clots 1

27 Vascular occlusive events Dasatinib: no vascular occlusive events 3 Bosutinib: no vascular occlusive events 4 Nilotinib: serious cases of PAOD 5,6 Cardiac and arterial occlusive events included in labeling information 3 Food and Drug Administration. Label information SPRYCEL. 4 Food and Drug Administration. Label information BOSULIF. 5 Food and Drug Administration. "Label information - TASIGNA." 6 Quintas-Cardama A., et al. Clin Lymphoma Myeloma Leuk 12(5):

28 Rationales TKIs may alter other tyrosine kinases class effect? 7 2nd generation TKIs have demonstrated higher efficacy on surrogate outcomes in the treatment of CML 7 Giles, F. J. et al. (2013). Leukemia 27(6):

29 Question Risk of CV occlusive events associated to new generation BCR-ABL TKIs in CML compared with imatinib? Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results 4

30 Systematic review and Metaanalysis Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Protocol: online registration: PROSPERO 2014: CDR Literature search: Scientific articles: Pubmed, scopus and Cochrane library Congress abstracts: ASH, ASCO, ESMO Clinical trial register: Study selection: Randomized clinical trials Comparing new generation TKIs vs imatinib Patients with CML 5

31 Systematic review and Metaanalysis Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Data collection: Vascular occlusive events (+ OS and MMR) Arterial occlusive events, venous occlusive events Study characteristics, population characteristics, JADAD scale Statistical analysis: Random-effects model Exception: fixed-effect model for venous occlusive events Effect size measure: Odds ratio using Peto method 6

32 Peto odds ratio Odds Ratio Peto Odds Ratio OR = AD BC Brockhaus AC, et al. Statistics in medicine. 2014;33(28):

33 Systematic review and Metaanalysis Problem formulation Literature search Study selection Data collection Statistical analysis Reporting the results Statistical analysis: Stratification By treatment Heterogeneity Q statistic I 2 value Publication bias: funnel plots Robustness: 1-way sensitivity analysis 8

34 Vascular occlusive events 8 Vascular occlusive events atment Group by Comparison Study name Statistics for each study Peto odds ratio and 95% CI Peto Lower Upper Relative odds ratio limit limit p-value weight sutinib Bosutinib NCT BELA 2,768 0,388 19,769 0, ,00 Bosutinib 2,768 0,388 19,769 0,310 atinib Dasatinib NCT NordCML006 8,092 0, ,341 0,296 3,30 atinib Dasatinib NCT ,389 0, ,385 0,317 3,31 atinib Dasatinib NCT DASISION 4,855 1,301 18,120 0,019 29,30 atinib Dasatinib NCT START-R 4,460 0,230 86,505 0,323 5,78 atinib Dasatinib NCT ,085 0,002 4,614 0,227 3,19 atinib Dasatinib NCT SPIRIT2 2,317 0,887 6,052 0,086 55,12 Dasatinib 2,913 1,428 5,942 0,003 otinib Nilotinib NCT ENESTnd 3,307 1,949 5,611 0,000 80,15 otinib Nilotinib NCT ENESTcmr 4,826 1,178 19,777 0,029 11,26 otinib Nilotinib NCT ENESTchina 7,334 0, ,606 0,319 1,46 otinib Nilotinib NCT LASOR 7,475 1,270 43,982 0,026 7,13 Nilotinib 3,700 2,305 5,940 0,000 atinib Ponatinib NCT EPIC 3,470 1,231 9,779 0, ,00 Ponatinib 3,470 1,231 9,779 0,019 Overall 3,418 2,379 4,909 0,000 0,01 0, Imatinib New generation TKI 8 Douxfils J, et al. JAMA Oncol

35 Vascular occlusive events 8 Vascular occlusive events atment Group by Comparison Study name Statistics for each study Peto odds ratio and 95% CI Peto Lower Upper Relative odds ratio limit limit p-value weight sutinib Bosutinib NCT BELA 2,768 0,388 19,769 0, ,00 Bosutinib 2,768 0,388 19,769 0,310 atinib Dasatinib NCT NordCML006 8,092 0, ,341 0,296 3,30 atinib Dasatinib NCT ,389 0, ,385 0,317 3,31 Bosutinib: 3 events/248 pts atinib Dasatinib NCT DASISION 4,855 1,301 18,120 0,019 29,30 atinib Dasatinib NCT START-R 4,460 0,230 86,505 0,323 Imatinib: 1 event/251 pts 5,78 atinib Dasatinib NCT ,085 0,002 4,614 0,227 3,19 atinib Dasatinib NCT SPIRIT2 2,317 0,887 6,052 0,086 55,12 Dasatinib 2,913 1,428 5,942 0,003 otinib Nilotinib NCT ENESTnd 3,307 1,949 5,611 0,000 80,15 otinib Nilotinib NCT ENESTcmr 4,826 1,178 19,777 0,029 11,26 otinib Nilotinib NCT ENESTchina 7,334 0, ,606 0,319 1,46 otinib Nilotinib NCT LASOR 7,475 1,270 43,982 0,026 7,13 Nilotinib 3,700 2,305 5,940 0,000 atinib Ponatinib NCT EPIC 3,470 1,231 9,779 0, ,00 Ponatinib 3,470 1,231 9,779 0,019 Overall 3,418 2,379 4,909 0,000 0,01 0, Imatinib New generation TKI 8 Douxfils J, et al. JAMA Oncol

36 Vascular occlusive events 8 Vascular occlusive events atment Group by Comparison Study name Statistics for each study Peto odds ratio and 95% CI Peto Lower Upper Relative odds ratio limit limit p-value weight sutinib Bosutinib NCT BELA 2,768 0,388 19,769 0, ,00 Bosutinib 2,768 0,388 19,769 0,310 atinib Dasatinib NCT NordCML006 8,092 0, ,341 0,296 3,30 atinib Dasatinib NCT ,389 0, ,385 0,317 3,31 atinib Dasatinib NCT DASISION 4,855 1,301 18,120 0,019 29,30 atinib Dasatinib NCT START-R 4,460 0,230 86,505 0,323 5,78 atinib Dasatinib NCT ,085 0,002 4,614 0,227 3,19 atinib Dasatinib NCT SPIRIT2 2,317 0,887 6,052 0,086 55,12 Dasatinib (overall) 2,913 1,428 5,942 0,003 otinib Nilotinib NCT ENESTnd 3,307 1,949 5,611 0,000 80,15 otinib Nilotinib NCT ENESTcmr 4,826 1,178 19,777 0,029 11,26 otinib Nilotinib NCT ENESTchina 7,334 0, ,606 0,319 1,46 otinib Nilotinib NCT LASOR 7,475 1,270 43,982 0,026 7,13 Nilotinib (overall) 3,700 2,305 5,940 0,000 atinib Ponatinib NCT EPIC 3,470 1,231 9,779 0, ,00 Ponatinib (overall) 3,470 1,231 9,779 0,019 Overall 3,418 2,379 4,909 0,000 0,01 0, Imatinib New generation TKI 8 Douxfils J, et al. JAMA Oncol

37 Limitations Lack of individual data Time to event Number of deaths due to VOE Lack of homogeneity in evaluation criteria (VOE) between studies 8,9 8 Yang EH, et al. Future Oncol. 11(14): Groarke JD, et al. The New England journal of medicine. 369(19):

38 Strengths Consistent with the signal Robustness confirmed by the sensitivity analysis No heterogeneity between studies No evidence of publication bias Published & unpublished studies included 14

39 Standard Error Funnel plot Funnel Plot of Standard Error by Log Peto odds ratio Log Peto odds ratio 15

40 6th NTHC symposium 18

41 Acknowledgments Douxfils Jonathan Mullier François Chatelain Christian Graux Carlos Dogné Jean-Michel 42

42 Thank you for your attention!

43 PRISMA protocol checklist Shamseer L, et al. Bmj. 2015;349:g7647.

44 PRISMA protocol checklist Shamseer L, et al. Bmj. 2015;349:g7647.

45 FEM vs REM Both: weights depend of the precision of the estimation (= of the overall study error variance) Difference REM vs FEM: definition of this variance Fixed-effect model Only 1 source of variation: the estimation error ε i = difference between common true mean and observed mean Within-study variance depends of: - The variance of individual observation - The size of the sample Random-effects model 2 sources of variation - The estimation error within study ε i - The estimation error between study ξ i Overall study error variance: combination of the variance of these 2 parameters Borenstein M, et al. 2010;1(2):

46 FEM vs REM Fixed-effect model Random-effect model ε 1 ε 1 ξ 1 Adapted from Borenstein M, et al. 2010;1(2): Adapted from Borenstein M, et al. 2010;1(2): Circle: true mean Square: observed mean (differs from the true mean because of estimation error)

47 Raw mean difference Effect size: means Only if studies used the same scale Standardized mean difference In case of different evaluation of the outcome (the scale of measurement differed) Division of the mean difference in each study by study s standard deviation Response ratios When the measure is unlikely to be 0, but has a natural 0 point (e.g. length)

48 Odds ratio vs risk ratio Divergence large when the outcome is common Avoid quantitative statements about OR Holcomb WL, Jr., et al. Obstetrics and gynecology. 2001;98(4):685-8.

49 Meta-analysis of observational studies Controversial: potential biases Diversity of study designs and populations Publication bias (could have particular impact) Recommendations: Use broad inclusion criteria Perform analysis relating suspected source of bias and variability Investigate heterogeneity Stroup DF, et al. JAMA. 2000;283(15):

50 Meta-analysis of single-arm clinical trials Controversial: lack of control: effect of site-specific variables Interrupted-time series (ITS) study For studies with multiple time-points before and after an intervention (at least 3 data points before and 3 after the intervention) Repeated measures study: if the measures are repeated in the same individuals Effective Practice and Organisation of Care (EPOC) Available at: 51

51 Effective Practice and Organisation of Care (EPOC) Available at: 52