8/6/15. Homework. Homework. Lecture 8: Systematic Reviews and Meta-analysis

Transcription

1 Lecture 8: Systematic Reviews and Meta-analysis Christopher S. Hollenbeak, PhD Jane R. Schubart, PhD The Outcomes Research Toolbox Homework Stata Example: stset surv_days, failure(died) sts test age_cat sts graph, by(age_cat) title("age") xtitle("days Since Liver Transplant") ytitle("percent Surviving") text( "P = ") stcox age4049 age5059 age60 female black ab1 ab2 ab3 ab4 obese rbc ssi Homework Percent 0.50 Surviving Age P = Days Since Liver Transplant Percent 0.50 Surviving Sex P = Days Since Liver Transplant female = 0 female = Percent 0.50 Surviving Surgical Site Infection P = Days Since Liver Transplant ssi = 0 ssi = 1 1

2 Homework No. of subjects = 753 Number of obs = 753 No. of failures = 143 Time at risk = LR chi2(12) = Log likelihood = Prob > chi2 = _t Haz. Ratio Std. Err. z P> z [95% Conf. Interval] age age age female black ab ab ab ab obese rbc ssi Homework R Example: sv2 <- Surv(dat1$surv_days, dat1$died) ~ dat1$ssi survdiff(sv2) plot(survfit(sv2), xlab="days since Treatment", ylab="percent Surviving", lty=c(1,2), col=c("black","grey75"), lwd=1, cex=2, mark.time=false) legend(2000, 1, c("ssi","no SSI"), lty=c(1,2), col=c("black", "grey75"), bty="n") text(2350,.68, "P = ) Homework Age Sex Surgical Site Infection Percent Surviving P = Percent Surviving Male Female P = 0.39 Percent Surviving SSI No SSI P = Days since Treatment Days since Treatment Days since Treatment 2

3 Homework coef exp(coef) se(coef) z Pr(> z ) dat1$age4049. dat1$age ** dat1$age *** dat1$female dat1$black dat1$ab dat1$ab dat1$ab dat1$ab dat1$ssi * --- Signif. codes: 0 *** ** 0.01 * exp(coef) exp(-coef) lower.95 upper.95 dat1$age dat1$age dat1$age dat1$female dat1$black dat1$ab dat1$ab dat1$ab dat1$ab dat1$ssi Concordance= 0.62 (se = ) Rsquare= (max possible= ) Likelihood ratio test= on 10 df, p= Wald test = on 10 df, p= Score (logrank) test = on 10 df, p= Introduction A systematic review identifies previous studies of a research question and attempts to arrive at an overall conclusion about the body of evidence The statistical methods used in a systematic review are called meta-analysis Cornerstone of evidence based medicine and comparative effectiveness 8 Example Bahekar et al., Am Heart J 2007;154:

4 Overview There are four elements of a well-executed metaanalysis 1. Identify studies with potentially relevant data 2. Impose eligibility restrictions on the studies 3. Abstract the data 4. Analyze and summarize the data 10 Identify Relevant Studies The most important part of the meta-analysis is the way that potential studies are identified The method must be systematic and explicit 11 Example DuPont and Page (1991) Interested in the relationship between menopausal estrogen replacement therapy and breast cancer Used MEDLINE to identify 556 articles 35 of these publications provided an estimate of breast cancer risk in women who took estrogen replacement therapy The reference lists of these 35 publications and those in a review article led to identification of 15 more publications 50 articles were included in the study 12 4

5 Search Algorithm Search personal files Computerized search of databases Eliminate from information in titles and abstracts Review remaining articles Review bibliographies of articles and review articles Consult experts in the field 13 Publication Bias Publication bias is the phenomenon that statistically significant results tend to be published Funnel Plot is a visual tool for detecting publication bias 14 Funnel Plots This area is relatively empty, suggesting publication bias for small trials Sample Size Sample Size Odds Ratio Odds Ratio 15 5

6 Publication Bias What to do? Try to get data from unpublished studies Abstracts from conference presentations There are statistical methods to deal with publication bias but they are beyond the realm of this course 16 Restricting the Literature Not all studies should be included in the analysis Eligibility criteria should be explicitly defined prior to the literature search Criteria should be based on sound reasoning, not convenience 17 Eligibility Considerations 1. Study designs 2. Years of publication 3. Languages 4. Choice among multiple publications 5. Sample size 6. Follow-up time 7. Similarity of treatment or exposure 8. Completeness of information 18 6

7 Example Yusuf et al. (1985) studied beta blockers after AMI Included all beta blockers Odds ratio for death was 0.77 (CI: ) OR for beta blockers with intrinsic sympathomimetic activity was 0.90 (CI: ) OR for beta blockers without intrinsic sympathomimetic activity was 0.69 (CI: ) Choice to include all studies biased the result toward the null 19 Abstracting Data Develop abstraction forms prior to starting the study Develop a written protocol for abstraction Provide training to abstractors Assess inter-abstractor reliability Revise forms and protocol 20 Statistical Methods Goal is to Compute an overall measure of effect Ratio measures (Odds, risk, rate ratios) Continuous measures Then assess the variability of the overall effect Usually a 95% confidence interval 21 7

8 Statistical Models First choose a statistical model Fixed Effects model Random Effects model 22 Fixed Effects Model The fixed effect model assumes all studies are independently drawn from the same population y i = + i i N(0, 2 ) Only source of variation in between studies 23 Fixed Effects Model 24 8

9 Random Effects Model The random effects model assumes each study is drawn from its own distribution, with its own mean and variance y i = i + i i N(, 2 ) i N(0, Captures both within and between study variation 2 ) 25 Random Effects Model 26 Dichotomous Effects Data must be 2x2 table data Fixed effects Mantel-Haenszel Any risk measure: odds ratio, risk ratio, risk difference Peto Odds ratio only, events must be common Random effects DerSimonian-Laird 27 9

10 Continuous Effects Fixed effects General Variance Based Method Random effects Random Effects Model 28 Fixed or Random Effects Methods based on both fixed and random effects are available for both types of effects measure The choice can have a significant effect on the results In most cases, the random effects model is more conservative and is preferred Tradeoff is that these methods are more difficult to implement 29 Fixed or Random Effects If the true model is fixed effects, then fixed and random give the correct answer If the true model is random effects then only the random effects model is correct Always safer to go with random effects 30 10

11 Heterogeneity The choice between fixed and random effects models is based on a statistical test for heterogeneity If the variance differs between studies, then there is heterogeneity Stata and R both provide a Woolf test for heterogeneity If heterogeneity is insignificant fixed effects model If heterogeneity is significant random effects model Inverse Variance Weighting The overall effect estimated by either the fixed or random effects model is a weighted average of the individual effects Weight is simply the inverse of the variance Variance term differs for fixed and random effects models 32 Example 33 11

12 Graphical Presentation A Forest Plot is the standard display for metaanalysis results The goal is to convey an impression of the overall body of evidence Important features Effect size is shown 95% confidence intervals are shown Overall effect estimate is shown Reference to the null or overall effect is shown 34 Study Study 1 Study 2 Study 3 Study 4 Study 5 Study 6 Study 7 Study 8 Study 9 Pooled Favors Treatment Favors Control Odds Ratio 35 Problems Larger error bars draw visual attention Larger error bars suggest weaker information Could modify the symbols to reflect the amount of information Sized proportional to their weights 36 12

13 Study Study 1 Study 2 Study 3 Study 4 Study 5 Study 6 Study 7 Study 8 Study 9 Pooled Favors Treatment Favors Control Odds Ratio 37 Conventional and cumulative meta-analysis of 33 trials of intravenous streptokinase for acute myocardial infarction. Mulrow C D BMJ 1994;309: by British Medical Journal Publishing Group Stata Code We use a user written package called metan to do meta-analysis net search metan Look for sbe24_3 Click Install Package Syntax is: 2x2 table: metan a b c d, options Other: metan effect se_effect, options Important options: or rr fixed random peto 13

14 Stata Example Consider the BCG meta-analysis Meta-analysis of clinical trials for a vaccine for tuberculosis Studies included randomized trials that reported 2x2 table data Stata Example Perform a Mantel-Haenzel meta-analysis of the trials metan treat_disease treat_no_disease control_disease control_no_disease Stata Example Study RR [95% Conf. Interval] % Weight M-H pooled RR Heterogeneity chi-squared = (d.f. = 12) p = I-squared (variation in RR attributable to heterogeneity) = 92.1% Test of RR=1 : z= p =

15 Stata Example Study ID RR (95% CI) % Weight Overall (I-squared = 92.1%, p = 0.000) 0.41 (0.13, 1.26) 0.20 (0.09, 0.49) 0.26 (0.07, 0.92) 0.24 (0.18, 0.31) 0.80 (0.52, 1.25) 0.46 (0.39, 0.54) 0.20 (0.08, 0.50) 1.01 (0.89, 1.14) 0.63 (0.39, 1.00) 0.25 (0.15, 0.43) 0.71 (0.57, 0.89) 1.56 (0.37, 6.53) 0.98 (0.58, 1.66) 0.64 (0.59, 0.69) Stata Example Is there significant heterogeneity? Solution: Use random effects model instead Also, report Odds Ratios instead of Risk Ratios metan treat_disease treat_no_disease control_disease control_no_disease, random or Stata Example Study OR [95% Conf. Interval] % Weight D+L pooled OR Heterogeneity chi-squared = (d.f. = 12) p = I-squared (variation in OR attributable to heterogeneity) = 92.7% Estimate of between-study variance Tau-squared = Test of OR=1 : z= 3.88 p =

16 Stata Example Study ID OR (95% CI) % Weight Overall (I-squared = 92.7%, p = 0.000) 0.39 (0.12, 1.26) 0.19 (0.08, 0.46) 0.25 (0.07, 0.91) 0.23 (0.18, 0.31) 0.80 (0.51, 1.26) 0.38 (0.32, 0.47) 0.20 (0.08, 0.50) 1.01 (0.89, 1.15) 0.62 (0.39, 1.00) 0.25 (0.14, 0.42) 0.71 (0.57, 0.89) 1.56 (0.37, 6.55) 0.98 (0.58, 1.66) 0.47 (0.32, 0.69) NOTE: Weights are from random effects analysis R Code To perform meta-analysis in R we use a package called rmeta install.packages( rmeta ) library(rmeta) IMPORTANT: rmeta wants the data in a different format than Stata! Stata wants cell counts rmeta wants event counts and totals R Code metan format rmeta format Create new variables: ma$treat_tot <- ma$treat_disease + ma$treat_no_disease ma$control_tot <- ma$control_disease + ma$control_no_diseas 16

17 R Code Three steps: Step 1: Create a meta-analysis object, meta.mh() or meta.dsl() Step 2: Summarize the object, summary() Step 3: Plot the object, plot() Step 1: Syntax is Fixed Effects: meta.mh(n.trt, n.ctrl, col.trt, col.ctrl, names=studyname, statistic=[ OR or RR ]) Random Effects: meta.dsl(n.trt, n.ctrl, col.trt, col.ctrl, names=studyname, statistic=[ OR or RR ]) R Results ma1 <- meta.mh(ma$treat_tot, ma$control_tot, ma$treat_disease, ma$control_disease, names=ma$study, statistic="or") summary(ma1) Fixed effects ( Mantel-Haenszel ) meta-analysis OR (lower 95% upper) Mantel-Haenszel OR = % CI ( 0.57,0.68 ) Test for heterogeneity: X^2( 12 ) = ( p-value 0 ) R Code plot(ma1) Study Reference Summary Odds Ratio 17

18 R Results ma2 <- meta.dsl(ma$treat_tot, ma$control_tot, ma$treat_disease, ma$control_disease, names=ma$study, statistic= OR") summary(ma2) Random effects ( DerSimonian-Laird ) meta-analysis OR (lower 95% upper) SummaryOR= % CI ( 0.32,0.69 ) Test for heterogeneity: X^2( 12 ) = ( p-value 0 ) Estimated random effects variance: 0.37 R Code plot(ma2) Study Reference Summary Odds Ratio Forest Plot Options The forest plot can be customized using the meta.colors() options Summary symbol color: summary="color" 95% CI line color: lines="color" Study outcome symbol color: box="black" Null hypothesis line color: zero="red" Color of text: text="white" Background color of graph: background="gray80" Color of axes: axes="white" 18

19 Forest Plot Options plot(ma1, colors=meta.colors(summary="gray50", lines="gray50", box="black", zero="red", text="white", background="thistle, axes="white")) Study Reference Summary Odds Ratio PRISMA Guidelines PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses An evidence-based standard for reporting in systematic reviews and meta-analyses PRISMA consists of: A 27-item checklist of items to include in your write-up A four-phase flow diagram to include as a figure in your writeup The standard change periodically so consult the website: PRISMA Checklist The PRISMA Checklist contains 27 issues to specifically address in your write-up Many are not specific to systematic reviews Describe the rationale for the review in the context of what is already known. Duh. Also suggests the level of detail needed in identifying information sources and conducting searches Highly recommended that you follow these guidelines 19

20 PRISMA Flow Di agram Four stages: Identification Screening Eligibility Inclusion You should create this diagram for your study and include it in your paper Reviewers are becoming savvy to PRISMA and will make you do it PRISMA Flow Diagram Included Eligibility Screening Identification # of records identified through # of additional records identified database searching through other sources # of records after duplicates removed # of records screened # of records excluded # of full text articles # of full text articles assessed for eligibility excluded, with reasons # of studies included in qualitative synthesis # of studies included in quantitative synthesis (meta analysis) Conclusions Systematic reviews are an important part of the comparative effectiveness toolkit Summarize what is known about the effectiveness of interventions Provide inputs in into decision analysis models and costeffectiveness models Meta-analysis is the set of statistical methods used to conduct a systematic review Goal is to summarize the body of knowledge surrounding a clinical question Not a shortcut to research 60 20

21 Homework There is a data set on the website (blockers.csv) 22 clinical trials of beta blockers after AMI Four variables: rt: number of patients who in the treatment group who died nt: total number of patients in the treatment group rc: number of patients in the control group who died nc: total number of patients in the control group 1. Perform a Mantel-Haenzel meta-analysis What is the pooled odds ratio? Is it significant? What is the pooled risk ratio? Is it significant? 61 Homework 2. Produce forest plots for both the odds ratio and the risk ratio 3. Is there significant heterogeneity? 4. Perform a Dersimonian-Laird random effects meta-analysis for the odds ratio How does the pooled odds ratio compare to the fixed effects model? 5. If you are using Stata, perform a Peto analysis 21