Evidence-Based Medicine and Publication Bias Desmond Thompson Merck & Co.

Evidence-Based Medicine and Publication Bias Desmond Thompson Merck & Co.

Meta-Analysis Defined A meta-analysis is: the statistical combination of two or more separate studies In other words: overview, pooling, data synthesis, and quantitative review

Why Perform A Meta- Analysis? Meta-analysis of multiple trials should enhance the precision of estimates of treatment effect from individual clinical studies Greater estimate of precision may: accelerate changes in treatment adoption avoid unnecessary or harmful treatments eliminate the need for further clinical trials Meta-analysis can be used to examine or establish a research hypothesis

Types of Meta-Analysis Traditional (aggregate data) metaanalysis Cumulative meta-analysis Individual patient meta-analysis

Meta-Analysis: Effect Size vs. Direction Anticipated Unexpected Meta-analyses are predicated on the assumption that effects are more likely to differ in size than in direction Line of equivalence

Features of the Forest Plot Weight (%) 5 30 5 7 28 22 3 Each study is represented by a point estimate and specified confidence interval (CI) Size of point estimate is proportional to the contribution each study makes to the metaanalysis. This is indicated by the weight Smaller studies tend to have wider CIs but this is not important in meta-analysis what matters is the consistency of the trend between studies Meta-analysis effect size estimate (including CI) appears as a diamond at the bottom of the plot. This is the grand mean Line of equivalence

Principles of Evidence-Based Medicine First principle There is a hierarchy of evidence Some evidence is stronger than others Use evidence from the top of the hierarchy (or as high as possible) for each clinical decision Second principle Evidence itself is never enough to make clinical decisions or recommendations Always use evidence and value judgment Guyatt et al 1999

Hierarchy of Evidence Meta-analysis of RCTs systematic review of RCTs Individual RCT Observational studies patient-important outcomes Basic research test tube, animal, human physiology Clinical experience nonsystematic clinical observation

Key Question What is the process of developing a clinical recommendation that follows from the principles of evidence-based medicine? Define the question Gather the evidence Summarize the evidence Make a judgment on the evidence Guyatt et al 1999

Gathering the Evidence Identifying and selecting evidence explicit and sensible process Systematic overview appropriate inclusion and exclusion criteria comprehensive search databases, citations, unpublished data reproducible assessments Hayward et al 1995

Summarizing the Evidence Quantitative summary Meta-analysis Relative risk reduction Absolute risk reduction Quality Summary Completeness of studies

Why Meta-Analysis? Meta-analysis of multiple trials provides more reliable indications of drug efficacy than studying individual trials separately (Guyatt et al., 1999) Meta-analysis provides guidance as to the approximate effect of treatment in patients studied in the trials and more importantly to the likely effect of treatment in future patients outside of the trials (Peto, 1987) Meta-analysis provides a logical framework to a research review (Dickersin & Berlin, 1992)

Well done Meta-Analyses are critical Components that facilitate the Practice of EBM Did the overview address a focused clinical question? Were the selection criteria used to select studies for inclusion appropriate? Is it likely that important relevant studies were missed? Publication bias Studies where this outcome was secondary endpoint Subgroups from large studies

Meta-analysis: The Key Assumption It is reasonable to assume that if trials address related questions then there there will be a tendency for the answers to come out in the same direction. It is this tendency that we are trying to detect in an overview.

The missing data problem Systematic reviews lose their credibility when done with missing data Missing at random Selection bias Poor quality and hence not published Deliberate: don t like the results Studies not accessible Positive studies are more likely than to be published Use of composite endpoints

Possible Effect of Publication Bias Reported Missing Studies Overviews of trials are based on the assumption that there was completeness of information

Meta-analysis: Corrected for Missing

Identifying Bias in Meta-Analysis (1) The Funnel Plot A plot of effect size versus study size Large studies are at the top of the plot (because effect estimate is usually relatively small) Small studies are at the bottom of the plot (because effect estimate is usually relatively large) If there is no bias, the plot is symmetrical If there is bias, the plot is asymmetrical The type of asymmetry may indicate the type of bias

Some Funnel Plot Profiles Publication bias Methodology bias Publication deficit in this area Effect size Effect size

Correcting for Publication Bias in Meta-Analysis What can be done to correct for publication bias in meta-analysis? Not very much! Correction of treatment effect estimates for bias should be avoided as such corrections may depend heavily on the assumptions made. Sterne JAC et al. BMJ 2001;323:101-5.

Completeness of Data A complete clinical trial registry will provide a path to ensure completeness of information Overviews are usually post hoc Affords us the opportunity of prespecification Meta-analysis is only as strong as the weakest link

Sensitivity Analysis Sensitivity analysis is a method for determining whether a particular study or factor has overly influenced the results of a meta-analysis Systematically omit data one study at a time to see whether the omission of that study has a substantial effect on the overall metaanalysis results Apply a range of reasonable assumptions about the value of the factor of interest to see whether the meta-analysis results are robust to variations in the value of that factor

Sensitivity Analysis: Is The Overall Treatment Effect Seen Consistently In Every Study? Grand mean of 4 studies 29% (p = 0.012) Study 1 omitted 18 0.34 Study 2 omitted 31 0.03 Study 3 omitted 34 0.002 Study 4 omitted 30 0.02 0.3 0.5 0.8 1 2 Relative risk

Sensitivity Analysis: Is The Overall Treatment Effect Seen Consistently In Every Study? Grand mean of 5 studies 35% (p = 0.005) Study 1 omitted 39 0.02 Study 2 omitted Study 3 omitted Study 4 omitted Study 5 omitted 31 0.03 34 0.01 30 0.02 27 0.02 0.3 0.5 0.8 1 2 Relative risk

An Essential Principle of EBM All data are important! Not all will contribute directly to the summary Need for completeness to make recommendation useful

Consequences of Publication Bias Can affect answers to key questions Are the results of the overview valid? What are the results? Will the results help in caring for patients? Guyatt et al 1993

Will the results help in Evidence- Based Clinical Care? Use of evidence in decision-making Allows identification of best evidence Estimate of magnitude of treatment effect Quantify the impact on individual patients Evidence to action Evidence-based medicine facilitates explicit acknowledgment of patient values in decision-making Facilitates choice of style and philosophy of patientdoctor relationship

Summary The big picture is what matters To appreciate the big picture you need an overview An overview is best achieved from metaanalysis of all relevant RCTs Failure to obtain all the evidence is likely to result in a biased selection Peto: Stat Med 1987; 6:233-234