Pros and Cons of Clinical Trials vs. Observational Studies Beth Devine PCORP Summer Institute July 14, 2015
Outline Randomized Trials Definitions: Pragmatic-explanatory continuum indicator summary (PRECIS) Examples Observational Studies Definitions and types of observational data Advantages and disadvantages of observational data research Good practices in observational data research Examples
Learning Objectives Define the RCT continuum Describe appropriate use of pragmatic vs. explanatory trials List and define the major types of observational studies Describe uses, advantages and disadvantages of the major types of observational studies Locate good research practice tools for use when conducting observational studies
Part I: RCTs
Benefit of RCTs Randomization is the ONLY way to guarantee unbiasedness, particularly as it relates to unknown or unrecorded prognostic factors!
Pragmatic-Explanatory Continuum Pragmatic describes trials that help users choose between options for care Explanatory describes trials that test causal research hypotheses Both are randomized Represents a spectrum Impossible to perform a purely pragmatic or purely explanatory trial Reflects judgments made by trialists in study design phase Thorpe. CMAJ. 2009;188(10):E47-E57
PRECIS Tool 10 extreme domains Explanatory Pragmatic Participants Restricted Take all comers Interventions Strict instructions Flexible instructions Seasoned practitioners/settings Full range of practitioners/settings Comparator Restricted (placebo) Usual practice Standardized Ordinary attention Follow-up Frequent/extensive No formal F/U; registries Compliance (participant) Direct/immediate/ surrogate Closely monitored/followed Objectively measured; Assessed under usual conditions Unobtrusive Adherence (provider) Closely monitored Unobtrusive Analysis Intent to treat All patients Thorpe. CMAJ. 2009;188(10):E47-E57
PRECIS Tool Thorpe. CMAJ. 2009;188(10):E47-E57
PRECIS Tool - Examples Thorpe. CMAJ. 2009;188(10):E47-E57
Part II: Observational Studies
Observational Studies Subject not randomized Treatments/exposure delivered in natural settings Cohort studies identify exposure; then outcome(s) Retrospective cross-sectional or longitudinal All data collected before commencement of study Prospective typically longitudinal Consequential outcomes data collected after commencement of study Case-control studies identify outcome; then exposure(s) Always retrospective - longitudinal
Advantages of observational studies Useful for characterizing a population Useful in CER/PCOR Avoids voluntary participation Generalizable to target population; often not the case in RCTs Faster and cheaper Data collected as part of larger surveillance goals Usually does not require expensive protocol
Disadvantages of observational studies Study design Treatments, exclusion/inclusion criteria/follow-up period etc. determined by the data at hand Outcomes All relevant outcomes may not be available Causal Inference eliminating bias Must deal with confounders Requires use of more advanced statistical techniques
A word about validity RCTs often produce internally valid estimates, but may not be externally valid (generalizable) Internal validity is NECESSARY but NOT SUFFICIENT for external validity. Observational studies cannot provide externally valid estimates if they are not internally valid.
Good Practice Recommendations CER (with observational data) only relevant when there is clinical equipoise In presence of strong treatment preferences it is difficult to control for confounding or bias Retrospective data are most useful here Specify hypothesis, up front Specify population, comparators, outcomes of interest
Good Practice Recommendations Specify study design Strongest design always includes a control group Cohort pre/post Good to assess one or more outcomes Can assess one or more exposures Case-control Good to assess rare outcome (usually one) Can assess many exposures Case-Crossover Designs Individuals serve as their own controls Case-Time-Control Designs Case-crossover design with external control group to control for temporal trends
Good Practice Recommendations Two types of bias Observed can address by including covariates or stratification Unobserved requires advanced techniques Address presence of treatment effect heterogeneity Issues in estimation sample size etc. Issues in interpretability of results level of aggregation Issues in generalizability what does the mean effect tell us What are the moderators of treatment effect mostly use baseline characteristics
Good Practice Recommendations Observational data invaluable source of data for CER Clear descriptions of hypothesis, study design and methods are important for a good observational CER study Confounding is main issue Often advanced statistical methods are needed to address confounding
ISPE Guidelines for Good Pharmacoepidemiology Practices (GPP). https://www.pharmacoepi.org/resources/guidelines_08027.cfm References Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014. http://www.effectivehealthcare.ahrq.gov/registriesguide-3.cfm. Strengthening the Reporting of Observational studies in Epidemiology (STROBE) guideline. http://www.strobe-statement.org/ REporting of studies Conducted using Observational Routinely-collected Data (RECORD). http://record-statement.org/ ISPOR, Good Pharmacoepidemiology Practices, 2008. ISPOR Good Research Practices for Observational Data (many) http://www.ispor.org/workpaper/practices_index.asp European Network of Centers for Pharmacoepidemiology and Pharmacovigilance (ENCePP). http://www.encepp.eu/ Luce BR et al. Principles for planning and conducting comparative effectiveness research. Journal of Comparative Effectiveness Research 2012; 1(5): 431-440.
Case Study I: Prospective Cohort CER Study of Intermittent Claudication (IC): Impact of Intervention Type on Patient Function and Health-Related Quality of Life Devine Alfonso-Cristancho Yanez, Edwards Patrick, Armstrong Devlin, Symons, Thomason, Meissner, Clowes, Lavallee, Kessler, Flum, and CERTAIN Collaborative Funded by AHRQ R01HS020025 (PI: Flum)
Patient Voices Clinician Offices Hospitals Long-term Care Facilities Clinical Practice Partners Evidence Generation Dissemination & Implementation
IC Study Methods Study Design: Multisite, longitudinal, prospective, observational cohort study conducted from 2011-2013 Aims: Compare baseline, 6 & 12 month functional, health-related quality of life and symptoms among subjects receiving medical management vs. surgical or endovascular procedures for treatment of intermittent claudication Hypothesis: At 12-months, surgical and endovascular procedures are associated with greater improvements in function, health-related quality of life, and symptoms than the medical management cohort
Lessons Learned from the IC Study Patient reported outcome (PRO) measures can be used as primary and secondary outcomes Recruitment efforts are often Intense Always adjust analyses for baseline characteristics Data collected from both electronic health records and directly from patients provides the opportunity to compare patient reported outcomes to clinically reported outcomes Engage a biostatistician to assist with study design, power calculations and analyses Infrastructure is expensive to build initially; once built, additional studies can be conducted for modest incremental investment
Case Study II: Retrospective Cohort Estimating the costs of atrial fibrillation and associated adverse events Forrester SH, Li M, Roth G, Devine EB
Atrial fibrillation (AF) study methods Study Design: Matched (1:4), retrospective cohort study of patients 18 years old with incident AF(ICD-9 427.31) between 2008 and 2010 Aim: Estimate the incremental costs of events (ischemic stroke, myocardial infarction, systemic embolism, intracranial hemorrhage, or GI bleed) in patients with AF
Lessons Learned from the A fib Study Time invested in developing study design and protocol, a priori, is a must When using administrative claims data, clearly define run-in period; date of index diagnosis; date of intervention; define adequate follow-up period Illustrations are helpful in refining these design characteristics draw it out! Control for baseline characteristics and potential confounders Engage a biostatistician to assist with study design, power calculations and analyses
Thank You! Questions? bdevine@uw.edu