Observational Studies and PCOR: What are the right questions? Steven Goodman, MD, MHS, PhD Stanford University PCORI MC Steve.goodman@stanford.edu
A tale of two cities? "It was the best of times, it was the worst of times, it was the age of Clinical Trials, it was the age of Observational studies, it was the epoch of Data Mining, it was the epoch of Pre-specification, it was the season of Discovery, it was the season of Decision, it was the spring of Effectiveness, it was the winter of Harms, we had Truth before us, we had Lies before us, we were all going direct to Causality, we were all going direct the other way--in short, the period was the present period, and some of its noisiest funders and policy makers insisted on its being perceived, for good or for evil, with a superlative degree of scientific rigor."
These questions appear almost daily
If you're a runner, you might have noticed this surprising headline from the April 5 edition of the Guardian: "Brisk walk healthier than running scientists." Or maybe you saw this one, which ran in Health magazine the very same day: "Want to lose weight? Then run, don't walk: Study. Both articles described the work of a herpetologist-turned-statistician at the Lawrence Berkeley National Laboratory named Paul T. Williams, who, this month, achieved a feat that's exceedingly rare in mainstream science: He used exactly the same dataset to publish two opposing findings.
Foundational equations E=mc 2 e πi = -1
Foundational equation of Epidemiology Pr(Outcome X=x) = Pr(Outcome Set(X=x) ) In English The probability of an outcome when we observe a risk factor is equal to when we actively set the risk factor to the same value. (This equation does not specify how one sets the variable.)
5. How should FDA factor in different kinds of safety evidence in considering different kinds of regulatory actions? Committee charge 1. What are the ethical and informed consent issues that must be considered when designing RCTs s to evaluate safety risks? 2. What are the strengths and weaknesses of various approaches, including observational studies, including patient registries, metaanalyses, including patient level data meta-analyses, and randomized controlled trials, to generate evidence about safety questions? 3. Considering the speed, cost, and value of studies, what types of follow-up studies are appropriate to investigate different kinds of signals (detected pre-approval or postapproval) and in what temporal order? 4. Under what circumstances should head-to-head randomized clinical trials for safety be required?
How to decide between an OS and an RCT Size of and nature of signal. Size of effect needed to justify policy change Time urgency Other potential causes of outcome; studying intended or unintended effects? Quality and availability of data Transportability Consider analytic approach, not just design.
What should it do? Part 1 FUNCTIONS. the methodology committee shall work to develop and improve the science and methods of comparative clinical effectiveness research by developing and periodically updating the following: (i) Methodological standards for research. Such methodological standards shall provide specific criteria for internal validity, generalizability, feasibility, and timeliness of research and for health outcomes measures risk adjustment The process for developing and updating such standards shall include input from relevant experts, stakeholders, and decisionmakers, and shall provide opportunities for public comment. Such standards shall also include methods by which patient subpopulations can be accounted for and evaluated in different types of research
What should it do? Part 2 (ii) A translation table that is designed to provide guidance and act as a reference for the Board to determine research methods that are most likely to address each specific research question.
Translation Framework
Emerging principles 1 Keep the research question and the design separate. 2 Focus on clarifying tradeoffs. 3 Expect that several versions of a translation tool might be needed for different decision-makers and different research categories. 4 Place individual research studies in the context of a research program. 5 The choice of study design must take into account the state of the art of research methodology.
The role of observational research In both of these reports, the role and place of observational research in the world of therapeutics is central. There seems to be a strong need to apply the same, somewhat formulaic rules of evidence/design familiar from the EBM world to observational data. This comes in through creating rules or at least a social consensus about what constitutes legitimate designs for various questions.
RCT- Observational canards Trait RCTs OSs Bias Low High Resources: Time, $ High Low Causal inference High Low Transportability Low/Medium High Reporting biases Low/Medium High Real world: Interventions, outcomes, comparators Low High Patient centered Low High Data quality High Low Ethicality Low High Sample size Low/medium High
Methods canards New statistical methods solve the inferential problems of OSs. Large amounts of data can overcome the problems of poor data. Electronic health records make high quality observational research much easier. The more personal the data, the better the prediction. Better prediction means better outcomes.
What is true The distinctions between OSs and RCTs are can be subtle. Interventions are changing, with more need to evaluate complex interventions. We have not just more data, but different data types and more complex data. We do have new, improved methods, and many unproven. We have new outcomes we should or can measure. We different decisions to make about how to use new decision tools (e.g. diagnostics) and interventions. We have more emphasis on transparency, reproducibility and wider engagement in the scientific process.
Basis for meeting plan Effect in practice True effect in study setting Average Effect Measured by Study Design Generalizability External validity Bias Random Error Effect observed in actual study Subgroup effects
Central questions Does the method address the question we are really interested in? Does it estimate the effect correctly for those to whom the results are applied? If not, how wrong could it be? Does it estimate the uncertainty in the effect estimate correctly? If not, how wrong could it be?
Central questions Which of the approaches discussed today and tomorrow would be sound enough to guide a treatment decision? If not, could they get there and what would it take?
PCORI Decisions Which methodologies should be used and how? Investment in the development of which new methodologies will yield the greatest benefit? Measurement Design Analysis Data-generation and linkage Decision making What methods should the next generation of PCOR researchers be taught?