The Second David Sackett Symposium THE RANDOMIZED TRIAL AS ROYALTY

Transcription

1 The Second David Sackett Symposium THE RANDOMIZED TRIAL AS ROYALTY John Concato, MD, MS, MPH Yale University School of Medicine VA Clinical Epidemiology Research Center 27 September 2013

2 DISCLOSURE John Concato has no potential for conflict of interest with this presentation

3 OBJECTIVES Understand specific strengths and limitations of observational studies and randomized trials Appreciate need for methodological and clinical judgment when evaluating patient-oriented research

4 Oxford Dictionary: DEFINITIONS OF ROYAL people of royal blood or status; member of a royal family the status or power of a king or queen the most successful, famous, or highly regarded members of a particular group

5 RCT AS KING OF STUDY DESIGN Grades of evidence for study design: I: At least one properly randomized, controlled trial (RCT) II-1: Well-designed trials without randomization II-2: Well-designed cohort or case-control analytic studies, preferably multi-site II-3: Multiple time series with or without the intervention, or dramatic results in uncontrolled experiments (e.g., penicillin or insulin) III: Opinions of respected authorities, based on clinical experience; descriptive studies, case reports, expert committee reports (US Preventive Services Task Force 1996)

6 STUDY DESIGN IN BROADER CONTEXT Validity study architecture: randomize, adjust for confounding, etc. source of data: primary, secondary, pooled, etc. Note: observational studies don t always use 2 o data Generalizability intended spectrum based on inclusion/exclusion criteria potential impact of participation itself (e.g., Hawthorne effect) Note: observational studies have inherent advantage

7 EMPHASIS RE: STUDY DESIGN Architecture Strengths Limitations Randomized trials a a Observational studies a a

8 LIMITATIONS OF RCTs Randomized trials on same topic often contradictory e.g., Horwitz Am J Med 1987;82:498 to be expected, but discordance is often overlooked Meta-analyses and large/simple RCTs often don t agree e.g., LeLorier New Engl J Med 1997;337:536 under-appreciated; attributed partly to heterogeneity RCTs can have restricted generalizability e.g., Coca JAMA 2006;296:1377 acknowledged, but implications are often minimized

9 RCT GENERALIZABILITY (cont d) Coca, et al., JAMA 2006;296:1377 (cont d) [Cardiovascular RCTs] do not provide adequate information on the effect of interventions on patients with renal disease Juurlink, et al., New Engl J Med 2004;351:543 Publication of [spironolactone RCT] was associated with abrupt increases in hyperkalemia-associated morbidity and mortality. Luce, et al., Ann Intern Med 2009;151:206 many RCTs are ill suited to meet the evidentiary needs implicit in the [Inst of Med] definition of [comp effectiveness research]

10 SPECTRUM OF OBSERVATIONAL STUDIES Indirect cause-and-effect evidence: ecological study (unit of observation is >1 patient/person) case series (no comparison group) historical control trial (non-concurrent comparison group) Direct cause-and-effect evidence: observational cohort study (with concurrent comparison) case-control study (including nested within cohort)

11 STRENGTHS OF OBSERVATIONAL STUDIES Strategies can promote validity of observational methods e.g., Horwitz Am J Med 1990;89:630 (including RCT-like restricted cohort, specific zero-time for intervention) Non-randomized treatments are effective/safe e.g., Tsimberidou J Clin Oncol 2009;27:6243 (including bleomycin, cisplatin, etoposide for testicular cancer) Observational studies and RCTs often agree head-to-head e.g., Concato New Engl J Med 2000;342:1887 ( welldesigned observational studies do not systematically overestimate [results when compared with RCTs] )

12 OBSERV STUDIES & RCTs AGREE (cont d) McKee, et al., BMJ 1999;319:312 one method does not give a consistently greater effect Benson and Hartz, New Engl J Med 2000;342:1887 [Observational results are] neither consistently larger than nor qualitatively different from those obtained in RCTs Ligthelm, et al., Clin Ther 2007;29:1284 observational studies can be conducted using the same exacting and rigorous standards as are used for RCTs

13 PREDICTION RE: RANDOMIZED TRIALS Alvan R. Feinstein

14 PROBLEM OF CONFOUNDING Confounding factors threaten validity of observ. study, whereas randomization tends to balance such factors

15 HOW TO DEAL WITH CONFOUNDING Several available options: conduct only randomized, controlled trials (per EBM) use methods that substitute for randomization (e.g., propensity scores, instrumental variables) use traditional observational methods rigorously Current emphasis is problematic: expectation of requiring RCT or complex model observational studies held in low regard

16 VARIABLE IMPACT OF CONFOUNDING Status of confounding factors implications for study: known, not accounted for major methodological weakness known & accounted for sound research practice unknown clinicians would not treat preferentially, reducing potential for bias Note: extent of problem varies (as with limitations of RCTs)

17 EVIDENCE-BASED MEDICINE (EBM) APPROACH Validity: whether results are true for patients enrolled in the trial or observational study Generalizability: whether the results generalize to a broader population of patients with condition Validity is considered to be of primary importance; generalizability is considered subsequently

18 BEYOND RANDOMISED VS. OBSERVATIONAL STUDIES Accuracy: whether results are true for patients who would receive exposure, intervention, etc.; Consider validity and generalizability together (Concato & Horwitz, Lancet 2004;363:1660)

19 MEDICINE-BASED EVIDENCE (MBE) In a medicine-based evidence approach, the primary emphasis is on clinically relevant issues of who and where were the patients, what and why were the treatments, & when and how were the outcomes measured as well as an assessment of validity and generalizability considered together and denoted as accuracy. (Concato JAMA 2012; 307:1641)

20 EXAMPLE #1: HORMONE REPLACEMENT THERAPY Research question: Does hormone replacement therapy (HRT) for post-menopausal women improve health outcomes? Observational studies: showed benefit of HRT regarding cardiovascular outcomes RCTs: showed early cardiovascular harm from HRT Conventional wisdom: RCTs are correct, observational studies are flawed; problem is inherent to type of research design

21 EXAMPLE #1: RCT/OBS STUDIES FOR HRT Relative risks or odds ratios from: Outcome RCTs Observational Colorectal ca 0.63 ( ) 0.66 ( ) Hip fracture 0.66 ( ) 0.75 ( ) Stroke 1.41 ( ) 1.45 ( ) Pulm emb 2.13 ( ) 2.1 ( ) (Grodstein New Engl J Med 2003;348:645)

22 EXAMPLE #1: RCT/OBS STUDIES FOR HRT Relative risks or odds ratios from: Outcome RCTs Observational Colorectal ca 0.63 ( ) 0.66 ( ) Hip fracture 0.66 ( ) 0.75 ( ) Stroke 1.41 ( ) 1.45 ( ) Pulm emb 2.13 ( ) 2.1 ( ) Coronary dz 1.29 ( ) 0.61 ( ) (Grodstein New Engl J Med 2003;348:645)

23 EXAMPLE #1: MEDICINE-BASED EVIDENCE Hormone replacement & coronary disease: Patients: Exposure: Outcome: time since menopause socioeconomic status dose of drug(s) duration of treatment timing of treatment duration of follow-up capture of early events? equal detection of MIs?

24 HRT & CV OUTCOMES: OBSERVATIONAL STUDIES Not adjusted for socioeconomic status: Adjusted for socioeconomic status: (Von Elm BMJ 2004;329:869)

25 TIMING OF HRT VS. ONSET OF MENOPAUSE For randomised trials, [the start of HRT] is the natural analysis because therapy starts at randomisation the first years of hormone replacement by combined oestrogen-progestin did increase coronary heart disease, which then waned. Most current users [in observational studies] were past the window wherein coronary heart disease risk was increased when data from the observational part of the [WHI trial] were reanalysed according to time since start of therapy, the same pattern emerged of an initial increase in risk, followed by a decrease. (Vandenbroucke Lancet 2009;373:1233)

26 EXAMPLE #1: IMPLICATIONS Laine Ann Intern Med 2002;137:290 Thus, rather than HRT keeping women healthy, healthy women were taking HRT. Vandenbroucke Lancet 2009;373:1233 Thus nothing was intrinsically wrong with the observational data; what went wrong was an analysis that had not taken into account that the effect of HRT might be different over time Neither design [RCT nor observational] help superior truth.

27 EXAMPLE #2: SCREENING FOR PROSTATE CANCER Observational studies: did not confirm benefit of screening with prostate-specific antigen (PSA), but were dismissed as inferior evidence RCTs: Two major RCTs were launched 20 years ago Superficial assessment: await pronouncement from RCTs, because observational studies are flawed Recent history: results from the trials eagerly anticipated as of 2009, yet RCTs on same topic often don t agree

28 EXAMPLE #2: RCT EXCLUSIVITY United States Preventive Services Task Force (USPSTF) 2008 recommendations on screening with PSA: Although the 2002 USPSTF review considered case-control studies and ecological data related to this key question, we excluded these study types from this part of the evidence update to avoid potential sources of confounding that are inherent in nonrandomized studies. (USPSTF Ann Intern Med 2008;149:192)

29 [King] Perspective when writing: ongoing RCTs may not be definitive. Statement (in Discussion): we should recognize that the [King s pronouncement] (i.e., results of the RCTs) may not provide a definite answer to the question of whether screening with PSA decreases mortality. (Cancer J 2009;15:7-12)

30 EXAMPLE #2: RCTs & OBSERVATIONAL STUDIES PSA screening and prostate-cancer mortality: RCT (Schroder 2009); HR = 0.80 ( ) RCT (Andriole 2009); HR = 1.13 ( ) Case-control (Weinmann 2004); OR = 0.70 ( ) Case-control (Concato 2006); OR = 1.13 ( ) Note: results of the two RCTs are contradictory ; RCT & observational study results overlap

31 EXAMPLE #2: ROYAL DECREE Amer Urol Assoc (AUA) 2009 Best-Practice update: Because there is now evidence from a RCT regarding a mortality decrease associated with PSA screening, the AUA is recommending PSA screening, as proposed in this document, for well-informed men who wish to pursue early diagnosis.

32 EXAMPLE #2: MEDICINE-BASED EVIDENCE Issues re: screening for prostate cancer & mortality: Patients: Exposure: Outcome: age; comorbidity indolent vs. aggressive cancer PSA alone (vs. velocity, density, etc.) with or without digital rectal exam? compared to usual care? overall vs. cause-specific mortality consider morbidity? duration of follow-up

33 SPECTRUM OF VIEWS ON RESEARCH DESIGN Randomized trials or observational tribulations? Only randomized treatment assignment can provide a reliably unbiased estimate of treatment effects perhaps we have not tried hard enough to convert the skeptics. (Pocock, New Engl J Med 2000;342:1907) Why there s no cause to randomize None of [the arguments in favor of randomization] supplies any practical reason for thinking of randomization as having unique epistemic power It is in fact very difficult to see any cogent reason for thinking as highly of RCTs as the medical community does (Worrall, Brit J Phil Sci 2007;58:451)

34 CRITICISM OF ROYAL STATUS GIVEN TO RCTs Decisions should be based on best available evidence, to avoid: RCT-myopia action only justified by RCTs Evidence-based paralysis no action without RCTs (Ziemer Arch Intern Med 2006;166:1672)

35 DISCUSSION IN LAY PRESS Sep 16, 2007 Do We Really Know What Makes Us Healthy? remain skeptical until a randomized trial is done.

36 RESPONSE AS LETTER-TO-EDITOR The take-home message for readers should be not to expect certainty from any individual study, observational or randomized. Progress in science is iterative and incremental in the long run, we get closer to truth as overall evidence accumulates on a given topic. Putting what we know into practice would prevent more disease than [bowing before the throne] of randomized trials. John Concato, M.D., The New York Times, Sep. 30, 2007

37 SUMMARY Science as experiment; science as observation The importance lies not in arguing about which methodology is better than the other, but what can be learned about disease activity and therapy from each type of study. (Chakravarty, Nat Clin Pract Rheumatol 2006;6:286) On the evidence for decisions [re:] therapeutic interventions Decision makers need to assess and appraise all the available evidence irrespective of whether it has been derived from randomised controlled trials or observational studies, and the strengths and weaknesses of each need to be understood. [There is] no shame in accepting that judgments are required. (Rawlins, Lancet 2008;372:2152)

38 TAKE-HOME MESSAGES Carefully-designed observational studies can address confounding adequately; limitations of RCTs are often overlooked; results from both designs are informative Patient-oriented research needs more Medicine-Based Evidence, i.e., more methodological and clinical judgment