Interpreting Evidence Based Research: Carpe Diem or Caveat Lector? Jonathan C. Routh, MD, MPH March 10, 2017

Transcription

1 Interpreting Evidence Based Research: Carpe Diem or Caveat Lector? Jonathan C. Routh, MD, MPH March 10, 2017

2 Disclosures I have no relevant financial relationships with the manufacturer(s) of any commercial product(s) and/or provider(s) of commercial services discussed in this CME activity. I am Principal Investigator on related grants: NIDDK K08-DK100534, Dendreon CRPC, Trent Foundation, Duke SCORES, Duke Health Fellows grants I do not intend to discuss an unapproved/investigative use of a commercial product/device in our presentation.

3 CME Objectives Describe basic concepts of study design and levels of evidence Define why certain study designs can (or cannot) answer certain types of research questions Identify the most frequent design & analysis mistakes in contemporary urology research

4 Study Design

5 Healthcare Research Spectrum Basic Science Clinical Science Healthcare Policy Translational Science Health Services Research

6 Study Design Simplified Clinical Research Descriptive research Who? What? Where? When? Ecological study Case reports/series Cross-sectional study Analytic research Why? Observational study Case-control study Cohort study Intervention study Randomized clinical trial

7 Randomized Trials RCTs most likely to produce valid conclusions about efficacy Outcome is most likely due to therapy Fundamental trade-off of RCTs: Internal validity for external validity Decreased bias but less clear generalizability RCTs are not necessarily truth Ultimately, an RCT is just one experiment! And any one experiment can be wrong

8 RCTs Are Great, But They Have Limits Smith & Pell. BMJ 327:1459, 2003.

9 Cohort Studies Cohort = follow-up = longitudinal study Retrospective vs. prospective: Important: how were data generated? Possibility of misclassification bias Other Observational Study Designs Case-control Case-crossover Case-cohort

10 Registry studies Advantages of multi-purpose registries: Can include varying exposures and outcomes Including ones you haven t thought of initially Limitations: Important baseline measures may not be assessed Many measured factors will never be used e.g., waist size Difficult to establish and maintain Long-term funding?

11 What is administrative data?

12 Example structure - KID

13 Potential Pitfalls in Study Design If your experiment needs statistics, you ought to have done a better experiment. Ernest Rutherford

14 Improving Validity Bias is a significant issue in much of the published urology literature So anticipate and minimize bias at the design phase Possible to reduce confounding with analysis, BUT: Often can t correct for bias at the analysis stage

15 How should you use data?

16 Big Data Pearls 1. Only ask what the data can answer 2. Beware of significant findings 3. Correlation does not equal causation 4. Validation is a researcher s best friend

17 Big Data Pearls 1. Only ask what the data can answer 2. Beware of significant findings 3. Correlation does not equal causation 4. Validation is a researcher s best friend

18 How is VUR being treated? Let s say you want to determine current VUR treatment patterns Possible Methods: 1. Survey clinicians 2. Perform an RCT 3. Query Big Data Possible Issues: 1. Surveys capture what we think we do 2. RCTs capture what we do under perfect conditions 3. Big Data captures real-world conditions

19 PHIS Database Endoscopic Injection Open Surgery Antibiotic Prophylaxis Nelson et al. J Urol, 2009

20 PHIS Database Herbst et al. J Urol, 2015

21 Probability of EI for Moderate VUR 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% A B C D E F G H I J K L M N O P Q R S T U V W Hospital Routh et al. Pediatrics

22 What can the data answer? Generally good at capturing events Practice patterns, surgeries, dispensed medications, readmissions, deaths, etc Generally NOT good at explaining events VUR grade, presence of bowel bladder dysfunction, preop PSA, tumor size, etc.

23 Admin Data Pearls 1. Only ask what the data can answer 2. Beware of significant findings 3. Correlation does not equal causation 4. Validation is a researcher s best friend

24 UNC vs. EI readmissions Linked state databases 1,802 UNC & 1,768 EI pts UNC pts more likely to be: Younger (4.8 v. 5.8 y) Male (30 v. 22%) Publically insured (53 v. 50%) EI associated with: Less readmissions (6 v. 2%, OR 0.4) Less ER visits (14 v. 7%, OR 0.5) More redo surgeries (1 v. 11%, OR 13) (p<0.001) Tejwani et al. In press, JPU

25 Common Statistical Errors: Absolute vs. Relative risk

26 Common Statistical Errors Overestimating the Importance of P We found no association between Medicaid insurance and loss of testis in 37 boys with testicular torsion 12% Blue Cross v. 67% Medicaid, p=0.07

27 Common Statistical Errors Overestimating the Importance of P We found a highly significant association between incidence of bladder exstrophy and Diet Cherry Cola consuption in our 300 million patient cohort OR , p<0.0001

28 Admin Data Pearls 1. Only ask what the data can answer 2. Beware of significant findings 3. Correlation does not equal causation 4. Validation is a researcher s best friend

29 Correlation & causation

30 Common Statistical Errors Overestimating the Importance of P We found a highly significant correlation between PSADT & spousal BMI r=0.1, p< Image from Wikipedia

31 Common Statistical Errors Our brains are trained to detect patterns Often correctly but not always

32

33

34

35 Admin Data Pearls 1. Only ask what the data can answer 2. Beware of significant findings 3. Correlation does not equal causation 4. Validation is a researcher s best friend

36 Do Not Trust Big Data It is very, very easy to make a small mistake in an admin database Small mistakes can have very large consequences!

37 How to Avoid Big Errors Make sure you understand the data Make sure your statistician understands the data Make sure you understand your statistician (& vice-versa) Validate, Re-validate, and Re-re-validate your data

38

39

40

41 Practical advice Keep in mind why and for whom data were generated Become good friends with somebody who intimately understands the data Do a validation study whenever possible