Take Home Messages Evidence, Anecdotes, and Snake Oil: A Primer on When to Believe

Size: px

Start display at page:

Download "Take Home Messages Evidence, Anecdotes, and Snake Oil: A Primer on When to Believe"

Annis Sherman
6 years ago
Views:

1 Take Home Messages Evidence, Anecdotes, and Snake Oil: A Primer on When to Believe Sumit R. Majumdar, MD MPH FRCPC FACP Professor of Medicine, Endowed Chair in Patient Health Management, AHFMR and AI-HS Health Scholar, Faculties of Medicine and Dentistry and Pharmacy and Pharmaceutical Sciences, University of Alberta Discount anecdotes and testimonials Be very skeptical of observations (and media reports about these studies) Understand that if it is too easy or too good to be true it probably is Wait until there is evidence that proves the new treatment is both safe and effective And all else equal, wait for the second study before buying it or trying it or believing it Scientific Method? Anecdotes then Observations then Hypotheses then Experimental Trials Why experiment? Chance (p <.5) Selection bias Measurement bias Regression to mean Volunteer / Hawthorne Secular trends Experiments = Trials (Systematic) Observations etc Anecdotes Anecdotes (Systematic) Observations Scotland has a very high rate of death from coronary dz and Scots don t eat much fish (< 5g per week) Japan has a lower rate of death from coronary dz and Japanese eat a lot of fish (~1g per day) Greenland has the lowest rate of death from coronary dz and Greenlanders eat even more fish (~4g per day) 1

2 852 Healthy Men Systematically Followed for 2 Years How Does Fish Oil* Reduce Cardiovascular Disease? Fish (yes) CHD Death (yes) Alive 6 [8%] 673 No Fish 18 [15%] 11 OR =.5 5% decrease p <.1 Reduces susceptibility to arrhythmias Decreases platelet stickiness Reduces triglycerides Retards growth of atherosclerotic plaques Anti-inflammatory Promotes NO induced vasodilation Lowers blood pressure Etc *Omega-3 Fatty Acids (AHA Scientific Statement, online update and ATVB) Explanations for ANY Findings Schematic for ANY Experiment (= TRIALS) Oxman and Guyatt: Selection bias Intervention The human mind is sufficiently fertile that there is no shortage of biologically plausible explanations to support almost any observation Starts R Controls Ends EXPERIMENTS FIRST, EXCEPT FOR MEDICAL RESEARCH, BECAUSE: Trial of Fish Oil (N-3 PUFA) to Prevent Heart Dz in 12, Healthy-ish People 1. Can t wait for evidence 2. Obvious so we don t need evidence 3. Anything new better than nothing R Fish oil capsules 4. No harms or downsides to the efforts Starts Olive oil controls Ends 2

Experiments = Trials Increasing Degrees of Truth (Systematic) Observations Anecdotes (Risk and Prevention Collaborative Group.

assertion claims to know intuitively "from the gut" or because it "feels right" without regard to evidence, logic, intellectual examination, or

Observations Vegetarians live longer People who say they eat more than 5 servings of fresh fruits and vegetables a day are less likely to get

3 Experiments = Trials Increasing Degrees of Truth (Systematic) Observations Anecdotes (Risk and Prevention Collaborative Group. N Engl J Med 213;368:18) Increasing Degrees of Truth Not Truthiness a quality characterizing a "truth" that a person making an argument or assertion claims to know intuitively "from the gut" or because it "feels right" without regard to evidence, logic, intellectual examination, or facts something that seems like truth the truth we want to exist (Dr Stephen Colbert DFA, 25) Anecdotes About Fruits and Vegetables (Systematic) Observations Vegetarians live longer People who say they eat more than 5 servings of fresh fruits and vegetables a day are less likely to get lung cancer Beta-carotene very common and easy-tosupplement vitamin across many fruits and vegetables and people with higher serum levels less likely to get lung cancer 3

8, Healthy Nurses Systematically Followed for 1 Years How Does Beta-Carotene Reduce Cancer? >5 servings (yes) None (<2 per day) Lung Cancer (yes) 91 [.6%] 139 [.9%] No Cancer 14,99 14,861 OR =.

2 Demonstrations of anti-carcinogenesis and tumor inhibition in both test-tubes and murine models of: Free-radical scavenger (anti-oxidant) Induction of apoptosis Enhanced gap junction communication

4 8, Healthy Nurses Systematically Followed for 1 Years How Does Beta-Carotene Reduce Cancer? >5 servings (yes) None (<2 per day) Lung Cancer (yes) 91 [.6%] 139 [.9%] No Cancer 14,99 14,861 OR =.7 3% decrease p =.2 Demonstrations of anti-carcinogenesis and tumor inhibition in both test-tubes and murine models of: Free-radical scavenger (anti-oxidant) Induction of apoptosis Enhanced gap junction communication Immune-modulation Etc TRIALS NEEDED FIRST, EXCEPT FOR BETA-CAROTENE, BECAUSE: Trial of Beta-Carotene to Prevent Lung Cancer in 29, Smokers 1. Can t wait for evidence 2. Obvious so we don t need evidence 3. Anything new better than nothing R Beta-carotene (2 mg/day; RDA = 5mg) 4. No harms or downsides its natural Starts Placebo controls Ends Beta Carotene and Cardiovascular Mortality Observations 31% decrease in cardiac deaths Trials 12% increase in cardiac deaths 18% increase in lung cancer (p=.1) 8% increase in (cardiac) mortality (p=.2) (Egger. BMJ. 1998;316:14) 4

5 Event Rates (%) Event Rates (%) Half a Million Systematic Observations of Flu Vaccine Effectiveness in Older People 42% Decrease (Jefferson et al. Lancet. 27;37:1199 and replicated in definitive cohort study [n=18 cohorts, 7k person-years] by Nichol et al. N Engl J Med. 27;357:1373) Benefits of Flu Vaccine Systematically Observed in Older People TRIALS NEEDED FIRST, EXCEPT FOR FLU VACCINES, BECAUSE: Vaccinated Not Vaccinated 2% decrease, 19% decrease, 16% decrease, NNT 556, NNT 585, NNT 893, p=.1 p=.2 p=.18 23% decrease, NNT 145, p<.1 IHD HF Stroke Any Event 1. Can t wait for evidence 2. Obvious so we don t need evidence 3. No harms or downsides to the efforts 4. Not giving flu vaccine - UNETHICAL (Nichol et al. N Engl J Med. 23;348:1322) Trial of Flu Vaccine in Elderly Benefits of Flu Jab in Older People One Randomized Trial (n=544) Selection bias Flu jab % decrease 1% decrease (-41 to +51%) (-44 to +79%) Flu Jab Placebo 8 R % increase (-41 to +6%) 2 Starts Saline jab Ends Serology Clinical Dz Death (Govaert et al. JAMA. 1994;272:1661) 5

6 24-Month Improvement (%) At Target (%) Systematic Observations: Early Referral to a Specialist Reduces Mortality 4% decrease p<.1 6% decrease p<.1 (Kinchen. Ann Intern Med. 22;137:479) Specialist Care vs Family Docs for Patients with Chronic Kidney Dz COMPUTERIZED PROMPTS DOB: 1918 Jan 12 SEX: F SBP <13mmHg LDL <2.6mmol/L A1c < 7% RAAS-i Specialist Family Doc Results Test Name Result Units Ref. Range Creatinine *166 umol/l 5-15 High Calculated GFR *25 ml/min/1.73m2 >59 Low Abnormality In Outpatients, estimated GFR is a more accurate marker of kidney function than creatinine. Chronic kidney disease is defined by egfr <6 ml/min/1.7.3m2 for 3 months or more. *Published guidelines recommend that patients with egfr <3 ml/min/1.73 m2 be referred to a Nephrologist. (see ) TRIALS NEEDED FIRST, EXCEPT FOR KIDNEY DZ RESEARCH, BECAUSE: 1. Can t wait for evidence 2. Obvious so we don t need evidence 3. Anything new better than usual care 4. No harms or downsides to the efforts Prompts and Early Specialist Referral 4 SBP < 13mmHg 21 LDL < 2.6 mmol/l Early Referral p=.8 p<.1 p<.1 p< A1c < 7% egfr loss > 4ml/min (Barrett. Clin JASN. 211;6:1241) 6

24-Month Improvement (%) Randomized Trial of Early Nephrologist Referral vs Usual Care By Family Physicians (N~5) 1 9 8 7 6 5 4 3 2 1 4 4 SBP < 13mmHg Family Doc 22 21 LDL < 2.

7 24-Month Improvement (%) Randomized Trial of Early Nephrologist Referral vs Usual Care By Family Physicians (N~5) SBP < 13mmHg Family Doc LDL < 2.6 mmol/l 34 Early Referral p=.76 p=.74 p=.64 p= A1c < 7% egfr loss > 4ml/min Application of the Scientific Method (Barrett. CANPREVENT Trial. Clin JASN. 211;6:1241) Selection Bias Questions or Comments? Normal function, cognition Good Patients Better Healthy- More prevention diet and User - meds (vitamins, HRT) lifestyle - screening (BMD, cancer) - immunizations (flu jab) Better Outcomes (Eurich, Majumdar. JGIM. 212;27:268) REASONS WE USE TO JUSTIFY NOT DOING TRIALS FIRST? 1. Can t wait for evidence 2. Obvious so we don t need evidence 3. Anything new better than nothing 4. No harms or downsides = Too hard, not worth it, can t be done (Smith, Pell. BMJ. 23; 327: 1459) 7

Peptic Ulcer Disease Common problem related to damage from too much stomach acid Triggered by stress, smoking, alcohol, poor diet, etc (Reasonably successfully) treated with bland diet then antacids

project chatted with a pathologist (Robin Warren) who said I keep seeing bacteria on biopsies of patients with really bad ulcers 1 consecutive patients cultured and 2 grew Helicobacter pylori Work

8 Peptic Ulcer Disease Common problem related to damage from too much stomach acid Triggered by stress, smoking, alcohol, poor diet, etc (Reasonably successfully) treated with bland diet then antacids + / - sometimes surgery and more recently with powerful acid suppressing medications Anecdotes and Observations No experimental proof 3 rd year resident (Barry Marshall) looking for a research project chatted with a pathologist (Robin Warren) who said I keep seeing bacteria on biopsies of patients with really bad ulcers 1 consecutive patients cultured and 2 grew Helicobacter pylori Work scoffed at by gastroenterologists, rejected at journals and at meetings Helicobacter pylori (N of 1) Trials Experiments = Trials Increasing Degrees of Truth Patient with ulcers and Helicobacter pylori infection Marshall had no ulcers and normal endoscopy (Systematic) Observations Drank a broth of bugs, expected symptoms in a few months nausea, vomiting, dyspepsia within 3 days Endoscopy shows florid infection and gastritis Anecdotes Took antibiotics and bismuth, cleared infection, resolved gastritis, and felt better now we know that infection the most common cause of ulcers and gastritis and (perhaps) stomach cancer 8

Rate of Events (%) Trial of Intercessory Prayer for CCU Patients (N ~ 4) Prayers R Starts Nothing Ends Intercessory Prayer for CCU Patients 3 Prayer Controls 2 2 14 13 17 14 22 1 8 3 Heart Failure

9 Rate of Events (%) Trial of Intercessory Prayer for CCU Patients (N ~ 4) Prayers R Starts Nothing Ends Intercessory Prayer for CCU Patients 3 Prayer Controls Heart Failure Cardiac Arrest Died * Did "Bad" (Byrd. South Med J. 1988; 81: 826) *All findings statistically significant except for death (Byrd. South Med J. 1988; 81: 826) Chelation Therapy EDTA binds metallic ions toxic to humans, including mercury, lead, cadmium as well as well as many needed cofactors like zinc, magnesium, and calcium Typical IV infusion = 1 L saline plus EDTA [plus heparin plus magnesium plus lidocaine plus B-vitamins plus 1-2g vitamin C]; usually need 3-4 infusions over 3-6 months plus maintenance treatments Costs $5 - $1 per infusion or $3-ish 9

Chelating Atherosclerotic Plaques Anecdotes About Chelation Ten patients who got EDTA felt better after treatment (Clarke, 1956 and Meltzer, 196) Most commonly used for chest pain, avoiding bypass,

Chelation Work? 1. 196-197 Roto-rooter hypothesis 2. 197-198 PTH and gradual plaque decalcification hypothesis 3. 198-199 Blocks free radicals which damage arteries and promote plaque 4.

Obvious so we don t need evidence 3. Anything new better than nothing 4.

10 Chelating Atherosclerotic Plaques Anecdotes About Chelation Ten patients who got EDTA felt better after treatment (Clarke, 1956 and Meltzer, 196) Most commonly used for chest pain, avoiding bypass, claudication, TIAs Also used for (in alphabetical order): arthritis, back pain, chronic fatigue, fibromyalgia, irritable bowel, sciatica, sleep apnea, stress, weight loss, whiplash, etc How Does Chelation Work? Roto-rooter hypothesis PTH and gradual plaque decalcification hypothesis Blocks free radicals which damage arteries and promote plaque to now Prevents cellular mutation of arterial cells which causes plaque TRIALS NEEDED FIRST, EXCEPT FOR CHELATION THERAPY, BECAUSE: 1. Can t wait for evidence 2. Obvious so we don t need evidence 3. Anything new better than nothing 4. No harms or downsides to the efforts Placebo Effects Odds ratio (95% CI) Trial of Chelation for Cardiac Patients (N = 84) Coronary Drug Project Research Group 198 w1 β blocker heart attack trial (men) 199 w2 β blocker heart attack trial (women) 1993 w3 Canadian amiodarone myocardial infarction arrhythmia trial 1999 w8 Cardiac arrhythmia suppression trial 1996 w4 Physicians health study 199 w16 West of Scotland prevention study 1997 w17 University Group Diabetes Project 197 w w18 R Chelation Total events: 581 (good adherence), 415 (poor adherence) Test for heterogeneity: χ 2 = 14 (P =.5) with I 2 = 51% Test for overall effect: Z = 4 (P <.1) 4% decrease Good adherence Poor adherence (Simpson SH et al. BMJ. 26;333:15-9) Starts Placebo (IV saline) Ends 1

Time in Seconds Time in Seconds PATCH Chelation Trial PATCH Chelation Trial 7 63 sec p<.1 7 9 sec p=.

6 3 3 2 2 1 1 Increase in Walk Time Increased Time to Fatigue Increase in Walk Time Increased Time to

11 Time in Seconds Time in Seconds PATCH Chelation Trial PATCH Chelation Trial 7 63 sec p< sec p= Chelation 31 sec p= Chelation Placebo 15 sec p= Increase in Walk Time Increased Time to Fatigue Increase in Walk Time Increased Time to Fatigue (Knudston. JAMA. 22; 287: 481) (Knudston. JAMA. 22; 287: 481) 2 1 PATCH Chelation Trial Favors Chelation Application of the Scientific Method p= Walk Time (s) Endurance Chest Pain Health Favors Placebo (Knudston. JAMA. 22; 287: 481) Trials Increasing Degrees of Truth (Systematic) Observations Increasing Degrees of Snake Oil-iness Anecdotes 11

North American Ginseng (Panax quinquefolius) Still not sure How Does It Work?

analgesia GABA-ergic receptor agonist Decrease insulin resistance Etc Ginseng Side Effects Headache, nausea, insomnia, restlessness Tachycardia,

Application of the Scientific Method 4% rate of colds untreated 6% absolute reduction (15% relative) in colds = 1 person taking Cold-FX for 17

12 North American Ginseng (Panax quinquefolius) Still not sure How Does It Work? Ginsenogens active ingredient, fairly closely related to steroids and thus: Immuno-modulation esp Natural Killer cells Tumor suppression Non-opioid analgesia GABA-ergic receptor agonist Decrease insulin resistance Etc Ginseng Side Effects Headache, nausea, insomnia, restlessness Tachycardia, palpitations Low blood sugars (esp those with diabetes) Serious bleeding (esp those taking aspirin) Etc 4 Trials of Cold-FX TM for Prevention (N=1) Application of the Scientific Method 4% rate of colds untreated 6% absolute reduction (15% relative) in colds = 1 person taking Cold-FX for 17 entire winter seasons would have 1 less cold Note: No significant effects observed in meta-analysis when looking at symptom DURATION or SEVERITY and NO TRIALS published studying infected patients (Ilersich. CBC Marketplace. 211) 12

Application of the Scientific Method Discount anecdotes and testimonials Be very skeptical of observations (and media reports about

13 It works best if you take it at the first signs of a cold Trials Increasing Degrees of Truth (Systematic) Observations Increasing Degrees of Snake Oil-iness Anecdotes Note: NO Trials of Reducing Cold Symptoms Once Infected When to Believe? Application of the Scientific Method Discount anecdotes and testimonials Be very skeptical of observations (and media reports about these studies) Understand that if it is too easy or too good to be true it probably is Wait until there is evidence that proves the new treatment is both safe and effective 13

14 And all else equal, wait for the second study before buying it or trying it or believing it! Questions or Comments? (Ioannidis. JAMA. 25;294:218 and PLoS. 25;8:e124) 14

The Healthy User Effect: Ubiquitous and Uncontrollable S. R. Majumdar, MD MPH FRCPC FACP

The Healthy User Effect: Ubiquitous and Uncontrollable S. R. Majumdar, MD MPH FRCPC FACP Professor of Medicine, Endowed Chair in Patient Health Management, Health Scholar of the Alberta Heritage Foundation,