Evaluating Effectiveness of Treatments Elenore Judy B. Uy, M.D.
OUTLINE 1. Why evaluate effectiveness 2. How to acquire evidence 3. How to appraise evidence Directness Validity Results 4. Our Case
WALA WALA WALANG HIMALA! x2 24x32 Acrylic on Canvas Geloy Concepcion 2012 Walang himalaaaaaaa!!!
1 WHY EVALUATE EVIDENCE
Half of what we learn in medical school is wrong 1960 s 1980 s 2010 2000 s
Clinical Questions Internal Medicine Residents 2 questions for every 3 outpatients 5 for every inpatient 29% pursued books (31%); journals (21%); others (17%) Barriers: lack of time (60%), forgot (29%) Am J Med 2000;109:218-33.
Millions of Articles 20 18 16 14 12 750,000 added in 2009 alone Diabetes: 150,000 since 1966 Hypertension: 4000 in October 2010 10 8 6 4 2 0 65 70 75 80 85 90 95 00 05 10 NATIONAL LIBRARY OF MEDICINE
The need for a new paradigm We have a continuing need to acquire medical information Knowledge constantly grows and changes We don t have enough time
EBM: a new paradigm A systematic approach to ACQUISITION, APPRAISAL, AND APPLICATION of research evidence To guide decisions In healthcare
Mrs. Cruz
Why evaluate effectiveness Complex medical conditions Limited Resources Each treatment needs to be effective
Mrs. Cruz Discharge Diagnosis Community Acquired Pneumonia Pulmonary Tuberculosis (probably inactive) Rheumatic Heart Disease Moderate Mitral Stenosis Transient Ischemic Attack Hypertension Dyslipidemia Medications 1. Azithromycin 500mg OD 2. Cefuroxime 500mg OD 3. Quadruple anti-koch s OD 4. Losartan 50mg OD 5. Metoprolol 50mg OD 6. Spironolactone 25 mg OD 7. Digoxin 0.25 mg OD 8. Warfarin 2.5 mg OD 9. Simvastatin 20mg HS 10. Multivitamins OD 11. Vitamin E OD
2 ACQUIRING THE EVIDENCE
Forming answerable clinical questions Patient Among adults Exposure is giving multivitamins Outcome beneficial? Morbidity? Mortality?
Searching for evidence
Gaziano J, Sesso HD. Multivitamins in the Prevention of Cancer in Men: The Physicians Health Study Ii Randomized Controlled Trial. JAMA 308, no. 18 (November 14, 2012): 1871 Acquisition of Evidence 1880. doi:10.1001/jama.2012.14641 Gaziano J, Sesso HD. Multivitamins in the Prevention of Cancer in Men: The Physicians Health Study Ii Randomized Controlled Trial. JAMA 308, no. 18 (November 14, 2012): 1871 1880. doi:10.1001/jama.2 012.14641
Directness Validity Results 3 APPRAISING THE EVIDENCE
APPRAISING DIRECTNESS Is this study likely to answer my clinical question?
Directness Clinical Question Question Research Question (Trial) Population Adult, female Adult, male Exposure Outcome Multivitamins Morbidity Mortality Multivitamins vs. Placebo Cancer
APPRAISING DIRECTNESS Is this study likely to answer my clinical question?
APPRAISING VALIDITY Is the difference in outcome events really because of the therapies that are being compared? How fair are the comparisons between the therapies being compared? Were the odds stacked in favor of one group over the other (bias)?
APPRAISING VALIDITY 1. Randomization 2. Allocation Concealment 3. Equal at baseline 4. Patient blinding 5. Caregiver blinding 6. Outcome assessor blinding 7. Analyzed to original group 8. Adequate follow-up rate
1 Randomization Look for: Coin Toss Table of random numbers Computer-generated random sequences
2 Allocation Concealment Look for: Sealed, opaque envelopes Third party allocates treatment Computerized allocation of treatment
3 Groups equal at baseline Look for the table of baseline characteristics (Table 1)
4-6 Blinding Make unaware of treatment assignment Who is blinded? Patient Caregiver Outcome Assessor
7 Analyzed to original group Look for: Patients analyzed to the groups to which they were originally randomized (regardless of compliance to treatment) Intention-to-treat analysis done Number of patients randomized=number of patients analyzed
8 Adequate follow-up Drop-outs Patients who leave the study Usually leave due to adverse events or dissatisfaction Outcomes unknown
8 Adequate follow-up Is the drop-out rate worrisome? Control Drop-out Rate Treatment Drop-out Rate Control Death Rate Treatment Death Rate Reported 1% 1% 2% 1% Worst Case 1% 1% 2% 2%
8 Adequate follow-up When to worry: 1. When there is gross imbalance in drop-out rates between groups 2. When drop-out rates are greater than event rates 3. When worst assumptions on what happened lead to opposite conclusions.
Appraisal: Validity Validity Criteria Randomly assigned to treatment groups? Allocation concealed? Baseline characteristics similar at the start of the trial? Patients blinded to treatment assignment? Caregivers blinded to treatment assignment? Outcome assessors blinded to treatment assignment? All patients analyzed in the groups to which they were originally randomized? Follow-up rate adequate? Assessment Yes Yes Yes Yes Yes Yes Yes No Best Case (Assume drop outs in treatment survived): 0.92 Worst Case (Assume drop outs in treatment died): 0.99
APPRAISING VALIDITY Is the difference in outcome events really because of the therapies that are being compared?
APPRAISING RESULTS Am I confident that treatment is effective?
APPRAISING RESULTS 1. How large was the treatment effect?
My sister is not a pig. She weighs 75% of what she used to weigh 60kg/80kg Relative Weight She lost25% of her weight (80kg- 60kg)/80kg Relative Weight Reduction She lost 20 kilograms 80kg 60kg - Absolute Weight Reduction
Risk The probability that an event will occur. -Carr, S. et. al. Developed Outcome Total Patients Treatment 6 100 Control 8 100 The risk of developing the outcome: Control (Rc) or Baseline risk : Treatment (Rt) : 8/100 6/100
Expressing the change in risk Comparing the risk of treatment (Rt) against the risk of control (Rc): Developed Outcome Total Patients Risk Treatment 6 100 6% Control 8 100 8% My risk is 75% of what it used to be 6 8 Rt Rc Relative Risk I lost 25% of my risk (8-6) 8 (Rc-Rt) Rc Relative Risk Reduction I lost 2% of my risk 8-6 Rc-Rt Absolute Risk Reduction
1 How large was the treatment effect My risk is now 75% of what it used to be Relative Risk Rt/Rc I lost 25% of my risk Relative Risk Reduction (Rc-Rt)/Rc I lost 2% of my risk Absolute Risk Reduction Rc-Rt
Relative Risk (RR) RR = Risk of Treatment (Rt) Risk of Control (Rc) Assuming a NEGATIVE outcome/event: RR Value Implication Interpretation RR>1 Rt>Rc Harm RR=1 Rt=Rc No effect RR<1 Rt<Rc Benefit
Point Estimates RR, RRR, ARR Best estimate of the true effect of a drug Misleading Does not accurately reflect the uncertainty surrounding the estimate due to sample size limitations Among adults who take multivitamins, the risk of developing cancer is decreased to 92% of the risk of those who do NOT take it
Interval Estimates Confidence Intervals Humbler, admits a range of possible values of the treatment effect Are more likely to be correct Provides best and worse case scenario Among adults who take multivitamins, the risk of developing cancer is decreased to 92% of the risk of those who do NOT take it [95% CI: 0.86, 0.998]
APPRAISING RESULTS 1. How large was the treatment effect? 2. How precise was the estimate of effect?
Precision
Interval Estimates Benefit 1 Harm
Appraisal: Results Incidence of total cancers: HR: 0.92 (95%CI: 0.86 to 0.998) 1. How large was the treatment effect? Among adults who take multivitamins, the risk of developing cancer is decreased to 92% of the risk of those who do NOT take it. 2. How precise was the estimate of the treatment effect? At best: the risk of developing cancer in those who take multivitamins is 86% the risk of those who do NOT take it. At worst: the risk of developing cancer is practically the same in those who do and do NOT take multivitamins (99.8% 100%)
APPRAISING RESULTS Am I confident that treatment is effective?
Recommendation: DO NOT recommend taking multivitamins It may offer no benefit over placebo HR: 0.92 (95%CI: 0.86 to 0.998)
4 OUR CASE: Mrs. Cruz
Mrs. Cruz Discharge Diagnosis Community Acquired Pneumonia Pulmonary Tuberculosis (probably inactive) Rheumatic Heart Disease Moderate Mitral Stenosis Transient Ischemic Attack Hypertension Dyslipidemia Medications 1. Azithromycin 500mg OD 2. Cefuroxime 500mg OD 3. Quadruple anti-koch s OD 4. Losartan 50mg OD 5. Metoprolol 50mg OD 6. Spironolactone 25 mg OD 7. Digoxin 0.25 mg OD 8. Warfarin 2.5 mg OD 9. Simvastatin 20mg HS 10. Multivitamins OD 11. Vitamin E OD
Mrs. Cruz Discharge Diagnosis Community Acquired Pneumonia-resolved Pulmonary Tuberculosis (probably inactive) Rheumatic Heart Disease Moderate Mitral Stenosis Transient Ischemic Attack Hypertension Dyslipidemia Medications 1. Metoprolol 50mg OD 2. Spironolactone 25 mg OD 3. Warfarin 2.5 mg OD 4. Simvastatin 20mg HS
OUTLINE 1. Why evaluate effectiveness 2. How to acquire evidence 3. How to appraise evidence Directness Validity Results 4. Our Case
Walang himala! Ang himala ay nasa puso ng tao, nasa puso nating lahat! Tayo ang gumagawa ng mga himala!
Thank you.
Relative Risk Reduction (RRR) RRR = Risk of Control (Rc) - Risk of Treatment (Rt) Risk of Control (Rc) Assuming a NEGATIVE outcome/event: RR Value Implication Interpretation RRR>0% Rt<Rc Benefit RRR=0% Rt=Rc No effect RRR<0% Rt>Rc Harm
Absolute Risk Reduction (ARR) ARR = Risk in Control (Rc)-Risk in Treatment (Rt) Assuming a NEGATIVE outcome/event: RR Value Implication Interpretation ARR>0% ARR=0% ARR<0% Rt<Rc Rt=Rc Rt>Rc Benefit No effect Harm
8 Adequate follow-up Is the drop-out rate worrisome? Control Drop-out Rate Treatment Drop-out Rate Control Death Rate Treatment Death Rate Reported 1% 10% 5% 5% Worst Case 1% 10% 5% 15%
8 Adequate follow-up Is the drop-out rate worrisome? Control Drop-out Rate Treatment Drop-out Rate Control Death Rate Treatment Death Rate Reported 10% 10% 10% 5% Worst Case 10% 10% 10% 15%