MEASURING THE SIZE OF A TREATMENT EFFECT: RELATIVE RISK REDUCTION, ABSOLUTE RISK REDUCTION, AND NUMBER NEEDED TO TREAT Hussein Hollands, MD, MS (epid) Peter J. Kertes, MD, CM, FRCS(C) 72 The purpose of this review article is to outline a few concepts that are useful for interpreting the results of a randomized clinical trial (RCT). Specifically, we will focus on measures of the size of a treatment effect relative risk ratio (RRR), absolute risk ratio, and number needed to treat (NNT). These concepts are important, specifically with respect to decisions to implement a treatment modality in clinical practice based on the results of a RCT. To solidify these ideas, we will work through some practical examples in ophthalmology. In this manuscript, we will assume that we are interpreting results from a well-designed RCT with a clinically important dichotomous negative outcome measure (eg, 15-letter loss of visual acuity) comparing a new treatment modality with a control. Statistical and Clinical Significance When interpreting the results of a RCT, one must make a distinction between the statistical significance of a result and the clinical significance of a result. The statistical significance refers to the precision of the treatment effect and is presented as either a P value or 95% confidence interval. These results are obtained from formal statistical testing. A P value represents the probability that an observed treatment effect is due to sampling error and a 95% confidence interval is a range of treatment effects in which we can be 95% confident that the true treatment effect lies. 1 The clinical significance, by contrast, refers to the size of treatment effect at which a physician feels adoption of the treatment modality would be justified in clinical practice. 2 4 We will review measures used to quantify the size of a treatment effect in the following sections. Of course, in order for a treatment modality to be considered clinically significant, statistical significance would have to be shown in a scientifically valid manner using a clinically important outcome measure. Statistical significance is necessary for clinical significance but is not sufficient. It is useful to understand how sample size relates to these concepts. A larger sample size will increase the precision of the treatment effect that is observed but will not change the size of the treatment effect. A larger sample size (and thus more outcome events) leads to more confidence in the result and a smaller P value or a narrower confidence interval around the point estimate. In fact, any treatment effect, in theory, can be found to be statistically significant if enough people are studied. However, even with a very large sample size, a given treatment effect (for example, the RRR) will remain similar. Measures of the Size of a Treatment Effect Most clinical trials will use a dichotomous harmful event such as a 15-letter loss of visual acuity as the primary outcome measure. The incidence rate of the primary outcome is measured in both treatment arms for the duration of the follow-up period. The size of the treatment effect versus a control is often reported as a relative risk (RR) or RRR but can also be reported as an absolute risk reduction (ARR) or NNT. Although results of a RCT are often reported with
complex figures and statistical analyses, it is often easiest to convert the main study results into a simple 2 2 contingency table to allow for clearer conceptualization. Table 1 shows a simple contingency table along with definitions for RR, RRR, ARR, and NNT. The RR is defined as the incidence rate of the outcome in the treatment group divided by the incidence rate of the outcome in the control group. The RRR is defined as a risk reduction (Table 1). If the treatment arm is effective in preventing disease then the RR will be less than 1.0; a smaller RR corresponds to a larger treatment effect. Again, the RR measures the size of a treatment effect and therefore does not change depending on sample size. Increasing sample size will increase the precision of the RR and thus make it more likely to be statistically significant. Another measure of the size of a treatment effect is the ARR, which is defined as the rate of the outcome in the control group minus the rate of the outcome in the treatment group. This can be a useful and intuitive statistic as it accounts for the absolute incidence of disease. 5 The ARR can also be mathematically described in an intuitive statistic called the NNT. The NNT is simply the inverse of ARR and is the number of patients that need to be treated with the intervention to prevent 1 negative outcome event. Essentially, the ARR and NNT confer the same information, except that the NNT offers superior conceptualization. As with RR, the ARR and NNT do not change with sample size, although the precision of these estimates will increase with a larger sample size. To understand the different information that RRR and ARR confer a clinical example is warranted. The Ocular Hypertension Treatment Study (OHTS) was a landmark RCT where over 1600 subjects with ocular hypertension (intraocular pressure between 21 and 32 mm Hg) and no clinical evidence of glaucoma were randomized to either topical glaucoma medications to reach a target intraocular pressure or no treatment. 6,7 The primary outcome measure was the development of glaucoma based on visual field and optic nerve abnormalities over a minimum 5-year follow-up period. Tables 2A and B show a contingency table along with RR and ARR at 60- month follow-up for the entire study population (Table 2A) and among only black patients (Table 2B). 6,7 These results were simplified from survival analyses that were published. Among all patients in the study, treatment with glaucoma medication conferred a 54% RRR (Po0.001). Among only black patients, treatment conferred a 48% relative risk reduction (Po0.001). From this information alone one may conclude that the group of black subjects did slightly less well than the group of all subjects. However, black subjects were more likely to develop glaucoma than the entire patient population (12.2% vs. 6.9%) regardless of treatment. Hence, the ARR conferred from treatment was higher amongst black subjects than the entire population (7.7% vs. 5.1%). In other terms, to prevent 1 patient with ocular hypertension from developing glaucoma over 5 years, only 13 black study patients would need to be treated (NNT = 13) versus 20 patients selected from the study at large (NNT = 20). Table 1. Generic 2 2 Contingency Table and Definitions Outcome (Harmful) Treatment group a b a þ b a/a þ b Control c d c þ d c/c þ d RR = Incidence outcome (tx grp)/incidence outcome (control) = (a/a þ b)/(c/c þ d). RRR = (1 RR) 100% = 100% [1 (a/a þ b)/(c/c þ d)]. ARR = Incidence outcome (control) Incidence outcome (tx grp) = (c/c þ d) (a/a þ b). NNT Equation = 1/ARR = 1/[(c/c þ d) (a/a þ b)]. (Harmful Outcome) 73
Table 2A. Results From the OHTS (All Patients) 7 Outcome (Glaucoma Progression) (Glaucoma Progression) Medical therapy 36 781 817 36/817 = 4.4% No treatment 78 741 819 78/741 = 9.5% RR = Incidence outcome (tx grp)/incidence outcome (control) = (a/a þ b)/(c/c þ d) = 0.46. RRR = (1 RR) 100% = 100% [1 (a/a þ b)/(c/c þ d)] = (1.0 0.46) 100% = 54%. ARR = Incidence outcome (control) Incidence outcome (tx grp) = (c/c þ d) (a/a þ b) = 9.5% 4.4% = 5.1%. NNT = 1/ARR = 1/[(c/c þ d) (a/a þ b)] = 1/0.51 = 19.6. This example illustrates the point that the RRR looks at the risk of disease in the treatment group relative to the control group as a ratio, but does not consider the incidence of the outcome. In contrast, the ARR will increase (and hence NNT decrease) when the overall incidence of the outcome in the study population is increased. Clearly, to fully understand the size of a treatment effect and its clinical importance, both RR and ARR (and thus NNT) should be considered. Table 3 shows some clinical examples from well-cited ophthalmology trials, including the ANti-VEGF Antibody for the Treatment of Predominantly Classic Choroidal Neovascularization in Age- Related Macular Degeneration (AN- CHOR) study, 8 Herpetic Eye Disease Study (HEDS I), Stromal Keratitis, Not on Steroids study, 9 and the Age-Related Eye Disease Study (AREDS). 10 The purpose of this table is to work through some practical examples of RR, ARR, and NNT, and is not to provide summary information from these clinical trials. Certain clinical results were used from these studies to illustrate conceptual points rather than clinical points. The reader is referred to the study manuscripts for more details on the study results. 8 10 Assessing Adverse Events In addition to assessing the size of a treatment effect in a RCT, it is important to quantitatively consider complication rates with a novel therapy. A useful summary statistic is the number needed to harm (NNH), or the number of patients that would need to be exposed to a treatment modality to produce an adverse event for 1 patient. This measure can be calculated for any adverse outcome that occurs more frequently in the treatment group than the control group and is related to attributable risk. The attributable risk is simply the incidence of the adverse event in the treatment group minus the incidence of the event in the control group. Furthermore, the NNH is simply the inverse of the attributable risk. An example where NNH can be clinically applicable is in the treatment of diabetic macular edema that persists Table 2B. Results From the OHTS Study (Black Subjects) 6 Outcome (Glaucoma Progression) (Glaucoma Progression) 74 Medical therapy 17 186 203 17/203 = 8.4% No treatment 33 172 205 33/205 = 16.1% RR = Incidence outcome (tx grp)/incidence outcome (control) = (a/a þ b)/(c/c þ d) = 0.52. RRR = (1 RR) 100% = 100% [1 (a/a þ b)/(c/c þ d)] = (1 0.52) 100% = 48%. ARR = Incidence outcome (control) Incidence outcome (tx grp) = (c/c þ d) (a/a þ b) = 16.1% 8.4% = 7.7%. NNT = 1/ARR = 1/[(c/c þ d) (a/a þ b)] = 1/0.077 = 12.9.
Table 3. Summary Statistics From 3 Major Clinical Trials in Ophthalmology Study Outcome Rate (Harmful Outcome) RR RRR(%) ARR(%) NNT ANCHOR* 0.5 mg Ranibizumab: 5/139 = 3.6% 0.10 90 32.1 2.8 Control: 51/143 = 35.7% HEDS Iw Antiviral þ Steroids: 13/49 = 26% 0.36 64 47 2 Antiviral Only: 41/57 = 73% AREDSz Antioxidants þ Zinc: 0.202 0.726 27 7.6 13 Placebo: 0.278 *In the ANCHOR trial, 8 patients with wet age-related macular degeneration were randomized to receive either Ranibizumab or verteporfin. The outcome measure reported here is loss of 15 or more letters from baseline visual acuity at 12 months. win the Herpetic Eye Disease I (HEDS 1) study 9 - Stromal Keratitis, Not on Steroids Trial, patients with active HSV stromal keratitis diagnosed clinically were randomized to receive either topical antivirals or topical antivirals plus topical steroids for a 10-week period in tapered doses. The outcome measure reported here is treatment failure before the completion of a 10-week course of trial medication. zdata reported here from the Age-Related Eye Disease Study (AREDS) 10 show results from patients with categories 3 and 4 dry age-related macular degeneration (advanced dry disease) who were randomized to either antioxidants plus zinc or placebo. These show the probability of development of advanced age-related macular degeneration over a 5-year period. after laser treatment. In 2006, Gillies and others 11 conducted a well-designed RCT and showed that patients with refractory diabetic macular edema receiving 4-mg intravitreal triamcinolone acetonide (IVTA) (average 2.6 treatments) were significantly more likely than patients receiving sham to gain 5 or more letters of best-corrected visual acuity after 2 years. However, IOP complications of the steroid were statistically and clinically significant. At 2 years, more patients in the IVTA group required initiation of glaucoma medication (44% vs. 3%), and more patients in the IVTA group required glaucoma filtration surgery (6% vs. 0%) when compared with patients receiving sham. Therefore, the attributable risk for requiring initiation of glaucoma medication was 41% (44% to 3%) and the NNH for this adverse event was 2.4 (1/0.41). In addition, the attributable risk for requiring glaucoma filtration surgery is 6% (6% to 0%) and the NNH is 17 (1/0.06). Consequently, the complication of requiring initiation of glaucoma medication in 1 patient occurred when only 2.4 patients were exposed to IVTA treatment. Analogously, glaucoma filtration surgery was required in 1 out of every 17 patients exposed to IVTA therapy. Conclusions In evidence-based medicine, the consideration of a novel treatment modality for application in practice usually requires reproducible results from a good quality RCT or meta-analysis showing a statistically significant treatment effect measured with a clinically meaningful primary outcome measure. As most new treatment modalities are associated with cost and potential adverse events, the treatment should also be considered clinically significant. In other words, the size of the treatment effect should be large enough to justify its use in clinical practice. A relative risk ratio with associated P value is often quoted in the literature. However, to gain a full understanding of the size of a treatment effect and thus its clinical implications, the ARR and NNT are important concepts. The NNT is especially easy to conceptualize and quantifies the number of patients that need therapy to prevent 1 harmful outcome. These statistics may not be reported outright in a published manuscript so it is worthwhile to be familiar with creating a simple 2 2 contingency table and performing the calculations by hand. References 1. Rosner B. Fundamentals of Biostatistics. 6th ed. Toronto: Thomson; 2006. 2. Kertes PJ, Johnson TM. Evidence-Based Eye Care. Philadelphia: Lippincott Williams & Wilkins; 2007. 75
3. Oxman AD, Sackett DL, Guyatt GH. Users guides to the medical literature. I. How to get started. The Evidence-Based Medicine Working Group. JAMA. 1993;270:2093 2095. 4. Sackett DL, Brian Haynes R, Tugwell P. Clinical Epidemiology a Basic Science for Clinical Medicine. Toronto: Little, Brown, and Company; 1988. 5. Oleckno W. Essential Epidemiology Principles and Applications. Prospect Heights, IL: Waveland Press; 2002. 6. Higginbotham EJ, Gordon MO, Beiser JA, et al. The Ocular Hypertension Treatment Study: topical medication delays or prevents primary open-angle glaucoma in African American individuals. Arch Ophthalmol. 2004;122:813 820. 7. Kass MA, Heuer DK, Higginbotham EJ, et al. The Ocular Hypertension Treatment Study: a randomized trial determines that topical ocular hypotensive medication delays or prevents the onset of primary open-angle glaucoma. Arch Ophthalmol. 2002;120:701 713; discussion 829 830. 8. Brown DM, Kaiser PK, Michels M, et al. Ranibizumab Versus verteporfin for neovascular agerelated macular degeneration. N Engl J Med. 2006;355:1432 1444. 9. Wilhelmus KR, Gee L, Hauck WW, et al. Herpetic Eye Disease Study. A controlled trial of topical corticosteroids for herpes simplex stromal keratitis. Ophthalmology. 1994;101: 1883 1895; discussion 1895 1896. 10. Age-Related Eye Disease Study Research Group. A randomized, placebo-controlled, clinical trial of high-dose supplementation with vitamins C and E, beta carotene, and zinc for age-related macular degeneration and vision loss: AREDS report no. 8. Arch Ophthalmol. 2001;119: 1417 1436. 11. Gillies MC, Sutter FK, Simpson JM, et al. Intravitreal triamcinolone for refractory diabetic macular edema: two-year results of a double-masked, placebo-controlled, randomized clinical trial. Opthalmology. 2006;113: 1533 1538. 76