GS3. Understanding How to Use Statistics to Evaluate an Article. Session Summary. Session Objectives. References. Session Outline

Transcription

1 GS3 Understanding How to Use Statistics to Evaluate an Article Reese H. Clark, MD Director of Research Pediatrix Medical Group Neonatologist Greenville Memorial Hospital, Greenville, SC The speaker has signed a disclosure form and indicated he has no significant financial interest or relationship with companies or the manufacturer(s) of any commercial product/service that will be discussed as part of this presentation. Session Summary In this session Dr. Clark will review how recent articles can be misleading based on the statistical interpretation of the data. Failure to understand the limitations of published papers can lead to changes in care that can hurt patients. Session Objectives Upon completion of this presentation, the participant will: understand how statistics influence the interpretation of important articles; be able to utilize various statistical techniques to evaluate the results of clinical studies. References Ambalavanan, N. & Whyte, R. (2003). The mismatch between evidence and practice. Clinical Perinatology; 30: Carlo, W., Finer, N.E., Walsh, M., et al. SUPPORT Study Group (2010). Early CPAP versus surfactant in extremely preterm infants. New England Journal of Medicine, 362: Cordoba, G. (2010). Definition, reporting, and interpretation of composite outcomes in clinical trials: Systematic review. British Medical Journal, 341: c3920. Smith, P.B., Ambalavanan, N., Li., L., et al. (2012). Approach to infants born at 22 to 24 weeks' gestation: Relationship to outcomes of more-mature infants. Pediatrics, 129: e1508 e1516 Stenson, B., Brocklehurst, P. & Tarnow-Mordi, W. for the UK & Australian & New Zealand Boost II trials. (2011). Increased 36-week survival in higher oxygen saturation targets in extremely preterm infants. New England Journal of Medicine, 364: Terrin, G., Passariello, A., De, C.M., et al. (2012). Ranitidine is associated with infections, necrotizing enterocolitis, and fatal outcome in newborns. Pediatrics, 129(1): e40-e45. van der Pol, R., Smits, M., van Wijk, M., Omari, T., et al. (2011). Efficacy of proton-pump inhibitors in children with gastroesophageal reflux disease: A systematic review. Pediatrics, 27: Session Outline See handout on the following pages. GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 1 of 14

2 Understanding How to Use Statistics to Evaluate an Article Reese H Clark, MD Disclosure Statements: Reese H. Clark, MD I have no relevant financial relationships to disclose or conflicts of interest to resolve. I will be discussing the H2 Blockers and Proton Pump Inhibitors (PPIs). These drugs are not approved by the FDA for use in neonates. I will discuss why the off-label use of any drug is dangerous. Goals To review how statistics influence the interpretation of important articles We will review the results of three clinical studies that should influence clinical care and evaluate them with using statistical techniques Most Important Concepts Power our ability to detect a difference if a difference exist. Small studies have no power to prove safety. Composite outcomes can lead to confusing results The absolute difference is more important than the relative difference. Decreasing something from 2 to 1% is a 50% reduction but so is 50% to 25%. Decreasing something from 2 to 1% alters the outcome for 1 patient in 100. Decreasing something from 50 to 25% decreases the rate by 25 in 100. Most Important Concepts Study types worst to best. Consensus expert opinion not based on RTC, case series, retrospective uncontrolled, retrospective ti with controls; prospective uncontrolled; prospective with controls; randomized control trial. Only prospective randomized control trials can establish cause and effect. All others can only suggest associations Selection bias is hard to correct. Site variation in practice and outcome influences the results. Biological plausibility do the results make sense. GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 2 of 14

3 Basic Descriptive Statistics Central Tendency Mean Mean = ( x) / N Median Value at which 50% of values are greater and 50% are smaller Mode Most common value Target ranges of oxygen saturation in extremely preterm infants. N Engl J Med 2010; 362: Carlo WA, et al. Support Study Group Support Trial Methods Randomized trial with a 2-by-2 factorial design to compare target ranges of oxygen saturation of 85 to 89% or 91 to 95% All infants were also randomly assigned to continuous positive airway pressure or intubation and surfactant infants who were born between 24 weeks 0 days and 27 weeks 6 days of gestation were enrolled The primary outcome was a composite of severe retinopathy of prematurity (defined as the presence of threshold retinopathy, the need for surgical ophthalmologic intervention, or the use of bevacizumab), death before discharge from the hospital, or both. Factorial Design What is a factorial design? In statistics, a full factorial experiment is an experiment whose design consists of two or more factors, each with discrete possible values or "levels", and whose experimental units take on all possible combinations of these levels across all such factors. Factorial designs allow clinical scientist to study the effect of each factor on the response variable, as well as the effects of interactions between factors on the response variable. GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 3 of 14

4 What are the factors being studied? Study Factors Or Variables Being Studied Support modes CPAP only compared to Early Surfactant Oxygen saturation targets Low (85 to 89%) compare to High (91 to 95%) Confounding Variable What are confounding variables? A confounding variable (also confounding factor, lurking variable, a confound, or confounder) is an extraneous variable in a statistical model that correlates (positively or negatively) with both the dependent variable and the independent variable. The methodologies of scientific studies must account for these variables. Experimental design (randomization) can achieve control of the confounding variable by creating a balance of the confounding event in treated and control patients Confounding Variables Statistical test (multivariate correction) can also be use to account (correct) for confounding variables The goal is to avoid a false positive (Type I) error; an erroneous conclusion that the dependent variables are in a causal relationship with the independent variable. We can establish associations retrospectively but causation has to be determined prospectively Potential Confounding Variables EGA Birth Weight Chorioamnionitis Antenatal Steroids Decision about weaning support Site of care GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 4 of 14

5 Composite Outcome What is a composite outcome? Combines one or more outcomes In the study we are discussing a composite of severe retinopathy of prematurity (defined as the presence of threshold retinopathy, the need for surgical ophthalmologic intervention, or the use of bevacizumab), death before discharge from the hospital, or both. Gloria Cordoba. Definition, reporting, and interpretation of composite outcomes in clinical trials: systematic review. BMJ 2010; 341:c3920 The use of composite outcomes is problematic. Components are often unreasonably combined, inconsistently defined, and inadequately reported. These problems will leave many readers confused, often with an exaggerated perception of how well interventions work. Problem with Combining Death and Other Outcomes. Factors associated with death are distinct and different from factors associated with ROP EGA and Birth Weight and severity of illness influence both Site variability in a decision to continue to offer care influences death significantly How oxygen is used influences ROP Reason for using composite Why use a composite outcome? Better measure of overall outcome Correct for competing outcomes Can not have ROP if you die. So you could find less ROP because you had more death. Solution is to report each component of the composite outcome GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 5 of 14

6 SUPPORT Study Group. N Engl J Med 2010;362: Does the Support Study control for confounding variables? What else do you want to know? When planning a trial the treatment groups need to be distinctively defined. Were they? Oxygen Groups Blinding was maintained with the use of electronically altered pulse oximeters (Masimo Radical Pulse Oximeter) that showed saturation levels of 88 to 92% for both targets t of oxygen saturation, ti with a maximum variation of 3%. For example, a reading of 90% corresponded to actual levels of oxygen saturation of 87% in the group assigned to lower oxygen saturation (85 to 89%) and 93% in the group assigned to higher oxygen saturation (91 to 95%). Actual Median Oxygen Saturation with Oxygen Supplementation in the Two Treatment Groups Were the two oxygen groups distinctive? SUPPORT Study Group of the Eunice Kennedy Shriver NICHD Neonatal Research Network. N Engl J Med 2010;362: What does this mean? Why median? GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 6 of 14

7 Efficacy or Equivalency or Non-inferiority? What are the components of statistical certainty? Efficacy trials are directed at proving that one therapy is better than another with 95% confidence Equivalence implies that the two therapies produce the same outcome If one therapy reduces health care cost, then we may only want to show that the two approaches produce similar outcomes Non-inferiority are done to show that patients treated with X do no worse than those treated with Y. Like efficacy but only one tail-test are used. Required sample size is smaller. Type 1 Error (alpha) A type I error, also known as a false positive, occurs when a statistical test rejects a true null hypothesis (null hypothesis states there is no difference). Concludes there is a difference between groups when there is not (Treatment worked when it really does not work). False positive (test says there is disease when there is not) The rate of the type I error is denoted by the Greek letter alpha (α) and usually equals the significance level (or size) of a test. Type II Error (beta) A type II error, also known as a false negative, occurs when the test fails to reject a false null hypothesis (no difference). Concludes that there is no difference between groups when in fact there is a difference. (Concludes the therapy did not work when in fact it really does work) False negative (test shows there is no disease when there is in fact there is disease) The rate of the type II error is denoted by the Greek letter beta (β). Power Probability of rejecting the null hypothesis when it is false Probability of detecting a difference if it really exists Power = 1 beta error rate (Type II error) Sample Size Calculations Dependent on: The absolute event rate in the population being studied The absolute difference between the two groups How certain you want to be in the measured difference GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 7 of 14

8 Effect of Sample Size On Confidence Interval and Probability that the Proportions are Different p=0.6 p=0.16 p<0.001 What is a relative risk and what is a odds ratio and how are they different? Does it really matter? RR vs. OR RR vs. OR Primary Outcome Low O2 High O2 ROP/Died A B Lived No C D ROP RR=E/F E=Percent With Outcome A/(A+C) F=Percent With Outcome D/(B+D) OR=(A/C)/(B/D) Definitions Are both the relative and absolute difference both important? Relative Risk -- The probability (risk) of developing ROP or dying if the patient was assigned to the low O2 group compared to low O2 group (% ROP or death low O2 group/ %ROP or death in high O2) Odds Ratios -- The odds ratio compares the relative odds of ROP or death in each group. GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 8 of 14

9 Definitions Relative Risk of Outcome Confidence intervals -- How certain are you that the observation falls within your measured result. Usually the number is 95% CI Standard Deviation -- a measure of average variance from the mean (Square root of {Sum(individual values - mean value) 2 /number of measurements} Standard Error of the Mean -- STD/Square root of the sample size. Decreased ROP or death Good confidence Effect but no Confidence No effect Increased ROP or death Good confidence Relative Risk of Outcome Relative Risk of Outcome The new therapy is better and no risker Effect but no confidence Equivalent but no confidence Equivalent Good confidence Relative Risk of Outcome Results (N Engl J Med 2010;362: ) The rates of severe retinopathy or death did not differ significantly between the lower- and the higheroxygen-saturation group (28.3% vs. 32.1%, respectively; RR, 0.90; 0.76 to 1.06; P=0.21, Absolute difference = 3.8%). Death before discharge occurred more frequently in the lower-oxygen-saturation oxygen group (19.9% 9% vs. 16.2%; RR, 1.27; 1.01 to 1.6; P=0.04, Absolute difference = 3.7%) Severe retinopathy among survivors occurred less often in this group (8.6% vs. 17.9%; RR, 0.52; 0.37 to 0.73; P<0.001, Absolute difference = 9.3%). There were no significant differences in the rates of other adverse events. Major Outcomes Why 2 definitions of death SUPPORT Study Group of the Eunice Kennedy Shriver NICHD Neonatal Research Network. N Engl J Med 2010;362: GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 9 of 14

10 Outcomes Boost II P<0.001, Adjusted RR = 0.52 P=0.085 Unadjusted Fisher s Exact Adjusted RR = 1.27 The U.K., Australian, and New Zealand BOOST II trials were designed to compare SpO2 targets of 85 to 89% versus 91 to 95%, with a primary outcome of survival without disability at 2 years corrected for gestation. A prospective meta-analysis of all the neonatal oxygen trials is planned. Boost II NEJM 364(17): 1680, April 28, 2011 Crosses 1 Among all 3631 infants, those randomly assigned to an SpO2 of 91 to 95% had a higher survival rate than those assigned to an SpO2 of 85 to 89% (mortality, 17.3% vs. 14.4%; relative risk for survival associated with higher SpO2 target, 1.21; 99.73% CI, 0.96 to 1.52; P = 0.015). 015) Among the 1055 infants in the U.K. and Australian trials who were treated after the change in the calibration algorithm, survival differences were greater (mortality, 21.8% vs. 13.3%; relative risk for survival associated with higher SpO2 target, 1.65; 99.73% CI, 1.09 to 2.49; P<0.001; test for interaction for pooled comparisons of old vs. new algorithm, P = 0.006). ROP and long term follow-up results not reported. NNT How do you calculate the number need to treat and number need to harm? The number needed to treat (NNT) is an epidemiological measure used in assessing the effectiveness of a health-care intervention, typically a treatment with medication. The NNT is the average number of patients who need to be treated to prevent one additional bad outcome (i.e. the number of patients that need to be treated for one to benefit compared with a control in a clinical trial). It is defined as the inverse of the absolute risk reduction (i.e. 1/risk reduction) GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 10 of 14

11 NNH The number needed to harm (NNH) is an epidemiological measure that indicates how many patients need to be exposed to a risk-factor over a specific period to cause harm in one patient that would not otherwise have been harmed. It is defined as the inverse of the attributable risk. Intuitively, the lower the number needed to harm, the worse the risk-factor. NNH is computed with respect to "exposure" and "non-exposure", and can be determined for raw data or for data corrected for confounders. A defined endpoint has to be specified. NNH is computed as 1/(p exposure-p non-exposure). NNT and NNH calculations NNT for severe ROP = 1/0.093 About 11 NNH for death 1/0.037 About 27 NNH/NNT ratio =27/11=2 This estimates that you should expect one additional death for every cases of severe ROP you prevented. The major problem with these calculation is it assumes we have measured with certainty the absolute differences. Across Site Variability Site Variation Most sites randomize within center Each site is a mini-trial Differences in care are variable within sites and may effect the efficacy of the drug being studied within that site steroids nosocomial sepsis nutrition saturation targets Site Variability in Proportion of Neonates Alive & Off Oxygen Treatment Study ( All Treated ) Difference between Group 1 and Group2 Alive and off oxygen PMA36 for Treatment Study Surfactant 2 better Surfactant 1 better GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 11 of 14

12 Approach to Infants Born at 22 to 24 Weeks Gestation: Relationship to Outcomes of More-Mature Infants. Brian Smith et al. Pediatrics 2012;129;e1508 Study included 3631 infants 22 to 24 weeks gestation and 5227 infants 25 to 27 weeks gestation. Among the 22- to 24-week infants, use of antenatal corticosteroids ranged from 28% to 100%, cesarean delivery from 13% to 65%, and resuscitation from 30% to 100% by center. Centers with higher rates of antenatal corticosteroid use in 22- to 24-week infants had reduced rates of death, death or retinopathy of prematurity, death or late-onset sepsis, death or necrotizing enterocolitis, and death or neurodevelopmental impairment in 25- to 27-week infants. A C Interventions in 22- to 24-week infants by center. A, Antenatal corticosteroid use by center; B, cesarean delivery rate by center; C, resuscitation rate by center. B Center variation in outcomes for 25- to 27-week infants (median, range). Conclusions This study suggests that physicians willingness to provide care to extremely low gestation infants as measured by frequency of use of antenatal corticosteroids is associated with improved outcomes for more mature infants. Pediatrics 2012;129:e1508 e1516 Site Effect The impact of site is as important as any therapy we are studying. Retrospective Studies Problems Not based on current data Selection bias Confounded by unmeasured variables GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 12 of 14

13 Efficacy of Proton-Pump Inhibitors in Children With Gastroesophageal Reflux Disease: A Systematic Review. Authors searched PubMed, Embase, and the Cochrane Database of Systematic Reviews for randomized controlled trials and crossover studies investigating efficacy and safety of PPIs in children aged d0t to 18 years with ithgerd Infants - PPIs were more effective in 1 study (compared with hydrolyzed formula) not effective in 2 studies equally effective in 2 studies (compared with placebo) for the reduction of GERD symptoms. CONCLUSIONS: PPIs are not effective in reducing GERD symptoms in infants. Terrin G, Passariello A, De CM et al. Ranitidine is associated with infections, necrotizing enterocolitis, and fatal outcome in newborns. Pediatrics 2012;129(1):e40-e45. Gastric acidity is a major nonimmune defense mechanism against infections. (Biological plausibility) The objective of this study was to investigate whether ranitidine treatment in VLBW infants is associated with (not the cause of) an increased risk of infections, NEC, and fatal outcome. van der Pol RJ, Pediatrics. 2011;127: Terrin G, Passariello A, De CM et al. Ranitidine is associated with infections, necrotizing enterocolitis, and fatal outcome in newborns. Pediatrics 2012;129(1):e40-e45. Newborns with birth weight between 401 and 1500 g or gestational age between 24 and 32 weeks, consecutively observed in neonatal intensive care units, were enrolled in a multicenter prospective observational study. The rates of infectious diseases, NEC, and death in enrolled subjects exposed or not to ranitidine were recorded. How did the authors try to avoid selection bias? Newborns with birth weight ranging between 401 and 1500 g or gestational age between 24 and 32 weeks, consecutively observed in 4 Italian NICUs Exclusion criteria were immunodeficiency, malformations, evidence of infections or NEC before enrollment, critical conditions (blood ph, 6.8, or hypoxia with persistent bradycardia for at least 1 hour), ranitidine therapy for fewer than 7 days, and hospitalization for fewer than 8 weeks. Evaluated 2 cohorts of very low birth weight (VLBW) newborns: those exposed or not exposed to ranitidine treatment. Terrin G, et al. Pediatrics 2012;129(1):e40-e45. Problems Did they succeed? Indications, dosage, and duration of ranitidine treatment were decided by the caregivers of each NICU, who were unaware of the study aims. Dose and duration did not alter the results reported Terrin G, et al. Pediatrics 2012;129(1):e40-e45. Terrin G, et al. Pediatrics 2012;129(1):e40-e45. GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 13 of 14

14 Terrin G, Passariello A, De CM et al. Ranitidine is associated with infections, necrotizing enterocolitis, and fatal outcome in newborns. Pediatrics 2012;129(1):e40-e45. Terrin G, Passariello A, De CM et al. Ranitidine is associated with infections, necrotizing enterocolitis, and fatal outcome in newborns. Pediatrics 2012;129(1):e40-e45. Multicenter cohort of 274 VLBW infants: 91 had taken ranitidine and 183 had not. The risk of infection was higher; 34/91 (37%) of infants exposed to ranitidine and 18/183 (10%) of the infants not exposed to ranitidine contracted infections (OR=5.5 [2.9, 10.4], P < 0.001). The risk of NEC was higher in ranitidine-treated VLBW infants (9.8 vs 1.6%; OR=6.6 [ ], P = 0.003) Mortality rate was significantly higher in newborns receiving ranitidine (9.9% vs. 1.6%, P =.003). 40% 35% 30% 25% 20% 15% 10% 5% 0% Exposed to Ranitidine Not exposed Infection NEC Mortality Terrin G, et al. Pediatrics 2012;129(1):e40-e45. Problem The best available evidence, however, is not always sound or valid evidence. Sometimes, when faced with a collection of reports that do not constitute good evidence, attempts to choose the best evidence become pointless; in this case, a statement of no good evidence is preferable. Ambalavanan N et al. Clin Perinatol 2003; 30: Terrin G, et al. Pediatrics 2012;129(1):e40-e45. GS3: HOW TO USE STATISTICS TO EVALUATE AN ARTICLE Page 14 of 14