the Orthopaedic forum Is There Truly No Significant Difference? Underpowered Randomized Controlled Trials in the Orthopaedic Literature

Transcription

1 2068 COPYRIGHT Ó 2015 BY THE JOURNAL OF BONE AN JOINT SURGERY, INCORPORATE the Orthopaeic forum Is There Truly No Significant ifference? Unerpowere Ranomize Controlle Trials in the Orthopaeic Literature Leath Abullah, BS, aniel E. avis, M, MSc, Peter. Fabricant, M, MPH, Keith Balwin, M, MPH, MSPT, an Surena Namari, M, MSc Backgroun: Ranomize controlle trials (RCTs) are consiere the gol stanar in evience-base meicine. Unerpowere RCTs that escribe comparative outcomes without significance are of questionable benefit. The present stuy hypothesizes that a substantial proportion of RCTs in the orthopaeic literature that o not note significant ifferences between groups are inaequately powere. Methos: Using the ISI Web of Science atabase, we searche all English-language journals in the orthopaeic category for RCTs publishe from January 2012 to ecember Qualifying articles were analyze with regar to whether the null hypothesis was rejecte (a positive stuy) for the primary outcome or if it was not (a negative stuy), whether a power analysis was performe, an whether the stuy was aequately powere. We performe a power analysis base on the primary outcome or outcomes of interest for the stuies that i not escribe a power analysis. Results: After inclusion an exclusion criteria were applie, 456 RCTs were selecte for complete review. Of those stuies, 215 (47.1%) ha negative finings an 241 (52.9%) ha positive finings for primary outcomes. Twenty-five stuies that faile to reject the null hypothesis note inaequate power in the stuy. On the basis of our own power analyses, we foun an aitional thirty-five negative stuies without power calculations to be unerpowere. Sixty (27.9%) of the 215 negative stuies were unerpowere. Following binary logistic regression, only the journal impact factor was a significant preictor of whether a stuy was unerpowere. Conclusions: If an RCT lacks aequate statistical power to ientify a clinically meaningful absence of a ifference between groups, there is an unacceptable risk of inappropriately failing to reject the null hypothesis. The present stuy continue Peer Review: This article was reviewe by the Eitor-in-Chief an one eputy Eitor, an it unerwent bline review by two or more outsie experts. It was also reviewe by an expert in methoology an statistics. The eputy Eitor reviewe each revision of the article, an it unerwent a final review by the Eitor-in-Chief prior to publication. Final corrections an clarifications occurre uring one or more exchanges between the author(s) an copyeitors. isclosure: None of the authors receive payments or services, either irectly or inirectly (i.e., via his or her institution), from a thir party in support of any aspect of this work. One or more of the authors, or his or her institution, has ha a financial relationship, in the thirty-six months prior to submission of this work, with an entity in the biomeical arena that coul be perceive to influence or have the potential to influence what is written in this work. In aition, one or more of the authors has a patent or patents, planne, pening, or issue, that is broaly relevant to the work. No author has ha any other relationships, or has engage in any other activities, that coul be perceive to influence or have the potential to influence what is written in this work. The complete isclosures of Potential Conflicts of Interest submitte by authors are always provie with the online version of the article. J Bone Joint Surg Am. 2015;97:

2 2069 IS THERE TRULY NO SIGNIFICANT IFFERENCE? foun that a sizable proportion of RCTs in orthopaeic surgery in which the null hypothesis is rejecte are inaequately powere. Researchers shoul consier this when esigning clinical trials, an journal eitors an reviewers shoul be wary of unerpowere RCTs when consiering manuscripts for publication. A well-esigne ranomize controlle trial (RCT) is generally consiere the gol stanar in clinical research an is assigne the highest level of evience (Level I) for clinical research. These types of stuies are often highly cite for evience-base clinical ecisions an are highly value by journal eitors. The unerlying conclusions of rejecting or retaining the null hypothesis are particularly epenent on statistical power. A priori power analyses are performe before beginning an RCT in orer to establish the sample size require to aequately ifferentiate between a true lack of clinically meaningful ifference rather than a false-negative fining (when a ifference may exist, but the stuy is unerpowere to etect it) (Fig. 1). By convention, 80% power is consiere aequate. In other wors, there is a 20% chance of falsely etermining there is no ifference between stuy groups. This threshol is calle the type-ii or beta error 1. Beyon the methoological importance of achieving aequate power, multiple stuies have propose that unerpowere RCTs are unethical 2,3. If an RCT falsely reports no clinical ifference, clinical ecision making coul be misguie. It has been propose that many stuies in the orthopaeicliteraturehaveinaequatesamplesizesanthus inaccurately escribe an absence of clinically meaningful ifferenceswhenonemayinfactexist 4. The purpose of the current stuy was to systematically evaluate RCTs in a representative sample from the recent orthopaeic literature an etermine (1) the prevalence of power analyses in the orthopaeic literature, (2) the proportion of negative stuies that were aequately powere compare with those not aequately powere, an (3) the factors that were preictive of aequate statistical power in negative stuies. We hypothesize that a substantial proportion of RCTs in the orthopaeic literature that o not note significant ifferences between groups are inaequately powere. Materials an Methos Search Strategy The methoology utilize in the present stuy has been previously use in the rheumatology literature 5. In orer to ientify orthopaeic RCTs, we searche the ISI Web of Science uner the orthopaeic category for English-language stuies, publishe from January 2012 to ecember 2013, containing the terms ranom, ranomize, ranomization, ranomly, controlle, or control. The resulting titles were then reviewe by one author (.E..) to eliminate uplicates or stuies that were not RCTs on the basis of the title. The abstracts were then further reviewe by two authors (.E.. an S.N.), an nonprospective RCTs were rejecte. isagreements between these two authors were reviewe by the entire team an resolve by consensus. Pilot stuies an seconary reviews of previous RCT ata were exclue. The full texts of the remaining stuies were thoroughly reviewe to confirm that the stuies were prospective RCTs, an they were assesse accoring to the criteria liste below. Evaluation for RCT Quality The quality of each stuy was etermine by the calculation of the Jaa score for each investigation 4. The Jaa score was calculate by giving 1 point for the presence of the following components: ranomization, ouble-blining, an a escription of withrawn subjects. Aitionally, by reviewing the acknowlegment section an statistics section of each article, we note whether a statistician ha been involve in the paper or if one of the primary authors hel an MPH or Ph in statistics. ata Extraction The stuies were reviewe by one author (L.A.) for ata extraction. The following variables were recore: significant presence of an effect of treatment on primary outcome(s) (i.e., a positive stuy), no significant ifference (i.e., a negative stuy), level of evience, stuy type, subject number, pathology stuie, outcome measures, statistical testing methos, reporte statistical power (or beta error), reporte sample-size aequacy, statistician involvement, subspecialty, an journal impact factor. Positive stuies were those in which a significant result (p < 0.05) was note in the primary outcome of interest, an therefore a type- II error coul not have been committe. Negative stuies were those in which the authors faile to reject the null hypothesis (Fig. 1). Post Hoc Power Analysis The primary outcome variable(s) in each stuy were etermine to be those explicitly state by the authors in the specific aims. Articles claiming no significant Fig. 1 A2 2 table illustrating statistical concepts an terminology. A positive stuy was one in which a significant ifference was foun in the primary outcome of interest; this enotes either a true ifference or a type-i error. A negative stuy was one in which the authors faile to reject the null hypothesis, either correctly or as the result of a type-ii error.

3 2070 IS THERE TRULY NO SIGNIFICANT IFFERENCE? TABLE I A Summary of All 456 Ranomize Controlle Trials Reviewe in the Current Stuy Characteristic No. (%) of Stuies Stuy conclusions Positive finings 241 (52.9) Negative finings 215 (47.1) Stuy level I 256 (56.1) II 199 (43.6) III* 1 (0.2) Stuy esign Crossover 5 (1.1) Parallel 451 (98.9) Subspecialty Ault reconstruction 119 (26.1) Trauma 91 (20.0) Rehabilitation 58 (12.7) Sports 50 (11.0) Spine 46 (10.1) Shouler an elbow 27 (5.9) Miscellaneous 22 (4.8) Han 18 (3.9) Foot an ankle 12 (2.6) Peiatrics 9 (2.0) Poiatry 3 (0.7) Tumor 1 (0.2) *While RCTs are by efinition Level I or II, one stuy self-reporte as a Level-III stuy an thus was reporte here as such. ifference between stuy groups in terms of primary outcome variables were reviewe for the presence of a power analysis. If one was not performe in the stuy, we performe a power analysis base on the primary outcome of interest utilizing one of the following methos, as appropriate. If a publishe minimal clinically important ifference (MCI) existe for the primary outcome of interest, we use the MCI to calculate a power analysis. The MCI is the smallest ifference in an outcome score that a patient perceives as beneficial. MCIs can provie the basis for etermining if significant ifferences in outcomes after treatment are clinically relevant. For example, Tashjian et al. etermine that patients with rotator cuff isease who are treate without surgery an have a 2-point change in the Simple Shouler Test score or a 12 to 17-point change in the American Shouler an Elbow Surgeons score experience a clinically important change in self-assesse outcome 6. If no MCI existe for the primary outcome of interest or if an article ha multiple outcomes, an MCI of one-half of a stanar eviation was use on the basis of ata suggesting that one-half of a stanar eviation correlates well with an MCI in health-relate quality-of-life stuies 7,8. The ifference between means was calculate, an the stanar eviation was calculate using an effect size of If an article investigate multiple primary outcomes, a Bonferroni correction was use. By iviing the conventionally accepte alpha (p < 0.05) by the number of primary outcomes an then converting that value to the appropriate two-sie z-score, a new, smaller p value threshol was calculate. Statistical Methos Continuous variables were compare using inepenent sample t tests. Binary or categorical variables were compare with a Pearson chi-square test for inepenence. A receiver operating characteristic (ROC) analysis was performe on continuous variables foun to be significant in orer to ichotomize them for the purpose of regression analysis. Binary logistic regression analysis was use to etermine inepenent preictors of whether a stuy was unerpowere or truly negative. We initially inclue all variablesan then use backwar likelihoo ratio methoology to eliminate variables in a stepwise fashion. A minimum criterion for inclusion in the final moel was p < All statistics were calculate with SPSS statistical software (SPSS). Source of Funing No internal or external funing was obtaine for the preparation of this manuscript. Results The initial search resulte in 1008 titles. After inclusion an exclusion criteria were applie, 456 full-text articles were reviewe (Fig. 2). There was an equal istribution between 2012 an 2013, an the inclue articles covere all areas of musculoskeletal care. Of the 456 articles inclue, 241 stuies (52.9%) note significant ifferences in the primary outcome between stuy groups (positive stuies) an 215 articles (47.1%) note no ifference (negative stuies) (Table I). Of the 215 negative stuies, twenty-five ha performe a power analysis an note that their stuy was unerpowere. An aitional forty-four (20.5%) of the 215 negative stuies ha not performe a power analysis. Our power analysis reveale that thirty-five of them ha inaequate sample size an were inaequately powere. As a result, sixty (27.9%) of 215 negative stuies were unerpowere an thus were unable to etermine if their lack of a significant ifference between stuy groups represente a lack of clinically important groupwise ifferences or if the stuy merely misse a ifference that i exist but was overlooke because of low statistical power (Table II). Of all 456 stuies, sixty-nine (15.1%) were negative stuies that ha inaequate or unreporte power. The mean sample size, stuy esign, mean length of follow-up, mean Jaa score, statistician involvement, an mean journal impact factor are summarize in Table III. There were no ifferences note in Jaa scores between RCTs with positive finings an those with negative finings (2.071 versus 2.060; p = 0.860). The mean impact factor of the journals TABLE II Summary of 456 RCTs in Which an a Priori Power Analysis Was Reporte an Those That Were Unerpowere or Ha No Power Analysis Reporte Power Analysis No. (%) of Stuies Power analysis reporte Yes aequate 299 (65.6) Yes unerpowere 34 (7.5) No 123 (27.0) Unerpowere stuies or no power analysis reporte (n = 157) Positive finings (rejecte the null) 88 (56.1) Negative finings (faile to reject 69 (43.9) the null)

4 2071 IS THERE TRULY NO SIGNIFICANT IFFERENCE? TABLE III A Summary of 215 Negative Stuies with Power Note or Calculate an Comparisons of the Unerpowere an Aequately Powere Negative Stuies Negative Stuies Unerpowere (N = 60) Aequate Power (N = 155) P Value* Stuy level (no.[%]) I 12 (20.0) 110 (71.0) <0.001* II 48 (80.0) 45 (29.0) <0.001* Mean no. of subjects (range) 72.0 (20-200) ( ) Mean follow-up (range) (mo) 26.3 (0-144) 18.4 (0-149) ouble-blin esign (no.[%]) Yes 12 (20.0) 42 (27.1) No 48 (80.0) 113 (72.9) Mean Jaa score (an stan. ev.) ± ± Input from statistician (no.[%]) Yes 12 (20.0) 49 (31.6) 0.09 No 48 (80.0) 106 (68.4) 0.09 Mean journal impact factor (range) 2.19 ( ) 2.6 ( ) *P values calculate using a two-taile Stuent t test for continuous variables an chi-square for count variables. P < whose articles were inclue in this stuy was The mean impact factor was slightly higher for all negative stuies (2.5) compare with positive stuies (2.3; p = 0.042). The mean impact factor for 155 negative stuies with aequate power was significantly higher than the mean for the sixty negative stuies that were unerpowere (2.6 versus 2.19; p = 0.001). In a binary logistic regression moel, only the journal impact factor was ientifie as an inepenent preictor of whether a negative stuy was unerpowere (p = 0.002). For each increase of 1 point in the impact factor, the os of a stuy being aequately powere increase by 1.82 times (95% confience interval [CI]: 1.26, 2.63). An ROC analysis showe an optimal inflectionpointofanimpactfactorof2.15.journalswithan impact factor of 2.15 ha a 2.35 (95% CI: 1.26, 4.37) times Fig. 2 Flowchart illustrating the categorization of 456 total RCTs inclue in the stuy. greater os of publishing an aequately powere negative stuy compare with journals with impact factors below this marker. iscussion As technological avancement continues an the cost of health care continues to increase, the importance of creating highquality clinical evience through RCTs is paramount to supporting best practices. Conucting RCTs that are unerpowere is a concern both because such stuies may inaccurately influence the clinical ecision-making process an because the results may unethically place patients at risk without the benefit of eluciating a meaningful conclusion 3. The present stuy of RCTs in the English-language orthopaeic literature foun that 27.9% of articles that o not fin a ifference in their primary outcome are in fact unerpowere to make such conclusions. Unerpowere RCTs are not limite to the orthopaeic literature. An analysis of the rheumatology literature by Keen et al. performe a similar examination of RCTs 5. Of 205 Phase- III RCTs in the rheumatology literature, eighty-six stuies (41.9%) were negative or ineterminate. Thirty-seven stuies (18.0%) note sample size calculations, an thirty-three (16.1%) were aequately powere. Forty-nine of the negative or ineterminate stuies i not escribe a power analysis. Of those negative stuies, ten were aequately powere, twentyseven were unerpowere, an twelve i not provie enough ata to perform an analysis. The results of the current stuy are consistent with those finings. Similarly, in the oncology literature, Bear et al. examine RCTs with negative finings that were presente at the annual meetings of the American Society of Clinical Oncology from 1995 to Four hunre an twenty-three negative stuies unerwent a post hoc analysis performe for aequate power for etecting small, meium,

5 2072 IS THERE TRULY NO SIGNIFICANT IFFERENCE? an large effect sizes. They foun that 89.4% of the stuies were unerpowere to etect a small ifference between groups, 67.4% were unerpowere to etect a meium ifference between groups, an 44.9% were unerpowere to etect a large ifference between groups. Improperly performe RCTs have been examine in the orthopaeic literature, particularly with respect to the Consoliate Stanars of Reporting Trials (CONSORT) criteria 10. The CONSORT criteria were efine in a Special Communication in The Journal of the American Meical Association in 1996 an inclue a twenty-one-item checklist pertaining to the methos, results, an conclusion sections of RCT stuy esigns 11. The goal was to help guie researchers in planning meaningful an complete stuy esigns. These criteria an checklists were further upate an amene in 2010 as it is a continually evolving guieline 12. In a 2011 review of the spine literature, Naunheim et al. examine thirty-two RCTs in three prominent spine journals, with the goal of evaluating their aherence to the CONSORT criteria 13. Interestingly, they foun that, when scoring the abstracts versus the boy of the text, there was a iscrepancy of 78% aherence versus 60%, respectively. McCormick et al. reviewe RCTs focuse on rotator cuff isorers using the CONSORT criteria an the Jaa score to etermine the quality of the stuies 10.Infifty-four stuies ientifie in the top six orthopaeic journals with the highest impact factors, they foun that 66% were consiere high quality (a Jaa score of 3), an the mean CONSORT criteria score was 70%. There was an association between a higher Jaa score an stuies that ha high CONSORT criteria scores 10. The authors foun that, of the eficiencies liste, the absence of a power analysis in 64.8% of the stuies was one of the most common. Importantly, we foun that unerpowere stuies were self-reporte as lower levels of evience an were publishe in journals with lower impact factors. This inicates that the review process of higher-impact journals may place greater emphasis on the power analyses of submitte articles. The present stuy oes have limitations. When calculating power for negative stuies in which no power calculation was reporte, we use one-half of a stanar eviation when MCI information was not available. This methoology is accepte in measures of health-relate quality of life, but may not be as accurate as power analyses performe for specific ata sets 8. Ieally, as suggeste by the International Committee of Meical Journal Eitors, primary outcomes an power analyses woul have been available in RCT registries (e.g., clinicaltrials. gov). Unfortunately, no stuy that neglecte to note a power calculation was foun in any RCTregistry 14. Aitionally, while the Jaa scale is a well-accepte tool to evaluate RCTs, further analysis using the CONSORT criteria woul have aie in etermining the quality of the reviewe stuies. Finally, in the present investigation, stuies that showe a significant ifference for primary outcome variables were consiere to be aequately powere. It is possible that a portion of these stuies woul have been unerpowere if a significant ifference between groups ha not been shown. In conclusion, the present stuy provies a comprehensive analysis of RCTs reporte in the orthopaeic literature over a recent two-year perio. These finings are similar to previously raise concerns in other meical specialties that stuies with high levels of evience frequently are not sufficiently powere to accurately raw conclusions from the reporte ata 10,13,15. This leas to both ethical ilemmas of ata reporting as well as the clinical ifficulty of incorporating clinical research into aily practice. It is critical to unerstan that when reporting the lack of clinical ifferences between stuy groups, aequate sample size must be ensure in orer to generate an acceptable level of type-ii error; yet, this was absent in 27.9% of the analyze stuies. As the clinical literature in orthopaeics continues to expan, researchers must strive to achieve aequate power an thus a relevant finings (both positive an negative) on which to base clinical ecisions. In aition, orthopaeic surgeons shoul pay particular attention to whether stuies are aequately powere to support their negative finings an journal eitors an reviewers shoul be wary of unerpowere RCTs when consiering manuscripts for publication. n Leath Abullah, BS rexel University College of Meicine, 2900 West Queen Lane, Philaelphia, PA aniel E. avis, M, MSc epartment of Orthopaeic Surgery, Thomas Jefferson University Hospital, 1025 Walnut Street, Suite 516, Philaelphia, PA Peter. Fabricant, M, MPH Keith Balwin, M, MPH, MSPT epartment of Orthopaeic Surgery, Chilren s Hospital of Philaelphia, 2 Woo Builing, Philaelphia, PA Surena Namari, M, MSc The Rothman Institute at Thomas Jefferson University, 925 Chestnut St, 5th Floor, Philaelphia, PA aress: Surena.Namari@rothmaninstitute.com References 1. Portney LG, Watkins MP. Founations of clinical research: applications to practice. 3r e. Upper Sale River, NJ: Pearson/Prentice Hall; Ewars SJ, Lilfor RJ, Braunholtz, Jackson J. Why unerpowere trials are not necessarily unethical. Lancet Sep 13;350(9080): Halpern S, Karlawish JHT, Berlin JA. The continuing unethical conuct of unerpowere clinical trials. JAMA Jul 17;288(3): Jaa AR, Moore RA, Carroll, Jenkinson C, Reynols J, Gavaghan J, McQuay HJ. Assessing the quality of reports of ranomize clinical trials: Is blining necessary? Control Clin Trials Feb;17(1):1-12.

6 2073 IS THERE TRULY NO SIGNIFICANT IFFERENCE? 5. Keen HI, Pile K, Hill CL. The prevalence of unerpowere ranomize clinical trials in rheumatology. J Rheumatol Nov;32(11): Tashjian RZ, eloach J, Green A, Porucznik CA, Powell AP. Minimal clinically important ifferences in ASES an Simple Shouler Test scores after nonoperative treatment of rotator cuff isease. J Bone Joint Surg Am Feb;92 (2): Cohen J. Statistical power analysis for the behavioral sciences. 2n e. Hillsale, NJ: Lawrence Erlbaum Associates; Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-relate quality of life: the remarkable universality of half a stanar eviation. Me Care May;41(5): Bear PL, Krzyzanowska MK, Pintilie M, Tannock IF. Statistical power of negative ranomize controlle trials presente at American Society for Clinical Oncology annual meetings. J Clin Oncol Aug 10;25(23): McCormick F, Cvetanovich GL, Kim JM, Harris J, Gupta AK, Abrams G, Romeo AA, Provencher MT. An assessment of the quality of rotator cuff ranomize controlle trials: utilizing the Jaa score an CONSORT criteria. J Shouler Elbow Surg Sep;22(9): Epub 2013 Mar Begg C, Cho M, Eastwoo S, Horton R, Moher, Olkin I, Pitkin R, Rennie, Schulz KF, Simel, Stroup F. Improving the quality of reporting of ranomize controlle trials. The CONSORT statement. JAMA Aug 28;276(8): Schulz KF, Altman G, Moher ; CONSORT Group. CONSORT 2010 Statement: Upate guielines for reporting parallel group ranomise trials. J Clin Epiemiol Aug;63(8): Epub 2010 Mar Naunheim MR, Walcott BP, Nahe BV, Simpson AK, Agarwalla PK, Coumans JV. The quality of ranomize controlle trial reporting in spine literature. Spine (Phila Pa 1976) Jul 15;36(16): e Angelis C, razen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeer TV, Sox HC, Van er Weyen MB; International Committee of Meical Journal Eitors. Is this clinical trial fully registere? A statement from the International Committee of Meical Journal Eitors. N Engl J Me Jun 9;352(23): Epub 2005 May owell E, ua S, ulai SK, Astone K, Mulpuri K. The quality of ranomize controlle trials in peiatric orthopaeics: Are we improving? J Peiatr Orthop Jul-Aug;35(5):