Improvement in the Quality of Randomized Controlled Trials Among General Anesthesiology Journals 2000 to 2006: A 6-Year Follow-Up

Transcription

1 Improvement in the Quality of Randomized Controlled Trials Among General Anesthesiology Journals 2000 to 2006: A 6-Year Follow-Up Mary Lou V. H. Greenfield, MPH, MS Jill M. Mhyre, MD George A. Mashour, MD, PhD James M. Blum, MD Eugene C. Yen, BS Andrew L. Rosenberg, MD BACKGROUND: We previously assessed all randomized controlled trials (RCTs) from four anesthesiology journals from January 2000 to December We identified key areas for improvement in the study protocol design and implementation and in data analyses. This study was repeated for the year 2006 to determine if improvements have occurred during the 6-yr interval. METHODS: All RCTs published in 2006 in four anesthesiology journals (Anesthesiology, Anesthesia & Analgesia, Anaesthesia, and Canadian Journal of Anesthesia) were retrieved using a MEDLINE search. Of 2164 articles published in 2006, 200 papers met these search criteria and were considered valid for analysis. We completed a 14-item, validated assessment tool used in our previous study to determine a quality score for each article. Four clinical reviewers each assessed 50 articles, and one reviewer assessed all 200 articles. Points were assigned by consensus. Scores were weighted and compared with the results from the year RESULTS: Quality scores improved from the year 2000 to 2006, from a mean overall quality score of 44% (95% CI 42, 46) to a mean score of 58% (95% CI 55, 60). Specific areas of study, quality assessment demonstrating improvement, included sample size estimates (52% vs 86%, P ), major end-points (44% vs 99%, P ), and discussion of side effects (68% vs 82%, P ). Low quality scores remained for randomization blinding (4% vs 19% P ), observer blinding to continuing studies (1% vs 5% P 0.116), and post- estimates in trials with negative outcomes (16% vs 18%, P 0.87). CONCLUSIONS: There appears to have been a general improvement in the overall quality of RCT reporting among the major anesthesiology journals from the year 2000 to However, many articles could be improved with respect to randomization blinding, observer blinding to continuing study results (i.e., no unplanned interim data analysis), and a full discussion of Type II error in negative trials. Responsibility to improve the quality of the anesthesiology literature rests with investigators to design, implement and report high quality RCTs, and with peer reviewers and journal editors to set the standard for manuscript reporting. Periodic reassessments of the literature can serve to improve and maintain the quality of clinical trials reporting. (Anesth Analg 2009;108: ) The randomized controlled trial (RCT) is considered the most valid method of comparing treatments and making inferences regarding cause and From the Department of Anesthesiology, University of Michigan Health System, Ann Arbor, Michigan. Accepted for publication December 17, Supported by Department of Anesthesiology, The University of Michigan. There are no relationships between any of the authors of this study and any company or organization with a vested interest in the outcome of this study. Address correspondence to Andrew L. Rosenberg, MD, Department of Anesthesiology, University of Michigan Health System, 4172 Cardiovascular Center SPC 5861, 1500 East Medical Center Dr., Ann Arbor, MI Address to arosen@umich.edu. Reprints will not be available from the author. Copyright 2009 International Anesthesia Research Society DOI: /ane.0b013e31819fe6d effect. 1,2 A RCT is a planned experiment in human subjects randomized to a test treatment or to a control treatment; subjects are enrolled, treated, and followed over the same time period. 3 The results of a RCT can directly and immediately affect patient care more than any other type of study design. 4 In 2004, Moher et al. 5 reported...if the conduct or reporting of RCTs is poor, treatments may be introduced that are less effective than was thought or that might even be ineffective. Examining the published reports of RCTs is the only way we have to determine the quality of their conduct and analysis. The medical literature has numerous meta-analysis and systematic review tools that synthesize evidence from RCTs to guide best clinical practices. 6 9 In the mid-1990s the consolidated standards of reporting Vol. 108, No. 6, June 2009

2 trials (CONSORT) checklist for reporting RCTs was published;* since then, these standards have been further developed and expanded with the expectation that the quality of RCTs will improve as editors and authors implement these standards for their journals and clinical trials. 10 Although anesthesiology journals frequently publish RCTs, as well as systematic reviews and meta-analyses, there have only been three studies in English of the quality of these types of studies in the anesthesiology literature since In 2005, we published the results of a quality assessment review of 279 RCTs published in anesthesiology journals (Anesthesiology, Anesthesia & Analgesia, Anaesthesia, and Canadian Journal of Anesthesia) from January through December The purpose of the previous review was to delineate specific areas for improvement in the conduct, implementation, analysis, and reporting of RCTs in the anesthesiology literature. The goal of the current study was twofold: to compare articles published in four anesthesiology journals in 2006 with those published in the same journals in 2000 to determine areas of improvement over the past 6 years; and to identify areas for future improvement. We hypothesized that there would be significant improvements in the quality of reporting of protocol designs and analyses. METHODS All human RCTs published between January 2006 and December 2006 in four anesthesiology journals (Anesthesiology, Anesthesia & Analgesia, Anaesthesia, and Canadian Journal of Anesthesia) were retrieved using the same methodology as described previously 11 (Fig. 1). Articles were obtained in portable document format and all identifiers (e.g., journal name, authors) were removed by an investigator (EY) who was not involved in further evaluation. Before beginning the analyses, the reviewers used the scoring instrument (Table 1) and scoring guide to independently assess 10 randomly selected articles not included in the main study. Reviewers met for two training sessions during which the training articles and the scores assigned by each reviewer were discussed and clarified for reliability purposes. Remaining study articles were randomly divided into four groups using a computer-generated randomization scheme, such that each reviewer (AR, JB, GM, JM) evaluated 50 articles. A fifth reviewer (MLG) evaluated all 200 articles. Scoring discrepancies were resolved by consensus. *An international group of clinical trialists, statisticians, and medical journal editors proposed the CONSORT initiative; it has since gained large acceptance by medical journals and editorial groups worldwide. CONSORT is a 22-item checklist and flow diagram intended for authors to adequately describe key study methodologies that are required for readers to evaluate the validity, limitations, and generalizability of clinical trials including details for the enrollment, interventions, allocation, follow-up, and statistical analysis of the clinical trial. Figure 1. Determining the study sample. To evaluate each article, we used a modified version of Chalmers quality assessment tool 6,14 ; the same tool was used in the 2000 quality review 11 (Table 1). The scale includes 14 quality domains, each of which has precise requirements for what must be recorded to achieve a certain weighted score. 15 Percentages (total score divided by total possible score) were assigned because some items were not applicable to the individual RCT under review and were not included in the denominator of possible points. This method of scoring has been used in numerous studies of clinical research. 16,17 For each element on the Chalmers scale score, frequencies were compared between all articles published in 2000 with all articles published in 2006 using Pearson s 2 test or Fisher s exact test as appropriate. Total quality scores were averaged Vol. 108, No. 6, June International Anesthesia Research Society 1917

3 Table 1. Comparison of Quality Scores in 2000 with Scores in 2006 Item number Quality measure reported (with possible point assignments for each quality score item measure) from 2000 a from 2006 b Reporting of study protocol 1 Control appearance or regimen Score 0 (appearance of control group different than 41 (20) 26 (17) 0.40 treatment group 0/3) Score 3 (appearance of control group same as treatment group 3/3) 160 (80) 128 (83) 2 Blinding of randomization allocation methods and concealment Score 0 (neither random allocation methods nor 182 (65) 78 (39) concealment reported 0/10) Score 5 (either random allocation methods or concealment 85 (31) 85 (42.5) reported, but not both reported 5/10) Score 10 (random allocation methods and concealment both reported 10/10) 12 (4) 37 (18.5) 3 Patient blinding techniques Score 0 (patient blinding techniques not described 0/8) 60 (21.5) 49 (25) Score 4 (statement that patients were blinded, but no 60 (21.5) 16 (8) description 4/8) Score 8 (patient blinding described 8/8) 159 (57) 133 (67) 4 Observer blinding to treatment Score 0 (all or some observers could have been blinded, 58 (23) 37 (20) but none were blinded 0/10) Score 4 (observers could have been blinded, but only 95 (37) 27 (14) some were blinded 5/10) Score 8 (all observers who could have been blinded 102 (40) 125 (66) were blinded 10/10) 5 Observer blinding to ongoing study results Score 0 (no observers were blinded 0/10) 273 (98) 189 (94.5) 0.12 Score 5 (some observers were blinded 5/10) 2 (1) 2 (1) Score 10 (all observers were blinded 10/10) 4 (1) 9 (4.5) 6 Evidence of sample size estimate and power analysis Score 0 (not included 0/3) 134 (48) 28 (14) Score 3 (included 3/3) 145 (52) 172 (86) 7 Testing compliance (patient compliance not applicable for most in-hospital studies ) d 8 Results of randomization on pretreatment variables and inclusion in analysis Score 0 (no Table 1 or equivalent describing baseline comparability of groups 0/3) Score 1.5 (fair description limited to race, age, sex, demographics 1.5/3) Score 3 (adequate description of baseline comparability of study groups 3/3) Reporting of data analysis 9 Major end points (statistical test and P value) Score 0 (neither statistical test reported nor P value reported 0/4) Score 1 (statistical test reported, but no P value reported 86 (32) 18 (9) 106 (39) 43 (21.5) (29) 139 (69.5) P c 3 (1) 0 (0) (55) 3 (1.5) 1/4, or statistical test not reported but P value reported) Score 4 (statistical test and P value reported 4/4) 122 (44) 198 (98.5) 10 Probability of a Type II error or post- estimates (in negative trials only) Score 0 (poor no discussion of probability of Type II error 0/3) 107 (72) 54 (71) Score 1.5 (fair possibility of Type II error noted 1.5/3) 18 (12) 8 (11) 0.87 Score 3 (good discussion of Type II error 3/3) 24 (16) 14 (18) (Continued) 1918 Quality of Randomized Control Trials ANESTHESIA & ANALGESIA

4 Table 1. Continued Item number Quality measure reported (with possible point assignments for each quality score item measure) from 2000 a from 2006 b 11 Confidence limits listed for endpoints representing differences in proportions, rates, or means Score 0 (no standard deviations or standard errors or 17 (6) 13 (6.5) 0.01 confidence intervals calculated 0/3) Score 1.5 (standard deviations or standard errors of 231 (83) 145 (72.5) endpoints included, but no confidence intervals calculated 1.5/3) Score 3 (confidence intervals calculated 3/3) 31 (11) 42 (21) 12 Statistical analyses (overall assessment of statistical methods used, including clarity and precision in reporting of study results) Score 0 (poor 0/4) 27 (10) 1 (0.5) Score 1 (fair 1/4) 106 (38) 18 (9) Score 2 (good 2/4) 115 (41) 102 (51) Score 4 (excellent 4/4) 31 (11) 79 (39.5) 13 Explanation of any patient withdrawals after randomization e Score 0 (no list of withdrawals, or withdrawals 15% 0/3) 27 (10) 52 (26) Score 1.5 (withdrawals listed with reason, representing 235 (84) 88 (44) 15% patients 1.5/3) Score 3 (statement of no withdrawals 3/3) 17 (6) 60 (30) 14 Discussion of adverse events and side effects related to study treatments Score 0 (no discussion of side effects present 0/3) 31 (11) 16 (8) Score 1.5 (incomplete report or discussion of side 60 (22) 21 (10.5) effects 1.5/3) Score 3 (side effects reported with appropriate statistical tests between groups 3/3) 188 (67) 163 (81.5) a 279 articles were evaluated in Not all point assignments frequencies will sum to 279 because some quality measures allowed for a not applicable assignment (0/0). For example, in 2000, 78 articles involved anesthesia or surgery regimens that could not be made to appear the same in both the treatment and control groups; therefore, only 200 articles were able to be assessed for this item. Of those 200 articles, 160 (80%) achieved a score of 3/3 and 41 (20%) achieved a score of 0/3. b 200 articles were evaluated in Not all point assignments frequencies will sum to 200 because some quality measures allowed for a not applicable assignment (0/0). For example, in 2006, 46 articles involved anesthesia or surgery regimens that could not be made to appear the same in both the treatment and control groups; therefore, only 154 articles were able to be assessed for this item. Of those 154 articles, 128 (83%) achieved a score of 3/3 and 26 (17%) achieved a score of 0/3. c Pearson 2 -square test or Fisher s exact test as appropriate. d There were only 1 of 200 studies in 2000 and 1 of 279 studies in 2006 in which this variable was applicable. e Withdrawals are defined as patients randomized but who did not receive assigned treatment, or who did not complete specified observation period, or who were removed during analysis. P c Figure 2. An analysis of variance (ANOVA) demonstrated significant improvement within individual journal pairs from 2000 to 2006 (P for each of the four journals). The mean paired difference from 2000 to 2006 for the four journals was significant at points (P ). across all articles published in each journal. Analysis of variance was used to compare the set of mean total quality scores for the four journals between 2000 and Statistical analyses were conducted using the Statistical Analysis System (SAS 9.1; SAS Institute, Cary, NC). RESULTS The mean weighted quality scores ( the standard deviation [sd]) for the four individual journals, from 2006, were 63% 16%, 53% 15%, 57% 16%, and 59% 14%. Figure 2 shows the 2006 quality scores Vol. 108, No. 6, June International Anesthesia Research Society 1919

5 compared with the 2000 scores. The overall mean quality score for the 200 studies from 2006 was 58% (95% confidence interval [CI] 55,60) compared with the mean quality score of 44% (95% CI 42,46) for the 279 studies from This represents an absolute increase in quality score from 2000 to 2006 of 15% 3% (P ). A comparison of quality score items among the journals from 2000 and 2006 is presented in Table 1. Six score items demonstrated improvement with scores above 50% between 2000 and 2006, including items #3, 4, 6, 8, 9, and 14. One score item (#1) remained essentially unchanged but above 80%. Another six quality items showed at least a trend towards improvement, but the item quality scores remained 50%, including items #2, 5, DISCUSSION Between 2000 and 2006, quality scores for RCTs reported in each of the four anesthesiology journals improved 15% (sd 3%). The 2006 mean quality score for anesthesiology RCTs was similar to evaluations of clinical RCTs in other medical fields (56% 16 and 58% 18 ) that have used the Chalmers scoring system. Despite the improvement in quality score, the current average quality score of 58% remains below 80%, the threshold that is considered sufficient to warrant acceptance of the study conclusions. 6 Particular items that contributed to the significantly higher 2006 quality scores (Table 1) were an increase in the number of articles reporting patient blinding to treatment, observer blinding to treatment, sample size estimates, the randomization results on important pretreatment variables, major study outcomes, and types of statistical tests and their significance levels, and side effects. Although almost all score items in 2006 demonstrated at least a trend toward improvement, several quality scores continue to indicate deficiencies both in lower scores assigned and showed less improvement between 2000 and 2006 than other score items. Score items #2 (randomization blinding), #5 (observer blinding to continuing study results), and #10 (post- estimates discussions, including the probability of Type II errors in negative studies) present the most objective opportunities for maximum improvement in the quality of reporting of RCTs. First, randomization blinding (item #2) requires two descriptions: 1) how the randomized allocation was generated; and 2) how the randomized allocation was concealed from study personnel before the study assignment for each patient. Appropriate randomization procedures must provide completely random assignments, by relying, for example, on random numbers tables or computer-generated randomization schemes. In many studies, randomization generation was either not described at all or was listed using unacceptable methods (e.g., subjects were randomized by hospital number or by shuffling envelopes containing assignments). Randomization concealment from study personnel was frequently not described in the studies we reviewed or was not described in enough detail to indicate a tamper-proof concealment (e.g., the assignment was in a closed envelope). Examples of acceptable descriptions of masking treatment assignments included: 1) sequentially numbered sealed and opaque envelopes containing group assignments opened at the time of patient allocation or 2) the blinded study drug was prepared by a hospital pharmacist who maintained the randomization allocation. Second, investigator blinding to ongoing study results (item #5) was described in 6% of the studies. Ideally, investigators should remain blind to the accumulating results of the study because knowledge of previous patient results may impact decisions to recruit particular patients, to encourage the recruitment process, or to continue or discontinue the study prematurely. Planned unblinded interim analyses should be outlined in the study protocol before study initiation (i.e., before any unblinding of investigators), completed with an accounting of the potential for bias and the impact on Type I error, and performed according to written study protocols, analysis design, unblinding strategies, and data monitoring committee charters. Reviews of accumulating data to ensure data quality may be completed without revealing the randomization code. If interim analyses with knowledge of randomized assignments are required to ensure patient safety, then an independent data safety monitoring board is appropriate. Third, of the 76 studies that failed to reject the primary null hypothesis in 2006, only 29% (54 of 76) addressed the possibility that such findings could have been due to a Type II error (item #10) or small sample size. Similar to the 2000 review, authors continued to interpret a lack of statistically significant difference in outcomes to indicate that the treatments were equally effective or that the new treatment was an acceptable alternative to the control. When the objective of a trial is to demonstrate equivalence, the null hypothesis is constructed as one of a difference between a standard and new treatment and the research hypothesis is constructed as one of equivalence between the two treatments; appropriate statistical methodology (e.g., 2 one-sided tests [TOST] for equivalency) should be applied. 19 When the objective of a trial is to demonstrate a difference, but the study results in a negative outcome, then to satisfy item #10, the investigators should discuss the probability of a Type II error. Post- estimates of a clinically interesting (but statistically nonsignificant) difference in negative trials and calculation of 95% confidence intervals around this difference help guide sample size estimates for future studies. Of interest, 53% of RCTs evaluated in 2000 in our previous review were negative trials; in 2006, 38% of studies had negative results. The smaller number of negative studies published in 2006 compared with 2000 raises the possibility of publication bias among 1920 Quality of Randomized Control Trials ANESTHESIA & ANALGESIA

6 the four anesthesiology journals in In the surgical literature, Hasenboehler et al. 20 found that 17% of articles presented negative results. They concluded that a bias towards publishing positive data disregards important information derived from unpublished negative studies. There are limitations to our approach to measuring and comparing the quality of reporting among RCTs in 2000 with RCTs in First, because a score item is not described in an article does not mean that it was not performed in the trial itself. However, it is generally accepted that the methodologic rigor of a study is likely reflected in the quality of its reporting. 21 Second, this 2006 review was limited to the four anesthesiology journals that were studied in 2000 and 2006; it is possible that other important anesthesiology journals could reflect higher or lower quality scores than of those measured in this study s four-journal review. Third, a tool other than the Chalmers system could have produced different results than found in our study. There is no clear gold standard for evaluating the reporting or quality of RCTs as evidenced by other tools available. 22,23 Lastly, we cannot exclude the possibility that the format of the articles reviewed could have enabled reviewer recognition of the journal in which it was published. The CONSORT checklist has been adopted by three of the four journals evaluated in this study: Anesthesiology and the Canadian Journal of Anesthesia have recommended the use of CONSORT since 2003, Anesthesia & Analgesia has required the use of CONSORT since Improvements in quality scores between 2000 and 2006 may reflect increasing use of the CONSORT checklist and a growing emphasis on the quality, consistency, and transparency of reporting. In conclusion, the results of this study indicate that the quality of RCTs in four leading anesthesia journals since 2000 has improved, particularly in the reporting of sample size estimation, in the description of the test statistics to be used and critical P values, and in the discussions of side effects. However, further improvement is warranted. Future efforts to enhance the quality of RCTs should be directed towards: 1) reporting how the randomization allocation is generated and concealed, 2) blinding investigators to ongoing study results, and 3) reporting complete analyses of negative outcomes to assess the probability of Type II error. REFERENCES 1. Rosenberg AL, Wei JT. Clinical study designs in the urologic literature: a review for the practicing urologist. Urology 2000;55: Altman DG. Better reporting of randomised controlled trials: the CONSORT statement. BMJ 1996;313: Meinert C. Clinical trials: design, conduct, and analysis. New York: Oxford University Press, Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276: Moher D, Altman DG, Schulz KF, Elbourne DR. Opportunities and challenges for improving the quality of reporting clinical research: CONSORT and beyond. CMAJ 2004;171: Chalmers TC, Smith H Jr, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A. A method for assessing the quality of a randomized control trial. Control Clin Trials 1981;2: DerSimonian R, Charette LJ, McPeek B, Mosteller F. Reporting on methods in clinical trials. N Engl J Med 1982;306: Detsky AS, Naylor CD, O Rourke K, McGeer AJ, L Abbe KA. Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol 1992;45: Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17: Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001;357: Greenfield ML, Rosenberg AL, O Reilly M, Shanks AM, Sliwinski MJ, Nauss MD. The quality of randomized controlled trials in major anesthesiology journals. Anesth Analg 2005;100: Pua HL, Lerman J, Crawford MW, Wright JG. An evaluation of the quality of clinical trials in anesthesia. Anesthesiology 2001;95: Halpern SH, Darani R, Douglas MJ, Wight W, Yee J. Compliance with the CONSORT checklist in obstetric anaesthesia randomised controlled trials. Int J Obstet Anesth 2004;13: Rochon PA, Gurwitz JH, Simms RW, Fortin PR, Felson DT, Minaker KL, Chalmers TC. A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Arch Intern Med 1994;154: Rochon PA, Gurwitz JH, Cheung CM, Hayes JA, Chalmers TC. Evaluating the quality of articles published in journal supplements compared with the quality of those published in the parent journal. JAMA 1994;272: Berghmans T, Paesmans M, Meert AP, Mascaux C, Lothaire P, Lafitte JJ, Sculier JP. Survival improvement in resectable nonsmall cell lung cancer with (neo) adjuvant chemotherapy: results of a meta-analysis of the literature. Lung Cancer 2005;49: Manzoli L, Schioppa F, Boccia A, Villari P. The efficacy of influenza vaccine for healthy children: a meta-analysis evaluating potential sources of variation in efficacy estimates including study quality. Pediatr Infect Dis J 2007;26: Le Quintrec JL, Bussy C, Golmard JL, Herve C, Baulon A, Piette F. Randomized controlled drug trials on very elderly subjects: descriptive and methodological analysis of trials published between 1990 and 2002 and comparison with trials on adults. J Gerontol A Biol Sci Med Sci 2005;60: Phillips KF. Power of the two one-sided tests procedure in bioequivalence. J Pharmacokinet Biopharm 1990;18: Hasenboehler EA, Choudhry IK, Newman JT, Smith WR, Ziran BH, Stahel PF. Bias towards publishing positive results in orthopedic and general surgery: a patient safety issue? Patient Saf Surg 2007;1:4 21. Altman DG, Dore CJ. Randomisation and baseline comparisons in clinical trials. Lancet 1990;335: Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther 2008;88: Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 1995;16:62 73 Vol. 108, No. 6, June International Anesthesia Research Society 1921