(2007) 31, 1554 1559 & 2007 Nature Publishing Group All rights reserved 0307-0565/07 $30.00 REVIEW www.nature.com/ijo What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials L Thabane 1,2, R Chu 3, K Cuddy 4 and J Douketis 4,5 1 Centre for Evaluation of Medicines, St. Joseph s Healthcare, Hamilton, ON, Canada; 2 Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, ON, Canada; 3 Department of Statistics, University of British Columbia, Vancouver, BC, Canada; 4 St Joseph s Healthcare, Hamilton, ON, Canada and 5 Department of Medicine, McMaster University, Hamilton, ON, Canada Background: Despite the large number of randomized controlled trials (RCTs) assessing weight loss interventions, no study has assessed the quality of reporting in these trials. Purpose: To assess the quality of reporting of RCTs of weight loss interventions and to identify predictors of reporting quality. Methods: The RCTs assessed were derived from a published systematic review of trials investigating the efficacy of weight loss interventions. For our study, two reviewers independently rated the quality of reporting in these trials, based on the Consolidated Standards for Reporting of Trials (CONSORT) Statement. We describe the quality of reporting using number (percent) of studies satisfying each of the 44 CONSORT criteria. We use generalized estimating equations (GEE) to fit a multivariable regression model to determine factors that are associated with the overall quality reporting score. Results: We assessed 63 RCTs, of which 25 were dietary-lifestyle trials, 22 were pharmacological trials and 16 were behaviorcognitive, exercise-lifestyle, or surgical trials. Less than half (46%) of the trials defined the primary outcome of the study; about 10% provided the description of the method of allocation concealment. Multivariable GEE results showed that the sample size, type of intervention (non-pharmacologic trials having lower scores than pharmacologic trials), and publication time relative to the CONSORT Statement publication in 1996 (publications after 1996 having higher scores) were strong predictors of the quality reporting score. Reporting a statistically significant result on the primary outcome was not significantly associated with the quality score. Conclusion: While the overall quality in reporting seemed to have improved since the publication of the revised CONSORT Statement in 1996, the reporting of some key methodologic aspects, such as clear description of primary outcome and method of allocation concealment, still requires improvements. Factors that are significantly associated with the overall quality reporting score can be used as surrogates in the review of protocols to enhance the quality of the final reports. (2007) 31, 1554 1559; doi:10.1038/sj.ijo.0803640; published online 24 April 2007 Keywords: weight loss; randomized trial; reporting quality Introduction As the worldwide prevalence of obesity increases at an alarming rate, 1 an increasing number of studies are being undertaken to investigate treatments aimed at inducing and maintaining weight loss in overweight and obese adults. 2 Correspondence: Dr J Douketis, Department of Medicine, St Joseph s Healthcare, Room F-544, 50 Charlton Ave. East, Hamilton, ON, Canada L8N 4A6. E-mail: jdouket@mcmaster.ca Received 31 October 2006; revised 21 February 2007; accepted 8 March 2007; published online 24 April 2007 Although most weight loss intervention studies use a randomized trial design, which is the gold standard to investigate the efficacy and safety of a new treatment, 3 the quality of randomized trials can be sub-optimal. This has the potential to invalidate or limit the generalizability of the trial results. 4 Furthermore, a lack of standardization in the manner which the trial methods and results are reported can also be inadequate, potentially limiting the interpretation of results and the pooling of trial results for meta-analysis. 5 Previous studies have assessed the quality of reporting of randomized trials within several clinical domains, including cardiovascular disease, oncology, clinical pharmacology and critical care medicine. 6 10 However, no study, to our knowl-
edge, has assessed the quality of reporting in randomized trials of weight loss interventions. This issue is important for two reasons. First, the management of obesity is frequently encountered in clinical practice, and clinicians should be aware of reporting quality in weight loss trials so as to better interpret the findings and apply them to patient care. Second, an assessment of reporting quality might identify gaps that need to be addressed in the design and reporting of future weight loss intervention trials. We, therefore, assessed the reporting quality of randomized trials of weight loss interventions. Our objectives were: (1) to describe the quality of reporting in these trials using a standardized assessment tool (the Consolidated Standards for Reporting of Trials (CONSORT) criteria); and (2) to identify predictors that can explain the variability in the overall score. Methods Data sources The studies in this analysis were derived from a published review assessing the long-term efficacy of weight loss interventions. 2 In this review, studies were selected by searching MEDLINE (1966 to September 2003), HealthSTAR (1975 to September 2003), and the Cochrane Controlled Trials Register (1990 to September 2003) for weight loss intervention studies, using the key words: obesity; overweight; body mass index; treatment; weight reduction; randomized controlled trial. English and non-english language articles were searched, but only articles with an English abstract were reviewed. Study selection criteria Included studies had the following characteristics: the study was a randomized controlled trial; the study population consisted of overweight or obese adults with a mean body mass index X25 kg/m 2 ; the study investigated a weight loss intervention; and the duration of patient follow-up was X1 year. Studies were excluded if they had one or more of the following characteristics: the study required patients to attain a weight loss target to qualify for study enrollment; the weight loss intervention was not approved for clinical use; or the weight loss intervention was approved only for short-term (o3 month) use. Data extraction We used the CONSORT Statement to assess the reporting quality of weight loss trials. 3 The CONSORT Statement was developed for randomized trials to provide standardized reporting criteria for the presentation of the rationale, methods, results and conclusions in randomized trials. For each trial, we determined whether each of the 44 CONSORT reporting criteria was satisfied, with a yes/no response. The CONSORT reporting criteria that were used for this assessment are presented in Appendix. A priori, we also identified eight CONSORT criteria relating to the study methods and results sections that were considered important determinants of the methodologic quality of randomized trials. There were four pre-specified key reporting criteria from the methods section: (1) primary and secondary study outcomes were defined; (2) power of the trial was provided; (3) an intention-to-treat analysis of results was performed and (4) method of allocation concealment was defined. There were four pre-specified key reporting criteria from the results section: (1) results were presented for primary and secondary outcomes; (2) results were presented as absolute numbers; (3) the number of subjects who had protocol deviation(s) with each intervention were reported and (4) the number of subjects who had adverse events/treatmentrelated side effects with each intervention were reported. An independent double review of included trials was done by two authors (RC, KC) to assess agreement regarding CONSORT criteria that were satisfied. Discrepancies in this assessment were resolved through discussion and consensus. Statistical analysis Cohen s analysis was done to test inter-reviewer agreement on study selection and reporting criteria satisfied. 11. We determined the mean (s.d.) number of studies that satisfied each of the 44 CONSORT reporting criteria and the number (percent) of studies that satisfied each criterion. We also assessed the mean (s.d.) number of studies that satisfied the pre-specified key criteria. For each study assessed, 1 point was given for each reporting criterion that was satisfied, and 0 points were given if the reporting criterion was not satisfied or was not clearly stated. We performed multivariable analysis using generalized estimating equations (GEEs) to explore factors that explain the variability in the reporting across studies. GEEs were chosen to account for the possible intrajournal correlation and an exchangeable correlation structure was assumed for these analyses. 12 The explanatory variables considered in these analyses included the sample size, whether the result of the primary analysis was statistically significant or not, and the type of intervention used in the trial (that is, whether it was drug, diet or other). The results of GEE analyses are expressed as coefficient, corresponding 95% confidence interval, associated P-value and estimate of the measure of intrajournal correlation. A priori, the criterion for statistical significance was set at a ¼ 0.05. For the multivariable model, multicolinearity was assessed using variance inflation factor (VIF) and variables with VIF410 were excluded from the model. 13 All analyses were undertaken using STATA software, Version SE 8.0. Results Studies included in the analysis There were 63 randomized trials assessed, of which 25 (40%) were dietary-lifestyle trials, 22 (35%) were pharmacological 1555
1556 trials, 9 (14%) were behavior-cognitive trials, 5 (8%) were exercise-lifestyle trials and 2 (3%) were surgical trials. The characteristics of these randomized trials are presented in Table 1. Agreement in data extraction In terms of inter-rater agreement for CONSORT criteria satisfied, there was good agreement (k ¼ 0.8 1.0) for 60 (70%) reporting criteria, fair agreement (k ¼ 0.6 0.8) for 23 (27%) reporting criteria and poor agreement (ko0.6) for 3 (3%) reporting criteria. Percentage of studies that satisfied the CONSORT criteria The extent to which the 44 CONSORT criteria were satisfied is presented in Table 2. Overall, 60% of reporting criteria were satisfied by the trials in this analysis. The reporting of criteria relating to the introduction section was generally good, as 90% of reporting criteria were satisfied in this section. The reporting of criteria relating to the methods section varied across studies: only 19% of studies satisfied the reporting criteria relating to treatment allocation; 75% of studies satisfied the reporting criteria relating to the description of the study protocol. The reporting of criteria relating to the results section was adequate, as 70% of reporting criteria were satisfied across studies. Percentage of studies that satisfied the key CONSORT criteria Methods section. In terms of reporting definitions of the primary and secondary outcomes, this was performed by 46 and 27% of studies, respectively. Statements in regard to the power of the study, and that an intention-to-treat approach was used, were satisfied by 43 and 54% of studies, respectively. A statement in regard to the method by which treatment allocation was concealed was satisfied by only 9.5% of studies. Table 1 Characteristics of included randomized trials: n ¼ 63 Study characteristic Number of studies (%) Year of publication Before 1990 3 (5) 1990 1994 9 (14) 1995 1999 22 (35) 2000 2004 29 (46) Type of weight loss intervention assessed Dietary-lifestyle 25 (40) Pharmacologic 22 (35) Exercise 9 (14) Behavioral-cognitive 5 (8) Surgical 2 (3) Results of primary study outcome Not statistically significant 16 (25) Statistically significant 47 (75) Results section. The reporting of the study outcomes and reporting of findings using absolute number was performed in 98 and 97% of studies, respectively. However, reporting of the number of study subjects who had protocol deviations or adverse events/ side effects with each treatment intervention was reported in 79 and 51% of studies, respectively. Predictors of the overall CONSORT reporting score The GEE results (Table 3) show that the sample size, timing of publication (whether the paper was published before or after 1996) and type of intervention are strong predictors of the overall score, although the impact of the sample size was minimal. The quality score improved significantly after 1996, with a mean increase (95% CI) ¼ 3.29 (1.50, 5.07; Table 2 Reporting of CONSORT criteria: n ¼ 63 Section Maximum possible score Maximum/minimum score obtained Mean score (s.d.) Title 1 1/0 0.40 (0.49) Abstract 1 1/0 0.70 (0.46) Introduction 3 3/2 2.86 (0.35) Methods Protocol 7 7/2 5.25 (1.45) Sample size 3 3/1 1.95 (0.91) Treatment allocation 4 3/0 0.75 (0.92) Blinding of treatment 3 3/0 1.27 (0.54) Statistical analysis 6 5/1 3.13 (1.07) Results 10 10/5 6.97 (1.09) Discussion 6 5/2 2.98 (0.73) Overall score 44 34/16 26.25 (4.51) Abbreviation: CONSORT, Consolidated Standards for Reporting of Trials. a 1 point given if reporting criterion satisfied; 0 points given if reporting criterion not satisfied or unclear if satisfied. Table 3 Multivariable analyses of factors associated with CONSORT reporting score Study features Coefficient (95% CI) P-value Sample size 0.001(0.000, 0.002) 0.014 Timing relative to the publication of CONSORT statement: Before or within 1996 0 After 1996 3.29 (1.50 5.07) o0.001 Results of the primary outcome: Not significant 0 Significant 1.14 ( 0.93 3.22) 0.279 Intervention category: Pharmacological 0 Dietary 2.70 ( 5.46, 0.05) 0.054 Others a 3.31 ( 6.10, 0.52) 0.020 Abbreviation: CI, confidence interval; CONSORT, Consolidated Standards for Reporting of Trials. Exchangeable correlation coefficient (ICC) ¼ 0.0432. a Other therapies include exercise therapy, behavioral therapy and surgical therapy.
Po0.001). Studies using dietary interventions had a lower mean score ( 2.70 ( 5.46, 0.05)) than those using drug interventions, although the decrease was not statistically significant (P ¼ 0.054); and other interventions (these include exercise therapy, behavioral therapy and surgical therapy) also had significantly lower mean scores than those using drug interventions ( 3.31 ( 6.10, 0.52); P ¼ 0.020). Assessment of the residuals yielded no serious violations of the model assumptions. Discussion There are two principal findings from this study that assessed the quality of reporting in randomized trials of weight loss interventions. First, the quality of reporting based on the CONSORT criteria was sub-optimal as only 60% of these criteria were reported in the 63 studies assessed. Second, key reporting criteria that may impact on the validity and generalizability of results were adequately reported. The validity of our findings are supported by our use of a standardized scoring system (CONSORT) to assess whether there was adequate reporting in randomized trials. Our finding of suboptimal reporting of obesity trials is consistent with sub-optimal reporting quality in randomized trials in other clinical domains. 8,9,14,15 Our findings question the adequacy of methods used to report the trial design and findings of weight loss intervention trials. These limitations have implications on the validity and generalizability of the study results. For example, the method of concealment of treatment allocation, an integral component of any randomized trial, was reported satisfactorily in only 9.5% of studies, leading a reader to question whether the randomization process was appropriate and, in turn, whether the study results were valid. In addition, sub-optimal reporting of patients who had study protocol deviations or who may have had adverse events or intervention-related side effects may lead a reader to question both the validity and generalizability of results. This issue is particularly relevant in weight loss intervention trials, where a considerable proportion of patients are lost to follow-up because of non-compliance to treatment or treatment-related adverse effects. The lack of reliable reporting of these aspects of the trial design and findings impedes the ability of the practicing clinician to apply the study findings to patients assessed in everyday clinical practice. We acknowledge several potential limitations of our study. First, we did not assess the quality of individual trials, as this addresses a different question that was beyond the scope of our study and has been addressed elsewhere. 2,4,5 Second, we limited our assessment to studies with at least one year of patient follow-up and, consequently, most weight loss intervention trials, which typically have a 3 6 month duration, were excluded. However, as obesity is considered a chronic condition, it follows that only the findings from long-term studies are relevant to clinical practice. Third, the assessment of whether reporting criteria were satisfied was based on a subjective review of the articles and may be subject to bias. However, studies were reviewed independently by two study authors and the good or fair interobserver agreement, as reflected by the k scores, supports the validity of our findings. Further, it is important to note that bad reporting does not necessarily imply poor conduct or methodology for the studies, 16,17 although other studies have found some association between the two. 15 Finally, we did not determine whether the funding source (industry vs non-industry) had an impact on reporting score, as this was beyond the scope of our study objectives. The results of our study have potential implications on the design of weight loss intervention trials. Our findings indicate that there is considerable room for improvement in the manner by which weight loss intervention trials are reported. The CONSORT criteria provide a reference point that can be used by investigators both in the design, execution and reporting of weight loss intervention trials. To summarize, we found that in weight loss intervention trials, standardized reporting criteria (CONSORT) were inadequately addressed. There is a need for future weight loss intervention trials to better adhere to the CONSORT reporting criteria. References 1 Controlling the global obesity epidemic. WHO 09/03/2003 http:// www.who.int/nut/obs.htm (Date of last access: October 6, 2006). 2 Douketis JD, Thabane L, Macie C, Williamson DF. Systematic review of long-term weight loss studies in obese adults: clinical significance and applicability to clinical practice. Int J Obes Relat Metabol Dis 2005; 10: 1153 1167. 3 Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al., for the CONSORT Group. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001; 134: 663 694. 4 US Preventive Services Task Force. Screening for obesity in adults: recommendations and rationale. Ann Intern Med 2003; 139: 930 932. 5 Douketis JD, Feightner JW, Attia J, Feldman WF, with the Canadian Task Force on Preventive Health Care. Periodic health examination, 1999 update: detection prevention and treatment of obesity. CMAJ 1999; 160: 513 524. 6 Krzyzanowska MK, Pintilie M, Brezden-Masley C, Dent R, Tannock IF. Quality of abstracts describing randomized trials in the proceedings of American Society of Clinical Oncology Meetings: guidelines for improved reporting. J Clin Oncol 2004; 22: 1993 1999. 7 Bath FJ, Owen VE, Bath PMW. Quality of full and final publications reporting acute stroke trials. A systematic review. Stroke 1998; 29: 2203 2210. 8 Mills E, Loke YK, Wu P, Montori VM, Perri D, Moher D et al. Determining the reporting quality of RCTs in clinical pharmacology. Br J Clin Pharmacol 2004; 58: 61 65. 9 Latronico N, Botteri M, Minelli C, Zanotti C, Bertolini G, Candiani A. Quality of reporting of randomised controlled trials in the intensive care literature. A systematic analysis of papers published in Intensive Care Medicine over 26 years. Intensive Care Med 2002; 28: 1316 1323. 10 Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S et al. Bad reporting does not mean bad methods for randomized trials: 1557
1558 observational study of randomized controlled trials performed by the radiation therapy oncology group. BMJ 2004; 328: 22 25. 11 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159 174. 12 Hardin J, Hilbe J. Generalized Linear Models and Extensions. Stata Press: College Station, Texas, 2001. 13 Kline RB. Principles and Practice of Structural Equation Modeling. Guilford Press: New York, NY, 1998. 14 Lai R, Chu R, Fraumeni M, Thabane L. Quality of randomized controlled trials reporting in the primary treatment of brain tumors. J Clin Oncol 2006; 24: 1136 1144. 15 Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 2005; 365: 1159 1162. 16 Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004; 328: 22 24. 17 Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ et al. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol 2004; 57: 1232 1236. Appendix Table A1 Presents CONSORT reporting criteria. Table A1 CONSORT statement individual reporting criteria satisfied Heading Reporting criterion Percent (95% CI) of studies that satisfied reporting criterion Title Identifies study as an RCT 39.7 (27.6 51.8) Abstract Has a structured format 69.8 (58.5 81.2) Introduction Gives background of the study 100 Gives rationale of the study 93.7 (87.6 99.7) States clinical objectives 92.1 (85.4 98.7) Methods Patients and interventions States planned study population 79.4 (69.4 89.4) Gives inclusion criteria for participants 93.7 (87.6 99.7) Gives exclusion criteria for participants 82.5 (73.2 91.9) Defines weight loss intervention 98.4 (95.3 100) Gives time period of intervention 98.4 (95.3 100) Defines primary outcome measure(s) 46.0 (33.7 58.3) Defines secondary outcome measures(s) 27.0 (16.0 37.9) Sample size Gives sample size 100 States and justifies the desired significance level 52.4 (40.0 64.7) States and justifies the desired power 42.9 (30.6 55.1) Statistical analysis States whether the trial is by intention-to-treat 54.0 (41.7 66.3) Defines and justifies the analysis methods used to compare primary 95.2 (90.0 100) outcomes among groups Gives method of interim analysis if applicable 6.3 (0.3 12.4) Gives method of subgroup analysis if applicable 14.3 (5.6 22.9) Gives method of adjusted analysis if applicable 50.8 (38.4 63.1) Gives stopping rules if interim analysis was performed 92.1 (85.4 98.7) Treatment allocation Gives generation of random allocation sequence 22.2 (12.0 32.5) Details any restriction of the randomization 38.1 (26.1 50.1) Defines method of allocation concealment 9.5 (2.3 16.8) Defines separation of generator from executor 4.8 (0 10) Blinding of treatment Gives mechanism 52.4 (40.0 64.7) States similarity of placebo if applicable 71.4 (60.3 82.6) States location of randomization code 3.2 (0 7.5) Results Gives recruitment period 11.1 (3.4 18.9) Gives follow-up period 46.0 (33.7 58.3) Gives results of primary and secondary outcomes 98.4 (95.3 100) Gives baseline characteristics by treatment group 95.2 (90.0 100) Gives results as absolute numbers 96.8 (92.5 100) Gives appropriately labeled participant flow chart 25.4 (14.6 36.1) Gives and justifies results of adjusted analyses if applicable 98.4 (95.3 100) Gives and justifies results of subgroup analyses if applicable 95.2 (90.0 100) Gives number of subjects who had protocol deviation(s) with each 79.4 (69.4 89.4)
Table A1 (continued) 1559 Heading Reporting criterion Percent (95% CI) of studies that satisfied reporting criterion intervention Gives adverse events/side effects with each intervention 50.8 (38.4 63.1) Discussion Addresses study objectives 98.4 (95.3 100) States source of bias 20.6 (10.6 30.6) States impression 15.9 (6.8 24.9) Beware multiple analyses 7.9 (1.3 14.6) States external validity of the trial findings 58.7 (46.6 70.9) Interprets results in the context of current evidence 96.8 (92.5 100) Key CONSORT reporting criteria in methods and results sections are in bold type.