Assessment of Risk of Bias Among Pediatric Randomized Controlled Trials

Assessment of Risk of Bias Among Pediatric Randomized Controlled Trials Michael T. Crocetti, Diane D. Amin and Roberta Scherer Pediatrics 2010;126;298-305; originally published online Jul 12, 2010; DOI: 10.1542/peds.2009-3121 The online version of this article, along with updated information and services, is located on the World Wide Web at: http://www.pediatrics.org/cgi/content/full/126/2/298 PEDIATRICS is the official journal of the American Academy of Pediatrics. A monthly publication, it has been published continuously since 1948. PEDIATRICS is owned, published, and trademarked by the American Academy of Pediatrics, 141 Northwest Point Boulevard, Elk Grove Village, Illinois, 60007. Copyright 2010 by the American Academy of Pediatrics. All rights reserved. Print ISSN: 0031-4005. Online ISSN: 1098-4275.

Assessment of Risk of Bias Among Pediatric Randomized Controlled Trials WHAT S KNOWN ON THIS SUBJECT: The introduction of bias into the design and conduct of RCTs can affect the believability of the results severely. The Cochrane Collaboration has designed a domain-based, evaluation tool to address internal validity and the risk of bias directly. WHAT THIS STUDY ADDS: Many pediatric RCT reports demonstrated high risk of bias for sequence generation and allocation concealment. Industry funding and assessment of behavioral/educational interventions were associated with high risk of bias, whereas trial registration was associated with lower risk of bias. abstract OBJECTIVE: The goal was to assess the risk of bias among pediatric, randomized, controlled trials (RCTs) reported in 8 high-impact journals. METHODS: We searched PubMed for all pediatric RCTs reported between July 1, 2007, and June 30, 2008, in 8 journals with high impact factors. Using Cochrane Collaboration methods for risk assessment, we evaluated all reports for risk of bias according to domain (ie, randomized sequence generation, allocation concealment, masking, incomplete outcome data, selective outcome reporting, and other). We used multiple logistic regression to test for associations between the presence of a high risk of bias according to domain and funding source, intervention type, trial registration, and multicenter status. RESULTS: Industry-funded RCTs were more likely to show a high risk of bias for sequence generation, compared with government-funded RCTs (adjusted odds ratio [aor]: 6.1 [95% confidence interval [CI]: 1.70 21.89]), and behavioral/educational trials were more likely to show a high risk of bias for sequence generation (aor: 2.8 [95% CI: 1.06 7.36]) and allocation concealment (aor: 4.09 [95% CI: 1.69 9.90]), compared with drug trials. Registered trials were less likely to have a high risk of bias for sequence generation, compared with nonregistered trials (aor: 0.33 [95% CI: 0.15 0.71]). CONCLUSIONS: Overall, we found a large proportion of pediatric RCT reports with a high risk of bias for sequence generation and allocation concealment. Factors associated with a high risk of bias included industry funding and assessment of behavioral/educational interventions, whereas trial registration was associated with a lower risk of bias. Pediatrics 2010;126:298 305 AUTHORS: Michael T. Crocetti, MD, MPH, a Diane D. Amin, PA-C, a and Roberta Scherer, PhD b a Department of Pediatrics, Johns Hopkins Bayview Medical Center, Baltimore, Maryland; and b Department of Epidemiology, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland KEY WORDS randomized, controlled trial, risk of bias, high-impact journal, Cochrane Collaboration ABBREVIATIONS RCT randomized, controlled trial OR odds ratio CI confidence interval aor adjusted odds ratio www.pediatrics.org/cgi/doi/10.1542/peds.2009-3121 doi:10.1542/peds.2009-3121 Accepted for publication Apr 26, 2010 Address correspondence to Michael T. Crocetti, MD, MPH, Johns Hopkins Bayview Medical Center, Department of Pediatrics, 4940 Eastern Ave, Baltimore, MD 21224. E-mail: mcrocet1@jhmi.edu PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275). Copyright 2010 by the American Academy of Pediatrics FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose. 298 CROCETTI et al

ARTICLES Randomized, controlled trials (RCTs) are considered standard for assessments of the efficacy of therapeutic interventions. RCTs typically are designed to assess the efficacy of a drug, medical or surgical procedure, or behavioral/educational intervention. RCTs provide the highest level of evidence because, when conducted properly, they have lower risk of bias than do other study designs. Bias is a systematic error in the design of a trial that can lead to underestimation or overestimation of the true intervention effect. 1 Standard RCT design characteristics, such as randomization, masking, accounting for loss to follow-up monitoring, and appropriate reporting, serve to minimize bias and to ensure internal validity. Numerous tools have been developed to assess the validity or quality of RCTs. 2 These tools typically consist of a scale or checklist of items thought to be critical in the design and reporting of RCTs. Unfortunately, there are numerous differences among the tools regarding categories for review, complexity, and weight assigned to the individual domains thought to be critical to the control of bias. 3 Furthermore, many of the scales concentrate on reporting quality, rather than the internal validity of a given trial. 3 In assessments of the risk of bias among RCTs, it is important to distinguish simple reporting of procedures from the actual design and conduct of a study. To address directly internal validity and risk of bias, the Cochrane Collaboration designed a domain-based, evaluation tool recommended for use in systematic reviews of trial reports. 1 The domains within the tool are sequence generation, allocation concealment, masking, handling of incomplete outcome data, selective outcome reporting, and other potential threats to validity. These domains were chosen on the basis of empirical evidence that they affect the magnitude and direction of the treatment response and thus are measures of internal validity. 1 The Cochrane Collaboration, domainbased tool for assessment of the risk of bias has been used in the setting of systematic review of RCTs. In this setting, levels of bias are assigned to trials within a particular research area and results are pooled to yield an assessment of the believability of the treatment effect. Accompanying the assessment is a clear statement of why a particular domain was assessed as having low, high, or unclear risk of bias. Recently, the interrater reliability of the Cochrane Collaboration, risk-ofbias tool was evaluated by Hartling et al. 4 The authors also compared this tool with 2 other assessment tools. They demonstrated that the interrater reliability was good for some domains but poor for others. Furthermore, the tool required longer to complete, compared with the other tools, and there were weak correlations between the overall risks of bias assessed with the Cochrane Collaboration, risk-of-bias tool and the other measures. We were interested in assessing whether the tool could be used to evaluate the overall quality of RCTs conducted on a specific research topic or within a medical discipline. RCTs have the potential to alter significantly the clinical care of patients and to improve health outcomes. The introduction of bias into the design and conduct of a RCT can severely affect the quality of the research. Potentially more important is the fact that bias can render the results of a RCT invalid. The use of invalid results in medical decision-making may subject patients to undue risk of harm and therapeutic interventions that have no benefit. Trial results are generally disseminated through journal publications, and a measure of the importance of a particular article or of the journal itself is the impact factor. 5 High-impact journals publish numerous influential RCTs, which often set the standard of care in medicine. The general medical journals with the highest-rated journal impact factors in the 2006 Journal Citation Reports were the New England Journal of Medicine, the Journal of the American Medical Association, and The Lancet. 6 There is concern that RCTs performed with pediatric populations are not well represented in general medical journals. 7 Barriers to the publication of pediatric RCTs may include a decreased burden of disease in children, compared with adults, or ethical issues related to the recruitment of pediatric patients. Pediatric RCTs may be preferentially submitted to and accepted by pediatric journals, rather than general medical journals. Nevertheless, it is critical that pediatric trials use proper trial design and conduct in assessments of therapeutic interventions, to minimize bias and to improve internal validity. By using the Cochrane Collaboration, risk-of-bias tool, the objective of this study was to assess the risk of bias among pediatric RCTs reported during a 1-year period in 3 general medical journals and 5 pediatric journals with the highest impact factors. METHODS We conducted a PubMed search to identify all pediatric RCT reports published between July 1, 2007, and June 30, 2008. We selected the 5 pediatric journals and 3 general medical journals with the highest rated impact factors. The pediatric journals were Pediatrics (impact factor: 5.01), the Journal of the American Academy of Child and Adolescent Psychiatry (impact factor: 4.77), the Journal of Pediatrics (impact factor: 3.99), Archives of Pediatrics and Adolescent Medicine (impact factor: 3.57), and the Pediatric Infectious Disease Journal (impact factor: 3.22). The general medical jour- PEDIATRICS Volume 126, Number 2, August 2010 299

nals were the New England Journal of Medicine (impact factor: 51.30), The Lancet (impact factor: 25.80), and the Journal of the American Medical Association (impact factor: 23.18). Additional search criteria included English language, RCT, and all-child (0 18 years of age). Trial reports were excluded if the study population was predominately adult, participants were assigned to study groups in a nonrandomized manner, or the study was a follow-up report from a RCT reported before July 1, 2007. We abstracted information from the trial reports regarding trial characteristics, including the number of authors, the country in which the study was conducted, a broad research category, multicenter status, and whether the trial had been registered. In addition, we documented funding sources (government, industry, internal hospital grant, multiple sources, none, or private foundation) and trial intervention type (educational/behavioral, medicine/drug, nutritional supplement, or vaccine). By using Cochrane Collaboration methods for risk assessment, we evaluated all reports for risk of bias within 6 domains, including randomized sequence generation; allocation concealment; masking of participants, personnel, and outcome assessors; incomplete outcome data reporting; selective outcome reporting; and other sources of bias. By using a priori definitions, we classified each domain according to risk of bias, as high, low, or unclear (definitions provided in the Appendix). The assessment of masking and incomplete outcome data reporting was based on the primary trial outcomes. Two reviewers scored each report, and a third reviewer adjudicated disagreements. Before the start of the study, each reviewer pretested the assessment tool on 1 RCT from each of the 8 journals. For analysis, we dichotomized bias as low risk or high risk (high risk or unclear). Because 98% of the trial reports were classified as having a low risk of bias for selective outcome reporting and other biases, these domains were not included in the final regression analysis. We performed a simple descriptive analysis, calculating means for continuous variables and proportions for categorical variables. We used multivariate logistic regression to test for an association between the presence of a high risk of bias according to domain and the independent variables of funding source, intervention type, author number, and trial registration status. Regression models were explored by using likelihood ratio tests, forward and backward stepwise selection for each independent variable, and interaction terms (Stata 9 [Stata, College Station, TX]). Colinearity among the independent variables in the final model was checked by using variance inflation factors. The mean variance inflation factor was 1.06 (range: 1.00 1.11). A large proportion of the covariate pattern (137 of 146 RCTs) consisted of unique values; therefore, we used the Hosmer-Lemeshow goodness-of-fit test to check the logistic model for each domain. Odds ratios (ORs) with 95% confidence intervals (CIs) are reported. A 2-sided P value of.05 was considered statistically significant. RESULTS Trial Report Characteristics We identified 244 trial reports that met the search criteria. Ninety-eight articles were excluded, which left 146 RCTs for review. Studies were excluded because they included only adult participants (25 studies), were follow-up studies (49 studies), or were not randomized (24 studies). The majority of RCTs (116 [79%] of 146 RCTs) were reported in pediatric journals (Table 1). The mean number of authors for each TABLE 1 RCT Characteristics (N 146) Characteristic n (%) Journal Pediatrics 56 (39) Journal of Pediatrics 21 (14) The Lancet 15 (10) Archives of Pediatrics and 13 (9) Adolescent Medicine Journal of the American Academy of 13 (9) Child and Adolescent Psychiatry Pediatric Infectious Disease Journal 13 (9) New England Journal of Medicine 8 (5) Journal of the American Medical 7 (5) Association Funding source Government 44 (30) Industry 36 (25) Multiple sources 32 (22) Private foundation 22 (15) No funding 10 (7) Internal hospital grant 2 (1) Type of research Behavioral/educational interventions 59 (40) Comparative drug trials 53 (36) Vaccine trials 18 (12) Nutritional supplement evaluations 16 (12) article was 9 (median: 7 authors [range: 2 29 authors]), and an investigator with a PhD degree was listed as an author in almost all of the reports (131 [90%] of 146 RCTs). Sixty-five (45%) of the studies were multicenter, and 86 (59%) reported a clinical trial registration number. Reported funding sources and types of research are presented in Table 1. Most of the trials were conducted in the United States (63 [43%] of 146 RCTs) or in multiple countries (18 [12%] of 146 RCTs). Eighteen different medical categories were represented, with infectious diseases (38 [26%] of 146 RCTs) and neonatology (27 [18%] of 146 RCTs) being most common. Risk of Bias The bias designation for all of the RCTs according to domain is shown in Fig 1. Seventy-six (52%) of the trial reports required adjudication for 1 ofthe risk-of-bias domains. A high/unclear risk of bias was found in 41% of reports (60 of 146 reports) for sequence generation, 57% (83 of 146 reports) for 300 CROCETTI et al

ARTICLES Number n = 146 160 140 120 100 80 60 40 20 0 FIGURE 1 Risk of bias according to Cochrane Collaboration domain. Unclear High Risk Low Risk DISCUSSION By using a domain-based tool, we assessed the risk of bias among pediatric RCTs reported in general medical and pediatric, high-impact journals during a 1-year period. The Cochrane Collaboration, domain-based, riskassessment tool was introduced as a potentially reliable way to assess the quality of individual RCTs. It uses predefined bias indicators within domains that have been shown empirically to affect trials internal validity by altering the size of the treatment effect. We found that pediatric RCTs are con- allocation concealment, 19% (28 of 146 reports) for masking, 11% (16 of 146 reports) for incomplete outcome data, 2% (3 of 146 reports) for selective outcome reporting, and 2% (3 of 146 reports) for other biases. 2 3 4 5 Tables 2 through 5 show unadjusted ORs and adjusted ORs (aors) for a high risk of bias for the sequence generation, allocation concealment, masking, and incomplete outcome reporting domains. Industry-funded RCTs were more likely (aor: 6.10 [95% CI: 1.70 21.89]) and privately funded RCTs (aor: 0.23 [95% CI: 0.06 0.97]) were less likely to show a high risk of bias for sequence generation, compared with governmentfunded RCTs. Behavioral/educational trials were more likely to show a high risk of bias for sequence generation (aor: 2.80 [95% CI: 1.06 7.36]) and allocation concealment (aor: 4.09 [95% CI: 1.69 9.90]), compared with drug trials. Registered trials (aor: 0.32 [95% CI: 0.14 0.73]) and trials funded by private foundations (aor: 0.23 [95% CI: 0.06 0.97]) were less likely to have a high risk of bias for sequence generation, compared with nonregistered trials. As the number of authors increased, trial reports were less likely to have a high risk of bias for incomplete outcome reporting (aor: 0.78 [95% CI: 0.62 0.98]). TABLE 2 High Risk of Bias for Sequence Generation Domain (N 146) Variable Unadjusted OR (95% CI) P aor (95% CI) P No. of authors 0.96 (0.89 1.03).26 1.01 (0.92 1.11).82 Registered 0.37 (0.19 0.74).005 0.32 (0.14 0.73).006 Funding Government Industry 2.56 (1.03 6.33).04 6.10 (1.70 21.89).005 Private 0.23 (0.06 0.89).03 0.23 (0.06 0.97).05 Internal grant 1.44 (0.08 24.63).80 2.74 (0.14 55.12).51 Multiple sources 0.57 (0.21 1.50).25 0.60 (0.21 1.71).34 None 2.17 (0.53 8.79).28 1.85 (0.41 8.36).42 Research type Medicine/drug Vaccine 1.65 (0.56 4.85).36 0.55 (0.14 2.21).40 Educational/behavioral 1.39 (0.65 2.96).39 2.80 (1.06 7.36).04 Nutritional supplement 0.55 (0.16 1.94).35 1.42 (0.31 6.50).65 The adjusted regression model included author number, registration status, funding source, and research type. ORs of 1 indicate a high risk of bias. Hosmer-Lemeshow goodness of fit: 28 9.02; P.34. TABLE 3 High Risk of Bias for Allocation Concealment Domain (N 146) Variable Unadjusted OR P aor (95% CI) P (95% CI) No. of authors 0.96 (0.89 1.03).28 1.02 (0.93 1.11).71 Registered 0.80 (0.41 1.57).52 0.69 (0.32 0.1.50).35 Funding Government Industry 0.65 (0.26 1.60).35 2.24 (0.68 7.38).19 Private 0.62 (0.22 1.77).37 0.65 (0.21 2.01).45 Internal grant 0.52 (0.03 8.86).80 0.70 (0.03 15.00).82 Multiple sources 0.46 (0.18 1.16).10 0.50 (0.18 1.35).17 None 0.78 (0.19 3.18).72 0.78 (0.17 3.54).75 Research type Medicine/drug Vaccine 0.56 (0.18 1.71).31 0.28 (0.07 1.06).06 Educational/behavioral 3.01 (1.37 6.62).006 4.09 (1.69 9.90).002 Nutritional supplement 1.44 (0.47 4.44).53 2.16 (0.62 7.46).22 The adjusted regression model included author number, registration status, funding source, and research type. ORs of 1 indicate a high risk of bias. Hosmer-Lemeshow goodness of fit: 28 7.40; P.49. PEDIATRICS Volume 126, Number 2, August 2010 301

TABLE 4 High Risk of Bias for Masking Domain (N 146) Variable Unadjusted OR P aor (95% CI) P (95% CI) No. of authors 0.95 (0.86 1.05).33 0.99 (0.88 1.11).83 Registered 0.91 (0.40 2.10).83 0.86 (0.34 2.17).76 Funding Government Industry 0.49 (0.14 1.73).27 1.09 (0.24 4.88).91 Private 1.14 (0.33 3.94).83 1.24 (0.34 4.50).74 Internal grant 3.89 (0.22 68.38).35 4.76 (0.23 98.73).31 Multiple sources 1.09 (0.36 3.31).88 1.27 (0.40 3.98).69 None 0.97 (0.18 5.40).98 0.97 (0.17 5.74).98 Research type Medicine/drug Vaccine 0.39 (0.04 3.38).39 0.41 (0.04 4.24).46 Educational/behavioral 2.66 (1.00 7.05).05 2.67 (0.92 7.75).07 Nutritional supplement 1.52 (0.34 6.70).58 1.52 (0.32 7.31).60 The adjusted regression model included author number, registration status, funding source, and research type. ORs of 1 indicate a high risk of bias. Hosmer-Lemeshow goodness of fit: 28 8.5; P.39. TABLE 5 High Risk of Bias for Incomplete Outcome Domain (N 146) Variable Unadjusted OR P aor (95% CI) P (95% CI) No. of authors 0.82 (0.68 0.99).04 0.78 (0.62 0.98).04 Registered 0.38 (0.13 1.10).07 0.55 (0.18 1.72).30 Funding Government Industry 1.02 (0.28 3.67).97 0.62 (0.12 3.24).57 Private 0.63 (0.12 3.43).60 0.65 (0.11 3.73).63 Internal grant Multiple sources 0.42 (0.08 2.24).31 0.41 (0.07 2.38).32 None 0.70 (0.08 6.60).76 0.36 (0.03 3.71).39 Research type Medicine/drug Vaccine 1.57 (0.35 7.04).56 0.55 (0.14 2.21).40 Educational/behavioral 0.89 (0.27 2.94).84 0.71 (0.20 2.56).60 Nutritional supplement 0.52 (0.06 4.69).56 0.45 (0.04 4.47).49 The adjusted regression model included author number, registration status, funding source, and research type. ORs of 1 indicate a high risk of bias. Hosmer-Lemeshow goodness of fit: 28 6.35; P.61. ducted throughout the world and across many medical categories; however, a large proportion of the trials were determined to have a high or unclear risk of bias for sequence generation and allocation concealment. Bias within these reporting domains may significantly affect the internal validity of the trial results, and caution should be taken in interpretation of the findings of these studies. The Cochrane Collaboration, domainbased, risk-assessment tool represents a method to assess objectively the internal validity of RCTs. The first domain in the tool, randomized sequence generation, protects against selection bias and increases the chance that, on average, predictor variables are balanced between intervention groups. We found a high or unclear risk of bias for sequence generation in 41% of the trial reports. Authors often reported that the trial was randomized without stating how the randomization sequence was generated. In this study, industry-funded RCTs were 6 times more likely to have a high or unclear risk of bias for sequence generation, compared with government-funded RCTs. Studies revealed that randomized drug trials funded by for-profit organizations such as industry sources were 4 to 5 times more likely to recommend the experimental drug, compared with trials funded by nonprofit organizations. 8,9 For results such as these to be believable, it is imperative that pediatric RCTs funded by industry sources exercise more care in their reporting of the methods used to assign participants to study groups. Allocation concealment further protects against selection bias. When performed properly, allocation concealment ensures unpredictability in the next treatment assignment. In 57% of trial reports, we found a high or unclear risk of bias for allocation concealment. RCTs in which allocation concealment is inadequate yield exaggerated treatment effects. 10,11 All RCTs should have methods in place to shield investigators and participants from knowing the next treatment assignment during the period of time from sequence generation until the participant is placed in a treatment group. We found that trials assessing behavioral or educational interventions were 4 times more likely to have a high or unclear risk of bias for allocation concealment, compared with drug trials. Masking of participants, personnel, and outcome assessors protects against information bias. We found that 19% of the trial reports had a high or unclear risk of bias for masking and, although results did not reach statistical significance (P.07), trials involving behavioral or educational interventions were 3 times more likely to have a high risk of bias for this domain, compared with drug trials. The most likely explanation for this is that masking participants and assessors to the intervention in behavioral/educational trials is extremely difficult or impossible, because of the underlying design. In addition, outcomes in behavioral/ educational trials tend to be subjective, which places such trials at higher 302 CROCETTI et al

ARTICLES risk of bias for an exaggerated treatment effect. 1 Attrition bias occurs when not all randomly assigned study participants complete the trial or some outcome assessments are not completed and not taken into account in the analysis. When data are missing, they should be balanced across intervention groups and an intention-to-treat analysis performed. In addition, statistical methods used to deal with missing data should be described. Wood et al 12 showed that missing outcome data often are not handled with adequate statistical methods in RCT reports published in high-impact general medical journals. Overall, 11% of the trial reports in this study had a high or unclear risk of bias for incomplete outcome data reporting. Trial reports were 22% less likely to be at high risk for incomplete outcome reporting for each additional author. It is not clear why increased numbers of authors would be associated with proper reporting of incomplete outcomes. In 2005, the International Committee of Medical Journal Editors developed a policy that information about RCTs should be deposited in a clinical trial registry. 13 The purpose of this policy was to make the details of trials with human subjects transparent to the public. ClinicalTrials.gov, a large, USbased, trial register, currently contains 70 000 studies from 164 countries. 14 The number of trials in the registry continues to increase, and this may be an indication that more journals are requiring trial registration numbers for pediatric RCT submissions. Clinical trial registration is required by 5 of the 8 high-impact journals included in this study and is encouraged by 2 others. We found that only 59% of pediatric RCTs (86 of 146 RCTs) reported trial registration numbers. Furthermore, for journals that require registration numbers, only 69% of RCTs (68 of 99 RCTs) reported registration numbers. On the basis of our findings, efforts to improve trial registration for pediatric RCTs are needed. Registered trials were 68% less likely to have a high risk of bias for sequence generation, compared with nonregistered trials. Increasing the number of pediatric RCTs that are registered would make trial information more accessible to the public and also could be a marker for a reduced risk of bias in trial design. There were some limitations in this study. The risk assessment was focused on pediatric RCT reports published during a 1-year period, and the majority of the reports were published in pediatric journals. Therefore, our findings may not be generalizable to RCTs performed with adult populations and pediatric RCTs reported in general medical journals. In addition, the domain-based, risk-assessment tool developed by the Cochrane Collaboration is not a validated instrument. Even with prespecified definitions, we found a large amount of disagreement among reviewers, in that 53% of studies required adjudication. These results are similar to those reported by Harlting et al 4 and reflect the amount of subjective judgment inherent in assessment of some of the domains of the Cochrane Collaboration, risk-ofbias tool. For a systematic review, the assessment is accompanied by a specific reason for scoring a domain as having a high, low, or unclear risk of bias, to ensure transparency. Our results and those of Hartling et al 4 suggest that using the tool without the accompanying explanations may be problematic. This limitation is highlighted by our assessment of selective outcome reporting and other biases. Although we found few studies with possible selective outcome reporting bias, this might be because of lack of content expertise or protocols to compare with the published report. Conversely, by using prespecified definitions with adjudication, we were able to identify factors associated with a high risk of bias. Finally, given the design of this study, we could assess only what was reported and not necessarily what actually occurred during the conduct of the study. CONCLUSIONS Pediatric RCTs are conducted throughout the world, across numerous medical categories, to test many different types of interventions. However, there was incomplete adherence to journal requirements for trial registration. Overall, we found a large proportion of pediatric trial reports with a high risk of bias for sequence generation and allocation concealment. Factors associated with a high risk of bias included industry funding and assessment of behavioral/educational interventions, whereas trial registration, private foundation funding, and increased author numbers were associated with a lower risk of bias. Results of pediatric trials with a high risk of bias should be interpreted with caution. APPENDIX: CODEBOOK FOR ASSESSMENT OF RISK OF BIAS Allocation Sequence Generation 1. Low risk: random number table, computer-generated, with minimization. 2. High risk: odd or even date of birth, rule based on date of admission, hospital or clinic record number, clinician judgment, participant preference, or intervention availability, or tossing coins, shuffling cards or envelopes, throwing dice, or drawing lots. 3. Unclear: randomization is stated but process is not described. PEDIATRICS Volume 126, Number 2, August 2010 303

Allocation Concealment 1. Low risk: central allocation (pharmacy-, telephone-, or Internetbased); for drug trials, medication containers are numbered and identical in appearance and placebo drugs are made to taste and to look like intervention drugs; envelopes numbered, sealed, and opaque. 2. High risk: any procedure in which participants or researchers could have foreseen allocation; assignment envelopes unsealed, seethrough, or not numbered; any alternation procedure that would allow an individual to predict allocation. 3. Unclear: insufficient evidence that allocation concealment was appropriate (eg, envelopes were used but it was not described whether they were numbered, sealed, and opaque). Masking Procedures 1. Low risk: masking procedures for all key participants, including study participants, treatment administrators, and outcome assessors, were described and it was unlikely that masking could have been broken; partially masked but outcome assessment was masked and unmasked participants were not thought to introduce bias; unmasked but outcome was not thought to be influenced by masking. 2. High risk: no masking or incomplete masking for an outcome that was likely to be affected by masking; masking procedures could have REFERENCES 1. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.1 [updated September 2008]. Baltimore, MD: Cochrane Collaboration; 2008. Available at: www.cochrane-handbook.org. Accessed November 10, 2008 been broken; unmasked participants could have introduced bias for those who were unmasked. 3. Unclear: insufficient evidence to state yes or no; study did not address masking. Incomplete Outcome Data 1. Low risk: no missing outcome data; reasons for missing data were unlikely to be related to outcomes (such as censored data); missing outcome data were balanced across intervention groups, there were similar reasons for missing data, and reasons were unrelated to outcomes; data were analyzed with intention-to-treat approach according to randomization group; statistical methods were described, were used to deal with missing data, and were not thought to introduce bias. 2. High risk: reasons for missing data were likely related to outcomes (ie, disproportionate numbers of dropouts in the intervention group because of adverse effects or exclusion of participants who failed to experience improvement); intention-to-treat analysis performed only for participants remaining in the study. 3. Unclear: insufficient reporting of dropouts and exclusions for clear decision; numbers randomized in each group were not clearly reported. 2. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995;16(1): 62 73 Selective Outcome Reporting 1. Low risk: all outcomes described are included and reported in analysis; for registered trials, all outcomes reported are included in analysis; all outcomes expected to have been collected for condition are reported. 2. High risk: not all prespecified outcomes are reported; 1 outcome is not reported completely; analytic methods or subsets of data used to report outcomes were not prespecified; authors fail to report key outcome expected for study. 3. Unclear: insufficient information for clear decision. Other Sources of Bias 1. Low risk: study seems to be free of other sources of bias. 2. High risk: for cluster-randomized trials, individuals were recruited to the study after clusters were randomized, complete clusters were lost from trial, or statistical methods were not used to account and to correct for clustering; for crossover trials, no or insufficient washout period and treatments were likely to have been carried over from one period to the next, so outcomes might differ depending on the order in which participants received treatments; study was stopped early before calculated sample size was reached, with no explanation given; extreme baseline imbalance between study groups. 3. Unclear: insufficient information for assessment of whether another source of bias exists. ACKNOWLEDGMENT Dr Scherer is a member of the CONSORT group and the Cochrane Collaboration. 3. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282(11): 1054 1060 4. Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of random- 304 CROCETTI et al

ARTICLES ised controlled trials: cross sectional study. BMJ. 2009;339:b4012 5. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295(1): 90 93 6. ISI Web of Knowledge. Journal Citation Reports. Available at: http://admin-apps.isiknowledge. com/jcr/jcr?sid 2DaK7OpgGi51pLo8CaG. Accessed June 7, 2010 7. Cohen E, Uleryk E, Jasuja M, Parkin PC. An absence of pediatric randomized controlled trials in general medical journals. J Clin Epidemiol. 2007;60(2):118 123 8. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? JAMA. 2003;290(7):921 928 9. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systemic review. BMJ. 2003;326(7400): 1167 1176 10. Pildal J, Hróbjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gøtzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36(4): 847 857 11. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273(5):408 412 12. Wood AM, White IR, Thompson SG. Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials. 2004;1(4):368 376 13. DeAngelis CD, Drazen JM, Frizelle FA, et al. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. JAMA. 2005; 293(23):2927 2929 14. US National Institutes of Health. About ClinicalTrials.gov. Available at: www. clinicaltrials.gov/ct2/home. Accessed April 1, 2009 Ratting Out the Enemy in Africa: It s hard to believe but a Dutch company has successfully trained African giant pouched rats, which have poor vision but superb senses of smell, to detect landmines. According to an article in The New York Times (Kristof ND, June 16, 2010), these large (30 inch long) rats are too light to set off mines, but upon exploring a suspected area, can point with their noses to buried mines and literally rat them out. In fact this particular species of rat, once trained over many months, can do as much land surveillance for mines in 20 minutes as a human can do in 2 days. What is even more impressive is that these rats can also use their noses to detect cases of tuberculosis. When sputum samples are given to the rats, they apparently can detect the bacteria through smell faster than a technician can look for them with a microscope. According to the article, a technician in Tanzania can screen 40 samples a day while one giant rat can do the same amount of work in only seven minutes. As to cost, the company notes that a years worth of bananas to feed one rat is only $36 and $100 will provide the materials for breeding these rodents. Noted by JFL, MD PEDIATRICS Volume 126, Number 2, August 2010 305

Assessment of Risk of Bias Among Pediatric Randomized Controlled Trials Michael T. Crocetti, Diane D. Amin and Roberta Scherer Pediatrics 2010;126;298-305; originally published online Jul 12, 2010; DOI: 10.1542/peds.2009-3121 Updated Information & Services s Permissions & Licensing Reprints including high-resolution figures, can be found at: http://www.pediatrics.org/cgi/content/full/126/2/298 This article cites 11 articles, 9 of which you can access for free at: http://www.pediatrics.org/cgi/content/full/126/2/298#bibl Information about reproducing this article in parts (figures, tables) or in its entirety can be found online at: http://www.pediatrics.org/misc/permissions.shtml Information about ordering reprints can be found online: http://www.pediatrics.org/misc/reprints.shtml