An absence of pediatric randomized controlled trials in general medical journals, 1985e2004

Transcription

1 Journal of Clinical Epidemiology 60 (2007) 118e123 An absence of pediatric randomized controlled trials in general medical journals, 1985e2004 Eyal Cohen a,b, *, Elizabeth Uleryk c, Mona Jasuja a, Patricia C. Parkin a,b a Division of Paediatric Medicine and the Paediatric Outcomes Research Team, Department of Paediatrics, The Hospital for Sick Children, University of Toronto, Faculty of Medicine, Toronto, ON, Canada b Population Health Sciences, The Hospital for Sick Children Research Institute, University of Toronto, Faculty of Medicine, Toronto, ON, Canada c The Hospital for Sick Children, Hospital Library, Toronto, ON, Canada Accepted 25 March 2006 Abstract Objective: There are numerous potential barriers to conducting randomized controlled trials (RCTs) in children. The purpose of this study was to compare the quantity, trends over time, characteristics, and quality of pediatric RCTs published in general medical journals (GMJs) with adult RCTs. Study Design and Setting: We conducted an electronic search of adult and pediatric RCTs from 1985e2004 and a manual search of published RCTs in the year 2000 in five high-impact GMJs (New England Journal of Medicine, Journal of the American Medical Association [JAMA], the Lancet, British Medical Journal [BMJ], Canadian Medical Association Journal [CMAJ]). Linear trends were identified and the 1-year sample was analyzed for publication characteristics (location of recruitment, sample size, number of centers, funding sources, and results) and quality scoring (Jadad score, intention-to-treat analysis, and citation frequency since publication). Results: Adult RCTs increased by 4.71 RCTs/year (95% confidence interval (CI) 3.62e5.80; P! 0.001), which was significantly higher (P! ) than pediatric RCTs, which increased by 0.4 RCTs/year (95% CI 0.02 to 0.9; P ). Adult RCTs were more likely to be hospital-based (P 5.001) and to involve more centers in multicenter studies (P ). Quality scores were similar, although adult RCTs were cited more frequently (P ). Conclusion: There may be significant barriers to the publication of high-quality pediatric RCTs in GMJs. Ó 2007 Elsevier Inc. All rights reserved. Keywords: Publications; Pediatrics; Randomized controlled trials; General medical journals; Citations; Controlled clinical trials 1. Introduction Clinical trials have led to important changes in clinical practice and health outcomes for children. Some examples of this are the marked reduction in mortality from childhood acute lymphoblastic leukemia and from extreme prematurity as a result of well-conducted multicenter trials [1,2]. However, pediatric practitioners and researchers commonly lament the lack of high-quality clinical trials in children [3]. This is particularly true of randomized controlled trials (RCTs), which are considered the gold standard of clinical trials, as they can provide unconfounded estimates of intervention effects when conducted appropriately [3,4]. * Corresponding author. Division of Paediatric Medicine, Department of Paediatrics, The Hospital for Sick Children, 555 University Avenue, Toronto, ON M5G 1X8, Canada. Tel.: ; fax: address: eyal.cohen@sickkids.ca (E. Cohen). General medical journals (GMJs) are an important source of influential pediatric publications. They are widely read by both generalists and specialists and are widely cited. An assessment of pediatric studies referenced in the Cochrane Database of Systematic Reviews revealed that four out of the top six journals cited were GMJs [5]. All of these GMJs have a higher impact factor than any clinical pediatric journal. Some authors have argued that GMJs under-report studies of diseases of poverty or studies from resource-poor countries and thus poorly represent the global burden of disease [6e8]. Many of these conditions disproportionately affect children (e.g., gastroenteritis, meningitis, nonintentional injury), and it can therefore be hypothesized that there may be under-reporting of trials of important diseases of childhood in GMJs. The objectives of this study are to assess the quantity, trend over time, characteristics, and quality of RCTs involving children published in GMJs and to compare these with RCTs involving adults /07/$ e see front matter Ó 2007 Elsevier Inc. All rights reserved. doi: /j.jclinepi

2 E. Cohen et al. / Journal of Clinical Epidemiology 60 (2007) 118e Methods 2.1. Electronic search strategy An electronic search of OVID MEDLINE was conducted to identify all published RCTs for the years 1985e2004 from five GMJs (New England Journal of Medicine, the Lancet, Journal of the American Medical Association [JAMA], British Medical Journal [BMJ], and Canadian Medical Association Journal [CMAJ]). These journals were chosen as they have the highest impact factor of the GMJs within the medicine, general, and internal category of the Institute for Scientific Information s journal citation reports for 2004 ( according to the formula described by Garfield [9,10]. The search strategy described by Robinson and Dickersin [11] was used. This method was designed as a highly sensitive search strategy for identifying RCTs. Identification of studies with an RCT design was done using Phase 1 of the strategy. In this phase of the strategy, publications are limited to two types ( randomized controlled trial or controlled clinical trial ) and five MeSH headings ( randomized controlled trials/ or random allocation/ or double-blind method/ or single-blind method/, and human/ ). All letters, comments, and editorial articles were excluded. Published studies were subdivided into agespecific (0e18 years; >19 years; both 0e18 and >19 years), and age nonspecific articles. A trial run of this search revealed that it was overly sensitive, yielding numerous methodological reviews without patient recruitment. Therefore, the strategy was further modified to remove MeSH headings in order to increase the specificity of the strategy. The final search strategy, as applied to one journal for 1 year, is shown in Table Validation of electronic search strategy In order to validate the electronic search strategy for RCTs, a manual search of these five GMJs was conducted for a single calendar year, This year was chosen by convenience to capture a sample of complete and correctly coded RCTs. RCTs were defined as studies in which interventions were allocated at random in either a parallel or a crossover design. Every original article was reviewed except annotations, case reports/series, review articles, or letters to the editor. Only publications in which patients were the subjects were included. Excluded, for example, were studies without patient recruitment, where the unit of study was a community, hospital, or physician Publication characteristics and quality assessment Data from the manual search of the five GMJs from 2000 were analyzed to assess publication characteristics. Using a standardized collection form, data were collected on the following variables: study population (pediatric vs. adult), location of recruitment (inpatient vs. outpatient; Table 1 Example of search strategy in OVID MEDLINE for determination of RCTs in 1985 for the British Medical Journal Set History Results Comment 1 (Randomized controlled trial or CCT). pt. 264,948 RCT or CCT as publication type 2 (BMJ or British medical 81,106 Journal title search journal or British medical journal clinical research ed). jn. 3 Limit 2 to yr ,728 Limit to specific year 4 Limit 3 to (comment or editorial or letter) 724 Limit to comment, editorial or letter as publication type 5 3 not 4 1,004 Eliminating editorials, comments or letters 6 1 and 5 88 Total number of RCTs or CCTs published in that 7 Limit 6 to all child!0e18 yearso 8 Limit 6 to all adult!19 plus yearso 27 Total number of pediatric RCTs published in that 63 Total number of adult RCTs published in that 9 6 not (7 or 8) 15 Total number of RCTs not indexed to an age group published in that 10 7 not 8 10 Total number of pediatric RCTs without adult subjects in that 11 8 not 7 46 Total number of adult RCTs without pediatric subjects in that 12 (7 or 8) not (10 or 11) 17 Total number of RCTs with both adult and pediatric subjects in that Abbreviations: CCT, controlled clinical trial; pt, publication type; jn, journal. community vs. hospital), sample size, number and location of centers involved (by continent), funding sources (industry sponsorship vs. no industry sponsorship), and results (positive vs. negative). In order to create appropriate comparative groups, we excluded trials in which both adults and children were enrolled (n 5 25) and studies which randomized adults but reported important pediatric outcomes (e.g., neonatal outcomes of pregnancy; n 5 14). A quality assessment was performed in a sample of publications. Since it was expected that there would be more publications involving adults than children, quality scoring was performed on all publications involving solely pediatric subjects (<18 years) and a 1:1 matched sample of publications involving only adults (>19 years) for each of the five GMJs. Sampling was performed by computergenerated block randomization for each journal. The Jadad score [12], a validated three-item quality assessment tool of

3 120 E. Cohen et al. / Journal of Clinical Epidemiology 60 (2007) 118e123 randomization, blinding, and description of withdrawals and drop outs was used. Scoring by two different assessors (EC, MJ) was performed and, in the event of any differences, the score was determined by consensus. This scale has been criticized for its limited applicability to some medical fields [13] and its inability to differentiate between poorly reported and poorly conducted trials [14]. Therefore, an additional assessment of the presence of intention-totreat analysis in studies with dropouts or withdrawals was performed. Finally, the frequency of citation since publication was determined using a cited reference search from the Institute for Scientific Information ( accessed in June Data analysis Comparisons of linear trends in the total number of RCTs published over time in studies categorized as adult, pediatric, both adult and pediatric, and no age specified were modeled using linear regression. An interaction term was included in order to determine if the slopes of the lines were different. A sensitivity analysis was performed assuming erroneous classification of studies with both adult and pediatric subjects and studies with no age specified. Validation of the electronic search strategy was assessed using the Cohen k statistic, comparing the number of RCTs in the electronic and manual search for the year 2000 for all five GMJs. Publication characteristics and quality assessment of adult and pediatric studies from the manual search for the year 2000 were compared using the c 2 test or the Manne Whitney U test. All P-values were two-tailed and analyses were conducted using SAS version 9 (Cary, NC, USA) and SPSS for Windows version 13 (Chicago, IL, USA). 3. Results 3.1. Electronic search strategy The electronic search strategy produced a total of 69,755 citations of which 5,420 (8% of all citations) were RCTs. Fig. 1 summarizes the breakdown of RCTs by age. In the 20 years reviewed, almost five times as many RCTs with only adult subjects (3,328) were published as compared to RCTs with solely pediatric subjects (701). Publication of RCTs with only adult subjects increased steadily over the 20-year period, with a mean of 166 RCTs per year ranging from 122 in 1985 to 223 in In contrast, publications of pediatric RCTs were more variable with a mean of 35 RCTs per year, a minimum of 28 in 1986 and 1987 and a maximum of 48 in Fig. 2 demonstrates secular trends over time in pediatric RCTs, adult RCTs, and RCTs with both adults and children published in GMJs. There was a significant increase in the number of adult RCTs published in the years 1985e2004 with a rise of 4.71 RCTs/ year (95% CI 3.62e5.80; P! ), which was significantly higher (P! ) than pediatric RCTs which increased by only 0.44 RCTs/year (95% CI 0.02 to 0.09; P ), and RCTs with both children and adults which increased by 0.23 RCTs/year (95% CI 0.34 to 0.80; P ). RCTs with no age identified decreased over time by 1.03 RCTs/year (95% CI 1.27 to 0.78; P! ). The difference in slopes between adult and pediatric RCTs persisted (P! ) after sensitivity analyses were performed assuming that all RCTs with no age identified and/or all RCTs with both pediatric and adult subjects were truly adult RCTs Validation of electronic search strategy There was strong agreement between the electronic and hand search for the year 2000 (k ; 95% CI 0.88e0.94) Publication characteristics and quality assessment Table 2 summarizes the data comparing adult and pediatric RCTs published in 2000 in the five GMJs. There was no significant difference in median sample size, or the proportion of multicenter, multinational, industry-funded, or nonnorth American studies. However, of multicenter RCTs, those involving adults were more likely to involve more centers. In addition, adult RCTs were more likely to Total References 69,755 Controlled Clinical or Randomized Controlled Trials (RCTs) 5420 (8% of all references) RCTs with age identified 4897 (90%) RCTs with no age identified 523 (10%) RCTs with age 0-18 y 701 (14%) RCTs with age 19+ y 3328 (68%) RCTs with both age groups 868 (18%) Fig. 1. Flow diagram of results of electronic search strategy of GMJs in OVID MEDLINE 1985e2004.

4 E. Cohen et al. / Journal of Clinical Epidemiology 60 (2007) 118e Adults Children Both Adults and Children 200 Number of RCTs Year Fig. 2. Trends in publication of RCTs by age in five GMJs 1985e2004. be hospital-based. Table 3 summarizes indicators of quality in adult and pediatric RCTs. The two raters agreed on assigned scoring in most (54/66; 82%) of the studies reviewed. Although there was no difference in Jadad scores or intention-to-treat analysis, studies involving adult subjects were cited more frequently than those involving children. 4. Discussion This study demonstrates that the average number of RCTs involving adult subjects published each year in five high-impact GMJs over 20 years has almost doubled while there has been no significant change in the number of RCTs published involving pediatric subjects. An analysis of 1 year of publications demonstrates that the characteristics of both adult and pediatric RCTs are similar except that adult RCTs are more frequently hospital-based and involve more centers. Quality scores of a sample of adult and pediatric RCTs are comparable although adult RCTs have been cited more frequently. A variety of explanations can be made for the paucity of pediatric RCTs published in GMJs. Diseases of children in the developed world are relatively uncommon compared with adult-onset disorders and the cost of carrying out clinical trials is relatively expensive. There are also unique ethical and logistical challenges in conducting clinical Table 2 Publication characteristics of pediatric and adult RCTs published in 2000 in five GMJs. All values expressed as n (%) [95% confidence interval] unless stated otherwise Pediatrics (n 5 34) Adults (n 5 202) P-value Sample size a 272 (29e13,677) 318 (7e46,551) 0.80 Multicenter 22 (65%) [48%e78%] 136 (67%) [61%e73%] 0.78 Number of centers a,b 9(2e53) 19 (2e1,032) 0.02 Multinational 6 (18%) [8%e33%] 55 (27%) [22%e34%] 0.23 Industry funded 18 (53%) [37%e69%] 91 (45%) [38%e52%] 0.41 Positive results 28 (82%) [67%e92%] 152 (75%) [69%e81%] 0.36 North America c 13 (38%) [24%e55%] 106 (52%) [46%e59%] 0.11 Hospital-based 18 (54%) [38%e70%] d 164 (85%) [80%e90%] e a expressed as median (range). b of multicenter trials. c based on the geographic location of the corresponding author. d unable to determine in one study. e unable to determine in 10 studies.

5 122 E. Cohen et al. / Journal of Clinical Epidemiology 60 (2007) 118e123 Table 3 Quality assessment of pediatric and adult RCTs published in 2000 in five general medical journals Pediatrics, (n 5 34) Adults, (n 5 34) P-value Jadad Score a,b 3(1e5) 3 (1e5) 0.65 Randomization a,b 1(0e2) 2 (0e2) 0.49 Double-blinding a,b 0(0e2) 0 (0e2) 0.46 Withdrawals/dropouts a,c 26 (76%) 32 (94%) 0.07 [60%e88%] [81%e98%] ITT d analysis done c,e 18 (53%) 20 (61%) 0.53 [37%e69%] [42%e74%] Citation frequency b,f 32 (1e267) 101 (2e991) a using the Jadad assessment scale [12]. b expressed as median (range). c expressed as n (%) (95% confidence interval). d ITT 5 intention-to-treat. e for studies with withdrawals and/or dropouts. f using the cited reference search from the Institute for Scientific Information ( accessed in June research in children. For instance, there are serious ethical concerns about obtaining proxy consent from caregivers to enroll children in trials and about providing financial incentives for parents of minors to enroll them in studies [15]. Pediatric educators have also identified problems in encouraging trainees to pursue research careers and challenges in obtaining funding for research training [16]. Until relatively recently, there has been little financial incentive for pharmaceutical companies to conduct studies in children on drugs that are used in adults [17]. This major problem has been highlighted by recent controversies over the safety of some of these medications, such as the reported increased risk of suicidality in adolescents taking anti-depressants [18]. Recent legislative efforts in the United States, such as the Food and Drug Administration (FDA) Modernization Act (1997) [17], providing market exclusivity to drug companies conducting pharmaceutical trials in children, and the Pediatric Research Equity Act (2003) which gives the FDA the authority to require drug manufacturers to perform pediatric testing on certain drug products, have been met with significant legal resistance [3,19]. Despite a reported increase in the number of drug trials being conducted in children [20], we have not yet detected any noticeable rise in publication of pediatric RCTs in GMJs in recent years, which may be attributable to publication lag time. In an analysis of 1 year of publications, the finding that pediatric studies involved fewer centers than adult studies and were less likely to be hospital-based is not surprising given the relatively small number of pediatric hospitals relative to general or adult hospitals. Increased citation frequency of adult studies may be related to study quality, but may also be related to the size of the research field. The fact that there are more journals geared toward health care workers who treat adults than those who treat children, may result in more citations of studies involving adults. Forrest et al. [21] reviewed pediatric outcomes research undertaken in the United States and published in 39 peerreviewed journals over the time period 1994e1999. Of the 314 citations that met their definition of pediatric outcomes research, only 10% had an RCT design. The authors suggested that pediatric outcomes research, which can inform policy and clinical practice, is in its nascent stages of development. Similarly, Cramer et al. [22] described a paucity of child-relevant evidence synthesis in their review of drug interventions in the Cochrane Database of Systematic Reviews. Campbell et al. [23] undertook a descriptive study based on hand searching 15 years of Archives of Diseases of Childhood (1982e1996) to identify all RCTs. A total of 249 RCTs were identified ( 15/year) with no apparent increase over time. Almost half had a sample size of less than forty, 18% were multicentered and 12% were set in the community. In contrast, we found that a sample of pediatric RCTs published in GMJs tended to have a larger sample size, more centers and more communitybased studies. This may reflect a bias toward publication of higher quality pediatric RCTs in GMJs. The electronic search strategy used in this study has been found to be highly sensitive for identifying RCTs. However, in order to improve specificity, it was necessary to make some modifications to the electronic search strategy. Although this modified strategy has not been previously studied, validation with 1 year of manually abstracted data supports its accuracy. Further, this strategy is limited in its ability to identify the subject age in all publications; 10% were found to have age unspecified and 18% were found to include subjects in both the pediatric and adult age groups. However, our sensitivity analysis demonstrated that even assuming that all of these publications were misclassified would not have changed the results of this study. In the manual search, 1 year of data were selected at convenience, primarily to validate the electronic search strategy. Interpretation of the secondary comparative analyses (study characteristics and quality) may be limited by a lack of power to detect more differences between the two groups. Finally, the study was limited to RCTs published in five GMJs with the highest impact factor, and therefore not necessarily representative of all of clinical medicine. Any attempt to extrapolate the results of this study to all RCTs needs to be made with caution, as many high-quality pediatric RCTs may be published in other general and subspecialty journals. 5. Conclusions/future studies The long-held contention that clinical studies in children are underrepresented needs to be supported with evidence, so that potential barriers can be identified and resolved with changes in policy such as those instituted in the United States for drug trials. Ideally, if the new movement toward registration of all clinical trials prior to publication becomes universal, as has been advocated by the editors of many GMJs [24], it will serve as a vital tool for assessing

6 E. Cohen et al. / Journal of Clinical Epidemiology 60 (2007) 118e systematic barriers to conducting clinical research in children and in other vulnerable populations. Acknowledgments We thank Derek Stephens and Eshetu Atenafu of the Clinical Research Support Unit at the Research Institute in the Hospital for Sick Children for their support in the statistical analyses of this study. We would also like to thank Dr. Catherine Birken from the Division of Paediatric Medicine at the Hospital for Sick Children for reviewing an earlier draft of this manuscript. References [1] Chessells JM. Treatment of childhood acute lymphoblastic leukemia: present issues and future prospects. Blood Rev 1992;6:193e203. [2] Soll RF, Andruscavage L. The principles and practice of evidencebased neonatology. Pediatrics 1999;103:215e24. [3] Caldwell PH, Murphy SB, Butow PN, Craig JC. Clinical trials in children. Lancet 2004;364:803e11. [4] Curry JI, Reeves B, Stringer MD. Randomized controlled trials in pediatric surgery: could we do better? J Pediatr Surg 2003;38:556e9. [5] Birken CS, Parkin PC. In which journals will pediatricians find the best evidence for clinical practice? Pediatrics 1999;103:941e7. [6] Obuaya CC. Reporting of research and health issues relevant to resource-poor countries in high-impact medical journals. Euro Science Edit 2002;28:72e7. [7] Horton R. Medical journals: evidence of bias against diseases of poverty. Lancet 2003;361:712e3. [8] Rochon P, Mashari A, Cohen A, Misra A, Laxer D, Streiner DL, et al. Relation between randomized controlled trials published in leading general medical journals and the global burden of disease. CMAJ 2004;170:1673e7. [9] Garfield E. Citation analysis as a tool in journal evaluation. Science 1972;178:471e9. [10] Garfield E. Journal impact factor: a brief review. CMAJ 1999;161: 979e80. [11] Robinson KA, Dickersin K. Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. Int J Epidemiol 2002;31:150e3. [12] Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17: 1e12. [13] Brouwers MC, Johnston ME, Charette ML, Hanna SE, Jadad AR, Bowman GP. Evaluating the role of quality assessment of primary studies in systematic reviews of cancer practice guidelines. BMC Med Res Methodol 2005;5:8. [14] Huwiler-Muntener K, Juni P, Junker C, Egger M. Quality reporting of randomized trials as a measure of methodologic quality. JAMA 2002;287:2801e4. [15] Budetti PP. Ensuring safe and effective medications for children. JAMA 2003;290:950e1. [16] McCabe LL. National Institutes of Health support for research and training: future of pediatrician scientists. Arch Pediatr Adolesc Med 1998;152:839e42. [17] Roberts R, Rodriguez W, Murphy D, Crescenzki T. Pediatric drug labelling: improving the safety and efficacy of pediatric therapies. JAMA 2003;290:905e11. [18] Richmond TK, Rosen DS. The treatment of adolescent depression in the era of the black box warning. Curr Opin Pediatr 2005;17:466e72. [19] Banner W. Off label prescribing in children. BMJ 2002;324:1290e1. [20] US Food and Drug Administration. FDA joins children s health groups to mark historic milestone for pediatric drugse100 drug labels have new pediatric information since Accessed December 28, Available at NEW01280.html. [21] Forrest CB, Shipman SA, Dougherty D, Miller MR. Outcomes research in pediatric settings: recent trends and future directions. Pediatrics 2003;111:171e8. [22] Cramer K, Wiebe N, Moyer V, Hartling L, Williams K, Swingler G, et al. Children in reviews: methodological issues in child-relevant evidence syntheses. BMC Pediatr 2005;21:38. [23] Campbell H, Surry SA, Royle EM. A review of randomised controlled trials published in Archives of Disease in Childhood from 1982e1996. Arch Dis Child 1998;79:192e7. [24] De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Is this clinical trial fully registered? A statement from the International Committee of Medical Journal Editors. CMAJ 2005;172: 1700e2.