Citation Characteristics of Research Published in Emergency Medicine Versus Other Scientific Journals

ORIGINAL CONTRIBUTION Citation Characteristics of Research Published in Emergency Medicine Versus Other Scientific From the Division of Emergency Medicine, University of California, San Francisco, CA * ; and the Department of Emergency Medicine, University of Florida at Jacksonville, Jacksonville, FL. Author contributions are provided at the end of this article. Received for publication June 23, 2000. Revision received January 8, 2001. Accepted for publication January 15, 2001. Reprints not available from the authors. Address for correspondence: Michael Callaham, MD, Box 0208, University of California, San Francisco, CA 94143-0208; 415-353- 1885, fax 925-253-9577; E-mail mlc@itsa.ucsf.edu. Copyright 2001 by the American College of Emergency Physicians. 0196-0644/2001/$35.00 + 0 47/1/114318 doi:10.1067/mem.2001.114318 Michael Callaham, MD * Ellen Weber, MD * Robert Wears, MD Study objective: We sought to examine how a cohort of published emergency medicine research is cited in scientific journals. Methods: Data were collected on all research submitted to the 1991 Society for Academic Emergency Medicine meeting and subsequently published. Outcome measures included all citations of these studies found in journals listed in the Science Citation Index, as well as the impact factors (citations per manuscript per year) of citing journals. Results: Two hundred four of the 493 submitted studies were published and met study entry criteria; the average article was cited 2.04 times a year during the study period. Twelve percent were never cited, and 39% were cited only once or twice. Thirty percent were published in non emergency medicine journals, and these were cited at least twice as often (and by almost 3 times as many journals) as apparently similar studies published in emergency medicine journals. The percentage of studies never cited by anyone was about threefold higher when published in emergency medicine journals. Forty-two percent of the citations of research published in emergency medicine journals came from within the specialty. Emergency medicine journals provided only 16% of the citations of emergency medicine research published in non emergency medicine journals because these studies were cited 3 times as often by authors in other disciplines. Rejection of research for presentation at the meeting did not predict the number or quality of citations or citing journals. Conclusion: Research submitted to the Society for Academic Emergency Medicine meeting and subsequently published is cited about as often as the average scientific journal article but receives more impact, is cited more widely, and is more likely to be cited by a broader range of authors when published by non emergency medicine journals. The ability of emergency medicine journals to compete with larger non emergency medicine journals for their larger audiences may help shape perceptions of the specialty. NOVEMBER 2001 38:5 ANNALS OF EMERGENCY MEDICINE 513

[Callaham M, Weber E, Wears R. Citation characteristics of research published in emergency medicine versus other scientific journals. Ann Emerg Med. November 2001;38:513-517.] INTRODUCTION An important part of the research process is publication in peer-reviewed journals. An equally important outcome, however, is the use and citation of these published articles by other researchers and authors. There have been no previously published reports examining the citation characteristics of individual research articles published in scientific journals. We conducted a study of all research submitted to the Society for Academic Emergency Medicine (SAEM) meeting in 1 year and subsequently published in peer-reviewed journals to determine how these studies were subsequently cited by other authors. METHODS The methods of the first phase of this study have been previously published. 1 In brief, all abstracts of scientific studies submitted to the SAEM in 1991 were examined. Independent of our study, each abstract was evaluated by 5 to 7 peer reviewers (members of the SAEM screening committee) and rated on a 5-point scale. Each submitted study was later randomly assigned in blinded fashion to 2 of the authors for evaluation and rated subjectively for scientific quality and newsworthiness, as well as categorized as to methodological design aspects, such as randomization, blinding, and control group. 1 A series of detailed, repeated searches of the Index Medicus (by means of MEDLINE), EMBASE, and Cochrane collaboration was conducted 4 years after the meeting to determine whether the study had been published in any journal listed in the National Library of Medicine. A broad search strategy was used with the last name only of the first, second, and third authors (one at a time sequentially) until a match was found. If there was any doubt on the basis of author names and publication title, the abstract or the full article was read to determine whether the correct study had been identified. If, after repeated searches with these various methods, no publication was found, the authors were contacted and asked to complete a questionnaire. 2 In March 2000, the same cohort of published articles was further examined to produce the new study reported here. The Science Citation Index (SCI) database was searched for citations by using the Web of Science (URL http://www.webofscience.com/) for every paper submitted to this meeting that had been published in a journal (on the basis of the results of the previously described search and questionnaire). For each published article in the cohort, a list was obtained of all citations of that article from publication to the time of the search. For the purposes of the current study, a standard time interval after publication was selected for evaluation. In this cohort (unlike basic science research), almost no articles were cited in less than 2 years after publication, and therefore, the evaluation period began 2 years after publication and extended for the next 3.5 years. This latter limit was chosen because most of the articles were published in the mid 1990s, and therefore, a longer time limit was not possible. By excluding the first 2 years, a citation rate that was comparable with the impact factor was obtained. Two outcome measures were determined. The total number of citations per year during the evaluation period was tallied for each article; this number is analogous to a journal s impact factor. 3 The impact factor equals the annual number of citations per published article in a journal, and this calculation was obtained for each citing journal from the SCI for the relevant year of publication. 3 The second outcome measure for each article was the average impact factor of the citing journal calculated from the individual impact factors of each journal in which a citation appeared. This measure provides an estimate of the importance, as well as the number of citations (eg, one citation in The New England Journal of Medicine [impact factor = 24.8] would count as more than 23 citations in Wilderness and Environmental Medicine [impact factor = 1.03]). For purposes of this study, journals with the primary focus of emergency medicine included Annals of Emergency Medicine, Academic Emergency Medicine, American Journal of Emergency Medicine, Prehospital and Disaster Medicine, Journal of Emergency Medicine, Pediatric Emergency Care, Accident and Emergency Medicine, Emergency Medicine Clinics of North America, and Archives of Emergency Medicine. Resuscitation was not considered an emergency medicine journal. Means, medians, and 95% confidence intervals (CIs) were calculated by using JMP software (version 3.16) for Macintosh (SAS Institute, Cary, NC). For clarifying the role of publication in emergency medicine versus non emergency medicine journals, citations outcomes were compared by using both multiple analysis of variance and recursive partitioning (nonparametric classification and regression trees [CARTs]) 4 by using CART 4.0 (Salford Systems, San Diego, CA). The study was approved by the Committee on Human Research of the University of California. 514 ANNALS OF EMERGENCY MEDICINE 38:5 NOVEMBER 2001

RESULTS Four hundred ninety-three studies were submitted to the 1991 SAEM meeting from a total of 144 institutions, 103 of which were formally affiliated with US medical schools. One hundred seventy-nine (36%) of the submitted abstracts were accepted for presentation at the meeting. As previously reported, 1 219 (44%) of the 493 studies submitted to this meeting were published in 44 peer-reviewed journals with CSI impact factors ranging from 0.23 to 24.5. One hundred forty-nine (68%) studies were published in 9 emergency medicine specialty journals. The remaining 70 studies were published in 35 other journals. Fifteen of the published articles appeared in journals that are not used as citation sources by SCI, are not reported in their database, and thus have no impact factor; these journals included Journal of Emergency Medicine (6 articles) and Prehospital and Disaster Medicine (6 articles). These 15 articles were therefore excluded from subsequent analysis because of absence of data, leaving 204 published manuscripts. Of the 204 publications eligible for study, 19 (12%) had no citations at all during the evaluation period; the remainder were cited a total of 1,446 times by 440 unique journals, all but 6 of which were from disciplines other than emergency medicine. Eighteen percent of citing Table 1. Citation characteristics of published articles. Published in Published in Emergency Non Emergency Medicine Medicine All Citation Characteristics Combined (n=143) (n=61) Average citations/y 2.04 (1.6 2.4) 1.5 (1.1 1.8) 3.4 (2.4 4.2) No. of citing journals 464 245 297 No. of citing journals per study 2.27 1.71 4.86 Average impact factor of citing 1.9 (1.7 2.1) 1.7 (1.4 1.9) 2.3 (2.0 2.6) journals Journal self-citations * (%) 14% (10 17) 16% (12 21) 10% (7 16) Citations/y by emergency 0.5 (0.4 0.6) 0.5 (0.4 0.6) 0.4 (0.3 0.6) medicine journals Citations/y by non emergency 1.5 (1.2 1.9) 0.9 (0.6 1.3) 2.9 (2.1 3.8) medicine journals Studies cited 1 or 2 times only 79 (39%) 60 (42%) 19 (31%) Studies never cited 24 (12%) 21 (15%) 3 (5%) Values are reported as mean (95% CI) where shown. Note one single extremely high value in emergency medicine journals (67 citations/y). * Refers to a study that is cited by another article in the same journal as that in which it was published. journals had an impact factor of greater than 3 compared with only 10% of all journals in the SCI. 5 The characteristics of these citations are detailed in Table 1. No differences in citations per year and average impact factor of the citing journal were present when journals were categorized as those with an impact factor of 1.5 or less (n=155) or those with an impact factor of greater than 1.5 (n=49), regardless of journal specialty. The top 10 citing journals for research studies published in emergency medicine versus non emergency medicine journals are summarized in Table 2. Twenty-one percent of citations of the American Journal of Emergency Medicine came from that same journal (journal self-citations); 32% of Academic Emergency Medicine citations were from the same journal, as were 22% of citations from Pediatric Emergency Care and 14% of citations from Annals of Emergency Medicine. Non emergency medicine journals self-cited only 11% of the time on average. The characteristics of studies published in emergency medicine versus non emergency medicine journals are reported in Table 3. Table 2. Top 10 citing journals by frequency of citation. No. Journal of Impact Citations Factor citing articles published in non emergency medicine journals Annals of Emergency Medicine 32 1.49 Academic Emergency Medicine 27 0.99 Journal of Trauma 21 1.62 Critical Care Medicine 19 4.3 Stroke 17 3.8 Chest 15 1.9 American Journal of Emergency Medicine 14 0.99 Resuscitation 10 1.8 Archives of Pediatric & Adolescent Medicine 8 1.4 The New England Journal of Medicine 8 24.8 Cumulative citations (% of total) 143 (48%) Mean=4.3 citing articles published in emergency medicine journals Academic Emergency Medicine 109 0.99 Annals of Emergency Medicine 89 1.49 American Journal of Emergency Medicine 31 0.99 Journal of Trauma 19 1.62 Resuscitation 15 1.8 Chest 15 1.9 Journal of Accident and Emergency Medicine 14 0.26 Critical Care Medicine 13 4.3 Pediatric Emergency Care 11 0.39 Archives of Pediatric and Adolescent Medicine 7 1.4 Cumulative citations (% of total) 323/770 (42%) Mean=1.5 NOVEMBER 2001 38:5 ANNALS OF EMERGENCY MEDICINE 515

Recursive partitioning was used to identify what role journal type (emergency medicine versus non emergency medicine) had in citation outcomes versus the impact factor of the publishing journal, which had a Spearman correlation of 0.54. Journal type was significant for both outcomes at the.001 level, and journal impact factor was significant at the less than.001 level, indicating that both contributed independently. There was no difference in the average citations per year, average impact factor of the citing journal, or number of citations by emergency medicine journals between articles rejected for presentation at the meeting and those accepted. DISCUSSION The publication of research in peer-reviewed journals is only an intermediate outcome, satisfying to authors but not necessarily useful to others. There is no way to measure how useful a published article is to clinicians, but we can measure some of its effect on other researchers and authors by examining how often and where they cite it in other articles. Such citations are a key measure of success for academic authors and have been shown to correlate with individual and institutional prestige as measured by other methods. 6 Twenty-five percent of Canadian medical school deans use them for evaluating faculty s publications for promotion (data are not available for US medical schools). 7 Table 3. Characteristics of studies published in the 2 types of journals. Emergency Non Emergency Medicine Medicine Study Characteristics (n=143) (n=61) SAEM committee selection scores, 2.02 (1.9 2.2) 1.98 (1.8 2.2) mean (95% CI) Author delphi panel scores: quality, 2.66 (2.5 2.8) 2.96 (2.7 3.2) mean (95% CI) Author delphi panel scores: news- 1.7 (1.6 1.8) 1.79 (1.7 1.9) worthiness, mean (95% CI) No. of subjects, mean (95% CI) 326 (212 440) 519 (271 768) Prospective observational design 34 26 (% of total in each category) Prospective interventional design (%) 26 41 Not randomized (specifically described, 64 44 appropriate to study type; %) Control group (%) 54 59 Positive results (%) 69 60 Author delphi panel refers to the ranking process described previously. 8 To our knowledge, an examination of how emergency medicine research is cited has never been previously performed. We used a study cohort of all the research presented at the 1991 SAEM meeting. Each of these studies was cited about twice a year (Table 1). Such a rate is roughly comparable with the citation rate of all material published in all journals in the SCI (Eugene Garfield, personal communication, May 2000). * Only 12% of these studies were never cited during the study period (even by their own authors) compared with 22% for all the international medical literature, 8 11% of AIDS articles, and 15% of biology articles. 9 This supports the view that emergency medicine research has achieved status comparable with that of other specialties, as does the fact that 30% of these studies met criteria for publication in journals outside of emergency medicine. However, the annual citation rate per study of 2 is below the average rate of 7 for AIDS articles published before 1993 (in an article addressing the issue of whether that AIDS research is of poor quality as demonstrated by citation rates 9 ). It is also lower than the average citation rate of 3.5 for articles on drugs and alcohol in 12 journals in the late 1980s. 10 Figures for all original medical research are not available, but the citation rate of 2 in this study is considerably below the comparable impact factor (citations per article per year) for most prestigious medical journals. The impact factors of emergency medicine journals are typically also well below this number (Table 2). (The New England Journal of Medicine has an impact factor of 24.8, Annals of Internal Medicine of 11.2, Circulation of 9.9, JAMA of 9.3, Critical Care Medicine of 4.3, Neurology of 4.6, and Pediatrics of 2.8.) Although all these studies were submitted to an emergency medicine meeting and concerned emergency medicine topics, 61 of them were published in journals outside of emergency medicine, presumably because of the increased prestige of these journals. The studies themselves did not seem to be larger, better designed, or otherwise different from those submitted to emergency medicine journals nor were they scored differently for scientific merit or newsworthiness (Table 3). The authors significantly increased their exposure to other researchers and authors by publishing in non emergency medicine journals. The average impact factor of the citing non emergency medicine journals was almost threefold that of emergency medicine journals. The SAEM studies published in non emergency medicine * However, the rate for all SCI journals includes all published material, including letters to the editor, meeting abstracts, editorials, and book reviews in the denominator. 8 If available, the rate for original research articles only would be expected to be significantly higher. 516 ANNALS OF EMERGENCY MEDICINE 38:5 NOVEMBER 2001

journals were cited at least twice as often (and by almost 3 times as many journals) as apparently similar studies published in emergency medicine journals (Table 2). The percentage of emergency medicine studies never cited by anyone was about threefold higher (15%) when published in emergency medicine journals than in non emergency medicine journals (Table 1), and all but 3 of these 24 never-cited studies were published in emergency medicine journals. By comparison, a study of the 33 most influential journals found a never-cited rate of only 2.2%. 6 One explanation for this difference might be the impact factor of the publishing journal and not its specialty topic. However, when we categorized the publishing journals as having an impact factor of above or below 1.5 (the maximum for emergency medicine journals), we found no difference in either of the citation outcome measures between the 2 groups of journals. Examining the issue further by means of recursive partitioning, both journal specialty type and journal impact factor contributed independently to citation outcomes. Thus, whether or not the publishing journal was an emergency medicine journal made a separate (and negative) effect on both number and quality of citations. Articles published outside of emergency medicine journals not only have more citations (in journals with higher impact factors) but are cited more broadly across specialties (Table 1). Articles in non emergency medicine journals are cited by others publishing in the same journal (self-citations) only 10% of the time, but in emergency medicine journals, that figure is 16%. Forty-two percent of the citations of research published in emergency medicine journals comes from other emergency medicine journals (ie, within the specialty). Emergency medicine research published in non emergency medicine journals is cited just as frequently by emergency medicine authors but also is cited 3 times as often by authors in other disciplines (Table 1). As reported previously, rejection for presentation at the SAEM meeting was not a predictor of publication or the impact factor of the publishing journal. 1 In the present study, we also found no difference in the average citations per year, average impact factor of the citing journal, or the number of citations by emergency medicine journals between papers rejected for presentation at the meeting and those accepted. Our study had a number of limitations. Citations of an article are only one limited measure of its effect and usefulness but also currently the only one that is quantifiable. To some degree, the higher citation rate in non emergency medicine journals might be due to a larger audience for a broader topic rather than the mere fact of where they were published; for example, an article on snake venom would be of interest to fewer researchers than one on shock and would be more likely to be published in an emergency medicine journal, regardless of its scientific rigor. These data represent only a single point on the maturation continuum of emergency medicine. Although emergency medicine research may hold its own with other clinical specialties in citations, emergency medicine journals face a difficult barrier in competing with larger non emergency medicine journals with their larger audiences. When emergency medicine researchers publish in non emergency medicine journals, their research is probably recognized as coming from our specialty only by those few who bother to check the authors academic department affiliation printed in small type. We believe that this is not comparable with the effect of reading an article in a journal with the words emergency medicine in large type on the masthead. Thus, emergency medicine may not get full credit as a specialty for major studies published in non emergency medicine journals, and its scientific reputation therefore largely depends on the scientific quality of emergency medicine journals. However, judging by citations, authors in other specialties are often unaware of emergency medicine journals. Thus emergency medicine researchers wishing to reach a broader audience will continue to be drawn to publish in journals outside the specialty. Author contributions: RW helped design the study, analyzed the data, and reviewed the manuscript. EW helped design the study, collected the data, and reviewed the manuscript. MC conceived and helped design the study, collected data, and wrote the manuscript. MC takes responsibility for the paper as a whole. REFERENCES 1. Callaham ML, Wears RL, Weber EJ, et al. Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting [published correction appears in JAMA. 1998;280:1232]. JAMA. 1998;280:254-257. 2. Weber EJ, Callaham ML, Wears RL, et al. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA. 1998;280:257-259. 3. Garfield E. SCI Journal Citation Reports: A Bibliometric Analysis of Science in the ISI Database. Philadelphia, PA: Institute for Science Information, Inc; 1996. 4. Breiman L, Friedman J, Olshen R, et al. Classification and Regression Trees. New York, NY: Chapman & Hall; 1984. 5. Garfield E. How can impact factors be improved? BMJ. 1996;313:411-413. 6. Garfield E. From citation indexes to informetrics: is the tail now wagging the dog? Libri. 1998;48:67-80. 7. Davies H, Langley J, Speert D. Rating authors contributions to collaborative research: the PICNIC study of university departments of pediatrics. Can Med Assoc J. 1996;155:877-882. 8. Schwartz C. The rise and fall of uncitedness. College and Research Libraries. 1997;48:19-29. 9. Brown P. Has the AIDS research epidemic spread too far? New Scientist. 1993;138:12-15. 10. Mo H, Da H. Citation analysis of 541 articles published in drug and alcohol journals 1984-1988. J Studies Alcohol. 1992;53:427-434. NOVEMBER 2001 38:5 ANNALS OF EMERGENCY MEDICINE 517