Citation accuracy is a fundamental aspect of scientific

Transcription

1 Original Research General Otolaryngology Reference Errors in Otolaryngology Head and Neck Surgery Literature Otolaryngology Head and Neck Surgery 2018, Vol. 159(2) Ó American Academy of Otolaryngology Head and Neck Surgery Foundation 2018 Reprints and permission: sagepub.com/journalspermissions.nav DOI: / Michael F. Armstrong 1 *, Joseph H. Conduff III 1 *, John E. Fenton, FRCSI (ORL-HNS) 2, and Daniel H. Coelho, MD 1 No sponsorships or competing interests have been disclosed for this article. Abstract Objective. Proper use of citation and quotation is crucial to the integrity of the medical literature. The purpose of this study was to determine the prevalence of quotation and citation errors in otolaryngology head and neck surgery (OHNS) journals and how they have changed over time. Study Design. Literature review. Methods. Fifty references were randomly selected from the first published issue of 2017 for 8 leading OHNS journals. These were analyzed for errors in citation (data elements by which the article is referenced) and quotation (factual inaccuracies of the reference). Citation errors were categorized as major, intermediate, or minor. Quotation errors were categorized as major or minor. Results were compared with data from 1997 articles. Results. Citation errors occurred in 17% of all references studied, with 34% classified as major. Quotation errors occurred in 9%, with 69% classified as major. There was no association between journal impact factor and total number of errors (r = 20.33, P =.42). This compares with a 37% citation error rate (32% major) and 17% quotation error rate (65% major) from Conclusion. Citation and quotation errors are still prevalent in the OHNS literature albeit decreased from previously reported data. Improvement in citation errors may be due to technological improvements in reference management. However, it is the continued responsibility of the authors, reviewers, and editors to further reduce error rates to maintain the integrity of our publications. Keywords citation, otolaryngology, publishing, impact factor, error, referencing Received December 22, 2017; revised February 27, 2018; accepted April 3, Citation accuracy is a fundamental aspect of scientific literature. New research develops in the context of the conclusions of prior studies. This is arguably of greater importance in medical literature, where the treatment of patients depends directly on clinical trial results, often referred to as evidence-based medicine. Correct and accurate citation use is generally taken for granted. With the rapid dissemination of knowledge through the internet, researchers have unprecedented access to information. While the benefits are apparent, ease of access may lead to carelessness in citations and even false conclusions. Reference errors generally take 2 forms: errors of citation and errors of quotation. Citation errors are defined as those involving the data elements by which the article is referenced. 1 Quotation errors are considered more serious and are defined as a citation in which the referenced article contradicts, fails to substantiate, or is unrelated to the authors statements. 2 Prior work by Fenton et al demonstrated the prevalence of citation and quotation errors in major otolaryngology head and neck surgery (OHNS) journals. 1 Much has changed since their findings were published.20 years ago. In addition to the recognition of the problem itself, internetand computer-based reference management programs may have, at least theoretically, resulted in improved accuracy. The objective of this study was to determine the current prevalence of reference errors in OHNS journals and, if changed, suggest the rationale behind that transformation. 1 Department of Otolaryngology Head and Neck Surgery, School of Medicine, Virginia Commonwealth University, Richmond, Virginia, USA 2 Department of Otorhinolaryngology Head and Neck Surgery, Graduate Entry Medical School, University of Limerick, Limerick, Ireland * These authors contributed equally to this article. Corresponding Author: Daniel H. Coelho, MD, Department of Otolaryngology Head and Neck Surgery, School of Medicine, Virginia Commonwealth University, PO Box , Richmond, VA , USA. daniel.coelho@vcuhealth.org

2 250 Otolaryngology Head and Neck Surgery 159(2) Methods The methodology is similar to that described by Fenton et al. 1 In addition to the 4 OHNS journals that they analyzed (Laryngoscope; Annals of Otology, Rhinology, and Laryngology; Clinical Otolaryngology; and Journal of Laryngology and Otology), 4 other OHNS journals (Head and Neck Journal for the Sciences and Specialties of the Head and Neck, JAMA Otolaryngology, Rhinology, and Otolaryngology Head and Neck Surgery) were selected as the clinical journals with the highest impact factors published at least 10 times per year. The first issue of each journal published in 2017 was evaluated. Articles excluded from evaluation included e-only articles and editorials. The references from each article were numbered sequentially, and 50 random numbers were computer generated for each of the 8 journals to determine which references to include in the study (ie, n = 50 references per journal). 3 Duplicate numbers and references that were inaccessible were rerandomized. Only referenced articles written in English were included in this study. Major citation errors were defined as those involving the data elements by which the article is referenced 1 :thesurname and initials of the first author, publication year, journal title, journal volume number, significant errors in the article title, supplement designation, and initial page number. Intermediate citation errors were defined as errors in the article title (incorrect spelling or omitted words) or incorrect page numbers. Minor citation errors included other features, such as punctuation errors, coauthor errors, or minor spelling errors, such as missing accents from first authors names. A major quotation error was defined as one in which the referenced article contradicted, failed to substantiate, or was unrelated to the authors statements. 2 Minor quotation errors included discrepancies in percentages or numbers, which had less of an impact on the overall conclusions of the referenced article. When.1 type of citation or quotation error was found, the error of greater magnitude was considered defining. Original articles were obtained through the Web of Science online database and VCU Libraries. Each original referenced article was read thoroughly and compared with the citation and content of the referring article. Two reviewers of equal medical experience classified the errors and discussed for agreement. When the authors could not agree over the type of error, a senior author (D.H.C.) broke the tie. Results were compared with Fenton et al data from 1997 (article published in 2000). 1 A chi-square test was used to compare the distribution of citation and quotation errors with the original 1997 data, and a Pearson correlation coefficient was calculated to assess the relation between journal impact factor and total number of errors. A P value \.05 was considered significant. Journal impact factor was taken from 2016 data from InCites Journal Citation Reports. 4 Results Of the 8 issues studied, 4658 total references met inclusion criteria (Table 1). There were no duplicates of the 400 references in the study sample (50 from each journal). Table 1. Total Number of References and Articles from 8 Otolaryngology Journal References Articles Table 2. Citation Errors Detected from 8 Otolaryngology Head and Neck Surgery Journals. No. of Citation Errors Journal (N = 400) a Major Intermediate Minor Total Head Neck J Sci Spec JAMA Otolaryngol Clin Otolaryngol Laryngoscope Rhinology Otolaryngol Head Neck Surg Ann Otol Rhinol Laryngol J Laryngol Otol Total a Per journal, n = 50. References per Article Head Neck J Sci Spec JAMA Otolaryngol Clin Otolaryngol Laryngoscope Rhinology Otolaryngol Head Neck Surg Ann Otol Rhinol Laryngol J Laryngol Otol Sixty-eight citations (17%) contained errors, of which 23 (34%) were major, 12 (18%) were intermediate, and 33 (49%) were minor (Table 2). For type of major citation errors, see Table 3. Additionally, 12 intermediate citation errors were detected: 10 included misspelled words or omitted insignificant words in the article title (eg, a, the ), and 2 had mistakes in the page numbers. There were 33 minor citation errors detected: 7 punctuation errors, 6 missing coauthor initials, 4 misspellings of coauthor surnames, 4 omissions of coauthors, 1 capitalization error, 1 typo, and 10 omitted diacritics for foreign author names. Thirty-six quotations (9%) contained errors, of which 25 (69%) were major and 11 (31%) were minor (Table 4). For type of major quotation errors, see Table 5. Eleven minor errors were detected. These included 9 percentage errors and 2 errors in numbers. The citation error rate of 17% (34% major) compares with a 37.5% citation error rate (32% major) from Likewise, a 9% quotation error rate (69% major) compares with a 17% quotation error rate (65% major) from There was no significant difference in the distribution of citation errors (P =.48) and quotation errors (P =.74) when compared with the 1997 data 1 (Tables 6 and 7).

3 Armstrong et al 251 Table 3. Types of Major Citation Errors in 8 Otolaryngology Type of Error Incorrect elements of article title 7 Incorrect author surname 4 Incorrect year 4 Incorrect journal 3 Incorrect author initials 2 Incorrect volume 1 Incorrect initial page number 1 Supplement omitted 1 Total 23 Table 4. Quotation Errors Detected from 8 Otolaryngology No. of Quotation Errors Journal (N = 400) a Major Minor Total Head Neck J Sci Spec JAMA Otolaryngol Clin Otolaryngol Laryngoscope Rhinology Otolaryngol Head Neck Surg Ann Otol Rhinol Laryngol J Laryngol Otol Total a Per journal, n = 50. Table 5. Types of Major Quotation Errors in 8 Otolaryngology Type of Major Quotation Error Unrelated 16 Failure to substantiate 7 Contradictory 2 Total 25 Table 6. Distribution of Citation Errors in Otolaryngology Head and Neck Surgery Journals: 2017 vs ,a Errors, n (%) Total Major 23 (33.8) 20 (31.7) 43 Intermediate 12 (17.6) 7 (11.1) 19 Minor 33 (48.5) 36 (57.1) 69 Total a There was no significant difference in the distribution of citation errors (P =.48). n n Table 7. Distribution of Quotation Errors in Otolaryngology Head and Neck Surgery: 2017 vs ,a Discussion Errors, n (%) Total Major 25 (69.4) 17 (65.4) 42 Minor 11 (30.6) 9 (34.6) 20 Total a There was no significant difference in the distribution of quotation errors (P =.74). This review of the contemporary otolaryngology literature reports citation and quotation error rates of 17% and 9%, respectively. The citation error rate demonstrated by this study is consistent with those of other medical specialties: 13.3% in burn surgery, 5 26% in orthopaedics, 6 and 11.1% in general surgery. 7 Conversely, the quotation error rate in the present study is lower than reports from orthopedics (20%- 38%) 6,8 but comparable to those from burn surgery (13.7%) 5 and general surgery (7.8%). 7 Recent meta-analyses reported quotation error rates across all medical literature between 14.5% 9 and 25.4%. 10 In 1997, Fenton et al reported overall citation and quotation error rates of 37% and 17%, respectively, in otolaryngology, which were consistent with the rates of other medical specialties at the time. 1 Results from the current study illustrate a decrease in overall citation and quotation error rates in the otolaryngology literature over 2 decades. Although the current study reports decreases in overall citation and quotation error rates, the prevalence of error severity appears to have remained consistent in the otolaryngology literature. This study yielded major citation and quotation error rates of 34% and 69%, respectively (Tables 2 and 4). These data are consistent with those of Fenton et al, who reported major error rates of 32% (citation) and 65% (quotation) in Despite overall error rates being consistent with those reported in other medical specialties, prevalence of error type is subject to much greater variability. Major citation and quotation error rates are, respectively, 2.3% and 80% for general surgery, 7 15% and 89% for orthopaedics, 8 and 3.3% and 50% for burn surgery. 5 The 69% major quotation error rate (Table 4) reported in this study is consistent with the recent meta-analysis by Mogull (64.8%). 9 Conversely, Jergas and Baethge reported a major quotation error rate of 11.9%. 10 Interestingly, in the only other published report of errors in otolaryngology, Adhikari found a comparable citation error rate of 30.1%, although this author studied only 63 articles in 2 volumes of 1 journal. 11 Variability in prevalence of error severity may lead to discrepancy in comparing reference errors among specialties. 9 This may be explained by differences in the methodology of error classification and variation in calculation of overall quotation error rates. 9 Additionally, a considerable

4 252 Otolaryngology Head and Neck Surgery 159(2) Table 8. Total Number of Citation and Quotation Errors in 8 Otolaryngology Head and Neck Surgery Journals Related to Impact Factor. a Journal (N = 400) b Total Errors Impact Factor Head Neck J Sci Spec JAMA Otolaryngol Clin Otolaryngol Laryngoscope Rhinology Otolaryngol Head Neck Surg Ann Otol Rhinol Laryngol J Laryngol Otol a Based on 2016 data from InCites Journal Citation Reports. 4 There was no relationship between journal impact factor and total number of errors (Pearson correlation coefficient = 20.33, P =.42). 15 b Per journal, n = 50. degree of human error exists in classifying a reference error as major, intermediate, or minor. 8,9 The variability in study design introduces some bias in the interpretation of metaanalyses published on the topic of reference error. Although meta-analyses possess the power of large sample sizes, multiple authors 9,10,12 commented on the heterogeneity among reference error studies as being a limitation to their extrapolation. Improper use of quotation in the medical literature distorts the ability to draw conclusions from research, fragments the image of respected journals, and can allow for propagation of false information. 1,9 Alternatively, citation errors create difficulty in locating references and can hinder authors credibility. 1 Unlike quotation errors, citation errors in this study were more likely to be minor errors, including punctuation, coauthor misspellings, and typos. However, a large proportion of citation errors were still classified as major (34%). As seen in Table 3, the most common major citation errors involved incorrect elements of article title, publication year, and author surnames. These types of major citation error were consistent with reports from other specialties. 5,7,8 Journal impact factor is an academic quality measure, which depends on the average number of citations to articles published in a journal. A citation error involving data elements by which the article is referenced may detract from the total citation count of the article and thus lower a journal s overall impact factor. Prior work comparing citation counts among databases commonly used to determine journal impact factor showed significant differences in the number of citations per article. 13 In 1997, Fenton et al demonstrated a correlation between journal impact factor and rates of reference error (Pearson correlation coefficient, 0.90; P \.05). 1 Higher impact factor journals had fewer rates of reference error than lower impact factor journals, and further studies supported this relationship. 1,12,14 However, the present study found no correlation between impact factor and reference error rates (Pearson correlation coefficient, 0.33; P =.42; Table 8). 15 Recognition of the error rate and education on the importance of accurate referencing by editors, reviewers, and authors have probably resulted in a substantial reduction in citation and quotation accuracy during the past decade in our specialty alone. Numerous authors have commented on factors that influence reference error rates in medical literature. 5,7,8,12 Technical editing by journals, reference management software, and improved author instructions have been discussed as contributing to improved reference error rates. However, there is sparse supporting evidence for any of these tactics. 5,12 The implementation of reference management software such as EndNote (ThomasReuters, Philadelphia, Pennsylvania) and RefWorks (ProQuest, Ann Arbor, Michigan) as well as the increasing reliance on electronic literature may have contributed to the decrease in overall reference error rates reported in this study. Although these data and comparable findings from other specialties would lend support to the important role of reference management systems, to date we could find no evidence to substantiate this theory. Despite improvement, the prevalence of error severity remains consistent. It is generally accepted that reference citation is the responsibility of the author. 1,8,12 However, continued improvement in technical editing by well-respected medical journals can also aid in the reduction of reference errors. Attempts to reduce the transgression rate further may need to include the appointment of a specific specialty-qualified journal editorial assistant to manually assess for reference accuracy in accepted articles. Further research is needed to determine which factors contribute to reference error and if comparing error rates among specialties is appropriate given the heterogeneous nature of the literature. It is promising to see improved overall rates of reference error in the present study, and it is the responsibility of the medical community to continue to strive for improvement in this crucial aspect of academic literature. Conclusion Citation and quotation errors are still prevalent in the OHNS literature albeit decreased from previously reported data. This study reports citation and quotation error rates of 17% and 9%, respectively. The prevalence of error severity has remained consistent in the otolaryngology literature. Improvement in citation errors may be due to technological improvements in reference management. However, it is the continued responsibility of the authors, reviewers, and editors to further reduce error rates to maintain the integrity of our publications. Author Contributions Michael F. Armstrong, data acquisition, draft of work, final approval, accountable for accuracy and integrity of the work; Joseph H. Conduff III, data acquisition, draft of work, final approval, accountable for accuracy and integrity of the work; John E. Fenton, conception, Critical revision, final approval, accountable for accuracy and integrity of the work; Daniel H. Coelho, conception, design, interpretation of data, critical revision, final approval, accountable for accuracy and integrity of the work.

5 Armstrong et al 253 Disclosures Competing interests: None. Sponsorships: None. Funding source: None. References 1. Fenton JE, Brazier H, De Souza A, Hughes JP, McShane DP. The accuracy of citation and quotation in otolaryngology/head and neck surgery journals. Clin Otolaryngol Allied Sci. 2000; 25: Evans JT, Nadjari HI, Burchell SA. Quotational and reference accuracy in surgical journals: a continuing peer review problem. JAMA. 1990;263: Random.org. True random number generator. Accessed September 14, Clarivate Analytics journal citation reports. Published Accessed September 12, Al-Benna S, Rajgarhia P, Ahmed S, Sheikh Z. Accuracy of references in burns journals. Burns. 2009;35: Davids JR, Weigl DM, Edmonds JP, Blackhurst DW. Reference accuracy in peer-reviewed pediatric orthopaedic literature. J Bone Joint Surg Am. 2010;92: Reddy MS, Srinivas S, Sabanayagam N, Balasubramanian SP. Accuracy of references in general surgical journals an old problem revisited. Surgeon. 2008;6: Luo M, Li CC, Molina DT, Andersen CR, Panchbhavi VK. Accuracy of citation and quotation in foot and ankle surgery journals. Foot Ankle Int. 2013;34: Mogull SA. Accuracy of cited facts in medical research articles: a review of study methodology and recalculation of quotation error rate. PLoS One. 2017;12:E Jergas H, Baethge C. Quotation accuracy in medical journal articles a systematic review and meta-analysis. PeerJ. 2015; 3:e Adhikari P. Accuracy of references in Indian Journal of Otolaryngology and Head & Neck Surgery. Indian J Otolaryngol Head Neck Surg. 2010;62: Wager E, Middleton P. Technical editing of research reports in biomedical journals. Cochrane Database Syst Rev. 2008;(4): MR Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA. 2009; 302: Lok CK, Chan MT, Martinson IM. Risk factors for citation errors in peer-reviewed nursing journals. J Adv Nurs. 2001;34: Brothersoft. How to calculate the P-value & its correlation in Excel Accessed October 3, 2017.