THE ESSENCE OF evidence-based medicine is the integration. Level of Evidence in Four Selected Rehabilitation Journals ORIGINAL ARTICLE

Transcription

1 299 ORIGINAL ARTICLE Level of Evidence in Four Selected Rehabilitation Journals Fatma U. Kocak, PhD, PT, Bayram Unver, PhD, PT, Vasfi Karatosun, MD ABSTRA. Kocak FU, Unver B, Karatosun V. Level of evidence in four selected rehabilitation journals. Arch Phys Med Rehabil 2011;92: Objective: To investigate the methodologic quality and level of evidence of publications in major peer-reviewed general rehabilitation journals (Archives of Physical Medicine and Rehabilitation [APMR], American Journal of Physical Medicine and Rehabilitation [AJPMR], Clinical Rehabilitation [CR], and Physical Therapy [PT]). Design: Descriptive, comparative. Main Outcome Measures: All the articles published in AJPMR, APMR, CR, and PT between January 2005 and December 2009 were investigated. Type of study and level of evidence were recorded for all articles. Selection and assessment of articles were based on the title and abstract by 2 independent raters. Results: The most frequently published reports were randomized controlled trials (12.7%), followed by cross-sectional studies (12.1%), case reports/case series (10.3%), validation studies (9.3%), cohort studies (8.9%), clinical trials (7.5%), case control studies (6.8%), and other study types (32.4%). When the articles were classified according to their level of evidence, level I studies most frequently appeared in CR (29.1%), followed by PT (11.0%), APMR (10.5%), and AJPMR (7.1%). Most of the meta-analyses (10) were in APMR, and there were none in AJPMR. Conclusions: Randomized controlled trials and meta-analyses form only a small proportion of articles published in the current rehabilitation literature. The numbers of randomized controlled trials and meta-analysis are comparable with those in other fields. Key Words: Meta-analyses; Randomized controlled trial; Journals; Evidence-based medicine by the American Congress of Rehabilitation Medicine THE ESSENCE OF evidence-based medicine is the integration of individual clinical expertise with the best available external clinical evidence from systematic research. 1 Advocates of EBM classify studies according to grades of evidence on the basis of the research design using internal validity (ie, the correctness of the results) as the criterion for hierarchic rankings. 2(p 1887) In the hierarchy of research study designs, prospective Rs and meta-analyses of several Rs are considered to provide the highest quality of evidence. 2-4 As professional scientists and physicians, our success is directly related to our ability to stay apprised of leading information and research in our field, and the most important source of this information is the scientific journals. 3 The data necessary to influence evidence-based practice should be presented in these journals. 3 Despite a number of studies assessing the quality of articles in various fields of medicine, 3-15 only a few investigations have analyzed the literature provided by physical therapy journals. 16,17 Milleretal 16 reviewed the contents of 4 physiotherapy professional journals (6 consecutive journal issues) published within a period of 18 months. However, a small number of issues and journals were reviewed. Paci et al 17 investigated 9 physical therapy journals during a 5-year period. However, not all of these were the journals that published general physical therapy studies; some of them were related to specific fields (ie, Journal of Neurologic Physical Therapy). A number of these journals did not appear in MED- LINE (ie, Physiotherapy), they were not available for every reader, and annual volume numbers of the journals were different (3, 4, 6, or 12 issues/y). Also, while some of these journals were included in the SCI journal list (ie, Journal of Orthopaedic and Sports Physical Therapy), others were included in the SCI expanded journal list (ie, Physiotherapy Research International), and some of them were excepted from the 2 (ie, Physiotherapy Theory and Practice). Paci 17 investigated only physical therapy journals. The aim of this study was to provide a benchmark for determining the current state of the science of the rehabilitation field, which would allow us to determine whether advances have been made over the decades ahead, compare the quality of our research to that of other fields (are we behind or ahead of comparable fields?), and examine the evidence base of rehabilitation research in 4 selected rehabilitation journals with international circulations over a period of 5 years. METHODS Journals were selected from the list of periodicals indexed by the Institute for Scientific Information Web of Science (these journals are the major peer-reviewed general rehabilitation journals listed in the rehabilitation category in SCI journal lists), 18,19 and according to the 2008 Journal Citation Report, they displayed the highest impact factor of all rehabilitation journals (APMR, PT, CR, and AJPMR had impact factors of 2.190, 2.159, 1.840, and 1.695, respectively). 19 Although this index lists 8 journals dedicated to rehabilitation AJPMR, From the School of Sports Sciences and Technology, University of Pamukkale, Denizli (Kocak); School of Physiotherapy (Unver), Department of Orthopedics, School of Medicine (Karatosun), University of Dokuz Eylül, Izmir, Turkey. No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit on the authors or on any organization with which the authors are associated. Correspondence to Fatma Unver Kocak, PhD, PT, Assistant Professor, School of Sports Sciences and Technology, Pamukkale University, 20020, Kinikli, Denizli, Turkey, fatmakunver@hotmail.com /11/ $36.00/0 doi: /j.apmr AJPMR APMR CR EBM PT R SCI List of Abbreviations American Journal of Physical Medicine and Rehabilitation Archives of Physical Medicine and Rehabilitation Clinical Rehabilitation evidence-based medicine Physical Therapy randomized controlled trial Science Citation Index

2 300 EVIDENCE IN REHABILITATION RESEARCH, Kocak Table 1: Grades of Evidence for the Purported Quality of Study Design Level I R without placebo R with placebo-controlled trial Level II Cohort study Cross-sectional study Controlled clinical trials without randomization Case control study Observational/descriptive study Survey Validation study Level III Clinical trials ( 10 subjects) Case report/case series ( 10 subjects) Reports of expert committees Editorial Letter to editor Commentary Perspective/expert opinion Review Systematic review: Nonsystematic review: Meta-analysis Systematic review Narrative review APMR, CR, PT, Journal of Electromyography and Kinesiology, Journal of Head Trauma Rehabilitation, Journal of Orthopaedic and Sports Physical Therapy, and Supportive Care in Cancer the last 4 were excluded because these journals publish only special topic articles and thus are not appropriate for comparison with the other 4. Because they are widely circulated major peer-reviewed general rehabilitation journals, AJPMR, APMR, CR, and PT were selected for investigation. All the articles published in AJPMR, APMR, CR, and PT between January 2005 and December 2009 were reviewed by 2 independent raters. All research articles, case series/reports, reviews, editorials, letters to the editor, commentaries, reports of expert committees, and perspectives were considered for the analysis. We excluded congress proceedings and evidence-in-practice series in PT. The articles were grouped and graded according to a modification of the U.S. Preventive Services Task Force system to grade the hierarchy of evidence (table 1). 20 They were differentiated into 3 categories of decreasing levels of evidence: meta-analysis, systematic review, and narrative review (see table 1). 17 Type of study and level of evidence were recorded for all articles. Selection and assessment of articles were based on the title and abstract. If the type of the article could not be determined through the title and the abstract, the article was evaluated by 2 independent raters reading the material and methods section. Interrater agreement was assessed using kappa values. RESULTS The distribution of study designs and subjects of articles by year in the 4 selected rehabilitation journals can be seen in table 2. The most frequently published reports were Rs (12.7%), followed by cross-sectional studies (12.1%), case reports/case series (noncontrolled before-after studies with 10 subjects were classified as case reports/case series) (10.3%), validation studies (9.3%), cohort studies (8.9%), clinical trials (noncontrolled before-after studies with 10 subjects were classified as clinical trials) (7.5%), and case control studies (6.8%). Other study types made up the remainder (32.4%) (table 3). When the articles were classified according to their level of evidence, level I studies most frequently appeared in CR (29.1%), followed by PT (11.0%), APMR (10.5%), and AJPMR (7.1%) (fig 1). Most of the meta-analyses (10) were in APMR. There were none in AJPMR (see table 2). The mean kappa value for interrater agreement was.82. DISCUSSION Recently there has been a progressive effort by researchers and journal editors to assess and improve the quality of published studies. 2-17,21 EBM continues to guide clinical decision-making based on the best available evidence in the literature. 13 We show that the numbers of Rs and metaanalyses are higher than in some disciplines 3,5,6,9,11,12,14,15 and lower than in others 7,8,10,13 and they are published with different frequencies among journals. The proportion of level I studies identified in rehabilitation in which quality methods were used compares well with other medical specialties, in particular surgery, family medicine, and community health, but there were more high-quality studies in anesthesia and general medicine journals. The reason for this may be that many issues in surgical areas do not lend themselves easily to this study design (ie, surgical interventions). Ethical concerns of Rs also play an important role. Another possible explanation is that the highest quality Rs in surgery or family medicine are submitted to general medical journals. The publication policies of individual journals might have affected the proportion of the Rs or meta-analyses. For example, CR gives high priority to Rs, but PT strongly encourages submission of validation studies and case reports. Rs and meta-analysis form only a small proportion of articles published in the current rehabilitation literature (see table 3). Thus, selected articles seem not to be sufficient for EBM. Of course, EBM is not restricted to Rs and metaanalyses, but the Rs and especially the systematic reviews of randomized trials are so much more likely to inform us and so much less likely to mislead us that they have become the criterion standard for judging whether a treatment does more good than harm. 3 However, some questions about therapy do not require Rs (successful interventions for otherwise fatal conditions) or cannot wait for trials to be conducted. If no R has been carried out for our patient s predicament, we must follow the trail to the next best external evidence and work from there. In fact, useful information for EBM may be provided by cohort or observational studies as well. 2 Rs were introduced into clinical medicine when streptomycin was evaluated in the treatment of tuberculosis, and they have become the criterion standard for assessing the effectiveness of therapeutic agents. 2 In the hierarchy of study designs, an R is thought to be the best way to control known and unknown confounding variables and to reduce bias. 4 The highest grade is reserved for research 4(p 1887) involving at least one properly conducted R, and the lowest grade is applied to expert opinion and case reports/case series. While useful to describe rare diseases and unusual observations, case reports/case series provide the weakest evidence (if any) for testing the benefit of one treatment over another. Observational studies, cross-sectional studies, cohort studies, and case control studies suffer from the lack of randomization, blindness, and the inclusion of control groups. Thus, these study designs are more prone to confounding

3 EVIDENCE IN REHABILITATION RESEARCH, Kocak 301 Journal Year/ Total R Table 2: Distribution of Study Design of Articles by Year in 4 Selected Rehabilitation Journals C C C-S C-C O/D S V AJPMR Total APMR Total CR Total PT Total General Total Abbreviations: C, cohort study; C-C, case-control study; C, controlled clinical trial without randomization; Com, commentary; C-S, cross-sectional study; 10, clinical trial ( 10 subjects); 10, case report/case series ( 10 subjects); E, editorial; LtE, letter to editor; MA, meta-analysis; NR, narrative review; O/D, observational/descriptive study; Per, perspective; REC, report of expert committee; S, survey; SR, systematic review; V, validation study. E LtE Com Per REC MA SR NR Total influences and fall at intermediate levels. 2,4,20 This hierarchic approach to study design has been promoted widely in individual reports, meta-analyses, consensus statements, and educational materials for clinicians. 4 Thus, in any clinical specialty, research primarily based on clinical trials, notably Rs and meta-analyses, is likely to provide the best quality of evidence. 4 Rs show optimal properties to minimize bias in the estimation of the treatment effect, which can otherwise easily be underestimated or exaggerated. 2 Because Rs play a crucial role in the evaluation of the efficacy of health care interventions, they may directly affect patient care more than any other study design 5 and are now legally required by the U.S. Food and Drug Administration as the proof of efficacy of a new drug prior to licensing. 5 Ideally, all trials on therapy should be Rs, because the design is subject to the least amount of bias. 10 Nonetheless, quantitative studies such as case reports or small case series may be the only type of evidence available for uncommon conditions. Also, for etiologic studies, cohort and case control studies are usually the best study design, because it is usually unethical or inappropriate to perform Rs. 10 Overall, the proportion of Rs in our study seems low (12.7%), but we can compare the proportion of such studies published in rehabilitation with other specialties, albeit within slightly different periods. When R ratios were compared, rehabilitation journals had more studies that used high-quality methods than pediatric surgery journals (0.3%), 11 plastic surgery journals (1.83%), 5 orthopedics journals (1.85%), 3 family medicine journals (3.4%), 15 com- Journal R C C Table 3: Proportion of Study Design in 4 Selected Rehabilitation Journals C-S C-C O/D S V AJPMR APMR CR PT General Abbreviations: C, cohort study; C-C, case-control study; C, controlled clinical trial without randomization; Com, commentary; C-S, cross-sectional study; 10, clinical trial ( 10 subjects); 10, case report/case series ( 10 subjects); E, editorial; LtE, letter to editor; MA, meta-analysis; NR, narrative review; O/D, observational/descriptive study; Per, perspective; REC, report of expert committee; S, survey; SR, systematic review; V, validation study. E LtE Com Per REC MA SR NR

4 302 EVIDENCE IN REHABILITATION RESEARCH, Kocak LEVEL I LEVEL II LEVEL III Fig 1. Proportion of level of evidence in 4 selected rehabilitation journals. Legend: 1. AJPMR, 2. APMR, 3. CR, 4. PT, 5. General. munity health journals (4%), 12 primary care journals (6%), 14 general surgery journals (7%), 9 and sports medicine journals (9.5%), 6 but fewer studies than rheumatology journals (16%), 7 ophthalmology journals (18.1%), 10 anesthesia journals (20.4%), 8 and general medicine journals (37%). 13 Our results were similar to the results of Paci, et al 17 who investigated 9 physical therapy journals (12.6%). However, our results were different from those of Miller et al 16 (ie, approximately 1 article an issue). This difference might depend on the evaluation by Miller 16 of fewer issues in a shorter period. The differences between our study and those of Paci 17 and Miller 16 are that we investigated only the journals that published on general physical therapy and rehabilitation subjects, were included in the SCI journal list, had the same volume number (12 issues/y), and were available for all readers, and we studied them for an extended period (5y). Therefore, we think our results better reflect the physical therapy and rehabilitation literature. However, an improvement of the number of Rs over the years has been found in the 4 rehabilitation journals reviewed (see table 2), and we believe that this fact represents an important move in the right direction for evidence-based rehabilitation research. Our results show that narrative review articles (105 articles) and editorials (63 articles) fill more pages than systematic reviews (96 articles) and meta-analyses (15 articles). This is unfortunate, because narrative review articles and editorials often have a greater impact on the reader than other types of articles, 8 despite their poorer quality and validity. The problem with narrative review articles is that the references and sources are often chosen haphazardly, with no apparent consistent structure, resulting in a high risk of bias. 8 Meta-analysis is the statistical synthesis of data from similar studies, and this enables individually small studies to be pooled into a meaningful result. 3,4 The technique of meta-analysis works best for Rs because they are more uniform in design and have fewer biases than observational studies. 3 Systematic reviews are secondary research in which scientific strategies are used to reduce bias in the systematic gathering, critical evaluation, and synthesis of all relevant studies on a specific subject. 8 Likewise, metaanalyses and systematic reviews must now be critical and not merely a reflection of the author s opinion on the subject matter. 21 Therefore, we need more meta-analyses and systematic reviews for physical therapy and rehabilitation practice. In our opinion, editors of journals other than APMR should explicitly state that they will no longer accept narrative reviews. Consequently, the overall paucity of meta-analyses and Rs noted in this study is of concern. There are several possible explanations for this. Rs require greater planning, potentially difficult patient recruitment, prospective clinical follow-up, and painstaking collection and analysis of data. 3,4 Thus, Rs are more expensive and time-consuming and call for greater commitment and cooperation between researchers, frequently from multiple centers in the case of infrequent disease types. Difficulty securing funding is likely to be a further barrier. In these respects, most non-r clinical trials tend to be easier to undertake, and undoubtedly the retrospective reporting of case series or individual case reports is even less taxing, perhaps underlying their popularity. 4 Also, metaanalyses are highly time-consuming and require specialized knowledge; therefore, it is doubtful that their numbers will increase. 8 Study Limitations There are some certain limitations of our study. First, we reviewed an unequal number of articles from each journal because of the differences between the total numbers of articles published in each journal (ie, CR 563 vs APMR 1711). Second, we did not assess the accuracy of study design reporting. Third, the raters were not blind to the journal in which the article had appeared, and this is a potential source of detection bias. Fourth, the search was restricted to the journals covered by the core index of the Institute for Scientific Information. Other rehabilitation journals (ie, Journal of Rehabilitation Medicine) provide evidence for clinical practice by publishing level I studies. Furthermore, rehabilitation information is not restricted to rehabilitation journals; other prestigious journals dedicated to general medicine also publish rehabilitation-related articles. CONCLUSIONS In conclusion, Rs and meta-analyses form only a small proportion of articles published in current rehabilitation literature. However, an improvement in the number of Rs and metaanalysis over the years has been found in the 4 rehabilitation journals reviewed (see table 2), and we believe that this fact represents an important move in the right direction for evidencebased rehabilitation research. Rehabilitation journals appear to be comparable to journals in the other fields. The types and levels of the study in rehabilitation journals can be reliably classified, and editors should continue to select and disseminate the highest quality research providing the best evidence available. References 1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn t. BMJ 1996;312: Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000;342: Kiter E, Karatosun V, Gunal I. Do orthopaedic journals provide high-quality evidence for clinical practice? Arch Orthop Trauma Surg 2003;123: Gnanalingham MG, Robinson SG, Hawley DP, Gnanalingham KK. A 30 year perspective of the quality of evidence published in 25 clinical journals: signs of change? Postgrad Med J 2006;82: Momeni A, Becker A, Antes G, Diener MK, Blumle A, Stark BG. Evidence-based plastic surgery: controlled trials in three plastic surgical journals (1990 to 2005). Ann Plast Surg 2009;62: Bleakley C, MacAuley D. The quality of research in sports journals. Br J Sports Med 2002;36: Ruiz MT, Alvarez-Dardet C, Vela P, Pascual E. Study designs and statistical methods in rheumatological journals: an international comparison. Br J Rheumatol 1991;30:352-5.

5 EVIDENCE IN REHABILITATION RESEARCH, Kocak Lauritsen J, Moller AM. Publications in anesthesia journals: quality and clinical relevance. Anesth Analg 2004;99: Horton R. Surgical research or comic opera: questions, but few answers. Lancet 1996;347: Lai TY, Leung GM, Wong VW, Lam RF, Cheng AC, Lam DS. How evidence-based are publications in clinical ophthalmic journals? Invest Ophthalmol Vis Sci 2006;47: Thakur A, Wang EC, Chiu TT, et al. Methodology standards associated with quality reporting in clinical studies in pediatric surgery journals. J Pediatr Surg 2001;36: Smith PJ, Moffatt ME, Gelskey SC, Hudson S, Kaita K. Are community health interventions evaluated appropriately? a review of six journals. J Clin Epidemiol 1997;50: Kuroki LM, Allsworth JE, Peipert JF. Methodology and analytic techniques used in clinical research: associations with journal impact factor. Obstet Gynecol 2009;114: Thomas T, Fahey T, Somerset M. The content and methodology of research papers published in three United Kingdom primary care journals. Br J Gen Pract 1998;48: Silagy CA, Jewell D, Mant D. An analysis of randomized controlled trials published in the US family medicine literature, J Fam Pract 1994;39: Miller PA, McKibbon KA, Haynes RB. A quantitative analysis of research publications in physical therapy journals. Phys Ther 2003;83: Paci M, Cigna C, Baccini M, Rinaldi LA. Types of article published in physiotherapy journals: a quantitative analysis. Physiother Res Int 2009;14: Unver B, Senduran M, Unver Kocak F, Gunal I, Karatosun V. Reference accuracy in four rehabilitation journals. Clin Rehabil 2009;23: Science - Thomson Reuters Preventive Services Task Force. Guide to clinical preventive services: report of the U. S. Preventive Task Force. Baltimore: Williams & Wilkins; Frontera WR, Grimby G, Basford J, Muller D, Ring H. Publishing in physical and rehabilitation medicine. Am J Phys Med Rehabil 2008;87: