Evidential strength of medical specialties: a comparative analysis

Transcription

1 Page 1 Evidential strength of medical specialties: a comparative analysis Kiran Kalyan Turaga, MBBS Integrating experience for the Master of Public Health. Advisor on the project: Dr. Steve Goodman, MD, MHS, PhD Johns Hopkins Bloomberg School of Public Health, 615 N. Wolfe Street, Baltimore, MD Key words: evidential strength, evidence based medicine, quality, journals, specialty, comparison Word count: 4288 (including tables and figures)

2 Page 2 Abstract Introduction: We undertook a journal review to compare medical specialties with regards to the type and quality of evidence generating studies published in the year 2002 in top specialty journals. Methods: We selected the top five journals in each specialty on the basis of the impact factor, which crossed a minimum cut-off for the total citations. A random sample of 200 articles was drawn from each database, and the original articles published were further classified according to whether they generated inferences for therapeutic/diagnostic strategies. These articles were then classified according to the type of study design, the number of subjects, the method of blinding and the specialty affiliations. Heterogeneity between the different specialties (with the Pearsons chi square statistic) was looked for and an index of relative evidential strength was calculated. Results: We found considerable heterogeneity in the specialties we looked at, anesthesiology, cardiology, dermatology, gastroenterology, hematology, nephrology, neurology, respiratory medicine and rheumatology, in the proportions of evidence generating studies. Cardiology had the highest number of multicenter randomized clinical trials (RCT) and meta-analysis of randomized trials (16.9% of therapeutic studies). Hematology had the highest mean number of subjects in the pooled groups of metaanalysis and RCT s (1930 per study for 6 studies) while anesthesiology had the highest proportion of studies in these pooled groups (39.8% of therapeutic studies). The studies were significantly different for whether the abstract was adequate for the study design, with anesthesiology contributing to most of the heterogeneity (35.1% of abstracts were inadequate). We developed an indicator of relative evidential strength and cardiology was ranked as having the highest evidential strength on the basis of our analysis. Conclusion: We suggest that specialties re-assess their literature for the differing adoption of evidence-based medicine as reflected by our study. However, the findings of our study have to be interpreted with caution and more as guidelines and indicators rather than rigid markers.

3 Introduction Evidential strength of medical specialties Page 3 From its philosophical origins in 19 th century Paris, evidence based medicine (EBM) has slowly but surely changed clinical practice over the years. Evidence based medicine propagates the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients 1. Evidence based medicine forms an important component of clinical decision-making, as an adjunct to individual clinical expertise and judgments. While there is still debate about the actual quantification of evidence and the levels of evidence, the application of evidence generated from research studies has become more widespread. Research literature has often been evaluated to see the quality of published studies with particular reference to randomized clinical trials, which have had a rapid ascendancy as sources of strong evidence. The literature in specialty journals can be viewed as a reflection of the evidential strength of the different specialties, where evidential strength can be loosely understood as a composite measure of the quality of evidence generated. However, most of these evaluations have been done within specialties and there have been few studies which have actually compared specialty literature with general medical literature 2. Medical sub-specialties have differed in the rate of adoption of the methods of evidence based medicine and while some specialties seem to have embraced the methods very well and integrated them into clinical practice, there is a perception that not all specialties have been as successful in doing so. While the volume of literature would be influenced by the burden of disease, funding opportunities and number of researchers, the authors believe that the quality of published literature in contributing to the body of evidence

4 Page 4 should be similar across different specialties. This study investigates the evidential strength of different specialties and we attempt to compare them on the basis of their evidential strength. Methodology We identified common medical subspecialties using the ISI Journal Citation Report 2001 which classifies journals by specialty 3. We used the database to identify the impact factor and the total citations for different journals in the year 2001 (the most current). Selection of journals: We included in our selection process only predominantly clinical journals, which we defined as journals in which clinical research articles comprise at least 40% of all original research published, as determined by a review of the current issue of the journal available to the reviewer on the 01/27/2003. We also included in our definition, journals which had separate sections for clinical research, which did not necessarily meet the above criterion. The journals that met our inclusion criteria were sorted by impact factor and the total cites. The top five journals on the basis of the impact factor, which also had over 10,000 total cites, were chosen. In cases where there were fewer than five journals, which met this criteria, we chose the journal with the highest impact factor with total cites as close to 10,000 as possible. The specialties that we included in our review were anesthesiology, cardiac and cardiovascular systems, clinical neurology, dermatology & venereal diseases, gastroeneterology & hepatology, hematology, internal medicine, nephrology, respiratory system, and rheumatology. Since the specialty classification in the JCR report was system based rather than purely specialty based (cardiovascular system as against cardiac surgery

5 Page 5 or cardiology), the journals reviewed also could have contained surgical specialty journals. No exclusion was made if a journal met the inclusion criteria two times, for instance a journal dealing with two different body systems (eg cardiovascular system and respiratory medicine) met the criteria two times, it was included both times (eg. Journal of Thoracic and Cardiovascular Surgery). Classification of articles: All articles published in the journals selected in the year 2002 were downloaded using the Endnote software (v Thomson ISI Researchsoft) from Pubmed (National Library of Medicine) database. Only those articles, which contained the term journal article in the notes section of the reference, were retained. This subset of articles includes mainly original research articles and occasionally review articles, letters, comments, historical articles, which are occasionally also labeled as journal articles. A more refined search strategy was not developed for the fear of exclusion of original research articles since the classification in the notes may not be always accurate. A random sample of 200 articles for each subspecialty was drawn using the random number generation tool in Microsoft Excel (an exception was made in internal medicine where 250 random articles were drawn). These abstracts were reviewed for study design, specialty affiliation (two), number of study subjects, method of blinding (if any) and whether the abstract was an adequate source for identifying the design of the study and blinding. When the abovementioned information was not available from the abstract or when there was no abstract, a reviewer obtained the original article and reviewed the methods and results section of the study to obtain the information. The reviewer was unblinded to the affiliations of the authors or to the journal to which the article belonged.

6 Page 6 Study design: The articles downloaded were classified according to the study design. Articles were classified as being therapeutic articles if they were original research articles on human subjects and included one of the following: Studies, which showed the direct effect of any intervention (prevention or treatment) or diagnostic strategy on clinical outcomes. Studies, which prognosticate or determine the risk factors for development of an outcome after an intervention or diagnostic strategy. Studies determining association of an intervention/diagnostic strategy with other factors. Case reports, which showed the effect of any intervention/diagnostic strategy. Systematic reviews/meta-analysis of therapeutic studies. Cost-effectiveness studies for diagnostic/therapeutic intervention. (Diagnostic studies were considered as therapeutic studies only if the authors had mentioned the use of this technique for diagnosis or prognostication in the hypothesis and not as an inference from the study. For e.g. If the authors found the levels of serum CRP elevated in atherosclerosis, and later commented on how CRP could be used as a diagnostic test it was not included. However, if they evaluated the use of CRP as a diagnostic marker for atherosclerosis it was included.) Articles not eligible as therapeutical articles were: Epidemiological studies looking at risk factors or natural histories of disease Case reports, which describe a certain disease as opposed to intervention strategies. Letters, comments, reviews, laboratory studies, editorials, historical articles,

7 Page 7 Only therapeutic articles, which met the above criteria were further classified according to their study design. Specialty affiliation: Unlike the selection of journals, specialty affiliation was chosen with focus on the specific sub-specialty(s) that the articles contributed evidence to. Number of study subjects: The total number of study subjects (combining all arms of the trial for instance or entire cohort) If an article contained two different studies, it was treated as two different studies and was included and excluded according to the criteria mentioned above. We split 4 articles into two study designs and one article into three study designs. Blinding: If the study did not mention the blinding status, it was assumed that the study was unblinded. Grades of evidence: The level of evidence was coded as follows: Level1.1: Meta-analysis of Randomized clinical trials (RCT) and Multi-center RCTs Level1.2: RCT and meta-analysis of observational studies Level2.1: Prospective cohort studies, controlled clinical trials, multi-center controlled clinical trials, systematic reviews Level2.2: Case control study, retrospective cohort study, case crossover, Level3.1: Case series, cross-sectional studies, diagnostic studies Level3.2: Case reports, descriptive study Other level: Guideline, Cost study, Consensus Statistical analysis: The data was collected by one reviewer and was entered into a Microsoft excel spreadsheet. The data were analyzed using Intercooled Stata (version 7.0, College Station, TX, USA). Proportions were tested using the chi square test or the fisher s exact test where necessary.

8 Page 8 Power of the study: The power of the study to detect a difference in the proportion of the of level 1 evidence generating studies was calculated as follows with the assumptions stated: Number of therapeutic articles per specialty=50% of total articles (200) =100 articles. Proportion of level1+2 evidence generating studies= 50% of total therapeutic articles 2. Proportion of level1 evidence generating studies=25% of total therapeutic articles. (assumption) Therefore the power to detect a 15% difference in the proportion=0.56 (if proportion higher), 0.74 (if proportion lower) at alpha =0.05. Evidential index: To calculate the evidential index, we calculated the median total number of patients by the level of evidence (1-3) and the proportion of level 1 evidence generating studies among all the therapeutic studies. A score was calculated on the basis of the product of the two and the specialties were ranked according to this score. Validation study: A second review of a subsample of 200 from the randomly selected articles is planned.

9 Page 9 Top 5 journals of each subspecialty selected All articles published in the year 2002 in specialty journals selected. (And 3149 articles from internal medicine) Anesthesiology2547 Dermatology(1667) Cardiac (3641) Gastro (2476) Hematology(3225) Neurology(2721) Nephrology(2525) Rheumatology 2035 Respiratory(2695) Endnote search Anesthesiology1728 Dermatology(1249) Cardiology (3066) Gastro (1957) Hematology(2893) Neurology(2139) Nephrology(2212) Rheumatology 1533 Respiratory(2184) 200 articles randomly chosen from each specialty (250 articles from internal medicine) Therapeutic articles classified by design, specialty affliation, number of subjects, blinding and abstract quality. Fig1: Study protocol. Numbers in parenthesis indicate the number of articles retrieved. The EndNote search strategy is described in the methods section. Internal medicine not represented in the figure. A total of 3149 articles of internal medicine were included after the end note search strategy Results: The journals selected are shown in table 1 below. Of the specialty systems selected, medicine (internal) had the highest mean impact factor (15.5) and total cites, while dermatology had the lowest mean impact factor (2.0). The range of the impact factors selected was and the lowest total number of cites selected was The total articles published in the year 2002 showed a difference between different specialties with the lowest being 1252 in dermatology and the highest 3149 in Medicine.

10 Page 10 Table 1. The journals selected based on the specialty classification of the JCR 2001 edition. Journal Impact Total cites Number of articles Number of factor published in 2002 * articles included Internal Medicine Annals of Internal Medicine JAMA Lancet NEJM BMJ Cardiac and cardiovascular systems American heart Journal American Journal of cardiology Circulation J American college of cardiology J thoracic and cardiovascular surgery Nephrology American Journal of kidney disease J American society of nephrology Kidney international Nephrology dialysis and transplantation Nephron Clinical Neurology Annals of neurology Archives of neurology Journal of neurology neurosurgery and psychiatry Neurology Stroke Gastroenterology & Heptology 1957 American Journal of gastroenterology Gatroenterology Gut Hepatology Journal of hepatology Respiratory medicine American Journal of respiratory critical care Chest European respiratory journal Journal of thoracic and cardiovascular surgery Thorax Rheumatology Arthritis and Rheumatism Annals of rheumatological disease Rheumatology Journal of rheumatology Clin exp rheumatol Anesthesiology Anesthesiology Anaesthesia Anesthesia Analegesia Brit Journal of Anesthesia Acta Anesthesiologica scandinavica

11 Page 11 of Journal Impact Total cites Number of articles Number factor published in 2002 * articles included Dermatology & venereal diseases Archives of dermatology British Journal of dermatology Journal of the American academy of dermatology Acta Dermatologica and venerologia Dermatological surgery Hematology Blood Transfusion British Journal of hematology Eur Journal of Hematology Annals of hematology American Journal of hematology *the articles are those which were obtained after filtering the original results with the search term journal article in end note software.. Of the 200 articles downloaded for each specialty, the articles were first classified as therapeutic or non-therapeutic. Since each article could be coded with two specialties, the total number of codes was greater than the 2054 articles. (Internal medicine (250), hematology (203), rheumatology (201), all the others 200). Since we coded articles, which had in their design two or more studies as separate articles, there were only 2048 unique articles among the 2054 articles. One article was split into 3 articles (dermatology) and five other articles were split into two (neurology, nephrology, respiratory medicine).

12 Page 12 Table 2: The number of articles by specialty (including the coding for both primary and secondary affiliations) Case series, Controlled Case crosssectional, Multicenter trials, control+ Consensus RCT+ Meta RCT+ Metaobservational prospective retrospective diagnostic Case statements, Specialty analysis cohort cohort studies reports guidelines Other Total Anesthesiology Cardiology Dermatology Gastroenterology Hematology Nephrology Neurology Pulmonary Rheumatology (Chi square statistic (56 df)=303.7, P< After excluding the column with consensus statements or guidelines and the other studies the chi square still remains significant p<0.001) Those articles, which were excluded from the analysis were investigated for any systematic errors by specialty. Anesthesiology (68.9%)and Hematology (65.8%) had the highest proportion of laboratory articles among those excluded. Table 3: Description of the articles not included by specialty (including both codings for specialty affiliations) Laboratory Image (or Others Total studies Non therapeutic epidemiological studies clinical vignettes) Specialty Anesthesiology Cardiology Dermatology Gastroenterology Hematology Nephrology Neurology Pulmonary Rheumatology Each cell represents the number of studies Combined levels of evidence: We collapsed the various levels of evidence into their primary levels of evidence since some of the values were very small. The results are shown in Table 4.

13 Page 13 Table 4: Collapsed table of the levels of evidence. Specialty Level1 Level2 Level3 Total Anesthesiology Cardiology Dermatology Gastroenterology Hematology Nephrology Neurology Pulmonary Rheumatology (Each cell represents the number of studies, chi square statistic(24)=79.43, p<0.001) We assessed the quality of the abstract in determining the design of the study and we performed a chi square analysis to understand the heterogeneity among different specialties in terms of the adequacy of abstracts (Table 5). However, after exclusion of anesthesiology from the analysis, the chi square statistic no longer remained significant (p>0.20). Table 5: The presentation of adequacy of abstracts to determine the design of the study by specialty Abstract adequate Abstract inadequate Abstract inadequate Total Abstracts Total for blinding for other inadequate Specialty reasons Anesthesiology Cardiology Dermatology Gastroenterology Hematology Nephrology Neurology Pulmonary Rheumatology (Each cell represents the number of studies, chi square analysis chi2 (8)=24.55, p=0.002) We graphically analyzed the number of subjects per specialty. The number of subjects was the sum of all the study subjects in the study. Since cross sectional studies and registry based studies could have very large sample sizes, we truncated the number of

14 Page 14 patients at 1000 for the graphical analysis (Fig 2&3) (and 10,000 for number of subjects by level of evidence) numberofsubjects Number of patients(truncated at 1000) anesthesia dermatology hematology neurology rheumatolog cardio gastro nephrology pulmonary Box plots by specialty of number of subjects Fig2: Plot of the total number of patients in each specialty numberofsubjects Number of patients(truncated at 10000) Box plots by level of evidence of number of subjects Fig 3: Plot of the number of patients by the level of evidence. Legend 1 Meta-analysis of RCTs and multicenter RCTs 2 RCT and meta-analysis of observational studies 3 Prospective cohort studies, controlled clinical trials, multicenter controlled clinical trials, systematic reviews 4 Case control study, retrospective cohort study, case crossover, 5 Case series, cross-sectional studies, diagnostic studies 6 Case reports, descriptive study

15 Page 15 Ranking of specialties: The median number of patients and the proportion of therapeutic studies was tabulated for each specialty by level of evidence (table 6). The product of the median number of patients and the proportion of level 1 studies was calculated and the specialties were ranked according to this index (table 7). Table 6: Median number of patients and proportion of total therapeutic studies by level of evidence. Median number of patients Proportion of therapeutic studies Median number of patients Proportion of therapeutic studies Median number of patients studies in level1 which are in level 2 which are in level 3 which Specialty studies level 1 studies level 2 studies level 3 Proportion of therapeutic Anesthesiology Cardiology Dermatology Gastroenterology Hematology Nephrology Neurology Pulmonary Rheumatology Table 7: Ranking on the basis of the proportion of level1 studies and the combined effect. Product of median number of patients Rank Specialty and the proportion of level1 studies Anesthesiology 24 4 Cardiology Dermatology Gastroenterology Hematology Nephrology Neurology Pulmonary Rheumatology are Discussion: The results from our review demonstrate the heterogeneity in the different specialties with regards to the proportions of evidence generating studies and abstract quality. Given the growing need for specialties to become more evidence based in practice, this study we hope, will provide impetus to specialties and allow them to look at their research with a new perspective. Although it is important for specialties to evaluate the research and the

16 Page 16 quality of the abstracts, the importance of using a comparison group cannot be underemphasized. The adoption of evidence based medicine and the differing evidential strength of the different journals could be partially explained by the differing morbidity and mortality in the different specialties. Since heart disease is the leading cause of mortality in the united states 4, there is a need for better quality of literature and larger studies in the cardiology literature. It is logical to hypothesize that specialties with greater morbidities will have better literature. Our study partially supports this hypothesis and we can see that cardiology and neurology (cerebrovascular diseases) (which are among the leading causes of death 4 ) are similarly ranked very high in terms of the proportions of quality studies published. However, there are exceptions to this and gastroenterology, hematology and anesthesiology do show a high proportion of better research study designs, in spite of not being as big contributors to morbidity as compared to neurology or cardiology. While it can be argued that the high volume of surgical procedures 5 is a proxy for morbidity in the case of anesthesiology, we would need to assess the quality of evidence in the surgical specialties before we can reach any inference. Table 8: Comparison of some statistics of different specialties. (NA=Not available) Number of NIH sponsored clinical trials 6 Number of specialty fellows/residents 7 Mean Journal Impact Mean number of cites Specialty Factor Anesthesiology NA Cardiology Dermatology Gastroenterology Hematology Nephrology NA Neurology Pulmonary Rheumatology

17 Page 17 Another hypothesis, which could explain better research in specialties, could be the number of clinical trials or the funding allotted (Table 8). Although the comparison suffers from an ecological fallacy, we compared the number of clinical trials currently sponsored by NIH by specialty and find that the number of trials (as a proxy measure for funding) actually does not reflect the same trends as those shown by the quality of research. However, the fact that our comparison is incomplete in its inclusion of specialties disallows us from comparing accurately and we might be unable to make correct inferences. The number of researchers in each field might also be an interesting aspect to the differing quality of research published. We used the American Medical Associations online database 7 to tabulate the number of residents/fellows in training by the different specialties. We do see that there is a range of the number of trainees and this might also potentially influence the quality of research for e.g. in the case of rheumatology. The heterogeneity in reporting study designs in the abstracts can however, not be attributed to any of the above and is merely a reflection of the quality of the journals and the emphasis on structured reporting. We do see that most journals have similar proportions for which we needed to verify the study design. However, it is only one specialty (anesthesiology, see table 5), which contributes to majority of the heterogeneity. After excluding the specialty, we found the heterogeneity in the comparison of specialties with regards to the adequacy of the abstract to be not significant (p>0.20). Classification of levels of evidence has been the source of much controversy, and authors have argued very strongly in favor of well designed observational studies 8 providing as good results as randomized clinical trials. We do not deny the fact that a good

18 Page 18 observational study could be much better than a poorly deigned randomized trial and they are continuums of evidence rather than rigid barriers between different studies. The classification of evidence we have used differs from that used by the USPTF 9 in that we differentiate between prospective and retrospective studies, and that we give equal weight to controlled trials and cohort studies 8. We understand the limitations of categorizing a continuum of evidence into rigid barriers and we suggest these levels of evidence only as a means of investigating the data, and not as rigid indicators. We have tried to develop a ranking system for specialties, which has the basic fallacy that all specialties are not comparable for different reasons. We propose this merely as a guide to the different specialties and suggest that individual specialties re-assess their literature with comparisons with different specialties, so that evidence based medicine can reach every specialty literature and ultimately trickle down to the patient. Our ranking system does not take into account the evidence from observational studies and this is a shortcoming of our system. One can argue that the random sample that we have drawn is not representative of the specialty. To counter this we used certain measures. Although the impact factor has been a much debated measure of quality of the study, we still find this to be the most suitable to select journals. However to ensure that we picked journals that published original articles, we used the 40% clinical criteria and weeded out pure review journals, which had high impact factors. Also we used the total cites as a measure of the circulation of the journal. We do understand that the total cites would depend on the number of articles published, but using it a rough measure to filter the journals yielded appropriate journals. A review of the selected journals by a few clinicians at the Johns Hopkins School Of

19 Page 19 Public Health was performed and we did not find any major criticisms to the choice of journals. The choice of therapeutic articles again has shades of subjectivity to it and the fact that we are dealing with small numbers ( articles per specialty), misclassification of these articles could significantly change the inferences. To ensure against such a misclassification bias, we propose to undertake a validation study by a second observer and a repeat classification of a subsample to calculate misclassification bias. Also the study proposes to increase the number of articles reviewed per specialty to increase the power of the study and to provide a better precision. We have attempted for the first time a measure of comparison of different specialties with reference to their evidential strengths. Although we understand the limitations of doing so, we hope that this study contributes to our understanding of the heterogeneity in the specialties in the adoption of the methods of evidence-based medicine. We do not intend our ranking methods to be infallible or rigid, rather we suggest they be used with caution and as indicators for specialties to take a look at their specialty journals.

20 References: Evidential strength of medical specialties Page Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. Bmj 1996; 312: Bhattacharyya N. Peer review: studying the major otolaryngology journals. Laryngoscope 1999; 109: ISI. Journal Citation Report, NVSS. Deaths: Leading causes for Vol. 50: Center for Diseases Control, NCHS. National Hospital Discharge and Ambulatory Surgery Data, /28/ NIH. Clinical Trials.gov. 04/27/ AMA. FREIDA Online specialty training search, /28/ Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000; 342: Preventive-Services-Task-Force. Guide to clinical preventive services: report of the U.S. Preventive Services Task Force. Baltimore: Williams & Wilkins, 1996.