Completeness of reporting of studies of diagnostic accuracy in Gastroenterology and Hepatology literature: a STARD assessment

Transcription

1 FM U P FACULDADE DE MEDICINA UNIVERSIDADE DO PORTO DEPARTMENT OF BIOSTATISTICS AND MEDICAL INFORMATICS Completeness of reporting of studies of diagnostic accuracy in Gastroenterology and Hepatology literature: a STARD assessment Adriana Meneses, med06155@med.up.pt Eduardo Vilela, med06002@med.up.pt Ana Fragoeiro, med06238@med.up.pt Inês Gonçalves, med06023@med.up.pt André Carvalho, med06194@med.up.pt João Sarmento, med06044@med.up.pt Daniela Machado, med06216@med.up.pt Luís Santos, med06065@med.up.pt Manuel Pinto, med06068@med.up.pt Pedro Alves, med06107@med.up.pt Sara Ferreira, med06128@med.up.pt Tiago Taveira, med06150@med.up.pt Adviser: Cristina Santos; csantos@med.up.pt Class: 2 Abstract Introduction: The increasing number of diagnostic tests over the years has led to a greater demand of articles concerning them. As such, the evaluation of their accuracy has become incredibly important as the information they contain can aid us identifying which tests are useful to perform. Several factors can threaten the internal and external validation of accuracy studies. Aim: To evaluate the quality of the articles which evaluate diagnostic tests in Gastroenterology and Hepatology on journals with the highest impact factor in the past 12 years, based on the STARD checklist. Methods: We searched MEDLINE with a valid strategy to identify articles on diagnostic accuracy and then we optimised it. We included articles which report on primary studies of diagnostic accuracy in Gastroenterology and Hepatology on humans. All eligible articles were read by two reviewers independently to determine if they met the inclusion criteria. The STARD statement was used to assess the quality of reporting. Each included article was read by two independent reviewers who determined whether each item of the checklist was described adequately in the text. Results: The average of the STARD items presented on the articles in each biennium was of 10,7 for the first (94/95); 14,7 for the following two (96/97 and 98/99); 14,6 in 00/01 and 16,6 for 02/03 and 04/05. The evolution was only statistically significant (p < 0,05) when comparing the 94/95 biennium with the following ones. Discussion: After this analysis, we are able to affirm that an increase of quality with elapsing of the time is present. Not being this evolution dependent of the STARD table, it comes from factors which were not of the scope of our analysis. In the journals with highest impact factor, the classification attributed through the STARD was waited to be quite higher. A disdain in relation to some of the STARD items was evident, nominated in the reproducibility of the diagnostic tests. We found diverse incompatibilities between the way authors write their articles and the form as the STARD considers they should be written. Key-words: Diagnostic accuracy; gastroenterology; hepatology; quality of reporting. Introdução à Medicina 2006/2007 1

2 Introduction As an extremely dynamic field, Medicine evolves everyday, integrating new ideas and concepts from all over the world. Gastroenterology and Hepatology, medical specialties that involve the diagnosis and treatment of diseases concerning the digestive tract and its organs (liver, gall bladder) has had an astonishing evolution over the last few decades, which is due mostly to the development of digestive endoscopy [1] technique that attempts to predict the presence (or absence) of a disease. The world of diagnostic tests is highly dynamic. New tests are developed at a fast rate, and the technology of existing tests is continuously being improved [2]. Yet, it is the clinical practice that is responsible for the appearance of diagnostic tests, essential tools on improving healthcare at a global level. The increasing number of diagnostic tests over the years has led to a greater demand of articles concerning them. However, exaggerated and biased results from poorly designed and reported diagnostic studies can trigger their premature dissemination and lead physicians into making incorrect treatment decisions [2]. As such, the evaluation of their accuracy has become incredibly important as the information they contain can aid us identifying which tests are useful to perform [3]. A rigorous evaluation of diagnostic tests before introduction into clinical practice could not only reduce the number of unwanted clinical consequences related to misleading estimates of test accuracy but also limit healthcare costs by preventing unnecessary testing [2]. To correctly interpret the results of clinical studies on testing, readers must understand the design, conduct and data analysis and be able to judge the internal validity and generalizations of the results [3]. A survey of studies of diagnostic accuracy published in four major medical journals between 1978 and 1993 showed that most diagnostic tests are still inadequately appraised [4]. Recent studies about the quality of reporting of diagnostic accuracy studies revealed that the quality of these tests were disappointingly poor [3];[5]. Several factors can threaten the internal and external validation of accuracy studies [3]. On this account, several checklists which help to evaluate the quality of studies about diagnostic tests have come up. The first was published in 2000 and had about 40 items concerning quality [6]. In January 2003, the guide for reporting studies of diagnostic accuracy: Standards for Reporting of Diagnostic Accuracy (STARD) was published simultaneously in 8 medical journals (Radiology; American Journal of Internal Introdução à Medicina 2006/2007 2

3 Medicine; British Medical Journal; Clinical Biochemistry; Clinical Chemistry; Clinical Chemistry of Laboratory Medicine; Lancet) [2];[8]. The objective of the STARD initiative is to improve the quality of the reporting studies in diagnostic accuracy. Complete and accurate reporting allows the reader to detect the potential for bias in the study (internal validity) and to access the possibility of generalization and application of the results (external validity) [2]. As such, given the facts we have just presented, we propose to evaluate the quality of the articles which evaluate diagnostic tests in Gastroenterology and Hepatology on journals with the highest impact factor in the past 12 years, based on the STARD checklist, and then, conclude if there was indeed some evolution in these type of studies. Also, we aimed to know whether the STARD initiative had some influence on it or not. Participants and Methods Data Sources We searched the US National Library of Medicine, MEDLINE, with a valid strategy to identify articles on diagnostic accuracy using the query [7] presented on Table 1. Then, we optimised it. The search was subsequently limited to publications released between 1994 and 2005 (twelve years time) and studies focusing on human subjects. The journals were selected according to their impact factor and the five ones with the highest impact factor in each year of our study were chosen (Table 2). After, we made a random selection of 50 articles per biennium (94/95; 96/97; 98/99; 00/01; 02/03; and 04/05) which were analysed by two independent reviewers. Table 1: General query, used to identify articles on diagnostic accuracy [7]. (((((((((((("sensitivity and specificity"[all Fields] OR "sensitivity and specificity/standards"[all Fields]) OR "specificity"[all Fields]) OR "screening"[all Fields]) OR "false positive"[all Fields]) OR "false negative"[all fields]) OR "accuracy"[all Fields]) OR (((("predictive value"[all Fields] OR "predictive value of tests"[all Fields]) OR "predictive value of tests/standards"[all Fields]) OR "predictive values"[all Fields]) OR "predictive values of tests"[all Fields])) OR (("reference value"[all Fields] OR "reference values"[all Fields]) OR "reference values/standards"[all Fields)) OR (((((((((("roc"[all Fields] OR "roc analyses"[all Fields]) OR "roc analysis"[all fields]) OR "roc and"[all fields]) OR "roc area"[all fields]) OR "roc auc"[all Fields]) OR "roc characteristics"[all Fields]) OR "roc curve"[all Fields]) OR "roc curve method"[all Fields]) OR "roc curves"[all Fields]) OR "roc estimated"[all Fields]) OR "roc evaluation"[all Fields])) OR "likelihood ratio"[all Fields]) Introdução à Medicina 2006/2007 3

4 Table 2: List of the journals with the highest impact factor in each year of the study. Year Journals with highest impact factor Year Journals with highest impact factor GASTROINTEST ENDOSC GASTROENTEROL CLIN N 1996 GASTROINTEST ENDOSC ALIMENTAR PHARM TERAP GASTROINTEST ENDOSC 2004 AM J GASTROENTEROL AM J PHISIOL-GASTR AM J GASTROENTEROL Study selection We included articles which report on primary studies of diagnostic accuracy in Gastroenterology and Hepatology on humans. The title, abstract and keywords of all eligible articles were read by two reviewers independently to determine if they met the inclusion criteria. All articles which met the following items were excluded: nonevaluation of diagnostic tests; diagnostic tests not applied on humans; articles without available abstract; articles written in other language rather than English or Portuguese; and review articles. Disagreements were discussed and resolved with a third reviewer on a consensus meeting. Introdução à Medicina 2006/2007 4

5 Data extraction The STARD statement [2] (Table 3) was used to assess the quality of reporting. The statement contains a list of 25 (twenty-five) items. Each included article was read by two independent reviewers and they determined whether each item of the checklist was described adequately in the text. Disagreements were discussed and resolved with a third reviewer on consensus meeting. Table 3: STARD checklist of items to improve the reporting of studies on diagnostic accuracy. Section and topic Item Describe TITLE/ABSTRACT/ The article as a study on diagnostic accuracy (recommend MeSH heading 'sensitivity 1 KEYWORDS and specificity') INTRODUCTION 2 The research question(s), such as estimating diagnostic accuracy or comparing accuracy between tests or across participant groups METHODS Participants 3 The study population: the inclusion and exclusion criteria, setting(s) and location(s) where the data were collected 4 Participant recruitment: was this based on presenting symptoms, results from previous tests, or the fact that the participants had received the index test(s) or the reference standard? 5 Participant sampling: was this a consecutive series of patients defined by selection criteria in (3) and (4)? If not specify how patients were further selected. 6 Data collection: were the participants identified and data collected before the index test(s) and reference standards were performed (prospective study) or after (retrospective study)? Reference standard 7 The reference standard and its rationale Test methods 8 Technical specification of material and methods involved including how and when measurements were taken, and/or cite references for index test(s) and reference standard 9 Definition and rationale for the units, cutoffs and/or categories of the results of the index test(s) and the reference standard 10 The number, training and expertise of the persons (a) executing and (b) reading the index test(s) and the reference standard 11 Whether or not the reader(s) of the index test(s) and reference standard were blind (masked) to the results of the other test(s) and describe any information available to them Statistical methods 12 Methods for calculating measures of diagnostic accuracy or making comparisons, and the statistical methods used to quantify uncertainty (e.g. 95% confidence intervals) 13 Methods for calculating test reproducibility, if done RESULTS Participants 14 When study was done, including beginning and ending dates of recruitment Clinical and demographic characteristics (e.g. age, sex, spectrum of presenting symptom(s), comorbidity, current treatment(s), recruitment center) How many participants satisfying the criteria for inclusion did or did not undergo the index test and/or the reference standard; describe why participants failed to receive either test (a flow diagram is strongly recommended) Introdução à Medicina 2006/2007 5

6 Reference standard 17 Time interval and any treatment administered between index and reference standard 18 Distribution of severity of disease (define criteria) in those with the target condition; describe other diagnoses in participants without the target condition Test results 19 A cross tabulation of the results of the index test(s) by the results of the reference standard; for continuous results, the distribution of the test results by the results of the reference standard 20 Indeterminate results, missing responses and outliers of index test(s) stratified by reference standard result and how they were handled 21 Adverse events of index test(s) and reference standard Estimation 22 Estimates of diagnostic accuracy and measures of statistical uncertainty (e.g. 95% confidence intervals) 23 Estimates of variability of diagnostic accuracy between subgroups of participants, readers or centers, if done 24 Measures of test reproducibility, if done DISCUSSION 25 The clinical applicability of the study findings The 25 (twenty-five) items of the STARD checklist can be divided into: item addressing title/abstract/keywords; introduction; study participants; test methods; statistical methods; results; and discussion. For each item of the STARD statement, the total number of articles reporting the elements mentioned in that item was summed. A total STARD score for each article was calculated by summing the number of reporting items (0-25 points possible). A higher score indicates a better quality of reporting. Six items (items 8, 9, 10, 11, 13 and 24) concern both the index test as well the reference test. If only the index test or the reference test is referred, a 0,5 point score was attributed. Statistical analysis After analysing every abstract, we calculated the kappa statistic in order to measure the difference between the means obtained by both reviewers. After scoring every article included in our study, we calculated that difference using the Intraclass Correlation Coefficient (ICC). We performed the One-Way ANOVA test with Post Hoc and Bonferroni to check for any significant evolution on these kind of studies. As summary measures we used the mean and standard deviation. The significance level set was 0,05. Data was analysed using the software SPSS (Statistical Package for Social Sciences). Introdução à Medicina 2006/2007 6

7 Results Abstract results Our strategy of limiting the research to studies made on humans and to the five journals with the highest impact factor in each year from 1994 to 2005 has resulted in the identification of 2141 publications (174 in 1994; 144 in 1995; 168 in 1996; 167 in 1997; 129 in 1998; 180 in 1999; 170 in 2000; 144 in 2001; 165 in 2002; 140 in 2003; 261 in 2004; and 299 in 2005). We took a random sample of 50 articles per biennium and after all abstracts having been read we selected 102 articles (Diagram 1), using the main non-inclusion criteria presented on the diagram. The agreement rate between reviewers was 84,6% and the kappa statistic turned up with 0,663 (p < 0,001). Diagram 1: Search and selection process of articles on diagnostic accuracy in Gastroenterology and Hepatology [5]. Introdução à Medicina 2006/2007 7

8 Article results After the selection of 102 articles according to the inclusion criteria, these were read and only 78 were classified using the STARD checklist (Diagram 1). Twenty-four articles were excluded either because they were not related to the analysis of diagnostic tests or because the full text could not be found. The degree of agreement between the two reviewers responsible for the classification of the articles was calculated using the ICC and a value of 0,885 was obtained, with a confidence interval of 95% (CI 95) between 0,823 and 0,925, p < 0,001. Evolution of articles quality The average of the items presented on the articles in each biennium was of 10,7 for the first (94/95); 14,7 for the following two (96/97 and 98/99); 14,6 in 00/01 and 16,6 for 02/03 and 04/05 (Table 4). Table 4: Average scores attributed to the articles in each biennium. Biennium Mean N Std. Deviation , , , , , , , , , , , ,67468 Total 14, ,30844 Our results reveal that the evolution was only statistically significant (p < 0,05) when comparing the 94/95 biennium with the following ones (Table 5; Graphic 1). Table 5: Comparison between 94/95 with the following biennia. Biennium Biennium Std. Error Sig. 95% Confidence Interval Upper Bound Lower Bound ,10778,010-7,3091 -, ,22024,027-7,6633 -, ,05675,007-7,0628 -, ,04335,000-9,0184-2, ,15511,000-9,4353-2,4208 Introdução à Medicina 2006/2007 8

9 Graphic 1: Evolution of article s scores using mean. Items of the STARD checklist The distribution of the STARD checklist s items is presented in Table 6. The items which are most infrequently present are 13 (16,7%) and 24 (11,5%), both concerning the reproducibility of the diagnostic tests mentioned on the article. On the other hand, items 2 (88,5%), 19 (83,3%) and 25 (97,4%) were present in most of the articles evaluated. Table 6: Frequencies and percentages of the STARD checklist items on analysed articles (n =78). Introdução à Medicina 2006/2007 9

10 Discussion The main conclusions of our work are that: (1) there was an evolution of the quality of the articles that evaluate diagnostic tests throughout the time. However, this evolution was insufficient, even in the most high-ranked journals; (2) the STARD checklist did not affect the quality of articles; (3) the relevance given to each item of that checklist by the authors was not equal. For base, we had the article analysis of 6 biennia, in order to be able to survey the existence or not of an independent secular evolution of other factors that not it STARD. Thus, and after the related analysis, we are able to affirm that an increase of quality with elapsing of the time is present. Not being this evolution dependent of the STARD table (since a previous ascent to the existence of the same one was verified), it comes from factors which were not of the scope of our analysis. Yet, and with speculative base, we suggest that this evolution is due to the acquisition of knowledge that occurs naturally in function of the time. However, to try to inquire the nature of these factors, other studies would have to be carried through. Moreover, and in the journals with highest impact factor, the classification attributed through the STARD remains, at this moment, around 17. It was waited of our part that the level of quality of articles in these journals was higher. Other carried through similar studies in different areas disclose equally that the quality of articles itself finds the same level significantly [3];[5];[9]. Thus, it has come to be denoted in the generality of the scientific community a lack of shrewdness in the accomplishment of this scientific article model. Of the analysis of the different frequency of items of the STARD, a disdain in relation to some of these was evident, nominated in the reproducibility of the diagnostic tests. Being the reproducibility an important point for the evaluation of a diagnostic test, since it refers to the capacity of a test to be able to be applied in different circumstances, getting the same results, it should not be left behind on this type of reports. The difference between the average results calculated by the ICC was not very high (0,885), meeting the expectations of a recently published work (N. Smidt et al., 2006) which demonstrates that the reproducibility of the STARD checklist is good (ICC = 0,79 [95% CI: 0,62 to 0,89]. Our study has a few limitations. First, the identification of studies of diagnostic accuracy is quite difficult, although we searched MEDLINE with a valid strategy to identify all the studies on diagnostic accuracy published in the past 12 years. Secondly, Introdução à Medicina 2006/

11 we only selected a random sample of 50 studies per year in the journals with the highest impact factor. So, there were many studies which were not taken into account. Thirdly, some of the selected articles could not be found, and thus, could not be taken into account too. We found diverse incompatibilities between the way authors write their articles and the form as the STARD considers they should be written. Two possibilities can explain what occurred: either the STARD fundaments are not appropriate, not having a direct application to these studies, or, on the other side, the error inhabits in the journals which should demand the application of this checklist as an essential tool on article evaluation. Acknowledgements Altamiro da Costa Pereira, PhD MD Cristina Santos, MD References [1] Communication from Professor Carlos Sofia, Coimbra University; [2] Bossuyt P, Reitsma J, Bruns D, Gatsonis C, Glasziou P, Irwig L, et al. Towards complete and accurate reporting o studies of diagnostic accuracy: the STARD initiative. Annals of internal medicine. 2003;138; [3] Coppus S, Veen F, Bossuyt P, Mol B. Quality of reporting of test accuracy studies in reproductive medicine: impact of the Standards for Reporting of Diagnostic Accuracy (STARD) initiative. Fertility and Sterility. 2006; [4] Reid MC, Lachs MS, Feinsteins AR. Use of methodological standards in diagnostic test research. Getting better but still not good. JAMA. 1995; [5] Smith N, Rutjes A, Windt A, Ostelo R, Reitsma J, Bossuyt P, et al. Quality of reporting of diagnostic Accuracy Studies. Radiology. 2005; [6] Bruns DE, Huth EJ, Magid E, Young DS. Towards a checklist for reporting of studies of diagnostic accuracy of medical tests. Clin Chem. 2000; Introdução à Medicina 2006/

12 [7] Devillé W, Buntins F, Bouter L, Montori V, Vet H, Windt D, et al. Conducting systematic reviews of diagnostic studies; didactic guideless. Medical research methodology. 2002; [8] Smidt N, Rutjes A, van der Windt D, Ostelo R, Bossuyt P, Reitsma J, et al. Reproducibility of the STARD checklist: an instrument to assess the quality of reporting of diagnostic accuracy studies. BMC Medical Research Methodology. 2006;6:12; [9] Siddiqui MAR, Azuara-Blanco A, Burr J. The quality of reporting of diagnostic accuracy. J. Ophthalmol. 2005;89; Introdução à Medicina 2006/