Citation indexes do not reflect methodological quality in lung cancer randomised trials

Transcription

1 Original article Annals of Oncology 14: , 2003 DOI: /annonc/mdg203 Citation indexes do not reflect methodological quality in lung cancer randomised trials T. Berghmans 1 *, A. P. Meert 1, C. Mascaux 1, M. Paesmans 1, J. J. Lafitte 2 & J. P. Sculier 1 1 Department of Internal Medicine, Institut Jules Bordet, Bruxelles, Belgium; 2 Service of Pneumology and Thoracic Oncology, CHU Calmette, Lille, France Received 28 August 2002; revised 12 December 2002; accepted 19 December 2002 Background: Citation factors are applied to assess scientific work despite the fact that they were developed commercially in order to compare competing journals. The aim of the present study was to determine whether there is a relationship between citation factors and a trial s methodological quality using published randomised trials in lung cancer clinical research. Material and methods: All of the randomised trials included in nine systematic reviews performed by the European Lung Cancer Working Party (ELCWP) were assessed using two quality scales (Chalmers and ELCWP). Results: One hundred and eighty-one articles were eligible. The median overall ELCWP and Chalmers quality scores were 61.8% and 49.0%, respectively, with a correlation coefficient (r s ) of 0.74 (P <0.001). A weak association was observed between citation factors and quality scores with the respective correlation coefficients ranging from 0.18 to 0.40 (ELCWP scale) and from 0.21 to 0.38 (Chalmers scale). American authors published trials significantly more often in journals with high citation factors than European or non-american authors (P <0.0001), despite no better methodological quality. Positive trials, which were significantly more likely to be published in journals with higher citation factors, were of no better quality than negative ones. Conclusion: Journals with higher citation factors do not appear to publish clinical trials with higher levels of methodological quality, at least for trials in the field of lung cancer research. Key words: bibliometry, citation factor, eurofactor, impact factor, lung cancer, prestige factor Introduction *Correspondence to: Dr T. Berghmans, Institut Jules Bordet, Rue Héger-Bordet 1, 1000 Bruxelles, Belgium. Tel: ; Fax: ; thierry.berghmans@bordet.be One field of bibliometry is the quantitative appraisal of the literature published in periodicals. This approach is frequently based on citation analysis and has various possible applications: management of library collections, comparison of competing journals, assessment of the impact of research groups or a country in particular fields and measurement of the amount of the scientific work [1, 2]. The first bibliometric studies ranking scientific journals according to the number of references to each journal were reported as early as 1917 [3] and 1927 [4]. The Science Citation Index was published in 1963 and the term impact factor (IF) was used to assess the journal rank [1]. The citation data are obtained from a database produced by the Institute for Scientific Information (ISI). The IF is defined as the recorded number of citations within a certain year to the items published in the journal during the two preceding years, divided by the number of such items [2, 5]. The journal IF will depend on the research field, with high IFs associated with journals covering a large research area with many references per article [6]. Editors are sometimes tempted to artificially increase the IF of their journal in several different ways, such as self-citation, incorporation of review articles which are cited more often than original reports [2, 5] or the selection of authors on the basis of previous publications, publishing those with a higher potential impact on citation [1]. As the number of references allowed in a journal article may be limited, authors refer more often to review articles, and as a result reviews generally have higher citation rates than original articles. In order to avoid some of the artificial increases in IFs, the prestige factor (PF) was developed, which only takes into consideration original articles and excludes review articles. Also, ISI includes in its database about 3200 journals from an estimated total of published worldwide, with a bias towards English language journals and English-speaking authors [6]. To reduce discrimination, Europeans developed their own citation factor, the eurofactor (EF), which compares only European journals. Citation factors were developed with the initial purpose of comparing competing journals in a commercial way. Later, the assumption was made that the journal is representative of the articles published in it and consequently that citation factors could represent a way of assessing the quality of the scientific work. In certain countries, IFs have been used to select individuals for academic promotions or to determine resource allocation [5]. However, the contribution of individual articles to the global IF of a journal is often disproportionate. The few most cited articles account for the majority of citations [6]. The aim of the 2003 European Society for Medical Oncology

2 716 present work was to determine whether there is a relationship between methodological quality, as assessed by specific quality scores, and various citation indexes (IF, PF and EF) in lung cancer randomised trials. Materials and methods The randomised trials included in nine systematic reviews (some with metaanalysis) conducted by the European Lung Cancer Working Party (ELCWP) evaluating various treatment modalities for lung cancer [7 15] were assessed. Five reviews were dedicated to small-cell lung cancer [7, 8, 10, 11, 14]; they assessed the role of prophylactic cranial irradiation, thoracic irradiation in limited disease, the respective role of cisplatin and etoposide in comparison with other drugs, the place of maintenance chemotherapy after induction treatment and the effect of colony-stimulating factors in addition to chemotherapy. Four reviews were related to the treatment of non-small-cell lung cancer [9, 12, 13, 15]; they investigated the role of chemotherapy in addition to local treatment in non-metastatic disease, the comparison of chemotherapy with best supportive care in stage IV tumours, the role of mitomycin C and the place of new drugs in chemotherapy. Articles were retrieved from a Medline search, completed by the references known by the authors or found in the articles. They must be related to the particular topic developed in each systematic review, include only lung cancer patients and be published in English, French or Dutch. Only fully published articles were considered. When trials were published more than once, each article was considered for this review and separately qualitatively assessed by the investigators, with the exception of articles reporting preliminary results without survival (toxicity analysis) or when updated results were presented in the form of a summary. A team of medical doctors and a biostatistician carried out the methodological quality evaluation. Consensual agreement on the scores attributed to each item for each trial was obtained during regular meetings requiring the presence of at least 75% of the investigators. The methodological quality of all randomised trials was assessed using two previously published quality scales, ELCWP [7 15] and Chalmers [16] (Appendices A and B). Scores are expressed as a percentage of the maximal theoretical value that can be obtained for the considered category. Higher scores reflect the better methodological quality of the trial. Three citation indexes were used: the IF, the PF and the EF. For IF, the value obtained from the last 1999 version available to the authors (IF99) and the value of IF at the date of publication of the article (IF Pub) were considered. IF Pub was researched by the librarian of the Institut Jules Bordet, by contacting the journals direct or by requesting the information from other scientific libraries. PFs and EFs were obtained from their respective web sites: and The result of a trial was considered to be conclusive if the statistical test comparing the different study arms, according to the primary objective as defined by the authors, gave a value of P <0.05 in favour or not of the experimental arm. If this was the case, the trial was termed positive ; otherwise, it was classified as negative. When no primary objective was clearly defined, we considered survival as the main study endpoint. Statistics were performed using the software Statistica (Statsoft, Tulsa, OK). Citation factors were treated as continuous variables. The association between two continuous variables was measured by the Pearson correlation coefficient or by the Spearman rank correlation coefficient, if the assumption of Gaussian distributions was not valid. Non-parametric Mann Whitney (for dichotomic variables) and Kruskal Wallis (for nominal variables with multiple classes) tests were done to compare the distributions of continuous variables, in cases of non-gaussian distributions, according to the value of a discrete variable. When the continuous variable presented with a normal Table 1. Quality scores of randomised phase III trials in the treatment of lung cancer Scale ELCWP Chalmers Median scores 61.8% 49.0% Correlations r s P r s P Chalmers 0.74 <0.001 Year of publication <0.001 No. of eligible patients ELCWP, European Lung Cancer Working Party. Table 2. Relationship between citation factors and quality scores No. of ELCWP Chalmers articles r s P r s P IF IF Pub < < PF EF EF, Eurofactor; ELCWP, European Lung Cancer Working Party; IF, impact factor; IF Pub, value of IF at the date of publication of the article; IF99, the value of IF obtained from the last 1999 version available to the authors; PF, prestige factor. distribution, Student s t-test was carried out to compare its distribution according to the value of a discrete variable. Chi-square statistics were used to assess the relationship between two dichotomic variables. Because of the multiple tests that we performed, we considered the difference between two comparisons as statistically significant for values of P < Results A total of 181 eligible articles were included in nine systematic reviews. They were published in 31 journals and one book in the period from 1977 to Values of citation factors were available for IF99, IF Pub, PF and EF in 171, 150, 175 and 39 articles, respectively. Quality scores The overall median ELCWP and Chalmers quality scores were 61.8% (range ) and 49% (range 9 86), respectively. A good correlation between the scores was observed with an r s of 0.74 (P <0.001). Quality scores were significantly correlated with year of publication more recent trials had better scores. The respective r s were 0.44 (P = 0.001) for ELCWP and 0.51 (P <0.0001) for Chalmers. A weak correlation between the scores and the number of eligible patients per article was found (r s = 0.19; P = 0.01). The results are summarised in Table 1. Relationship between citation factors and quality scores A weak relationship was observed between the value of citation factors and quality scores (Table 2). The correlation coefficient ranged from 0.18 to 0.40 when citation factors were correlated

3 717 Table 3. Ranking of journals with more than three publications according to the ELCWP and Chalmers quality scores, IF 99 and PF citation factors Rank ELCWP Chalmers IF99 PF 1 Ann Intern Med JNCI NEJM NEJM 76.7% 63.1% Ann Oncol Br J Cancer JNCI JNCI 69.6% 58.0% JNCI Ann Oncol Ann Intern Med JCO 69.3% 56.5% Br J Cancer NEJM JCO Ann Intern Med 68.4% 56.1% NEJM Ann Intern Med Cancer Cancer 66.7% 55.1% JCO Lung Cancer Br J Cancer Br J Cancer 63.8% 52.1% Semin Oncol JCO Ann Oncol Int J Rad Oncol 60.2% 50.3% Eur J Cancer Eur J Cancer Int J Rad Oncol Ann Oncol 59.9% 48.5% Lung Cancer Cancer Semin Oncol Eur J Cancer 59.8% 48.4% Cancer Semin Oncol Eur J Cancer Lung Cancer 59.7% Int J Rad Oncol Int J Rad Oncol Lung Cancer Semin Oncol 29.0% 28.2% Journals were classified in descending order according to their methodological median quality score and compared with their citation factor value. The position of the journals when classified according to the quality scores is not equivalent with the citation factors ranking. Ann Intern Med, Annals of Internal Medicine; Ann Oncol, Annals of Oncology; Br J Cancer, British Journal of Cancer; Eur J Cancer, European Journal of Cancer; ELCWP, European Lung Cancer Working Party; Int J Rad Oncol, International Journal of Radiation Oncology, Biology and Physics; JCO, Journal of Clinical Oncology; JNCI, Journal of the National Cancer Institute; NEJM, New England Journal of Medicine; Semin Oncol, Seminars in Oncology. with ELCWP scale and from 0.21 to 0.38 with the Chalmers scale. We separated the trials according to quartiles of distribution of the citation factors and compared the quality scores among the four constructed groups. Analyses were performed according to each specific citation factor. No statistically significant difference was found for the ELCWP score between the groups constructed on the basis of IF99 (P = 0.10), PF (P = 0.07) or EF (P = 0.02). A significant quality difference was found among trials for which the values of IF Pub (P = ) were available, i.e. studies with the highest IFs also had the highest quality scores. No difference in the Chalmers score was noted when considering IF 99 (P = 0.06), PF (P = 0.03) and EF (P = 0.02); a significant quality difference was found when IF Pub (P < ) was considered. To avoid the possibility of non-representative samples, the same analysis was carried out after the exclusion of journals reporting less than four articles, leaving 11 journals for comparison. These journals were classified in descending order according to their methodological quality scores and their citation factor values for comparison (Table 3). The respective values of the correlation coefficient between the citations factors and the ELCWP and Chalmers scores were as follows: IF 99, 0.18 (P = 0.03) and 0.18 (P = 0.03); IF Pub, 0.26 (P = 0.004) and 0.29 (P = 0.001); PF, 0.13 (P = 0.12) and 0.17 (P = 0.04); and EF, 0.53 (P = 0.004) and 0.38 (P = 0.05), respectively. This absence of correlation is described in Table 3. The position of the journals when classified according to quality score is not equivalent with the citation factor ranking. Relationship between citation factors and quality score according to the geographical distribution of the authors The vast majority of authors originated from Europe (n = 100 articles) and America (n = 54). Asian, intercontinental co-

4 718 Table 4. Median values for citation factors according to the authors geographical distribution Europe America Asia Australia Intercontinental P IF 99 (n) (96) (50) (13) (5) (7) < IF Pub (n) (86) (43) (13) (5) (3) PF (n) (99) (50) (14) (5) (7) < IF, impact factor; IF Pub, value of IF at the date of publication of the article; IF99, the value of IF obtained from the last 1999 version available to the authors; PF, prestige factor. Table 5. Comparison of European and American median quality scores Studies a ELCWP Chalmers Europe America P Europe America P IF99 (n) 64.1% (96) 61.4% (50) % (96) 47.4% (50) IF Pub (n) 63.7% (86) 61.3% (43) % (86) 47.5% (46) PF (n) 63.7% (99) 61.4% (50) % (99) 47.4% (50) a Studies for which citation factor values were available. IF, impact factor; IF Pub, value of IF at the date of publication of the article; IF99, the value of IF obtained from the last 1999 version available to the authors; PF, prestige factor. Table 6. Comparison of European and American quality scores according to the median date of publication Median date of publication Less than Greater than or equal to Europe (n = 37) America (n = 34) Europe (n = 49) America (n = 9) IF Pub (median values) P < P = ELCWP (median) P = 0.42 P = % 58.2% 68.4% 66.7% Chalmers (median) P = 0.59 P = % 45.9% 60.4% 51.3% ELCWP, European Lung Cancer Working Party; IF, impact factor; IF Pub, value of IF at the date of publication of the article. operative and Australian trials accounted for 15, seven and five articles, respectively. The median value of citation factors according to the geographical distribution of the authors is presented in Table 4. We did not consider EF since only two American trials were published in European journals. We observed a statistically significant difference in citation factor values according to continent. We compared European and American authors for citation factors and quality scores. We found that American academics published statistically significantly more often than Europeans in journals with high citation factors (P <0.0001), whichever type of citation factor was used for comparison. Nevertheless, we detected no difference in methodological quality when the ELCWP scale was applied, and a borderline significant difference, favouring European authors, when the Chalmers scale was used (Table 5). We performed a second analysis, dividing the trials according to the median date of publication (Table 6) as European trials were published more recently than the American trials (P <0.0001). When considering trials published before the median date of publication, we observed that American authors published in journals with statistically significantly higher citation factors without methodological quality difference. The same results were found when studies published at or after the median date of publication were included in the analysis, except that the difference for citation factors, although present, was no longer statistically significant. The same analysis comparing American and all non-american authors gave the same conclusions (data not shown). Impact of trial significance on quality scores and citation factors Of the 181 articles, 73 were positive trials, where a statistically significant difference, according to the primary objective, was found among the treatment arms. Geographical distribution had

5 719 no effect on trial positivity; both when American and European trials (44.4% versus 36%; P = 0.31) or American and all non- American studies (44.4% versus 38.6%; P = 0.46) were compared. Positive trials were published significantly more often in journals with higher citation factors (Table 7). However, there was no quality difference between positive and negative studies (Table 8). When European trials were considered apart, the same observation was made (data not shown). When analysing the relationship between citation factors values and trial positivity for American authors, the difference was not significant (IF99, P = 0.03; PF, P = 0.05; IF Pub, P = 0.47). No difference in quality scores was detected. Discussion When analysing randomised trials conducted in the field of lung cancer research, we observe a weak, although statistically significant, relationship between citations factors (IF, PF or EF) and the methodological quality of articles, as assessed by two different quality scales. Some information can be drawn from this study. Trial quality cannot be evaluated using citation factors, which are frequently misused for academic work assessment. Journals with higher citation factors do not present articles of better quality, although trials with statistically significant results are published more often in those journals. There is a discrepancy between American and European authors, the former publishing their papers in journals with higher citation factors without concomitant better methodological quality. All these points, as well as Table 7. Comparison between positive and negative trials for citation factors Positive trials Negative trials P IF 99 a (n) (69) (102) IF Pub a (n) (59) (91) PF a (n) (71) (104) EF a (n) (9) (30) 0.41 a Expressed as median values. EF, Eurofactor; IF, impact factor; IF Pub, value of IF at the date of publication of the article; IF99, the value of IF obtained from the last 1999 version available to the authors; PF, prestige factor. potential methodological problems due to the design of this study, need to be successively discussed. The use of citation factors as a quality measure implies that the quality of published articles is a direct function of the value of a journal s citation factor. Although in many of the journals the selection of articles is part of a peer-reviewed process, it is dependent on two or three individuals whose decision is not only a function of the quality of the work, but may also be influenced by subjective arguments. For example, when the authors know the name of the reviewer, peer-review is of a higher quality than for unsigned forms [18]. Also, editors can interfere with this process for non-scientific reasons, so that controversial publications, without presumption of their quality, can be selected because they lead to a substantial increase in the number of citations. Some editorial policies can artificially increase the value of citation factors without having any positive impact on the quality of the publications; for example, by choosing well recognised authors that are more likely to be cited, by asking the authors to select, in their references, articles published in the journal (self-citation), by including more review articles or by modifying the number of published articles each year. Frequently, citation factors are misused for academic work assessment and determination of resource allocation. When citation factors are used in this way, it is assumed that they are a direct representation of the intrinsic quality of a trial and that articles edited in the same journal are of similar quality. Nevertheless, citation factors do not represent the individual citation rate of a paper given that only a few articles account for the majority of citations [2, 6], and uncited articles have the same IF as frequently cited ones. Moreover, in journals with a large number of articles in each issue, the quality of the articles is more variable and the frequency of citation is reduced, lowering the IF. Also, the value of citation factors is dependent on the research field, which requires similar units to be applied when IFs are used for comparison. Thus, not only does the number of articles in a journal and their source have to be taken into account when scientific work assessment is performed, but also the methodological quality and their scientific relevance have to be taken into consideration. Journals with higher citation factors do not systematically publish articles with the highest quality scores. There are some explanations for this, which reflects a potential publication bias. Table 8. Comparison between positive and negative trials according to quality scores ELCWP (median) Chalmers (median) Positive trials Negative trials P Positive trials Negative trials P IF % 61.5% % 49.4% 0.59 IF Pub 63.2% 61.5% % 51.0% 0.84 PF 63.6% 61.5% % 49.4% 0.55 EF 59.0% 58.3% % 45.6% 0.88 ELCWP, European Lung Cancer Working Party; IF, impact factor; IF Pub, value of IF at the date of publication of the article; IF99, the value of IF obtained from the last 1999 version available to the authors; PF, prestige factor.

6 720 We found that statistically significant trials are published more often in journals with high IFs, even though their methodological quality is equivalent to non-significant studies. It was previously reported that negative trials are published more often in the native authors language [19]. But, in the ISI database, there is a bias towards journals published in English; for example, only two German social science journals are included in the ISI database in comparison with 542 in a German database [6]. Therefore, non-significant trials can be published in journals that are not included in the citation factors analysis. For different reasons, authors may be attracted by journals with high citation factors, which can be more interested in publishing positive trials with a higher potential of citation than negative studies. Why do American authors choose to publish their work in journals with statistically higher citation factors than European or non-american authors? Neither significance of the trial results nor better methodological quality were sufficient explanations for such a difference; it is probably due to biases in the system of calculating citation factors. In the ISI database, more than half of all citations concern American scientists (language bias). Furthermore, American authors are particularly prone to cite each other, raising the citation rate of American science 30% above the world average [6]. Also, non-american authors can have difficulties in publishing trials in American journals, partly because English is not their native language. So, the apparent American leadership in this field is partly due to the high citation factors of the journals in which they choose to publish, to the self-citation rates and the national citation bias; however, it does not reflect the intrinsic quality of their scientific work, at least in the case of lung cancer research [5]. Citation factors have been created with a commercial purpose in mind and this needs to be preserved. They can be useful to authors when choosing which journal should publish their work, based on the distribution of the journal, assessed by the citation factors, and its ability to deliver the information. However, the reader must take note that publishing in journals with high citation factors is not a guarantee of scientific quality. When analysing a trial, one needs to take into consideration not only the journal where the study was published, but also its scientific relevance and its intrinsic quality, which are not measured by citation factors. There are some limitations when generalising the results of our study outside of the lung cancer field. To determine whether there is a correlation between citation factors and the methodological quality of articles, we considered articles reporting studies performed in a same clinical research subfield, avoiding the potential bias of comparisons between different scientific domains. In lung cancer treatment research, mainly small to moderate size randomised trials with equivocal results (level of evidence II) are available. The observed effects are small, needing meta-analyses to determine statistically significant differences between treatment arms. Extrapolation of the conclusions drawn from this study to large prospective randomised trials (level of evidence I), including thousands of patients, as is the case in the field of cardiac research, must be performed with circumspection. We tried to avoid other potential methodological biases. We considered only randomised studies to keep some homogeneity in trial design. All studies were extracted from previous systematic reviews performed by the ELCWP, for which articles were selected after an extensive review of the literature. All lung cancer randomised trials, published on the specified topics, in English, French or Dutch were eligible. Studies published in other languages are not included in this analysis, but the majority of articles published in non-english language journals are not included in the ISI database [5], so it seems unlikely that including them would change our results. Quality analysis was performed using two different quality scales, Chalmers and ELCWP. The use of two scales, read independently by different investigators with consensual agreement on each item in regular meetings (in which the majority of the authors needed to be present) and the multiplicity of the items allowed consistency in the quality assessment. The classical P value threshold of 0.05 was not used in this study because of the multiplicity of tests that were performed. We decided to apply a P value threshold of (0.05 divided by the number of statistical tests) to reduce the risk of observing by chance a falsely statistically significant result. Not only does the statistical significance need to be considered but also the clinical relevance when interpreting the results of statistical tests. The potential relationship between quality scores and citation factors was evaluated by the Spearman rank correlation coefficient. The association is better as this coefficient is closer to 1 (in absolute value). For example, although we observed a statistically significant correlation between PF and quality score, the correlation coefficient was far from 1. If an association exists, it is weak. In conclusion, although a statistically significant but weak correlation was found between citation factors and the methodological quality of randomised trials in the field of lung cancer research, there is no convincing evidence that citation factors can be used to assess the quality of published scientific work. The use of citation factors must be restricted to commercial purposes, such as the comparison of different journals. Their application for the assessment for academic work is limited by their incapacity to evaluate the quality and scientific relevance of a publication. Other quality indicators are needed which take into consideration not only the journal in which the article is published and the nationality of the authors, but also the content of the paper. Quality scales, such as we have developed, are possible alternatives for objectively assessing the scientific importance and the potential contribution of scientific articles, at least in literature dealing with lung cancer. Acknowledgements The authors would like to thank Mrs Bormans, librarian at the Institut Jules Bordet, for her important help in retrieving the impact factor values, and all the physicians participating to the meta-analyses: F. Branle, A. Burniat, C. Castaigne, L. Ghisdal, Y. Lalami, F. Lemaitre, I. Louviaux, S. Luce, B. Martin, C. Sotiriou, E. Steels, F. Vallot, J. M. Verdebout and Ph. Vermylen.

7 721 References 1. Garfield E. How can impact factors be improved? BMJ 1996; 313: Similowski T, Derenne JP. Bibliometry of biomedical periodicals. Rev Mal Respir 1995; 12: Cole JR, Eales NB. The history of comparative anatomy. Sci Prog 1917; 11: Gross PLK, Gross EM. College libraries and chemical education. Science 1927; 66: Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997; 314: Seglen PO. Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthop Scand 1998; 69: Luce S, Paesmans M, Berghmans T et al. Revue critique des études randomisées évaluant le rôle de la radiothérapie thoracique adjuvante à la chimiothérapie dans le traitement du cancer bronchique à petites cellules au stade limité. Rev Mal Respir 1998; 15: Sculier JP, Berghmans T, Castaigne C et al. Maintenance chemotherapy for small cell lung cancer: a critical review of the literature. Lung Cancer 1998; 19: Sculier JP, Berghmans T, Castaigne C et al. Best supportive care or chemotherapy for stage IV non-small cell lung cancer. Van Houtte P, Klastersky J, Rocmans P (eds): Progress and Perspectives in Lung Cancer. Berlin, Germany: Springer 1999; Mascaux C, Paesmans M, Berghmans T et al. A systematic review of the role of etoposide and cisplatin in the chemotherapy of small cell lung cancer with methodology assessment and meta-analysis. Lung Cancer 2000; 30: Meert AP, Paesmans M, Berghmans T et al. Prophylactic cranial irradiation in small cell lung cancer: a systematic review of the literature with meta-analysis. BMC Cancer 2001; 1: Sculier JP, Berghmans T, Paesmans M et al. La place de la chimiothérapie dans le traitement des cancers bronchiques non à petites cellules, non métastatiques. Rev Med Brux 2001; 22: Sculier JP, Ghisdal L, Berghmans T et al. The role of mitomycin in the treatment of non-small cell lung cancer: a systematic review with metaanalysis of the literature. Br J Cancer 2001; 84: Berghmans T, Paesmans M, Lafitte JJ et al. Role of granulocyte and granulocyte macrophage colony stimulating factors in the treatment of small-cell lung cancer: a systematic review of the literature with methodological assessment and meta-analysis. Lung Cancer 2002; 37: Meert AP, Berghmans T, Lafitte JJ et al. Which progress have the new agents brought for chemotherapy of advanced non-small cell lung cancer? Eur Respir Rev 2002; Chalmers TC, Smith H Jr, Blackburn B et al. A method for assessing the quality of a randomized control trial. Control Clin Trials 1981; 2: Luukkonen T. Bibliometrics and evaluation of research performance. Ann Med 1990; 22: Walsh E, Rooney M, Appleby L, Wilkinson G. Open peer review: a randomised controlled trial. Br J Psychiatry 2000; 176: Egger M, Zellweger-Zahner T, Schneider M et al. Language bias in randomised controlled trials published in English and German. Lancet 1997; 350: Appendix A. The ELCWP quality score for phase III studies A. Study protocol description 1. Description of the selection criteria for the study 2. Randomisation method 3. Treatment description 4. Work-ups 5. Evaluation criteria description 6. Statistical methods and trial objectives description B. Study analysis report 1. Analysis timing 2. Patients characteristics 3. Survival data and analysis 4. Antitumoural response data and analysis 5. Toxicity data and analysis 6. Prognostic factors for survival 7. Discussion Appendix B. Chalmers scoring system A. Internal validity items 1. Randomised methods 2. Analysis of efficacy of randomisation 3. Blinding in evaluation of response to treatment 4. Blinding in evaluation of interim results 5. Compliance 6. Follow-up schedules 7. A priori estimate of sample size 8. Withdrawal 9. Analysis of withdrawals 10. Response evaluation 11. A posteriori estimate of study power B. External validity items 1. Patients characteristics 2. List of eligible but not enrolled patients 3. Description of the therapeutic regimen 4. Timing of events 5. Discussion of side effects