Editorial. An audit of the editorial process and peer review in the journal Clinical Rehabilitation. Introduction

Transcription

1 Clinical Rehabilitation 2004; 18: Editorial An audit of the editorial process and peer review in the journal Clinical Rehabilitation Objective: To investigate the editorial process on papers submitted to a scienti c journal. Design: Descriptive and correlational analysis. Setting: A single specialist rehabilitation journal from mid-1999 to mid Subjects: The Editor, peer reviewers and authors. Interventions: Submitted papers were initially screened, the majority were subject to review, and a decision was made on whether or not to publish. Reviewers scored papers on 11 items using a 0 (bad) 10 (good) scale. Measures: Time delays and rating of each paper on a pro-forma. Results: The number of papers submitted each year increased from 136 to 209. Between 19% and 31% were rejected without review and 17 24% were rejected later. The proportion accepted dropped from 64% to 47%. The median delay between arrival and rst author contact in papers subject to review was stable between 67 and 76 days. Agreement between reviewers was low with an intraclass correlation coef cient (ICC) varying between 0.12 and 0.27 and disagreement of 3 or more points in 32 51%. The main factor associated with eventual acceptance was the sum of the overall recommendation score given by the two reviewers, but 20% of the variance in nal decision was unexplained by the reviewers scoring. The number of randomized trials published increased from 5 in 1996 to 21 in Conclusions: The in uence of the Editor on nal decisions remains signi cant, particularly for papers on the margin ; decisions on publication in this journal are guided by but not determined by reviewers. Introduction Many authors nd the process of editorial handling of their submitted papers arbitrary and mysterious. Open reviewing on the web may slightly demystify the process eventually, but only to those who have submitted papers. This editorial throws a little light on the process undertaken in this journal, limited by the restricted data collected. A scienti c journal performs many functions. From the author s perspective they include: being one way of publicizing research; improving their curriculum vitae; increasing their departmental status; and becoming known to other researchers. From the publisher s perspective the primary function is to generate pro t, or at least no loss! The journal s Editor may enjoy the power and prestige the job brings (so they say!). The wider scienti c community and general public may use it as a source of information which is assumed (with little justi cation) to be valid. This wide variety of functions leads different people to have different expectations of the Arnold / cr726ed

2 118 Editorial editorial process. The Editor and the publishers wish to select papers that are in line with the journal s area of interest, that are scienti cally valid and that will increase the prestige (and hence readership and sales) of the journal. They also hope that the review process will improve articles accepted. The author wants their article accepted, and if not hopes for some constructive feedback that might help in submitting the paper elsewhere. The author also wants a quick decision. Many readers and some authors think that the review process guarantees the truth of the article; they believe that all valid articles are accepted and that all articles accepted are valid. In reality the peer review process is awed, 1 with marked disagreement among reviewers and many well-known instances of awed or even fraudulent research passing peer review, but it is likely that peer review does at least help improve the quality of presentation of articles. Moreover, even if peer review were more consistent, a selection process would still be needed because many papers of equal quality are submitted and usually only a proportion can be published. This editorial informs readers and authors about the editorial processes used by this journal. Method All papers formally submitted to Clinical Rehabilitation are registered on a database recording items such as title, authors, contact information, dates of various events, and names of reviewers chosen. The review process has been discussed before. 2 In brief, the Editor reads every paper, and rejects a proportion immediately on various grounds such as being inappropriate for this journal, scienti cally extremely awed, excessively long, or obviously not suf ciently important. Papers are not rejected simply because they are too long, or badly written, or do not follow the instructions to given authors (see The majority of papers are then sent to two reviewers who are given guidance ( and a score sheet which they may complete (see Appendix). This score sheet asks for an overall recommendation and for scoring on different aspects of the article. When returned, the Editor reads the paper again, considers the comments made by the reviewers including the scores given on each part, and makes a decision. The Editor writes to the author accepting or rejecting the paper, explaining why the paper has not been accepted or what changes are needed to make it acceptable. The data presented here primarily relate to papers submitted between 1 July 1999 and 30 June Furthermore, the data set is not complete on every paper: Some papers are returned by the Editor without any external review; occasionally items of data are missing; and not all reviewers use the scoring sheets (which is no problem to the Editor!). The data relating to outcome and delays in the process are analysed descriptively. The data on the scores given by two reviewers was analysed descriptively and inter-rater reliability was also analysed using SPSS 11.5 to calculate the intraclass correlation coef cient (ICC) with a random effects model. The relationship between reviewer gender and profession and scores given and the nal decision made was analysed using logistic regression analysis. The relationship between the scores by an individual reviewer was analysed using correlation coef cients. Results Table 1 shows the process followed for 716 papers submitted between 1 July 1999 and 30 June 2003, by yearly groups (1 July to 30 June). The increase in numbers submitted is obvious. More importantly, in the absolute number accepted rose considerably, leading to a backlog which is only now being cleared. One hundred and forty-seven (20%) were rejected without review. Forty-three papers (6%) are still undecided. For the 530 (74%) papers sent out for review and with a decision made, half of all authors had a rst response from the Editor within 10 or 11 weeks, but for some there was a longer delay. The speed of response has remained roughly constant. Half of all authors submit the nal, accepted version of their paper within 8 10 weeks, though again there can be much longer delays. Table 2 shows that the number of evaluative studies, primarily randomized controlled trials,

3 Editorial 119 Table 1 Flow of papers through editorial process ; 1 July to 30 June Number submitted Immediate rejection 26 (19%) 27 (15%) 30 (16%) 64 (31%) Later rejection 23 (17%) 45 (24%) 66 (35%) 39 (19%) Accepted 87 (64%) 111 (60%) 87 (47%) 68 (33%) Still pending (2%) 38 (18%) Delay to reply (days) n = 110 n = 156 n = 150 n = 114 Minimum % Median % Maximum Mean (SD) 81.3 (48.0) (56.6) 89.8 (60.1) 75.0 (34.3) Reply to acceptance (days) n = 87 n = 104 n = 80 n = 64 Minimum % Median % Maximum Mean (SD) (101.7) (100.4) 79.1 (57.5) 68.2 (46.7) Table 2 Evaluative studies published Year RCTs RCTs CCTs CCTs full abstract full abstract RCTs full, randomized controlled trials, full paper published; RCTs abstract, randomized controlled trials, only abstract published; CCTs full, controlled clinical trials, full paper published; CCTs abstract, controlled clinical trials, only abstract published. has increased almost every year since the journal started. (Earlier data were published in 1995.) 3 During the period under review, we had scores from both reviewers for 193 (36%) of the 530 papers where a decision was made after review, and about 60% of these were accepted. Some of the remaining 337 papers were received before scores were requested routinely, but in the majority only one reviewer used the score sheet. Only papers where both reviewers gave scores have been used in subsequent analyses. Table 3 shows that the level of agreement between reviewers is low. The mean disagreement between reviewers was 0 (as expected) with a standard deviation of 3. Only 25 papers (13%) displayed a level of disagreement greater than ± 1 standard deviation. Table 4 shows that generally the summed score of the two reviewers across all domains signi cantly (p <0.01) differentiated those papers accepted from those rejected. The differences between reviewers did not relate to acceptance or rejection. Given the score range is 0 20 for the sum of both reviewers, those papers accepted typically scored 12 or more on any summed characteristic; those rejected 9 or less. However, the summed total score (overall judgement by the reviewer) appeared to be the key characteristic here, as judged by the magnitude of t in the t-test and the visual separation of the two groups (Figure 1). Given this, it might follow that a pass mark of 10 out of 20 would appear to be in operation. Such a mark gives 88% sensitivity for acceptance, with 71% speci city (i.e., correctly predicts 71% of those that were rejected); only 20% of papers are misclassi ed using this single indicator. A logistic regression model con rms that this

4 120 Editorial Table 3 Score difference between two reviewers in 193 papers Question Max. diff. Differences ICC % 0 points % 1 or 2 points % >2 points Overall Readability Abstract Introduction Method description Method measures Method design Method analysis Results text presentation Results tables/ gures Discussion weaknesses Discussion extrapolation ICC, intraclass correlation coef cient; Max. diff., maximum difference between the scores of the two reviewers. Figure 1 Summed score of overall advice from reviewers for those accepted and rejected. summed total score is a signi cant (and the only) predictor of acceptance, with a predictive model accuracy of 81%, and the odds of acceptance increase by 1.77 times for each total point (95% con dence interval (CI): ), with a Nagelkerke pseudo R 2 value of However, it does appear that other characteristics in uence how a reviewer judges the overall score. All characteristics, for example, the score for introduction, methods and analysis, correlate highly with the judgement of the reviewers as expressed by their overall total score (correlation range ; Spearman rank correlation). Furthermore, all individual characteristics discriminate signi cantly across the pass fail cut point of the total score (Mann Whitney; p<0.01).

5 Editorial 121 Table 4 Mean (SD) of the summed scores of both reviewers on papers accepted and rejected Question Rejected Accepted Overall 7.2 (2.8) 11.8 (2.8) Readability 9.9 (3.3) 13.5 (3.0) Abstract 10.2 (2.9) 13.6 (3.1) Introduction 9.5 (2.8) 13.0 (3.1) Method description 9.0 (3.4) 12.8 (3.0) Method measures 8.9 (3.6) 12.6 (3.6) Method design 8.1 (3.4) 12.0 (3.4) Method analysis 8.0 (3.1) 11.7 (2.8) Results text presentation 8.5 (3.2) 12.3 (3.2) Results tables/ gures 7.9 (3.5) 11.9 (3.4) Discussion weaknesses 7.2 (3.3) 10.7 (3.6) Discussion extrapolation 7.1 (2.6) 11.0 (3.4) All t-tests p < An exploratory regression analysis with summed total score as dependent variable showed that many individual characteristics contributed independently to the total score. Design, results and introduction showed the highest beta coef cients. Scores were not related to the gender or profession of the reviewer. In practice, only one in ve papers are not predicted by this simple summed total score. Presumably these are where the Editor s judgement is most exercised. Overall it is possible to classify papers into true positives (both reviewers summed scores = 10+ and editorial decision was acceptance; 46%), true negatives (both reject 34%), false positives (reviewers scores imply acceptance but Editor rejects 6%) and false negatives (reviewers score implies rejection but Editor accepts 14%). Looking at the summed total scores across these categories it becomes clear that the false negatives (those below the pass score for the reviewers, but accepted by the Editor) are at the margins of acceptance, and generally have a higher score than those rejected (Figure 2). On the other hand, those rejected by the Editor have scores generally in the lower quartile of the acceptance range. Discussion This is the rst time that this journal has published data on the process of handling submitted articles. The most interesting nding is that, Figure 2 Sum score on overall advice according to status of acceptance. Both reject : sum score 9/20 or less, and Editor rejects (n = 66); Both accept : sum score 10/20 or greater and Editor accepts (n = 89); Editor a rev rej : sum score 9/20 or less, but Editor accepts (n = 27); Editor rej rev accep : Sum score 10/20 or greater, but Editor rejects (n = 11).

6 122 Editorial Figure 3 Difference in total score between reviewers. despite a low degree of agreement between reviewers, their combined recommendation does predict acceptance or rejection in the majority (80%) of cases. The analysis also suggests that the Editor usually makes personal choices when a paper is on the margin. Finally, the data con- rm that an increasing number of papers are being submitted, but an increasing number are also being rejected without review. The data on numbers submitted and the outcome is reasonably robust. Every paper formally submitted is registered, and dates are known and entered for each action (arrival, rst reply to author, nal acceptance). However, over the last few years authors have sent draft versions of papers for an initial opinion by . These are not registered. The Editor usually gives an initial response (without prejudice) and those judged unlikely to be accepted are often not submitted formally. Consequently the gures are starting to underestimate the total number submitted, and hence to overestimate the proportion accepted. The data on timings are reasonably secure. Papers may wait a few days before actual registration (even editors have holidays!). Furthermore, from the author s point of view, there are additional delays in the postal system. The time from registration to writing to the authors con- tains several activities. The paper must be sent (by post) to reviewers, often not in the UK. The reviewer must read and consider the paper, write a reply and post it back. When both are back the Editor has to read the paper, and to read the reviews, and then to make a decision. It is likely that the time devoted to post and thought amounts to at least two weeks, which means that many reviewers are providing prompt replies within the four weeks requested. A few reviewers never respond at all, and a small number only respond after a delay (but often with good reason; reviewers also have holidays and other work). The data on the scores given by reviewers are selected data, in that they only refer to papers where both reviewers replied and used the score sheet. The factors underlying this selection are unknown. Furthermore, the analysis of the relationship between the reviewers scores (which includes an overall recommendation on acceptance) and the Editor s nal decision is clearly speci c to this Editor and journal. Nonetheless the ndings are of interest. We con rm other research 1 in showing a relatively low rate of agreement between reviewers on most matters. Despite this, the summed total score given by reviewers was strongly associated with the actual decision taken by the Editor,

7 Editorial 123 which is surprising given the unreliability of the judgements. However, it would appear that the various facets of quality of the paper as judged by the reviewers, such as the Introduction or Methods, do in uence this total score. These aspects also signi cantly discriminated for acceptance, but a logistic regression model supported the evidence for the sensitivity of the simple summed total (overall judgement) score. Where the Editor accepted papers which fell below the obvious pass mark (10+), most seemed to be just at the margins of failure. The Editor rejected relatively few papers which the reviewers thought were good. There was no evidence to suggest that such decisions were associated with disagreement by the reviewers, rather the opposite, with only one in 12 such decisions falling in the group where Editor and pass mark disagreed. Of course, some decisions will be affected by matters other than the pure quality of the paper. For example, a paper may be rejected if a similar one is already awaiting publication. Not only may the reviewers who provide scores be a biased sample, but also it is impossible to know to what extent the Editor simply accepted the general thrust of the combined reports of the reviewers. The Editor believes that he makes a decision independently before reading the reviews in detail, but maybe he is misguided in this belief. In other words the Editor may have subconsciously added the scores and come to a decision. However, the Editor does need to select between papers that are of similar quality because more papers of quality are submitted than can be published. This selection is dif cult, and ultimately capricious. The factors that the Editor believes that he takes into account are: Relevance to the readership This is not easy, because we have little information about the actual readers, and certainly reviewers also disagree markedly on whether some papers are relevant to the readership. It is probable that the readership covers a vast range of interests. The main questions the Editor asks are Does it potentially help a clinician who sees patients regularly? and Is it interesting?. Type of study Articles that provide evaluation on treatments and systematic reviews are given highest priority. Any randomized controlled study of reasonable quality will be accepted, and almost all systematic reviews unless a similar review has been accepted or published recently. Balance of papers The journal has (we think) a wide and international readership, and so the journal tries to publish articles covering a reasonable spread of topics. Sometimes papers will be speci cally accepted or rejected in the light of other papers being considered or published. The dif culty in selecting papers was recently illustrated. Members of the editorial committee were asked to indicate in one issue three papers they would have de nitely chosen, and three they would not have chosen. As with peer review, there was virtually no agreement! At present the Editor checks his decision by writing a letter to each author explaining the decision. This ensures careful thought, and hopefully assures the author that a considered decision has been made. This editorial has described and analysed the process of selecting papers for publication in Clinical Rehabilitation. The results surprised the Editor by showing a reasonable process that seems to re ect the opinions of the reviewers. Authors and readers may have different views, and any comments or suggestions would be most welcome. Acknowledgements We would like to thank all reviewers for their hard work, and especially all reviewers who completed the score sheets. All reviewers are named in the last issue published each year. Derick Wade and Alan Tennant References 1 Jefferson TO, Alderson P, Davidoff F, Wager E. Editorial peer-review for improving the quality of reports of biomedical studies (Cochrane Methodology Review). In: The Cochrane Library, Issue 3. Oxford: Update Software, Wade DT. The review process. Clin Rehabil 1998; 12: Wade DT. Randomised and controlled clinical trials in Clinical Rehabilitation. Clin Rehabil 1995; 9:

8 124 Editorial Appendix The scoring questions This sheet is aimed to help you, the reviewer, to undertake a fair, complete and yet ef cient review and is to help me, the Editor, to decide on the fate of the paper. It is not intended to replace a free-text commentary which is always welcome, but it is intended to ensure some consistency. Any suggestions on it are most welcome. Please rate the paper on each topic on the scale given. Overall advice: Accept (minor or no modi cation) Reject (no merit whatsoever) Readability and understandability Good, minor editing only Appalling, major rewrite needed Abstract A full and fair summary Inaccurate &/or incomplete Introduction A fair review, justi es study Biased, incomplete, does not set the scene Methods: description Good description, easily understood and complete Incomplete &/or badly described Methods: measures used Appropriate, good measures used Inappropriate or no measures used Methods: the design of the study The design is good The design is poor Methods: analytical approach and statistical methods Correct, and simplest possible for study Incorrect or unnecessarily complex Results: presentation in text Well presented and complete Muddled or incomplete presentation Results: presentation in tables & gures Helpful, easily understood, complete Muddled, incomplete, wasteful Discussion: covers weaknesses of study Fair discussion, placing in context Fails to mention any weaknesses Discussion: extrapolation and speculation Reasonable conclusions, justi ed from data Conclusions not justi ed at all