Methods in Research on Research The Peer Review Process. Why Evidence Based Practices Are Needed? Isabelle Boutron METHODS team Research Centre of Epidemiology Biostatistics Sorbonne Paris Cité Paris Descartes University
Acknowledgements MiRoR project Joint doctoral training program, to train 15 PhD students in Methods in Research on Research in the field of clinical research (http://mirorejd.eu) funded by Marie Skłodowska-Curie Actions. 7 European Universities and 10 International Partners Projects -Planning -Conduct -Reporting -Peer review
The peer review system Central to the scientific community Gatekeeper of the scientific publications To improve the quality of manuscripts The number of journals and manuscripts is increasing 8 million researchers publishing 2.5 million articles in 28,000 peerreviewed English-language journals/year The annual revenue is $10 billion A system relying mainly on work performed voluntarily by academic researchers International Association of Scientific, Technical and Medical Publishers STM Report March 2015 3
The peer review system 2.7 billion US dollars / year* *Look H, Summary report: Rightscom Ltd for JISC Collections; 2010
The performance of the system is questioned. «material often obtained in the most rigourous kind of biomedical experimentation is rated as publishable or not by a system that has rarely been subjected to any analysis, let alone one that is rigourous» Franz Ingelfinger, editor of the NEJM (1967 to 1976) 5
Level of evidence of peer review At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. Published in Issue 2, 2008 6
The performance of the system is questioned. Submission of 304 versions of a fake paper with fatal flaw to open-access journals. More than half of the journals accepted the paper, failing to notice its fatal flaws. For DOAJ publishers, that completed the review process, 45% accepted the paper. 7
Impact of peer review on reporting of RCTs Comparison of the submitted and accepted version of manuscripts of RCTs published in BMC journals Changes were limited A median of 11% (range 1-60%) words deleted and 20% added (range 2-88%). Peer reviewers often fail to detect important deficiencies in the reporting Peer reviewers requested relatively few changes for reporting Hopewell S, BMJ, 2015 8
Detection of selective reporting of outcomes 34% examined information registered on a trial registry. 9
Can peer reviewers reduce spin? Assessment of the manuscript submitted, peer reviewers comments, and final manuscript of non-randomized studies assessing a therapeutic intervention published in BMC Series journals (2011-2013). 55% of submitted manuscripts, peer reviewers identified at least one example of spin Of the spin identified by peer reviewers 67% were completely deleted, 16% partially deleted 17% not removed in the final published article. For 15%, peer reviewers requested adding some spin Lazarus, J Clin Epidemiol. 2016 10
Level of spin before and after peer review in the abstract conclusion Before peer-review After peer-review High N=66 N=61 Moderate Low No N=24 N=24 N=10 N=17 N=28 N=26 76% Peer reviewers failed to identify spin in abstract conclusions Lazarus, J Clin Epidemiol. 2016 11
Two versions of a well-designed randomized controlled trial that differed only in the direction of the finding of the principal study end point 238 reviewers at 2 journals (assigned at random) were more likely to recommend the positive version of the test manuscript for publication than the no-difference version (97% vs 80%, P<0.001) detected more errors in the no-difference version than in the positive version (mean 0.85 vs 0.41, P<0.001) awarded higher methods scores to the positive manuscript than to the (identical) no-difference manuscript (8.24 vs 7.53, P=0.005) 12
Influence of authors prestige 119 reviewers were randomized to assess a fabricated manuscript with the prestigious authors names and institutions masked or visible Reviewers were more likely to recommend acceptance when the prestigious authors names and institutions were visible than when they were redacted 87% vs 68%; RR, 1.28 [95% CI, 1.06-1.39], P =.02 They gave higher ratings for the methods. 13
The process is not clearly defined and standardized Who are the reviewers? Who can be a reviewer? How are they identified? What are the core competencies? Should reviewers be trained and how? What documents should be peer-reviewed? Manuscript Appendices CONSORT checklist Registry Protocol, SAP CSR? Raw data? 14
The tasks expected from a peer-reviewer are not realistic More than 200 different tasks identified The tasks involved different level of expertise and different backgrounds Statistical and methodologic expertise Content expertise (novelty, relevance, external validity etc) None Verification (adherence to guidelines, consistency with registries) Formatting 15
Multiple tasks asked to reviewers (important tasks for reviewers are not congruent with important tasks for editors) Chauvin, BMC Medicine, 2016 The most important task for peer reviewers (evaluating the risk of bias) was clearly requested by 5 % of editors. The task most frequently requested by editors (provide recommendations for publication), was rated in the first tertile only by 21 % of all participants.
17
Several interventions are implemented or proposed to improve the system 18
19
What is the evidence? A Systematic Review and Meta-analysis of Interventions to Improve the Quality of Peer Review Only 22 reports of RCTs Only 7 were published over the past 10 years Interventions assessed Blinding Open peer review Training Use of checklist Adding experts Most are performed in one specific journal Low methodologic quality Bruce, BMC Med, 2016
RCTs and Peer review There are no barrier to the conduct of RCTs Large sample size is available (manuscripts, reviewers) Consent with an opt out approach facilitate recruitment Randomisation, allocation concealment are easy to perform Some outcomes are routinely collected Blinding outcome assessment is easily feasible 21
RCTs and Peer review : Methodological research is needed What is the best study design for assessing interventions to improve the peer review process? Design: RCTs, cluster RCTs, pairwise comparisons, Stepped wedge cluster RCT of journals, time series analysis etc Unit of randomisation Manuscripts Reviewers Type of manuscript (real, fabricated) What will be the outcome? Quality of the peer review report (editor s subjective assessment, validated scales to assess the quality of the peer review report Quality of the final manuscript (how is high quality defined, who is to decide?) Need of CORE outcome set Ethical issues Impact on the final decision
Modeling (Agent-based model) Kovanis Plos One 2016 Kovanis Scientometrics. 2016 To approach the complexity of the scientific publication system and compare different systems Explore sustainability of the system Compare different systems (cascade etc)
A call to substantially increase the amount of research conducted by journals in journalology and meta-research. Participate in Randomized Controlled Trials (eg about peer review improvement) Share their empirical data to help modeling Share manuscripts submitted, reviews, revised manuscripts to develop qualitative and quantitative research on the impact of peer review 24
Conclusions So peer review is a flawed process, full of easily identified defects with little evidence that it works. Nevertheless, it is likely to remain central to science and journals because there is no obvious alternative. The most important question with peer review is not whether to abandon it, but how to improve it. (R. Smith) We need rigorous studies to tell us the pros and cons of these approaches. (D Rennie) 25