General reviewer guidelines can be found here:

Transcription

1 Guidelines fr reviewers Registered Reprts are a frm f empirical article in which the methds and prpsed analyses are pre-registered and reviewed prir t research being cnducted. This frmat f article seeks t neutralise a variety f inapprpriate research practices, including inadequate statistical pwer, selective reprting f results, undisclsed analytic flexibility, and publicatin bias (Chambers, 2013). General reviewer guidelines can be fund here: The review prcess fr Registered Reprts is divided int tw stages. In Stage 1, reviewers assess study prpsals befre data is cllected. In Stage 2, reviewers cnsider the full study, including results and interpretatin. Stage 1 manuscripts will include nly an Intrductin, Methds (including prpsed analyses), and Pilt Data (where applicable). In cnsidering papers at Stage 1, reviewers will be asked t assess: The significance f the research questin(s). The lgic, ratinale, and plausibility f the prpsed hyptheses. The sundness and feasibility f the methdlgy and analysis pipeline. Whether the clarity and degree f methdlgical detail wuld be sufficient t replicate exactly the prpsed experimental prcedures and analysis apprach. Whether the authrs prvide a sufficiently clear and detailed descriptin f the methds t prevent undisclsed flexibility in the experimental prcedures r analysis pipeline. Whether the authrs have cnsidered sufficient utcme-neutral cnditins (e.g. absence f flr r ceiling effects; psitive cntrls) fr ensuring that the results btained are able t test the stated hyptheses. Fllwing Stage 1 peer review, manuscripts will be accepted, ffered the pprtunity t revise, r rejected utright. Manuscripts that pass peer review will be issued an in principle acceptance (IPA), indicating that the article will be published pending successful cmpletin f the study accrding t the exact methds and analytic prcedures utlined, as well as a defensible and evidence-bund interpretatin f the results. Fllwing cmpletin f the study, authrs will cmplete the manuscript, including Results and Discussin sectins. These Stage 2 manuscripts will mre clsely resemble a regular article frmat. The manuscript will then be returned t the reviewers, wh will be asked t appraise: Whether the data are able t test the authrs prpsed hyptheses by passing the apprved utcme-neutral criteria (such as absence f flr and ceiling effects) Whether any changes t the Intrductin are reasnable and d nt alter the ratinale r hyptheses Whether the authrs adhered precisely t the registered experimental prcedures Whether any unregistered pst hc analyses added by the authrs are justified, methdlgically sund, and infrmative Whether the authrs cnclusins are justified given the data Please nte that editrial decisins will nt be based n the perceived imprtance, nvelty, r clarity f the results. 1

2 Guidelines fr authrs Registered Reprts are a frm f empirical article in which the methds and prpsed analyses are pre-registered and reviewed prir t research being cnducted. The crnerstne f this article frmat is that a significant part f the manuscript will be assessed prir t data cllectin. Initial submissins will include a descriptin f the key research questin and backgrund literature, hyptheses, experimental prcedures, analysis pipeline, a statistical pwer analysis (r Bayesian equivalent), and pilt data (where applicable). Initial submissins will be triaged by an editrial team fr scientific significance. Thse that pass triage will then be sent fr in-depth peer review (Stage 1). Fllwing review, the article will then be either rejected r accepted in principle fr publicatin. Fllwing in principle acceptance (IPA), the authrs will then prceed t cnduct the study, adhering exactly t the peer-reviewed prcedures. When the study is cmplete the authrs will submit their finalised manuscript fr re-review (Stage 2) and will uplad their raw data and labratry lg t a free and publicly accessible file-sharing service. Pending quality checks and a sensible interpretatin f the findings, the manuscript will be published regardless f the results. The review prcess fr Registered Reprts 2

3 Stage 1: Initial manuscript submissin and review Due t the high vlume f submissins, the editrial team will select nly the mst scientifically prmising manuscripts fr in-depth peer review. Stage 1 submissins shuld include the manuscript (details belw) and a brief cver letter. Authrs are welcme t submit presubmissin enquires fr advice n the likely suitability f a study as a Registered Reprt. Hwever, please nte that the editrial bard will nt agree t send manuscripts fr in-depth review until a cmplete Stage 1 submissin has been cnsidered. The cver letter shuld include: A brief scientific case fr cnsideratin. Authrs are encuraged t refer t the likely replicatin value f the research (Nsek et al., 2012). High-value replicatin studies are welcme and will be treated with equal pririty t nvel studies. A statement cnfirming that all necessary supprt (e.g. funding, facilities) and apprvals (e.g. ethics) are in place fr the prpsed research. Nte that manuscripts will be cnsidered nly fr studies that are able t cmmence immediately. An anticipated timeline fr cmpleting the study if the initial submissin is accepted. A statement cnfirming that the authrs agree t share their raw data and labratry lg fr all published results. A statement cnfirming that if the authrs later retract their paper, they agree t the Jurnal publishing a shrt summary f the pre-registered study under a sectin Retracted Registratins. Manuscript preparatin guidelines Stage 1 Fr general guidelines fr manuscript preparatin see: Initial Stage 1 submissins shuld include the fllwing sectins: Intrductin A review f the relevant literature that mtivates the research questin and a full descriptin f the experimental aims and hyptheses. Methds Full descriptin f prpsed sample characteristics, including criteria fr subject inclusin and exclusin, and detailed descriptin f prcedures fr defining utliers. Prcedures fr bjectively defining exclusin criteria due t technical errrs (e.g. defining what cunts as excessive head mvement during fmri) r fr any ther reasns must be dcumented, including details f hw and under what cnditins subjects wuld be replaced. A descriptin f experimental prcedures in sufficient detail t allw anther researcher t repeat the methdlgy exactly, withut requiring further infrmatin. These prcedures must be adhered t exactly in the subsequent experiments r any Stage 2 manuscript will be summarily rejected. Please nte that reviewers at Stage 1 will be asked t specifically cnsider whether the stated experimental prcedures cntain sufficient detail t prevent undisclsed prcedural flexibility. Prpsed analysis pipeline, including all preprcessing steps, and a precise descriptin f all planned analyses, including apprpriate crrectin fr multiple cmparisns. Any cvariates r regressrs must be stated. Cnsistent with the guidelines f Simmns et al. (2011), prpsed analyses invlving cvariates must be reprted with and withut the cvariate(s) included. Neurimaging studies must dcument in advance, and in precise detail, the cmplete analysis pipeline frm raw data nwards. Where analysis decisins are cntingent n the utcme f prir analyses, these cntingencies must be specified and adhered t. Only pre-planned analyses can be reprted in the main Results sectin f Stage 2 submissins. Hwever, unplanned pst hc analyses will be admissible in a separate sectin f the Results (see belw). 3

4 Pilt Data Studies invlving Neyman-Pearsn inference must include a statistical pwer analysis. Estimated effect sizes shuld be justified with reference t the existing literature. T accunt fr existing publicatin bias, which leads t verestimatin f true effect sizes (Hedges and Vevea, 1996; Lane and Dunlap, 1978), pwer analysis must be based n the lwest available r meaningful estimate f the effect size. The a priri pwer (1 ) must be 0.9 r higher fr all prpsed statistical tests. In the case f highly uncertain effect sizes, a variable sample size and interim data analysis will be permissible but with inspectin pints stated in advance, apprpriate Type I errr crrectin fr peeking emplyed (Strb, 2006), and a final stpping rule fr data cllectin utlined. Fr studies invlving analyses with Bayes Factrs, the predictins f the thery must be specified s that a Bayes factr can be calculated. Authrs shuld indicate what distributin will be used t represent the predictins f the thery and hw its parameters will be specified. Fr example, will yu use a unifrm up t sme specified maximum, r a nrmal/half-nrmal t represent a likely effect size (Dienes, 2011), r a JZS/Cauchy with a specified scaling cnstant (Ruder et al., 2009)? The parameters need nt be stated in advance, but where unstated, authrs must indicate what aspect f data will be used t set thse parameters. Fr inference by Bayes factrs, authrs shuld guarantee testing participants until the Bayes factr is either mre than 3 and less than 0.33 t ensure clear cnclusins. When using Bayes factrs, adjustments fr multiple cmparisns are nt required. Fr advice n Bayes factrs, prspective authrs are invited t cntact the editrial bard: crtex@ed.ac.uk Full descriptins must be prvided f any utcme-neutral criteria that are required fr successful testing f the stated hyptheses. Such reality checks might include the absence f flr r ceiling effects, r psitive cntrls. Please nte that reviewers will be asked t judge whether the manuscript includes sufficient specificatin f reality checks. Timeline fr cmpletin f the study and prpsed resubmissin date if registratin review is successful. Extensins t this deadline can be negtiated with the actin editr. Optinal. Can be included t establish reality checks, effect size estimatins, feasibility, r prf f principle. Any pilt experiments will be published with the final versin f the manuscript and will be clearly distinguished frm data btained fr the main experiment(s). Stage 1 submissins that are judged by the editrial bard t be f sufficient quality and scientific significance will be sent fr peer review. In cnsidering papers at the registratin stage, reviewers will be asked t assess: The significance f the research questin(s). The lgic, ratinale, and plausibility f the prpsed hyptheses. The sundness and feasibility f the methdlgy and analysis pipeline. Whether the clarity and degree f methdlgical detail wuld be sufficient t replicate exactly the prpsed experimental prcedures and analysis apprach. Whether the authrs prvide a sufficiently clear and detailed descriptin f the methds t prevent undisclsed flexibility in the experimental prcedures r analysis pipeline. Whether the authrs have cnsidered sufficient utcme-neutral cnditins (e.g. absence f flr r ceiling effects; psitive cntrls) fr ensuring that the results btained are able t test the stated hyptheses. Fllwing Stage 1 peer review, manuscripts will be rejected utright, ffered the pprtunity t revise, r accepted. Manuscripts that pass peer review will be issued an in principle acceptance (IPA), indicating that the article will be published pending successful cmpletin f the study 4

5 accrding t the exact methds and analytic prcedures utlined, as well as a defensible and evidence-bund interpretatin f the results. Please nte that any deviatin frm the stated experimental prcedures, regardless f hw minr it may seem t the authrs, will be lead t summary rejectin f the manuscript. If the authrs wish t alter the experimental prcedures fllwing IPA but still wish t publish their article as a Registered Reprt then the manuscript must be withdrawn and resubmitted as a new Stage 1 submissin. Nte that registered analyses must be undertaken, but additinal unregistered analyses can als be included in a final manuscript (see belw). Stage 2: Full manuscript review Once the study is cmplete, authrs prepare and resubmit their manuscript fr full review, with the fllwing additins: Submissin f raw data and labratry lg Raw data must be made freely available and will be upladed t the Elsevier Editrial System. Data files shuld be apprpriately time stamped t shw that data was cllected after IPA and nt befre. Other than pre-registered and apprved pilt data, n data acquired prir t the date f IPA is admissible in the Stage 2 submissin. Raw data must be accmpanied by guidance ntes, where required, t assist ther scientists in replicating the analysis pipeline. Authrs are encuraged t uplad any relevant analysis scripts. The authrs must cllectively certify in the resubmissin Cver Letter that all nn-pilt data was cllected after the date f IPA. A basic labratry lg must als be prvided utlining the range f dates during which data cllectin tk place. Revisins t the Backgrund and Ratinale The stated hyptheses cannt be altered r appended. Hwever, it is acceptable fr the tne and cntent f an Intrductin t be shaped by the results. Mrever, depending n the timeframe f data cllectin, new relevant literature may have appeared between registratin review and full manuscript review. Therefre, authrs will be allwed t update part f the Intrductin. Results & Discussin These will be similar t standard Research Reprts but with added requirements. The utcme f all registered analyses must be reprted in the manuscript, except in rare instances where a registered and apprved analysis is subsequently shwn t be lgically flawed r unfunded. In such cases, the authrs, reviewers, and editr must agree that a cllective errr f judgment was made and that the analysis is inapprpriate. In such cases the analysis wuld still be mentined in the Methds but mitted with justificatin frm the Results. It is reasnable that authrs may wish t include additinal analyses that were nt included in the registered submissin. Fr instance, a new analytic apprach might becme available between IPA and full review, r a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, apprpriately caveated, and reprted in a separate sectin f the Results titled Pst hc analyses. Authrs must be careful nt nt base their cnclusins entirely n the utcme f statistically significant pst hc analyses. Authrs will be required t reprt exact p values and effect sizes fr all inferential tests using the Neyman-Pearsn apprach. The resubmissin will ideally be cnsidered by the same reviewers as in the registratin stage, but culd als be assessed by fresh reviewers. In cnsidering papers at Stage 2, reviewers will be asked t decide: Whether the data are able t test the authrs prpsed hyptheses by passing the apprved utcme-neutral criteria (such as absence f flr and ceiling effects) 5

6 Whether any changes t the Intrductin are reasnable and d nt alter the ratinale r hyptheses Whether the authrs adhered precisely t the registered experimental prcedures Whether any unregistered pst hc analyses added by the authrs are justified, methdlgically sund, and infrmative Whether the authrs cnclusins are justified given the data Crucially, reviewers will be infrmed that editrial decisins will nt be based n the perceived imprtance, nvelty r clarity f the results. Thus while reviewers are free t enter such cmments n the recrd, they will nt influence editrial decisins. Manuscript withdrawal and Retracted Registratins It is pssible that authrs with IPA may wish t withdraw their manuscripts fllwing r during data cllectin. Pssible reasns culd include technical errr r an inability t cmplete the study due t ther unfreseen circumstances. In all such cases, manuscripts can f curse be withdrawn. Hwever, the jurnal will publicly recrd each case in a sectin called Retracted Registratins. This sectin will include the authrs, prpsed title, the abstract frm the apprved Stage 1 submissin, and brief reasn(s) fr the failure t cmplete the study. Partial retractins are nt pssible; i.e. authrs cannt publish part f a registered study by selectively retracting ne f the planned experiments. Such cases must lead t retractin f the entire paper. Incremental Registratins Authrs have the ptin t add experiments t apprved submissins. In such cases the apprved manuscript will be cnsidered accepted fr publicatin, and authrs will be able t prpse additinal experiments fr Stage 1 cnsideratin. Where these experiments wuld extend the apprved submissin (as ppsed t being part f new submissins), the editrial team will seek t fast-track the review prcess. This ptin may be particularly apprpriate where an initial experiment reveals a majr serendipitus finding that warrants fllw-up within the same paper. In cases where an incremented submissin is rejected (at either Stage 1 r 2), authrs will retain the ptin f publishing the mst recently apprved versin f the manuscript. Fr further advice n specific scenaris fr incremental registratin, authrs are invited t cntact the editrial bard (crtex@ed.ac.uk). References Chambers CD. Registered Reprts: a new publishing initiative at Crtex. Crtex, 49(3): , Dienes Z. Bayesian Versus Orthdx Statistics: Which Side Are Yu On? Perspectives n Psychlgy Science, 6(3): , Hedges LV & Vevea, JL. Estimating effect size under publicatin bias: Small sample prperties and rbustness f a randm effects selectin mdel. Jurnal f Educatinal and Behaviral Statistics, 21(4): , Lane DM & Dunlap WP. Estimating effect size: Bias resulting frm the significance criterin in editrial decisins. British Jurnal f Mathematical and Statistical Psychlgy, 31(2): , Nsek BA, Spies JR, & Mtyl M. Scientific utpia: II. Restructuring incentives and practices t prmte truth ver publishability. Perspectives n Psychlgical Science, 7(6): , Ruder J, Speckman PL, Sun D, Mrey RD (2009). Bayesian t tests fr accepting and rejecting the null hypthesis. Psychnmic Bulletin & Review, 16(2): ,

7 Simmns JP, Nelsn LD, and Simnshn U. False-psitive psychlgy: Undisclsed flexibility in data cllectin and analysis allws presenting anything as significant. Psychlgical Science, 22(11): , Strube MJ. SNOOP: A prgram fr demnstrating the cnsequences f premature and repeated null hypthesis testing. Behavir Research Methds, 38(1): 24 27, Sftware available frm 7