Registered reports as a solu/on to publica/on bias and p-hacking

Transcription

1 Registered reports as a solu/on to publica/on bias and p-hacking Chris Chambers Cardiff University Brain Research Imaging Centre (CUBRIC) School of Psychology, Cardiff University chambersc1@cardiff.ac.uk 1

2 Results-driven culture distorts incen/ves What s best for science High quality research, published regardless of outcome What s best for scien/sts Producing a lot of good results see Nosek, Spies & Motyl (2012). Perspec'ves on Psychological Science, 7(6):

3 ~92% posiyve Fanelli (2010) What happens you put researchers under pressure to get good results? Publication bias! Lack of data sharing! ~70% failure Wicherts et al (2006) Publish or conduct next experiment! Generate and specify hypotheses! Lack of replication! 1 in 1000 papers Makel et al (2012) Interpret data! p-hacking! Analyse data & test hypotheses! ~50-90% prevalence John et al (2012) Kerr (1998) ~50-100% prevalence John et al (2012) p-hacking! Collect data! Design study! Low statistical power! ~50% chance to detect medium effects Cohen (1962); Sedlmeier and Gigerenzer (1989); Bezeau and Graves (2001)

4 h]ps://acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research

5 Why is this happening? Because we place too much importance on the results of research and not enough on the processes that produce them Results make science exciyng but judging the quality of science (and scienysts) according to the results is soc science

6 Fixing this requires a change in mindset Philosophy: What gives hypothesis-tesyng its scienyfic value is the QUESTION it asks the QUALITY of the method it uses never the RESULT it produces If we accept this philosophy then editorial decisions at journals should be blind to results

7 Registered Reports Four central aspects of the Registered Reports model: Researchers decide hypotheses, experimental procedures, and main analyses before data collecyon Part of the peer review process takes place before experiments are conducted Passing this stage of review virtually guarantees publicayon Original studies and high-value replicayons are welcome

8 How it works Authors submit STAGE 1 manuscript with IntroducYon, Proposed Methods & Analyses, and Pilot Data (if applicable) Are the hypotheses well founded? Stage 1 peer review Are the methods and proposed analyses feasible and sufficiently detailed? Is the study well powered? ( 90%) Have the authors included sufficient posi've controls to confirm that the study will provide a fair test? If reviews are posiyve then journal offers in-principle acceptance (IPA), regardless of study outcome (protocol not published yet)

9 How it works Authors do the research Authors resubmit completed STAGE 2 manuscript: Introduc/on and Methods (virtually unchanged) Results (new): Registered confirmatory analyses + unregistered exploratory analyses Discussion (new) Data deposited in a public archive Stage 2 peer review Did the authors follow the approved protocol? Did posi've controls succeed? Manuscript published! Are the conclusions jus'fied by the data?

10 None of these things ma]er

11 Published examples at Cortex Reproducible detailed, repeatable methods high staysycal power (2-3x > sample sizes) Transparent accompanied by open data & materials outcomes of confirmatory and exploratory analyses disynguished Credible no publicayon bias no hindsight bias no selecyve reporyng h]p:// See also: Social Psychology special issue: h]p://econtent.hogrefe.com/toc/zsp/45/3

12 Current State 75 journals have adopted Registered Reports 59 permanently 16 as part of a special issue Fields covered Life/medical sciences: neuroscience, nutriyon, psychology, psychiatry, biology, cancer research, ecology, clinical & preclinical medicine Social sciences: poliycal science, financial and accounyng research Physical sciences: chemistry, physics, computer science etc. 69 fully completed Stage 2 RRs (~ provisionally accepted & awaiyng data) h]ps://

13 Registered Reports at Royal Society Open Science Now available in all STEM areas, from physics to psychology h]p://rsos.royalsocietypublishing.org/content/registered-reports

14 Registered Reports at Nature Human Behaviour 14

15 Registered Reports at BMC Medicine The first Registered Reports model for clinical trials Prevents hidden outcome switching (AKA outcome reporyng bias; see h]p:// Eliminates publicayon bias and ensures all trials are published regardless of outcome Raises the issue of whether all clinical trials should be published as Registered Reports 15

16 Curated list h]ps://cos.io/rr/

17 Frequently asked quesyons

18 1. Are Registered Reports suitable for my field? Applicable to any field engaged in hypothesis-driven research where one or more of the following problems apply: Publica/on bias Significance chasing (e.g. p-hacking) Post hoc hypothesizing (hindsight bias) Low sta/s/cal power Lack of direct replica/on Not applicable for Purely exploratory science } Methods development No hypothesis tesyng 2. Will this limit explorayon or sygmayze exploratory research? No. The are no restric/ons on the repor/ng of unregistered exploratory analyses. Confirmatory and exploratory analyses are simply reported separately in the final paper What sygmayzes exploratory research is post hoc hypothesizing to fit a deducyve framework Exploratory research is simply not valued in its nayve form

19 Exploratory Reports at Cortex coming in December 2017 De-emphasis on a priori hypotheses and p values Greater emphasis on parameter esymayon and hypothesis generayon h]p://neurochambers.blogspot.de/2017/07/open-ended-open-science.html

20 Going further Can we integrate grant funding and Registered Reports? Registered Reports funding model Authors submit their research proposal before they have funding. Following review by the both the funder and the journal, the strongest proposals are offered financial support by the funder AND in-principle acceptance for publicayon by the journal. Grant funded and ar/cle(s) accepted on same day! 20

21 Registered Report: Journal/Funder Partnerships Journals/publishers Nico'ne and Tobacco Research PLOS Biology PLOS ONE Royal Society Open Science BMC, including BMC Medicine Funders Cancer Research UK Pfizer Children s Tumor FoundaYon CHDI h]ps://doi.org/ /ntr/ntx081

22 Registered Report: Journal/Funder Partnerships h]p://blogs.plos.org/everyone/2017/09/26/registered-reports-with-cv/

23 Informa/on Hub at the Center for Open Science These slides: h]ps://osf.io/d4x5/ h]ps://cos.io/rr/ Detailed FAQs Table comparing journal features For more info, me or David Mellor at the COS