TRANSPARENCY AND OPENNESS IN SCIENCE

Transcription

1 TRANSPARENCY AND OPENNESS IN SCIENCE Marjan Bakker 24 May 2016

2 A CRISIS

3 A CRISIS IN PSYCHOLOGY

4 A CRISIS FRAUD 58 PAPERS Diederik Stapel See for the current count

5 A CRISIS FEELING THE FUTURE Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of personality and social psychology, 100(3), 407.

6 REPORTING ERRORS p =.06 Bakker, M., & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 1-13.

7 QUESTIONABLE RESEARCH PRACTICES Occurrence of QRPs John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science,

8

9 QUESTIONABLE RESEARCH PRACTICES From Franco, A., Malhotra, N., & Simonovits, G. (2016). Underreporting in Psychology Experiments Evidence From a Study Registry. Social Psychological and Personality Science, 7(1), 8-12.

10

11 A CRISIS REPLICATION PROBLEMS Bargh et al. (1996) Doyen et al. (2012)

12 A CRISIS REPLICATION PROBLEMS Reproducibility Project: Psychology (RPP) Replications of 100 experimental and correlational studies. Thirty-six percent of replications had significant results Mean effect size of the replication effects was half the magnitude of the mean effect size of the original effects. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. Doi: /science.aac4716

13 ONLY IN PSYCHOLOGY???

14 FRAUD Dipak K. Das Medicine Hwang Woo-Suk Stem cell research Anil Potti Cancer Research Jan Hendrik Schön Physicist Yoshitaka Fujii Medicine Bengu Sezen Chemistry Marc Hauser Evolutionary Biologist

15 REPLICATIONS Experimental Economics Replication Project In a systematic replication project of experimental studies published in high-impact economics journals 61% replicated. Reproducibility Project for Cancer Biology is still running Failures to replicate medical findings Approximately US$28B/year is spent on preclinical research that is not reproducible From Freedman LP, Cockburn IM, Simcoe TS (2015) The Economics of Reproducibility in Preclinical Research. PLoS Biol 13(6): e doi: /journal.pbio

16 REPLICATIONS From Klein & Roodman 2005

17

18 TIME FOR TRANSITION Sharing of data and materials Preregistration

19 SHARING OF DATA AND MATERIALS CERN Data from LHC experiments

20 SHARING OF DATA AND MATERIALS No data despite promise 20% No reply 14% Data shared after 1st request 11% Data shared after reminders 16% Sharing was not common in psychology Undeliverable 4% Only 27% of researchers who published in an APA journal (psychology) could share their data (Wicherts, Borsboom, Kats & Molenaar, 2006). All of them had signed the APA Certification of Compliance With APA Ethical Principles, which includes the principle on sharing data for reanalysis. Refused/unable to share data 35% Two thirds of Dutch researchers sees saving their data on their current computer as proper archiving (Voorbrood 2010)

21 ADVANTAGES OF SHARING DATA AND MATERIALS Ensures that data is not lost Makes it possible to find (and correct) errors Makes secondary (novel) analyses possible (e.g., meta-analyses, novel statistical techniques) Reuse of materials for replication and follow-up research Wicherts, J.M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? Intelligence, 40, doi: /j.intell

22 SHARING OF DATA AND MATERIALS Forced by many funding agencies (NIH, NSF, the Wellcome Trust, MRC, DFG, NWO) Forced by some journals (e.g., PlosOne) Encouraged with Badges (e.g., Psychological Science) More and more forced by universities Data packages (Tilburg University) More repositories Open Science Framework Figshare Dataverse Etc

23

24 PREREGISTRATION Preregistration involves registering the hypothesis, study design, and data-analysis plan prior to data collection. Confirmatory and exploratory research is clearly separated Prevention of researcher degrees of freedom

25 PREREGISTRATION Open Science Framework AsPredicted.org Badges Registered reports

26

27 REGISTERED REPORTS More than 25 journals have adopted registered reports E.g., Cortex, AIMS Neuroscience, Attention, Perception, and Psychophysics, Cognition and Emotion, Cognitive Research, Comprehensive Results in Social Psychology, Drug and Alcohol Dependence, European Journal of Neuroscience, Experimental Psychology, Human Movement Science, International Journal of Psychophysiology, Journal of Accounting Research, Journal of Business and Psychology, Journal of European Psychology Students, Journal of Personnel Psychology, Journal of Media Psychology, Nicotine & Tobacco Research, NFS Journal, Perspectives on Psychological Science, Royal Society Open Science, Stress & Health, Working, Aging and Retirement

28 THE $1,000,000 PREREGISTRATION CHALLENGE One thousand researchers will win $1,000 each for publishing results of preregistered research.

29 A bright future for science? Thanks!