Hacking the experimental method to avoid Cognitive biases

Transcription

1 Hacking the experimental method to avoid Cognitive biases Cyril Pernet, PhD Centre for Clinical Brain Sciences Neuroimaging Sciences, University of Edinburgh 3 rd March Warwick /m9.figshare

2 The experimental method ensures logical thinking (deduction/induction) Reproducibility, repeatability and replicability what are we talking about Make neuroimaging open and reproducible hack the exp method against your cognitive biases

3 The experimental method +Psychology%27s+Scientific+Methods

4 The scientific method Theory/Model, tools/methods, data/observations, verification

5 The scientific method (Popper = falsification) Publication Hypothesis (temporary truth) Interpretation against prediction (falsification) Study design & Prediction analysis collection

6 Reproducibility, repeatability and replicability

7 Reproducibility, repeatability and replicability Repeatability: repeat evaluations under identical conditions (can you re-run your own analysis?) Reproducibility: re-compute the same things (is your research transparent so that someone can re-do your analysis?) Replicability: achieve consistent results (using a new sample can we obtain the same results?) Statistical analyses aim at statistical generalizability: inferring from a sample to a target population. Scientific generalizability, on the other hand, refers to applying a model based on a particular target population to other populations. Kenett & Shmueli (2015). Nature Methods 12

8 Reproducibility, repeatability and replicability Reproducibility (we can redo your work) gives credibility to a result while replicability (we can get same kind of results) increases evidential weight. With Method reproducibility, we need to ensure that someone else can redo the same (=sharing data and code), while with result reproducibility (replicability), we want the same results from different data. In this case, only equivalence of effect sizes matter and not binary results (significant or not). Inferential reproducibility refers to conclusions obtained vis-à-vis a hypothesis of interest and is best obtained by using cumulative evidence. Goodman, Fanelli & Ioannidis (2016). Science Translational Medicine 8

9 Computational complexity Method reproducibility, is particularly challenging for neuroimaging CLICKING ON A GUI IS NOT REPRODUCIBLE sharing AND code sharing (literate programming) Pipelines (with/without container) are better (PSOM, AA, Nipype) 6912 analysis paths Tons of

10 Scientific Reproduction is a continuum Repeatability of your own analysis Reproduction of someone else analysis Direct reproduction (same experiment with new sample) = Replication Systematic reproduction (obtain the same finding under different conditions) Conceptual reproduction (replicate some findings proving the existence of a concept using different paradigms) Pernet & Poline (2014). GigaScience 4-15

11 What to expect? The inherent variability in biological systems means there is no expectation that results will necessarily be precisely replicated. Issues are that main results / conclusion don t replicate experimental design and power statistical analysis because of cognitive bias Begley & Ioannidis (2015) Circ Research 116

12 Make neuroimaging open and reproducible

13 Not addressed today Publication Interpretation for/against prediction Hypothesis low statistical power Study design & Prediction Power analyses Quantitative prediction Effect size Bayesian statistics Sharing Full reporting (COBIDAS) analysis collection

14 We conclude that the apparent failure of the Reproducibility Project to replicate many target effects can be adequately explained by overestimation of effect sizes (or overestimation of evidence against the null hypothesis) due to small sample sizes and publication bias in the psychological literature. ~ thresholded maps ~ raw stat maps

15 The researchers cognitive biases and how to hack your way through them

16 What s a cognitive bias? A cognitive bias refers to the systematic pattern of deviation from norm or rationality in judgment, whereby inferences about other people and situations may be drawn in an illogical fashion (Wikipedia). A cognitive bias is a mistake in reasoning, evaluating, remembering, or other cognitive process, often occurring as a result of holding onto one's preferences and beliefs regardless of contrary information. Psychologists study cognitive biases as they relate to memory, reasoning, and decision-making. ( Let s think about cognitive bias Nature editorial Oct 2015

17 The scientific method (hack 1) Publication Interpretation for/against prediction analysis Hypothesis collection Study design & Prediction IKEA effect: consumers place artificially high value on products that they have built themselves your own design and analysis script/method might not be has good as what you think Design and analysis plan are better when externally reviewed (pre-registration?) Use validated code (why writing a SVM when you have many tested libraries) New code need unit testing

18 The scientific method (hack 2) Publication Hypothesis Texas sharp-shooter effect: firing off a few rounds and then drawing a bull s eye around the bullet holes Interpretation for/against prediction analysis P-hacking collection Study design & Prediction trying many analyses techniques until is start showing some results. Use registered analyses to test hypotheses (otherwise, it s called exploratory analysis, and it s fine too, but don t pretend) Outcomes variables are known in advance

19 The scientific method (hack 3) Publication Hypothesis Confirmation bias: carefully Interpretation for/against prediction analysis P-hacking collection Study design & Prediction debugging analyses and debunking data that counter a favoured hypothesis, while letting evidence in favour of the hypothesis slide by unexamined Use data blinding (removing outliers that do not fit the hypothesis) Use reverse inference tools to generate all alternative interpretations (neurovault and neurosynth)

20 The scientific method (hack 4) Publication Interpretation for/against prediction analysis Hypothesis P-hacking collection Study design & Prediction Hindsight bias: inclination, after an event has occurred, to see the event as having been predictable, despite there having been little or no objective basis for predicting it. Hypothesizing After the Results are Knowns (Kerr, 1998) and Selecting Hypothesized Areas after Results are Known (Poldrack et al 2017) Preregistration Independent ROI (related to circularity analyses too)

21 TAKE HOME MESSAGE Fight your biases! Don t trust your own code (unit test) and use validated code when available register hypotheses and outcome variables Use blinding, independent ROI, independent map/activation reading tools (reverse inference) Share data (raw and stat maps) and code This allows to attain (at very low cost) a certain level of replicability and statistical generalizability