CIS : Computational Reproducibility

Transcription

1 CIS : Computational Reproducibility Introduction Dr. David Koop

2 About Me 3rd Year at UMass Dartmouth Research Interests: - Visualization - Computational Provenance & Reproducibility - Data Science Research Projects: - VisTrails: - Interactive Provenance - Geographic Visualization - Fluid Interaction Techniques 2

3 About You Previous topics courses? Reading research literature? Writing research papers? Presenting research material? Topics: - Version Control? - Data Management (SQL, nosql) - Virtual Machines - Containers (e.g. Docker, etc.) - Provenance - Workflows 3

4 4

5 Course Structure Topics rotate: not the same course as last year Designed to focus on current research topics Reading Papers Presenting and Discussing Material Relevant Assignments One Midterm Final Project - Reproduce existing work, or - Related research project 5

6 Course Information Registration - Make sure you have registered in COIN - If you have not registered, please see me after class Policies - Academic Honesty: Do not cheat, you will receive a zero (0). Repeat offenders will fail the course. - Late Policy: 3 late days for assignments only 6

7 Reproducibility and Science 7

8 Gay Marriage Canvassing Study (2015) Change in Support for Same Sex Marriage Canvassing Treatment Direct Contact Court Decisions Canvassing Treatment Secondhand Contact Experimental Condition Same Sex Marriage Script by Gay Canvasser Recycling Script by Gay Canvasser Same Sex Marriage Script by Straight Canvasser Recycling Script by Straight Canvasser Court Decisions Days Since Treatment Note: The first vertical dashed line represents the canvassing intervention, which was administered between Internet survey waves 1 and 2. The second vertical dashed line represents the U.S. Supreme Court decisions striking down California s ban on same-sex marriage. The Y-axis is opinion change between the baseline survey and subsequent survey waves, with higher scores indicating more support for same-sex marriage. Points represent mean values, bars display 95% bootstrap confidence intervals. [LaCour and Green, Science, 2014 (Retracted 2015)] 8

9 Gay Marriage Canvassing Study (2015) Change in Support for Same Sex Marriage Canvassing Treatment Direct Contact Court Decisions Canvassing Treatment Secondhand Contact Experimental Condition Same Sex Marriage Script by Gay Canvasser Recycling Script by Gay Canvasser Same Sex Marriage Script by Straight Canvasser Recycling Script by Straight Canvasser Court Decisions Days Since Treatment Note: The first vertical dashed line represents the canvassing intervention, which was administered between Internet survey waves 1 and 2. The second vertical dashed line represents the U.S. Supreme Court decisions striking down California s ban on same-sex marriage. The Y-axis is opinion change between the baseline survey and subsequent survey waves, with higher scores indicating more support for same-sex marriage. Points represent mean values, bars display 95% bootstrap confidence intervals. [LaCour and Green, Science, 2014 (Retracted 2015)] 8

10 Gay Marriage Canvassing Study (2015) Change in Support for Same Sex Marriage Canvassing Treatment Direct Contact Court Decisions Canvassing Treatment Secondhand Contact Experimental Condition Same Sex Marriage Script by Gay Canvasser Recycling Script by Gay Canvasser Same Sex Marriage Script by Straight Canvasser Recycling Script by Straight Canvasser Court Decisions Days Since Treatment Note: The first vertical dashed line represents the canvassing intervention, which was administered between Internet survey waves 1 and 2. The second vertical dashed line represents the U.S. Supreme Court decisions striking down California s ban on same-sex marriage. The Y-axis is opinion change between the baseline survey and subsequent survey waves, with higher scores indicating more support for same-sex marriage. Points represent mean values, bars display 95% bootstrap confidence intervals. [LaCour and Green, Science, 2014 (Retracted 2015)] 8

11 Reproducing Results Frequency LaCour (2014) Study 2, Baseline Feeling Thermometer "Irregularities in LaCour (2014)", Broockman, Kalla, and Aronow, 2015 Tried their own pilot study and were unable to get similar results Found published data matched other data from different studies CCAP Frequency Feeling Thermometer [Broockman et al., 2015] 9

12 Frequency Frequency Reproducing Results Simulated Wave LaCour (2014) Study 2, Baseline CCAP Sample Feeling Thermometer CCAP Remaining Uncertainties Feeling Thermometer LaCour (2014) Wave "Irregularities in LaCour (2014)", Broockman, Kalla, and Aronow, 2015 Tried their own pilot study and were unable to get similar results Found published LaCour (2014) data Wave matched 1 other data from different studies We do not have access to the same-sex marriage question in CCAP, so we cannot evaluate the similarities of LaCour (2014) s same-sex marriage question to the CCAP on that item. The claimed treatment e ect was heterogeneous by canvasser attributes and the posted replication file does not have canvasser identifiers, so it is di cult to perform diagnostics on the responses of those assigned to treatment. The data for the abortion study reported at gess/cis/cis-dam/cis_dam_2015/colloquium/papers/lacour_2015.pdf in LaCour (2015) is not currently publicly available. References [Broockman et al., 2015] 9

13 Retraction Green, one of the two authors, requested retraction after these questions arose " survey data could not be traced to any originating Qualtrics source files" (emphasis added) " failure to produce the raw data coupled with the other concerns noted above undermines the credibility of the findings" (emphasis added) Science retracted the paper on May 28,

14 fmri [J. Moren (via Flickr)] 11

15 fmri 12

16 fmri Determine activity in the brain based on blood flow Enabled Studies: - Drug addiction - Human empathy for robots - Reaction to poetry and prose Costs: - One Scan: ~$600/hour - Machine: $3 million [Lunau, vice.com, 2016] 13

17 Determining Brain Activity from fmri "Neuroscience has long struggled over just how useful fmri data is at showing brain function. In the early days these fmri signals were very small, buried in a huge amount of noise, says Elizabeth Hillman, a biomedical engineer at the Zuckerman Institute at Columbia University. A lot of this noise is literal: noise from the scanner, noise from the electrical components, noise from the person s body as it breathes and pumps blood." - N. Stockton, Wired 14

18 Statistical Corrections fmri data comes in as voxels (3D pixels) that are activated or not Have to compare to background (contrast) and some will be selected as activated There is often a greater likelihood that adjacent voxels will also be activated, but we need statistics to determine how likely this is "If you do a lot of tests, at least some of them will come out positive, even if they are not real. These are called false positives, and they are something you really want to watch out for." To solve this problem, there are various methods for correcting the multiple comparisons, but this also means that you lose a lot of statistical power. In other words, you get rid of your false positives, but it might mean you don't see things that are really there, you might find false negatives instead." [Scicurious, Scientific American Blog, 2012] 15

19 Determining Brain Activity from fmri "[R]esearchers use statistical software to help them separate the signal from the noise. And when these malfunction, they lead to false positives: indications of brain activity when none exist" IgNobel Prize in Neuroscience: The dead salmon study - N. Stockton, Wired [Bennet et al., 2009] 16

20 Studying fmri How to run a large-scale study to validate results? - Data sharing! - Use real data from 499 healthy people Error rate was 70%! Found a bug in one of the software packages used for fmri analysis (fixed in 2015) What about past results? ~3,500 results may have issues ["Cluster failure: Why fmri inferences for spatial extent have inflated falsepositive rates", A. Eklund et al., 2016, PNAS] 17

21 fmri Study Conclusions "These results question the validity of a number of fmri studies and may have a large impact on the interpretation of weakly significant neuroimaging results." "Due to lamentable archiving and data-sharing practices it is unlikely that problematic analyses can be redone." "Brain imagery has this tradition of showing a picture, but the data underlying that image is never shared" (via T. Nichols via Wired) Journals should require authors to include their statistical evaluations. 18

22 Gene Names and Excel 19

23 Gene names and Excel SEPT2 9/2/2016, MARCH1 3/1/2016 First cited in 2004 Blog posts: Studied supplemental data from 18 journals, 35,175 Excel files Increased at an annual rate of 15% in the past five years Not just Excel: e.g. Apache OpenOffice Calc ["Gene name errors are widespread in the scientific literature", M. Ziemann et al., 2016,] 20

24 Affected Gene Lists ["Gene name errors are widespread in the scientific literature", M. Ziemann et al., 2016,] 21

25 Ten Simple Rules for Reproducible Computational Research G. K. Sandve, A. Nekrutenko, J. Taylor, E. Hovig

26 Rules for Reproducible Computational Research Rule 1: For Every Result, Keep Track of How It Was Produced Rule 2: Avoid Manual Data Manipulation Steps Rule 3: Archive the Exact Versions of All External Programs Used Rule 4: Version Control All Custom Scripts Rule 5: Record All Intermediate Results, When Possible in Standardized Formats [Sandve et al., 2013] 23

27 Rules for Reproducible Computational Research Rule 6: For Analyses That Include Randomness, Note Underlying Random Seeds Rule 7: Always Store Raw Data behind Plots Rule 8: Generate Hierarchical Analysis Output, Allowing Layers of Increasing Detail to Be Inspected Rule 9: Connect Textual Statements to Underlying Results Rule 10: Provide Public Access to Scripts, Runs, and Results [Sandve et al., 2013] 24

28 Initiatives Stanford Center for Reproducible Neuroscience SIGMOD Reproducibility Badges Biostatistics Journal Center for Open Science (Psychology) 25

29 Next Class Scientific Writing and Presentation 26