HOW CAN RESEARCH FINDINGS BECOME MORE RELIABLE? ANTHONY MVEYANGE WORLD BANK /EAST FELLOW

Transcription

1 HOW CAN RESEARCH FINDINGS BECOME MORE RELIABLE? ANTHONY MVEYANGE WORLD BANK /EAST FELLOW E A S T A F R I C A I M PA C T E VA L U AT I O N W O R K S H O P A N D E V I D E N C E S U M M I T

2 META-RESEARCH As we have discussed, big problems include: Data-mining Selective reporting Publication bias Lack of transparency

3 HOW DO WE IMPROVE? Research transparency! 1. Study registration and pre-analysis plans 2. Results-neutral publishing 3. Replications 4. Publish /share all study results 5. Sharing: data, code, surveys, readme files

4 REGISTERING STUDIES Creating a public record of a study and basic information about the study (e.g., intervention, outcomes, location, dates) Register prior beginning the intervention IPA + J-PAL to pre-register on the AEA registry (

5 WHAT IS THE AEA REGISTRY? Portal for registering RCTs in the social sciences, established by the American Economics Association Established in 2013 ~580 trials registered

6 WHAT IS THE AEA REGISTRY?

7 OTHER REGISTRIES:

8 REWARD FOR REGISTRATION

9 WHY REGISTER A TRIAL? Address publication bias As a long standing requirement + increasingly a focus in the social sciences as well. Clinicaltrials.gov (185k + studies registered)

10 HOW AND WHEN TO REGISTER? How? Register by making a user account on socialscienceregistry.org 18 required fields: basic information such as trial title, country, intervention start/end dates, outcomes, etc Project staff (e.g. research managers) may register a trial for PIs, but PIs must log in to approve and submit the registration. More detailed how to guide and video in Box (link)/sharepoint

11 PRE-ANALYSIS PLANS Detailed write-ups about the study hypotheses, outcomes, and planned analysis Curbs against p-hacking or cherry picking AEA registry does NOT require preanalysis plans

12 WHY HAVE A PRE-ANALYSIS PLAN? Registration makes your work more credible!!! We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after (Kaplan and Irvin, 2015, PLOS).

13 SHARE ALL RESULTS (INCLUDING NULL RESULTS!) Publication bias leads to a skew towards positive (exciting) results Problem: how do we disseminate null results, if journals tend to accept positive results at greater rates? One solution: reporting all results in a public registry. For example: the FDA in the US requires that all results be reported on clinicaltrials.gov within one year after trial completion. For social science studies, the results could be reported in the place they are registered (e.g. AEA registry) Alltrials is a group that campaigns to push for all clinical trials to report results

14 RESULTS- NEUTRAL PUBLISHING Submitting a protocol with details on methodology of study and planned analysis, before carrying out the study. The article is then accepted based on the design, before any data are collected. Because the study is accepted in advance, the incentives for authors change from producing the most beautiful story to producing the most accurate one. - Chris Chambers, editor of Cortex (OSF citation)

15 RESULTS- NEUTRAL PUBLISHING 20 journals have adopted the registered reports model so far (list). Examples: AIMS Neuroscience Attention, Perception & Psychophysics Comparative Political Studies Comprehensive Results in Social Psychology Cortex Drug and Alcohol Dependence elife Experimental Psychology Frontiers in Cognition Journal of Business and Psychology Nutrition and Food Science Journal Perspectives on Psychological Science Social Psychology Working, Aging, and Retirement

16 REPLICATIONS As valuable as they are, replications are relatively rare. What can the research community do to increase replications being done and shared?

17 REPLICATION INITIATIVES: 3ie s replication program Reprodocibility projects in psychology and cancer biology Many labs project in psychology Replication wiki in economics

18 DATA-SHARING: WHAT? Making data, code, and other materials publicly available An article in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete set of instructions (and data) which generated the figures. Dr. David Donoho, Stanford stats professor

19 DATA-SHARING: WHY? Facilitates replication of published results Re-use of data for further studies and meta-analysis Promote better quality data/code/metadata

20 THE DATA-SHARING MOVEMENT Funders: Holdren memo, February 2013 Foundations- Gates, Wellcome Trust (and 15 other funders who signed joint statement), many others Journals and associations: Top economics journals e.g. AER, Econometric, AEJ American Political Science Association- new ethics guidelines for data-sharing Open science groups advocating for open data: Berkeley Initiative for Transparency in the Social Sciences Center for Open Science Meta-research institute at Stanford

21 RESEARCH TRANSPARENCY AT IPA IPA has adopted new data publication guidelines and policy: sharing data by 3 years following endline data collection. Data repository for data from RCTs (including IPA/J-PAL and other collaborating organizations)

22 DATA FROM RCTS IN THE SOCIAL SCIENCES

23 DATA FROM RCTS IN THE SOCIAL SCIENCES

24 WHAT TO SHARE? 1. Datasets Recommended: cleaned, study dataset (PII removed!) Minimally: publication dataset- code/data underlying publication Ideal: start to finish reproducibility 2. Readme files explainin relation between data and code, as well as any further data documentation 3. Surveys 4. Study-level metadata

25 WHERE TO SHARE Data repositories: Dataverse (used by IPA and J-PAL) Can also create a personal dataverse (free- training later!). Figshare Dryad ICPSR (not free to deposit data, but curation services are offered) Many others for specific types of data, see Re3data.org

26 STEPS BEFORE SHARING DATA/CODE: 1. Checking usability of data e.g., that there are variable labels/value codes 2. Ensuring no sensitive data is released: Checking that all direct identified (names, IDs, etc) are removed. Indirect identifiers such as geospatial data removed Thinking carefully about whether variables can be recombined to identifuy participants 3. Replicating the study, in the sense of checking that data/code produce the published results 4. Checking that ReadMe files provide good instructions for replication 5. Ensuring variable/file/study-level metadata is shared

27 WHAT IS FULLY REPRODUCIBLE RESEARCH The ideal: sharing start to finish data/code to permit replication from initial raw dataset to final tables Includes all code (variable construction and cleaning as well). Key point sharing isn t all or nothing: sharing some data/code e.g. underlying the published results is better than sharing nothing!

28 GETTING DATA/CODE INTO SHAPE EARLY ON Preparing data/code early on is crucial, since there are serious limitations in what can be shared and understood later on, if good practices aren t followed

29 GETTING DATA/CODE INTO SHAPE E.g., Variable naming and labeling Commenting code Master do files Headers/footers ReadMe files..more detail in IPA s best practice manual for managin data and code

30 TOOLS FOR REPRODUCIBLE RESEARCH Version control for code using git Dynamic documents using Markdoc (Stata) or Rmarkdown (R) which generate tables in papers directly from the code Open science framework for collaborative workflow (session later!)

31 INCENTIVES TO SHARE DATA? Data citation and shifting tenure/promotion committees to rewarding transparency, rather than just publication.

32 INCENTIVES TO SHARE DATA? Transparency and openness promotion (TOP) guidelines (Science, 2015)