Fixing the replicability crisis in science. Jelte M. Wicherts
|
|
- Rosa Harrison
- 5 years ago
- Views:
Transcription
1 Fixing the replicability crisis in science Jelte M. Wicherts 1
2 Supporting responsible research practices Diminishing biases in research Lowering false positive ratios Fixing the replicability crisis in science Enhancing the trust in scientific findings Making science more efficient Improving scientific reproducibility Empowering the truth in science Scaring away scientific cowboys 2
3 Empirical cycle Observe (literature) Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) 3
4 Success rates across the sciences Source: Fanelli, D. (2010). Positive results increase down the hierarchy of the sciences. PloS one, 5(4), e
5 Fraud Observe (literature) Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) 5
6 How to counter scientific misconduct Improve regulations & procedures Training in responsible conduct of research Lower questionable research practices Enhance transparency and accountability 6
7 Open science practices Heightens reproducibility and data re-use Leads to loss of sleep among scientific fraudsters Sources: Wicherts, J. M. (2011). Psychology must learn a lesson from fraud case. Nature, 480, 7. Wicherts, J. M. & Bakker, M. (2012). Publish (your data) or (let the data) perish! Why not publish your data too? 7 Intelligence, 40,
8 HARKing Observe (literature) Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) 8
9 y(t) HARKing White Noise time the infamous one-year dip! (at t=365) 9
10 HARKing 10
11 Explorative research 11
12 Suboptimal designs Observe (literature) Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) 12
13 Poor design 13
14 Study number Distribution under H 0 Distribution under H A Power N l = Actual effect size
15 N % effects are likely to be inflated estimates of the cts, given the problems associated with small escribed above. Power failure in neuroscience esults described in this section are based on meta-analyses, and we should be appropriately in extrapolating from this limited evidence. less, it is notable that the results are so conith those observed in other fields, such as the results indicated that the median statistical power of these studies was 8% across 461 individual studies aging and neuroscience studies that we have above. ions ions for the likelihood that a research finding true effect. Our results indicate that the avertical power of studies in the field of neurosciobably no more than between ~8% and ~31%, sis of evidence from diverse subfields within ience. If the low average power we observed ese studies is typical of the neuroscience lits a whole, this has profound implications for A major implication is that the likelihood that inally significant finding actually reflects a true mall. As explained above, the probability that h finding reflects a true effect (PPV) decreases ical power decreases for any given pre-study and a fixed type I error level. It is easy to show even lower than the 8 31% range we observed. Ethical implications. Low average power in neuroscience studies also has ethical implications. In our analysis of animal model studies, the average sample size of 22 animals for the water maze experiments was only sufficient to detect an effect size of d = 1.26 with Source: Button, K. S. et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, of meta-analyses (%) on the right axis. There is a clear Power (%) Figure 3 Median power of studies included in neuroscience meta-analyses. The figure shows a histogram of median study power calculated for each of the n = 49 meta-analyses included in our analysis, with the number of meta-analyses (N) on the left axis and percent bimodal distribution; n = 15 (31%) of the meta-analyses
16 Fixing the power! Powerful designs collaborate 16
17 File-drawer problem Observe (literature) Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) 17
18 Publication bias When is a study truly failed? Blackboard in the office of a couple of PhD students 18
19 Power intuitions Marjan Bakker asked 291 psychologists to indicate their: -typical effect size -typical sample size -typical power β α=0.05 I usually aim for subjects per cell of the experimental design, which is typically what it takes to detect a medium effect size with.80 probability. Actual power =.35 80% of respondents overestimated the power of their studies Source: Bakker, M., Wicherts, J. M., Hartgerink, C. H. J., & van der Maas, H. L. J. (2016). Researcher s intuitions about power in psychological research. Psychological Science, 27,
20 Poor statistical intuitions Researchers over overly optimistic to find evidence when they are right Source: Bakker, M., Wicherts, J. M., Hartgerink, C. H. J., & van der Maas, H. L. J. (2016). Researcher s intuitions about power in psychological research. Psychological Science, 27,
21 Failed study /fāld stuhd-ee/ 1. Is an empirical study in which unforeseen problems occurred during the data collection 2. Colloquial expression used in the sciences before 2018 to denote studies with (disappointing) nonsignificant outcomes that were deemed unpublishable Source: van Assen, M. A., van Aert, R. C., Nuijten, M. B., & Wicherts, J. M. (2014). Why publishing everything is more effective than selective publishing of statistically significant results. PLOS ONE, 9, e84896
22 Overly positive reporting Observe (literature) Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) 22
23 Selective outcome reporting Stress Fysiol. Measure Observ. behavior Selfreport p<.05 p<.05 p>.05 23
24 Evidence for this in clinical trials Source: Compare trials project (Ben Goldacre et al.) 24
25 Errors in the reporting of statistical results p =.06 Source: Bakker, M. & Wicherts, J. M. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43,
26 Source: Nuijten, M. B., Hartgerink, C.H.J., Van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016). The Prevalence of Statistical Reporting Errors in Psychology ( ). Behavior Research Methods 26
27 Fixing misreporting & selective reporting Reporting guidelines -STROBE -PRISMA -ARRIVE -STARD -CARE Peer review with checklists Statcheck & other tools 27
28 P-Hacking Observe (literature) Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) 28
29 P-hacking Remove outliers (Z > 2 ) p>.05 Call this a failed study Perform new study p>.05 p<.05 Add 10 cases p<.05 p>.05 Redo analysis with adapted? dependent var. p>.05 p<.05 Effect! Planned analysis p<.05 Write paper Misreport the p-value as being <.05 29
30 Many ways to analyse the data. imply many ways to reach the stars* *p<.05 Source: Wicherts et al. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies. A checklist to avoid p-hacking. Frontiers in Psychology, 7,
31 P-hacking pervasive? 12.5% articles misreporting p-values 87.5% articles using more subtle ways of p-hacking?? 31
32 Scientists are only human This one SHOULD really be higher! If not my reviewers will kill my paper There MUST be something wrong with this analysis or with these data And I can forget about getting tenure And I cannot buy the house I wanted 32
33 Pre-register studies Specify hypotheses & analyses in advance Publish the pre-registration Or publish a Registered Report (RR) in which the peer review is focused on the rationale, hypotheses, and methods (and article is published regardless of the results) Sources: Chambers, C. D. (2013). Registered Reports: A new publishing initiative at Cortex. Cortex. Wagenmakers et al. (2012). An Agenda for Purely Confirmatory Research. Perspectives on Psychological Science, 7,
34 Pre-registration challenge 34
35 Big money Observe (literature) Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) 35
36 Big money The business model of publishers is not necessarily in line with goals of furthering science. Is non-profit publishing the answer? Robert Maxwell 36
37 Big money Incentivize the right behaviors 37
38 Lack of replication Observe (literature) Observe (literature) Evaluate (present) Hypothesize Evaluate (present) Hypothesize Test (collect & analyze data) Predict (Set-up exp.) Test (collect & analyze data) Predict (Set-up exp.) Use cross-validation or other holdout sample techniques 38
39 Problems and how to fix them Problems Scientific misconduct Observer bias & effects File drawer problem HARKing Errors in reporting & selective outcome reporting P-hacking Lack of replication Solutions Open data & regulations Blinding Power, banning failed study Pre-registration Reporting guidelines, reviewer checklists statcheck Pre-registration Incentivize & analyze sensibly 39
40 The first principle is that you must not fool yourself and you are the easiest person to fool. Richard P. Feynman 40
41 Metaresearch.nl Hilde Augusteijn Marjan Bakker Marcel van Assen Amir Abdol Michele Nuijten Coosje Veldkamp + Esther Maassen Andrea Stoevenbelt Robbie van Aert Linda Dominguez Alvarez Chris Hartgerink Paulette Flore Olmo van den 41
Registered Reports: Peer reviewed study pre-registration. Implications for reporting of human clinical trials
Registered Reports: Peer reviewed study pre-registration Implications for reporting of human clinical trials Chris Chambers Professor of Cognitive Neuroscience Cardiff University Brain Research Imaging
More informationErrors and biases in Structural Equa2on Modeling. Jelte M. Wicherts
Errors and biases in Structural Equa2on Modeling Jelte M. Wicherts 1 De empirical cycle Observe Evaluate Theorize Test Predict 2 Grey zone map Remove outliers (Z > 2 ) P>.05? P>.05 P
More informationCURRICULUM VITAE MARJAN BAKKER PERSONAL WORK EXPERIENCE EDUCATION
CURRICULUM VITAE MARJAN BAKKER Tilburg School of Social and Behavioral Sciences Department of Methodology and Statistics Tilburg University PO Box 90151 5000 LE Tilburg The Netherlands m.bakker_1@uvt.nl
More informationChristopher Chambers Cardiff University. Rob McIntosh University of Edinburgh. Zoltan Dienes University of Sussex
1 Christopher Chambers Cardiff University Zoltan Dienes University of Sussex Rob McIntosh University of Edinburgh Klaus Willmes-von Hinckeldey University of Aachen 2 Problems with everyday research practices
More informationTRANSPARENCY AND OPENNESS IN SCIENCE
TRANSPARENCY AND OPENNESS IN SCIENCE Marjan Bakker 24 May 2016 A CRISIS A CRISIS IN PSYCHOLOGY A CRISIS FRAUD 58 PAPERS Diederik Stapel See www.retractionwatch.com for the current count A CRISIS FEELING
More informationTilburg University. Conducting meta-analyses based on p values van Aert, Robbie; Wicherts, Jelte; van Assen, Marcus
Tilburg University Conducting meta-analyses based on p values van Aert, Robbie; Wicherts, Jelte; van Assen, Marcus Published in: Perspectives on Psychological Science Document version: Publisher's PDF,
More informationHacking the experimental method to avoid Cognitive biases
Hacking the experimental method to avoid Cognitive biases Cyril Pernet, PhD Centre for Clinical Brain Sciences Neuroimaging Sciences, University of Edinburgh 3 rd March 2017 - Warwick - 10.6084/m9.figshare.4696429
More informationSTI 2018 Conference Proceedings
STI 2018 Conference Proceedings Proceedings of the 23rd International Conference on Science and Technology Indicators All papers published in this conference proceedings have been peer reviewed through
More informationEstimation of effect sizes in the presence of publication bias: a comparison of meta-analysis methods
Estimation of effect sizes in the presence of publication bias: a comparison of meta-analysis methods Hilde Augusteijn M.A.L.M. van Assen R. C. M. van Aert APS May 29, 2016 Today s presentation Estimation
More informationChanges to NIH grant applications:
Changes to NIH grant applications: Changes to application guide instructions for preparing the Research Strategy. Three new required sections have to be incorporated into the 12 page limit of the Research
More informationPooling Subjective Confidence Intervals
Spring, 1999 1 Administrative Things Pooling Subjective Confidence Intervals Assignment 7 due Friday You should consider only two indices, the S&P and the Nikkei. Sorry for causing the confusion. Reading
More informationCurriculum Vitae. Michèle B. Nuijten, MSc. December 19, 2016
Curriculum Vitae Michèle B. Nuijten, MSc December 19, 2016 Personal Information Born February 9th 1990, Utrecht, The Netherlands Work Address Tilburg University, Department of Methodology and Statistics,
More informationARE RESEARCH FINDINGS RELIABLE? SAINT KIZITO OMALA MAKERERE UNIVERSITY
ARE RESEARCH FINDINGS RELIABLE? SAINT KIZITO OMALA MAKERERE UNIVERSITY 2 0 1 6 E A S T A F R I C A I M P A C T E V A L U AT I O N W O R K S H O P A N D E V I D E N C E S U M M I T RELIABLE RESEARCH When
More informationBayesian Mixture Modeling of Significant P Values: A Meta-Analytic Method to Estimate the Degree of Contamination from H 0
Bayesian Mixture Modeling of Significant P Values: A Meta-Analytic Method to Estimate the Degree of Contamination from H 0 Quentin Frederik Gronau 1, Monique Duizer 1, Marjan Bakker 2, & Eric-Jan Wagenmakers
More informationCurriculum Vitae. Michèle B. Nuijten, MSc. May 10, 2017
Curriculum Vitae Michèle B. Nuijten, MSc May 10, 2017 Personal Information Born February 9th 1990, Utrecht, The Netherlands Work Address Tilburg University, Department of Methodology and Statistics, P.O.
More informationThe Replication Crisis Explained N. WAGNER & R. TERNES MI AIR NOV. 2018
The Replication Crisis Explained N. WAGNER & R. TERNES MI AIR NOV. 2018 Replication as the Cornerstone of Science We all know that the scientific method relies on careful observation, prediction & hypothesis
More informationScientific Ethics. Modified by Emmanuel and Collin from presentation of Doug Wallace Dalhousie University
Scientific Ethics Modified by Emmanuel and Collin from presentation of Doug Wallace Dalhousie University Outline What is scientific ethics Examples of common misconduct Known rates of misconduct Where
More informationUC Davis UC Davis Previously Published Works
UC Davis UC Davis Previously Published Works Title Introduction to the Special Section on Improving Research Practices: Thinking Deeply Across the Research Cycle Permalink https://escholarship.org/uc/item/54f712t3
More informationIntroduction to Research Methods
Introduction to Research Methods 8-10% of the AP Exam Psychology is an empirical discipline. Psychologists develop knowledge by doing research. Research provides guidance for psychologists who develop
More informationRegistered reports as a solu/on to publica/on bias and p-hacking
Registered reports as a solu/on to publica/on bias and p-hacking Chris Chambers Cardiff University Brain Research Imaging Centre (CUBRIC) School of Psychology, Cardiff University Email: chambersc1@cardiff.ac.uk
More informationUnderreporting in Psychology Experiments: Evidence From a Study Registry
Article Underreporting in Psychology Experiments: Evidence From a Study Registry Social Psychological and Personality Science 2016, Vol. 7(1) 8-12 ª The Author(s) 2015 Reprints and permission: sagepub.com/journalspermissions.nav
More informationHOW CAN RESEARCH FINDINGS BECOME MORE RELIABLE? ANTHONY MVEYANGE WORLD BANK /EAST FELLOW
HOW CAN RESEARCH FINDINGS BECOME MORE RELIABLE? ANTHONY MVEYANGE WORLD BANK /EAST FELLOW 2 0 1 6 E A S T A F R I C A I M PA C T E VA L U AT I O N W O R K S H O P A N D E V I D E N C E S U M M I T META-RESEARCH
More informationThe Replication Paradox: Combining Studies Can Decrease Accuracy of Effect Size Estimates. Michèle B. Nuijten, Marcel A. L. M.
The Replication Paradox: Combining Studies Can Decrease Accuracy of Effect Size Estimates Michèle B. Nuijten, Marcel A. L. M. van Assen, Coosje L. S. Veldkamp, and Jelte M. Wicherts Tilburg University
More informationCONSORT 2010 checklist of information to include when reporting a randomised trial*
CONSORT 2010 checklist of information to include when reporting a randomised trial* Section/Topic Title and abstract Introduction Background and objectives Item No Checklist item 1a Identification as a
More informationLess Story and more Reliability in cognitive neuroscience. David E. Huber, Kevin W. Potter, and Lucas D. Huszar. University of Massachusetts, Amherst
Less Story and more Reliability in cognitive neuroscience David E. Huber, Kevin W. Potter, and Lucas D. Huszar University of Massachusetts, Amherst Contact Office: 425 Tobin Hall Office Phone: (413) 545-1559
More informationThe Royal College of Pathologists Journal article evaluation questions
The Royal College of Pathologists Journal article evaluation questions Previous exam questions Dorrian CA, Toole, BJ, Alvarez-Madrazo S, Kelly A, Connell JMC, Wallace AM. A screening procedure for primary
More informationSystematic Reviews. Simon Gates 8 March 2007
Systematic Reviews Simon Gates 8 March 2007 Contents Reviewing of research Why we need reviews Traditional narrative reviews Systematic reviews Components of systematic reviews Conclusions Key reference
More informationChecking the counterarguments confirms that publication bias contaminated studies relating social class and unethical behavior
1 Checking the counterarguments confirms that publication bias contaminated studies relating social class and unethical behavior Gregory Francis Department of Psychological Sciences Purdue University gfrancis@purdue.edu
More informationCurriculum Vitae. Michèle B. Nuijten, MSc. November 15, 2017
Curriculum Vitae Michèle B. Nuijten, MSc November 15, 2017 Personal Information Born February 9th 1990, Utrecht, The Netherlands Work Address Tilburg University, Department of Methodology and Statistics,
More informationRESEARCH ARTICLE. Dipartimento di Psicologia Generale, Università di Padova, Italy
Journal of Scientific Exploration, Vol. 30, No. 1, pp. 10 15, 2016 0892-3310/16 RESEARCH ARTICLE Prospective Statistical Power: Sample Size Recommendations for the Investigation of the Main Parapsychological
More informationETHICS OF PUBLICATION
ETHICS OF PUBLICATION Ethics in Psychological Science Beyond how participants are treated, its of huge importance to be honest in reporting data Data most often comes from unverifiable sources (e.g., anonymous
More informationEQUATOR Network: promises and results of reporting guidelines
EQUATOR Network: promises and results of reporting guidelines Doug Altman The EQUATOR Network Centre for Statistics in Medicine, Oxford, UK Key principles of research publications A published research
More informationSAMPLING AND SAMPLE SIZE
SAMPLING AND SAMPLE SIZE Andrew Zeitlin Georgetown University and IGC Rwanda With slides from Ben Olken and the World Bank s Development Impact Evaluation Initiative 2 Review We want to learn how a program
More informationStep 2 Challenging negative thoughts "Weeding"
Managing Automatic Negative Thoughts (ANTs) Step 1 Identifying negative thoughts "ANTs" Step 2 Challenging negative thoughts "Weeding" Step 3 Planting positive thoughts 'Potting" Step1 Identifying Your
More informationMore than 3Rs Improving the validity and reproducibility of animal research
CAWS Symposium 2017 Purdue University, West Lafayette, IN May 18th, 2017 More than 3Rs Improving the validity and reproducibility of animal research Hanno Würbel Division of Animal Welfare Vetsuisse Faculty,
More informationEPSE 594: Meta-Analysis: Quantitative Research Synthesis
EPSE 594: Meta-Analysis: Quantitative Research Synthesis Ed Kroc University of British Columbia ed.kroc@ubc.ca March 14, 2019 Ed Kroc (UBC) EPSE 594 March 14, 2019 1 / 39 Last Time Synthesizing multiple
More informationChapter 1.1. The Process of Science. Essential Questions
Chapter 1.1 The Process of Science Essential Questions 1. How does scientific inquiry answer questions about the living world? 2. How do observations lead to scientific inferences? 3. How do scientific
More informationGuidelines for reviewers
Guidelines for reviewers Registered Reports are a form of empirical article in which the methods and proposed analyses are pre-registered and reviewed prior to research being conducted. This format of
More informationScience, Society, and Social Research (1) Benjamin Graham
Science, Society, and Social Research (1) Nuts and Bolts My computer croaked, so no clickers today We will start collecting clicker data for grades next Thurs Discussion sections start next week Homework
More informationUNIT II: RESEARCH METHODS
THINKING CRITICALLY WITH PSYCHOLOGICAL SCIENCE UNIT II: RESEARCH METHODS Module 4: The Need for Psychological Science Module 5: Scientific Method and Description Module 6: Correlation and Experimentation
More informationCritical Thinking A tour through the science of neuroscience
Critical Thinking A tour through the science of neuroscience NEBM 10032/5 Publication Bias Emily S Sena Centre for Clinical Brain Sciences, University of Edinburgh slides at @CAMARADES_ Bias Biases in
More informationClever Hans the horse could do simple math and spell out the answers to simple questions. He wasn t always correct, but he was most of the time.
Clever Hans the horse could do simple math and spell out the answers to simple questions. He wasn t always correct, but he was most of the time. While a team of scientists, veterinarians, zoologists and
More informationEmotional Intelligence: The Foundation of Leadership
CASAGRANDE CONSULTING aligning performance with mission Emotional Intelligence: The Foundation of Leadership Georgia Campus Chiefs Training Conference March 2015 Inattentional Blindness We think we see
More informationThe importance of good reporting of medical research. Doug Altman. Centre for Statistics in Medicine University of Oxford
The importance of good reporting of medical research Doug Altman Centre for Statistics in Medicine University of Oxford 1 Why reporting matters CONSORT and other reporting guidelines EQUATOR Network Other
More informationEXERCISE: HOW TO DO POWER CALCULATIONS IN OPTIMAL DESIGN SOFTWARE
...... EXERCISE: HOW TO DO POWER CALCULATIONS IN OPTIMAL DESIGN SOFTWARE TABLE OF CONTENTS 73TKey Vocabulary37T... 1 73TIntroduction37T... 73TUsing the Optimal Design Software37T... 73TEstimating Sample
More informationCarrying out an Empirical Project
Carrying out an Empirical Project Empirical Analysis & Style Hint Special program: Pre-training 1 Carrying out an Empirical Project 1. Posing a Question 2. Literature Review 3. Data Collection 4. Econometric
More informationImproving reporting for observational studies: STROBE statement
Improving reporting for observational studies: STROBE statement Technical meeting on the reporting of human studies submitted for the scientific substantiation of health claims EFSA Parma 20 November 2013
More informationEssential Skills for Evidence-based Practice Understanding and Using Systematic Reviews
J Nurs Sci Vol.28 No.4 Oct - Dec 2010 Essential Skills for Evidence-based Practice Understanding and Using Systematic Reviews Jeanne Grace Corresponding author: J Grace E-mail: Jeanne_Grace@urmc.rochester.edu
More informationUnderstanding and Building Emotional Resilience
Understanding and Building Emotional Resilience @howtothrive Agenda Introduction to resilience Consider from a personal/parent perspective Discussion and practice Introduction to the Penn Resilience Programme
More informationEmotional Intelligence
Emotional Intelligence 1 Emotional Intelligence Emotional intelligence is your ability to recognize & understand emotions in yourself and others, and your ability to use this awareness to manage your behavior
More information9 research designs likely for PSYC 2100
9 research designs likely for PSYC 2100 1) 1 factor, 2 levels, 1 group (one group gets both treatment levels) related samples t-test (compare means of 2 levels only) 2) 1 factor, 2 levels, 2 groups (one
More informationImplementing scientific evidence into clinical practice guidelines
Evidence Based Dentistry Implementing scientific evidence into clinical practice guidelines Asbjørn Jokstad University of Oslo, Norway 15/07/2004 1 PRACTICE GUIDELINES IN DENTISTRY (Medline) 2100 1945
More informationSample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty
723724PSSXXX10.1177/0956797617723724Anderson et al.sample-size Planning for Accurate Power research-article2017 General Article Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting
More informationFixed Effect Combining
Meta-Analysis Workshop (part 2) Michael LaValley December 12 th 2014 Villanova University Fixed Effect Combining Each study i provides an effect size estimate d i of the population value For the inverse
More informationModule 2/3 Research Strategies: How Psychologists Ask and Answer Questions
Myers PSYCHOLOGY 8 th Edition in Modules Module 2/3 Research Strategies: How Psychologists Ask and Answer Questions 1 The Need for Psychological Science Psychologists, like all scientists, use the scientific
More informationMalhotra - Interview (London 2018)
Malhotra - Interview (London 2018) Dr. Aseem Malhotra: One of the reasons we have a complete healthcare system failure is because there is a root of an epidemic of misinformed doctors and misinformed patients.
More informationPublishing Your Study: Tips for Young Investigators. Learning Objectives 7/9/2013. Eric B. Bass, MD, MPH
Publishing Your Study: Tips for Young Investigators Eric B. Bass, MD, MPH Learning Objectives To apply a logical approach to organizing & presenting your work in a manuscript To recognize the importance
More informationRecent Developments: Should We WAAP Economics Low Power?
Recent Developments: Should We WAAP Economics Low Power? Tom Stanley School of Business and Law, Deakin University, Melbourne, Australia; Stanley@Hendrix.edu & Julia Mobley Professor of Economics, Emeritus
More informationDescribe what is meant by a placebo Contrast the double-blind procedure with the single-blind procedure Review the structure for organizing a memo
Business Statistics The following was provided by Dr. Suzanne Delaney, and is a comprehensive review of Business Statistics. The workshop instructor will provide relevant examples during the Skills Assessment
More informationPromoting Transparent Reporting of Conflicts of Interests. and Statistical Analyses at the Journal of Sex Research
1 Promoting Transparent Reporting of Conflicts of Interests and Statistical Analyses at the Journal of Sex Research John K. Sakaluk, Statistical Consultant, Journal of Sex Research Department of Psychology
More informationSource: Emotional Intelligence 2.0 by Travis Bradberry and Jean Greaves Copyright 2009 by Talent Smart
The following presentation, in large part, discusses the findings and work of Travis Bradberry, Ph.D. and Jean Graves, Ph.D. Their work can be found in greater detail in their most recent book Emotional
More informationTHOUGHTS, ATTITUDES, HABITS AND BEHAVIORS
THOUGHTS, ATTITUDES, HABITS AND BEHAVIORS Ellen Freedman, CLM Law Practice Management Coordinator Pennsylvania Bar Association I ve been thinking a lot lately about how we think, what we think, and what
More informationMadhukar Pai, MD, PhD Associate Professor Department of Epidemiology & Biostatistics McGill University, Montreal, Canada
Madhukar Pai, MD, PhD Associate Professor Department of Epidemiology & Biostatistics McGill University, Montreal, Canada Email: madhukar.pai@mcgill.ca Readers often look to the Reviewers discussion and
More informationWhat s Really True? Discovering the Fact and Fiction of Autism
What s Really True? Discovering the Fact and Fiction of Autism Beth MacLehose Dempsey Middle School, Delaware, Ohio In collaboration with Catherine Rice, National Center on Birth Defects and Developmental
More informationFame: I m Skeptical. Fernanda Ferreira. Department of Psychology. University of California, Davis.
Fame: I m Skeptical Fernanda Ferreira Department of Psychology University of California, Davis fferreira@ucdavis.edu February 20, 2017 Word count: 1492 1 Abstract Fame is often deserved, emerging from
More informationIntentional Action and Side Effects in Ordinary Language. The chairman of the board of a company has decided to implement
Intentional Action and Side Effects in Ordinary Language JOSHUA KNOBE Knobe, J. (2003). Intentional Action and Side Effects in Ordinary Language. Analysis, 63, 190-193. The chairman of the board of a company
More informationHealth Coaching {re} Defined: What health coaching is... And what it s not
Health Coaching {re} Defined: What health coaching is... And what it s not DATE What You ll Discover: What health coaching is NOT... and what health coaching is The difference between health coaches and
More informationTiming Your Research Career & Publishing Addiction Medicine
Timing Your Research Career & Publishing Addiction Medicine Jeffrey H. Samet, MD, MA, MPH Chief, General Internal Medicine John Noble MD Professor in General Internal Medicine & Professor of Public Health
More informationSupplemental Materials: Facing One s Implicit Biases: From Awareness to Acknowledgment
Supplemental Materials 1 Supplemental Materials: Facing One s Implicit Biases: From Awareness to Acknowledgment Adam Hahn 1 Bertram Gawronski 2 Word count: 20,754 excluding acknowledgements, abstract,
More informationPSYC1024 Clinical Perspectives on Anxiety, Mood and Stress
PSYC1024 Clinical Perspectives on Anxiety, Mood and Stress LECTURE 1 WHAT IS SCIENCE? SCIENCE is a standardised approach of collecting and gathering information and answering simple and complex questions
More informationRegistered Reports - Guidelines for reviewers
Registered Reports - Guidelines for reviewers Registered Reports are a form of publication of empirical research in which the methods and proposed analyses are pre-registered and reviewed prior to the
More informationRunning head: Estimating Replicability 1. Z-Curve: A Method for the Estimating Replicability Based on Test Statistics in Original Studies
Running head: Estimating Replicability 1 Z-Curve: A Method for the Estimating Replicability Based on Test Statistics in Original Studies Ulrich Schimmack and Jerry Brunner University of Toronto Mississauga
More informationThe QUOROM Statement: revised recommendations for improving the quality of reports of systematic reviews
The QUOROM Statement: revised recommendations for improving the quality of reports of systematic reviews David Moher 1, Alessandro Liberati 2, Douglas G Altman 3, Jennifer Tetzlaff 1 for the QUOROM Group
More informationCRITICAL APPRAISAL WORKSHEET 1
For Residents CRITICAL APPRAISAL WORKSHEET 1 AN ARTICLE ON THERAPY This worksheet should be submitted along with the completed Submission Form to your supervisor. It is based on the articles on therapy
More informationDescribe what is meant by a placebo Contrast the double-blind procedure with the single-blind procedure Review the structure for organizing a memo
Please note the page numbers listed for the Lind book may vary by a page or two depending on which version of the textbook you have. Readings: Lind 1 11 (with emphasis on chapters 10, 11) Please note chapter
More informationThe Power of Feedback
The Power of Feedback 35 Principles for Turning Feedback from Others into Personal and Professional Change By Joseph R. Folkman The Big Idea The process of review and feedback is common in most organizations.
More informationStill important ideas
Readings: OpenStax - Chapters 1 11 + 13 & Appendix D & E (online) Plous - Chapters 2, 3, and 4 Chapter 2: Cognitive Dissonance, Chapter 3: Memory and Hindsight Bias, Chapter 4: Context Dependence Still
More informationLecture 4: Evidence-based Practice: Beyond Colorado
Lecture 4: Evidence-based Practice: Beyond Colorado A. Does Psychotherapy Work? Many of you are in the course because you want to enter the helping professions to offer some form of psychotherapy to heal
More informationResponse to the ASA s statement on p-values: context, process, and purpose
Response to the ASA s statement on p-values: context, process, purpose Edward L. Ionides Alexer Giessing Yaacov Ritov Scott E. Page Departments of Complex Systems, Political Science Economics, University
More informationOverview. Survey Methods & Design in Psychology. Readings. Significance Testing. Significance Testing. The Logic of Significance Testing
Survey Methods & Design in Psychology Lecture 11 (2007) Significance Testing, Power, Effect Sizes, Confidence Intervals, Publication Bias, & Scientific Integrity Overview Significance testing Inferential
More informationA Guide to Reading a Clinical or Research Publication
A Guide to Reading a Clinical or Research Publication For people living with a rare disease, being able to read and understand the information found in a clinical or research publication can be especially
More informationToday the overuse of opioids is a problem. Many of
A PPENDIX B A Word About Opioid Use Today the overuse of opioids is a problem. Many of these opioids are prescribed and thus are legal. Other people steal or buy opioids on the streets. These are illegal.
More informationRunning head: PRIOR ODDS IN COGNITIVE AND SOCIAL PSYCHOLOGY 1. The Prior Odds of Testing a True Effect in Cognitive and Social Psychology
Running head: PRIOR ODDS IN COGNITIVE AND SOCIAL PSYCHOLOGY 1 The Prior Odds of Testing a True Effect in Cognitive and Social Psychology Brent M. Wilson and John T. Wixted University of California, San
More informationSTUDIES OF THE ACCURACY OF DIAGNOSTIC TESTS: (Relevant JAMA Users Guide Numbers IIIA & B: references (5,6))
STUDIES OF THE ACCURACY OF DIAGNOSTIC TESTS: (Relevant JAMA Users Guide Numbers IIIA & B: references (5,6)) Introduction: The most valid study design for assessing the accuracy of diagnostic tests is a
More informationAssessing risk of bias
Assessing risk of bias Norwegian Research School for Global Health Atle Fretheim Research Director, Norwegian Institute of Public Health Professor II, Uiniversity of Oslo Goal for the day We all have an
More informationEvidence-based Imaging: Critically Appraising Studies of Diagnostic Tests
Evidence-based Imaging: Critically Appraising Studies of Diagnostic Tests Aine Marie Kelly, MD Critically Appraising Studies of Diagnostic Tests Aine Marie Kelly B.A., M.B. B.Ch. B.A.O., M.S. M.R.C.P.I.,
More informationAppendix B. Contributed Work
Appendix B Contributed Work This appendix contains a list of the contributed publications and software packages until the completion of this PhD project. The PDF version of the dissertation contains links
More informationWhat is Science 2009 What is science?
What is science? The question we want to address is seemingly simple, but turns out to be quite difficult to answer: what is science? It is reasonable to ask such a question since this is a book/course
More informationScientific Thinking Handbook
Making Observations An observation is an act of noting and recording an event, characteristic, behavior, or anything else detected with an instrument or with the senses. Observations allow you to make
More informationSession 6: Assessing Measurement Validity
The following slides are the property of their authors and are provided on this website as a public service. Please do not copy or redistribute these slides without the written permission of all of the
More information2 Critical thinking guidelines
What makes psychological research scientific? Precision How psychologists do research? Skepticism Reliance on empirical evidence Willingness to make risky predictions Openness Precision Begin with a Theory
More informationResearch Methods in Social Psychology. Lecture Notes By Halford H. Fairchild Pitzer College September 4, 2013
Research Methods in Social Psychology Lecture Notes By Halford H. Fairchild Pitzer College September 4, 2013 Quiz Review A review of our quiz enables a review of research methods in social psychology.
More informationDEPARTMENT OF ECONOMICS AND FINANCE SCHOOL OF BUSINESS UNIVERSITY OF CANTERBURY CHRISTCHURCH, NEW ZEALAND
DEPARTMENT OF ECONOMICS AND FINANCE SCHOOL OF BUSINESS UNIVERSITY OF CANTERBURY CHRISTCHURCH, NEW ZEALAND A Primer on the Reproducibility Crisis and Ways to Fix It W. Robert Reed WORKING PAPER No. 21/2017
More informationEFFECTIVE MEDICAL WRITING Michelle Biros, MS, MD Editor-in -Chief Academic Emergency Medicine
EFFECTIVE MEDICAL WRITING Michelle Biros, MS, MD Editor-in -Chief Academic Emergency Medicine Why We Write To disseminate information To share ideas, discoveries, and perspectives to a broader audience
More informationPower & Sample Size. Dr. Andrea Benedetti
Power & Sample Size Dr. Andrea Benedetti Plan Review of hypothesis testing Power and sample size Basic concepts Formulae for common study designs Using the software When should you think about power &
More informationDescriptive Research Methods. Depending on the type of question researchers want answered, will depend on the way they investigate it
Descriptive Research Methods Depending on the type of question researchers want answered, will depend on the way they investigate it Naturalistic Observation Method that measures people (organisms), in
More informationEvidence on questionable research practices: The good, the bad, and the ugly
Evidence on questionable research practices: The good, the bad, and the ugly and how effective are the interventions? George C. Banks University of North Carolina at Charlotte Presentation for CARMA, November
More informationHealthy Self. Bell Ringer. Class Period
Healthy Self Name Class Period Bell Ringer Fill out the bell ringer at the beginning of each class period. I f you are ABSENT for a bell ringer you can copy the information from my website www.missgarfield.weebly.com.
More informationPsychology 205, Revelle, Fall 2014 Research Methods in Psychology Mid-Term. Name:
Name: 1. (2 points) What is the primary advantage of using the median instead of the mean as a measure of central tendency? It is less affected by outliers. 2. (2 points) Why is counterbalancing important
More information