Why randomize? Rohini Pande Harvard University and J-PAL.
|
|
- Octavia Gregory
- 5 years ago
- Views:
Transcription
1 Why randomize? Rohini Pande Harvard University and J-PAL
2 Agenda I. Background on Program Evaluation II. What is a randomized experiment? III. Advantages and Limitations of Experiments IV. How wrong can you go: Vote 2002 campaign V. Conclusions
3 What is Program or Impact Evaluation? Program evaluation is a set of techniques used to determine if a treatment or intervention works. The treatment could be determined by an individual, or by a social program. To determine the causal effect of the treatment we need knowledge of counterfactuals, that is, what would have happened in the absence of the treatment? The true counterfactual is not observable: Observational studies, or associations, will not estimate causal effects The key goal of all program/impact evaluation methods is to construct or mimic the counterfactual.
4 Program Evaluation: The problem of Causal Inference What would have happened absent the program? How do we compare a world in which a program happens to a hypothetical world in which it doesn t? The program either happens or it doesn t. We cannot create a world in which the program never happened. So how do we measure this Counterfactual? We cannot do this exactly. But we can identify groups who are similar and measure their outcomes.
5 Counterfactuals We compare a group that is given the program to a group that is not given the program. Terminology: Treated (T): Group affected by program Control (C): Group unaffected by program Every program evaluation establishes a control group Randomized: Use random assignment of the program to create a control group which mimics the counterfactual. Non-randomized: Argue that a certain excluded group mimics the counterfactual.
6 What is a good counterfactual? When the only difference between Treatment and Control group is the program. In the absence of the program, Treatment and Control Units would have behaved the same. Key question: Are there other unobservable factors that might affect the treated units but not the control units? We call these unobservable factors selection bias. Presence of selection bias is rampant in observational studies.
7 Intuition Observational studies are contaminated with selection bias of unknown magnitude and direction. The selection bias is a consequence of the control and treatment groups differing in ways that we do not observe. Answer from Observational Study= Treatment Effect + Selection Bias If selection bias is positive, an observational study overstates the treatment effect; if negative, we understate this effect.
8 Intuition Observational studies are often used to understand the effect of exercise on health, taking extra classes on SAT scores and video games on exam performance. All are contaminated with selection bias. Key objective of program evaluation is finding or creating situations where the selection bias term is non-existent or very small. Good impact studies are those with small or zero selection bias. This will happen when: Treatment and Control groups differ only by exposure to the program. If they differ in the absence of the program, then we re not estimating a causal effect.
9 Types of Impact Evaluation Methods (1) Randomized Evaluations: Use coin-flip to classify subjects into Treatment and Control Groups. Give Treatment to the Treatment Group. Control group provides the counterfactual Also known as: Random Assignment Studies Randomized Field Trials Social Experiments Randomized Controlled Experiments
10 Types of Impact Evaluation Methods (Cont) (2) Non-Experimental or Quasi-Experimental Methods Simple Difference: Compare outcomes of treatment and Control group, where control group wasn t exposed to program for `exogenous reasons Differences-in-Differences Compare changes over time between treatment group and control group Statistical Matching Identify control group based on observables Instrumental Variables Predict treatment as function of variables which don t directly affect outcome of interest.
11 II WHAT IS A RANDOMIZED EVALUATION?
12 The basics Start with simple case: Take a sample of program applicants Randomly assign them to either: Treatment Group - receives treatment Control Group - not allowed to receive treatment (during the evaluation period)
13 Random Assignment Suppose 50% of a group of individuals are randomly `treated to a program (without regard to their characteristics). If successfully randomized, individuals assigned to the treatment and control groups differ only in their exposure to the treatment. Implies that the distribution of both observable and unobservable characteristics in the treatment and control groups are statistically identical. Had no individual been exposed to the progam, the average outcomes of treatment and control group would have been the same, and there would be no selection bias!
14 Basic Setup of a Randomized Evaluation Target Population Potential Participants Evaluation Sample Random Assignment Treatment Group Control Group Participants No-Shows Based on Orr (1999)
15 Some variations on the basics Assigning to multiple treatment groups Assigning to units other than individuals or households Health Centers Schools Local Governments Villages Important factors: What is the decision-making unit At what level can data be collected?
16 Key Steps in conducting an experiment 1. Design the study carefully What is the problem? What is the key question to answer? What are the potential policy levers to address the problem? 2. Collect baseline data and randomly assign people to treatment or control 3. Verify that assignment looks random 4. Monitor process so that integrity of experiment is not compromised
17 Key Steps in conducting an experiment (cont) 5. Collect follow-up data for both the treatment and control groups 6. Estimate program impacts by comparing mean outcomes of treatment group vs. mean outcomes of control group 7. Assess whether program impacts are statistically significant and practically significant
18 III - ADVANTAGES AND LIMITATIONS OF RANDOMIZED EVALUATIONS
19 Key advantage of experiments Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment, any difference that subsequently arises between them can be attributed to the treatment rather than to other factors
20 Validity A tool to assess credibility of a study Internal validity: relates to ability to draw causal inference, i.e. can we attribute our impact estimates to the program and not to something else External validity: relates to ability to generalize to other settings of interest, i.e. can we generalize our impact estimates from this program to other populations, time periods, countries, etc? Random assignment versus Random sampling
21 Other advantages of experiments Relative to results from non-experimental studies, results from experiments are: Less subject to methodological debates Easier to convey More likely to be convincing to program funders and/or policymakers
22 Limitations of experiments Despite great methodological advantage of experiments, they are also potentially subject to threats to their validity. For example, Internal Validity (e.g. Hawthorne Effects, survey non-response, noshows, crossovers, duration bias, etc.) External Validity (e.g. are the results generalizable to the population of interest?) It is important to realize that some of these threats also affect the validity of nonexperimental studies
23 Other limitations of experiments Measure the impact of the offer to participate in the program Depending on design it may be possible to understand of the mechanism underlying the intervention. Costs (though need to think about costs of getting the wrong answer and costs ) Partial equilibrium
24 Other limitations of experiments Ethical issues Some public policies are implemented in randomized fashion due to lack of resources, for ethical reasons or to remove discretion from the system. In such cases researchers can evaluate the policy immediately e.g. in which Indian villages are leader positions reserved for woman Exploit Pilots or phase-in due to budgetary constraints
25 A nuanced view of Randomization Randomization offers best gold-standard answer for any treatment that we want to offer on a universal, compulsory, or mandatory basis. Vaccinations. Access to education until a certain level. Medical treatment for a certain condition. Universal Health Insurance. A randomization is less useful for situations where the effect of the treatment varies across individuals, and where individuals will be allowed to self-select into treatment. Attending Business School. Elective Surgery.
26 IV HOW WRONG CAN YOU GO: THE VOTE 2002 CAMPAIGN
27 Case 1 Vote 2002 campaign Intervention designed to increase voter turnout in 2002 Phone calls to ~60,000 individuals Only ~35,000 individuals were reached Key Question: Did the campaign have a positive effect (i.e. impact) on voter turnout? 5 methods were used to estimate impact
28 Methods 1-3 Based on comparing reached vs not-reached Method 1: Simple difference in voter turnout, (Voter turnout) reached (Voter turnout) not reached Method 2: Multiple Regression controlling for some differences between the two groups Method 3: Method 2, but also controlling for differences in past voting behavior between the two groups
29 Impact Estimates using Methods 1-3 Estimated Impact Method pp * Method pp * Method pp * pp=percentage points; *: statistically significant at the 5% level
30 Methods 1-3 Is any of these impact estimates likely to be the true impact of the Vote 2002 campaign?
31 Reached vs. Not Reached Reached Not Reached Difference Female 56.2% 53.8% 2.4 pp* Newly Regist. 7.3% 9.6% -2.3 pp* From Iowa 54.7% 46.7% 8.0 pp* Voted in % 63.3% 8.3 pp* Voted in % 37.6% 9.0 pp* pp=percentage points *: statistically significant at the 5% level
32 Method 4: Matching Similar data available on 2,000,000 individuals Select as a comparison group a subset of the 2,000,000 individuals that resembles the reached group as much as possible Statistical procedure: matching To estimate impact, compare voter turnout between reached and comparison group
33 An illustration of matching Source: Arceneaux, Gerber, and Green (2004)
34 Impact Estimates using Matching Estimated Impact Matching on 4 covariates 3.7 pp * Matching on 6 covariates 3.0 pp * Matching on all covariates 2.8 pp * pp=percentage points; *: statistically significant at the 5% level
35 Method 4: Matching Is this impact estimate likely to be the true impact of the Vote 2002 campaign? Key: These two groups should be equivalent in terms of the observable characteristics that were used to do the matching. But what about unobservable characteristics?
36 Method 5: Randomized Experiment Turns out 60,000 were randomly chosen from population of 2,060,000 individuals Hence, treatment was randomly assigned to two groups: Treatment group (60,000 who got called) Control group (2,000,000 who did not get called) To estimate impact, compare voter turnout between treatment and control groups Do a statistical adjustment to address the fact that not everyone in the treatment group was reached
37 Method 5: Randomized Experiment Impact estimate: 0.4 pp, not statistically significant Is this impact estimate likely to be the true impact of the Vote 2002 campaign? Key: Treatment and control groups should be equivalent in terms of both their observable and unobservable characteristics Hence, any difference in outcomes can be attributed to the Vote 2002 campaign
38 Summary Table Method Estimated Impact 1 Simple Difference 10.8 pp * 2 Multiple regression 6.1 pp * 3 Multiple regression with 4.5 pp * panel data 4 Matching 2.8 pp * 5 Randomized Experiment 0.4 pp
39 VI CONCLUSIONS
40 Conclusions Good public policies require knowledge of causal effects. Causal effects are estimated only if we have good counterfactuals. In the absence of good counterfactuals, analysis is contaminated with selection bias. Be suspicious of causal claims that come from observational studies. Randomization offers a solution to generating good counterfactuals.
41 Conclusions If properly designed and conducted, social experiments provide the most credible assessment of the impact of a program Results from social experiments are easy to understand and much less subject to methodological quibbles Credibility + Ease =>More likely to convince policymakers and funders of effectiveness (or lack thereof) of program
42 Conclusions (cont.) However, these advantages are present only if social experiments are well designed and conducted properly Must assess validity of experiments in same way we assess validity of any other study Must be aware of limitations of experiments
43 THE END
Impact Evaluation Methods: Why Randomize? Meghan Mahoney Policy Manager, J-PAL Global
Impact Evaluation Methods: Why Randomize? Meghan Mahoney Policy Manager, J-PAL Global Course Overview 1. What is Evaluation? 2. Outcomes, Impact, and Indicators 3. Why Randomize? 4. How to Randomize? 5.
More informationTRANSLATING RESEARCH INTO ACTION. Why randomize? Dan Levy. Harvard Kennedy School
TRANSLATING RESEARCH INTO ACTION Why randomize? Dan Levy Harvard Kennedy School Your background Course Overview 1. What is evaluation? 2. Measuring impacts (outcomes, indicators) 3. Why randomize? 4. How
More informationCase study: Telephone campaigns to encourage child vaccination: Are they effective?
Case study: Telephone campaigns to encourage child vaccination: Are they effective? This case study is based on Comparing experimental and matching methods using a field experiment [ ] by Kevin Arceneaux,
More informationWhat can go wrong.and how to fix it!
What can go wrong.and how to fix it! Megha Pradhan Policy and Training Manager, J-PAL South Asia Kathmandu, Nepal 29 March 2017 Introduction Conception phase is important and allows to design an evaluation
More informationThreats and Analysis. Shawn Cole. Harvard Business School
Threats and Analysis Shawn Cole Harvard Business School Course Overview 1. What is Evaluation? 2. Outcomes, Impact, and Indicators 3. Why Randomize? 4. How to Randomize? 5. Sampling and Sample Size 6.
More informationImpact Evaluation Toolbox
Impact Evaluation Toolbox Gautam Rao University of California, Berkeley * ** Presentation credit: Temina Madon Impact Evaluation 1) The final outcomes we care about - Identify and measure them Measuring
More informationThreats to Validity in Experiments. The John Henry Effect
Threats to Validity in Experiments Office Hours: M 11:30-12:30, W 10:30-12:30 SSB 447 The John Henry Effect Housekeeping Homework 3, due Tuesday A great paper, but far from an experiment Midterm review
More informationInstrumental Variables Estimation: An Introduction
Instrumental Variables Estimation: An Introduction Susan L. Ettner, Ph.D. Professor Division of General Internal Medicine and Health Services Research, UCLA The Problem The Problem Suppose you wish to
More informationThreats and Analysis. Bruno Crépon J-PAL
Threats and Analysis Bruno Crépon J-PAL Course Overview 1. What is Evaluation? 2. Outcomes, Impact, and Indicators 3. Why Randomize and Common Critiques 4. How to Randomize 5. Sampling and Sample Size
More informationGlossary From Running Randomized Evaluations: A Practical Guide, by Rachel Glennerster and Kudzai Takavarasha
Glossary From Running Randomized Evaluations: A Practical Guide, by Rachel Glennerster and Kudzai Takavarasha attrition: When data are missing because we are unable to measure the outcomes of some of the
More informationThe Practice of Statistics 1 Week 2: Relationships and Data Collection
The Practice of Statistics 1 Week 2: Relationships and Data Collection Video 12: Data Collection - Experiments Experiments are the gold standard since they allow us to make causal conclusions. example,
More informationLecture II: Difference in Difference and Regression Discontinuity
Review Lecture II: Difference in Difference and Regression Discontinuity it From Lecture I Causality is difficult to Show from cross sectional observational studies What caused what? X caused Y, Y caused
More informationCASE STUDY 2: VOCATIONAL TRAINING FOR DISADVANTAGED YOUTH
CASE STUDY 2: VOCATIONAL TRAINING FOR DISADVANTAGED YOUTH Why Randomize? This case study is based on Training Disadvantaged Youth in Latin America: Evidence from a Randomized Trial by Orazio Attanasio,
More informationTRANSLATING RESEARCH INTO ACTION
TRANSLATING RESEARCH INTO ACTION This case study is based on a current study by Esther Duflo and Tavneet Suri. J-PAL thanks the authors for allowing us to use their project 17 Key Vocabulary 1. Equivalence:
More informationDoes Loss Aversion Motivate Collective Action?
Does Loss Aversion Motivate Collective Action? Kevin Arceneaux Temple University Chris Mann University of Miami David Nickerson University of Notre Dame Background Fundamental problem of causal inference
More informationLecture II: Difference in Difference. Causality is difficult to Show from cross
Review Lecture II: Regression Discontinuity and Difference in Difference From Lecture I Causality is difficult to Show from cross sectional observational studies What caused what? X caused Y, Y caused
More informationAbdul Latif Jameel Poverty Action Lab Executive Training: Evaluating Social Programs Spring 2009
MIT OpenCourseWare http://ocw.mit.edu Abdul Latif Jameel Poverty Action Lab Executive Training: Evaluating Social Programs Spring 2009 For information about citing these materials or our Terms of Use,
More informationEvaluating Social Programs Course: Evaluation Glossary (Sources: 3ie and The World Bank)
Evaluating Social Programs Course: Evaluation Glossary (Sources: 3ie and The World Bank) Attribution The extent to which the observed change in outcome is the result of the intervention, having allowed
More informationIntroduction to Applied Research in Economics Kamiljon T. Akramov, Ph.D. IFPRI, Washington, DC, USA
Introduction to Applied Research in Economics Kamiljon T. Akramov, Ph.D. IFPRI, Washington, DC, USA Training Course on Applied Econometric Analysis June 1, 2015, WIUT, Tashkent, Uzbekistan Why do we need
More informationMethods for Addressing Selection Bias in Observational Studies
Methods for Addressing Selection Bias in Observational Studies Susan L. Ettner, Ph.D. Professor Division of General Internal Medicine and Health Services Research, UCLA What is Selection Bias? In the regression
More informationExperiments. ESP178 Research Methods Dillon Fitch 1/26/16. Adapted from lecture by Professor Susan Handy
Experiments ESP178 Research Methods Dillon Fitch 1/26/16 Adapted from lecture by Professor Susan Handy Recap Causal Validity Criterion Association Non-spurious Time order Causal Mechanism Context Explanation
More informationPublic Policy & Evidence:
Public Policy & Evidence: How to discriminate, interpret and communicate scientific research to better inform society. Rachel Glennerster Executive Director J-PAL Global Press coverage of microcredit:
More informationSupplement 2. Use of Directed Acyclic Graphs (DAGs)
Supplement 2. Use of Directed Acyclic Graphs (DAGs) Abstract This supplement describes how counterfactual theory is used to define causal effects and the conditions in which observed data can be used to
More informationMethods of Randomization Lupe Bedoya. Development Impact Evaluation Field Coordinator Training Washington, DC April 22-25, 2013
Methods of Randomization Lupe Bedoya Development Impact Evaluation Field Coordinator Training Washington, DC April 22-25, 2013 Content 1. Important Concepts 2. What vs. Why 3. Some Practical Issues 4.
More informationChapter 13. Experiments and Observational Studies. Copyright 2012, 2008, 2005 Pearson Education, Inc.
Chapter 13 Experiments and Observational Studies Copyright 2012, 2008, 2005 Pearson Education, Inc. Observational Studies In an observational study, researchers don t assign choices; they simply observe
More informationChapter 13 Summary Experiments and Observational Studies
Chapter 13 Summary Experiments and Observational Studies What have we learned? We can recognize sample surveys, observational studies, and randomized comparative experiments. o These methods collect data
More informationThe Regression-Discontinuity Design
Page 1 of 10 Home» Design» Quasi-Experimental Design» The Regression-Discontinuity Design The regression-discontinuity design. What a terrible name! In everyday language both parts of the term have connotations
More information9 research designs likely for PSYC 2100
9 research designs likely for PSYC 2100 1) 1 factor, 2 levels, 1 group (one group gets both treatment levels) related samples t-test (compare means of 2 levels only) 2) 1 factor, 2 levels, 2 groups (one
More informationREVIEW FOR THE PREVIOUS LECTURE
Slide 2-1 Calculator: The same calculator policies as for the ACT hold for STT 315: http://www.actstudent.org/faq/answers/calculator.html. It is highly recommended that you have a TI-84, as this is the
More informationThe Stable Unit Treatment Value Assumption (SUTVA) and Its Implications for Social Science RCTs
The Stable Unit Treatment Value Assumption (SUTVA) and Its Implications for Social Science RCTs Alan S. Gerber & Donald P. Green Yale University From Chapter 8 of Field Experimentation: Design, Analysis,
More informationYou can t fix by analysis what you bungled by design. Fancy analysis can t fix a poorly designed study.
You can t fix by analysis what you bungled by design. Light, Singer and Willett Or, not as catchy but perhaps more accurate: Fancy analysis can t fix a poorly designed study. Producing Data The Role of
More informationMS&E 226: Small Data
MS&E 226: Small Data Lecture 10: Introduction to inference (v2) Ramesh Johari ramesh.johari@stanford.edu 1 / 17 What is inference? 2 / 17 Where did our data come from? Recall our sample is: Y, the vector
More informationYou must answer question 1.
Research Methods and Statistics Specialty Area Exam October 28, 2015 Part I: Statistics Committee: Richard Williams (Chair), Elizabeth McClintock, Sarah Mustillo You must answer question 1. 1. Suppose
More informationEXPERIMENTAL RESEARCH DESIGNS
ARTHUR PSYC 204 (EXPERIMENTAL PSYCHOLOGY) 14A LECTURE NOTES [02/28/14] EXPERIMENTAL RESEARCH DESIGNS PAGE 1 Topic #5 EXPERIMENTAL RESEARCH DESIGNS As a strict technical definition, an experiment is a study
More informationRegression Discontinuity Design
Regression Discontinuity Design Regression Discontinuity Design Units are assigned to conditions based on a cutoff score on a measured covariate, For example, employees who exceed a cutoff for absenteeism
More informationAP Statistics Chapter 5 Multiple Choice
AP Statistics Chapter 5 Multiple Choice 1. A nutritionist wants to study the effect of storage time (6, 12, and 18 months) on the amount of vitamin C present in freeze dried fruit when stored for these
More informationExperimental design (continued)
Experimental design (continued) Spring 2017 Michelle Mazurek Some content adapted from Bilge Mutlu, Vibha Sazawal, Howard Seltman 1 Administrative No class Tuesday Homework 1 Plug for Tanu Mitra grad student
More informationECONOMIC EVALUATION IN DEVELOPMENT
ECONOMIC EVALUATION IN DEVELOPMENT Autumn 2015 Michael King 1 A plan to end poverty.. I have identified the specific investments that are needed [to end poverty]; found ways to plan and implement them;
More informationMATH& 146 Lesson 6. Section 1.5 Experiments
MATH& 146 Lesson 6 Section 1.5 Experiments 1 Experiments Studies where the researchers assign treatments to cases are called experiments. When this assignment includes randomization (such as coin flips)
More informationIntroduction to Applied Research in Economics
Introduction to Applied Research in Economics Dr. Kamiljon T. Akramov IFPRI, Washington, DC, USA Training Course on Introduction to Applied Econometric Analysis November 14, 2016, National Library, Dushanbe,
More informationDifference-in-Differences in Action
Difference-in-Differences in Action Centre for Evaluation Seminar Series Timothy.Powell-Jackson@lshtm.ac.uk Department of Global Health and Development London School of Hygiene and Tropical Medicine Overview
More informationSYMSYS 130: Research Methods in the Cognitive and Information Sciences (Spring 2013)
SYMSYS 130: Research Methods in the Cognitive and Information Sciences (Spring 2013) Take Home Final June 20, 2013 Instructor' s Responses Please respond to the following questions with short essays (300-500
More informationThis exam consists of three parts. Provide answers to ALL THREE sections.
Empirical Analysis and Research Methodology Examination Yale University Department of Political Science January 2008 This exam consists of three parts. Provide answers to ALL THREE sections. Your answers
More informationIntroduction to Applied Research in Economics
Introduction to Applied Research in Economics Dr. Kamiljon T. Akramov IFPRI, Washington, DC, USA Regional Training Course on Applied Econometric Analysis June 12-23, 2017, WIUT, Tashkent, Uzbekistan Why
More informationREMINDER: Fill out your course evaluation by Sunday. Also, CAOS and SATS extended to Sunday, December 8 23:50.
REMINDER: Fill out your course evaluation by Sunday. Also, CAOS and SATS extended to Sunday, December 8 23:50. STA220 review session: Thursday December 5, 10:00 12:00, SS 2110 For more information: http://www.artsci.utoronto.ca/current/exam_jam
More informationChecking the counterarguments confirms that publication bias contaminated studies relating social class and unethical behavior
1 Checking the counterarguments confirms that publication bias contaminated studies relating social class and unethical behavior Gregory Francis Department of Psychological Sciences Purdue University gfrancis@purdue.edu
More informationKey questions when starting an econometric project (Angrist & Pischke, 2009):
Econometric & other impact assessment approaches to policy analysis Part 1 1 The problem of causality in policy analysis Internal vs. external validity Key questions when starting an econometric project
More informationExperimental Design Part II
Experimental Design Part II Keith Smolkowski April 30, 2008 Where Are We Now? esearch eview esearch Design: The Plan Internal Validity Statements of Causality External Validity Statements of Generalizability
More informationPSY 250. Designs 8/11/2015. Nonexperimental vs. Quasi- Experimental Strategies. Nonequivalent Between Group Designs
Nonexperimental vs. Quasi- Experimental Strategies PSY 250 Quasi and Non-experimental designs Chapter 10 Resemble experiments but contain some threat such as a confounding variable that can not be completely
More informationPrinciple underlying all of statistics
Sampling Bias Principle underlying all of statistics Drawing inferences from a suitable sample of a population is far less labor intensive than, but can be equally as informative as, studying the entire
More informationSOME NOTES ON THE INTUITION BEHIND POPULAR ECONOMETRIC TECHNIQUES
SOME NOTES ON THE INTUITION BEHIND POPULAR ECONOMETRIC TECHNIQUES These notes have been put together by Dave Donaldson (EC307 class teacher 2004-05) and by Hannes Mueller (EC307 class teacher 2004-present).
More informationStudy Design. Study design. Patrick Breheny. January 23. Patrick Breheny Introduction to Biostatistics (171:161) 1/34
Study design Patrick Breheny January 23 Patrick Breheny Introduction to Biostatistics (171:161) 1/34 in the ideal world In an ideal world, We have a list of everyone in the population of interest We randomly
More informationinvestigate. educate. inform.
investigate. educate. inform. Research Design What drives your research design? The battle between Qualitative and Quantitative is over Think before you leap What SHOULD drive your research design. Advanced
More informationWhat Behaviors Do Behavior Programs Change
What Behaviors Do Behavior Programs Change Yingjuan (Molly) Du, Dave Hanna, Jean Shelton and Amy Buege, Itron, Inc. ABSTRACT Utilities behavioral programs, such as audits and web-based tools, are designed
More informationA Review of Counterfactual thinking and the first instinct fallacy by Kruger, Wirtz, and Miller (2005)
A Review of Counterfactual thinking and the first instinct fallacy by Kruger, Wirtz, and Miller (2005) BY RICHARD THRIPP EXP 6506 UNIVERSITY OF CENTRAL FLORIDA NOVEMBER 5, 2015 What is counterfactual thinking?
More informationMath 124: Modules 3 and 4. Sampling. Designing. Studies. Studies. Experimental Studies Surveys. Math 124: Modules 3 and 4. Sampling.
What we will do today Five Experimental Module 3 and Module 4 David Meredith Department of Mathematics San Francisco State University September 24, 2008 Five Experimental 1 Five 2 Experimental Terminology
More informationSampling Controlled experiments Summary. Study design. Patrick Breheny. January 22. Patrick Breheny Introduction to Biostatistics (BIOS 4120) 1/34
Sampling Study design Patrick Breheny January 22 Patrick Breheny to Biostatistics (BIOS 4120) 1/34 Sampling Sampling in the ideal world The 1936 Presidential Election Pharmaceutical trials and children
More informationStatistical Sampling: An Overview for Criminal Justice Researchers April 28, 2016
Good afternoon everyone. My name is Stan Orchowsky, I'm the research director for the Justice Research and Statistics Association. It's my pleasure to welcome you this afternoon to the next in our Training
More informationCausality and Treatment Effects
Causality and Treatment Effects Prof. Jacob M. Montgomery Quantitative Political Methodology (L32 363) October 24, 2016 Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 1 / 32 Causal
More informationMath 124: Module 3 and Module 4
Experimental Math 124: Module 3 and Module 4 David Meredith Department of Mathematics San Francisco State University September 24, 2009 What we will do today Experimental 1 What we will do today Experimental
More informationCreative Commons Attribution-NonCommercial-Share Alike License
Author: Brenda Gunderson, Ph.D., 2015 License: Unless otherwise noted, this material is made available under the terms of the Creative Commons Attribution- NonCommercial-Share Alike 3.0 Unported License:
More informationDesigned Experiments have developed their own terminology. The individuals in an experiment are often called subjects.
When we wish to show a causal relationship between our explanatory variable and the response variable, a well designed experiment provides the best option. Here, we will discuss a few basic concepts and
More informationStat 13, Intro. to Statistical Methods for the Life and Health Sciences.
Stat 13, Intro. to Statistical Methods for the Life and Health Sciences. 0. SEs for percentages when testing and for CIs. 1. More about SEs and confidence intervals. 2. Clinton versus Obama and the Bradley
More informationDo Your Online Friends Make You Pay? A Randomized Field Experiment on Peer Influence in Online Social Networks Online Appendix
Forthcoming in Management Science 2014 Do Your Online Friends Make You Pay? A Randomized Field Experiment on Peer Influence in Online Social Networks Online Appendix Ravi Bapna University of Minnesota,
More informationAnalysis A step in the research process that involves describing and then making inferences based on a set of data.
1 Appendix 1:. Definitions of important terms. Additionality The difference between the value of an outcome after the implementation of a policy, and its value in a counterfactual scenario in which the
More informationVersion No. 7 Date: July Please send comments or suggestions on this glossary to
Impact Evaluation Glossary Version No. 7 Date: July 2012 Please send comments or suggestions on this glossary to 3ie@3ieimpact.org. Recommended citation: 3ie (2012) 3ie impact evaluation glossary. International
More informationUnderlying Theory & Basic Issues
Underlying Theory & Basic Issues Dewayne E Perry ENS 623 Perry@ece.utexas.edu 1 All Too True 2 Validity In software engineering, we worry about various issues: E-Type systems: Usefulness is it doing what
More informationWho? What? What do you want to know? What scope of the product will you evaluate?
Usability Evaluation Why? Organizational perspective: To make a better product Is it usable and useful? Does it improve productivity? Reduce development and support costs Designer & developer perspective:
More informationVALIDITY OF QUANTITATIVE RESEARCH
Validity 1 VALIDITY OF QUANTITATIVE RESEARCH Recall the basic aim of science is to explain natural phenomena. Such explanations are called theories (Kerlinger, 1986, p. 8). Theories have varying degrees
More informationPopulation. population. parameter. Census versus Sample. Statistic. sample. statistic. Parameter. Population. Example: Census.
Population Population the complete collection of ALL individuals (scores, people, measurements, etc.) to be studied the population is usually too big to be studied directly, then statistics is used Parameter
More informationRandomized Evaluations
Randomized Evaluations Introduction, Methodology, & Basic Econometrics using Mexico s Progresa program as a case study (with thanks to Clair Null, author of 2008 Notes) Sept. 15, 2009 Not All Correlations
More informationInstrumental Variables I (cont.)
Review Instrumental Variables Observational Studies Cross Sectional Regressions Omitted Variables, Reverse causation Randomized Control Trials Difference in Difference Time invariant omitted variables
More informationRisk Aversion in Games of Chance
Risk Aversion in Games of Chance Imagine the following scenario: Someone asks you to play a game and you are given $5,000 to begin. A ball is drawn from a bin containing 39 balls each numbered 1-39 and
More informationAugust 29, Introduction and Overview
August 29, 2018 Introduction and Overview Why are we here? Haavelmo(1944): to become master of the happenings of real life. Theoretical models are necessary tools in our attempts to understand and explain
More informationSTA Module 1 Introduction to Statistics and Data
STA 2023 Module 1 Introduction to Statistics and Data 1 Learning Objectives Upon completing this module, you should be able to: 1. Classify a statistical study as either descriptive or inferential. 2.
More informationIntroduction: Research design
Introduction: Research design 17.871 1 Statistics 1. a. In early use, that branch of political science dealing with the collection, classification, and discussion of facts (especially of a numerical kind)
More informationChapter 9 Experimental Research (Reminder: Don t forget to utilize the concept maps and study questions as you study this and the other chapters.
Chapter 9 Experimental Research (Reminder: Don t forget to utilize the concept maps and study questions as you study this and the other chapters.) In this chapter we talk about what experiments are, we
More informationConducting Strong Quasi-experiments
Analytic Technical Assistance and Development Conducting Strong Quasi-experiments Version 1 May 2015 This report was prepared for the Institute of Education Sciences (IES) by Decision Information Resources,
More informationAP Psychology -- Chapter 02 Review Research Methods in Psychology
AP Psychology -- Chapter 02 Review Research Methods in Psychology 1. In the opening vignette, to what was Alicia's condition linked? The death of her parents and only brother 2. What did Pennebaker s study
More informationAbstract Title Page Not included in page count. Title: Analyzing Empirical Evaluations of Non-experimental Methods in Field Settings
Abstract Title Page Not included in page count. Title: Analyzing Empirical Evaluations of Non-experimental Methods in Field Settings Authors and Affiliations: Peter M. Steiner, University of Wisconsin-Madison
More informationIntroduction to Program Evaluation
Introduction to Program Evaluation Nirav Mehta Assistant Professor Economics Department University of Western Ontario January 22, 2014 Mehta (UWO) Program Evaluation January 22, 2014 1 / 28 What is Program
More informationExperimental Methods. Policy Track
Experimental Methods Policy Track East Asia Regional Impact Evaluation Workshop Seoul, South Korea Nazmul Chaudhury, World Bank Reference Spanish Version & French Version also available, Portuguese soon.
More informationPrevious Example. New. Tradition
Experimental Design Previous Example New Tradition Goal? New Tradition =? Challenges Internal validity How to guarantee what you have observed is true? External validity How to guarantee what you have
More informationQuizzes (and relevant lab exercises): 20% Midterm exams (2): 25% each Final exam: 30%
1 Intro to statistics Continued 2 Grading policy Quizzes (and relevant lab exercises): 20% Midterm exams (2): 25% each Final exam: 30% Cutoffs based on final avgs (A, B, C): 91-100, 82-90, 73-81 3 Numerical
More informationRecognizing Ambiguity
Recognizing Ambiguity How Lack of Information Scares Us Mark Clements Columbia University I. Abstract In this paper, I will examine two different approaches to an experimental decision problem posed by
More informationRecent advances in non-experimental comparison group designs
Recent advances in non-experimental comparison group designs Elizabeth Stuart Johns Hopkins Bloomberg School of Public Health Department of Mental Health Department of Biostatistics Department of Health
More informationSampling for Success. Dr. Jim Mirabella President, Mirabella Research Services, Inc. Professor of Research & Statistics
Sampling for Success Dr. Jim Mirabella President, Mirabella Research Services, Inc. Professor of Research & Statistics Session Objectives Upon completion of this workshop, participants will be able to:
More informationResearch Process. the Research Loop. Now might be a good time to review the decisions made when conducting any research project.
Research Process Choices & Combinations of research attributes Research Loop and its Applications Research Process and what portions of the process give us what aspects of validity Data Integrity Experimenter
More informationobservational studies Descriptive studies
form one stage within this broader sequence, which begins with laboratory studies using animal models, thence to human testing: Phase I: The new drug or treatment is tested in a small group of people for
More informationMargin of Error = Confidence interval:
NAME: DATE: Algebra 2: Lesson 16-7 Margin of Error Learning 1. How do we calculate and interpret margin of error? 2. What is a confidence interval 3. What is the relationship between sample size and margin
More informationPsychology 205, Revelle, Fall 2014 Research Methods in Psychology Mid-Term. Name:
Name: 1. (2 points) What is the primary advantage of using the median instead of the mean as a measure of central tendency? It is less affected by outliers. 2. (2 points) Why is counterbalancing important
More informationIntroduction: Research design
Introduction: Research design 17.871 1 Statistics 1. a. In early use, that branch of political science dealing with the collection, classification, and discussion of facts (especially of a numerical kind)
More informationEconomics 345 Applied Econometrics
Economics 345 Applied Econometrics Lab Exam Version 1: Solutions Fall 2016 Prof: Martin Farnham TAs: Joffré Leroux Rebecca Wortzman Last Name (Family Name) First Name Student ID Open EViews, and open the
More informationValidity and Quantitative Research. What is Validity? What is Validity Cont. RCS /16/04
Validity and Quantitative Research RCS 6740 6/16/04 What is Validity? Valid Definition (Dictionary.com): Well grounded; just: a valid objection. Producing the desired results; efficacious: valid methods.
More informationDo Get-Out-The-Vote Calls Reduce Turnout? The Importance of Statistical Methods for Field Experiments
Do Get-Out-The-Vote Calls Reduce Turnout? The Importance of Statistical Methods for Field Experiments Kosuke Imai Department of Politics, Princeton University August 19, 2003 Abstract In their landmark
More informationExperimental Research Design
Experimental Research Design OJMARRH MITCHELL The design of research is fraught with complicated and crucial decisions. Researchers must decide which research questions to address, which theoretical perspective
More informationMethodology for Non-Randomized Clinical Trials: Propensity Score Analysis Dan Conroy, Ph.D., inventiv Health, Burlington, MA
PharmaSUG 2014 - Paper SP08 Methodology for Non-Randomized Clinical Trials: Propensity Score Analysis Dan Conroy, Ph.D., inventiv Health, Burlington, MA ABSTRACT Randomized clinical trials serve as the
More informationTEACHING HYPOTHESIS TESTING: WHAT IS DOUBTED, WHAT IS TESTED?
Australasian Journal of Economics Education Vol. 5. Numbers 3 & 4, 008 1 TEACHING HYPOTHESIS TESTING: WHAT IS DOUBTED, WHAT IS TESTED? Gordon Menzies 1 University of Technology, Sydney E-mail: gordon.menzies@uts.edu.au
More informationResearch Designs. Inferential Statistics. Two Samples from Two Distinct Populations. Sampling Error (Figure 11.2) Sampling Error
Research Designs Inferential Statistics Note: Bring Green Folder of Readings Chapter Eleven What are Inferential Statistics? Refer to certain procedures that allow researchers to make inferences about
More informationCHAPTER NINE DATA ANALYSIS / EVALUATING QUALITY (VALIDITY) OF BETWEEN GROUP EXPERIMENTS
CHAPTER NINE DATA ANALYSIS / EVALUATING QUALITY (VALIDITY) OF BETWEEN GROUP EXPERIMENTS Chapter Objectives: Understand Null Hypothesis Significance Testing (NHST) Understand statistical significance and
More information