Experimental Political Science and the Study of Causality

Transcription

1 A/ Experimental Political Science and the Study of Causality From Nature to the Lab REBECCA B. MORTON New York University KENNETH C. WILLIAMS Michigan State University CAMBRIDGE UNIVERSITY PRESS

2 Acknowledgments page xv I INTRODUCTION 1 The Advent of Experimental Political Science The Increase in Experimentation in Political Science Is the Increase in Experimentation Real? How "New" Is Experimental Political Science? Political Science Experiments in the 1950s and 1960s The Rise in Experimentation in the 1970s and 1980s Not New Research, But New in Prominence Is It the Artificiality? Why Experiments Have Received More Interest Is It Technology? Inability of Existing Data to Answer Important Causal Questions New Research Questions Is Political Science Now an Experimental Discipline? Why Study Causality? What Is Experimental Reasoning? Using Experiments as a Guide for Research Broadly The Welcoming Discipline Heritage Matters The Advantages of the Welcoming Nature The Disadvantages of the Welcoming Nature Purpose of This Book Our Audience Plan of This Book 26

3 vi II EXPERIMENTAL REASONING ABOUT CAUSALITY 2 Experiments and Causal Relations Placing Experimental Research in Context Technical Background Causes of Effects Versus Effects of Causes Causes of Effects and Theoretical Models Effects of Causes and Inductive Research An Example: Information and Voting Setting Up an Experiment to Test the Effects of a Cause The Data We Use What Is an Experiment? Examples of Information and Voting Experiments What Is Not an Experiment? Experimental Versus Observational Data Chapter Summary Appendix: Examples 58 3 The Causal Inference Problem and the Rubin Causal Model Variables in Modeling the Effects of a Cause The Treatment Variable Variables That Affect Treatments The Dependent Variable Other Variables and Variable Summary Manipulations Versus Treatments Why Are Manipulations Sometimes Called Treatments? When Treatment Variables Cannot Be Manipulated When Manipulations Affect Treatments Only Indirectly The Rubin Causal Model Defining Causal Effects The Causal Inference Problem and Observational Data Causal Inference Problem and Experimental Data Design Versus Analysis Measures of Causality Average Unconditional Treatment Effects Average Conditional Treatment Effects Other Treatment Effects 95

4 vii 3.6 The Stable Unit Treatment Value Assumption What Does It Mean for <5; to Represent the Causal Effect? Implications of SUTVA for Inductive Experimental Work Advantages of RCM Chapter Summary 99 Controlling Observables and Unobservables Control in Experiments Controlling Observables in Experiments Controlling Unobservables in Laboratory Experiments Control Functions in Regressions When an Observable Variable Cannot Be Controlled or Manipulated A Digression on Dealing with Voting as a Dependent Variable The Switching Regression Model Selection on the Observables or Ignorability of Treatment How Reasonable Is the Ignorability of Treatment Assumption? When Ignorability of Treatment Holds but Observables Affect Potential Choices Political Science Examples Using Regression Control Methods with Experimental Data Using Time to Control for Confounding Unobservables Time and Data Structures Panel Data and Control of Unobservables Political Science Example: A Panel Study of Turnout Propensity Scores in Regressions Use of Controls and Propensity Scores Without Regression Control by Matching Propensity Scores and Matching Nonparametric Preprocessing and Matching Political Science Example Causal Effects Through Mediating Variables Chapter Summary 138

5 viii 5 Randomization and Pseudo-Randomization RCM-Based Methods and Avoiding Confounding The Ideal Instrumental Variable Definition of an Ideal IV Is Random Assignment of Manipulations in Experiments an Ideal IV? When Assignment of Treatment Is Not Independent of Potential Choices Potential Violations of Independence in Random Assignment Using Experimental Design to Solve Independence Problems Solving Independence Problems After an Experiment or with Observational Data When an IV or Assignment Is Not a Perfect Substitute for Treatment Potential Problems of Substitutability in Random Assignment Using Experimental Design to Solve Substitutability Problems Solving Substitutability Problems After the Experiment Missing Data When Might Data Be Missing? Using Experimental Design to Reduce Missing Data Dealing with Missing Data After an Experiment Manipulation and Time Chapter Summary Adding in Formal Theory Formal Theory and Causality What Is a Formal Model? Using an RCM Approach with Predictions from Nonformal Models The Theoretical Consistency Assumption Addressing the Theoretical Consistency Assumption Using an RCM Approach with Predictions from Formal Models The FTA Process Essence of FTA Theory and Stress Tests 204

6 ix 6.5 FTA and the Design Stage of Experimental Research The Formal Model: An Example Summary of the Features of the Model Predictions of the Model Designing a Theory Test Designing a Stress Test FTA and the Analysis Stage The Growing Importance of the Analysis Stage The Analysis Stage and FTA Experiments FTA, Field Experiments, and Observational Data Chapter Summary 250 III WHAT MAKES A GOOD EXPERIMENT? Validity and Experimental Manipulations Validity of Experimental Research Deconstructing Internal Validity Statistical Validity Causal Validity and the Identification Problem Construct Validity Summary of Internal Validity Deconstructing External Validity External, Statistical, and Ecological Validity Establishing External Validity Is External Validity Possible Without Satisfying Internal Validity? Chapter Summary 276 Location, Artificiality, and Related Design Issues Levels of Analysis Location of Experiments Survey Experiments Internet Experiments Lab-in-the-Field Experiments Is Field More Than a Location? Why Laboratories? Why the Field or the Internet? Which Is Better? Baselines and Comparisons in Experiments Making Comparisons Determining the "Right" Baseline 310

7 x Experiments with Theoretical Baselines Multiple Comparisons Artificiality in Experiments Experimental Effect Dealing with Experimental Effects Desirability of Artificiality Experimental Cross-Effects Experiments as a Behavioral Testbed for Methodologies Chapter Summary Choosing Subjects On the Use of Students as Subjects How Often Are Students Used as Subjects? Why Use Students as Subjects? Worries About Students as Subjects Internal Validity and Subject Pools Representativeness and Statistical Validity Construct Validity External Validity and Subject Pools Experiments Conducted with Multiple Subject Pools Meta-Analysis and Subject Pool Comparisons Chapter Summary Money and Subjects Subjects' Motivations Financial Incentives, Theory Testing, and Validity How Financial Incentives Work in Theory Testing Financial Incentives Versus Intrinsic Motivations Is Crowding Out by Financial Incentives a Problem? Induced Value Theory Risk Aversion Risk Aversion and Repetition Other Incentive Mechanisms Home-Grown Values Grades Recruiting Mechanisms Motivating Subjects Without Explicit Incentives Experimental Relevance and Validity Task Information and Validity Chapter Summary 398

8 xi IV ETHICS 11 History of Codes of Ethics and Human Subjects Research Codes of Ethics and Social Science Experiments Early Professional Codes of Ethics The Nuremberg Code Regulation in the United States The Impetus for Regulation The Belmont Report and the Common Rule The Common Rule and Social Science Research The Current System Regulations in Other Countries Cross-Country Research and Regulations Chapter Summary Appendix A: Code of Federal Regulations: Title 45, Public Welfare, Department of Health and Human Services, Part 46, Protection of Human Subjects Subpart A: Basic HHS Policy for Protection of Human Research Subjects Subpart B: Additional Protections for Pregnant Women, Human Fetuses, and Neonates Involved in Research Subpart C: Additional Protections Pertaining to Biomedical and Behavioral Research Involving Prisoners as Subjects Subpart D: Additional Protections for Children Involved as Subjects in Research Appendix B: Categories of Research That May Be Reviewed by the Institutional Review Board (IRB) Through an Expedited Review Procedure Applicability Research Categories Ethical Decision Making and Political Science Experiments Expected Benefits and Costs in Experiments Expectations, Probabilities, and Magnitudes Expected Benefits from Experimental Research Expected Costs from Experimental Research IRB Assessment of Benefits and Risks 473

9 xii 12.2 Other Criteria in the Common Rule Subject Selection Informed Consent Chapter Summary Appendix: Sample Consent Letter for Laboratory Experiments Deception in Experiments Deception in Political Science Experiments What Is Deception? Types of Deception Deceptive Purpose Deceptive Materials and Information Deceptive Identities Arguments for Deception Control over Perceived Artificiality and Subjects' Motivations Experiments on Rare Situations Deception as a Method of Lowering Costs of Experiments Deception as an Educational Benefit The Societal Costs of Prohibiting Deception Deception Is Part of Everyday Life Objections to Deception Ethical Concerns Methodological Concerns Effects of Deception Are Harms from Deception Minimal? Does Deception Affect Subjects' Behavior in Future Experiments? Debriefing and Removing Effects of Deception The Conditional Information Lottery Procedure and Avoiding Deception How Much Deception Should Be Allowed in Political Science Experiments? Chapter Summary 520 V CONCLUSION 14 The Future of Experimental Political Science The Promise of Collaboration The Difficulties Ahead 528

10 xiii 15 Appendix: The Experimentalist's To Do List The Target Population and Subject Pools Nonspecific Target Populations Specific Target Populations Location of the Experiment Traditional Laboratory Experiments Lab-in-the-Field Experiments Internet Experiments Survey Experiments Field Experiments Motivation and Recruitment of Subjects Financial Incentives Related to Subjects'Choices Other Types of Incentives Related to Subjects' Choices Motivating Subjects in Other Ways and the Role of Frames and Scripts Subject Recruitment Relationship to Theory, Manipulations, and Baselines When Working from a Formal Theory When Working from a Nonformal Theory or a Prediction That Is Inspired by a Formal Theory Operationalizing Random Assignment Random Assignment in Laboratories, Both Traditional and in the Field Random Assignment Outside the Laboratory Considering Ethical Issues Involved in the Experiment and Securing IRB Approval Benefit and Cost Evaluation IRB Approval Deception Post-Experimental Analysis 538 References 539 Author Index 571 Subject Index 581