Causality and Treatment Effects

Causality and Treatment Effects Prof. Jacob M. Montgomery Quantitative Political Methodology (L32 363) October 24, 2016 Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 1 / 32

Causal inference Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 2 / 32

Overview Thinking about causality Average treatment effects Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 3 / 32

We do not live in a univariate world So far, we ve done univariate inference. Moving forward we will do bivariate inference. This is useful: With the right study design (e.g., experimental research) Why? This is a powerful, powerful way to do research Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 4 / 32

We do not live in a univariate world So far, we ve done univariate inference. Moving forward we will do bivariate inference. This is useful: With the right study design (e.g., experimental research) Why? This is a powerful, powerful way to do research Building block for more advanced models NOTE: In many situations (e.g., civil war), we don t have experimental data. This means we need to think about all relevant predictors and control variables. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 4 / 32

We do not live in a univariate world So far, we ve done univariate inference. Moving forward we will do bivariate inference. This is useful: With the right study design (e.g., experimental research) Why? This is a powerful, powerful way to do research Building block for more advanced models NOTE: In many situations (e.g., civil war), we don t have experimental data. This means we need to think about all relevant predictors and control variables. We will finish the semester by talking about multiple regression as one method for attempting to deal with this. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 4 / 32

Causality In political science we want to make causal claims. X Y What does this mean? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 5 / 32

Causality In political science we want to make causal claims. X Y What does this mean? Let s do this a bit more formally for the case of an experiment (the easiest way to think about it). Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 5 / 32

Causal inference We will use T to represent a treatment variable. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 6 / 32

Causal inference We will use T to represent a treatment variable. For a categorical treatment T i = 1 if unit i receives the treatment 0 if unit i receives the control Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 6 / 32

Causal inference We will use T to represent a treatment variable. For a categorical treatment T i = 1 if unit i receives the treatment 0 if unit i receives the control We let yi 1 represent the outcome of the ith unit if the treatment is given. We let yi 0 represent the outcome of the ith unit if the control is given. One of these is observed, the other is the counterfactual what would have been observed if the other treatment have been given? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 6 / 32

Causal inference We will use T to represent a treatment variable. For a categorical treatment T i = 1 if unit i receives the treatment 0 if unit i receives the control We let yi 1 represent the outcome of the ith unit if the treatment is given. We let yi 0 represent the outcome of the ith unit if the control is given. One of these is observed, the other is the counterfactual what would have been observed if the other treatment have been given? The causal effect of T i will then be yi 1 yi 0 Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 6 / 32

Causal inference We will use T to represent a treatment variable. For a categorical treatment T i = 1 if unit i receives the treatment 0 if unit i receives the control We let yi 1 represent the outcome of the ith unit if the treatment is given. We let yi 0 represent the outcome of the ith unit if the control is given. One of these is observed, the other is the counterfactual what would have been observed if the other treatment have been given? The causal effect of T i will then be yi 1 yi 0 Ex., My theory is that individuals who watched this TV ad will be more likely to vote for Ted Cruz than if they didn t watch it. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 6 / 32

Average treatment effects We cannot measure individual level causal effects Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 7 / 32

Average treatment effects We cannot measure individual level causal effects We can estimate the population average treatment effect by looking at those who received the treatment and those who did not. ATE = mean(yi 1 yi 0) ATE = mean(yi 1) mean(y i 0) Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 7 / 32

Average treatment effects We cannot measure individual level causal effects We can estimate the population average treatment effect by looking at those who received the treatment and those who did not. ATE = mean(yi 1 yi 0) ATE = mean(yi 1) mean(y i 0) Each group acts as a counterfactual for the other Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 7 / 32

Average treatment effects We cannot measure individual level causal effects We can estimate the population average treatment effect by looking at those who received the treatment and those who did not. ATE = mean(yi 1 yi 0) ATE = mean(yi 1) mean(y i 0) Each group acts as a counterfactual for the other Ex., My theory is that those individuals who watched this TV ad will be more likely to vote for Mitt Romney on average than those who didn t watch it. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 7 / 32

The fundamental problem of causal inference The fundamental problem of causal inference is that at most one of yi 0 and yi 1 can be observed. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 8 / 32

The fundamental problem of causal inference The fundamental problem of causal inference is that at most one of yi 0 and yi 1 can be observed. We can think of each of these as potential outcomes. However, we can only observe one. The other is the counterfactual. Estimation of causal effects requires some combination of: Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 8 / 32

The fundamental problem of causal inference The fundamental problem of causal inference is that at most one of yi 0 and yi 1 can be observed. We can think of each of these as potential outcomes. However, we can only observe one. The other is the counterfactual. Estimation of causal effects requires some combination of: certain research designs that approximate potential outcomes Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 8 / 32

The fundamental problem of causal inference The fundamental problem of causal inference is that at most one of yi 0 and yi 1 can be observed. We can think of each of these as potential outcomes. However, we can only observe one. The other is the counterfactual. Estimation of causal effects requires some combination of: certain research designs that approximate potential outcomes randomization Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 8 / 32

The fundamental problem of causal inference The fundamental problem of causal inference is that at most one of yi 0 and yi 1 can be observed. We can think of each of these as potential outcomes. However, we can only observe one. The other is the counterfactual. Estimation of causal effects requires some combination of: certain research designs that approximate potential outcomes randomization statistical adjustment Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 8 / 32

Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 9 / 32

Stand up if your student ID ends in: 1 3 4 6 9 Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 10 / 32

Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 11 / 32

Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 12 / 32

Take away Causal effects rely on unobserved counterfactuals Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 13 / 32

Take away Causal effects rely on unobserved counterfactuals At best, we can estimate average treatment effects comparing those who received a treatment with those who don t. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 13 / 32

Take away Causal effects rely on unobserved counterfactuals At best, we can estimate average treatment effects comparing those who received a treatment with those who don t. In order for this to work, each group must be identical (on average) in every way except the treatment. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 13 / 32

Take away Causal effects rely on unobserved counterfactuals At best, we can estimate average treatment effects comparing those who received a treatment with those who don t. In order for this to work, each group must be identical (on average) in every way except the treatment. The best way to achieve this is through random assignment (i.e., experiments) Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 13 / 32

Take away Causal effects rely on unobserved counterfactuals At best, we can estimate average treatment effects comparing those who received a treatment with those who don t. In order for this to work, each group must be identical (on average) in every way except the treatment. The best way to achieve this is through random assignment (i.e., experiments) What if this assumption is not met? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 13 / 32

Confounders and causality PROBLEM: This only works if the two groups are, on average, otherwise identical Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 14 / 32

Confounders and causality PROBLEM: This only works if the two groups are, on average, otherwise identical If the two groups differ on other factors that also cause yi 1 and y 0 this is a confounding relationship. If this is the case, our counterfactual is wrong and we can make no causal claim. i, Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 14 / 32

Confounders and causality PROBLEM: This only works if the two groups are, on average, otherwise identical If the two groups differ on other factors that also cause yi 1 and y 0 this is a confounding relationship. If this is the case, our counterfactual is wrong and we can make no causal claim. Take Away If you aren t controlling for all other relevant variables (through randomization or statistical methods), you cannot make a valid causal claim. i, Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 14 / 32

Thinking about confounding variables Direct causal relationships: X 1 Y Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 15 / 32

Thinking about confounding variables Direct causal relationships: Spurious relationships: X 1 Y Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 15 / 32

Thinking about confounding variables Direct causal relationships: Spurious relationships: X 1 Y X 2 X 1 AND X 2 Y 2 Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 15 / 32

Thinking about confounding variables Direct causal relationships: Spurious relationships: X 1 Y Chain relationships: X 2 X 1 AND X 2 Y 2 Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 15 / 32

Thinking about confounding variables Direct causal relationships: Spurious relationships: X 1 Y Chain relationships: X 2 X 1 AND X 2 Y 2 X 1 X 2 Y Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 15 / 32

Thinking about confounding variables Direct causal relationships: Spurious relationships: X 1 Y Chain relationships: Multiple causation: X 2 X 1 AND X 2 Y 2 X 1 X 2 Y X 1 Y AND X 2 Y Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 15 / 32

Thinking about confounding variables Direct causal relationships: Spurious relationships: X 1 Y Chain relationships: Multiple causation: X 2 X 1 AND X 2 Y 2 X 1 X 2 Y X 1 Y AND X 2 Y Direct and indirect causation: X 1 Y AND X 1 X 2 AND X 2 Y Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 15 / 32

Write down one of each type of claim for this data. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 16 / 32

Being a responsible causal analyst Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 17 / 32

Less silly examples Vitamin C? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 18 / 32

Less silly examples Vitamin C? Flossing? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 18 / 32

Less silly examples Vitamin C? Flossing? Drinking while pregnant? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 18 / 32

Less silly examples Vitamin C? Flossing? Drinking while pregnant? Fox News? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 18 / 32

At a minimum we need to show... Association What we will be doing this rest of the semester Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 19 / 32

At a minimum we need to show... Association What we will be doing this rest of the semester Correlation, contingency tables, regression coefficients,... Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 19 / 32

At a minimum we need to show... Association What we will be doing this rest of the semester Correlation, contingency tables, regression coefficients,... Association = causation Temporal order For T i to cause Y i it must come before Y in time order Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 19 / 32

At a minimum we need to show... Association What we will be doing this rest of the semester Correlation, contingency tables, regression coefficients,... Association = causation Temporal order For T i to cause Y i it must come before Y in time order Post hoc ergo propter hoc Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 19 / 32

At a minimum we need to show... Association What we will be doing this rest of the semester Correlation, contingency tables, regression coefficients,... Association = causation Temporal order For T i to cause Y i it must come before Y in time order Post hoc ergo propter hoc After this, therefore because of this Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 19 / 32

At a minimum we need to show... Association What we will be doing this rest of the semester Correlation, contingency tables, regression coefficients,... Association = causation Temporal order For T i to cause Y i it must come before Y in time order Post hoc ergo propter hoc After this, therefore because of this Temporal order does = causation (e.g., every superstition ever) Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 19 / 32

At a minimum we need to show... Eliminate alternative explanations Suppose there is an association and a proper time order. We stil cannot infer causation. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 20 / 32

At a minimum we need to show... Eliminate alternative explanations Suppose there is an association and a proper time order. We stil cannot infer causation. Rather, we must test for all alternative explanations. Only if all of these have been resolved can we claim causation. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 20 / 32

At a minimum we need to show... Eliminate alternative explanations Suppose there is an association and a proper time order. We stil cannot infer causation. Rather, we must test for all alternative explanations. Only if all of these have been resolved can we claim causation. How can we do this? Experimental control Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 20 / 32

At a minimum we need to show... Eliminate alternative explanations Suppose there is an association and a proper time order. We stil cannot infer causation. Rather, we must test for all alternative explanations. Only if all of these have been resolved can we claim causation. How can we do this? Experimental control Statistical control (Stay tuned...) Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 20 / 32

Problem: It can be subtle (Mount 2010) Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 21 / 32

A case study Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 22 / 32

Problem: It can be subtle Rogers, Coffman, and Bergman: While the authors control for certain variables, their research only implies there is a relationship between parental involvement and student performance. This caveat is important; the existence of a relationship does not tell us what causes what. Think of it this way: If you had two children, and one was getting A s and the other C s, which of them would you help more? The C student. An outsider, noticing that you ve spent the school year helping only one of your children, might infer that parental help caused that child to earn lower grades. This of course would not be the case, and inferring causation here would be a mistake. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 23 / 32

If you see a surprising result, be skeptical Occasional Notes 35 Nobel Laureates per 10 Million Population 30 25 20 15 10 5 0 r=0.791 P<0.0001 The Netherlands Poland Portugal Greece Japan China Brazil Belgium Canada Italy Spain United States France Australia Sweden Austria Denmark Finland Ireland Norway United Kingdom Switzerland Germany 0 5 10 15 Chocolate Consumption (kg/yr/capita) Figure 1. Correlation between Countries Annual Per Capita Chocolate Consumption and the Number of Nobel Laureates per 10 Million Population. Discussion about 14 Nobel laureates, yet we observe 32. Lecture 13 (QPM 2016) Causality and Treatment Considering Effects that in this instance the October observed 24, 2016 24 / 32

Example: Can social pressure increase turnout? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 25 / 32

Comparing two populations We have two independent samples, and we want to compare them. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 26 / 32

Comparing two populations We have two independent samples, and we want to compare them. Our data will look like this. Variable 1 Variable 2 (Outcome or response) (Explanatory or grouping) 1 1 0 0 1 0 0 1.. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 26 / 32

Example: Social Pressure and Turnout Gerber, A., Green, D., and Larimer C.W. 2008 Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment. American Political Science Review 101(1): 33-48. Obviously, most of these treatments worked. (Note: The analysis of this experiment is actually somewhat more complicated than this due to the way individuals were assigned to treatment groups.) Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 27 / 32

Do Politicians Racially Discriminate? Broad Question Is racial discrimination against blacks still a problem in the political sphere? Puzzle Do legislators discriminate against individual requests for constituency service on the basis of race? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 28 / 32

Tuesday, October 25, 2011 Tuesday, October 25, 2011 Who do you think they are? Jake Mueller DeShawn Jackson Jake Mueller Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 29 / 32

Comparing legislators responsiveness Questions: Do legislators answer a higher proportion of emails from the citizens they believe are white, Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 30 / 32

Comparing legislators responsiveness Questions: Do legislators answer a higher proportion of emails from the citizens they believe are white,... even though both black & white don t signal party affiliation?... even though both black & white signal being Republican?... even though both black & white signal being Democrat? Experiment The sample includes states legislators in 44 U.S. states with a valid e-mail address in September 2008. Race was signaled by randomizing whether the email was signed and sent from an email account with the name Jake Mueller or DeShawn Jackson. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 30 / 32

Comparing legislators responsiveness Questions: Do legislators answer a higher proportion of emails from the citizens they believe are white,... even though both black & white don t signal party affiliation?... even though both black & white signal being Republican?... even though both black & white signal being Democrat? Experiment The sample includes states legislators in 44 U.S. states with a valid e-mail address in September 2008. Race was signaled by randomizing whether the email was signed and sent from an email account with the name Jake Mueller or DeShawn Jackson. The text of the email was also manipulated so as to signal the partisan preference of the email sender. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 30 / 32

Comparing legislators responsiveness Questions: Do legislators answer a higher proportion of emails from the citizens they believe are white,... even though both black & white don t signal party affiliation?... even though both black & white signal being Republican?... even though both black & white signal being Democrat? Experiment The sample includes states legislators in 44 U.S. states with a valid e-mail address in September 2008. Race was signaled by randomizing whether the email was signed and sent from an email account with the name Jake Mueller or DeShawn Jackson. The text of the email was also manipulated so as to signal the partisan preference of the email sender. The cross-tabulation between race & partisan preference gives six treatments (or groups). Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 30 / 32

Comparing legislators responsiveness Questions: Do legislators answer a higher proportion of emails from the citizens they believe are white,... even though both black & white don t signal party affiliation?... even though both black & white signal being Republican?... even though both black & white signal being Democrat? Experiment The sample includes states legislators in 44 U.S. states with a valid e-mail address in September 2008. Race was signaled by randomizing whether the email was signed and sent from an email account with the name Jake Mueller or DeShawn Jackson. The text of the email was also manipulated so as to signal the partisan preference of the email sender. The cross-tabulation between race & partisan preference gives six treatments (or groups). The outcome variable is the response (or lack thereof) to any e-mails. Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 30 / 32

Results: Table 1?? Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 31 / 32

Class business Midterms PS posted today. Quiz for next class over t-tests Lecture 13 (QPM 2016) Causality and Treatment Effects October 24, 2016 32 / 32