Lecture II: Difference in Difference. Causality is difficult to Show from cross

Transcription

1 Review Lecture II: Regression Discontinuity and Difference in Difference From Lecture I Causality is difficult to Show from cross sectional observational studies What caused what? X caused Y, Y caused X Omitted Variable Bias/Confounding In some cases you can say whether the estimate is an upper-bound or lower bound estimate Other times impossible to sign bias since omitted variables bias the coefficient of interest positively and negatively. Net effect impossible to determine a-priori 2 Review (cont.) Discussed Randomized Control Trials as a simple (but not necessarily practical) way to solve the causality problem Randomization works because we can be sure about temporal precedence Randomization works because treatment t t and control groups are balanced on observables and un-observables Regression Discontinuity (RD) Arbitrary Threshold determines whether or not a unit gets assigned to Treatment or Control group Anti-Discrimination law only applies to firms with at least 15 employees Cancer screening guidelines recommend that screening begin for asymptomatic individuals at a certain age mammography screening begins for asymptomatic women at age 4 colorectal cancer screening begins for asymptomatic individuals at age 5 prostate cancer screening begins for asymptomatic men at age 5 3 4

2 Regression Discontinuity (cont.) In this research design being above or below some threshold implies you are in the treatment group Infants below a certain birth weight receive different types of care Look for a change in magnitude of the outcome variable right around this prespecified threshold Regression Discontinuity (cont.) RD can provide unbiased estimates of the relationship between X and Y The key is that we know precisely the assignment mechanism to the treatment and control groups Primary assumption is that all the other variables that affect the outcome of interest change smoothly across the threshold This is verifiable in the data 5 6 Regression Discontinuity (cont.) Important to note there is NO assignment of individuals to treatment and control groups for the same set of covariates This is unlike what happens in most matching estimators where we find very comparable groups based on set of covariates except that some in the group get a treatment and some don t So we have to use the fact that we observe units with values very close to the threshold to get an estimate of the effect of X on Y Where Do Discontinuities come from (I)? Discontinuities arise from a variety of situations Merit High test score results in the treatment being allocated Thresholds can be determined by combinations of variables (GPA, Grades, Ranking etc.) Cost-effectively target an intervention Screening for cancer Screening is costly-want it to be based on the incidence of disease in the population Environmental interventions based on level of pollution 7 8

3 Where Do Discontinuities Come from? (II) Resources can t be allocated continuously Class size is a discrete variable RDdesigninPracticeI in Two types of regression discontinuity Sharp Regression Discontinuity Probability(Treatment) = 1 if X i >= C All units with X >= C are assigned to the treatment All units with X< C are assigned to control RD Estimate is: E[Y X>=C] E[Y X<C] 9 1 Probability of Assignment - Sharp Regression Discontinuity i i Design Regression Discontinuity Example (Legal Drinking Age) probability of treatment Treatment Criteria 11 12

4 RD in practice II Fuzzy Regression Discontinuity Design Probability of receiving treatment does not have to be 1 at the threshold For example individuals above some threshold could be offered a treatment The offer does not lead all individuals to take up treatment As an example think of a voucher scheme that allows people to move to different neighborhoods. For some individuals voucher amount offered is not enough to get them to comply Probability of Assignment-Fuzzy Regression Discontinuity i i Design Treatment Criteria 1 probability of treatment Potential RD Outcome RD in Practice III Outcome Treatment criteria Estimates from RD design are useful only for providing treatment effects for sub- populations i.e. the subpopulation around the threshold RD design does not provide an overall average treatment effect like what we get from Randomized control trials Implies limited external validity 15 16

5 RD in practice III (cont.) Evaluating the outcome Use graphs If you don t see a change in the mean of the outcome variable around the threshold then there is likely no effect First important specification check Would be nice to verify whether decision rule is strictly being followed or is being gpotentially manipulated Administrators of programs have leeway and can potentially confound the research design Researcher no longer knows the assignment mechanism You can check for manipulation by looking at covariates of individuals near the threshold Also check for manipulation by looking at frequencies of variables RD in Practice III (cont.) Second important specification check: Look for other discontinuities in the independent variables that are comparable in magnitude to the one found near the specified threshold If you find others that you can t explain then you question this design RD in Practice III (cont.) Third important specification check distributions of other covariates near pre- specified threshold should be similar RD in Practice (IV) How do you formalize this in the regression framework? Depending on the type of dependent variable (ordinal versus categorical) you can generally use standard regression techniques Ordinary least squares for ordinal variables Logistic regression for categorical variables Running the regression Two key dependent variables The ordinal forcing variable A dummy variable for being in the group being affected by the treatment Treatment effect estimated from coefficient on the dummy variable 19 2

6 RD in Practice (V) What sample should you run the regression on? Ideally you want to stay very close to the discontinuity. Why? But staying close to the discontinuity means you might have to drop lots of observations from your analysis Tradeoff between efficiency (standard errors) and bias (the true relationship between X and Y) Specification checks Adjusting for the other covariates in the regression framework should not change the treatment effect identified in the simple regression Run a separate regression including other observable variables Compare treatment effects generated from the two specifications Alternatively use the covariates as dependent variables in the simple regression and show that the dummy variable is not related the covariate Study Designs Longitudinal Time Component X 1 - Observe data on Y only from post ttreatment t t(x) period 1 X 2 Observe data on Y from pre and post treatment periods 1 2 X 3 Observe data on Y from pre and post treatment; observe a longer pre period Study Designs (cont.) Common features Temporal precedence is established NO CONTROL GROUP» No counterfactual» No way to understand how your outcome would have changed if the treatment t t had not been applied Difference in Difference I 1 X is the pre-period period treatment group data 2 is the post intervention treatment group data 3 is the pre-period control group data 4 is the post intervention control group data 23 24

7 Difference in Difference I (cont.) Difference in Difference I (cont.) Let s Compare to 1 X 2 design How is this different from difference-in-difference design?` No control group, leads to the strong assumption that over time, without an intervention, dependent variable of interest would not have changed 1 X Accounts for the fact that the independent variable might change even if there were no intervention The control group provides the counterfactual Simplest representation is 2 X 2 Table Diff. in Diff. Estimate= [E(Y T1 ) E(Y T )] [E(Y C1 ) E(Y C )] Same result even if you calculate = [E(Y T1 ) E(Y C1 )] [E(Y T ) E(C T )] E(Y T ) E(Y T1 ) E(Y C ) E(Y C1 ) Difference in Difference I (cont.) You are subtracting out the change in the control group from the change in the treatment group If treatment had no effect what does this imply about the magnitudes of the two terms? The two differences are equal If the treatment had an effect then either the first term is bigger than the second term (positive effect) or the second term is bigger than the first term (negative effect) Difference in Difference II Why is Diff. and Diff. powerful? MAIN REASON: We have a control group and temporal ordering is clear Another problem with cross-sectional studies is that we worry about unobserved and hard to measure differences between the treatment and control group In the difference in difference estimate, Unobserved differences across treatment and control that stay constant over time are differenced out Another way of saying this is that these unobserved unchanging characteristics effect the level but not the changes 27 28

8 Difference in Difference III Problems with Difference in Difference Estimation Lets remember What made RCT powerful We knew the assignment mechanism: RANDOMIZATION Note that there is no randomization in Diff. in Diff Consequently we are still concerned with some of the usual problems from cross-sectional studies Difference in Difference III (cont.) Main Concern is History How can we be sure that other interventions are also not simultaneously occurring with treatment? For ex. Some states in an effort to reduce smoking might enact anti-smoking laws in public spaces Very possible that the states that enact anti-smoking laws simultaneously enact other anti-smoking measures as well (increase advertising, increase taxes etc.) For these changes not to bias the difference-in difference estimate we would have to argue that the control group also enacted these other changes at the same time.» Or we would have to adjust for them explicitly in the regression 29 3 Difference in Difference III (cont.) Specification Checks Plot pre intervention trends over time for dependent variables separately by treatment and control groups. If the trends are parallel in treatment and control groups and you see sudden change after intervention then you are potentially safe If trends are not parallel then possible bias from other sources Create False Treatments and Redo estimation For ex. If intervention happened in 199, assign intervention in treatment group to 1989 and see if you still find an effect If you find an effect then likely that something else is driving your findings Difference in Difference III (cont.) Use an outcome that shouldn t be affected by the intervention and redo estimation Estimate the impact of second hand smoke on child health Policy change that impacts that are targeted to reducing smoking specific age groups Examine smoking rates in age groups outside the range of targeted age groups

9 Cites Bruce Meyer, Natural and Quasi-Experiments in Economics, Journal of Business and Economic Statistics, Vol. 13 (2), pp Card and Krueger, Minimum Wages and Employment: A case study of the fast-food industry in NJ and Pennsylvania, American Economic Review, Vol. 84(4), pp Imbens, Lemieux, Regression Discontinuity Designs: A Guide to Practice, NBER Working Paper,