Review Lecture II: Difference in Difference and Regression Discontinuity it From Lecture I Causality is difficult to Show from cross sectional observational studies What caused what? X caused Y, Y caused X Omitted Variable Bias/Confounding In some cases you can say whether the estimate is an upper-bound or lower bound estimate Other times impossible ibl to sign bias since omitted variables bias the coefficient of interest positively and negatively. Net effect impossible to determine a-priori. 2 Review (cont.) Discussed Randomized Control Trials as a simple (but not necessarily practical) way to solve the causality problem Randomization works because we can be sure about temporal precedence Randomization works because treatment and control groups are balanced on observables bl and un-observables Review (cont.) Also quickly presented some other commonly used research designs X 01 - Observe only data from post treatment (X) 01 X 02 Observe data from pre and post treatment periods 01 02 X 03 Observe data from pre and post treatment; observe a longer pre period Common Feature of all these designs is that there is NO CONTROL GROUP 3 4
Difference in Difference I 01 X 02 03 04 01 is the pre-period treatment group data 02 is the post intervention ti treatment t t group data 03 is the pre-period control group data 04 is the post intervention ti control group data 5 Difference in Difference I (cont.) Let s Compare to 01 X 02 design How is this different from difference-in-difference design?` No control group, leads to the strong assumption that over time, without an intervention, dependent variable of interest would not have changed 01 X 02 03 04 Diff. in Diff. treatment design accounts for the fact that dependent variable might change even if there were no interventioni Similar to the RCT framework the control group provides the counterfactual 6 Difference in Difference I (cont.) Difference in Difference I (cont.) Simplest representation is 2 X 2 Table Diff. in Diff. Estimate= [E(Y T1 ) E(Y T0 )] [E(Y C1 ) E(Y C0 )] Same result even if you calculate = [E(Y T1 ) E(Y C1 )] [E(Y T0 ) E(C T0 )] E(Y T0 ) E(Y T1 ) E(Y C0 ) E(Y C1 ) You are subtracting out the change in the control group from the change in the treatment group If treatment had no effect what does this imply about the magnitudes of the two terms? The two differences are equal If the treatment t t had an effect then either the first term is bigger than the second term (positive effect) or the second term is bigger than the first term (negative effect) 7 8
Difference in Difference II Why is Diff. and Diff. powerful? MAIN REASON: We have a control group Another problem with cross-sectional studies is that we worry about unobserved and hard to measure differences between the treatment t t and control group In the difference in difference estimate, Unobserved differences across treatment and control that stay constant over time are differenced out Another way of saying this is that these unobserved unchanging characteristics effect the level but not the changes 9 Difference in Difference III Problems with Difference in Difference Estimation Lets remember What made RCT powerful We knew the assignment mechanism: RANDOMIZATION Note that there is no randomization in Diff. in Diff Unit of observation (for ex. state) still chooses whether or not to get treatment Choice leads to the potential problem that treatment and control groups are different Consequently we are still concerned with some of the usual problems from cross-sectional sectional studies 10 Difference in Difference III (cont.) Main Concern is History How can we be sure that other interventions are also not simultaneously occurring with treatment? For ex. Some states in an effort to reduce smoking might enact anti-smoking laws in public spaces Very possible that the states that enact anti-smoking laws simultaneously enact other anti-smoking measures as well (increase advertising, i increase taxes etc.) For these changes not to bias the difference-in difference estimate we would have to argue that the control group also enacted these other changes at the same time.» Or we would have to adjust for them explicitly in the the regression 11 Difference in Difference III (cont.) Specification Checks Plot pre intervention trends over time for dependent variables separately by treatment and control groups. IF trends are parallel in treatment and control groups and you see sudden change after intervention then you are potentially safe If trends are not parallel then possible bias from other sources Create False Treatments t and Redo estimation For ex. If intervention happened in 1990, assign intervention in treatment group to 1989 and see if you still find an effect If you find an effect then likely that something else is driving your findings 12
Difference in Difference III (cont.) Use an outcome that shouldn t be affected by the intervention and redo estimation Difference in Difference IV Still Other Concerns Policy intervention is tied to outcome Difference in Difference will overstate true effect Mean reversion is again a potential problem My sense is that this is only a problem for some outcomes (wages is a good ex.) Long term effect ect might be difficult to estimate Estimate is most reliable right after intervention Long term effects likely confounded by other variables Functional Form Means or Logs 13 14 Card & Krueger - An Example What is the Effect of a Minimum Wage increase on employment? Theory says rise in wages should lead to less employment Firms are profit-maximizing already, taxing one input (labor) should lead to a decrease in it s use Card & Krueger (cont.) NJ enacted a state law that increased the minimum wage from 4.25 to 5.05 Effective April 1, 1992 Card and Krueger(1994) use a Diff. in Diff. research design to examine whether this change led to lower employment Control group is Pennsylvania where the minimum wage did not change over this time period 15 16
Card & Krueger (cont.) Card & Krueger Look at the effects in Fast Food Industry, Why? Leading employer of low-wage workers Easier to measure prices, employment and wages in this industry Survey Burger King, KFC, Wendy s and Roy Roger s chains Exclude McDonalds because McDonalds had a poor response rates to surveys in previous work Initial survey conducted in late February and early March 1992, A month before the NJ minimum wage increase Secondary Survey conducted in November and December 1992 Card & Krueger (cont.) Around 80% response rate in pre-period 90% of these 80% responded in post-period One Key question: Is the wage increase in N.J. meaningful? Yes, average starting wage in New Jersey restaurants increased by 10% (4.61 to 5.05) In wave 1: 31% had a starting wage of 4.25 In Pennsylvania, In wave 1, average starting wage in Pennsylvania was 4.63 and In wage two there was no change 17 18 NJ PA Card & Krueger: Results Avg. Full Time Employees Before Avg. Full Time Employees After 20.44 21.03 23.33 21.17 Diff in Diff Estimate: [21.03-20.44] [21.17-23.33] [.59]-[-2.16]=2.75 Standard Error on estimate is 1.36 Conclusion: Estimate is positive but not statistically significant at the 5% level C&K Results (cont.) Lets compare to the 01X02 design Question: Given the C&K data what would you have concluded about the effect of the increase in minimum wage if you used this design? This simpler design would have said that the effect of the minimum wage hike is positive and the magnitude=.59 The Diff. in Diff. estimate also says the effect ect of the minimum wage hike is positive but the magnitude is now 2.7 Including a control group increases the 01X02 estimate by a factor of close to 5 19 20
C&K Results (cont.) Regression Framework Each observation in the data is a store Dependent variable is Change in employment Independent variables include region, chain dummies (burger king etc.) State Dummy for whether or not in New Jersey Regression coefficient on State Dummy: 2.33 On average the law leads to an increase of 2.33 employees But standard d error on the estimate t is 1.33 so not statistically ti ti different from zero C&K-Other Specifications (cont.) Some stores not affected if they are already above the minimum wage Create a GAP variable 0 for stores in Pennsylvania 0 for stores in NJ whose wage is already above 5.05 (5.05-initial wage)/initial wage for other NJ states % increase in wages Again find a positive effect but not statistically significant 21 22 C&K-Other Specifications (cont.) % change in employment in the dependent d variable Exclude management employees Include part time workers in employment Exclude stores along the coast of NJ These stores might have a different seasonal pattern Finally surveyors called some stores in NJ more often to get data. Exclude these stores from sample NONE OF THESE CHANGES AFFECT THE BASIC RESULTS C&K - Other Specifications (cont.) Non Wage-Offsets Offset raise in minimum wage by reducing non-wage compensation (fringe benefits) Main fringe benefit is free and reduced price meals Do not find any changes in this measure Future wage offsets Reduce the rate at which salaries increase Examined the average time to first pay raise 23 24
Regression Discontinuity (RD) Arbitrary Threshold determines whether or not a unit gets assigned to treatment or Control group Anti-Discrimination law only applies to firms with at least 15 employees Rabbinic Scholar Maimonides says Class size cannot exceed 40, if so must group student into smaller classes For ex. 42 students means average class size is 20.5 80 students means two classes of size 40 but 81 students means average class size of 27 Regression Discontinuity (cont.) In this research design being above or below some threshold implies you are in the treatment group Look for a Change in magnitude of the outcome variable right around this pre- specified threshold 25 26 Regression Discontinuity RD can provide unbiased estimates of the relationship between X and Y If the assignment criteria is explicitly followed there are no concerns of omitted variables bias The key is that we know precisely the assignment mechanism to the treatment and control groups Regression Discontinuity (cont.) Important to note there is NO assignment of individuals to treatment and control at the same value of X This is unlike what happens in most matching estimators where we find very comparable groups based on set of covariates except that some in the group get a treatment and some don t So we have to use the fact that we observe units with values very close to the threshold to get an estimate of the effect of X on Y 27 28
Regression Discontinuity (cont.) This research design might make you think of 01 X 02 But it s not? Why is that? There is no time component We are not sure X is the only change happening RD design in Practice I Two types of regression discontinuity Sharp Regression Discontinuity W i = 1 if X i >= C All units with X >= C are assigned the treatment All units with X< C are assigned to control RD Estimate is: E[Y X>=C] E[Y X<C] 29 30 Probability of Assignment Potential RD Outcome 1 12 0.8 10 0.6 0.4 02 0.2 probability of treatment Outco ome 8 6 4 2 0 0 2 4 6 8 10 0 0 1 2 3 4 5 6 7 8 9 10 Treatment Criteria Treatment criteria 31 32
RD in practice II Fuzzy Regression Discontinuity Design Probability of receiving does not have to be 1 at the threshold For ex. Individuals above some threshold could be offered a treatment t t The offer does not lead all individuals to take up treatment As an example think of a voucher scheme that allows people to move neighborhoods. For some individuals voucher amount offered is not enough to get them to comply 33 RD in Practice II (cont.) The key is that offer is only made to individuals who are above threshold Note that it would be wrong to estimate treatment effects by comparing individuals who were offered treatment at threshold C but did not take it, with individuals who were offered treatment at threshold C but did take it. 34 RD in Practice III Estimates from RD design are useful only for providing treatment effects for sub- populations i.e. the subpopulation around the threshold RD design does not provide an overall average treatment t t effect like what we get from Randomized control trials Implies limited external validity 35 RD in practice III (cont.) Threshold h can be determined d by complicated procedures Colleges can create a numerical rating for financial aid that is based on a function of several variables (SAT, family Income, etc.) Would be nice to verify whether decision i rule is strictly being followed or being potentially manipulated Administrators of programs have leeway and can potentially confound the design Researcher no longer knows the assignment mechanism You can check for this by looking at the numbers of individuals near the threshold 36
RD in Practice III (cont.) Use Graphs If you don t see a change in the Mean of the Outcome variable around the threshold then likely no effect First important specification check: Look for other discontinuities in the Dependent variable that are comparable in magnitude to the one found near the specified thresholdh If you find others that you can t explain then you question this design RD in Practice III (cont.) Second Important Specification Check Mean values of other variables near pre- specified threshold should be similar Inclusion of these covariates in a regression specification should NOT change your treatment effect 37 38 Cites Bruce Meyer, Natural and Quasi-Experiments i in Economics, Journal of Business and Economic Statistics, Vol. 13 (2), pp.151-161 Card and Krueger, Minimum Wages and Employment: A case study of the fast-food industry in NJ and Pennsylvania, American Economic Review, Vol. 84(4), pp. 772-793. Imbens, Lemieux, Regression Discontinuity Designs: A Guide to Practice, NBER Working Paper, 13039. 39