Quasi-experimental analysis Notes for "Structural modelling". Martin Browning Department of Economics, University of Oxford Revised, February 3 2012 1 Quasi-experimental analysis. 1.1 Modelling using quasi-experiments. I prefer the term causal reduced form (CRF) analysis to quasi-experiments or natural experiments. CRF s attempt to mimic a randomised clinical trial ( a controlled experiment ). For this, there is some factor/variable which is changed randomly for one group (the treatment group) but not for another (the control group). Then some outcome of interest is compared between the two groups. If the assignment to treatment and control is genuinely random then we have a controlled experiment. This is quite rare in economics. If we do not have genuine controlled assignment then we have to rely on randomisation that is either natural or arises from some human agency. In a review of the natural experimental literature, Rosenzweig and Wolpin (Journal of Economic Literature, 2000) identify only ve sources of natural experimental variation: these include the weather, having twins and the sex of a new born. If these were the only sources of exogenous variation that we allowed then we would not be able to address the vast majority of questions we have. For example, how is variation in any of these going to help in determining the e ects of unemployment bene ts on unemployment duration? In practice most researchers in this area use quasi-experimental variation to determine causal e ects. These are variations in how some people are treated relative to others that may be thought of as random conditional on observables. Considering the unemployment bene t example, the quasi-experimental variation might come because of a change in bene t levels or rules. Once we have some quasi-experimental variation there are a wide variety of statistical methods for the analysis. These include: matching (a nonparametric form of OLS); di in di ; regression discontinuity; timing of events, instrumental variable estimation etc.. The best reference for these things is the book Mostly harmless econometrics by Angrist and Pischke. Wooldridge, chapter 18 is also recommended for the details. 1
For a counter-blast to the tidal wave of CRF analysis, see Instruments of development... by Angus Deaton. 1.2 Example of a natural experiment. In the data it is seen that couples whose rst child is a girl are more likely to divorce. Here the treatment is having a girl and the outcome is duration to divorce. This is causal (in Rubin sense) since the assignment of sex of child is random. It is also a reduced form: in a model of divorce we might want to control for the sex of a rst child but the fact itself does not tell us much. We need a structural model to infer why the divorce probability is higher if a girl. This probably requires an explicit model of who initiates the divorce. Such a model would have other implications that are subject to testing. Note, as well, that this source of exogenous variation does not tell us anything about why people without children divorce. As well as being potentially useful in a model of divorce, this could be in a model that had divorce as an endogenous independent variable. An example would be men s wages at age 40 in which marital status was included as an explanatory variable. Then having a female child rst is a candidate instrument for divorce. Think about whether it is likely to be a good instrument. 1.3 Example of a quasi-experiment. A classic example of a quasi-experiment arises in the research into the e ect of class size on children s scholastic outcomes. This is an important issue since smaller class sizes obviously cost more and we would not want to have small classes if there is no bene t. Suppose we have a data set of classes with the information on the number of children in each class and the average end-of-year test score of the children in each class. A simple regression of the test score on class size does not have any obvious interpretation since class size depends on many factors that may also be correlated with test scores (that is, class size is endogenous). Genuine experimental assignment of class sizes would be impractical (and unethical). So we have to look for something that is correlated with class size but uncorrelated with test scores. One example of such variation arises from the fact that most school authorities have an upper bound on class size. If the number of children exceeds this bound then two classes are formed that are much smaller. This gives quasi-experimental variation in class sizes. Suppose the maximum class size is 25. This gives a sharp distinction between students who are in a year with 25 students (one class) and a year with 26 students (two classes each of 13). This example is instructive. Suppose, for example, that using this analysis, we found that a decrease in class size of 12 students raised the average grade at the end of a course by 3 marks. How can we use this? Presumably, the interesting application is for predicting the e ect of a change in school policies that lowered the mandated maximum class size, from, say, 26 to 24. The CRF analysis does not help much. First, the e ect is found for a localised e ect 2
whereas the policy change would a ect all schools. Second, the e ect is for a large change in class size (25 to 13) as against the small size proposed by the government. This requires interpolation - is the e ect linear, quadratic or what? Third, we only estimate a mean e ect but what of the distributional impact? Maybe small classes are better for weaker students and we may value increases in their grades more. 1 Finally, the e ect does not hold everything else constant. To see this suppose that the average grade for class i is q i and this depends linearly on four factors, class size, c i ; teacher quality, t i ; parental input at home, p i and the average ability of the students in the class, a i : q i = 0 + 1 c i + 2 t i + 3 p i + 4 a i (1) The parameter of interest is 1. This is the partial derivative @q=@c. This is not what we observe in the class splitting analysis since all of the other factors could also change: school administrators might assign a higher quality teacher to larger classes: @t @c > 0 parents might help their children more at home if they are in a big class: @p @c > 0 the selection into the school may not be random. If parents of high ability children move their children to schools with smaller class sizes, we have @a @c < 0 The observed e ect is the total derivative: dq dc = @t 1 + 2 @c + @p 3 @c + @a 4 @c It is obvious that even if 1 is zero, the overall e ect may be positive or negative, unless the other factors do not a ect grades ( k = 0 for k = 2; 3; 4) or the other factors do not change. Without a model of how other relevant factors will respond, we cannot hope to identify the parameter of interest for policy purposes. Importantly, the changes in other factors induced by the quasi-experiment might be very di erent to those induced by an nationwide change. The class size CRF analysis does, however, impose a discipline on a structural model. However the latter is set up, it must predict the observed change in the average grade for the class size changes seem in the quasi-experiment. In that sense, the quasi-experimental result is a reduced form. 1 The converse is also possible if teachers in large classes have to devote too much time to weaker students. (2) 3
2 Relative merits of CRF and structural modelling 2.1 Advantages of CRF analyses 1. They do not require much theory. Generally the theory is to justify conditional independence assumptions (the randomness of the treatment). These justi cations are usually relatively informal (and sometimes not even stated explicitly!). 2. It is easy for non-econometricians to understand the analysis. 3. Robust (nonparametric). 4. They sometimes give credible answers to well de ned and interesting questions. 5. They are reduced forms and can (and should) be used in estimation of structural models. 2.2 Disadvantages of CRF analyses: 1. Not always possible to nd suitable control groups. For many policies we cannot conduct any analysis. Or we can only identify uninteresting e ects. 2. Usually only mean e ects are identi ed. Even analyses that allow for heterogeneous e ects do not identify enough to allow us to incorporate estimates in reasonable policy simulations. 3. No possibility of conducting welfare analyses. 4. Only give answers to what would have happened if policy had not been implemented (ex post analysis). Cannot robustly be extended to the analysis of novel policies (ex ante analysis). No interpolation (or extrapolation) is allowed. 2.3 Advantages of structural analyses: 1. All assumptions are explicit and out there for everyone to see. 2. The results are in a form that can be used in simulations or in a CGE or a DSGE. 3. Portability. If the model is fully parametric then a researcher can report all of the results so that anyone else can use the results without having the original data. 4. The estimates can be used for ex ante policy analysis as well as ex post analysis. 5. Coherent welfare analysis is possible. 4
2.4 Disadvantages of structural analyses: 1. All assumptions are explicit and out there for everyone to see. 2. Non-trivial models are very di cult. Thought intensive. Computer intensive. 3. Di cult for non-economists to understand. 4. Non-robust. Estimates of outcomes of interest may be sensitive to general model formulation or to particular assumptions concerning functional forms. 5. Crude in the sense that we cannot handle much in the way of complexity. 5