Estimation of treatment effects: recent developments and applications

Transcription

1 Empir Econ DOI /s Estimation of treatment effects: recent developments and applications Bernd Fitzenberger Michael Lechner Jeffrey Smith Received: 2 December 2012 / Accepted: 6 December 2012 Springer-Verlag Berlin Heidelberg 2012 Abstract The literature on the estimation of treatment effects has matured in economics. The potential outcomes framework guides the estimation of the causal effect of economic choices or policy interventions. The application of methods from the treatment effects literature has spread from the analysis of the effects of labor market programs and the wage return to education to other areas in economics. This special issue involves methodological developments and state-of-the art applications of B. Fitzenberger (B) University of Freiburg, Platz der Alten Synagoge 2, Freiburg, Germany bernd.fitzenberger@vwl.uni-freiburg.de B. Fitzenberger IFS, London, UK B. Fitzenberger M. Lechner J. Smith IZA, Bonn, Germany B. Fitzenberger ROA, Maastricht, The Netherlands B. Fitzenberger ZEW, Manheim, Germany M. Lechner Swiss Institute for Empirical Economic Research (SEW), University of St. Gallen, Varnbüelstr. 14, 9000 St. Gallen, Switzerland Michael.Lechner@unisg.ch URL: M. Lechner CEPR, London, UK M. Lechner CESifo, Munich, Germany

2 Fitzenberger et al. methods to estimate treatment effects in various areas of economics. The contributions illustrate the emphasis within the treatment effects literature on the separate but related issues of heterogeneous treatment effects and identification. The careful, high-quality substantive applications collected here show just how much applied work on policyrelevant topics has benefitted from the methodological developments in the treatment effects literature. At the same time, many of the substantive papers make important methodological contributions as well. Keywords Estimation of treatment effects Identification Heterogeneity JEL Classification C31 J68 C14 1 Overview This special issue of Empirical Economics presents a strong set of papers, each of which either develops new methodology related to the evaluation of the treatment effects of programs or policies or applies recent methodological developments in new contexts or both. Taken together, the papers in this issue illustrate a number of important themes and show important developments over the last decade relative to the state of the literature a decade ago, e.g. as surveyed by Heckman et al. (1999) regarding the evaluation of labor market programs and as discussed by Card (2003) regarding the estimation of the wage return to education. First, the papers in this issue illustrate the spread of treatment effects methods across fields from their beginning in the literature on evaluation of labor market programs, more recently to the literature on educational interventions, and from there on to essentially every other field within applied economics and to many others outside it. The applications in our papers include both traditional applications in labor and education as well as applications in trade and development. The applications show the high policy relevance of research using these methods. Second, the papers in this special issue illustrate the emphasis within the treatment effects literature on the separate but related issues of heterogeneous treatment effects and identification. Twenty years ago, most papers estimated the effect of programs or policies, implicitly assuming a common effect across all treated or potentially treated units. The intervening years have been unkind to this way of thinking, and justifiably so. All but one of the papers in the special issue pays close attention to heterogeneous treatment effects and to the particular average or averages of such treatment effects identified by the estimation strategy employed. J. Smith Department of Economics, University of Michigan, 238 Lorch Hall, 611 Tappan Street, Ann Arbor, MI , USA econjeff@umich.edu. J. Smith NBER, Cambridge, MA, USA

3 Estimation of treatment effects The papers herein also demonstrate the salutary effects of the overarching emphasis on quality of identification in the treatment effects literature. While much work remains on this score, in general the best applied work throughout economics now justifies, rather than just stating, the assumptions required for identification of the treatment effect of interest. Such justification can take the form of careful analysis of institutions and policy changes, of reference to economic theory, or of building upon previous empirical evidence, or some combination of these. What we do not see (and do not miss!) in our papers, or in the higher quality end of the applied literature more generally, is naïve and mechanical applications of econometric methods. Finally, the papers in our special issue embody the relative intellectual maturity of the treatment effects literature. The past 20 years have seen most of the low-hanging methodological fruit picked clean from the tree of knowledge. Instead, our papers largely offer refinements and improvements, often quite valuable ones, on existing methods and procedures. This is all to the good. While much remains to be learned on many dimensions, much has already been learned as well. The careful, high-quality substantive applications collected here show just how much applied work on policyrelevant topics has benefitted from the methodological developments in the treatment effects literature. We now briefly describe and remark on each of the individual papers in the special issue. For this purpose, we have grouped them into four categories methodological papers, papers addressing topics in the economics of education, papers on the evaluation of active labor market programs, and a residual other category while recognizing that many of the substantive papers make important methodological contributions as well. 2 Methodological papers Lehrer and Kordas The paper by Lehrer and Kordas (2013) makes a large contribution to the surprisingly small literature on the semi-parametric estimation of the conditional probability of participation (the propensity score) for use in econometric evaluation methods such as propensity score matching that assume selection on observed variables. Lehrer and Kordas focus their attention on the potential for the Binary Regression Quantile (BRQ) estimator of Manski to replace the commonly used parametric logit and probit models in estimation of the propensity scores. BRQ can be thought of as the analog of quantile regression for binary dependent variables. While the logit and probit estimators can be interpreted as non-parametric given certain promises regarding increased flexibility in the conditioning variables as the sample sizes increase, common practice simply puts in the chosen covariates as main effects and stops there, regardless of sample size. Relative to this (unfortunate) norm, BRQ greatly increases flexibility in conditioning. The authors compare their estimator to standard parametric models and to the estimator of Klein and Spady (1993) via both a thoughtful Monte Carlo analysis and via (as is seemingly required in every analysis of matching and weighting estimators) analysis of the experimental data from the US National Supported Work

4 Fitzenberger et al. Demonstration and related non-experimental comparison groups drawn from the US Current Population Survey and the Panel Study of Income Dynamics, first analyzed in LaLonde (1986). The key finding is that using BRQ rather than another method matters, but only matters in a substantively important way when standard parametric models strongly misspecify the error distribution in the treatment equation or when the treatment effect varies in a highly non-monotonic way with the probability of treatment. More broadly, this study provides important new evidence on the performance of some under-used estimators from the literature and reinforces the notion that applied researchers should worry more than they do about the quality of their propensity score models. Lee Lee (2013) considers the application of balancing tests in the context of treatment effect estimators, such as matching and weighting, that assume selection on observed variables (or unconfoundedness, to use the awkward term common in the statistics literature). The application of balancing tests in matching parallels the common use of covariate balance tests in experimental contexts as a check on the performance of random assignment. In the narrower context of propensity score matching, balancing tests function as specification tests for parametric propensity score models. If a given specification fails to balance the included conditioning variables, then the researcher adopts a more flexible one. Balancing tests provide no information about the truth or falsity of the conditional independence assumption required for matching to remove the selection bias resulting from non-random participation in treatment. The state of the literature prior to Lee s paper featured a variety of balancing tests and little in the way of systematic theory or evidence to guide the researcher in selecting among them. Lee s contribution is to partially clear the resulting methodological fog. He starts by showing the sensitivity of impact estimates to alternative balancing test approaches as well as the poor size properties of existing tests in particular contexts. Lee then highlights the superior performance of permutation-based tests. He finds these tests to have good size properties and good power for detecting misspecifications of the propensity score model. Overall, this paper represents an important contribution to a literature that is smaller than it should be as well as a quite helpful guide to applied researchers. Boes Treatment analyses based on average outcomes do not immediately generalize to the case of ordered responses because the expectation of an ordinally measured variable does not exist. Boes (2013) extends the classical evaluation literature in this direction. His proposed remedy shifts the focus of the evaluation to distributional effects. Assuming a threshold crossing model on both the ordered potential outcomes and the binary treatment variable, and leaving the distribution of error terms and functional forms unspecified, this paper discusses bounds on the underlying

5 Estimation of treatment effects treatment effects and illustrates the construction of the bounds in a simulated data example. 3 Papers on the economics of education Farré, Klein, and Vella Farré et al. (2013) readdress the classical problem of estimating the causal effect of education in the setting of a constant treatment effect conditional on covariates. In previous work, two of the authors (Klein and Vella) established identification via conditional second moments in a control function approach that bypasses the need for instruments when estimating the coefficient on an endogenous treatment variable and which models the conditional variances semi-parametrically. This strategy relies on the fact that under certain conditions the control function involves the ratio of the standard deviations of the error terms from the first- and second-stage equations. Identification follows if this ratio varies with the covariates, i.e., conditional heteroskedasticity in either the first stage or the second-stage equation generally suffices for identification and no exclusion restriction is required. However, the non-parametric aspect of the necessary sequential estimation procedure may not be attractive for practical purposes. Computational difficulties may arise with implementing the semi-parametric estimators of the conditional variances as suggested in previous work by Klein and Vella. The paper by Farré et al. (2013) outlines how the estimator can be implemented parametrically. The use of parametric assumptions is accompanied by a large reduction in computational and programming demands. The paper provides a Monte Carlo study to demonstrate the advantages of the parametric approach under correct specification. The approach is then illustrated by estimating the causal effect of education using a sample drawn from the National Longitudinal Survey of Youth 1979 cohort. Accounting for endogeneity increases the estimate of the return to education from 6.8 to 11.2%, a result which is in the range of standard IV estimates. These findings correspond to a negative correlation between the error terms in the first-stage and second-stage equations, which contradicts a simple ability bias story. The authors argue that this finding is consistent with over-education effects. Unfortunately, the authors do not show the properties of their estimator under heterogeneous treatment effects an issue addressed explicitly by the broader literature on IV estimators and Local Average Treatment Effects (LATEs); see Card (2003) for an assessment. Klein Klein (2013) provides a methodological contribution illustrated in the setting of estimating the causal effect of education. Many studies debate how the unobserved dependence between the monetary return to college education and selection into college can be characterized. Klein (2013) considers a setting with heterogeneous treatment effects wherein individuals choose their level of education based on the perceived returns. For the case of a binary treatment indicator, the study develops a quite practical semi-parametric local instrumental variables estimator for identified features

6 Fitzenberger et al. of a flexible correlated random coefficient model. These identified features relate directly to the marginal and average treatment effects typically considered in policy evaluation. The estimation approach assumes independence between all covariates, including the instruments, and the error terms of the first stage and the second-stage equations. This assumption implies monotonicity of the effect of the propensity score on the treatment indicator. When the outcome equation includes a continuously distributed covariate, the approach even works with only discrete instruments; this is useful for applied work as institutional rules providing identification often yield only discrete instruments. Under specific support conditions on the propensity score, the approach allows estimation of both the average treatment effect on the treated as well as the average treatment effect for the non-treated under selection on unobserved variables. It also allows the assessment of whether or not comparative advantage guides selection into treatment. The study illustrates the application of the estimator by using British data to estimate the labor market effects of a college education. The study carefully discusses the choice of covariates in light of the stringent independence assumption required for estimation. The empirical results indicate that causal effects of college systematically differ between actual college graduates and actual college non-graduates. They are on average higher for college graduates and positively related to selection into college for 96 percent of the individuals. The dependence between selection into college and the effects of college is strongest for individuals with low math test scores at the age of 7, individuals with less educated mothers, and for working-class individuals. The effects of college have a strong positive dependence on unobserved ability for working-class individuals in the U.K.; the author concludes that policymakers may want to encourage more high ability individuals from this background to attend college. Bell and Bradley Bell and Bradley (2013) address an important practical problem that arises in many evaluations: what do when the initially untreated group eventually gets the treatment. In the context of, for example, experimental evaluations of education programs, getting schools or districts to participate often requires the promise that the randomly assigned control group will receive the treatment only a year or two later than the randomly assigned treatment group. This same issue crops up in other settings, such as the experimental evaluation of Mexico s conditional cash transfer program Progresa, which relied on random assignment of the timing of program roll-out, and studies that use standard panel data methods to evaluate policies that jurisdictions adopt at different times, but that ultimately get adopted in all jurisdictions. Rather obviously, once all units receive treatment, standard treatment effect estimators no longer estimate the impact of treatment versus no treatment. Instead, they estimate the impact of the timing of treatment. The contribution of this paper lies in formalizing the assumptions under which the impacts of treatment versus no treatment in early periods can be subtracted from the outcomes of initially untreated units in later periods after they begin to receive treatment in order to retrieve a consistent estimate of the impact of treatment versus no treatment at longer durations after the initiation of treatment. In addition to laying out this estimator, the authors do a nice job of describing how the

7 Estimation of treatment effects necessary assumptions can fail in the context of evaluations of educational interventions. These lessons generalize easily to other contexts. We particularly like this aspect of the paper, which provides a clear blueprint for applied researchers looking to justify the applications of these methods. Overall, this paper represents an important guide for evaluation practice as well as a useful methodological contribution. 4 Papers on the evaluation of active labor market programs Cockx, Goebel, and Robin The study by Cockx et al. (2013) estimates a continuous-time, competing-risks, mixed proportional hazard duration model for transitions out of unemployment based on data grouped at a quarterly frequency. The competing hazards refer to transitions to employment and to the start of treatment. For implementation, the authors assume piecewise constant baseline hazards within each quarter. The study uses the timing of events approach to identify the treatment effect, where the duration until start of treatment is right-censored if the duration until employment is shorter. Accounting for unobserved heterogeneity allows the authors to control for non-random selection into treatment. As noted by the authors, the timing of events approach builds on the assumption that the unobserved heterogeneity affects the transition to regular employment throughout the unemployment spell, whereas the treatment may only influence this transition from the start of the treatment onwards. Section 5.2 in Cockx et al. (2013) provides a careful discussion of identification. In particular, the estimation approach assumes no anticipation of the exact date of either transition modeled, although the agent may have knowledge of the distributions of the competing durations. Cockx et al. (2013) investigate whether income support for low-paid, part-time workers in Belgium increases the rate of transition from unemployment to non-subsidized, regular employment. The empirical analysis uses a sample of long-term unemployed young women as defined by their labor market histories from 1998 to The results suggest that participation in the policy has a substantively and statistically significantly positive effect on the transition to regular employment. Participation reduced the survival rate in unemployment by 27 percentage points 1 year after the start of the program. The time spent in the program did not affect the transition to regular employment. Zabel, Schwartz, and Donald Zabel et al. (2013) reanalyze the Self-Sufficiency Project (henceforth, SSP), a widely studied research and demonstration project in Canada that attempted to make work pay by providing generous earnings subsidies to welfare recipients who found fulltime work within a specific time interval. SSP is a prime example of a randomized welfare-to-work experiment in which the treatment compensates for a limited time period for disincentives to work caused by welfare payments without cutting the value of welfare for those not working. The SSP treatment aimed to increase work, and thus work experience, enough that many participants would experience wage gains and, as a result, choose to remain employed even after the subsidies ran out. The

8 Fitzenberger et al. study also provides an example of the general point that even with experimental data researchers may require non-experimental methods to answer specific questions of interest. In this case, the authors seek to estimate the effect of the treatment on the wages of those who work, breaking down those who work into always takers (here called the non-incentivized ) who would have worked with or without the subsidy, and compliers (here called the incentivized ) who worked because of the subsidy and would not have worked without it. The study carefully accounts for selection into these two subgroups and for more general selection into work when estimating the wage effects. The non-incentivized subgroup is identified by propensity score matching based on the estimated propensity to find a job among the members of the control group. The remaining subsidy recipients constitute the incentivized group. To estimate the wage effects, the authors use random assignment as an instrument. The study finds evidence of large and statistically significant relative differences in wage progression of approximately 9 percentage points during the 3-year supplement period for the incentivized group. The impact for the non-incentivized group is much smaller (at most 3 percentage points). There is also some limited evidence that the non-incentivized group in New Brunswick and the incentivized groups in both New Brunswick and British Columbia (the two locations at which the SSP demonstration took place) continued to work more after the 3-year supplement period ended. Cavaco, Fougère, and Pouget The study by Cavaco et al. (2013) investigates the effects of a French retraining program, called Conventions de conversion, on the reemployment rate of displaced workers. This program was intended to improve the reemployment prospects of the displaced by giving them retraining and job seeking assistance for a period of 6 months beginning just after displacement. The empirical analysis applies matching methods to non-experimental data collected by the French Ministry of Labour. The matching estimates show that this program succeeded in increasing the employment rate of trainees by approximately 6 percentage points in the medium-term, defined as the second and third years after entry into the program. This improvement results primarily from an increase in the rate of reemployment in regular jobs, defined as jobs covered by long-term labor contracts. Fitzenberger, Orlanski, Osikominu, and Paul The detailed study by Fitzenberger et al. (2013) estimates the impact of short-term training for the unemployed in Germany during two distinct time periods: and They apply state-of-the-art matching estimators in the context of a dynamic treatment assignment framework. This framework avoids the conditioning on future outcomes implicit in approaches that compare unemployed workers who do and do not participate within a certain interval. As argued by Fredriksson and Johansson (2008), that approach leads to a downward bias in estimates of program impact because one important reason that individuals may not participate in a program within a particular interval is that they find a job. The dynamic treatment assignment

9 Estimation of treatment effects approach produces estimates of the impact of training versus waiting at each period following the start of an unemployment spell, where the waiting choice includes the possibility of training in the future. Only individuals still unemployed at a given duration feature in the estimation for that duration. Examination of short-term training programs from an earlier era allows the authors both to look at long term impacts on their outcomes of interest employment, earnings, and participation in longer-term training programs and to compare the impacts of the earlier programs to the more recent programs from the 2000s. The authors find generally positive effects of shortterm training on the outcomes they consider for both time periods, with the effects largest for those who start the training in months 7 12 of their unemployment spell and for those who receive a version of short-term training more focused on skill development and less focused on activation and monitoring. 5 Other papers In recent years, the econometric methods for the estimation of treatment effects developed and initially applied mainly of the contexts of educational treatments and active labor market programs have seen growing application in other substantive literatures. The two papers in this category illustrate that trend. Becker and Egger Becker and Egger (2013) provide an application of treatment effects methods to a topic at the intersection of industrial organization and trade. In particular, they analyze the effects of product versus process innovations on a firm s propensity to export. Product innovation is a key factor for successful market entry in models of creative destruction and Schumpeterian growth. Process innovation helps secure a firm s market position given the characteristics of the products it supplies. Both modes of innovation are expected to raise a firm s propensity to export. Following the new trade theory, the paper conjectures that product innovation is relatively more important in that regard. The contribution of the study is to explicitly take into account the endogeneity of product and process innovations when estimating their impact on the choice to export. The study uses linked panel data for a long time period from two unique firm-level surveys in Germany that include data on the determinants of innovations and on the business environment. To account for self-selection of firms into the two types of innovation, a propensity score matching estimator is employed for multiple binary treatments in a panel context. The researchers control for past values of innovation expenditures as well as indicators for various innovation motives and impediments as well as other continuous variables measured at the firm and industry levels. To estimate the standard errors for the treatment effects, the authors use both analytical standard errors ignoring the estimation error in the propensity score and block-sub-sampling to account for the latter in the context of nearest neighbor matching. The study finds that pursuing both process and product innovations leads to a higher export probability than not pursuing any kind of innovation. Furthermore, product innovations dominate relative to process innovations for the decision to export. Process innovations raise

10 Fitzenberger et al. a firm s probability of exporting only when combined with product innovations, and they have a small positive impact on a firm s export-to-sales ratio at the intensive margin. Arpino and Aassve The paper by Arpino and Aassve (2013) aims to estimate the causal effect of fertility on households economic well-being, an issue that has received considerable attention in development studies and policy analysis. In this paper, the authors discuss several strategies for causal inference, stressing that their validity should be judged based on the assumptions that can plausibly be formulated in a given application, which, obviously, depends on the richness of the available data. The authors compare methods relying on the unconfoundedness (or conditional independence) assumption, including regressions and propensity score matching, with instrumental variable methods. The discussion in this paper is of general relevance as it suggests a set of guidelines that are useful for choosing an appropriate identification and estimation strategy. 6 Concluding remarks In conclusion, we want to thank our authors for submitting their research to the special issue. We think the average quality of the finished products is quite high. We also thank the authors for what in some cases was an extraordinary amount of work to produce revisions that satisfied the editor, the reviewers, and the authors themselves. References Arpino B, Aassve A (2013) Estimating the causal effect of fertility on economic wellbeing: data requirements, identifying assumptions and estimation methods. Empir Econ (this issue) Becker S, Egger P (2013) Endogenous product versus process innovation and a firm s propensity to export. Empir Econ (this issue) Bell SH, Bradley MC (2013) Calculating long-run impacts of social programs with staggered implementation once all research sample members receive the treatment. Empir Econ (this issue) Boes S (2013) Nonparametric analysis of treatment effects in ordered response models. Empir Econ (this issue) Card D (2003) Estimating the return to schooling: progress on some persistent econometric problems. Econometrica 69(5): Cavaco S, Fougère D, Pouget J (2013) Estimating the effect of a retraining program on the re-employment rate of displaced workers. Empir Econ (this issue) Cockx B, Goebel C, Robin S (2013) Can income support for part-time workers serve as a stepping-stone to regular jobs? An application to young long-term unemployed women. Empir Econ (this issue) Farré L, Klein R, Vella F (2013) A parametric control function approach to estimating the returns to schooling in the absence of exclusion restrictions: an application to the NLSY. Empir Econ (this issue) Fitzenberger B, Orlanski O, Osikominu A, Paul M (2013) Déjà Vu? Short-term training in Germany and Empir Econ (this issue) Fredriksson P, Johansson P (2008) Dynamic treatment assignment the consequences for evaluations using observational data. J Bus Econ Stat 26: Heckman JJ, LaLonde R, Smith J (1999) The economics and econometrics of active labor market programs. In: Ashenfelter O, Card D (eds) Handbook of labour economics, vol 3, , North-Holland, Amsterdam

11 Estimation of treatment effects Klein T (2013) College education and wages in the U.K.: estimating conditional average structural functions in nonadditive models with binary endogenous variables. Empir Econ (this issue) Klein R, Spady R (1993) An efficient semiparametric estimator for binary response models. Econometrica 61(2): LaLonde R (1986) Evaluating the econometric evaluations of training programs with experimental data. Am Econ Rev 76(4): Lee W-S (2013) Propensity score matching and variations on the balancing test. Empir Econ (this issue) Lehrer S, Kordas G (2013) Matching using semiparametric propensity scores. Empir Econ (this issue) Zabel J, Schwartz S, Donald S (2013) An analysis of the impact of the self-sufficiency project on wages. Empir Econ (this issue)