Methods of Reducing Bias in Time Series Designs: A Within Study Comparison

Transcription

1 Methods of Reducing Bias in Time Series Designs: A Within Study Comparison Kylie Anglin, University of Virginia Kate Miller-Bains, University of Virginia Vivian Wong, University of Virginia Coady Wing, Indiana University

2 Background Randomized experiments have long been established as the gold standard for addressing causal questions. However, experiments are not always feasible or desired, so observational methods are also needed. When a series of observations on the same outcome are available, an Interrupted Time Series (ITS) or difference-in-difference (DID) design may be used to assess treatment effects. In practice, changes may arise because of the introduction of a law or a change in an existing program. With increased state and federal emphasis on longitudinal data systems, these types of designs are becoming more common in educational evaluations (see Dee & Jacob, 2011). Despite the popularity of the time series designs, studies of their ability to produce unbiased treatment effect estimates have yielded mixed results (Somers, Zhu, Jacob, & Bloom, 2013; St.Clair, Hallberg, & Cook, 2016). Several issues can arise in the analysis of ITS. The purpose of this paper is to use a within-study comparison (WSC) to assess empirically whether these methodological approaches work in producing unbiased causal estimates. Purpose In this study, we draw on a large-scale, multi-site longitudinal dataset with many covariates to describe the performance of various interrupted time series designs in conjunction with matching methods. We use an RCT benchmark to evaluate the performance of (1) a simple interrupted time series (ITS) with no comparison group, (2) panel designs, including DID and CITS methods, and (3) DID and CITS designs with matched comparison group members. In addition, results from our study will provide guidance on two common methodological issues that arise in the analysis of repeated measures designs. The first is matching of units across sites or states, where covariates may vary due to contextual and measurement differences (e.g. using income to match units across geographical areas that may have different cost of living). The second is appropriate model specification for CITS and DID designs. We examine similarities and differences between the two methods, highlighting required assumptions for each approach. Population and Program The study employs experimental data from the Cash and Counseling Demonstration Project, which evaluated the effects of a consumer-directed care program on Medicaid recipients outcomes (cite). The data include monthly Medicaid expenditures for 12 months prior to the intervention (pretest), and 12 months after the intervention (posttest). Medicaid participants in Arkansas, New Jersey, and Florida were randomly assigned to treatment and control conditions, where the treatment consisted of Medicaid recipients selecting their own services using a Medicaid-funded account and the control consisted of local agencies selecting services for Medicaid recipients. Research Design For the WSC, we used experimental data to create different non-experimental comparisons and compared to the experimental treatment effects at 4, 9, and 12 months after treatment to assess performance. In the simple ITS design, we used only treatment cases within state, and then

3 estimated the changes in the intercept and slope once the intervention was introduced. Second, we evaluated the performance of comparative ITS designs for ruling out history and selection threats. For each state, we used data from the experimental controls in the other two states to form the comparison group. Third, we looked at the performance of multiple pretests for ruling out selection differences between treatment and comparison groups, as well as for ruling out spurious treatment effects due to incorrect specifications of time trend data. For these analyses, we examined the performance of ITS estimates with all pretest time points, and compared these results to the experimental benchmark. Fourth, we used estimated propensity scores to weight units that are observationally similar on a rich set of covariates to evaluate the performance of CITS and DID methods. Data Analysis We assessed correspondence using two metrics: the standardized bias and the root mean squared error (RMSE). First, we estimated corresponding experimental treatment effects for each subpopulation, and then calculate the standardized bias for each model using the following formula: T = ˆt ne - ˆt re s, ˆt ne ˆt re where and are the non-experimental and experimental treatment effect estimates (respectively), and s is the standard deviation of the outcome estimated from the experimental control group. The second performance statistic we consider is more holistic because it contains information about both bias and statistical precision. To compute the mean square error statistic, we implemented each design in 500 different bootstrap samples and stored estimated treatment effect parameters from each replicate. To estimate the RMSE statistic for a given nonexperimental treatment effect parameter, we centered the point estimate of the parameter from each bootstrap replicate around its corresponding experimental benchmark. Then we squared these deviations from the benchmark and computed the average of the squared deviations across the 500 bootstrap replicates. Formally, the statistic we work with is MSE(π q ) = 1 B (π B q π (b) RCT ) 2 b=1, where b = 1 B indexes the bootstrap replicates. We report the average bias and RMSE for each design type (e.g., simple DID, ITS, etc.) across different states by regressing bias on indicators of the design type. Findings We report a subset of the findings in Table 1; additional analyses are forthcoming. Overall, many of the time series designs were able to replicate the experimental benchmark results as evidenced

4 by the relatively small standardized bias. This is true even when drawing comparison units from other states, so long as the pretreatment and outcome variables are standardized to ensure comparability in both the scale of the measure and the way in which it is constructed. However, bias tended to become larger the further the estimates were from the onset of treatment. Additionally, matching methods did not seem to improve the initial ITS and CITS estimates, and frequently made the estimates less precise. Conclusions and Implications The results of this WSC provide further evidence that panel designs are able to approximate a randomized experiment. Somewhat surprisingly, matching methods did not generally improve the performance of these designs despite significant covariate imbalance. These findings have yet to be compared to the estimates obtained when outcome measures have not been adjusted for differences. These findings will suggest ways in which evaluators deal with potential threats to ITS designs.

5 References Dee, T. S., & Jacob, B. (2011). The impact of No Child Left Behind on student achievement. Journal of Policy Analysis and management, 30(3), Somers, M. A., Zhu, P., Jacob, R., & Bloom, H. (2013). The Validity and Precision of the Comparative Interrupted Time Series Design and the Difference-in-Difference Design in Educational Evaluation. MDRC. St. Clair, T., Cook, T. D., & Hallberg, K. (2014). Examining the internal validity and statistical precision of the comparative interrupted time series design by comparison with a randomized experiment. American Journal of Evaluation, 35(3),

6 Table 1. Standardized Bias and RMSE across different non-experimental designs Bias Month 4 Bias Month 9 Bias Month 12 RMSE Month 4 RMSE Month 9 RMSE Month 12 Simple ITS Linear Saturated Simple DID Unweighted PS Weighted DID + Linear Unweighted PS Weighted DID + Pre-Post Unweighted PS Weighted DID + Group + Pre-Post Unweighted PS Weighted Fixed Effects Unweighted PS Weighted