The article by Wilson and Chermak (2011, this issue) addresses an important topic:

Transcription

1 POLICY ESSAY C O M M U N I T Y - D R I V E N V I O L E N C E R E D U C T I O N P R O G R A M S Crime policy and informal social control Megan Ferrier N o r t h w e s t e r n U n i v e r s i t y Jens Ludwig U n i v e r s i t y o f C h i c a g o The article by Wilson and Chermak (2011, this issue) addresses an important topic: What policy levers outside of the criminal justice system can help reduce crime and violence in our nation s most distressed and dangerous urban areas? Wilson and Chermak try to answer that question by evaluating the effects of the One Vision One Life program in Pittsburgh, PA. One Vision involved a collaborative problem-solving process resulting in a multipart intervention that included continuing data analysis to guide program activities; the use of street workers to try to connect high-risk young people to social services such as job training and substance-abuse counseling; and some violenceinterruption activity intended to de-escalate situations in which retaliatory and other types of violence were likely. In what follows we first discuss whether these analyses are likely to have isolated the causal effects of the One Vision intervention, which is the key question that determines whether we can use these results to guide crime-policy decisions. Unfortunately we think the answer is no, which is mostly a result of the nature of the intervention and how it was implemented rather than from any fault of the investigators themselves. We then offer some thoughts about the underlying logic behind One Vision, including the (usually implicit) theory behind these sorts of kitchen sink interventions that risk factors have more-thanadditive effects on criminal behavior, as well as the practical challenges associated with asking real-world organizations to engage in complicated problem-solving and collaborative activities. We close with some discussion about the great value of learning more about how Thanks to Wesley Skogan for helpful suggestions. This essay was supported in part by grants from the Joyce, MacArthur and McCormick foundations to the University of Chicago Crime Lab. All opinions are our own. Direct correspondence to Megan Ferrier, Institute for Policy Research, Northwestern University, 2040 Sheridan Road, Evanston, IL, ( m-ferrier@northwestern.edu). DOI: /j x C 2011 American Society of Criminology 1029 Criminology & Public Policy Volume 10 Issue 4

2 Policy Essay Community- Driven Violence Reduction Programs to stimulate more of what we hypothesize might be the most likely active ingredient behind programs like One Vision informal social control. The Threat of Omitted Variables Bias in Evaluating One Vision One Vision was implemented in two target areas in Pittsburgh (Northside and the Hill District) in May 2004, and then it was expanded to another target area (on the Southside) in May The quasi-experimental research design takes advantage of the fact that the intervention was implemented in some places ( treatment neighborhoods ) but not in others ( control areas ), to compare the trends in crime preintervention and postintervention in the treatment areas with crime trends in the control areas over the same time period. The key assumption behind this standard difference-in-differences (DD) research design is that the crime trends in the comparison areas tell us something about what would have happened in the treatment (program) areas had there been no intervention. Put differently, the DD design assumes that crime trends in the comparison areas are informative about the counterfactual crime outcomes that the treatment areas would have experienced absent One Vision. Although this assumption is not directly testable, we can assess the assumption s plausibility by examining whether the treatment and control areas have similar crime trends during the period before the treatment is implemented (see Angrist and Pischke, 2009; Bassi, 1984; Heckman and Hotz, 1989; for an application to crime research, see Ludwig and Cook, 2000). Evidence for divergent crime trends between treatment and control areas during the pretreatment period reduces our confidence that any differences in crime trends between treatment and control areas after the intervention can be attributed solely to the intervention itself. Based on this sort of model specification test, the raw data presented in Figures 1 through 3 in Wilson and Chermak (2011) raise concerns about the potential for omitted variables bias in the article s key empirical estimates. For example, Figure 3 shows that from 2001 to 2002 (i.e., during the pretreatment period), the number of gun assaults in the control areas roughly doubled (from approximately 200 to approximately 400). In contrast, the trend in gun assaults over the same period for two of the study s key treatment areas (the Hill district and the Southside areas) looks fairly flat in Figure 3. Examination of the other figures reveals similar patterns that suggest that the control areas may not provide useful estimates for the counterfactual crime trends we would have expected in the treatment areas absent One Vision. Of course Figures 1 through 3 in Wilson and Chermak (2011) are just raw data, and the analysts do attempt to make the treatment and control areas more similar by using propensity score matching to adjust for a variety of census-tract level measures of neighborhood sociodemographic composition, as well as for 1 year (2003) of pretreatment crime levels (homicide, aggravated assault, and gun assaults). Table 1 in Wilson and Chermak suggests that the propensity score matching seems to do reasonably well in adjusting for 1030 Criminology & Public Policy

3 Ferrier and Ludwig census tract characteristics because the average tract characteristics for the control group after reweighting the data with the estimated propensity scores look fairly similar to the average tract characteristics of the treatment group. But even after reweighting, the average pretreatment crime rates in the treatment and control areas continue to be quite different. For example, the homicide rate in 2003 (the pretreatment period) in the target areas was equal to 0.41; the homicide rate in the nontarget areas before the propensity score weighting was equal to 0.61 (approximately 50% higher than in the treatment areas), whereas even the reweighted comparison area average is still A more formal test of similarity in pretrends between treatment and control areas would be to reestimate the full DD model with covariates using only data from the pretreatment period, selecting different pretreatment years to define placebo treatment indicators, which should be statistically indistinguishable from zero if the DD assumptions are met. As the key to the credibility of the DD design is to have similar pretrends in the outcome of interest between the treatment and control areas, perhaps a better way to construct the analysis would have been to focus explicitly on matching treatment with candidate control areas using the entire history of pretreatment crime data that are available. This method would be more likely to yield a valid comparison group than Wilson and Chermak s (2011) approach, which seemed to match on just 1 year of pretreatment crime data together with sociodemographic tract attributes that are imperfect predictors of local-area crime rates. Moreover, because Figures 1 3 suggest the two different treatment areas seem to follow different pretreatment crime trends, there would be value in constructing different weighted groups of control areas or synthetic controls for each treatment area (see Abadie, Diamond, and Hainmueller, 2010, Abadie and Gardeazabal, 2003). Our concern about the threat of omitted variables bias is relevant for thinking about the substantive questions Wilson and Chermak s (2011) article raises about why Pittsburgh s One Vision program was apparently less effective than similar interventions implemented elsewhere, such as Chicago CeaseFire or Baltimore s Safe Streets Project. Wilson and Chermak devote a great deal of discussion to differences across programs in staffing, emphasis, and implementation. But under Occam s razor, a simpler and more likely reconciliation of the different program results is that the true uncertainty band around the One Vision estimates is simply too large to rule out the null hypothesis that all of the programs have similar impacts. Who knows whether alternative synthetic-control methods would have been able to create more suitable comparison areas and to narrow the uncertainty band around the One Vision impact estimates. The fundamental problem is that in Pittsburgh, as in so many places around the country, policy makers and practitioners want to know what their new pilot programs are achieving, but they do not design and implement these programs in advance in ways that will maximize the chances of a successful impact evaluation later on. Random assignment is one way to increase the odds of learning how the program works, but it is not the only way. For example, in some applications, candidate target areas Volume 10 Issue

4 Policy Essay Community- Driven Violence Reduction Programs could be systematically ranked on some set of criteria, which would allow for a regression discontinuity (RD) study (see Owens and Ludwig, 2011). Policy makers and practitioners who want to say that they have done a real evaluation too often just implement the program in some way that is convenient, and then they hope that researchers can come in afterward and work some econometric magic to uncover program impacts. But even methodological magic has its limits. KISS? Wilson and Chermak s (2011) discussion about how One Vision was implemented highlights the difficulty that government and nonprofit organizations often have in carrying out complex problem-solving activities and multipronged, collaborative interventions. Problem-oriented policing, for example, even in cities like San Diego, CA, that have made long-term efforts to support, facilitate, and encourage its use, is in practice rare at large scale despite evidence of its effectiveness, likely because it can be so challenging and demanding to implement (Cordner and Biebel, 2005). Implementation becomes even more complicated when more than one organization is involved, in part because even basic things like data sharing are so difficult. Agencies and organizations have strong incentives to protect information, partly to reduce the risks of revealing mistakes, making them reluctant to share and then analyze and use more integrated data. Note that the real-world consequences of these implementation challenges are, if anything, likely to be understated in the program evaluation literature. The reason is that many of the highest quality studies that use randomized experimental designs are analogous to what medical researchers call efficacy trials, which test small hot house programs that are carried out with high fidelity under the watchful eye of bespectacled, laptop-toting researchers, rather than effectiveness trials that test interventions as they would actually be implemented at-scale in the real world. Given the great difficulty of implementing complicated interventions in the real world, it is useful to reexamine the (often implicit) conceptual framework behind the sort of multicomponent kitchen sink or synergistic program that One Vision represents. 1 The underlying theory for many kitchen sink interventions is that risk or protective factors have interactive (more-than-additive) effects, which implies that programs should act simultaneously on multiple fronts in order to make the biggest possible difference with available resources. More formally, let Y i be some measure of criminal involvement for individual (i), and let X 1i and X 2i be different candidate protective factors (say, human capital, and health capital, which in this application could be thought of as being drugfree ). The theory described by Equation 1 is that coefficient β 3 is negative and large (in 1. For a discussion of alternative ways of testing the basic logic behind kitchen-sink experiments without having to implement a kitchen-sink program, see Ludwig, Kling, and Mullainathan (2011) Criminology & Public Policy

5 Ferrier and Ludwig absolute value) relative to the main effects β 1 and β 2. Y i = β 0 + β 1 X 1i + β 2 X 2i + β 3 (X 1i X 2i ) + υ i (1) One empirically testable prediction of the synergy theory implied by equation (1) is that relatively lower-risk people should benefit relatively more from social policy interventions. The intuition is that if X 1i and X 2i exert an important interaction effect on the key outcome of interest, Y i, then an intervention that tries to (say) improve human capital, X 1i, should have a more pronounced impact on those who already have a high level of health capital, X 2i. Yet in real-world evaluations, evidence often shows us that the people who have the highest levels of baseline risk and so are presumably more severely disadvantaged on multiple dimensions that affect the outcome are the most responsive to social policy interventions, even to those interventions that try to influence just one or two relevant risk or protective factors. For example, cognitive-behavioral programs seem to have more pronounced impacts on those criminal-justice involved people who have the most extensive prior criminal records (Landenberger and Lipsey, 2005). In the Tennessee STAR class-size reduction experiment, minority children and poor children (i.e., those at highest risk for educational failure) benefit the most from smaller elementary school class sizes (Krueger and Whitmore, 2001). In a study of the long-term effects of Head Start, arrest rates seem to decline relatively more among African Americans than among whites (Garces et al., 2002). These findings are more consistent with a theory of diminishing marginal returns to social program investments than with the synergy theory described by Equation 1. The intuition behind diminishing marginal returns to behavioral change is quite straightforward: giving an extra thousand dollars to a low-income single-parent household on the South Side of Chicago should do more to change the schooling outcomes and delinquency risk of children in that home than would giving that thousand dollars to the parents of, say, Lindsay Lohan. (More formally the implication is that equation 1 should include squared terms for X 1i and X 2i, and that the coefficients on these squared terms are large in absolute value relative to the coefficient on the interaction of X 1i and X 2i ). Given the conceptual uncertainty about the value of trying to intervene on all fronts simultaneously, and the practical difficulties of implementing complicated multipart interventions in the real world, policy analysts and policy planners should think about whether there is value in prioritizing simplicity in policy design. We might heed the design principle articulated by aerospace engineer Kelly Johnson of Lockheed Skunk Works, who famously coined the term Keep it simple stupid (i.e., the KISS principle) For more information, go to en.wikipedia.org/wiki/kiss_principle. Volume 10 Issue

6 Policy Essay Community- Driven Violence Reduction Programs Promoting Informal Social Control Even if we were not concerned about the threat of omitted variables bias with Wilson and Chermak s (2011) estimates of One Vision s impacts, it would still be difficult to disentangle empirically the mechanisms through which the intervention reduced crime given the program s multiple components. With that caveat in mind, our own guess is that the most potentially important active ingredient with One Vision is the attempt to stimulate informal social control. This approach strikes us as more likely to be a key mechanism than, say, connecting youth to social services, which assumes that information (rather than motivation or the limited availability of high-quality social services) is the key barrier to service receipt by high-risk youth. One reason to suspect that informal social control might be a key mechanism through which programs like One Vision might be able to reduce crime comes from the seminal work by Sampson, Raudenbush, and Earls (1997), showing that cross-neighborhood variation in violent crime rates is strongly predicted by neighborhood informal social control or by what they call collective efficacy (see also Sampson, Morenoff, and Gannon-Rowley, 2002). What remains poorly understood are the specific ways in which public policy can causally intervene to strengthen the capacity of neighborhoods to carry out informal social control particularly distressed, dangerous neighborhoods. The Chicago Ceasefire program tries to strengthen local informal social control by hiring former gang-involved people to serve as violence interrupters and intervene to prevent retaliatory violence (Skogan, Hartnett, Bump, and Dubois, 2008). One practical challenge with the Chicago Ceasefire model is that keeping full- or even part-time people on the payroll is expensive, and government at every level in the United States is extremely budget constrained. Another potential challenge is political risk-averse policy makers may worry about providing funding to an intervention that hires former gang members, given the risk for relapse is not zero and (fairly or not) the public relations downside from even a single relapse by a formerly gang-involved person on the public payroll could be considerable. Are there alternative ways to stimulate informal social control? Our conversations with Columbia University sociologist Sudhir Venkatesh (personal communications, March 9, 2011) suggest the different roles that various neighborhood residents already play (at least in some parts of Chicago s high-crime South Side) in trying to defuse and de-escalate potentially violent events, particularly those involving young people. In our view, we would find great value in learning more about the types of residents who are involved with informal social control, what specific activities they carry out, and how public policy can help support them in their roles (and help overcome whatever barriers they face). Ethnographic research of the sort that Venkatesh carries out, which may help inform the design of new pilot programs and randomized intervention studies, would be an important complement to the sort of quantitative evaluation evidence presented in the One Vision study Criminology & Public Policy

7 Ferrier and Ludwig References Abadie, Alberto and Javier Gardeazabal The economic costs of conflict: A case study of the Basque Country. American Economic Review, 93: Abadie, Alberto, Alexis Diamond, and Jens Hainmueller Synthetic control methods for comparative case studies: Estimating the effect of California s tobacco control program. Journal of the American Statistical Association, 105: Angrist, Joshua D. and Jorn-Steffen Pischke Mostly Harmless Econometrics: An Empiricist s Companion. Princeton, NJ: Princeton University Press. Bassi, Laurie J Estimating the effect of training programs with non-random selection. The Review of Economics and Statistics, 66: Cordner, Gary and Elizabeth Perkins Biebel Problem-oriented policing in practice. Criminology & Public Policy, 4: Garces, Eliana, Duncan Thomas, and Janet Currie Longer term effects of Head Start. American Economic Review, 92(4): Heckman, James J. and Joseph V. Hotz Choosing among alternative nonexperimental methods for estimating the impact of social programs: The case of manpower training. Journal of the American Statistical Association, 84: Krueger, Alan B. and Diane M. Whitmore The effect of attending small class in the early grades on college-test taking and middle school test results: Evidence from Project STAR. Economic Journal, 111: Landenberger, Nana A. and Mark W. Lipsey The positive effects of cognitivebehavioral programs for offenders: A meta-analysis of factors associated with effective treatment. Journal of Experimental Criminology, 1: Ludwig, Jens and Philip J. Cook Homicide and suicide rates associated with implementation of the Brady Handgun Violence Prevention Act. Journal of the American Medical Association, 284: Ludwig, Jens, Jeffrey R. Kling, and Sendhil Mullainathan Mechanism experiments and policy evaluations. Journal of Economic Perspectives, 25: Owens, Emily and Jens Ludwig Regression Discontinuity Designs for Criminology Research. Working Paper. Cornell University, Ithaca, NY. Sampson, Robert J., Jeffrey D. Morenoff, and Thomas Gannon-Rowley Assessing neighborhood effects : Social processes and new directions in research. Annual Review of Sociology, 28: Sampson, Robert J., Stephen W. Raudenbush, and Felton Earls Neighborhoods and violent crime: A multilevel study of collective efficacy. Science, 277: Skogan, Wesley G., Susan M. Hartnett, Natalie Bump, and Jill Dubois Evaluation of CeaseFire-Chicago. Washington, D.C.: U.S. Department of Justice, Office of Justice Programs, National Institute of Justice. Wilson, Jeremy M. and Steven Chermak Community-driven violence reduction programs: Examining Pittsburgh s One Vision One Life. Criminology & Public Policy. This issue. Volume 10 Issue

8 Policy Essay Community- Driven Violence Reduction Programs Megan Ferrier is a research associate at Northwestern University s Institute for Policy Research and a research associate at the University of Chicago Crime Lab. Jens Ludwig is the McCormick Foundation Professor of Social Service Administration, Law, and Public Policy at the University of Chicago, director of the University of Chicago Crime Lab, and research associate at the National Bureau of Economic Research Criminology & Public Policy