Deterrence and the death penalty Why the statistics should be ignored

Transcription

1 Deterrence and the death penalty Why the statistics should be ignored Is the death penalty a deterrent? Does it dissuade potential murders from killing? Many studies say it does, many studies say it does not; but all of the research about deterrence and the death penalty done in the past generation should be ignored said a recent headline summarising a report from the National Research Council. Daniel Nagin co-edited the report. He explains why the statistics are no guide. Introduction In April 2012 the National Research Council released the report Deterrence and the Death Penalty 1. I chaired the committee and co-edited the report with John Pepper (University of Virginia). Among the seven members of the committee was James Q. Wilson of Pepperdine University who, sadly, died shortly before the release of the report. The report is dedicated to Jim with the inscription I ve tried to follow the facts wherever they land. The committee endeavoured to follow Jim s sage advice. This summary of the committee s report draws heavily on that document. In 1976, the Supreme Court decision Gregg v. Georgia (428 US 153) ended a four-year moratorium on executions in the United States. That moratorium had resulted from the same court s 1972 decision in Furman v. Georgia (408 US 238). The decision in Gregg reaffirmed the Supreme Court s acceptance of the use of the death penalty provided certain criteria were met: fundamentally, that it not be applied arbitrarily or capriciously. There must be rational standards that determined when it was imposed and when it was not. After Gregg, executions resumed in some states. There was, naturally, debate as to whether the death penalty in fact was a deterrent. In the immediate aftermath of Gregg, a National Research Council report in 1978 reviewed the evidence that had been gathered up to the mid-1970s 2. That review was highly critical of the earlier research and concluded that available studies provide no useful evidence on the deterrent effect of capital punishment. During the 35 years since Gregg, and particularly in the past decade, many further studies have renewed the attempt to estimate the effect of capital punishment on homicide rates. Most researchers have used post-gregg data to examine the statistical association between homicide rates and the legal status or the actual implementation of the death penalty or both. The studies have reached widely varying, even contradictory, conclusions. Some studies conclude that executions save large numbers of lives; others conclude that executions actually increase homicides; and still others conclude that executions have no effect on homicide. Commentary on the scientific validity of the findings has sometimes been acrimonious. The National Research Council on which I served was convened against this backdrop of conflicting claims about the effect of capital punishment on homicide rates. The committee addressed three main questions laid out in its charge: 1. Does the available evidence provide a reasonable basis for drawing conclusions about the Some US states have the death penalty. Some do not. But other factors overwhelm simple analyses of the deterrent effects 2014 The Royal Statistical Society 9

2 Original Death Chamber at the Red Hat Cell Block in the Louisiana State Penitentiary. The electric chair is a replica of the original. Photo: Lee Honeycutt magnitude of capital punishment s effect on homicide rates? 2. Are there differences among the extant analyses that provide a basis for resolving the differences in findings? Are the differences in findings due to inherent limitations in the data? Are there existing statistical methods and/ or theoretical perspectives that have yet to be applied that can better address the deterrence question? Are the limitations of existing evidence reflective of a lack of information about the social, economic, and political underpinnings of homicide rates and/or the administration of capital punishment that first must be resolved before the deterrent effect of capital punishment can be determined? 3. Do potential remedies to shortcomings in the evidence on the deterrent effect of capital punishment have broader applicability for research on the deterrent effect of non-capital sanctions? The committee s overall conclusion was: [R]esearch to date on the effect of capital punishment on homicide is not informative about whether capital punishment decreases, increases, or has no effect on homicide rates. Therefore, the Committee recommends that these studies not be used to inform deliberations requiring judgments about the effect of the death penalty on homicide. Consequently, claims that research demonstrates that capital punishment decreases or increases the homicide rate by a specified amount or has no effect on the homicide rate should not influence policy judgments about capital punishment. In other words, claims that the death penalty deters murderers and claims that it does not deter murderers should both be rejected. The evidence does not exist to back them up. The committee was heartened by the widespread and consistently favourable media attention given to the report. As an example, on April 27th, 2012 a New York Times editorial observed: One of the most frequently made claims about the death penalty is that it deters potential murderers But a distinguished committee of scholars working for the National Research Council has now reached the striking and convincing conclusion that all of the research about deterrence and the death penalty done in the past generation should be ignored. Shortcomings in existing research Why should it be that the myriad of statistical analyses that have been carried out are, in effect, valueless? The post-gregg studies are usefully divided into two categories based on the type of data analysed. One category, panel data studies, analyse sets of states or counties measured over time usually from around 10

3 1970 to These studies use various forms of regression to relate homicide rates to variations, over time and across states or counties, in the legal status of capital punishment and/ or the frequency of executions. The second category, time series studies, generally study only a single geographic unit. The geographic unit may be as large as a nation or as small as a city. These studies usually examine whether there are short-term changes in homicide rates in that geographic unit in the aftermath of an execution. Sadly, statistical and logical shortcomings abounded in both these types of studies. Some studies played the useful role, either intentionally or not, of demonstrating the fragility of claims to have or not to have found deterrent effects. However, even these studies suffer from two intrinsic shortcomings that in and of themselves severely limit what can be learned from them about the effect of the death penalty on homicide rates from an examination of the death penalty as it has actually been No studies take different alternatives to the death sentence into account; no studies consider the potential murderer s perception of his risk of execution administered in the United States in the past 35 years. One is that none specify the noncapital sanction components of the sanction regime for the punishment of homicide. The other is the use of incomplete or implausible models of potential murderers perceptions of and response to the capital punishment component of a sanction regime. Without this basic information it is impossible to draw credible findings about the effect of the death penalty on homicide. We shall deal with both of these in detail below. The report describes data collection and research initiatives aimed at correcting these flaws. However, even if these initiatives were ultimately successful, it is important that empirical analyses based on the new sanctions data and knowledge of perception also innovate in the statistical methods used to estimate effects. Existing studies use strong and unverifiable assumptions to identify the effects of capital punishment on homicides. The report also describes statistical methods that allow researchers to avoid this pitfall. The report also notes that commentary on research findings often pits studies claiming to find statistically significant deterrent effects against those finding no statistically significant effect. Studies of the latter type are sometimes loosely interpreted as showing that capital punishment has little or no effect on the homicide rate. The report emphasises an elementary point of logic about hypothesis testing that is too often forgotten failure to reject a null hypothesis does mean that the null hypothesis is correct. We found many studies that failed to find an effect. We found no study that found there was no effect, or that claimed to affirmatively demonstrate that the effect is small. Incomplete specification of the sanction regime for homicide Capital and non-capital options are available to punish homicides. What penalties other than death are applied to convicted murders? It is a very relevant consideration. Properly understood, the relevant question regarding the deterrent effect of capital punishment is the differential deterrent effect of execution in comparison to the deterrent effect of other available or commonly used penalties. The report emphasises differential because it is important to recognise that even in states that make more intense use of capital punishment, most convicted murderers are not sentenced to death but to a lengthy prison sentence often life without the possibility of parole. Sometimes in some states non-life sentences are given; sometimes the sentence is life but with the hope of parole. These different penalties carry different deterrence effects. None of the studies that we reviewed took into account the severity of non-capital sanctions for murder in the states whose statistics they analysed. There are sound reasons for expecting that the severity of the non-capital sanctions for homicide varies systematically with the availability of the use of capital punishment, the intensity of use of capital punishment, or both. For example, the political culture of a state may affect the frequency of use of capital punishment and also the severity of non-capital sanctions for homicide. Thus, any effect that these non-capital sanctions have on homicides may contaminate any estimated effect of capital punishment. Potential murderers perceptions of and responses to capital punishment A by-product of the above is that no studies consider how the capital and non-capital components of the regime combine in affecting the behaviour of potential murderers. Only the capital component of the sanction regime has been studied, and this in itself shows both a serious conceptual flaw and a serious data flaw in the entire body of research. Several factors make the attempts by the panel studies to specify the capital component of state sanctions regimes uninterpretable. First, the findings are very sensitive to the way the risk of execution is specified. Second, there is no logical basis for resolving disagreements about how this risk should be measured. Should it be the percentage of convicted murderers sentenced to death, or the percentage actually executed, or some other measure? Third, much of the panel research simply assumes that potential murderers respond to the objective risk of execution. In fact, of course, potential murderers do not know the objective risk of execution. They have neither the data not the means of analysing it. As detailed in Chapter 4 of the report, there are significant complexities in computing this risk even for a well-informed researcher, let alone for a potential murderer. Among these complexities are that only 15% of people who have been sentenced to death since 1976 have actually been executed, and that a large fraction of death sentences are subsequently reversed. None of the measures that are used in the research can be said to be a better measure of the risk of execution than any others. Thus, even if one assumes that a potential murderer s perceived risk corresponds to the actual risk, there is no basis for arbitrating the competing claims about what is the right risk measure. The committee is also sceptical that potential murderers can possibly calibrate the objective risk, whatever it is; hence, there is good reason to believe that their perceived risk deviates from the objective risk. The research does not address this issue about how potential murderers perceptions of capital punishment risk and, more generally, of non-capital sanction risks are formed. 11

4 The time series studies come in many forms studies of a single execution event, studies of many events, and studies with a cross-polity dimension but a common feature is that none of them attempts to specify even the capital component of the overall sanction regime. This is a crucial shortcoming and is exemplified in the time series analyses that examine the association between deviations of number of executions from a fitted trend line and deviations of homicides from a fitted trend line. (Most forms of time series analysis de-trend the data and, in the case of capital punishment studies, analyse the association of deviations from the trend line in homicides and number of executions.) The studies are silent on two key questions: (1) Why are potential murderers attentive to the trend line in the number of executions (if indeed they are attentive)? (2) Why do they respond to deviations from the trend line (if indeed they do respond)? If time series analysis finds that homicide rates are not responsive to such deviations, it may be that potential murderers are responding to the trend line in executions but not to deviations from it. For example, a rising trend in the number of executions might be perceived as signalling a toughening of the sanction regime, which might deter potential murderers. Alternatively, if the time series analysis finds that homicide rates are responsive to such deviations, the question is why. One possibility is that potential murderers interpret the deviations as new information about the intensity of the application of capital punishment that is, they signal a change in the part of the sanction regime relating to application of capital punishment. If so, a deviation from the execution trend line may cause potential murderers to alter their perceptions of the future course of the trend line, which in turn may change their behaviour. Yet, even accepting this idea, a basic question persists. Why should the trend lines fitted by researchers coincide with the perceptions of potential murderers about trends in executions? There are no studies that include empirical analyses on the question of how potential murderers perceive the risk of sanctions; there is therefore no basis for assuming that the trend line specified by researchers corresponds to the trend line (if any) that is perceived by potential murders. If researchers and potential murderers do not perceive trends the same way, then time series analyses do not correctly identify what potential murderers perceive as deviations. Because of this basic flaw in the research, the committee concluded that it had no basis for assessing whether the findings of time series studies reflect a real effect of executions on homicides or are artefacts of models that incorrectly specify how deviations from a trend line cause potential murderers to update their forecasts of the future course of executions. Strong and unverifiable assumptions To identify the effect of capital punishment on homicide, researchers invariably rely on a range of strong and unverified assumptions. In part (as discussed above), this reflects the lack of basic information on the relevant sanction regimes for homicide and the associated perceptions of risk. None of the studies accounts for the non-capital component of the sanction regime, and potential murderers risk perceptions are assumed to depend on observable frequencies of arrest, conviction, and execution. The ad hoc choices of alternative models of risk perceptions lead to very different inferences on the effects of capital punishment, and none of them is inherently any more justifiable than any other. As discussed in Chapter 6 of the report, additional data and research on sanction regimes and risk perceptions may serve to reduce this form of model uncertainty. However, even if these uncertainties are fully reconciled, a more fundamental problem is that outcomes of counterfactual sanction policies are unobservable. That is, there is no way to determine what would have occurred if a given state had a different sanction regime. Thus Texas has executed more murderers than any other state in recent years; but we cannot know how many murders there would have been in Texas if capital punishment had been abolished there. That is, the data alone cannot reveal that information. The standard procedure in capital punishment research has been to impose sufficiently strong assumptions to yield definitive findings on deterrence. For example, a common assumption is that sanctions are random across states or years, as they would be if sanctions had been randomly assigned in an experiment. Another example is that the response of criminality to sanctions is homogeneous across states and years. To overcome some of the uncertainties, some studies use instrumental variables to identify deterrent effects, but this use requires yet other assumptions. The use of strong assumptions hides the problem that the study of deterrence is plagued by model uncertainty and that all the assumptions used in the research lack credibility. Next steps The committee report makes three recommendations for research. They address the three key criticisms of the death penalty research literature. Two of these address deficiencies in the data and knowledge base that must necessarily be corrected before research on the deterrent effect of capital punishment can possibly be informative. Data are presently unavailable on the non-capital component of state sanction regimes in the post-gregg era. One recommendation is that these data be assembled not only for homicide but also for other offences. Like homicide, reliable state-level data on the available sanctions and the frequency of their Deficiencies in data must be corrected; we need research on how potential murderers assess the risk of execution use for all types of crime is unavailable. This is a shocking data deficiency. The second recommendation is that research on sanction risk perceptions be greatly expanded, so that knowledge of how potential murderers, and criminals more generally, perceive sanction risk can be based on more than speculation. This too is an obvious prerequisite for any assessment about deterrent effects. The report acknowledges that progress on these two recommendations will not come quickly or easily. Moreover, even if the recommendations are successfully implemented, they will not provide certitude about the deterrent effect of capital punishment. Instead the goal is to lay the basis for research that provides evidence which is not as good as certainty but is a lot better than baseless speculation. The third recommendation concerns the use of methods that will not only increase the evidentiary value of research findings but also better communicate the uncertainty about that value. The different and often conflicting 12

5 findings in capital punishment deterrence research reflect different choices of assumptions, most of which cannot be supported and which the report demonstrates are not credible. Furthermore, the state of social science knowledge does not support a unique model that can be used to identify the effects of capital punishment under the current US regime or to permit the evaluation of deterrence under alternative regimes. The study of deterrence is plagued by model uncertainty. The failure of the existing research to address the issue of model uncertainly is evident in the debate initiated by John Donohue and Justin Wolfers 3, who challenged claims made by a broad set of researchers about deterrence. Much of their challenge involved demonstrations of how small changes in the models used in the various studies led to very different estimates of deterrence effects. In some cases they changed from positive to negative or vice versa, and in others they eliminated statistical significance. The third recommendation, which is labelled addressing model uncertainty with weaker assumptions, describes two possible approaches for addressing uncertainty about the correct statistical model. One is called model averaging and the other involves partial identification. Model averaging can be seen as a generalisation of the approach Donohue and Wolfers use to demonstrate the fragility of the research findings based on panel regression analysis. They demonstrate fragility across a selected set of model specifications. This approach, which is a common form of sensitivity analysis in statistical research, is vulnerable to the criticism of cherry picking model specifications that demonstrate fragility. Model averaging attempts to avoid this criticism by exhaustively exploring the sensitivity of findings over an entire predetermined set of model specifications. In a paper commissioned by the committee, which has appeared in a special issue of the Journal of Quantitative Criminology, Durlauf et al. 4 demonstrate this approach in a reanalysis of data used 10 years earlier by Dezhbakhsh et al. 5 Their analysis explores a model space of panel regression models defined over all combinations of four distinct model dimensions that were selected not only to reflect uncertainty about technical considerations like functional form but also uncertainty about the correct behavioural model. Their estimates of the effect of an execution range from 31 net lives saved to 98 net lives lost. Partial identification takes an entirely different approach to addressing model uncertainty. It does not attempt to estimate a single point value of the desired effect. Instead it produces a range over which the true effect will lie. This range should not be confused with a confidence interval which reflects uncertainty due to sampling variability. Instead it reflects uncertainty about the exact value of the true effect due to model uncertainty. Committee member Charles Manski has been a leading advocate of this approach. He recommends estimating a succession of partial identification effect-size intervals under increasingly specific modelling assumptions Judgements about capital punishments have to be made. But it must not be claimed that they rest on evidence; the evidence on capital punishment is too weak to guide any decisions about the phenomenon under investigation, which in this context is the effect of the legal status or intensity of use of capital punishment on homicide rates. In the partial identification framework, the benefit of making more assumptions is that the breadth of the effect-size intervals narrows, thereby providing a clearer indication of the true effect size. The downside is that disagreement about the credibility of all the modelling assumptions is likely to increase. Manski and Pepper 6 demonstrate this approach in the capital punishment context in the above mentioned special issue of the Journal of Quantitative Criminology. Conclusion The report emphasises that deterrence is but one of many considerations relevant to rendering a judgement on whether the death penalty is good public policy. Even though the scholarly evidence on the deterrent effect of capital punishment is too weak to guide decisions, this does not mean that people should have no views on capital punishment. Judgement about whether there is a deterrent effect is still relevant to policy, but that judgement should not be based on existing research or justified by quoting research. As we have demonstrated, existing research is too greatly flawed to justify any conclusion. Just as important, the committee did not investigate the moral arguments for or against capital punishment or the empirical evidence on whether capital punishment is administered in a non-discriminatory and consistent fashion, whether the risk of mistaken execution is acceptably small, or how the cost of administering the death penalty compares to other alternatives. All of these issues are relevant to making a judgement about whether the death penalty is good public policy. The committee s charge was also limited to assessing the evidence on the deterrent effect of the death penalty on murder, not the deterrent effect of other sanctions on crime more generally. We did not, for example, look at whether longer prison sentences deterred crime. Thus, the negative conclusion on the informativeness of the evidence on the former issue should not be construed as extending to the latter issue because the committee did not review the very large body of evidence on the deterrent effect of non-capital sanctions. References 1. National Research Council (2012) Deterrence and the Death Penalty, D. S. Nagin and J. V. Pepper (eds). Washington, DC: National Academies Press. 2. National Research Council (1978) Deterrence and Incapacitation: Estimating the Effects of Criminal Sanctions on Crime Rates, A. Blumstein, J. Cohen, and D. S. Nagin (eds). Washington, DC: National Academies Press. 3. Donohue, J. and Wolfers, J. (2005) Uses and abuses of empirical evidence in the death penalty debate. Stanford Law Review, 58, Durlauf, S., Fu, C. and Navaro, S. (2013) Capital punishment and deterrence: Understanding disparate results. Journal of Quantitative Criminology, 29(1), Dezhbakhsh, H., Rubin, P. and Shepherd, J. (2003) Does capital punishment have a deterrent effect? New evidence from postmoratorium panel data. American Law and Economics Review, 5, Manski, C. and Pepper, J. (2013) Deterrence and the death penalty: Partial identification analysis using repeated cross sections. Journal of Quantitative Criminology, 29(1), Daniel S. Nagin is Teresa and H. John Heinz III University Professor of Public Policy and Statistics at Carnegie Mellon University. 13