Risk Assessment Update: ARREST SCALES February 28, 2018 DRAFT

Size: px

Start display at page:

Download "Risk Assessment Update: ARREST SCALES February 28, 2018 DRAFT"

Bruno Reynolds
5 years ago
Views:

1 SUMMARY: In December 2017 the Commission voted to replace number of prior convictions with number of prior arrests as a predictor in the risk assessment scales. Over the past months staff has prepared the arrest scales, and (as with the conviction scales previously) has conducted racial impact analyses. The racial impact examinations reveal that half of the OGS-specific arrest scales perform differently for Black offenders versus White offenders: in the language of the risk assessment and predictive analytics literature, the scales are biased. This memo presents these results. While staff cannot recommend adoption of the biased scales, we explored several options and suggest a way forward. Based on the Commission s direction from the December 2017 meeting, staff prepared final scales using number of prior arrests as an input measure. While the general approach to developing risk scales remained the same, the process did involve the total recreation of scales. There were some differences in the predictor variables that were significant in the new arrestbased models, and therefore there were some differences in the final scales besides the replacement of prior convictions with prior arrests. As Table 1 shows, the arrest-based scales were generally a few percentage points more accurate when compared to the conviction-based scales. Table 1. Comparison of Accuracy (AUCs) for Conviction and Arrest Scales Conviction Arrest Difference OGS % 65.86% 2.72% OGS % 67.91% 3.00% OGS % 69.64% 2.11% OGS % 69.13% 1.20% OGS % 72.70% 1.98% OGS % 71.88% 1.82% OGS % 70.13% 1.43% OGS % 70.95% -0.28% OGS % 70.44% 1.08% Person 64.25% 66.00% 1.75% Average Difference: 1.68% Page 1 of 7

2 RACIAL BIAS TESTS As previously discussed in the staff memo from November 2017, the Commission has always been mindful of outcomes that differ by race group. Concerns over racial disparities were one of the driving forces that led to the inception of the Commission and the creation of the guidelines decades ago. Differential outcomes continue to be common across social, economic, and criminal justice indicators throughout the United States, and studies consistently find that minorities are more likely to be arrested, detained, and imprisoned. However, recidivism rates also differ by race group. In the context of risk assessment, since the outcome differs by race, assessing the potential bias of an instrument involves more than merely counting whether the high- and low-risk predictions differ by race. Instead, the salient question is whether the instrument appears to predict differently for one group compared to the other. Scholars in this field rely on a set of empirical approaches designed to test for this sort of bias. 1 Following this literature, we performed comparisons of the Area Under the Curve 2 statistics for White and Black individuals, and regression models which tested for race interactions 3 among the key decision groups of low, typical, and high risk. Table 2 presents the results of both the AUC and regression interaction examinations. Cells with a statistical significance indicator (*, **, or ***) indicate scales that fail these bias tests. 1 See Flores, A., Bechtel, K., & Lowencamp C. (2016). False Positives, False Negatives, and False Analyses: A Rejoinder to Machine Bias: There s Software Used Across the Country to Predict Future Criminals. And It s Biased Against Blacks. Federal Probation, 80(2), ; Monahan, J., Skeem, J., & Lowencamp C. (2017). Age, Risk Assessment, and Sanctioning: Overestimating the Old, Underestimating the Young. Law and Human Behavior. 2 The AUC scores range from 0-1 and an AUC of.5 indicates that the predictive ability of the scale is no better than chance. An AUC of.75 means that if you were to randomly select both a recidivist and a non-recidivist from the data, the recidivist would have a higher risk score 75 percent of the time. For these purposes, we tested whether the predictive accuracy differed by race group to a statistically significant degree. 3 The regression models examine the form of the relationship between the risk score and recidivism. The risk scale should predict recidivism similarly across racial groups, meaning the slope of the regression line for Black and White offenders should be similar. The difference between low risk offenders and typical risk offenders, or the difference between typical risk offenders and high-risk offenders should be similar for all subgroups. Page 2 of 7

3 Table 2. Race Differences in Prediction and Bias Tests, Arrest Scales Area Under the Curve (AUC) Analysis Form of the Relationship Bias Test OGS 1 ** * OGS 2 *** * OGS 3 *** ** OGS 4 OGS 5 OGS 6 *** ** OGS 7 OGS 8 OGS 9-14 Person *** * NOTES: * p <.05; ** p <.01; *** p <.001. Cells with theses statistical significance symbols indicate scales that fail the respective racial bias test. Page 3 of 7

4 First, the AUC tests showed statistically significant differences in accuracy for White and Black offenders in 5 of the 10 scales, indicating bias for these scales. 4 The more important analyses were the logistic regressions that examined what is known as the form of the relationship between recidivism and risk prediction, also indicated in Table 2. If the risk assessment scales perform similarly across race, the slope of the relationship between recidivism and the risk score (or risk designation) should be similar across groups. 5 The logistic regression analyses test for this. As indicated in Table 2 above, the race interactions were statistically significant in half of the scales, again indicating racial bias for these scales. To illustrate the form of the relationship concept, consider Figure 1 below. The plot on the left shows OGS 2 which failed the racial bias test; after accounting for the risk predictions, the plotted recidivism prediction lines cross: the prediction instrument is operating differently for Black and White offenders. 6 The right side of Figure 1 illustrates OGS 7. Although the slopes of the lines for White and Black individuals are not perfectly parallel, they are roughly parallel and do not cross, illustrating the regression model results we conclude that the instrument operates similarly for Black and White offenders for OGS While the AUC test is frequently performed in the literature to examine racial bias and therefore was included in our analyses, it is less pertinent for evaluating the Commission s proposed risk assessment. The reason is that AUCs consider differences in risk scores that are not translated into differences in decisions in the Commission s approach to implementing the risk instruments. For example, with the arrest input measure for OGS 5, all offenders with between 5 and 10 points are designated as typical risk. The AUC test may find a meaningful distinction between scores of 5 and 6, 6 and 7, etc., but because all of these scores indicate typical risk, these are arguably distinctions that do not make a difference in risk category. Observers may still find this information relevant, since the three risk categories are derived from the full scales, but the instrument classifications of low, typical, and high are the ones integral to the Commission s proposed use of risk assessment. Accordingly, we place greater emphasis on the form of the relationship examinations, which test for differences along these three key categorizations. 5 Flores et al., 2016; see also Skeem and Lowenkamp, The figure is based on margins plots of average marginal effects. 7 A note on statistical power: Previously, we reported that with the conviction scales, the regression interactions (or form of the relationship) test was significant only for OGS 3. However, this appeared to be driven by the very large size of the OGS 3 sample at around 46,000 individuals. When we performed the same tests on a random subsample from OGS 3 the results were no longer statistically significant. We took a similar approach here with OGS 3, and also for OGS 1 and OGS 2 (both of which have Ns of well over 10,000). For these arrest scales, the apparent bias in OGS 3 was no longer statistically significant for the subsample, but the bias in OGS 1 and OGS 2 persisted. Page 4 of 7

5 Figure 1. Form of the Relationship Plots IMPLICATIONS FOR THE RISK ASSESSMENT MANDATE Given that half of the arrest-based scales failed to meet the standards related to racial bias in predictive instruments, staff considered several alternative approaches. We ultimately recommend moving forward with the conviction scales. We did investigate whether we could re-construct variations of the arrest scales that would be free from racial bias. This proved problematic for three reasons: (1) remedial scales that pass the bias tests lose predictive power or accuracy; (2) remediating the scales requires collapsing variables and losing important distinctions related to criminal conduct (see example below); Page 5 of 7

6 and (3) the remedial scale construction changes the approach to building the scales resulting in inconsistency. As an example, we attempted to remediate the OGS 1 arrest scale. This attempt involved collapsing the prior arrest measure. For example, the points allocated on the basis of the number of prior arrests had to be changed as follows: Initial Arrest Scales (Biased) Remediated Arrest Scales Priors Points Priors Points 0 arrests 0 points 0-1 arrests 0 points 1 arrest 1 point >1 arrests 1 point 2-3 arrests 2 points 4-5 arrests 3 points >5 arrests 4 points This version 8 passed both the form of the relationship and AUC test, but reduced the overall accuracy of the scale by 1%. And as shown directly above, it required collapsing those with 0 and 1 prior arrests, and also collapsing all offenders with more than one arrest into a single group. Based on these efforts to remediate the problematic arrest scales, staff recommends proceeding with the conviction scales. While these remedial efforts were exploratory and not definitive, the process involved surrendering seemingly important information about prior criminal conduct: instead of distinguishing among five different categories of prior arrest groupings, the factor was collapsed to just two. Further, the difference in accuracy between the problematic arrest scales and the conviction scales was already marginal with the improvement varying between a -.28% to 3.00% and averaging 1.68% overall when using the arrest scales rather than the conviction scales (see Table 1 above). Constructing remedial arrest scales would result in further loss of accuracy, sacrificing the appeal of arrests over convictions in the first 8 We first attempted to truncate this scale (0 arrests = 0 points / 1 arrest = 1 point / 2-3 arrests = 2 points / >3 arrests = 3 points). This effort reduced the overall accuracy of the scale by about.5%, passed the form of the relationship test, but still failed the AUC test. Page 6 of 7

7 place. It would also result in more inconsistency in the approach (as with collapsing the 5- category prior offenses variable into a 2-category one for some scales, but retaining the 5- category approach for others). CONCLUSION Staff recommends proceeding with the conviction scales that were presented for the Commission s consideration in December Whether the Commission pursues this option, or some other course, the current arrest scales fail to pass the standard measures of racial bias and therefore staff is unable to recommend proceeding with these arrest scales. Page 7 of 7

RACE, RISK, & BIAS: What is the evidence? SARAH L. DESMARAIS, PH.D.

RACE, RISK, & BIAS: What is the evidence? SARAH L. DESMARAIS, PH.D. NORTH CAROLINA STATE UNIVERSITY Justice And Mental Health Collaboration Program And Second Chance Act National Conferences December 17,