The Validity of Violence Risk Estimates: An Issue of Item Performance

Transcription

1 Psychological Services Copyright 2007 by the American Psychological Association 2007, Vol. 4, No. 1, /07/$12.00 DOI: / The Validity of Violence Risk Estimates: An Issue of Item Performance Jeremy F. Mills Bath Institution and Carleton University Daryl G. Kroner Pittsburgh Institution Toni Hemmati Carleton University Typically, research conducted on the cross-validation or generalization of risk assessment schemes focuses on the aggregate score accuracy of the schemes within the new sample or population. Often overlooked when the schemes are examined in their aggregate form is the performance of the individual items. This study looks at the association between the items of the HCR-20 (C. D. Webster, K. S. Eaves, D. Douglas, & S. D. Wintrup, 1995) and the Violence Risk Appraisal Guide (VRAG; C. D. Webster, G. T. Harris, M. E. Rice, C. Cormier, & V. L. Quinsey, 1994) and violent recidivism in a sample of predominantly violent offenders. The results show that a number of the items from each scale do not distinguish between violent recidivists and nonrecidivists and that the presence of these items potentially reduces the predictive accuracy of the instruments. In addition, the inclusion of items that do not discriminate between recidivists and nonrecidivists potentially undermines the validity of the risk assessment process. Discussion centers on the application of prediction schemes and their individual risk factors in forensic practice. Keywords: violence, risk, recidivism, HCR-20, VRAG Jeremy F. Mills, Department of Psychology, Bath Institution, Bath, Ontario, Canada, and Department of Psychology, Carleton University, Ottawa, Ontario, Canada; Daryl G. Kroner, Department of Psychology, Pittsburgh Institution, Kingston, Ontario, Canada; Toni Hemmati, Department of Psychology, Carleton University, Ottawa, Ontario, Canada. The views expressed in this article are those of the authors and do not necessarily reflect the views of the Correctional Service of Canada. Correspondence concerning this article should be addressed to Jeremy F. Mills, Department of Psychology, Bath Institution, P. O. Box 1500, Bath, Ontario, K0H 1G0 Canada. Millsjf@csc-scc.gc.ca Actuarial prediction of risk has come under fire for a number of reasons: ethics (Mathiesen, 1998; Norko, 2000), accuracy, and the tendency to exclude clinical considerations (Litwack, 2001). Notwithstanding these arguments from detractors, actuarial risk assessment has become a key component in the prediction of violence across many settings. Grove and Meehl (1996) advocated for the actuarial approach to risk assessment primarily on the basis of superior accuracy. They pointed out that, although actuarial and clinical methods may consider the same variables in making a prediction, the two methods do not necessarily agree, and the authors stated emphatically that, disagreement is not a theory or philosophical preference; it is an empirical fact (p. 300). An actuarial system is viewed as an efficient and accurate method of combining factors relevant to the prediction of the outcome. The underlying assumption is that these factors, individually, are relevant to the outcome. Mossman (2000) identified other advantages of actuarial prediction beyond that of accuracy. When used properly, they are systematic and impartial. They also have a transparency that clinical judgment lacks... actuarial judgments are based on data and an explicitly prescribed method of combining those data. This makes actuarial methods open to inspection, questioning, and when necessary, critique (p. 277). Dawes, Faust, and Meehl (1989) similarly commented that actuarial methods are explicit and facilitate informed criticism (p. 1673). The qualities of systematic and impartiality speak to 1

2 2 MILLS, KRONER, AND HEMMATI the reliability (the reduction of measurement error) of actuarial methods of risk assessment. The quality of transparency speaks to the clarity of the assessment process and how the risk estimation is made and applied to the individual. Whereas the accuracy of a prediction scheme is determined by the performance of the aggregate scores in predicting those who failed, the transparency of the prediction scheme is determined by knowing the individual items or factors and how they contribute to the individual s score. Accuracy of aggregate score prediction is often the focus in studies involving the cross-validation and generalization of risk assessment schemes. Grove and Meehl (1996) commented on the concern that there is potential for error of actuarial generalization to populations in which data has not been gathered. Specifically, they considered weak the argument that readjustments were needed to overcome the attenuation of optimal prediction when a measure was generalized to other populations. Grove and Meehl s points are well taken; it is quite possible for an actuarial instrument to predict better-thanchance findings across populations that are different from the sample on which it was developed. For example, the accuracy of the Violence Risk Appraisal Guide (VRAG; Webster, Harris, Rice, Cormier, & Quinsey, 1994) in predicting violence in the original sample as represented by area under the curve (AUC) was.76. AUC is a statistic of receiver operating characteristic (ROC) analysis, and it represents the likelihood that a randomly chosen recidivist would have a higher score than a randomly chosen nonrecidivist. In this instance, the AUC of.76 means that a randomly chosen recidivist would have a higher score than a nonrecidivist 76% of the time. AUCs that exceed.70 are considered large effect sizes. Upon cross-validation to a Swedish sample of mentally disordered offenders, the AUC of the VRAG was lower in absolute terms (AUC.68; Grann, Belfrage, & Tengstrom, 2000). Similarly, the VRAG predicted general (AUC.67) and violent (AUC.60) recidivism in a sample of violent offenders (Kroner & Mills, 2001). Grove and Meehl were correct that actuarial risk assessment schemes can be robust across a variety of samples. However, one cannot conclude that all factors within the prediction scheme are relevant to the outcome if the observed accuracy is predicated on aggregate analysis. Observed aggregate score accuracy, therefore, does not necessarily mean that all items (risk factors) are relevant to violence prediction in that sample. It is expected that an instrument s accuracy will attenuate upon cross-validation. If this reduction in accuracy is due to items not discriminating between recidivists and nonrecidivists to the same degree as in the original sample ( the slight nonoptimality of beta coefficients or other statistical parameters... ; Grove & Meehl, 1996, p. 301), then the validity of risk factors remains. However, if the reduction in accuracy is due to the failure of some items to discriminate between recidivists and nonrecidivists then the validity of the items (and the instrument) is challenged. An item s failure to generalize may be due in part to the method of item selection. For example, stepwise procedures may result in the inclusion of sample specific variables. In fact, many predictive cross-validation studies do not examine the instruments at the item level. One exception to this is the crossvalidation of the Rapid Risk Assessment for Sex Offense Recidivism (RRASOR; Hanson, 1997) and the Static-99 (Hanson & Thorton, 1999) by Sjostedt and Langstrom (2001). Sjostedt and Langstrom (2001) examined the individual risk factors included in the RRASOR and Static-99. They found that most (7 of 10) risk factors discriminated between sexual recidivists and nonsexual recidivists when analyzed by either odds ratios or Pearson s r. However, despite the items individual performance, the RRASOR and Static-99 were predictive of sexual recidivism (AUCs.72 and.76, respectively). Similarly, a study of the generalizability of the Level of Service Inventory Revised (Andrews & Bonta, 1995) to a sample of predominantly violent offenders showed that not all domains (combined item scores) discriminated between recidivists and nonrecidivists (Mills, Kroner, & Hemmati, 2003). On the basis of aggregate analysis, Grove and Meehl (1996) were correct in asserting the robustness of actuarial prediction across samples. However, applying these instruments individually could have significant repercussions. For example, if an offender scored 1 on each of the three items found not to discriminate in the Sjostedt and Langstrom (2001) study, the overall score could be 3 points higher and classified at a considerably higher risk level than that of another of-

3 PREDICTING VIOLENCE IN OFFENDERS 3 fender who scored lower on those three items. This difference may have a minimal impact in aggregate analysis but a significant impact at the individual level. The purpose of this study is to take advantage of the transparency of risk prediction schemes and examine the generalizability of individual factors (scale items) in the prediction of violence. Specifically, we examine the efficacy of two instruments used to predict violent behavior: the HCR-20 and VRAG. Furthermore, we studied individual items performance within the context of statistical robustness of the aggregate scores. The two instruments were developed using two separate methodologies. The HCR-20 was developed on the basis of the rationale development of items after an extensive review of the literature. Furthermore, the HCR-20 was designed to communicate risk through a structured professional judgment approach. That is, items are considered (separately and collectively), and a risk estimate of low, moderate, or high is made by the professional. The VRAG was developed using a purely statistical procedure based on the file information of mentally disordered offenders. The VRAG was designed to communicate risk through a purely actuarial approach, assigning risk probabilities to the instrument s scores. Examining the HCR-20 and VRAG will permit a comparison of the items performance against a single outcome from two different development strategies. We tested the items of the HCR-20 and the VRAG to determine whether they discriminate between violent recidivists and nonrecidivists within an offender population. It is hypothesized that both the HCR-20 and VRAG aggregate scale scores will predict violent recidivism but that not all items within the scales will discriminate between violent recidivists and nonrecidivists. In short, the scales in aggregate will generalize but the items will not. Participants Method Participants were 83 volunteers drawn from a population of incarcerated men sentenced to 2 years or more. Receiving a sentence of 2 years or more is the criterion for entering federal custody in Canada. The ages of the participants ranged from 18 to 55 years, with a mean of 27.9 (SD 8.1; Mdn 26 years). The participants racial composition was as follows: Caucasian (n 67), African American (n 7), Native North American (n 5), and Asian (n 4). Participant s most serious index (confining) offenses were as follows: assaultive (n 51), robbery (n 18), criminal negligence/driving (n 7), arson (n 6), and drug related (n 1). The mean number of convictions and incarcerations for these participants was 12.9 (SD 13.5) and 4.6 (SD 5.3), respectively. Participants were not paid for their involvement in the study. Measures HCR-20 (Webster, Eaves, Douglas, & Wintrup, 1995). The HCR-20 is a structured clinical judgment approach to risk assessment. The 20- item instrument is broken down into three components: Historical (10 items; H1 H10), Clinical (5 items; C1 C5) and Risk Management (5 items; R1 R5 [see Table 1 for a description of all items]). Items are scored 0, 1, or 2. A score of 0 means that the available information contraindicates the presence of the item, a score of 1 means that the available information suggests the possible presence of the item, and a score of 2 means that the available information indicates the presence of the item. The HCR- 20 s concurrent validity and interrater agreement (intraclass correlation [ICC].82) has been demonstrated in an offender sample (Douglas & Webster, 1999). Predictive validity of the HCR-20 within an offender sample has also been demonstrated (Douglas, Yeoman, & Boer, 2005). The HCR-20 has also been shown to differentiate between offenders who were violent in prison and those who were not violent (Belfrage, Fransson, & Strand, 2000). Other research with offenders has shown the HCR-20 to have a moderate effect size when predicting minor institutional misconduct (AUC.68) and a small effect size when predicting major misconduct (AUC.57). The Historical component has been shown to be predictive of future violence in mentally disordered offenders (Grann, Belfrage, & Tengstrom, 2000). VRAG (Webster et al., 1994). The VRAG is a 12-item instrument developed to predict violent recidivism within a psychiatric sample (See Table 1 for a description of all items). The

4 4 MILLS, KRONER, AND HEMMATI Table 1 Items of the HCR-20 and VRAG HCR-20 VRAG H1. Previous violence V1. Psychopathy H2. Young age at first violent incident V2. Elementary school maladjustment H3. Relationship instability V3. Personality disorder H4. Employment problems V4. Age at index offense H5. Substance use problems V5. Separated from parents under the age of 16 H6. Major mental illness V6. Failure on prior conditional release H7. Psychopathy V7. Nonviolent offense history H8. Early maladjustment V8. Never married H9. Personality disorder V9. Diagnosis of schizophrenia H10. Prior supervision failure V10. Victim injury C1. Lack of insight V11. Alcohol abuse C2. Negative attitudes V12. Female victim of index offense C3. Active symptoms of major mental illness C4. Impulsivity C5. Unresponsive to treatment R1. Plans lack feasibility R2. Exposure to destabilizers R3. Lack of personal support R4. Noncompliance with remediation attempts R5. Stress Note. The HCR-20 consists of three components: Historical (H; 10 items), Clinical (C; 5 items), and Risk Management (R; 5 items). The Violence Risk Appraisal Guide (VRAG) consists of 12 items (V1 V12). initial items were selected according to the ability of each item to discriminate between violent recidivists and nonrecidivists. A stepwise discriminant analysis was used to determine which items together had the strongest predictive power. Each item was then given a weighting of1or 1 for every 5% difference from the mean recidivism rate of 31%. The total score of the VRAG ranges from 26 to 38, with higher scores reflecting a greater propensity of violent reoffending. Interrater correlations for the items are above.80, and kappas are above.70 (Rice, Harris, & Cormier, 1992; Rice, Harris, & Quinsey, 1990). Three items (separation from parents before age 16, elementary school adjustment, and scores on the Psychopathy Checklist Revised [PCL R; Hare, 1991, 2003) refer to development issues; four items are based on the index offense (age at index offense, victim injury, alcohol abuse, gender of victim); and the other items are diagnosis of personality disorder, diagnosis of schizophrenia, marital status, and failure on prior conditional release. Prediction results with the VRAG have shown the instrument to predict violent recidivism with psychiatric patients for 2- and 10-year follow-up periods (Grann et al., 2000; Rice & Harris, 1995). With nonpsychiatric offenders, the VRAG is able to distinguish between violent and nonviolent offenders (Loza & Dhaliwal, 1997). Violence variables. We followed up with all participants after their release from custody. Both offender correctional files and official police records were examined. Participants were classified according to their postrelease performance as nonrecidivist or violent recidivist in keeping with the methodology of the original VRAG study. Offenses committed while incarcerated were not counted. Violent offenses included uttering threats, assault (on someone other than a peace officer), sexual assault, armed robbery, and robbery with violence. If violence had been used in a robbery through assault, threat, or use of a weapon, that offense was categorized as a violent offense even if the resulting conviction specified robbery only. Robbery (specifically without violence) and assault of a peace officer (typically committed in attempts to flee at the time of arrest) were not classified as violent offenses. The rationale for this exclusionary decision is that robbery offenses include theft from a person where threat may only be implied or not present at all. Furthermore, the offense of assault on a peace officer predominantly occurs at the time of ar-

5 PREDICTING VIOLENCE IN OFFENDERS 5 rest (resisting arrest) and does not reflect the risk of violence to the public at large but is specifically directed toward criminal justice personnel during their intervention. Withdrawn and dismissed charges were counted as offenses only when there was compelling file information to indicate that the participant did commit the offense, as in this example: The participant had been charged for assault causing bodily harm after police responded to an emergency call. The participant s partner had been beaten, rushed to the emergency room, and admitted and treated but later petitioned the crown to withdraw charges. Additionally, the days of opportunity to reoffend were calculated for each participant. Days of opportunity are the number of days a participant is free to commit an offense of any kind. This included all days not incarcerated or not on an escorted temporary absence, as well as any days spent while whereabouts were unknown. These days began at release but also included any days of unescorted temporary absence before release. Not included were days spent in custody on suspension, arrest, days spent in holding prior to a court appearance, or time spent in provincial custody. The average follow-up period for all offenders in the sample was 4.6 years (SD days). The number of days to an act of violent recidivism ranged from 16 days to 4.3 years (M days, SD days). Procedure The participants in this study had been included in a previous study by Kroner and Mills (2001). However, in the present analysis, the average follow-up period increased to 1,675 days from the previously reported 790 days. In addition, the definition of violence was modified to reflect more closely the definition used in the VRAG original study. As a result, the base rate for postrelease violence increased from 12.6% to 35.0%. The sample was drawn from volunteers who participated in the psychological assessment process. We conducted the assessments within 8 to 12 weeks of the inmates arrival into federal custody. The purpose of the assessments was to determine the level of risk, to make recommendations for programming, and to identify appropriate supervision strategies. The participants were approached before testing and completion of a psychological assessment. At that time, the inmates were asked if they would consent to participation in the study, and agreement would be indicated by the signing of a consent form. Ratings were made by trained clinicians with at least a masters-level education. Ratings of the HCR-20 and VRAG were based on information from an interview (typically 2 hr in length), correctional file review, and police records. Kroner and Mills (2001) reported ICCs of.85 for the HCR-20 and.95 for the VRAG within this sample. Results The ranges, means, and standard deviations for both the HCR-20 and VRAG are reported in Table 2. The base rate for violent offending was 35% (29 violent recidivists and 54 nonrecidivists). Pearson s r and AUC (ROC analysis) are also reported in Table 2. Overall, the HCR-20 was somewhat more strongly related to violent Table 2 Descriptive Statistics Violent recidivism Measure Range M SD r AUC HCR **.72 H **.67 C ***.75 R **.71 VRAG *.67 Note. AUC area under the curve; H, C, and R the Historical (H; 10 items), Clinical (C; 5 items), and Risk Management (R; 5 items) components of the HCR-20; VRAG Violence Risk Appraisal Guide. * p.05. ** p.01. *** p.001.

6 6 MILLS, KRONER, AND HEMMATI recidivism than was the VRAG. When we examined the HCR-20 using the item groupings of Historical, Clinical and Risk Management, the Clinical items were most strongly related to the violence. Table 3 contains the correlations of the individual items with the number of days without violent recidivism. Negative relationships were expected because a person high on a risk factor is expected to reoffend. The analysis showed that 15 of 20 HCR-20 items were significantly related to the number of days to a violent offense, whereas only 5 of 12 items of the VRAG were related to the number of days to a violent offense. Among the items of the HCR-20 most strongly related (r.30) to time to reoffense were noncompliance (R4), exposure to destabilizers (R2), age at first violent offense (H2), previous violence (H1), and negative attitudes (C2). Only one item of the VRAG exceeded a correlation of.30 with days to violent reoffense: failure on prior conditional release (V6; r.41). Table 3 Correlations Between the Number of Days Without Recidivism and the HCR-20 and VRAG Items HCR-20 VRAG Item r Item r H1.33 ** V1.28 * H2.36 ** V2.25 * H3.26 * V3.08 H4.28 * V4.16 H5.25 * V5.21 H6.05 V6.41 *** H7.12 V7.20 H8.19 V8.10 H9.04 V9 H10.27 * V10.25 * C1.28 * V11.26 * C2.30 ** V12.06 C3.11 C4.26 * C5.29 ** R1.26 * R2.33 ** R3.27 * R4.43 *** R5.26 * Note. The HCR-20 consists of three components: Historical (H; 10 items), Clinical (C; 5 items), and Risk Management (R; 5 items). The Violence Risk Appraisal Guide (VRAG) consists of 12 items (V1 V12). * p.05. ** p.01. *** p.001. The next step in the analyses was to examine the unique variance contribution to outcome for each of the variables in the scales. Two regression analyses were undertaken. The variables of the HCR-20 and VRAG were respectively entered into Cox regression equations predicting violence: The beta, standard error, and Wald statistics for each item are reported in Table 4. Beta is the estimated coefficient of the covariate, and the Wald statistic is the beta divided by the standard error all squared and is used to test whether beta is different from zero. Overall, there was sufficient collinearity between the items to keep most from contributing unique variance at the p.05 level. Exceptions to this finding in the HCR-20 were the Clinical component s variable of negative attitudes and the Risk Management component s variables of lack of personal support and noncompliance with remediation. The individual item s efficacy to differentiate violent recidivists from nonrecidivists was examined next. For each item, we conducted a t test comparing those who reoffended violently with those who did not 1 (see Table 5). A directional hypothesis was made for this analysis (one-tailed, p.05) only when testing for differences between groups because of the directional relationship of the items with recidivism in prior research. For the H-10 scale, the items of substance abuse problems, major mental illness, psychopathy, and personality disorder did not significantly distinguish between groups. Of the five items in the C-5 scale, only one item active symptoms of major mental illness did not distinguish between the violent recidivist and nonrecidivist groups. All items of the R-5 scale distinguished between the groups. More than half of the items of the VRAG did not distinguish between groups at the p.05 (one-tailed) level. The items that were significantly different between groups were the PCL R score, elementary school maladjustment, separation from parents under the age 1 Both parametric t tests and nonparametric chi-square tests were calculated on each of the items. Both methods identified the same variables (one-tailed p.05) with two exceptions: The H-10 items Previous Violence and Prior Supervision Failure met the threshold for t tests only. Given that our approach to significance testing was a liberal onetailed approach to be as inclusive as possible, only the t test results are reported.

7 PREDICTING VIOLENCE IN OFFENDERS 7 Table 4 Cox Regression Statistics for the HCR-20 and VRAG Items HCR-20 VRAG Item B SE Wald Item B SE Wald H V H V H V H V H V H V * H V H V H V9 H V C V C * V C C C R R R * R ** R Note. The HCR-20 consists of three components: Historical (H; 10 items), Clinical (C; 5 items), and Risk Management (R; 5 items). The Violence Risk Appraisal Guide (VRAG) consists of 12 items (V1 V12). * p.05. ** p.01. of 16, failure on prior conditional release, and alcohol abuse. In addition, the AUC was calculated for each item, thereby providing an index of predictive efficiency. Individually, most items produced a small or moderate effect size. The final step of the analyses was to modify both the HCR-20 and VRAG, removing the items that were not related to violent outcome in our sample. This was done, and the scores were recalculated. The strength of the relationship between the modified versions of the instruments and violent outcome was then analyzed in the same manner as that for the original instruments using Pearson s r and AUC. The relationship with outcome improved little in absolute terms over the original scales for both instruments: HCR-20 (r.40, AUC.73); VRAG (r.32, AUC.69). Discussion The results of our study support the hypothesis that aggregate scale scores indicate predictive accuracy despite the failure of individual items to discriminate violent recidivists from nonrecidivists. This finding holds true for both instruments studied; however, more items from the VRAG failed to predict recidivism than did items from the HCR-20. These findings permit the present study to shed some light on how the approach to scale development affects the generalizability of that scale to other samples. Taken in aggregate form, the relationship of the scores of the HCR-20 and VRAG with violent recidivism indicate that they readily generalize to our sample of predominantly violent offenders. However, of the 20 items of the HCR-20, 5 (25%) were not related to violent recidivism. This is substantively different from the items of the VRAG, where 7 of 12 items (58%) were not related to violent recidivism. In both instances, there are items that are not associated with violent recidivism, but more items of the VRAG, both proportionately and in absolute terms, are not related to violent outcome. We speculate that the reason for the VRAG items relatively poorer performance has to do with the original scale development (method of item selection). The HCR-20 was developed rationally on the basis of clinical experience

8 8 MILLS, KRONER, AND HEMMATI Table 5 Comparisons Between Violent Recidivists and Nonrecidivists and Predictive Accuracy for the HCR-20 and VRAG Items HCR-20 VRAG Item t p AUC (SE) Item t p AUC (SE) H *.61 (.06) V *.62 (.06) H ***.66 (.06) V ***.67 (.06) H ***.64 (.06) V (.07) H **.63 (.06) V (.07) H (.06) V *.60 (.07) H (.06) V ***.64 (.06) H (.07) V (.07) H **.63 (.06) V (.07) H (.07) V9 H *.61 (.06) V (.06) C ***.65 (.06) V ***.66 (.06) C ***.68 (.06) V (.07) C (.07) C ***.67 (.06) C **.64 (.06) R **.65 (.06) R ***.64 (.06) R *.60 (.06) R ***.70 (.06) R **.65 (.06) Note. The HCR-20 consists of three components: Historical (H; 10 items), clinical (C; 5 items), and Risk Management (R; 5 items). The Violence Risk-Appraisal Guide (VRAG) consists of 12 items (V1 V12). AUC area under the curve. * p.05. ** p.01. *** p.001. anchored in empirical findings. This method sought out items representing constructs of known relevance to violence prediction across different populations. In contrast, the VRAG was empirically derived from an array of file information on men admitted to a maximumsecurity psychiatric facility for treatment (detention) or for assessment only. The decision for each item s inclusion in the scale was predicated only on the robustness of its statistical relationship with the outcome of serious violent offending within that single sample. Given the statistical rigor with which the items were chosen, there is little doubt that, for that specific sample, there is no better combination of variables more strongly associated with violent reoffense. However, what makes those variables the best choice for the original sample (maximizing sample specific variance) potentially serves to reduce the generalizability of the items to other samples. Our results are consistent with other research that shows overall predictive accuracy for aggregate scale scores, despite the failure of individual items to discriminate recidivists from nonrecidivists (Sjostedt & Langstrom, 2001). The failure of certain items to generalize may be due to either (a) the lack of a relationship between the item (construct) and violence within a specific sample or (b) the manner in which the item measures the construct. With regard to the former, Steadman (2000) suggested that the accuracy of psychiatric risk assessment could be improved in the future for specific subgroups within specified time frames and locations (p. 266). The idea of specificity, in our view, should extend to the constructs being used, the sample in which they are used and for the outcome being predicted. For example, sexually deviant cognitions are important in the prediction of sexual recidivism in a sample of sex offenders (Hanson & Harris, 2000). It is unlikely that the construct of sexually deviant cognitions would generalize to the prediction of violence within a sample of non-sex offenders. However, there may indeed be constructs that do generalize between these very different groups. Hanson and Harris (2000) found that noncooperation with supervision added to the prediction of sexual recidivism. This is very

9 PREDICTING VIOLENCE IN OFFENDERS 9 much akin to the HCR-20 item of noncompliance with remediation attempts, which broadly includes supervision and management issues. Generalization of items and the constructs they represent must not be assumed but proven, as this item outcome relationship is central to the strength of actuarial prediction and transparency in decision making. The present study also permits an examination of how the HCR-20 and VRAG items measure similar constructs and the effect on the item outcome relationship. Several items of similar content are consistent in either predicting or not predicting the outcome of violence. The VRAG item of diagnosis of schizophrenia corresponds closely with the H-10 item of major mental illness and C-5 item of active symptoms of major mental illness. All three items were not related to violent recidivism. Similarly, the VRAG item of personality disorder corresponds with the H-10 item of personality disorder, and again, neither item is related to outcome. The VRAG item of elementary school maladjustment corresponds with the H-10 item of early maladjustment, the VRAG item of failure on prior conditional release corresponds with the H-10 item of prior supervision failure, and all items are related to violent offending. Not all items of shared content area shared a similar relationship with the outcome of violence. The VRAG item of marital status was a dichotomous item of ever married (includes common-law marriage) or never married. This VRAG item was not related to outcome but a similar item of relationship instability in the H-10 was significantly related to violence. The H-10 item of relationship instability is specific to intimate or nonplatonic relationships. The rating is guided by the individual s ability to form and maintain stable long-term relationships (Webster, Douglas, Eaves, & Hart, 1997, p. 33). Measuring the success of intimate relationships, as opposed to measuring their presence or absence, appears to make the difference in the relationship with violence. Alcohol abuse as measured by the VRAG is significantly related to violence, but a similar item in the H-10 measuring substance abuse problems was not related to violence. The H-10 item requires the assessor to identify problems of daily functioning associated with the abuse of substances. The VRAG item provides five indexes of alcohol abuse (parental alcoholism, teenage alcoholism, adult alcohol problem, alcohol involved in prior offenses, and alcohol involved in index offense). Two possibilities seem evident to explain the difference: Either the presence of teenage alcohol abuse and parental alcoholism is having an impact, or the VRAG s exclusive focus on alcohol, as opposed to other substances, is the source of the relationship with violence. Our belief is that the latter is the more likely explanation. As an example, Zamble and Quinsey (1997) compared assaulters and robbers on measures of alcohol and drug use. Their conclusion was that assault was preceded by the highest levels of alcohol intake, while the use of certain other drugs was differentially associated with robbery (p. 104). Because our violence outcome variable did not include robbery without violence, it is possible that the presence of drug abuse in the H-10 item would inhibit the association with our outcome variable. The final interscale comparison is made between the VRAG measure of the PCL R and the H-10 measure of the PCL R. The PCL R score was the same, as the same rater rated both instruments, yet the VRAG representation of psychopathy was significantly related to violent recidivism whereas the H-10 measure was not. The VRAG representation of the PCL R divides the score into one of six categories, with a range of weightings from 5 to 12. The H-10 representation divides the PCL R score into one of three categories 20, 20 29, and 29. The only explanation offered for this interesting finding points to the issue of construct representation. The manner in which the VRAG represents the PCL R provides more variance reflecting the various levels of recidivism, and this representation is superior to that of the H-10. This particular finding points out that how a construct is represented and reflected in an item may enhance or dampen the relationship between the construct and the outcome. Therefore, it seems to be imperative that, before discarding a construct from inclusion in an actuarial scheme, the construct should be thoroughly tested using multiple representations in its relationship with the outcome of interest. Not only do the HCR-20 and VRAG represent different developmental origins and methodologies, but they also represent different methods of applying the information measured to the estimation of violence risk. The HCR-20

10 10 MILLS, KRONER, AND HEMMATI uses structured professional (clinical) judgment. This process uses a mechanistic (anchoreditem) approach to scoring the items but then uses clinical judgment to integrate the factors and assign a level of risk (low, moderate, or high). Clinical override permits the clinician to attach proportionally greater meaning to any one risk factor and thereby produce a risk statement greater than an overall score would indicate. The VRAG s actuarial approach produces a numerical score. The predictive bin that contains the score has an associated probability of violent recidivism. Our findings regarding the poor performance of the VRAG items may help to explain the lack of generalizability of the VRAG prediction bins in a correctional sample (Mills, Jones, & Kroner, 2005). For example, Mills, Jones, and Kroner (2005) found that the probabilities associated with the nine VRAG bins in the original mental health sample of development did not replicate to an offender sample. Furthermore, two bins had associated probabilities of recidivism below that of the immediately preceding lower risk bin. Although aggregate analysis indicated predictive accuracy, there was an accuracy failure at the point where the instrument was applied to the individual: the probability bins. Therefore, the presence of items that are not related to violent recidivism can provide misleading information that could lead to inaccurate estimates of the risk for violence. The results of this study offer interesting comparisons between instruments of very different developmental orientations and the generalizability of those scales items and the constructs measured. The results underscore the statistical robustness of actuarial instruments in that a number of items within the instrument may not be predictive of recidivism, yet the overall statistical performance of the instrument can still be quite acceptable. Paradoxically, this statistical robustness may mask the compromises to predictive accuracy that result from poor individual item performance. It is our position that incremental predictive validity, whether applied to an actuarial instrument such as the VRAG or a structured clinical judgment scheme such as the HCR-20, will be found in the specificity of the construct outcome relationship for different populations of interest. This will be necessary from a statistical point of view to optimize prediction and also from an applied point of view to ensure the accuracy of clinical conclusions, thereby engendering greater confidence among decision makers in the value of risk assessment as applied to a specific case. The primary weakness of the present study is the relatively small sample size. Smaller samples lack power for observing effects that may exist; hence, there may have been some items that could discriminate between violent recidivists and nonrecidivists that were not found to do so within this sample. This weakness is somewhat mitigated by the relaxed statistical threshold (one-tailed, p.05) for an item s identification as able to discriminate. Even doubling the probability to.20 as the threshold would only have admitted two additional items on both the HCR-20 and VRAG, permitting very similar conclusions to be drawn. A fairly robust violence base rate provides confidence in the results of the aggregate analysis. Future research on risk estimation instruments, particularly cross-validation or generalizability studies, would do well to examine and report the performance of individual items. The need to recalibrate risk estimation instruments is not a weakness of the actuarial system but simply good science. Dawes et al. (1989) cautioned against applying actuarial instruments blindly to new situations and settings without any performance checks (p. 1671), citing such an approach as a threat to predictive validity. Sound psychometric approaches to other areas of psychological measurement (e.g., IQ measurement) often result in culture-, gender-, and age-specific norms. In the absence of such normative information, clinicians should issue a preface or caveat for their estimates of risk accordingly. Mossman (2000) identified three strengths of actuarial assessment: They are said to be systematic, impartial, and transparent. Transparency permits an examination of the items and risk factors to ensure that they are related to outcome within the sample in which the instrument is being used. For example, within the original VRAG sample, committing the index offense against a female victim was negatively associated with violent recidivism. However, within the present sample, that same item did not discriminate between recidivists and nonrecidivists. Assuming that this finding is replicated, how does the mental health professional

11 PREDICTING VIOLENCE IN OFFENDERS 11 explain to a decision maker that an individual has a greater score (representing an increased risk for violence) because he assaulted a male, when the male female victim distinction is not relevant to that individual. In this instance, there would be a temptation for researchers to fall back on the aggregate findings, pointing to the fact that, despite the item s performance overall, the instrument discriminates between recidivists and nonrecidivists 67% of the time. However, research has shown that identifying specific risk factors and the management of those factors is a preferred form of risk communication. Heilbrun et al. (2000) compared a number of forms of risk communication that included prediction-oriented, management-oriented, and descriptive approaches. The approach most preferred was the management-oriented approach, which identified specific risk factors and interventions necessary to address the risk factors. This finding underscores the importance of our findings in this study. If practitioners use a prediction scheme to assess risk and to guide them in identifying the most salient risk factors, it is imperative that each risk factor be related to future violence. This holds true regardless of whether the approach to risk assessment is actuarial (percent likelihood) or structured clinical judgment (low, moderate, high). In conclusion, risk factors as represented by items within a prediction scheme should be routinely examined to ensure that they generalize across samples and that they discriminate between violent and nonviolent people. Stakeholders in the ongoing evaluation of prediction schemes include criminal justice systems, which rely upon the outcome of violence risk estimates, researchers in the fore of risk assessment and development, publication vehicles that review and communicate scientific findings, and clinicians who are on the front lines of applying psychological knowledge to the individual client. Each stakeholder s contribution differs. Systems can proactively encourage and facilitate research. Researchers will be most effective when they pursue improvements in explanatory theory, construct development, and measurement, and least effective when they exclusively defend the use of specific schemes in which they are personally vested. Journal editors and reviewers must include the unglamorous but necessary cross-validation studies and must be willing to publish results from applied settings in which science is practiced and methodologies may be less than perfect. The most difficult task falls to clinicians, who have the unenviable duty to sift through the growing literature and sometimes contradictory findings (to which this article now contributes) to judiciously apply nomothetic results to idiographic situations. Nonetheless, clinicians who routinely conduct risk assessments should regularly evaluate these assessments with the full aid of all other stakeholders. Within the context of the present study, ongoing analysis at the item level could eventually lead to items that better measure the risk factor of interest and items that generalize across samples. References Andrews, D. A., & Bonta, J. L. (1995). The Level of Service Inventory Revised. Toronto, Ontario, Canada: Multi-Health Systems. Belfrage, H., Fransson, G., & Strand, S. (2000). Prediction of violence using the HCR-20: A prospective study in two maximum-security correctional institutions. Journal of Forensic Psychiatry, 11, Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, Douglas, K. S., & Webster, C. D. (1999). The HCR-20 violence risk assessment scheme: Concurrent validity in a sample of incarcerated offenders. Criminal Justice and Behavior, 26, Douglas, K. S., Yeomans, M., & Boer, D. P. (2005). Comparative validity analysis of multiple measures of violence risk in a sample of criminal offenders. Criminal Justice and Behavior, 32, Grann, M., Belfrage, H., & Tengstrom, A. (2000). Actuarial assessment of risk for violence: Predictive validity of the VRAG and the historical part of the HCR-20. Criminal Justice and Behavior, 27, Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical statistical controversy. Psychology, Public Policy, and Law, 2, Hanson, R. K. (1997). The development of a brief actuarial risk scale for sexual offence recidivism. (User Report ). Ottawa, Ontario, Canada: Department of the Solicitor General of Canada. Hanson, R. K., & Harris, A. J. R. (2000). Where should we intervene? Dynamic predictors of sex-

12 12 MILLS, KRONER, AND HEMMATI ual offence recidivism. Criminal Justice and Behavior, 27, Hanson, R. K., & Thornton, D. (1999). Static-99: Improving actuarial risk assessments for sex offenders. (User Report ) Ottawa, Ontario, Canada: Department of the Solicitor General of Canada. Hare, R. D. (1991). The Hare Psychopathy Checklist Revised (manual). Toronto, Ontario, Canada: Multi-Health Systems. Hare, R. D. (2003). The Hare Psychopathy Checklist Revised (2nd ed.). Toronto, Ontario, Canada: Multi-Health Systems. Heilbrun, K., O Neill, M. L., Strohman, L. K., Bowman, Q., & Philipson, J. (2000). Expert approaches to communicating violence risk. Law and Human Behavior, 24, Kroner, D. G., & Mills, J. F. (2001). The accuracy of five risk appraisal instruments in predicting institutional misconduct and new convictions. Criminal Justice and Behavior, 28, Litwack, T. R. (2001). Actuarial versus clinical assessments of dangerousness. Psychology, Public Policy, and Law, 7, Loza, W., & Dhaliwal, G. K. (1997). Psychometric evaluation of the Risk Appraisal Guide (RAG): A tool for assessing violent recidivism. Journal of Interpersonal Violence, 12, Mathiesen, T. (1998). Selective incapacitation revisited. Law and Human Behavior, 22, Mills, J. F., Jones, M. N., & Kroner, D. G. (2005). An examination of the generalizability of the LSI-R and VRAG probability bins. Criminal Justice and Behavior, 32, Mills, J. F., Kroner, D. G., & Hemmati, T. (2003). Predicting violent behavior through a static/stable variable lens. Journal of Interpersonal Violence, 18, Mossman, D. (2000). Commentary: Assessing the risk of violence Are accurate predictions useful? Journal of the American Academy of Psychiatry and the Law, 28, Norko, M. A. (2000). Commentary: Dangerousness A failed paradigm for clinical practice and service delivery. Journal of the American Academy of Psychiatry and the Law, 28, Rice, M. E., & Harris, G. T. (1995). Violent recidivism: Assessing predictive validity. Journal of Consulting and Clinical Psychology, 63, Rice, M. E., Harris, G. T., & Cormier, C. (1992). Evaluation of a maximum security therapeutic community for psychopaths and other mentally disordered offenders. Law and Human Behavior, 16, Rice, M. E., Harris, G. T., & Quinsey, V. L. (1990). A follow-up of rapists assessed in a maximumsecurity psychiatric facility. Journal of Interpersonal Violence, 5, Sjostedt, G., & Lanstrom, N. (2001). Actuarial assessment of sex offender recidivism risk: A crossvalidation of the RRASOR and the Static-99 in Sweden. Law and Human Behavior, 25, Steadman, H. J. (2000). From dangerousness to risk assessment of community violence: Taking stock at the turn of the century. Journal of the American Academy of Psychiatry and Law, 28, Webster, C. D., Douglas, K. S., Eaves, D., & Hart, S. D. (1997). HCR-20: Assessing risk for violence Version 2. Vancouver, British Columbia, Canada: Simon Fraser University and British Columbia Forensic Psychiatric Services Commission. Webster, C. D., Eaves, D., Douglas, K. S., & Wintrup, A. (1995). The HCR-20 scheme: The assessment of dangerousness and risk. Vancouver, British Columbia, Canada: Simon Fraser University and British Columbia Forensic Psychiatric Services Commission. Webster, C. D., Harris, G. T., Rice, M. E., Cormier, C., & Quinsey, V. L. (1994). The Violence Prediction Scheme: Assessing dangerousness in high risk men. Toronto, Ontario, Canada: Centre of Criminology, University of Toronto. Zamble, E., & Quinsey, V. L. (1997). The criminal recidivism process. Cambridge, United Kingdom: Cambridge University Press. Received August 31, 2005 Revision received January 9, 2006 Accepted June 8, 2006