What to Measure in Client Behavior Alec Boros, Ph.D. Research Manager, Oriana House, Inc. Rebecca Fischbein, Ph.D. Research Specialist, Oriana House, Inc. Asli Buldum, M.A. Research Specialist, Oriana House, Inc.
Using Assessments to Measure Offender Change Measuring Change Responsivity Principle Focus on Three Assessments Criminal Thinking Scale Adverse Childhood Experiences Short Inventory of Problems Issues to Consider with using an Assessment Implementation Interpretation Application Tracking Discussion
Where does our presentation fit in? Fidelity (2-C) Quality Assessments: Overview of ODRC s ORAS Quality Assurance Tracking Behavioral Change (2-C)Behavior Management System for Residential Programs Risk Assessments for Specialized Populations (3-D)Specialized Populations, When Is the ORAS Not Enough? Reponsivity Assessments (2-B) Using Assessments to Measure Offender Change
Change We could measure all kinds of change in community corrections Staff competence in service delivery Clients in knowledge of cognitive and skill development Risk Need scores Criminal and anti-social behavior Drug/alcohol use Barriers
Why is it important to measure change during programming? Maximizes rare treatment resources Allows for timely adjustment in client programming Provides feedback to offender Program effectiveness Have better outcomes One of the neglected areas is in Responsivity, and also one of the most confusing
Relationship of Targeted Interventions: Neglected Areas Desired Outcome Fidelity Responsivity Treatment Plan Risk Dosage Need Treatment
Responsivity: General vs. Specific Responsivity are individual factors that interfere with or facilitate learning Addressing responsivity can have a major impact on the effectiveness of other programming
Responsivity: General vs. Specific General Responsivity- is associated the use of the most effective correctional programming to change the criminogenic needs of offenders Specific Responsivity Use cognitive behavioral interventions that take into account characteristics of the individual. Failure to address can hinder treatment efforts
Specific Responsivity: The changing focus Not only most ignored but also the most difficult to define Definitions have changed over time Three approaches of responsivity: 1. Match treatment approach with learning style of offender 2. Match characteristics of staff member with that of the offender 3. Match skills of the therapist with the type of program Catering intervention to fit the offender Barriers All of the above
Responsivity: Internal vs. External Internal Responsivity Factors Characteristics of the individual offender Demographic More difficult to assess and accommodate factors such as personality and intelligence They can contribute to the engagement of offenders into treatment and the development of therapeutic alliance
Responsivity: Internal vs. External External Responsivity Factors The interaction between Facility, Staff and Client characteristics Facility Staff Client
How can we categorize these responsivity areas? Gender Race Age Ethnicity Religion Demographic Characteristics Peers Motivation Trauma Literacy Attitude/Thinking Style Family relations Personality Dynamic Characteristics Programming Intelligence Communication style Learning Style Static or Near-Static Characteristics
Selecting an Appropriate Responsivity Assessment
Sample of Assessments Used at OHI Assessment Construct Our Recommendation Adverse Childhood Experiences (ACE) TCU Criminal Thinking Scales (TCU CTS) TCU Family and Friends Psychological Inventory of Criminal Thinking Style (PICTS) Assesses exposure to childhood emotional, physical or sexual abuse and household dysfunction Measures 6 criminal thinking styles Family relationships, family drug use, peer socialization, peer criminality Assesses 8 criminal thinking styles Under review. Currently developing norms to examine how we can use information regarding adverse childhood experiences to guide programming. Yes. Good assessment for determining client thinking errors prior to programming. Can be used to guide programming as well as one-on-one client-staff interactions. Also, an effective measure of client change when pre and post test scores are compared Under review. Currently developing norms to examine how we can use information regarding social functioning (family/friends) to guide programming. It depends. An excellent, well-validated assessment of preintervention thinking as well as pre and post test change, but maybe not be practical due to length and difficulty in scoring
Sample of Assessments Used at OHI Assessment Construct Our Recommendation PTSD Diagnostic Scale (PDS) Short Inventory of Problems (SIP) University of Rhode Island Change Assessment Score (URICA) Assesses severity of Post Traumatic Stress Disorder (PTSD) Symptoms stemming from a traumatic incident Measure 5 life areas that could be affected by drug/alcohol use in past 3 months Motivation to change as assessed by Transtheoretical Model of Change Yes. Good for assessment of PTSD symptoms to determine whether client requires a referral for PTSD treatment. Also can be used as a post-test to determine changes in PTSD symptoms. Please note: self report assessment that is administered and scored by a clinician. Yes. Good assessment for determining problems related to drug and alcohol use. Can be used to provide insight into areas where client requires assistance. Also, an effective measure of client change and improvement in client functioning when pre and post test scores are compared. Yes. Good assessment for determining the client s initial motivation to change. Can be adapted to address any problem. We found it is not effective as a post-test and cannot effectively be used to determine change in motivation.
Texas Christian University Criminal Thinking Scale (TCU CTS) What is criminal thinking? Psychological Inventory of Criminal Thinking Walters, 1995 Criminal Thinking Scale Knight et al., 2006 TCU CTS Scales Personal Irresponsibility Entitlement Power Orientation Justification Cold Heartedness Criminal Rationalization Definition Blaming others/external factors for criminal behavior. Feeling of privilege Need for power/ control over others Minimalization of seriousness of antisocial acts Callousness Negative attitude toward law and authority figures
TCU CTS Sample Items Knight, K., Garner, B. R., Simpson, D. D., Morey, J. T., & Flynn, P. M. (2006). An assessment for criminal thinking. Crime and Delinquency, 52(1), 159-177
TCU CTS - Scoring Scoring Assessment and scoring guide can be found online at: http://www.ibr.tcu.edu/pubs/datacoll/cj trt.html Higher scores indicate greater criminal thinking
CTS: Scoring Breakdown
CTS: Tracking Individual Scores
TCU CTS - Interpretation How can CTS scores be used? How we use at OHI: Intake Sample Personal Irresponsibility homework Exit Evaluate programming success
CTS: Tracking Many Client Scores
TCU CTS Evaluation Results 546 Cuyahoga County probation clients who underwent Thinking for a Change programming completed the TCU CTS intake and exit assessment Pre-Test Post-Test T-Test Results Average Intake Score for 546 clients who completed post-test Average post-test score p-value Entitlement 17.4 17.5.85 Justification 18.8 18.3.04* Power Orientation 22.3 21.9.18 Cold Heartedness 25.0 24.8.48 Criminal Rationalization 28.4 27.8.05* Personal Irresponsibility 21.5 20.6.00* Note. *indicates a statistically significant difference between scores at Intake and Post-Test; significance is indicated at 95%.
Assessing Trauma Why measure trauma? The Adverse Childhood Experiences (ACE) Scale The Post Traumatic Diagnostic Scale (PDS)
In more detail: The ACE What does the ACE measure? How was it developed? Constructs Childhood psychological abuse Childhood physical abuse Childhood dysfunctional household Childhood exposure to sexual abuse
In more detail: The ACE Sample items: Filitti, M. J., Anda, R. F., Nordenberg, D., et al. (1998). Relationship of childhood abuse and household dysfunction to many of the leading causes of death in adults. American Journal of Preventative Medicine, 14(4) 245-258.
What does the ACE tell us? Higher scores indicate greater risk Adverse Childhood Experiences are major risk factors for leading causes of illness and death as well as poor quality of life in the United States http://www.cdc.gov/ace/
Prevalence Rates: The ACE Percentage of participants who experienced negative childhood experiences http://www.cdc.gov/ace/prevalence.htm
Prevalence Rates: Female Criminal Justice Populations Number of Adverse Childhood Experiences Women in Study by Messina & Grella (2006)* (sample size of 491) Women at OHI (sample size of 59) 0 15.7% 11.9% 1 16.7% 22.0% 2 21.8% 15.3% 3 14.0% 18.6% 4 10.6% 8.5% 5 or more 21.2% 23.7% *Messina, N. & Grella, C. (2006). Childhood trauma and women s health outcomes in a California prison population. American Journal of Public Health. 96(10): 1842 1848.
In more detail: The ACE How can we track ACE data? How can we use ACE information? Referral for further evaluation and treatment
Additional Trauma Screening Instrument: Post Traumatic Diagnostic Scale (PDS) http://www.pearsonassessments.com For use by psychologists, social workers, and counselors Forty-nine item measure, assesses severity of PTSD symptoms related to a single traumatic event Must be purchased Can be used over time to monitor changes in symptoms Use at OHI Foa, E., Cashman, L., Jaycox, L., & Perry, K. (1997). The validation of a self-report measure of PTSD: The Posttraumatic Diagnostic Scale. Psychological Assessment, 9, 445-451. Foa, E. (1996). Posttraumatic Diagnostic Scale Manual. Minneapolis, MN: National Computer Systems.
SIP Short Inventory of Problems Measures the negative consequences of substance abuse across five domains: Physical Interpersonal Intrapersonal Impulse Control Social Responsibility Was originally developed for alcohol use, but has recently been validated on drug using populations (Bender, Griffin, Gallop, and Weiss, 2007).
Where does the SIP come from? Allensworth-Davies, Cheng, Smith, Samet, Saitz. (2012). The Short Inventory of Problems-Modified for drug use (SIP-DU): Validity in a primary care sample. The American Journal of Addictions, 21:257-262.
More about the SIP Self-administered Higher scores indicate increased levels of drugrelated consequences An effective measure of client change and improvement in client functioning when pre and post test scores are compared Provides insight into areas where clients require assistance
SIP Sample Items Bender, R. E., Griffin, M. L., Gallop, R. J., & Weiss, R. D. (2007). Assessing negative consequences in patients with substance use and bipolar disorders: psychometric properties of the short inventory of problems (SIP). The American Journal on Addictions American Academy of Psychiatrists in Alcoholism and Addictions, 16(6), 503-509.
SIP Sample Items per Domain Physical Because of using drugs, I have not eaten properly. Interpersonal A friendship or close relationship has been damaged by my drug use. Intrapersonal My drug use has gotten in the way of my growth as a person. Impulse Control When using drugs, I have done impulsive things that I regretted later. Social Responsibility I ve had money problems because of my drug use.
SIP How is it utilized at OHI? To measure the severity of substance-related problems among drug court participants at intake, 6 months, and discharge Baseline measures are compared with follow-up measures Provides an indication of client s progress while in the program
SIP Tracking Scores SPSS spreadsheet snapshot
SIP Sharing Results SIP can not only be used to determine program effectiveness, but also to help inform treatment staff. At OHI, the research department had created individualized reports to share with treatment staff. Treatment staff was to consider this report in program planning for client needs.
SIP Individualized Reports Example of an individualized report:
SIP Presenting Results Used in determining program effectiveness: Analysis show significant reductions in problems related to drug use: Average SIP Scores for Graduates Over Time (Physical Subscale) SIP Physical Scores 6 5 4 3 2 1 0 Intake 6 Month Follow-Up Discharge Time
Some Lessons Learned 1. Must be sure to not only train staff how to administer/score assessments but how to use them 2. Be sure to use a tracking method that is right for your project, staff and resources 3. Use the same tool throughout your organization to measure a construct
Discussion 1. What are some assessments that you have used in your own agencies? 2. How have you used the results of the assessments?
Questions?