Innovative Evaluation Techniques International Crime Prevention Conference Montreal, November 7-8, Donna Smith-Moncrieffe, Public Safety Canada

Transcription

1 Innovative Evaluation Techniques International Crime Prevention Conference Montreal, November 7-8, 2017 Donna Smith-Moncrieffe, Public Safety Canada

2 Summary New and Emerging Issues in Crime Prevention Traditional evaluation methods - Prior challenges with Quantitative methods Innovative Evaluation Methods Making Recommendations for Policy Challenges and Opportunities 1

3 Advancing the Evidence in Crime Prevention CSCCB research division, crime prevention uses various evaluation research methodologies to determine the most cost-effective practices that address known risk factors for criminal involvement. After PS has contributed 18 years of evaluation research in crime prevention, we know that given that program interventions interact with other programs, policies, cultures and contexts, newer methodologies are required to determine what works in crime prevention. Contexts (culture) Policies New programs 2

4 Traditional Designs and Challenges Type of Evaluation Design Challenges experienced at PS, Research Division RCT In government programs, the evaluator is not in a position to randomly assign the youth to a treatment and a comparison group as per the design requirements Ethical issues Contribution agreements are focussed on paying for youth in treatment groups, not youth in control groups Quasi-experiment (Delayed comparison group) Quasi-experiment (with comparison group) Comparison group is formed with youth on a waiting list where T1 and T2 pre and post tests are completed Challenge is that the comparisons at the six month and 12 month post test (T3 & T4) are not possible as the delayed comparison group youth are usually returned to the treatment group Comparison group is formed with youth from a different community where they receive minimal services All aspects of internal validity (i.e. maturity, regression to the mean, history etc..) are still a threat unless covariates are assessed to determine how they impacted on the outcome 3

5 New Crime Prevention Issues and populations require New Assessment Techniques Cyberbullying Qualitative Techniques New applications with social media Innovative Qualitative Techniques Exiting Prostitution Radicalization Mixed Method Techniques 4

6 Innovative Qualitative Evaluation Techniques Qualitative Analysis: Social Media Most Significant Change Contribution Analysis Photo Voice Example: Stopit app to prevent cyberbullying Example: Bill C- 36 Exiting Prostitution Example YMCA Plusone multi-site evaluation Example: Life Skills Treatment Program with Indigenous populations 5

7 Innovative method #1: Using Social Media 6

8 Evaluation of the STOPit cyberbullying prevention application Evaluation Objectives: to assess the potential relevance and performance of an innovative cyberbullying platform called STOPit Methods: Environmental Scan, Comparative Analysis, Focus Groups (school-aged youth (aged 12-17) and adult professionals) using social media techniques Innovative method: To stimulate thinking and provide the participants with an opportunity to test the STOPit app, the evaluators designed and presented a set of three scenarios: Scenario 1: WhatsApp group conversation between a group of friends speaking negatively about another student at school and planning to physically harm him. Scenario 2: imessage conversation between two friends. One friend is exhibiting signs of depression and thoughts of self-harm. Scenario 3: Instagram feed in which multiple students are shaming and bullying another student by posting explicit and hurtful comments on her pictures, uploading altered and embarrassing images, and threatening to share sensitive images. 7

9 Innovative Technique : Use of simulation to elicit reactions to cyberbullying scenario 8

10 Findings: STOPIt App ( simulated scenarios) Simulated scenarios Would report Would not report Scenario 1: WhatsApp group conversation (group of friends disrespecting and threatening to harm another student that is not part of the group) 11 38% 18 62% Scenario 2: imessage conversation between two friends (one friend demonstrating signs of depression and self-harm) 27 93% 2 7% Scenario 3: Instagram feed (group of students is bullying, ridiculing, and shaming one student) 16 55% 13 45% 9

11 Findings: STOPIt App cont d Findings demonstrated promising results for preventing and reducing cyberbullying: Youth found the STOPit app to be highly functional, useful and innovative, particularly with regards to its anonymous feature and the possibility of sending fast text or video/image reports; Youth indicated that trust, reputation, and credibility were necessary precursors for their potential use and acceptance of the platform; and Technical functionality and capabilities, reporting and follow-up features, ease-of-use, and privacy protection are also highly important to youth. 10

12 Innovative Method #2: Most Significant Change 11

13 Bill C-36 Exiting Prostitution In 2013, the Supreme Court of Canada found Canada s existing prostitution legislation unconstitutional, and gave the Government one year to develop a legislative response. In June 2014, the Government announced Bill C-36, The Protection of Communities and Exploited Persons Act. Under this new legislation, individuals who sell sexual services are viewed as victims of exploitation, and, for the first time, the purchase of sexual services is considered a criminal offence. The Government also announced $20 million in social program funding to help providers of sexual services transition out of prostitution. There is an emerging need to find an evaluation method for marginalized target groups, hidden populations and small participant groups 12

14 Most Significant Change Evaluation Method (MSC) MSC works best with the following type of programs (Davies & Dart, 2005) complex program produces diverse and emergent outcomes focused on social change requires participatory approach designed with repeated contact between field staff and participants struggling with conventional monitoring systems highly customized services to a small number of beneficiaries (i.e exiting prostitution). 13

15 Evaluation Example #1: MSC method for Women s Support and Recovery Program Evaluation method being used to determine the efficacy of exiting prostitution: MSC interviewees will be randomly selected from the participant cohort. Criteria for inclusion will include a minimum of 6 months participation in the program and willingness to participate in an in-depth interview process. Every two years, 8 cases will be randomly selected for in-depth review and Most Significant Change storytelling (2 in transitional housing and 6 community participants). At the end of the project, the 16 stories will be shared with a group of program participants who will be asked to select the top six stories. These six stories will be shared with the advisory committee who will select the top three stories for inclusion in the final evaluation 14

16 Innovative Method # 3: Contribution Analysis 15

17 Evaluation Study #3: YMCA Plusone Program Multi-Site Program Overview: A model mentoring program aimed at addressing antisocial behaviour in youth at risk of criminal involvement or youth already involved in crime. Challenge: Evaluators were unable to secure a comparison group for any of the program sites. There was a risk that causal attribution could not be examined Innovative Method: Evaluators will employ a Contribution Analysis to determine key elements that contribute to changes in risk factors including: peer associations, anti-social involvement, drug use, and anger/impulsivity. Data collected will be compared against the theory of change, as well as the assumptions and risks to determine the validity of the theory of change.. 16

18 Contribution Analysis Cont d Following the recommendations of Mayne (2004, 2012), Sridharan and Nakaima (2012) the steps in contribution analysis for the YMCA multi-site will include: Setting out the cause and effect issue; Developing a plausible theory of change; Gathering evidence; Assembling and assessing the contribution story; Developing a timeline and trajectory of impact; Clarifying and prioritizing linkages; Including intended and unintended outcomes; Revising the contribution story in light of new evidence. Eliminating or reducing the causes of alternative explanations 17

19 Innovative Method #4: Photovoice 18

20 Photo Voice (Wang & Burris, 1997) a qualitative method used for community-based participatory research to document and reflect reality an empowering and flexible process that combines photography with grassroots social action participants are asked to express their points of view or represent their communities by photographing scenes that highlight research themes photographs are collaboratively interpreted through discussions in both small and large groups, and narratives can be developed that explain how the photos highlight a particular research theme narratives are then used to promote dialogue to mobilize and help policymakers better understand and change the community 19

21 Innovative Methods #4 Photo Voice in the Life Skills Treatment Program The goals of the Life Skills Treatment (LST) program are to prevent and reduce substance use amongst adolescents and to promote healthy alternatives to risky behaviour. The LST program was delivered in Edmonton, Alberta, by the Ben Calf Robe Society, and offered to Aboriginal and non-aboriginal youth. As a research tool, Photo Voice is particularly intended to empower vulnerable peoples and allow them to tell their stories in a manner that values knowledge and personal experience. The LST evaluation incorporated mixed methods, and Photo Voice activities supplemented the traditional data collection activities. 20

22 Photo Voice as an innovative method Evaluators came on-site to facilitate Photo Voice activities, and brought digital or disposable cameras to allow youth to take their photos. Youth were asked to take photos that represented their experiences in the LST program and benefits of the program to them personally. Youth were able to provide their comments orally, supporting the oral tradition of the cultures involved in the program. Youth participants were asked to discuss and describe the photographs they took and the photographs' meaning to them. Each description was analyzed with the respective photographs. Selected photos and descriptions that were most illustrative of LST themes were used in the final report. 21

23 Photovoice example #1 Photovoice exercises also demonstrated youth's learning regarding the dangers of smoking. One junior high youth took a photo of a cigarette butt on the school grounds (Photo 6). Photo 6 "Think before you smoke" The student said she took this photo because it reminded her of the consequences associated with smoking. "You have to think before you smoke...what will happen... What's in it...?" 22

24 Photovoice Example #2 One junior high youth photographed a gym banner and discussed friendship and respecting personal differences. - -"It reminds me of friendship. There are one, two, three friends who are respecting each other. It reminds me of how each one is different. People are different. You don't have to look the same to be friends." 23

25 Photo Voice Findings (LST Program) Observations from key informant interviews, focus groups, and Photo Voice exercises confirmed results from a standardized questionnaire that was administered, and suggests that the program has continued to have positive impacts on youth's understanding of the effects of substance use. Photo Voice findings demonstrated the negative opinions program youth formed towards smoking, to the extent that some of the youth had begun to take anti-smoking messages home and were encouraging their parents to quit smoking. Photo Voice enabled youth to record and reflect on the program; promote dialogue about important issues through discussion and photographs; and helped others understand the impacts of the program. Findings from Photo Voice activities provided additional insights into the experiences of youth in the program. 24

26 How Did this Work Contribute to Policy Development Commons Standing Committee: Evaluation results were used to demonstrate the feasibility of implementing potential pay for performance models Results are used for accountability and performance measures strategy Results are used to identify gaps in policies 25

27 Summary New evaluation and innovative techniques should supplement traditional methods Benefits include increased recruitment into programming, capacity to isolate key elements that contribute to change and better cultural adaptation Challenges: Some of the qualitative responses cannot be monetized for cost benefit analyses and pay for performance models Opportunities: To continuously identify the appropriate balance between innovative techniques and traditional methods to achieve valid results about how programs contribute to change in desired crime prevention outcomes. 26

28 References Davies, R., & Dart, J. (2005). The Most Significant Change (MSC) Technique: A Guide to Its Use. Gelder, Gingras & Associations (2017) Research Summary: Evaluation of the Stopit App to Prevent Cyberbullying Malatest & Associates. (2014). Evaluation of the Life Skills Training Program. Final Report. Submitted to Public Safety Canada (Unpublished report). Mayne, J., (2004). Reporting on outcomes: Setting performance expectations and telling performance stories. The Canadian Journal of Program Evaluation, 16, Mayne, J., (2012). Contribution analysis: Coming of age? Evaluation, 18, Patton, M., (2012). A utilization-focused approach to contribution analysis. Evaluation, 18, Sridharan, S., & Nakaima, A. (2012). Towards an evidence base of theory-driven evaluations: Some questions for proponents of theory-driven evaluation. Evaluation. 3, Wang, C., & Burris, M. A. (1997). Photovoice: Concept, methodology, and use for participatory needs assessment. Health education & behavior, 24(3),

29 Merci Contributors to this Presentation: Donna Smith-Moncrieffe Christina Wright Wendy Ryan Giselle Rosario CSCCB Research Division, Public Safety Canada 28