Signs of Safety (SOS)

Transcription

1 Signs of Safety (SOS) Year I Implementation Evaluation Report PREPARED FOR: NANCY ANDREWS & NANCY DALE, CHILDREN S AID SOCIETY OF TORONTO August 14, 2015 Authored by: Violeta Dubov, Deborah Goodman, Rameez Mahmood, Jeremy Howe, & Patricia Appleton, Child Welfare Institute, Children s Aid Society of Toronto in partnership with the SOS Practice Leadership Committee of the Children s Aid Society of Toronto

2 Signs of Safety Evaluation 8/14/2015 SIGNS OF SAFETY (SOS) : Executive Summary BACKGROUND. The Signs of Safety (SOS) is a clinical framework for child protection that was developed through the 1990s by Andrew Turnell and Steve Edwards. SOS arose out of a need for a structured approach to child protection casework and the decreasing morale of child protection staff. Today, the SOS approach is increasingly practiced in numerous jurisdictions of various countries, including Australia, UK, USA, The Netherlands, Japan and Canada. SOS was formally implemented at the Children s Aid Society of Toronto (CAST) in March METHODOLOGY. This evaluation report prepared by the Child Welfare Institute (CWI) aims to provide an overview of Year I SOS implementation at CAST by collecting input from multiple stakeholders, specifically front-line workers, SOS Practice Leadership Committee, Intake supervisors, and clients. A mixed-method research design was utilized to meet the objectives of the evaluation; quantitative data included training surveys, Supervisor SOS Fidelity Rating Checklist, and Parental Rating Checklist while qualitative data included client interviews, focus groups, file reviews, and qualitative survey questions. The main findings in relation to the anticipated outcomes are presented next. OUTCOME 1: PROTECTION AND NON-PROTECTION STAFF TRAINED IN INTRODUCTORY SOS. Over 350 child protection workers and supervisors received two-part SOS training in 2013, of which 34%-40% completed an evaluation survey. In 2014, training and skill development continued for protection staff and management while non-protection service staff were introduced to SOS in late The majority of survey respondents (76% and higher) rated all satisfaction questions positively while over half of participants found they gained between 60%-100% new knowledge from the curriculum. The main criticisms were repetitive content and insufficient practice based learning. OUTCOME 2: ADVANCE STAFF SKILLS IN SOS ASSESSMENTS, SAFETY PLANNING AND FAMILY ENGAGEMENT. Six protection supervisors evaluated a total of 30 workers on their teams using the Supervisor SOS Fidelity Rating Checklist, a 35-item scale assessing workers practice skills and use of SOS. The analysis of that data showed that while the majority of their workers (over 80%) are highly proficient in many skills associated with best clinical practice (e.g. engagement with family), over half of the workers lacked proficiency in SOS-specific skills (e.g. use of SOS tools with children; using Mapping for assessment of the family and case-planning). The analysis also found that workers who had more training and more advanced SOS skills were rated as more proficient by their supervisors on all skills assessed; the majority of these positive relationships reached statistical significance. Data obtained from focus groups with front-line workers (n=5), Practice Leads (n=9), and Intake supervisors (n=12) corroborates the gaps in SOS knowledge and use across protection staff and management. However, most participants reported that the use of SOS at CAST has brought structure to the work, lead to more collaboration with families, and improved risk assessments and safety planning. File reviews yielded similar gaps where SOS Super Users (n=9), who have more advanced SOS skills, were more likely at a statistically significant level to write in evidence of SOS use in their file recordings (e.g. Danger/harm statements, identify strengths, specific goals/plans) than Regular Intake (n=21). OUTCOME 3: IMPROVED FAMILY SATISFACTION WITH CAS SERVICES. Clients who had recent child welfare involvement with a worker proficient in the use of SOS were interviewed using the Parent Feedback Checklist (n=9). The majority of clients were highly satisfied with their workers approach (e.g. humility, honesty, reliability) and skill (e.g. engagement with children; conducting a balanced assessment). Six clients were asked to compare this most recent worker to previous child welfare workers they had; half indicated that their most recent worker had superior skills to previous workers while the other half indicated that their most recent worker s high level of skill and support was comparable to previous workers. OUTCOME 4: IMPROVED WORKER JOB SATISFACTION. According to report by focus group participants as well as qualitative comments made on the Supervisor SOS Fidelity Rating Checklist, implementing SOS led to improved experience for front-line staff who feel more job satisfaction and pride in their work. Despite existing challenges with the implementation process, users of the SOS framework experience improved practice, better partnership with clients, and beneficial impact on the safety and functioning of children and their families. CONCLUSION. This report demonstrates preliminary evidence that the anticipated outcomes of Year I SOS implementation were achieved, but only for pockets of front line workers and protection supervisors. Therefore, the next step in SOS implementation should focus on increasing the knowledge and skill of staff across the agency who do not have the same proficiency as SOS Super Users or Practice Leads. As for child welfare clients, their increased satisfaction with CAST services is an important finding, despite the small sample size. As a next step, it is important to compare longitudinal outcomes related to child safety of clients who received SOS-informed service versus those who did not. 1

3 Signs of Safety (SOS) 8/14/2015 Table of Contents EXECUTIVE SUMMARY INTRODUCTION BRIEF HISTORY CORE PRINCIPLES PRACTICE ELEMENTS FRAMEWORK AND TOOLS EVIDENCE BASE SIGNS OF SAFETY IMPLEMENTATION AT CHILDREN S AID SOCIETY OF TORONTO ANTICIPATED OUTCOMES OF SIGNS OF SAFETY IMPLEMENTATION EVALUATION METHODOLOGY EVALUATION TOOLS TRAINING SURVEYS (N=138;119) FOCUS GROUPS (N=26) SUPERVISOR FIDELITY RATING CHECKLIST (N=30) CLIENT INTERVIEWS (N=9) FILE REVIEWS (N=30) DATA ANALYSIS STRENGTHS & LIMITATIONS OF THE EVALUATION EVALUATION RESULTS TRAINING RESULTS QUANTITATIVE FINDINGS QUALITATIVE FINDINGS FOCUS GROUPS RESULTS FRONT-LINE WORKERS FOCUS GROUP (N=5) PRACTICE LEADS FOCUS GROUPS (N=9) SUPERVISORS FOCUS GROUP (N=12) SUMMARY OF FOCUS GROUP DATA (N=26) SUPERVISOR SOS FIDELITY RATING CHECKLIST RESULTS BACKGROUND AND DEMOGRAPHICS RATING CHECKLIST IMPACT OF WORKER LEVEL OF SKILL ON RATINGS IMPACT OF YEARS OF SOS PRACTICE ON RATINGS IMPACT OF SOS RELATED TRAINING ON RATINGS QUALITATIVE WORKER OUTCOMES CLIENT INTERVIEWS QUANTITATIVE FINDINGS QUALITATIVE FINDINGS FILE REVIEWS SUMMARY OF YEAR I IMPLEMENTATION APPENDIX A - SOS LOGIC MODEL APPENDIX B - SUPERVISOR SOS FIDELITY RATING CHECKLIST APPENDIX C - PARENT FEEDBACK CHECKLIST APPENDIX D - FILE REVIEW TOOL APPENDIX E - SUPERVISOR SOS FIDELITY CHECKLIST RESULTS

4 Signs of Safety Evaluation 8/14/2015 List of Tables Table 1. Respondents by training type... 8 Table 2. Training satisfaction ratings... 9 Table 3. Summary of focus group data Table 4. Workers SOS knowledge and skill level Table 5. Parent Feedback Checklist descriptives Table 6. Client Feedback regarding caseworkers Table 7. File review comparison between Regular Intake Workers and SOS Super Users Table 8. Year I SOS implementation Logic Model Table 9. Supervisor SOS Fidelity Checklist: Humility & integrity Table 10. Supervisor SOS Fidelity Checklist: Transparency Table 11. Supervisor SOS Fidelity Checklist: Strengths Table 12. Supervisor SOS Fidelity Checklist: Focus on safety Table 13. Supervisor SOS Fidelity Checklist: Participatory & collaborative Table 14. Supervisor SOS Fidelity Checklist: Listening & understanding Table 15. Supervisor SOS Fidelity Checklist: Harm & danger Table 16. Supervisor SOS Fidelity Checklist: Clearly defining safety Table 17. Supervisor SOS Fidelity Checklist: Enhancing safety Table 18. Supervisor SOS Fidelity Checklist: Engaging family members Table 19. Correlational analysis of the relationship between worker variables and ratings List of Figures Figure 1. Percentage of new knowledge gained by participants... 9 Figure 2. SOS implementation rating of team vs. agency by Intake supervisors Figure 3. Number of years respondents supervised the workers being rated Figure 4. Number of years workers practiced SOS Figure 5. Number of workers vs. total days of training per each training type Figure 6. Mean rating scores of beginner SOS workers VS. more advanced SOS workers Figure 7. Mean rating scores of all workers vs. their years of SOS practice Figure 8. Mean rating scores of all workers vs. the number of training days they completed

5 Signs of Safety (SOS) 8/14/ Introduction 1.1 Brief History The Signs of Safety (SOS) approach was initiated in the 1980s and developed through the 1990s by Andrew Turnell and Steve Edwards in Western Australia. Since then, SOS has been increasingly recognized as the leading approach to child protection in numerous jurisdictions of various countries. SOS arose out of a need for a structured and systematic approach to child protection casework as well as the high turnover and decreasing morale of child protection staff (Turnell, 2012). The SOS model was seeking to improve child welfare practices and outcomes and was grounded on what works in practice for the front-line workers and service recipients. An international community of front-line practitioners aided the development of the SOS approach by describing their work, their struggles, and their successes; this appreciative inquiry method is a core principle of SOS as well as the driving force behind its continuous evolution (Turnell, 2012, p. 7). While the innovative aspect of the approach and its tools is exciting in itself, it is important to always keep in mind that the main goal of SOS is a shared focus on child safety. 1.2 Core Principles The SOS approach mainly relies on three core principles: PRINCIPLE 1: Constructive Working Relationships Constructive working relationships and partnerships between professionals and families are central to the SOS framework. According to Turnell (2012), research suggests that such collaboration leads to the best outcomes for vulnerable children and acts as a key to human change and growth. PRINCIPLE 2: Engaging in Critical Thinking and Fostering a Stance of Inquiry Referring to the research of Eileen Munro in child protection, Turnell (2012) emphasizes that it is important for child protection workers to avoid coming to a determination that they know the truth about a family or situation; such a position can prevent workers from accepting or critically examining new information that doesn t conform to their original position. Instead, Turnell (2012) urges protection workers to adopt a stance of questioning and inquiry. PRINCIPLE 3: Remaining Grounded in Everyday Practice According to Turnell (2012), the paternalistic approach that was often seen between child protection workers and their clients also exists in the relationship of upper management and academics towards front-line practitioners. Turnell (2012) emphasizes that this command and control social work, which is not productive and alienating, should be abandoned in favour of an approach grounded in the everyday good-practice experiences (p. 11). 1.3 Practice Elements In their original publication, Turnell and Edwards (1999) outlined six key practice elements, which have been largely derived from Solution Focus Brief Therapy (SFBT) and modified to meet the needs of child protection work: 1. UNDERSTAND THE POSITION OF EACH FAMILY MEMBER aids the worker to recognize and respond to the uniqueness of each person and each family. 2. FIND EXCEPTIONS TO THE MALTREATMENT finding times when the problem happens less or doesn t occur helps the client understand what they did in the past and can do in the future to move towards the preferred outcome. 3. DISCOVER STRENGTHS AND RESOURCES it is important to notice and name the unique strengths and resources the client has, as it allows the client to envision the possibility of change within their own means. 4. FOCUS ON GOALS involving the family in identifying their own goals and comparing these with the agency s goals in a transparent manner aids the working relationship and the safety of the child. 5. SCALE SAFETY AND PROGRESS a straightforward way to assess the perspectives of clients and workers as well as any changes that might have occurred over time along the continuum from danger to safety. 6. ASSESS WILLINGNESS, CONFIDENCE AND CAPACITY using scaling questions, increased willingness, confidence, and capacity of the family to follow through with plans will increase the likelihood of the successful implementation. 4

6 Signs of Safety Evaluation 8/14/ Framework and Tools The SOS approach aims to utilize a comprehensive approach to risk while simultaneously exploring strengths and safety, with the full involvement of all professionals and family stakeholders. This risk assessment framework is carried using a one page map (Assessment and Planning Form) addressing four domains: 1. WHAT WE ARE WORRIED ABOUT - past harm to the child, future danger, and any existing complicating factors; 2. WHAT S WORKING WELL - existing family strengths and safety; 3. WHAT NEEDS TO HAPPEN - to obtain the required safety; and 4. A SAFETY JUDGMENT - on a scale of zero to ten, where ten means that the child protection case may close and zero means certainty of re-abuse. This one page map is used to guide safety planning by creating a specific set of rules that indicate how the children will be safe in the future. Another key aspect of the SOS approach is involving the voices of children using specific tools, such as THREE HOUSES, WIZARDS AND FAIRIES, SAFETY HOUSE, and WORDS AND PICTURES. Using these tools, child protection workers have tangible ways to help children express themselves, understand why child protection services are involved, and be involved in the safety planning, all in a developmentally appropriate way. 1.5 Evidence Base Studies evaluating SOS implementation and outcomes have been completed in multiple countries, including but not limited to Australia, USA, Canada, the Netherlands, Japan, and the UK. Although a thorough literature review is beyond the scope of this report, these are the main findings on the SOS use to date (as cited in NSPCC, 2013): 1. Improvement in front-line workers job-satisfaction; 2. Improvement in workers-client relationship, which was found to be vital to successful child protection outcomes; 3. Increased involvement of families in casework processes; 4. SOS risk assessment process is effective; clarifies concerns, risk, and safety; 5. Very preliminary data from two USA jurisdictions show positive changes in child protection statistics. 1.6 Signs of Safety Implementation at Children s Aid Society of Toronto Planning for SOS implementation commenced in 2012 with the agency s Practice Leadership Committee (PLC) getting exposure to the SOS framework and introducing it to the agency. Other practice models (i.e. Supervision Model, Family Centered Conferencing) were examined with regards to collaboration with SOS. Connected Families, an organization that has been helping agencies adopt SOS since 2008, provided consultation on implementation, clinical practice and training. SOS was formally implemented at the Children s Aid Society of Toronto (CAST) in March 2013 and the first round of training occurred in April 2013; two days of training were delivered by CAST trainers and two more days delivered by Connected Families. Supervisors received an additional day of training from Connected Families. To support continuous learning and development, internal resources have been put into place: Weekly practice enhancement sessions to promote peer SOS knowledge exchange and enhance application Ongoing introductory and Connected Families training, safety planning, and branch management training. Turnell often notes that agency-wide implementation of a new clinical framework is a ten year journey. Therefore, with over a year of SOS implementation at CAST now complete, the journey is yet at its infancy. 1.7 Anticipated Outcomes of Signs of Safety Implementation The full list of anticipated short-term, intermediate, and long-term outcomes can be viewed in the Logic Model (see Appendix A). Focusing on Year I of implementation, the main anticipated outcomes were: OUTCOME 1: Protection and non-protection staff trained in introductory SOS OUTCOME 2: Advance staff skills in SOS assessments, safety planning and family engagement OUTCOME 3: Improved family satisfaction with CAS services OUTCOME 4: Improved worker job satisfaction 5

7 Signs of Safety (SOS) 8/14/ Evaluation Methodology The Child Welfare Institute (CWI) has been requested to take the lead with the evaluation of SOS implementation. Joint planning between CWI and Practice Leadership Committee (PLC) resulted in the following evaluation objectives for Year I of SOS implementation: OBJECTIVE 1: To evaluate training of staff in SOS OBJECTIVE 2: To evaluate ability of staff to implement SOS philosophy and practice OBJECTIVE 3: To evaluate the effectiveness of management s support of SOS OBJECTIVE 4: To evaluate the strengthening of services to families through SOS practice A combination of quantitative and qualitative research methodology was utilized to meet the objectives of the evaluation. Data were collected using surveys, focus groups, interviews, and file reviews. 2.1 Evaluation Tools Training Surveys (N=138; 119) Two-part SOS training was completed in Spring and Fall of The trainings were evaluated using a standardized online survey, routinely used by CWI. The survey included 11 multiple choice questions regarding course content, instructors, and applicability to one s work. There was one open ended question asking for additional feedback and recommendations as well as a comment section following every multiple choice question for deeper understanding of trainees experience. Out of a total of 355 training participants, 138 SOS Part I trainees and 119 SOS Part II trainees completed the evaluation survey Focus Groups (N=26) Three separate focus groups were conducted with front-line protection workers (n=5), Practice Leads (n=9), and Intake supervisors (n=12) between February and December of The focus group questions were designed by CWI researchers and aimed to assess the agency s current stage of SOS implementation as well as the benefits and challenges encountered. In addition, feedback and recommendations with regards to moving forward were gathered Supervisor Fidelity Rating Checklist (N=30) The Supervisor Fidelity Rating Checklist is a component of a larger research project undertaken by SOS Fidelity Workgroup, which is comprised of international members. The aim is to develop fidelity measures for the SOS approach and learn how SOS is being used by child protection organizations. The Checklist contains 35 rating scales assessing workers practice skills and use of SOS, six open-ended questions and background information to be completed by supervisors (see full Checklist in Appendix B). Six CAST Intake and Family Service supervisors evaluated five of their workers using the Checklist during Summer and Fall of The tool was completed using an online platform created by the SOS Fidelity Workgroup, allowing CAST supervisors to submit their data directly to the Workgroup. The data were then de-identified and shared with CWI as per a data sharing agreement for the purposes of this evaluation Client Interviews (N=9) Client interviews were completed using the Parent Feedback Checklist, which is another tool developed by the SOS Fidelity Workgroup. The tool contains 19 rating scales assessing parental experience with their most recent child protection worker (see full Checklist in Appendix C). As opposed to the Supervisor Checklist, data from the Parent Checklist WERE NOT submitted to the Fidelity Workgroup or shared with anyone outside of CWI. A researcher from CWI completed the tool with parents via phone interviews. Interviewees were selected using specific criteria and were compensated with $25 for their participation. A total of nine interviews were completed in two rounds of interviews that took place in March 2014 and February

8 Signs of Safety Evaluation 8/14/ File Reviews (N=30) A total of 30 CAS Intake files were randomly selected for review from all files opened by Intake between April 1 and June 30 of 2014 and transferred to Ongoing Family Services. The goal of the file reviews was to assess the use of SOS principles across Intake workers in order to examine the impact that SOS use has on family outcomes. Of the 30 files reviewed, nine belonged to Intake workers who had additional training in SOS and were known to be ahead in their SOS practice when compared to the agency as a whole; these nine workers will be referred to as SOS SUPER USERS. The remaining 21 files belonged to Intake workers without such classification, which will be referred to as REGULAR INTAKE. The contents of file transfer recordings were reviewed by CWI researchers using a File Review Tool (see Appendix D) developed by CWI in collaboration with PLC. 2.2 Data Analysis All quantitative and qualitative data were analyzed by CWI. No one other than the research staff had access to the data (with exception of the SOS Fidelity Working Group having access to the Supervisor Fidelity Rating Checklist as per the data sharing agreement). Qualitative data included client interviews, focus groups, file reviews, and qualitative survey questions, which were analyzed for thematic content. Quantitative data included training surveys, Supervisor SOS Fidelity Rating Checklist, and Parental Rating Checklist and were inputted and analyzed by CWI using the Statistical Package for the Social Sciences, version 20 (SPSS). Data is presented in aggregate, and no individualized agency data is reported. 2.3 Strengths & Limitations of the Evaluation The main strength of this evaluation is the in-depth data collection from multiple stakeholder groups using mixed qualitative and quantitative methodology. These diverse sources and types of data increase confidence in the findings obtained. In addition, this evaluation report adds new knowledge, although preliminary, in an area where little research exists to date. With SOS becoming the leading clinical framework to child protection casework in multiple countries, more insight is needed into the implementation of SOS as well as its impact. While every effort was made to mitigate data collection limitations, there were some challenges in collecting the data, mainly the variable exposure employees and departments had to SOS. More specifically, given the agency-wide change in practice brought on by the implementation of SOS, there was a need for a rapid evaluation process to examine whether the new clinical framework was beneficial, and if so, to what extent. While the need for swift results is understandable, conducting an evaluation at such early phase of SOS implementation limits the sample sizes, the breadth of the analysis possible at this stage as well as the applicability of the findings. Although common in evaluation research, there are challenges associated with retrospective file reviews. It is a method in harnessing or utilizing information that was not originally collected for research and evaluation purposes. However, given that the evaluation goal was to explore preliminary client outcomes and capture possible differences between SOS Super Users and Regular Intake groups, file reviews were seen as a unique opportunity and an appropriate method given the exploratory nature of this evaluation. The file review findings should be interpreted with caution as the total sample size of 30 is not necessarily representative. This sample size was chosen due to feasibility within a short time period. Finally, despite the high importance placed on SOS implementation and the production of an evaluation report, there were competing and overpowering CPIN-related priorities that rendered it impossible to allocate SOS the full attention it deserves. 7

9 Signs of Safety (SOS) 8/14/ Evaluation Results 3.1 Training Results The first round of SOS training took place between April and June of 2013 (Spring) and the second round of training took place between October and November of 2013 (Fall). Three types of training took place during these times: 1. SOS PART I, INTRODUCTION TO SOS - the Spring introduction training was designed for all protection staff and protection supervisors and was delivered in two days by CAST trainers. Topics covered were the philosophy behind SOS, practice principles, skillful use of authority, and partnering with families. The use of questions was introduced as well as danger statements and tools within the SOS framework. It should be noted that during the Fall session, this training was reduced to one day due to feedback stemming from the Spring session. 2. SOS PART II, ADVANCED PRACTICE FOR FRONT LINE WORKERS both Spring and Fall sessions were delivered to protection staff in two days by Connected Families. Training curriculum included an in-depth understanding of SOS and focused on the use of the questioning approach and various clinical tools. 3. SOS PART II, ADVANCED PRACTICE FOR PROTECTION SUPERVISORS child protection supervisors received three days of training from Connected Families. Please note that for the first two days of this training, supervisors were training together with front line workers while the third day of training was for supervisors only. All trainees were asked to complete an evaluation survey following the training. The number of survey respondents per training type ranged from 32% to 40% of total trainees. Table 1 summarizes of the number of trainees and survey respondents within each training type in the Spring and Fall sessions. Readers should be conscious of the overlap between attendants of the three training types. Please note that additional training for protection workers, intensive training for protection supervisors, and introductory training for non-protection service staff took place in late 2014; however, only the evaluation of the three training types described above has been included this report. TABLE 1. RESPONDENTS BY TRAINING TYPE Spring 2013 Fall 2013 TOTAL Training Type Trainees Survey Respondents Trainees Survey Respondents Trainees Survey Respondents SOS Part I (40%) SOS Part II for Front Line Workers (34%) SOS Part II for Protection Supervisors (32%) Quantitative Findings Survey respondents were asked to rate several SATISFACTION STATEMENTS related to course content, trainer delivery, and course applicability. Satisfaction ratings were made on a five-point scale, ranging from Poor to Excellent. The majority of respondents rated all questions positively (either Excellent, Very Good, or Good ), where the combined ratings of Fair and Poor never exceeded 22% of the respondents. Table 2 provides a summary of these results; please note that for the purposes of this analysis, participants of SOS PART II FOR PROTECTION SUPERVISORS were combined with SOS PART II FOR FRONT LINE WORKERS given that they had identical two days of training and the low number of supervisors who completed the evaluation survey. Interestingly, there was a statistically significant difference in satisfaction ratings of respondents between the Spring and Fall sessions. Specifically: The Mean ratings of SPRING TRAINEES in SOS Part I were all lower than the Mean ratings of FALL SOS Part I trainees. The differences that reached statistical significance at p<.05 are highlighted in pink. This indicates that the changes made in the curriculum and length of the training between seasons was favourable. The Mean ratings of SPRING TRAINEES in SOS Part II were all higher than the Mean ratings of FALL SOS Part II trainees. The differences that reached statistical significance at p<.05 are highlighted in orange. 8

10 Signs of Safety Evaluation 8/14/2015 Curriculum TABLE 2. TRAINING SATISFACTION RATINGS Questions M (of 5) Spring 2013 Fall 2013 SOS Part I SOS Part II SOS Part I SOS Part II % Good & M (of % Good & M (of % Good & M (of Very Good & 5) Very Good & 5) Very Good & 5) Excellent Excellent Excellent (N=105) (N=94) (N=33) (N=25) 100% New 80% New 60% New 40% New 20% New Fall: SOS Part II (N=119) 0% 10% 20% Participants SOS Part I (N=138) 30% 40% % Good & Very Good & Excellent Q1. The curriculum content was % % % % clear and relevant Q2. The workshop % 4.01* 95% % 3.64* 80% expanded/enhanced my understanding of the topic Q3. The workshop exercises % 4.18* 95% % 3.79* 83% stimulated learning and encouraged discussions/questions Q4. The instructor was % % % % knowledgeable and effective in delivery of the subject matter Q5. The instructor was prepared for % % % % the course and was well organized Q6. The instructor was open to % % % % diverse populations Q7. The instructor effectively % 4.18* 98% % 3.68* 84% addressed challenging and difficult content/issues/questions Q8. The likelihood that I will apply % % % % the course learning to my work Q9. My overall rating is % % % % Q10. The likelihood that I will recommend this workshop to colleague/co-worker is % 4.14* 93% % 3.60* 80% *Approaching statistical significance at p=.07 Survey respondents were also asked what PERCENTAGE OF THE CURRICULUM WAS NEW to them. No significant differences were found between the responses of trainees in the Spring sessions and trainees in the Fall sessions. Therefore, the Spring and Fall responses were combined and are presented in Figure 1. Evidently, over half of survey respondents in both SOS I and SOS II training (66% and 73% respectively) indicated that between 60% and 100% of the curriculum was new to them. This indicates that the majority of participants learned quite a bit in both parts of the training. 9 FIGURE 1. PERCENTAGE OF NEW KNOWLEDGE GAINED BY PARTICIPANTS

11 Signs of Safety (SOS) 8/14/ Qualitative Findings Participants were asked to COMMENT ON THEIR SATISFACTION RATINGS. There is some contrast between the quantitate results, which were summarized above and participant comments; while the participants ratings on the various satisfaction items outlined above were mostly positive, the majority of the comments made by trainees appear to be negative. This can possibly be attributed to the fact that feedback comments are often thought of as an opportunity to point out areas for improvement, and thus will inherently suggest the negative aspects of the workshop. Another possibility is that people who are dissatisfied tend to be more vocal. Those who made negative comments typically rated questions lower than the majority; this indicates that the majority who rated questions higher typically abstained from commenting. On average, 12% of respondents chose to provide written comments on individual questions. The most frequent feedback was that the training content had a lot of repetition from previous training sessions and that there was little expansion of their knowledge on the subject. This feedback came up in both training types (SOS I and II) and all sessions (Spring and fall), indicating repetition of content within each training as well as between trainings. As mentioned previously, PLC has been gradually exposing the agency to SOS since Therefore, it is understandable that multiple workers who engaged in SOS learning early on had preexisting knowledge of the introductory topics covered. However, there were many more workers who did not engage in early SOS learning and did find that the majority of the curriculum presented was new to them, which is evident in Figure 1. Therefore, as part of an agency-wide implementation strategy, it was necessary to begin training at the lowest common denominator of SOS knowledge and skill. Sample quotes of participants on both ends of the spectrum: I went into this training thinking I knew a lot but it turned out I didn t. This training expanded my knowledge. I felt as if it was unnecessary to have four days of this training, especially since most of the information overlapped. Too much time focused on the theory and not on the application of SOS. In addition to comments regarding rating questions, survey respondents were asked to provide RECOMMENDATIONS for future learning related to SOS. Of a total of 257 survey responses across training types and sessions, 35% provided additional written feedback: THEME 1: Repetition of information between training sessions Participants expressed that they would ve liked less overlap and repetition between the SOS training sessions. The second SOS training did not provide a lot of different information from the first. There was a lot of overlap in terms of the theory, and I would have liked to see more time spent on the tools (e.g. safety house). THEME 2: Practice based approach rather than theory based approach Participants mentioned they would ve benefitted more from learning practical application of SOS concepts and tools into their practice with clients. In my opinion, if the training had more practical components (for example how we would or should start using this in our everyday practice), it would have increased its benefits and clarity. While I learned concepts that I found interesting and useful, it would have been great to have learned a tentative guide, for how these tools could start to be integrated into practice. THEME 3: Exposure to applicable tools with diverse clients Participants expressed that in addition to learning more practical applications of SOS, they also would have liked example of cases that incorporate both diversity and complexity that we often encounter within Toronto. This model was presented well, however the examples used to better understand the concerns were quite simple and did not generally reflect the complexity of what we deal with in Toronto. Most of the examples also dealt with adolescents There was very little, if any detailed examples of working with families of babies and toddlers, especially those that have generationally been involved with the child welfare system. 10

12 Signs of Safety Evaluation 8/14/ Focus Groups Results Focus groups were conducted with three different stakeholder groups in order to get their perspective on SOS implementation and impact within the agency; front-line workers, practice leads, and supervisors were able to provide a unique perspective by virtue of their different roles within the agency. All focus group participants were asked questions that were broadly categorized around what was going well, not going well, and needed to change Front-Line Workers Focus Group (n=5) Five CAST front line workers participated in a focus group in March The following qualitative report organizes the responses into themes within each of the topics discussed. The themes are ranked in order of significance. SOS UPTAKE AND IMPLEMENTATION Workers were asked to rate the Society on a scale of 1to 5 with regards to SOS uptake and implementation. The responses averaged at 2.5 indicating the beginning of SOS momentum and emerging use of SOS framework, tools, and language in practice across some (but not all) teams. Front line staff also shared that uptake and full implementation are not the same thing. Three other CAS none of those are doing SOS...full implementation is not just about us it is all the way through the entire system WHAT IS GOING WELL? THEME 1: Improved Parent Experience through Partnership Approach Most workers (n=4) reported a positive change towards a better working relationship with families through SOS emphasis on partnering with the family. As a result of the framework encouraging collaborating with the families on keeping the child safe, workers felt that families cooperated and engaged with the process more. Particularly, they personally noticed a change in how families react to their involvement now compared to before SOS use. In addition, it was mentioned that the partnership approach allows for better and more open communication with the family. Where it has impacted with my practice dealing with denied child abuse, and working with them different allows to unstuck the argument more balanced in treating them as human beings and I feel better as a human being which decreases the defensiveness. THEME 2: Increased Worker Satisfaction Workers mentioned a number of points that related to their work satisfaction as a result of implementing SOS. A number of workers (n=3) felt that due to SOS partnership approach, families are responding better to their involvement and hence they feel they are accomplishing their job better. The partnership has been important working with a family now mom is really upset and to have honest and open communication and having really challenging conversations had really good experience with 3 houses/3 columns family had found it life changing as a worker the SOS allows us to be proud of our work THEME 3: Improved Practice and Risk Assessment Most workers (n=3) commented that SOS techniques improve practice by focusing on risk in the right ways. Workers felt that SOS does a great job as a framework to assess risk in a more balanced way and plan for safety with the family. One worker commented that safety planning with SOS is more meaningful. THEME 4: Utility of SOS Tools Workers felt that SOS tools were useful in bringing the child s voice into the conversation and better allowed them to think through safety and risk, resulting in a higher quality of work. bringing the voice of the child into the investigation developing the relationship with the kids they feel ok enough to talk to me this allows me to take what they said to then work with the parents and use of 3 houses really helps 11

13 Signs of Safety (SOS) 8/14/2015 WHAT IS NOT GOING WELL? THEME 1: Requires More Time There was general agreement among the workers that SOS tools and methods take longer to implement. At the same time, a number of workers noted that the extra time is worthwhile because cases do not come back. SOS does take longer but I don t go back and I don t reopen and I do transfer frustrating that I use the framework but it is not followed up on I had a 45 min call with a worker about why I was transferring a case...they felt that supports equals safety but I had two danger statements and mom hitting this child. THEME 2: Not Fully Supported at Agency A number of workers (n=3) felt that they had to practice SOS in isolation because it does not have enough uptake in the agency and is not fully supported. Workers commented that they had to gain allies and persuade people to see things from SOS point of view. They found it difficult that even though they may use SOS, others may not follow up with SOS. Particularly, it was noted that while SOS is the agency s adopted framework, management and supervisors are lagging behind in their knowledge and implementation of SOS which is a challenge for front-line workers looking to their supervisors for support. Easy to go back to the old ways because everyone is doing it that way and everyone will support you vs doing it SOS way and do it differently practicing the SOS in isolation and my work is trying to convince my supervisor I can keep the child safe with SOS HOW CAN WE IMPROVE? A number of suggestions were offered by workers on how to achieve thorough implementation of SOS at the agency: Get management and supervisors extensively trained in SOS; and they should make SOS use non-optional for themselves and staff; Allocate funding towards sustaining SOS at the agency; Management should be loud and visible in their support and use of SOS; Various techniques need to be employed to help front-line workers get exposure to SOS, such as practice sessions, shadowing of SOS experts, discussions during supervision, and sharing successes and challenges Practice Leads Focus Groups (n=9) Nine CAST Practice Leads from Toronto and Scarborough branches participated in a focus group in February The following qualitative report organizes the responses into themes within each of the topics discussed. The themes are ranked in order of significance of how frequently and strongly they were expressed. SOS UPTAKE AND IMPLEMENTATION Practice Leads were also asked to rate the Society on a scale of 1to 5 with regards to SOS uptake and implementation. Similarly to front-line workers, the responses averaged at 2.7 indicating the beginning of SOS momentum. However, there was quite a range in the responses; one participant indicated no training in his/her department to date while another participant indicated a longer journey with SOS. WHAT IS GOING WELL? THEME 1: Increased Job Satisfaction and Enthusiasm for Staff A number of practice leaders (n=4) identified that SOS use by staff has resulted in greater job satisfaction and enthusiasm about the work. Another key element identified that SOS framework and tools allow for deeper conversations with clients; these help the workers feel proud of their work. People s enthusiasm for their work, I have seen a real shift in people s enjoyment of the work. They learn more, getting in much deeper conversations and making connection and relationships, that is something I am very proud of 12

14 Signs of Safety Evaluation 8/14/2015 THEME 2: Improved Parent Experience through Collaborative Approach A number of workers (n=4) reported that families are finding the new approach useful and positive. It was noted that families are feeling CAS involvement is helpful and particularly that their voice is being heard in the process. Notably, the SOS tools are being received very well by the families such as mapping, scaling, safety plans, etc. Finally, it was also mentioned that SOS has helped reduce the anxieties of clients around CAS involvement. Some feedback, after a mapping, services recipients talking about that this felt really good, they felt like they had a voice, they say it reduced my anxiety about working with the agency, when people can voice this, or people who have received the services it feels good. THEME 3: Utility of SOS Approach and Tools Another area that a number of practice leaders (n=4) touched upon was the usefulness of the SOS approach and tools. The framework presents an order and consistency for the workers to rely upon. For families, the approach and tools foster greater collaboration, understanding, empathy and reduced anxiety. had a mom say she trusts me cause she thinks I am here for her kids, cause of the questions we are asking, the way we frame them. THEME 4: High Level of Skill among Staff A number of practice leaders (N=3) spoke explicitly about their observation that many staff have significantly improved in their knowledge and implementation of SOS. Through webinars and trainings the practice leaders got the sense that staff are significantly improving in their use and implementation of SOS at the agency. It was also noted that SOS is increasingly being practiced by staff at the agency. We had a webinar with connected families it become clear to me, the level of skill across the agency. I was shocked. It s amazing seeing the level of skill and how happy people are. WHAT IS NOT GOING WELL? THEME 1: Lack of Buy in from Management and Staff There was consensus among the practice leaders that parts of the agency are lagging behind with regards to various elements of SOS. For one it was identified that management does not understand SOS well and hence does not use it, such as in supervision. Practice leaders emphasized that if management wants SOS implemented in the agency, management will have to champion SOS in the agency and get visible in their support and use of SOS. One Practice Lead noted resistance to the SOS framework at the staff level as an issue. They [management] need to have bigger voice in this they are so pivotal in setting the tone and expectations, and there is a language that happens with SOS and everyone has to understand it and speak it so it is common language between branches and services so it s all the same. Been a bit of a struggle to grow people s perceptions that this framework is different than what we are doing if you can t get that piece then it s hard to move forward. THEME 2: Split Priorities A majority of supervisors (n=4) expressed that the agency keeps taking on various initiatives that sideline progress on SOS. They noted that a sustained effort and resources need to be invested in implementing SOS throughout the agency. It was also mentioned that some initiatives are said to be complementary to SOS but no direction is given on how they are complementary, and how they can be integrated. the struggle is all the initiatives aren t integrated so we talk about it but we don t go much beyond just talking about it for example with supervision model, family centered models, no integration of all of them together what does that mean for a front line or middle management position? I think I confuse workers at times. 13

15 SOS Implementation Rating Signs of Safety (SOS) 8/14/2015 HOW CAN WE IMPROVE? Get management and supervisors extensively trained in SOS; make SOS use non-optional across agency; Management should be loud and visible in their support and use of SOS; Management must better integrate (or prioritize) the various initiatives that have been taken on along with SOS; Management can showcase success stories of SOS on CASTLE, for example. This will help workers endorse and implement SOS; Practice leaders emphasized the need for workers to practice implementing SOS even if it means starting with one scaling question; Workers should be constantly talking about implementing SOS and create a culture of learning around it; Some practice leaders suggested that SOS use with families should be videotaped to showcase how it works and display its effectiveness. These videos can be used in team meetings Supervisors Focus Group (n=12) Twelve CAST Intake supervisors participated in a focus group in December The following qualitative report organizes the responses into themes within each of the topics discussed. The themes are ranked in order of significance of how frequently and strongly they were expressed. SOS UPTAKE & IMPLEMENTATION Similarly to front-line workers and Practice Leads, CAST supervisors were asked to rate the level of SOS uptake and implementation. However, as opposed to the other two focus groups, supervisors were asked to provide two ratings; one for their own team and one for the agency as a whole. Another change was that the supervisors were asked to rate on a 10-point scale (as opposed to 5-point scale) using an Implementation Map Template provided by Connected Families; this tool was not available for the other focus groups as they were conducted much earlier in the year (Feb- Mar 2014). On average, supervisors rated their own teams a little higher than the agency as a whole (M Team =2.86 vs. M Agency =2.75). The gaps per tram can be observed in Figure 3. These average ratings both indicate that the agency is finishing the Skill Building stage of implementation and is at the verge of Building Depth stage. Although these results cannot be directly compared to those of the other focus groups due to the different scale used, it is interesting to note that front line workers and practice leads felt that CAST was half way through the implementation scale while 9 months later supervisors felt CAST was only a quarter through the implementation scale. This findings may be in part due to a common phenomenon that occurs with continuous learning, where the more one knows about a subject the more one realizes that there is so much more to know; if this is indeed occurring at the supervisory level at the end of Year I of SOS implementation, this is could be considered as a positive outcome, indicating progress in the implementation journey P1 P2 P3 P4 P5 P6 P7* P8 P9 P10 P11 P12 Intake Supervisors FIGURE 2. SOS IMPLEMENTATION RATING OF TEAM VS. AGENCY BY INTAKE SUPERVISORS Team CAST * Missing 14

16 Signs of Safety Evaluation 8/14/2015 WHAT IS GOING WELL? THEME 1: Improved Parent Experience through Collaborative Approach A majority of supervisors reported a drastic positive change in the quality of work with families as a result of implementing SOS. Supervisors noted that their staff felt the less prescriptive approach really resonated with families. It allowed the whole process to be more of a collaboration with families, while acknowledging that parents are the experts about their lives and their children. A number of supervisors spoke of experiences where families with a history of involvement with CAS found the new approach with SOS the most positive so far. Some parents will say it s the most helpful time we ve been involved I wish I [parent] had this experience the first time around Asking the family what they are worried about is big. Not just what we are worried. This gives a sense of balance and relief. THEME 2: Increased Job Satisfaction for Staff A large proportion of supervisors identified that implementing SOS throughout the agency has led to increased job satisfaction for staff. Most supervisors agreed it was because of SOS focus on taking a collaborative approach with parents that workers felt their work improved and had a greater impact, thus leading to more job satisfaction. A number of supervisors reported that workers are happy with SOS implementation in supervision with techniques like appreciative inquiry. Some supervisors commented they have started using SOS as part of their performance management which has been welcomed by staff. I ve had job satisfaction moments with workers. They report more job satisfaction Aha moments. Workers are happier, more satisfied, they feel like they are more successful in engaging clients, not bossing clients around so much. THEME 3: Hearing the Child s Perspective Most supervisors were in agreement that one of the most unique and beneficial thing about SOS is involving children. Supervisors reported that children s voices and drawings are often extremely impactful for parents. A 5-yr old had drawn a picture very impactful and helpful to parent regarding the issues bringing the child s voice in better; allows us to deal better with risk you have to discuss as you have kids perspective. I ve had parents start crying sometimes. It comes as a surprise to them how much their children know. Parents get to hear the children voices and it s impactful. THEME 4: Utility of SOS Tools Almost all the supervisors agreed upon the usefulness of the various SOS tools: for supervisors, workers, and families. A number of supervisors talked about the positive impact of using the three columns and three houses. They noted that it gives structure to challenging cases. In addition, a few supervisors brought up the usefulness of safety goals and genograms. Finally, within supervision, the utility of appreciative inquiry was noted by some supervisors. They [workers] appreciated the appreciative inquiry because they can identify what they ve done well, without being asked. With challenging cases there is more of a structure the 3 houses are far superior What I see the most of is the common language helps get rid of the noise what is the actual harm, what is the danger helps us weed out when we align too much with one parent over another. 15

17 Signs of Safety (SOS) 8/14/2015 WHAT IS NOT GOING WELL? THEME 1: No Time! Supervisors were unanimous in their agreement that they, and their staff, have no time to properly implement SOS in their practice. In particular, they parsed the issue into two parts: (1) an overwhelming workload does not allow for SOS to be fully implemented, and (2) SOS as a practice framework requires a lot of time. Some supervisors noted that management has given mixed messages, requiring that SOS be implemented but then not reducing workload to actually allow it to occur. We are so busy...30 cases waiting, all workers with 14 cases no appropriate resources and time to ensure the SOS Framework can get implemented. The framework is not optimum for intake. Family services is different. Intake is go go go! Have no control over volume. THEME 2: A Gap in Learning and Leadership A number of varied points were identified as issues related to the leadership surrounding SOS implementation. First, a number of supervisors expressed frustration at the method chosen for implementation, which has not been effective; supervisors questioned how SOS leaders were picked and some expressed concerns about their level of expertise. Others expressed that since, as supervisors, they were not trained in SOS before their staff, they are not able to support staff learning. In addition, a large number of supervisors felt that most staff has not had their SOS training needs met (e.g. lack of specialized training, workload preventing partaking in Friday sessions). Finally, another prominent point was that at the upper management level, only one of the Directors is fully on-board and well-versed in SOS which has been very challenging. The management staff weren t trained first and the leaders weren t trained to support staff in the change process when you have a staff being mentored in SOS and I haven t received the SOS Training, don t know the tools, don t know the language. it created challenges not sure why the decision is made but a re-do would be to train management first. Small group of people is getting lots of training and support and the [rest] are not invited to that part of training The most trained are the ones with reduced workload... Giving mixed messages is not good; we want you to implement but then don t give time to learn Workers are consulting informally with these experts but that s also a problem. Who is teaching who There is a disconnect in the process of consulting Had a worker go off and ask one of them and implemented an SOS technique that took so much time and totally did not need to be done. If I was asked I would have prioritized. THEME 3: Resistance and Push-back A few supervisors mentioned various ways they have faced resistance and push-back from their staff around implementation of SOS. A few brought up the issue of some staff belonging to other agencies where SOS has not been adopted and hence are resistant to accepting it. A supervisor commented that some staff resisted it until external trainers came the internal training was not winning them over. Another supervisor expressed frustration that SOS is drastically changing practice, and there has not been a safe forum to discuss concerns, issues, pros, and cons. concerns with some of the messages of my cases don t come back because we do SOS, which shame the worker. depends on who you speak to some embrace it and some will resist to the bitter end especially up to the past training which was from an outside resource and it allowed staff to conceptualize it better vs. the internal training London dropped it [SOS], Peel dropped it, CCAS won t do it SOS is changing practice no unscheduled visits, no visits to the school, communicating the investigation with parents not had a safe forum to discuss these changes and concerns. 16

18 Signs of Safety Evaluation 8/14/ HOW CAN WE IMPROVE? THEME 1: Reduce Workload Reducing the workload of staff and supervisors would help them in learning and implementing SOS better and more thoroughly (supervisors acknowledged that having a reduced workload is unlikely to take place); Time to allow staff to consult more and implement certain SOS tools that require more time. THEME 2: Implement a Better Delivery Process The process of selecting SOS leads needs to be more fair and transparent; The need to train management and supervisors first so staff could be better supported in their use of SOS; The importance of fostering a better sharing and collaboration culture within the organization; mutual encouragement, learning, and communication would aid the implementation process; Implementation that is tailored to the unique needs/challenges of particular departments; Upper management should lead the implementation by example and role modeling (e.g. within supervision). THEME 3: More Training Opportunities More training opportunities would benefit management and staff; Having training at multiple times during the year to fit staff schedules; Training specific to the populations their staff serve Summary of Focus Group Data (n=26) TABLE 3. SUMMARY OF FOCUS GROUP DATA Front-Line Workers (n=5) Practice Leads (n=9) Implementation Score M=2.5 of 5 M=2.7 of 5 What s going well What s not going well What needs to change Same across groups Different Same Different Same Different Intake Supervisors (n=12) Team M=2.86 of 10 Agency M=2.75 of 10 Improved parent experience through partnership/collaborative approach Increased job satisfaction and enthusiasm for front-line workers/staff Utility of SOS approach and tools (improved practice and risk assessment, hearing the child s perspective and bringing voice into the work) Order, consistency & structure to the work, especially in challenging cases Appreciative inquiry, safety goals & genograms useful for work & supervision Highly level of skill among staff Resistance/push back/lack of practice at worker level Lack of buy in/leadership at management level A gap in learning/knowledge at management level No time to sufficiently learn or practice SOS Split agency priorities No time to sufficiently learn or practice SOS Leadership & visible support from management towards agency-wide full use of SOS Additional training and opportunities to practice SOS for front-line workers Management extensively trained in SOS Agency-wide continuous culture of learning of SOS Funding & resources towards SOS implementation Better integration of agency initiatives with SOS (i.e. Supervision Model) Showcase SOS success stories Videotaped SOS w/families Reduced workload to allow learning & practice to occur

19 Signs of Safety (SOS) 8/14/2015 Number of Workers 3.3 Supervisor SOS Fidelity Rating Checklist Results As mentioned in the Evaluation Tool section, the Supervisor Fidelity Rating Checklist is a tool developed by an international SOS Fidelity workgroup. The checklist contains 35 rating scales, six open-ended questions, and background information to be completed by supervisors for a worker on their team. A summary of the responses provided by six supervisors for a total of 30 workers is presented next. To cater to readers of various professions, the bulk of the highly detailed quantitative results have been placed in an appendix and only the main findings and their relevance are presented in this section Background and Demographics Responding supervisors were asked to provide background information on the workers they were evaluating. Specifically, the information provided was on the number of years supervising the worker as well as the level of SOS skill, experience, and training that the worker has. Results are presented next. Years supervising the worker The supervisors responses to the number of years they were supervising each worker is provided in Figure 4. It appears that half of the 30 evaluated workers were being supervised by the responding supervisors for at least 4 years. Almost half of the remaining 15 workers (n=7) were supervised for 1 year or less. This resulted in an overall average of 3.27 years of supervision or less Number of Years M=3.27, SD=1.87, Median = 3.5 FIGURE 3. NUMBER OF YEARS RESPONDENTS SUPERVISED THE WORKERS BEING RATED Worker SOS Skill Level Supervisors were asked to rate their workers SOS skill level. Table 5 below summarized the responses. Evidently, the majority of the workers (67%) were evaluated as either beginners or early learners with respect to their knowledge and skill level in SOS. It is important to keep this breakdown in mind when interpreting the responses on the upcoming rating scales. TABLE 4. WORKERS SOS KNOWLEDGE AND SKILL LEVEL # % 1. At the novice or beginner skill level 5 17% 2. Early learner skill level 15 50% 3. Moderate skills 3 10% 4. Substantial skills 5 17% 5. Advanced skills 2 7% TOTAL % 18

20 Signs of Safety Evaluation 8/14/2015 Number of Workers Number of Workers Number of Years Workers Practiced SOS Given that CAST is at its first year of implementation, it is natural to find that 19 of 30 (63%) workers were described as having up to 1 year of SOS practice. The remaining 11 workers had 2-5 years of SOS practice experience, which brought the Mean number of years up to 1.6. However, the Median remained at 1.0 and is a more accurate representation of the majority of workers M=1.6, SD=1.03, Median = FIGURE 4. NUMBER OF YEARS WORKERS PRACTICED SOS Number of Years Estimated Days of SOS Training Responding supervisors were asked to estimate the number of days of training that workers had within each of six training categories: 1. INTRODUCTORY TRAINING: Exposure to the model and introduction to mapping 2. INTENSIVE PARTICIPATORY TRAINING: Focus on practicing skills in small groups 3. CONSULTATIONS: Calls with a SOS trainer or consultant who provides case consultations 4. GATHERINGS: Practitioners meet and share their casework in conference settings 5. OTHER SOS TRAINING: Focusing on a specific skill or practice (e.g. questioning skills, safety planning, creating safety networks) 6. OTHER RELEVANT TRAINING: Not specifically SOS, but supports SOS practice The mean number of training days per person was 9.4 and the total training for 30 workers was 283 days. However, there is evidence in the data that a few workers received a particularly large amount of training, therefore are skewing the results (Median total days of training = 7). As evident in Figure 6, there is quite a range in the number of training days received by workers, with very few workers (n=5) receiving intensive training in SOS. The impact of this discrepancy in SOS training and the overall skill level and experience using SOS is examined more closely later on Total Days of Training Introductory Intensive Consultations Gathering Other SOS Non-SOS 19 FIGURE 5. NUMBER OF WORKERS VS. TOTAL DAYS OF TRAINING PER EACH TRAINING TYPE

21 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q22 Q23 Q24 Q25 Q26 Q27 Q28 Q29 Q30 Q31 Q32 Q33 Q34 Q35 Signs of Safety (SOS) 8/14/2015 Mean Scale Ratings Rating Checklist Participating supervisors were asked to rate the performance of their workers by reading the introductory phrase at the beginning of each section and selecting the most appropriate response between Never ( 0 ) and Always ( 10 ) on a checklist containing 35 scaling items. These checklist items are clustered into two main themes, (1) INDICATORS OF GOOD PRACTICE and (2) SIGNS OF SAFETY DIMENSIONS OF PRACTICE, as well as several subthemes, each focusing on a different aspect of the SOS approach to child protection work (see Appendix B). The supervisory ratings of their workers were analyzed and described in detail in Appendix E. The main finding from this analysis is: The majority of workers (over 80%) were rated highly (scores of 7 to 10) on all 14 INDICATORS OF GOOD PRACTICE, where a score of 10 signifies the skill is always practiced by the worker. However, ratings were less high on the SOS DIMENSIONS OF PRACTICE where half of the rated workers or less received 7-10 ratings on the majority of the 21 items. Particularly low scores were seen on items related to use of Mapping and Words and Pictures, implying workers use these tool close to never. Nevertheless, over 80% of workers received 7-10 ratings on their Engagement Skill with Family Members and between 50%-71% of workers received 7-10 ratings on items related to Enhancing Safety Impact of Worker Level of Skill on Ratings In order to examine the impact of worker s level of SOS skill on the ratings provided by their supervisors, a series of comparative and analytical analyses was performed. First, the five worker skill levels listed in Table 4 were clustered into two groups, BEGINNERS versus NON-BEGINNERS. The reason for this clustering was the low number of workers in each of the skill levels (2 to 5 workers), with the exception of Early Learner (n=15). Therefore, a comparison of five groups would be highly inaccurate and presumptuous. Second, ratings for beginner versus non-beginner workers were compared across all 35 rating items (see Appendix E for full quantitative results). A t-test comparison found a statistically significant difference where p<.05 on almost all questions, indicating that WORKERS WITH MORE ADVANCED SOS KNOWLEDGE AND SKILL WERE CONSISTENTLY RATED HIGHER THAN WORKERS WHO ARE NEW TO SOS OR HAVE BEGINNER SKILLS ONLY (SEE FIGURE 6). Beginner Advanced Scale Items FIGURE 6. MEAN RATING SCORES OF BEGINNER SOS WORKERS VS. MORE ADVANCED SOS WORKERS 20

22 Signs of Safety Evaluation 8/14/2015 Mean Scale Ratings The difference between beginner and more advanced workers was particularly profound on items Q15 through Q18, which pertain to the use of SOS mapping. Not surprisingly, it appears that workers with more advance SOS skills are much more likely to use mapping for multiple purposes than workers with beginner SOS skills (see means for both groups in Appendix E). For item Q19, which pertained to the use of SOS tools with children, the difference between the two groups of workers was still strong and statistically significant at p=.001, but much less profound than the former; this implies that front line workers, who are both beginners and more advanced SOS practitioners, use SOS tools with children more frequently and readily than they use mapping Impact of Years of SOS Practice on Ratings A correlational analysis was conducted to assess the strengths of the relationship between the number of years workers practiced SOS to the ratings they received by their supervisors (see Appendix E for full results). THE YEARS OF SOS PRACTICE HAD A SIGNIFICANT POSITIVE CORRELATION TO THE MAJORITY OF RATING ITEMS, INDICATING THAT AS THE FORMER INCREASED SO DID THE LATTER (see Figure 7) R² = Years of SOS Practice FIGURE 7. MEAN RATING SCORES OF ALL WORKERS VS. THEIR YEARS OF SOS PRACTICE Strong positive relationship (r s >.50) was found on items Q11, Q12, Q13, Q15, Q16, Q17, Q18, Q19, Q21, Q22, Q23, and Q25. The strengths of these correlations imply that the ratings given to workers can increase alongside with workers increasing time spent using SOS. The strongest three correlations (r s =.65 to r s =.69) were obtained on items Q15-Q18 (use of mapping) and Q23 (use of scaling questions) Impact of SOS Related Training on Ratings A correlational analysis was conducted to assess the strengths of the relationship between the number of days of SOS related training workers had to the ratings they received by their supervisors (see Appendix E for full results). THE DAYS OF TRAINING HAD A SIGNIFICANT POSITIVE CORRELATION TO THE MAJORITY OF RATING ITEMS, INDICATING THAT AS THE FORMER INCREASED SO DID THE LATTER (see Figure 8). Strong positive relationship (r s >.50) was found on items Q5, Q7, Q10, Q15-Q18, Q20, Q24-Q31, and Q35. The strongest three correlations were obtained on items Q20 (r s =.78; distinguishing between past harm and future dangers), Q24 (r s =.82; distinguishing between strengths and acts of protection), and Q30 (r s =.76; checking in to determine if safety plan is working). Item Q4 was only approaching statistically significance (p=.056) with medium strength correlation (r s =.35). Items Q6, Q11, Q19, Q32-Q34 were weakly correlated with days of training (r s =.15 to r s =.29) and were not statistically significant. 21

23 Signs of Safety (SOS) 8/14/2015 Mean Scale Ratings R² = Days of Training FIGURE 8. MEAN RATING SCORES OF ALL WORKERS VS. THE NUMBER OF TRAINING DAYS THEY COMPLETED Qualitative Worker Outcomes Supervisors were asked what else did you see the worker do to achieve the key case outcomes. Supervisors identified four areas where workers excelled, to help their clients: THEME1: Communication and Collaboration The worker uses respectful and simplistic language to ensure the family understands the key issues and is able to participate in serious discussions. Works hard on building a collaborative service plan. Clearly explains what worker is worried about. When communicating with parents, worker often looks at impact of the child protection worries onto the children; Takes direction from the families; and involves the family s professional supports in the service plan. THEME2: Good casework and clinical skills Worker begins intervention from service recipients strengths as opposed to area of needs; Worker has emotional intelligence; and Worker uses Family Centered Conferencing and Family Group Conferencing. The worker engages well with families, does quite a bit of work partnering and building trusting relationships. THEME3: Workers incorporated SOS into their practice The worker is open to learning about SOS and is asking questions differently using scaling questions to get a sense of how the risks might be viewed by the family Using Family Centered Conferences to bring service recipients and their personal and professional supports together in consolidating a safety plan and communicating worries in one setting where everyone hears the same messages uses the service plan which outlines the objectives and tasks to address the worries in ongoing visits and provides a copy of the service plan to service recipients that outlines SMART goals. THEME4: Advocates for the client This worker is very skilled at advocating for clients and ensuring their voice has been heard. The worker is reflective and will come and speak to me about what might be barriers for the family and will reflect on own role. Additional qualitative feedback on the checklist itself can be seen in Appendix E. 22

24 Signs of Safety Evaluation 8/14/ Client Interviews Potential clients were selected to be contacted for an interview based on the following criteria: 23 Families who had an open CAST file for at least 30 days within the last six months; Families who were closed to CAST for at least one month; Families who experienced SOS as the guiding approach during the most recent file opening; Families with previous file openings with CAST where SOS was not the guiding approach. Once appropriate clients were selected based on the above criteria, the most recent CAST worker for the family was asked to contact the primary care provider and ask for their consent to have CWI researchers contact them via telephone for the purposes of SOS evaluation. The first round in March 2014 identified 14 clients who consented to have contact with CWI, of which four interviews were successfully completed. The second round in February 2015 identified eight clients who consented to have contact with CWI, of which five interviews were successfully completed. Consenting clients (n=9) completed the Parent Feedback Checklist, which contains 19 rating questions and two open ended questions Quantitative Findings The 19 rating questions pertained to the clients most recent experience with CAST worker. Clients were asked to select the most appropriate response between Never ( 0 ) and Always ( 10 ). A summary of the responses provided by six supervisors for a total of 30 workers is presented in Table 6. TABLE 5. PARENT FEEDBACK CHECKLIST DESCRIPTIVES Questions Descriptives N Min Max Mean SD Median How often has your worker Q1. Listened to you & made you feel like she really understood your family? Q2. Treated the people in your family as unique people? Q3. Done what she said she would do? Q4. Noticed what s working well in your family regarding the care and safety of your child(ren)? Q5. Told you what s working well in your family regarding the care and safety of your child(ren)? Q6. Been direct with your family about what CAS is concerned about? Q7. Approached your family with a sense of humility clearly explain their opinion to you but they accept that their option could be wrong? Q8. Talked clearly about specific caring behaviours you need to demonstrate that the children will be safe? Q9. Helped you identify friends and extended family who could be part of a safety network to support you and your child(ren)? Q10. Explained how safe they thought your child(ren) were in a way that everyone involved, including the child(ren), could understand? Q11. Actively involved your family and your safety network to address the worries about your child(ren)? Q12. Made sure your child(ren) understood what was happening? Q13. Made sure the voices of your child(ren) were heard? Q14. Involved you in the decisions about how to build safety for your child? Q15. Asked questions about what was working well, what is not working well, and what needed to happen to help everyone understands the situation? Q16. Asked how safe you thought your child(ren) were? Q17. Helped your family develop an achievable safety plan? Q18. Made you feel like you had choices about how to solve the concerns? Q19. Checked in with your family about whether the safety plan was working?

25 Signs of Safety (SOS) 8/14/2015 As evident from the Standard Deviations (SD), there was a large variability in the responses to some of the rating items; this is why a median measure is provided in addition to the mean. LOOKING AT THE MEDIANS, WITH THE EXCEPTION OF Q17, ALL THE RATING ITEMS ARE AT OR ABOVE 8 AND FIVE OF THE ITEMS ARE AT THE MAX SCORE OF 10. For Q17 ( helped your family develop an achievable safety plan? ) with the lowest median score of 6, there was a jump between lower and higher ratings by parents, perhaps highlighting the complexities in the lives of child-welfare involved clients. Readers should also pay attention to the number of respondents for each question (N) as for a few the response rate was quite low (Q8, Q9, Q19). Clients who declined to respond to these questions mostly felt that the questions were not applicable to their particular situation while some of the respondents didn t like what the question was asking; the latter was particularly the case with Q9 ( Helped you identify friends and extended family who could be part of a safety network to support you and your child(ren)? ) Qualitative Findings What else did the worker do to help your family? Seven clients spoke of the workers personal qualities, attitude, and professional capabilities. Table 6 provides a summary of the responses by theme. Six of the interviewed clients spoke to how their most recent CAS experience compared to previous experiences with different workers. HALF OF THE CLIENTS (N=3) INDICATED THAT THEIR MOST RECENT EXPERIENCE WAS BETTER THAN BEFORE DUE TO A SUPERIOR WORKER. TABLE 6. CLIENT FEEDBACK REGARDING CASEWORKERS # Engagedwith child 3 Non-threatening 3 Generally supportive and helpful 2 Nice/kind/caring worker 2 Connected to resources 2 Ensured safety & stability 1 Good assessment of family 1 Advocated 1 The other half of the clients indicated that they had positive experiences with their most recent CAS worker, which was similar to previous experiences they had with Toronto CAS. Several direct client quotes have been included below: [Worker] very clearly communicated with my kids, why [worker] was there, what [worker] was hoping to do and that [worker] was there for them. My kids said this was the first time they ever felt that the worker was listening to them. Not sure if you hiring standards have changed, or you have better training now, but my current worker has been so much better. [Worker] was able to assess the situation and see everything was fine. Workers in the past weren t able to assess the situation, couldn t figure out what s wrong and what s right. [Most recent worker] connected with all my resources, talked to everyone being able to do that can give you better assessment of situation. Kind, nurturing and caring people [workers], care about what they are doing, not authority figures, genuine in concerns weren t here with a personal agenda, just to make sure kids were safe and that everyone were mentally and emotionally stable they did a good job. [Worker] spoke to landlord to help get locks changed for free No difference from previous involvement both times I felt supported. She [recent worker] just came in and explained why she was there and asked questions on computer. It was a survey she was reading of, it wasn t personal. She wasn t making me feel like a horrible person, she wasn t attacking me by asking me all these questions- she had to ask the questions because it was her job. the other workers made me feel like they are attacking you and your family, extremely uncomfortable situation to be. 24

26 Signs of Safety Evaluation 8/14/ File Reviews A total of 30 CAS Intake files were randomly selected for review from all files opened to Intake between April 1 and June 30 of 2014 and transferred to Ongoing Family Services. In order to ensure equal representation across all Intake teams, 10% of transfer files from each team were randomly selected. Please note that Intake teams belonging to the Child and Youth Advocacy Centre (CYAC) were excluded from selection due to their specialized work with the most severe of child abuse cases. The goal of the file reviews was threefold: To assess the use of SOS principles across Intake workers; 2. To collect information on family variables, which SOS aims to have a positive impact on; and 3. To compare family outcomes as a function of SOS use by their caseworker. Of the 30 files reviewed, nine belonged to SOS SUPER USERS, who are Intake workers who had additional training in SOS and were ahead in their SOS practice when compared to the agency as a whole. The remaining 21 files belonged to REGULAR INTAKE, Intake workers without such classification. The contents of file transfer recordings were reviewed by CWI researchers using a File Review Tool (see Appendix D) developed by CWI in collaboration with PLC. To meet the goals outlined above, the File Review Tool collected information in two areas: 1. FAMILY CHARACTERISTICS previous CAS openings; referral source; family court involvement; verification of protection allegations; Family Risk Ratings; number of children in the family; the children s residence; and any changes in the children s caregivers and/or residence from opening of CAS file to the point of file transfer. 2. USE OF SOS PRINCIPLES evidence of SOS practice, which was assessed using ten questions developed by CAST SOS trainers and Practice Leaders. The ten questions sought evidence of: (1) focus on engagement of client; (2) clear communication of protection concerns; (3) statement of family strengths or protective factors; (4) use of danger and harm statements; (5) an outlined plan; (6) safety goals; (7) involvement of family s support network; (8) use of EARS questioning; (9) use of SOS Tools; and (10) identification of what works well for the client. Each question received one point if there was clear evidence of use of the principle within the transfer recordings, a half point if there was unclear or inconclusive evidence and a score of zero if there was no evidence. The scored questions were added up to an overall SOS IMPLEMENTATION SCORE of 10 possible points. Results of the file review are summarized in Table 7 below. TABLE 7. FILE REVIEW COMPARISON BETWEEN REGULAR INTAKE WORKERS AND SOS SUPER USERS Regular Intake (n=21) SOS Super Users (n=9) Previous File openings 76% of files previously open 78% of files previous open M=2.75 (SD=2.02) M=3 (SD=2.24) Referral source Self = 9%; Other = 91% Self = 33%; Other =67% Family court involvement 14% 0% Verification of original allegations 62% verified 56% verified Verification of additional allegations 62% verified 67% verified Family Risk Assessment Moderate Risk = 52%; High Risk = 48% Moderate Risk = 78%; High Risk = 22% Number of children in the family M=1.90 (SD=.89) M=1.56 (SD=1.13) Children s residence 19% of children changed their residence 0% of children changed their residence SOS implementation score M=4.79 (SD=1.91)* M=8.06 (SD=.85)* There was a significant difference found in the SOS implementation scores (as evident from transfer file recordings) between Regular Intake and SOS Super Users, where p<.01, indicating that SOS Super Users implemented SOS principles in their transfer recordings (and, possibly, casework) much more often than Regular Intake. There was no significant difference found on family variables at point of file transfer to Ongoing Family Services between Regular Intake and SOS Super Users; this is not surprising given the small sample size and the short time period that intake workers have with families. Going forward, as SOS implementation across the agency continues, it will be possible to increase the size of the SOS Super User comparison group as well as follow up on the outcomes of these 30 families after receiving Ongoing Family Service. This would make it possible to shed more light on the difference SOS implementation makes in the lives of child welfare clients.

27 Signs of Safety (SOS) 8/14/ Summary of Year I Implementation Signs of Safety (SOS) has been implemented as the clinical framework to child protection casework at the Children s Aid Society of Toronto (CAST) in March This evaluation report aimed to provide an overview of Year I SOS implementation at CAST by collecting input from multiple stakeholders, specifically front-line workers, SOS Practice Leadership Committee, Intake supervisors, and clients. Even at this early phase of the implementation journey, there is preliminary evidence to the positive impact that SOS has been having on child protection work and child welfare involved clients. This section provides a summary of the main findings discussed throughout this evaluation report in relevance to the anticipated outcomes of SOS implementation at CAST. 4.1 Outcome 1: Protection and non-protection staff trained in introductory SOS In order to adopt the SOS framework, CAST s child protection workers and protection supervisors (N~350) received two-part SOS training during Spring and Fall of In 2014, training and skill development continued for protection staff and management while non-protection service staff was introduced to SOS in Fall and Winter of Only 2013 training evaluation results are presented at this time. A total of 40% of 354 SOS Part I trainees and 34% of 355 SOS Part II trainees completed the evaluation survey. A majority of respondents (over 76%) rated all satisfaction questions positively. SOS Part I Spring trainees gave lower ratings than Fall trainees while SOS Part II Spring trainees gave higher ratings than Fall trainees. Moreover, over half of all SOS Part I trainees (66%) and nearly three quarters of SOS Part II trainees (73%) indicated that between 60% and 100% of the curriculum was new knowledge for them. However, respondents who provided written feedback (35%) often noted that the content was repetitive and there was not enough practice-based teaching that included applicable examples and use of SOS tools. 4.2 Outcome 2: Advance staff skills in SOS assessments, safety planning and family engagement Quantitative and qualitative data from multiple stakeholders indicate that although the skills of multiple workers did advance, a variable level of SOS proficiency and application exists among protection staff and management. Based on the supervisors evaluation of their workers on the SUPERVISOR SOS FIDELITY RATING CHECKLIST (n=30), most workers (over 80%) are highly proficient in many skills, including humility, transparency, working with strengths, focusing on and enhancing safety, and engaging with various family members while working collaboratively. Qualitative comments from supervisors indicated that SOS brought structure into the casework and helped workers engage clients, conduct thorough and balanced assessments, build collaborative and meaningful safety plans, and reflect on their own practice. However, at least half of the workers or more were not sufficiently proficient in multiple SOS-specific skills, in particular the use of Mapping and Words and Pictures. To shed further light on these results, inferential statistical analyses was performed and found a significant difference between the ratings of workers with less advanced and more advanced SOS knowledge and skill: (1) A t-test comparison found that workers with more advanced knowledge in SOS were consistently rated higher on all checklist items; (2) the number of years of SOS practice of workers had a significant positive correlation to the majority of rating items, indicating that as the former increased so did the frequency with which workers practiced the skills assessed by their supervisors; (3) The number of days workers spent in training also had a significant positive correlation to their ratings by supervisors on all 35 items. Based on FOCUS GROUPS conducted with front-line workers (n=5), SOS Practice Leads (n=9), and Intake supervisors (n=12), most participants agreed that SOS approach has useful tools that bring structure to child welfare work, lead to more collaboration and partnership with families, and improve the practice of front-line workers by enhancing risk assessments and safety planning. With that being said, it was also recognized by focus groups participants that there is an existing gap, where some staff at front line and management levels are behind in their level of SOS knowledge and use; some of the challenges mentioned were lack of buy-in, insufficient time and opportunity given to learn SOS, split agency priorities, and inadequate integration of various models and initiative on the agency s agenda. 26

28 Signs of Safety Evaluation 8/14/2015 According to a FILE REVIEW comparing SOS Super Users to Regular Intake, there was a significance difference in the level of implementation of key SOS principles as evident in the recordings of Intake files transferred to Ongoing Family Services, where SOS Super Users appeared to implement SOS much more often. These results combined with focus group and Supervisor Fidelity Rating Checklist results imply that additional training and practice may bring Regular Intake up to a more advanced level of practice. 4.3 Outcome 3: Improved family satisfaction with CAS services Client interviews were conducted using the PARENT FEEDBACK CHECKLIST (n=9) where clients, having received services from a protection worker practicing the SOS framework, rated their worker on 19 items pertaining to worker s skill and the client s satisfaction. As evident from the interviews, many clients were highly satisfied with their worker, most frequently with the worker s ability to do what was promised, engage with children and ensure their voice was heard, approach the family with humility, and conduct a balanced assessment that includes strengths, worries, and next steps. Three of the interviewed clients indicated that their most recent CAST worker was superior to their previous CAST workers, implying that SOS implementation made a difference in practice with clients. Another three interviewed clients indicated that they were pleased with their most recent CAS worker as well as with their former CAST worker, highlighting the high level of skill CAST workers have. It is important to note that there was high variability in the responses, in particular on an item assessing the worker s ability to help the family develop an achievable safety plan; this could attest to the complexities in the lives of childwelfare involved clients and the possibility that the use of a checklist is insufficient to thoroughly assess clients experience, especially with such a small sample size. Nevertheless, this very preliminary evidence suggests that the use of SOS contributed to improved family experience and satisfaction with CAS services. 4.4 Outcome 4: Improved worker job satisfaction According to FOCUS GROUPS participants, which included front-line workers, practice leads, and Intake supervisors, implementing SOS led to improved experience for front-line staff who feel more job satisfaction due to their improved practice, better partnership with clients, and the beneficial effects these have had on the safety and functioning of children and their families. Qualitative comments from supervisors on the SOS FIDELITY RATING CHECKLIST support this finding by stating that through the use of SOS Framework, workers feel more proud of their work. Despite their satisfaction, workers also mentioned existing challenges, such as lack of support from management and peers in their use of SOS as well as the additional demand on their time and workload that learning and practicing this new framework requires. 4.5 Conclusion This report demonstrates preliminary evidence that the anticipated outcomes of Year I SOS implementation were achieved but only for pockets of front line workers and protection supervisors. Therefore, the next step in SOS implementation should focus on increasing the knowledge and skill of staff across the agency who do not have the same proficiency as SOS Super Users or Practice Leads. As for child welfare clients, their increased satisfaction with CAST services is an important finding, despite the small sample size. As a next step, it is important to investigate beyond client satisfaction and compare longitudinal outcomes, of which the most important are related to child safety, of clients who received SOS-informed service versus those who did not. 27

29 Signs of Safety (SOS) 8/14/ Appendix A - SOS LOGIC MODEL TABLE 8. YEAR I SOS IMPLEMENTATION LOGIC MODEL Program Plan: Signs of Safety: YEAR 1 (2013/14) Inputs Activities Outputs Target Group Resources Activities, Tasks, Deliverables Client, Budget Strategies Group Program Staff TOTAL COST: SOS Clinical Administration * Complete case notes * Supervision * SOS Implementation Advisory Committee *SOS Practice Leader Committee *Liaison with Connected Families * Develop SOS Trainers # protection workers trained in introductory SOS # nonprotection & support staff trained in SOS # CAST Practice Leaders trained in advanced SOS #SOS Trained trainers #Teams fully using SOS Practice Enhancement Groups running SOS Curriculum Children youth and families involved with CAST through protection services Outcome Plan Outcome Statements Short-Term Mid-Term (Impact) STAFF TRAINED IN SOS (YEAR 1) Protection & Workers produce nonprotection improved assessments and use danger & harm workers have statements basic Workers produce knowledge & improved safety plans skills in SOS Supervisors model Supervisors approach during support supervision learning Long-Term (Impact) Increasing level of SOS practice correlate to reduced reopenings, reduced future risk of harm, & increased worker satisfaction ADVANCE STAFF SKILLS IN SOS ASSESSMENT AND FAMILY ENGAGEMENT (YEAR 1) Workers practice SOS routinely Workers treat families as partners Workers produce balanced assessments using danger & harm statements as well as family strengths and resources Workers are skilled at engaging clients and building productive relationships Workers competent SOS practice correlates with child safety, reduce future risk to children, and higher family engagement. IMPROVED FAMILY SERVICE SATISFACTION (YEAR 1) Workers Families engage with partner with worker and feel heard and parents, validated children and Families have a clear their support understanding of why CAS networks is involved and what needs to happen IMPROVED CHILD/YOUTH SAFETY (YEAR 2) Workers are The child s voice is using SOS meaningfully incorporated tools to bring into safety planning out the child s voice Families are Families know what to do included in to meet safety planning for Families have extended their their support networks children s safety. Families view CAS as a resource or support and are highly satisfied with CAS SOS service Improved safety plans correlate to reduced reopenings, reduced future risk of harm, & > worker satisfaction 28

30 Signs of Safety Evaluation 8/14/ Appendix B - Supervisor SOS Fidelity Rating Checklist 29

31 Signs of Safety (SOS) 8/14/

32 Signs of Safety Evaluation 8/14/

33 Signs of Safety (SOS) 8/14/

34 Signs of Safety Evaluation 8/14/ \

35 Signs of Safety (SOS) 8/14/

36 Signs of Safety Evaluation 8/14/

37 Signs of Safety (SOS) 8/14/ Appendix C - Parent Feedback Checklist 36