Central LHIN Phase 2 Report. Andrew Szeto & Arla Hamer October

Transcription

1 Central LHIN Phase 2 Report Andrew Szeto & Arla Hamer October

2 1 OPENING MINDS: CHANGING HOW WE SEE MENTAL ILLNESS As part of its 10-year mandate, The Mental Health Commission of Canada (MHCC) embarked on an antistigma initiative called Opening Minds (OM) to change the attitudes and behaviours of Canadians towards people with a mental illness. OM is the largest systematic effort undertaken in Canadian history to reduce the stigma and discrimination associated with mental illness. OM is taking a targeted approach, initially reaching out to healthcare providers, youth, the workforce, and media. OM s philosophy is not to reinvent the wheel, but rather to build on the strengths of existing programs from across the county. As a result, OM has actively sought out such programs, few of which have been scientifically evaluated for their effectiveness. Now partnering with over 80 organizations, OM is conducting evaluations of the programs to determine their success at reducing stigma. OM s goal is to replicate effective programs nationally. A key component of programs being evaluated is contact-based educational sessions, where target audiences hear personal stories from and interact with individuals who have recovered or are successfully managing their mental illness. The success of contact-based anti-stigma interventions has been generally supported throughout international studies as a promising practice to reduce stigma. Over time, OM will add other target groups. For more information, go to: 2 BACKGROUND 2.1 Phase I: Mental Illness and Addictions: Understanding the Impact of Stigma In the spring of 2009, Opening Minds issued a Request for Interest (RFI), seeking existing programs aimed at reducing stigma among its initial target groups of healthcare providers and youth. Mental Illness and Addictions: Understanding the Impact of Stigma created and implemented by the Central Local Health Integration Network (LHIN) in Ontario was chosen by OM as a partner and pilot site for evaluations. The evaluation of this program was positive. Pre post results showed that the program was effective at reducing stigma. As a promising practice in stigma reduction, Mental Illness and Addictions: Understanding the Impact of Stigma has been adapted and implemented in other healthcare settings in Canada.

3 2.2 Pilot Sites: Vaughan Community Health Centre (VCHC) and North York General Hospital (NYGH) The project first identified an acute and a primary care site. In order to choose the pilot sites, all primary and acute healthcare sites in the Central LHIN (which had participated in the 2010 project) were approached to ascertain their interest, availability and willingness to participate. Based on these criteria, the Anti-Stigma Task Group selected Vaughan Community Health Centre as the primary care pilot site and North York General Hospital as the acute care pilot site. Vaughan Community Health Centre is a not-for-profit, community-governed organization that provides clinical and social services to the residents of the City of Vaughan in Ontario. North York General (NYGH) is a leading community academic hospital in Toronto, Ontario, providing a wide range of acute care, ambulatory and long-term care services at three sites. At this site, only emergency and mental health staff received Understanding Stigma Phase II program. 2.3 Understanding Stigma Phase II Although positively evaluated and successfully implemented elsewhere, the evaluation results also indicated that some of the positive changes gained by participants were not maintained at a three-month follow-up. One of the recommendations from the evaluation report was to implement additional booster sessions after the initial training to reduce or stop this regression back to baseline levels over time. Understanding Stigma Phase II is the follow-up project to the original Central LHIN project. Phase II incorporates the learning and strengths from the original program with additional booster sessions to create a more refined and potentially more effective program. In order to facilitate success, a site and department specific approach was implemented. At Vaughan Community Health Centre (VCHC), since staff had identified a lack of confidence and a skill deficit when working with individuals with mental health and addiction issues, it was decided to use the Walk a Mile in My Shoes simulation. This program provides real life examples of mental illness and addiction issues that present to healthcare providers. Specific skills were modeled by nurses and knowledge is developed through participatory exercises provided. Handouts related to illustrated mental health and addiction issues were also available. The project appreciates the sharing of this resource by Ontario Shores for Mental Health Sciences and the University of Ontario Institute of Technology for follow-up session use. At North York General Hospital (NYGH), two different follow-up strategies were developed. In the Emergency Department, where staff also identified confidence and skill deficits in working with individuals with mental health and addiction issues, the Walk a Mile in My Shoes simulation was selected. The program was integrated into the internal hospital intranet as an online booster session, allowing staff to access skill based learning in a way and at a time that met their learning needs. Since staff in the Mental Health Department were knowledgeable about the clinical skills needed to work with individuals with mental health and addiction issues, it was decided to use an experiential

4 model with this staff group. Realistic scenarios were developed and discussed in small groups. This approach is consistent with research that highlights the benefits of tailoring the approach and conveying anti-stigma training in smaller sized groups (1). 3 EVALUATION METHODS 3.1 Study Design and Procedures In general, the study design was a pre and post-test design with multiple follow-ups. At VCHC, questionnaires were given out just before the workshop, right after the workshop, right after the booster session, about one month after the booster session 1, three months following the booster session, and six months following the booster session. The procedures at NYGH were similar as at VCHC with the exception that no questionnaires were given out after the online booster session but questionnaires were given out after the group-based booster session. At all timepoints, the questionnaire package contained a question that asked participants to list three words that come to mind when they see the term mental illness (the analyses of this data is not included in this report), the 12-item version of the Opening Minds Scale for Healthcare Providers (2), and a short demographic questionnaire. At the three- and six-month follow-ups, several questions soliciting feedback about the workshops were also included. For the first four timepoints, participants completed paper questionnaires, with the three- and six-month follow-ups being completed online. At the pre-workshop timepoint, participants read a letter of information and completed a consent form before they completed the questionnaire. In this questionnaire, they were also asked to create a unique identifier consisting of their mother s initials and the year of high school graduation. This identifier was requested at the beginning of subsequent timepoints and used to link the participants responses over time. For the online follow-ups, participants were ed a link to the online questionnaire, which they could follow to complete. 3.2 Opening Minds Scale for Healthcare Providers (OMS-HC) The 12-item version of the OMS-HC was used assesses healthcare providers stigmatizing attitudes towards people with a mental illness. The questionnaire asks participants the extent to which they agree or disagree with various statements, with each statement rated on a 5-point scale from strongly agree, to strongly disagree. Total scale scores for the OMS-HC were created by summing across all 12 items of the measure. This results in a total score that can range from 12 to 60, with lower scores indicating less stigmatizing attitudes. The scale reliability for the OMS-HC was determined by calculating Cronbach s alpha at pre, post, and follow-ups. For VCHC, the OMS-HC had poor internal consistency at all timepoints (Cronbach s alpha for 1 This instance of the questionnaire was given due to the low rate of matching by the questionnaire given right after the booster session to pre- and post-test questionnaires

5 pre =.69, post =.42, booster 1 =.62, booster 2 =.65, three-month follow-up =.55, and follow-up =.59). In contrast, the OMS-HC had good internal consistency at all timepoints for NYGH (Cronbach s alpha for pre =.79, post =.79, booster 1 =.78, three-month follow-up =.74, and follow-up =.82). 3.3 Evaluation of Initiative Four workshop impact and evaluation questions were asked at the three- and six-month follow-ups. The first two questions asked participants to rate the extent they agree or disagree with the statements My participation in the Understanding Stigma Training program has helped me be more aware of my attitudes towards individuals with mental health and addiction issues and As a result of participating in the Understanding Stigma Training program, I have made a change in how I work with individuals with mental health and addictions issues. The third question asked participants to select from a list of 6 resource materials or program components (anti-stigma DVD, presentations by individuals with mental health problems, posters with key message, pen with key message, resource manual, and follow-up sessions) that they felt had that greatest impact. Lastly, participants were asked to comment if they would change anything about the training workshops or follow-up sessions to make them better. 3.4 Demographics A full demographic questionnaire was included at the pre and six-month timepoints. This questionnaire asked participants for their gender, age, professional status and years of practice, four questions about prior experience with mental illness (i.e., treated for a mental illness, interact with someone with mental illness, have family or friend with a mental illness, treated a person with mental illness), if they would like more training with mental illnesses, and any additional comments. At all other timepoints (i.e., post, booster, and three-month), participants were asked for their gender and age. Finally, participants were asked whether they had attended the initial workshop and/or the booster sessions at the follow-up timepoints (i.e., booster, three-month, and six-month). 4 VCHC RESULTS 4.1 Participation in Surveys The number of participants completing the surveys varied across the 6 timepoints: 29 for pre-workshop, 24 at post-workshop, 23 at booster 1, 30 at booster 2, and 13 (with 1 participant excluded from some analyses because of missing OMS-HC items) at both the three- and six-month follow-ups. 4.2 Participant Demographics Although some demographic data was collected at all timepoints, only those from the pre-workshop questionnaires are reported here. The demographic information collected at all other timepoints was

6 used mainly to assist the matching of participants questionnaires over the timepoints. Table 1 displays the demographic characteristics of the respondents at pre-workshop. The sample consisted mainly of female respondents. As well, most respondents fell between 30 to 49 years of age. Table 1. Demographic Characteristics of Respondents from VCHC n (30) % Gender Female Male Did not specify Age group Did not specify Professional Status Nurse Physician Social Worker Administration Manager/Director Other Did not specify Years in Practice % 10.0% 6.7% 13.3% 36.7% 36.7% 10.0% 3.3% 3.3% 13.3% 10.0% 23.3% 3.3% 36.7% 10.0% Avg years 4.3 Participants Experiences with Mental Illness The figures below depict the participants experiences with mental illness. Figure 1 displays the distribution of responses to the question have you ever been treated for a mental illness? Most participants responded that they had not been treated for a mental illness, with only 10% reporting they had been treated. Figure 2 displays the distribution of responses to the question have you ever interacted with someone with a mental illness? 90% of participants reported they had interacted with someone with a mental illness. Figure 3 and Figure 4 displays the distribution of responses to the question do you know a family member or close friend who has a mental illness? and have you ever treated a person with a mental illness? respectively. Two-thirds of participants reported they knew either a family member or friend that had a mental illness. However, only half of the current sample had treated someone for a mental illness.

7 Figure 1. Have you ever been treated for a mental illness? 80.0% 10.0% 10.0% No Yes Prefer not to answer/did not respond Figure 2. Have you ever interacted with someone with a mental illness? 90.0% 7.0% 3.0% No Yes Prefer not to answer/did not respond Figure 3. Do you know a family member or close friend who has a mental illness?

8 Figure 4. Have you ever treated a person with a mental illness? 47.0% 50.0% 3.0% No Yes Prefer not to answer/did not respond 4.4 Opening Minds Scale for Healthcare Providers (OMS-HC)

9 4.4.1 OMS-HC: Total Scores Table 2 displays the average total scores for each timepoint. The pre-workshop average was the highest at 30.3 with the six-month follow-up being the lowest at All other averages scores fell consistently between 27.6 and Table 2. Average total scores for the OMS-HC for all timepoints Timepoint # of Participants Average Total Score Pre-Workshop Post-Workshop Booster Booster Month Follow-up Month Follow-up Table 3 displays the paired t-test results for the scores from pre- to post-workshop and pre- to booster 2. All other comparisons (i.e., pre and booster 1, pre and three-month follow-up, and pre and six-month follow-up) returned 5 or less matched pairs. Due to the small numbers of matched pairs, these results are not displayed. This table shows that OMS-HC scores at both post-workshop and booster 2 were significantly lower than the scores at pre-workshop, suggesting that the workshop was effective at reducing stigma directly after the interventions and approximately 2 months after the intervention. Table 3. Paired t-test results for pre- to post-workshop and pre-workshop to booster 2 Timepoint # of Matched Pairs Pre-Workshop Average Total Score Average Total Score at that Timepoint Percentage Decrease Statistical Significance Post-Workshop % t = 3.87 (p =.001) Booster % t = 2.15 (p =.047)

10 Figure 5 displays the percentage of participants whose OMS-HC scores improved, worsened, or showed no change from pre- to post-workshop and from pre-workshop to booster 2. Consistent with previous Opening Minds healthcare provider evaluations, 71% of the participants had scores that improved from pre- to post-workshop, with only 21% having scores that worsened. The results did differ somewhat when examining score changes by participant type from pre to booster 2, with 65% showing a score improvement and 35% showing a score deterioration over that time. Figure 5. Total score change from pre- to post-workshop and from pre-workshop to booster 2 Total Score Change Pre to Post Total Score Change Pre to Booster % 21.0% No change Improved Worsen 65.0% 35.0% 8.0% OMS-HC: Stigma Content Areas and Item Analyses The 12-item version of the OMS-HC can be grouped by similar content into two different content areas of stigma: general attitudes towards mental illnesses and attitudes towards disclosing about mental illnesses. Table 4 displays participants pre- and post-workshop responses on the 5-point rating scale collapsed into three categories (i.e., strongly disagree and disagree; neither agree nor disagree; and strongly agree and agree) on all items of the OMS-HC scores grouped by the stigma content area. This table also displays the change from pre- to post-workshop for three collapsed categories.

11 Table 4. Participant responses on the (collapsed) rating scale for pre- and post-workshop and percentage changes from pre to post for all items (grouped stigma content areas) Number Content Area Item / Pre Post Post - Pre Neither Agree nor Agree/ Agree / Neither Agree nor Agree/ Agree / Neither Agree nor Agree/ Agree 1 Attitudes I am more comfortable helping a person who has a physical illness than I am helping a person who has a mental illness Attitudes 5 Attitudes If a person with a mental illness complains of physical symptoms (e.g. nausea, back pain or headache), I would attribute this to their mental illness. Despite my professional beliefs, I have negative reactions towards people who have mental illness Attitudes There is little I can do to help people with mental illness Attitudes More than half of people with mental illness don t try hard enough to get better Attitudes Healthcare providers do not need to be advocates for people with mental illness Attitudes I struggle to feel compassion for a person with a mental illness If I were under treatment for a mental illness I would not disclose this to any of 2 Disclosure my colleagues. I would be more inclined to seek help for a mental illness if my treating healthcare 4 Disclosure provider was not associated with my workplace Disclosure I would see myself as weak if I had a mental illness and could not fix it myself Disclosure I would be reluctant to seek help if I had a mental illness * Disclosure If I had a mental illness, I would tell my friends *this item was reverse scored. Please note that strongly disagree and disagree equate to non-stigmatizing responses for all items in this table.

12 Generally, most items regardless of content area had a more positive response distribution (i.e., higher percentage of strongly disagree/disagree and/or lower percentage of strongly agree/agree) comparing post- to pre-workshop. There were, however, three items from the attitudes content area that had at least a 20% increase from pre- to post-workshop, while there were none for the disclosure items. Item number 5 had no increase in the responses in the strongly disagree/disagree category but there was an already high rate at pre-workshop (i.e., 91.7%). As well, there was also decrease of 4.2% of responses in the strongly agree/agree category. There were also 3 items with decreases in the strongly disagree/disagree category from pre- to post workshop (i.e., more stigmatizing). For two of these items (Items 2 and 10), this was offset by similar decreases in strongly agree/agree category. For Item 12, there was about a 5% shift from the strongly disagree/disagree category to the neither agree nor disagree category from pre- to post-workshop. Finally, all items, except one, had decreases or no change in the percentage change in the strongly disagree/disagree category from pre- to post-workshop. This exception was Item 11 indicating that participants agreed more to the statement that healthcare providers do not need to be advocates for people with mental illnesses after the workshop OMS-HC: Cumulative Percent of Non-Stigmatizing Responses (Threshold of Success) Another way to examine the results of the OMS-HC is to see how many participants reached a threshold of success on the measure; in other words, how many participants responded to a certain number of items on the OMS-HC in a non-stigmatizing way. To examine this threshold, each response on the OMS- HC was recoded to either non-stigmatizing (strongly disagree and disagree) or stigmatizing (neither agree or disagree, strongly agree, and agree) and the number of non-stigmatizing responses summed for each participant. A threshold of above 80%, or 10 out of 12 correct (i.e., non-stigmatizing responses), corresponding to an A grade for an educational intervention, was used as an indication of the threshold of success. Figure 6 displays the cumulative percentages of participants who had non-stigmatizing responses for each possible score out of 12 at pre and post. The percentage of participants that were beyond the threshold before the workshop was at about 8% of the sample, with this increasing more than 2 times to about 21% of the sample after the workshop. There was also a large gain for those scoring just below the threshold (or 9 of 12). Here, the percentage doubled from pre- (21%) to post-workshop (42%). The gains seen here are comparable, if not greater than, other Opening Minds healthcare provider evaluations.

13 Figure 6. Cumulative Percent of Non-Stigmatizing Responses on OMS-HC for Pre and Post 100% 90% 80% Threshold of Success 80% 70% 60% 50% 40% 30% 20% 10% Pre-test Post-test 0% all Evaluation of Initiative As indicated earlier, four questions were used to assess participants perceptions of Understand Stigma Phase II initiative, at both the three- and six-month follow-up questionnaires. The first two questions asked participants if the initiative helped them be more aware of their attitudes towards individuals with mental health and addiction issues and if the initiative changed how they work with individuals with mental health and addictions issues. Table 5 shows the results for these two questions at both follow-up timepoints. A majority of participants, approximately 67% or greater, agreed that the initiative has helped them be more aware of their attitudes and changed how they worked with someone with a mental illness. One participant disagreed to the statements. The average score also indicates that the initiative had a positive impact as the score fell between neither agree nor disagree and agree.

14 Table 5. Response distribution and mean for the two workshop rating scale evaluation questions for threeand six-month follow-ups Response Scale # of Responses (1) (2) Neither Agree nor Agree (4) (3) Agree (5) Average Score three-month follow-up six-month Followup helped me be more aware of my attitudes towards individuals with mental health and addiction issues I have made a change in how I work with individuals with mental health and addictions issues helped me be more aware of my attitudes towards individuals with mental health and addictions issues I have made a change in how I work with individuals with mental health and addictions issues (15.4%) (15.4%) (16.7%) 11 (84.6%) 11 (84.6%) 10 (83.3%) (8.3%) 3 (25.0%) 8 (66.7%) Respondents were also asked to indicate the project component that had the greatest impact for them (see Table 6). For both timepoints, two-thirds of the participants selected the presentation given by the person with lived experience of a mental illness had the greatest impact on them. Finally, participants were asked to comment on any changes that would improve the workshops or booster sessions. There was generally a low response rate to this question as only 2 comments were generated at the three-month follow-up and 1 at six-month. One respondent suggested more participation by those with lived experience would improve the training. Another participant suggested more time was needed to work through the DVD scenarios. The last comment indicated that sensitivity training for supervisors was needed.

15 Table 6. Project components that had the greatest impact for each respondent for three- and six-month Follow-ups Project Components with Greatest Impact # of Responses Anti-Stigma DVD Presentation with Individual with Lived Experience Posters with Pens with Key Key Message Message Resource Manuals Follow-up Sessions three-month Follow-up six-month Follow-up 12 1 (8.3%) 10 (83.3%) (8.3%) 11 2 (18.2%) 7 (63.6%) (9.1%) 1 (9.1%) 4.6 Summary of Results for VCHC The results from VCHC are encouraging. The pre- and post-workshop comparison showed a statistically significant decrease (i.e., improvement in scores) on the OMS-HC score. This was also the case for the preworkshop and booster 2 comparison. More importantly, there does not appear to be a regression in the gains back towards baseline levels overtime (see Table 2). Particularly, the OMS-HC score at three- and six-month follow-up timepoints appear to be similar to the post-workshop score. This finding would suggest that the booster sessions helped maintain the gains from the initial workshop. Despite these encouraging findings, no firm conclusions can be made at this point due to the low number of follow-up surveys completed and their lack of matching to previous timepoints. In addition, participants believed that the Understanding Stigma Phase II program had impacted them positively. They indicated that the program helped them be more aware of their stigmatizing attitudes, and more importantly, helped them change how they work with individuals with a mental illness (see Table 5). 5 NYGH RESULTS 5.1 Participation in Surveys Similar to VCHC, the number of participants completing the surveys at NYGH varied across the 5 timepoints: 323 for pre-workshop, 307 at post-workshop, 88 at the booster, 147 at the three-month follow-up, and 49 at the six-month follow-up.

16 5.2 Participant Demographics Again to maintain brevity, only those demographics from the pre-workshop questionnaires are reported here. Table 7 displays the demographic characteristics of the respondents at pre-workshop. The sample consisted mainly of female respondents. As well, participants age was evenly distributed across the first four age categories with each having around 20%. For NYGH, the largest occupation group attending the workshops were nurses, consisting of half of the sample. Table 7. Demographic characteristics of respondents from NYGH n (323) % Gender Female Male Did not specify Age group Did not specify Professional Status Nurse Physician Social Worker Administration Manager/Director Other Did not specify Years in Practice % 17.3% 0.9% 20.4% 23.8% 22.9% 23.2% 6.2% 0.6% 2.8% 50.2% 0.6% 6.8% 11.8% 10.8% 18.6% 1.2% Avg years 5.3 Participants Experiences with Mental Illness The figures below depict the NYGH participants four types of experiences with mental illness. Figure 7 shows that 80% of the pre-workshop sample has not been treated for a mental illness with 15% having been treated. A large majority have also interacted with someone with a mental illness (see Figure 8), know a close friend or family member with a mental illness (see Figure 9), and have treated someone with a mental illness (see Figure 10).

17 Figure 7. Have you ever been treated for a mental illness? 80.0% 15.0% 5.0% No Yes Prefer not to answer/did not respond Figure 8. Have you ever interacted with someone with a mental illness? 93.0% 3.0% 4.0% No Yes Prefer not to answer/did not respond

18 Figure 9. Do you know a family member or close friend who has a mental illness? 72.0% 22.0% 6.0% No Yes Prefer not to answer/did not respond Figure 10. Have you ever treated a person with a mental illness? 74.0% 21.0% 5.0% No Yes Prefer not to answer/did not respond

19 5.4 Opening Minds Scale for Healthcare Providers (OMS-HC) OMS-HC: Total Scores Table 8 displays the average total scores for each timepoint. The pre-workshop average was the highest at 28.4, with the booster average 2 being the lowest at All other averages scores fell between 25.5 and Table 8. Average total scores for the OMS-HC for all timepoints Timepoint # of Participants Average Total Score Pre-Workshop Post-Workshop Booster Month Follow-up Month Follow-up Table 9 displays paired t-tests between the pre-workshop scores and the four other timepoints. Comparing pre-workshop scores to post-workshop and booster scores revealed a 9% and 11% decrease, respectively, with both comparisons being statistically significant. Scores did not seem to regress back towards baseline levels at three- and six-month follow-up. However, the pre-workshop scores for the three- (i.e., 27.4) and six-month (i.e., 27.5) comparisons appear to be lower than the original preworkshop score with all 280 participants (i.e., 28.6). It is possible that these lower baseline scores contributed to the non-significant comparisons. 2 The booster assessment was only offered to the participants who completed the small group booster session and not for those participants that completed the walk a mile in my shoes online booster session.

20 Table 9. Paired t-test results between pre-workshop and the other four timepoints Timepoint # of Matched Pairs Pre-Workshop Average Total Score Average Total Score at that Timepoint Percentage Decrease Statistical Significance Post-Workshop % t = (p <.001) Booster % t = 3.45 (p =.001) 3 Month Followup 6 Month Followup % No % No Table 10 displays the breakdown of OMS-HC scores for those who did and did not attend the initial training or the booster sessions at the three-month questionnaires. The results from this table suggest that those who attended the initial workshop had a lower score than those who did not attend (i.e., a quasi-control group). Similarly, those receiving a booster session had lower OMS-HC scores than their counterparts who did not attend the booster session. Additional analyses were also conducted to examine the effects of the two different boosters used at NYGH. For those who participated in the small group booster, there was a significant difference in OMS- HC scores from pre-workshop to right after the booster session (see Table 11). This difference appeared to remain at the three- and six-month follow-ups although the difference was only statistically significant for the six-month follow-up but marginally significant at the three-month follow-up. For those who participated in the online Walk a Mile in my Shoes booster session, results appear to be less positive (see Table 11). These participants did not complete the questionnaires directly after the booster sessions but those who did complete the three-month follow-up, there was no significant difference from their preworkshop scores. There appeared to be a large difference between the pre-workshop score and the sixmonth follow-up score. This was not a significant difference however. It is also interesting to note that this group who participated in the online booster session also did not have a significant pre- and postworkshop difference on their OMS-HC score. Given this non-significant finding and the generally small sample size, it is difficult to pinpoint the differential efficacy of the two different booster sessions.

21 Table 10. Comparing participants scores that did and did not attend the initial training or booster sessions at the three-month timepoint Yes No Timepoint # of Participants Average Total Score # of Participants Average Total Score Three-month Followup: did you attend initial training? Six-month Follow-up: did you attend booster training?

22 Table 11. Comparing the two booster sessions used at NYGH Small Group Booster Session Timepoint # of Matched Pairs Pre-Workshop Average Total Score Average Total Percentage Score at that Decrease Timepoint Statistical Significance Post-Workshop % t = 3.16 (p =.003) Booster % t = 3.45 (p =.001) three-month Follow-up six-month Follow-up % No % t = 3.69 (p =.01) Timepoint Walk a Mile in My Shoes Online Booster Session # of Matched Pairs Pre-Workshop Average Total Score Average Total Percentage Score at that Decrease Timepoint Statistical Significance Post-Workshop % No Booster three-month Follow-up six-month Follow-up % No % No Figure 11 displays the percentage of participants whose scores got better, got worse, or had no change from pre- to post-workshop, the booster, and the two follow-up timepoints. With the exception of the figure for pre to three-month follow-up, the distributions appear to be consistent with other Opening Minds evaluations, with approximately 60-70% of participants having scores that improved over time and approximately a quarter of participants having scores that worsened scores overtime. It is unclear why the pre to the three-month follow-up distribution was different from the others, especially with the pre to six-month follow-up distribution returning to a more typical distribution. This seems more anomalous given the similar average scores for the follow-up timepoints (see Table 9). Is it possible this specific set

23 of participants was not representative of the larger sample or contained an uneven proportion of a specific demographic (e.g., occupation). Another method for examining program impact is to determine the minimum detectable change (MDC) statistic. If a participant s change in score over two timepoints is greater than the MDC, this participant s change reflects a true change in attitude rather than a change due to statistical error or chance. The calculated MDC for the OMS-HC scale is This suggests that a score increase or decrease of 6.5 points or more reflects a true change in attitude for the current participants. As highlighted in Table 12, when the MDC statistic was applied to participants score changes from pre to post program, pre to booster, and pre to the 3 month follow-up, the number of participants who actually became more stigmatizing was minimal (2-6%). When the MDC was applied to the scores that improved, the percent of participants that truly became less stigmatizing ranged from 12% to 20%. 4 Figure 11. Total score change from pre-workshop to post-workshop, three-month follow-up, and sixmonth follow-up Total Score Change Pre to Post 2.0% 42.0% Improved Total Score Change Pre to Boost 11.0% 56.0% No Change Worsen 63.0% 26.0% Total Score Change Pre to 3 Mon Total Score Change Pre to 6 Mon 10.0% Improved 25.0% 70.0% 20.0% No Change Worsen 70.0% 27.5% 3 The MDC for the OMS-HC scale was calculated based on a standard error of measurement (SEM) of 2.80 [from test-retest results on the full scale, see(2)] and a z score of 1.65 (90% confidence level). The formula for calculating this statistic is as follows: MDC=SEM* 2*z score associated with confidence level of interest. For the current analysis, the MDC should be considered approximate, as OM does not yet have test-retest results for the 12-item version of the scale. 4 Due to the small number of matched participants for the Vaughan site, the MDC was not used in the analyses.

24 Table 12. Participants who became more or less stigmatizing based on the Minimal Detectable Change statistic Percent that became more stigmatizing Percent that became less stigmatizing Pre to Post 2.1% 12.9% Pre to Booster 2.5% 20% Pre to three-month Follow-up 6.0% 12.0% OMS-HC: Stigma Content Areas and Item analyses Table 13 displays the percentage of participants responses across the collapsed categories on the items of the OMS-HC for pre- and post-workshop. This table also displays the change from pre- to post-workshop for three collapsed. All 12 items had positive percentage changes from pre- to post-workshop in the strongly disagree/disagree category, equating to less stigmatizing responses across the two timepoints. Concomitantly, there were less participant responses in the strongly agree/agree category at postworkshop than pre-workshop for all but two items. There were also four items that had a 10% increase in the strongly disagree/disagree category across the two timepoints. 4 Due to the small number of matched participants for the Vaughan site, the MDC was not used in the analyses.

25 Table 13. Participant responses on the (collapsed) rating scale for pre- and post-workshop and percentage changes from pre to post for all items (grouped stigma content areas) Number Content Area Item / Pre Post Post - Pre Neither Agree nor Agree/ Agree / Neither Agree nor Agree/ Agree / Neither Agree nor Agree/ Agree 1 Attitudes 3 Attitudes 5 Attitudes I am more comfortable helping a person who has a physical illness than I am helping a person who has a mental illness. If a person with a mental illness complains of physical symptoms (e.g. nausea, back pain or headache), I would attribute this to their mental illness. Despite my professional beliefs, I have negative reactions towards people who have mental illness Attitudes There is little I can do to help people with mental illness Attitudes More than half of people with mental illness don t try hard enough to get better Attitudes Healthcare providers do not need to be advocates for people with mental illness Attitudes I struggle to feel compassion for a person with a mental illness If I were under treatment for a mental illness I would not disclose this to any of 2 Disclosure my colleagues. I would be more inclined to seek help for a mental illness if my treating healthcare 4 Disclosure provider was not associated with my workplace Disclosure I would see myself as weak if I had a mental illness and could not fix it myself Disclosure I would be reluctant to seek help if I had a mental illness * Disclosure If I had a mental illness, I would tell my friends *this item was reverse scored. Please note that strongly disagree and disagree equate to non-stigmatizing responses for all items in this table.

26 Generally, most items regardless of content area had a more positive response distribution (i.e., higher percentage of strongly disagree/disagree and/or lower percentage of strongly agree/agree) comparing postto pre-workshop. There were, however, three items from the attitudes content area that had a 20% increase from pre- to post-workshop, while there were none for the disclosure items. Item number 5 had no increase in the responses in the strongly disagree/disagree category but there was an already high rate at pre-workshop (i.e., 91.7%). As well, there was also decrease of 4.2% of responses in the strongly agree/agree category. There were also 3 items with decreases in the strongly disagree/disagree category from pre- to post workshop (i.e., more stigmatizing). For two of these items (Items 2 and 10), this was offset by similar decreases in strongly agree/agree category. For Item 12, there was about a 5% shift from the strongly disagree/disagree category to the neither agree nor disagree category from pre- to post-workshop. Finally, all items, except one, had decreases or no change in the percentage change in the strongly disagree/disagree category from pre- to post-workshop. This exception was Item 11 indicating that participants agreed more to the statement that healthcare providers do not need to be advocates for people with mental illnesses after the workshop OMS-HC: Cumulative Percent of Non-Stigmatizing Responses (Threshold of Success) Figure 12 displays the cumulative percentages of participants who had non-stigmatizing responses for each possible score out of 12 at pre and post. The largest percentage change came at just below the threshold (i.e., 9). At this level, there was about an 18% increase from pre- to post-workshop. It is also important to note that more than 50% of the participants scored at this level or higher at post-workshop. Although the gains at post-workshop are not as great for VCHC, there are generally higher percentages at both pre- and post-workshop across the various response levels.

27 Figure 12. Cumulative Percent of Non-Stigmatizing Responses on OMS-HC for Pre and Post 100% 90% 80% Threshold of Success 80% 70% 60% 50% 40% 30% 20% 10% Pre-test Post-test 0% all Evaluation of Initiative The same evaluation questions were also completed by NYGH participants. The results of these questions are displayed in Table 14. Responses were generally positive, although not as positive as for VCHC respondents. At both timepoints, 63% of participants agreed that the initiative has helped them be more aware of their attitudes. Only about 40% of participants, however, agreed that the initiative has changed how they worked with people with mental health problems. The average scores were slightly lower than the average scores for VCHC.

28 Table 14. Response distribution and mean for the two workshop rating scale evaluation questions for threeand six-month follow-ups Response Scale # of Responses (1) (2) Neither Agree nor Agree (4) (3) Agree (5) Average Score three-month follow-up six-month Followup helped me be more aware of my attitudes towards individuals with mental health and addiction issues I have made a change in how I work with individuals with mental health and addictions issues helped me be more aware of my attitudes towards individuals with mental health and addictions issues I have made a change in how I work with individuals with mental health and addictions issues (8.9%) 10 (7.4%) (6.8%) 15 (11.3%) 28 (20.7%) 54 (40.6%) 44 2 (4.5%) 3 (6.8%) 8 (18.2%) 45 4 (8.9%) 8 (17.8%) 15 (33.3%) 67 (49.6%) 48 (36.1%) 29 (65.9%) 17 (37.8%) 18 (13.3%) (5.3%) (4.5%) (2.2%) 3.1 Participants at NYGH, similar to those at VCHC, believed the presentation and the DVD were the program components with the greatest impact (see Table 15). Participants were also solicited at both follow-up timepoints for their comments on how to improve the program. There were 37 comments at the three-month follow-up and 8 comments at the six-month follow-up. Generally, comments were positive (e.g., enjoyed the speakers) with the most prevalent being the need for more training of this type, the continuation of this program, or offering this program to all staff at the hospital (12 comment, e.g., Please expand this program to more and more audiences ). Some more constructive comments indicated that more time should be allotted to the workshops (4 comments, e.g., Not enough time for discussion ), the DVD portrayed healthcare providers in a negative light (4 comments, e.g., I found the video to be more accusing of people which in turn I think made people feel defensive ), and the speaker with lived experience was not typical of those with a mental illness (4 comments, e.g., the speaker with mental illness was not the best example of the average person with mental illness ).

29 Table 15. Project components that had the greatest impact for each respondent for three- and six-month Follow-ups Project Components with Greatest Impact # of Responses Anti-Stigma DVD Presentation with Individual with Lived Experience Posters with Key Message Pens with Key Message Resource Manuals Follow-up Sessions three-month Followup (22.3%) 79 (65.3%) 2 (1.7%) 0 3 (2.5%) 10 (8.3%) six-month Follow-up 38 8 (21.1%) 26 (68.4%) 0 1 (2.6%) 0 3 (7.9%) 5.6 Summary of Results for NYGH In general, the results from NYGH reflect those found at VCHC. There was a statistically significant positive change in participants OMS-HC scores from pre- to post-workshop. Table 8 also indicates that scores stay consistent post-workshop. There were, however, no significant differences between pre- and the three- and six-month follow-ups (see Table 9). This lack of significant findings could be due to the lower baseline scores (i.e., pre-workshop) of the sub-samples that completed both the pre-workshop and the follow-up questionnaires than those who did not. The small group booster session also appears to have been effective at maintaining the gains of the program. For example, there was approximately 10-11% decrease on OMS-HC scores at the booster, and three- and six-month follow-ups (see Table 11). Moreover, the initial gains measured for this booster session group from pre- to post-workshop augmented to about 11% after this booster session. The Walk a Mile in My Shoes online booster session appeared not to fair as well. There was no significant difference from pre-workshop to the three-month follow-up (see Table 11). However, this group also did not have pre- and post-workshop differences either. These results are quite intriguing and do point to one booster session over the other; however, they need further corroboration due to the lack of experimental controls groups, the low baseline scores for those who attended the small group booster session, participant attrition, and lack of matching over time. Participants perceptions of the program s impact was positive but more tempered compared to VCHC. Over 50% of participants believed the program helped them be more aware of their stigmatizing attitudes. Although still substantial, only about 40% agreed that the program helped them change their behaviour in working with people with a mental illness. Finally, participants comments were quite positive regarding the speakers with lived experience and believe this aspect of the program had the greatest impact (see Table 15).

30 6 SUMMARY AND CONCLUSIONS As with its predecessor, Understanding Stigma Phase II was successful at improving healthcare provider attitudes and behavioural intentions towards mental illness and persons with a mental illness). Despite some methodological challenges (e.g., lack of matching of individual participants at different timepoints, no control groups) of collecting good data in a large operating hospital setting, the addition of booster sessions (in two of three types) also appears to have reduced, or even stopped, the regression of gains to baseline levels overtime as seen in many other Opening Minds evaluations. These results offer valuable insights for antistigma programmers in the healthcare field, and for other target groups alike. In addition to successes reflected in the evaluation data, success was also reflected in the implementation of long-term sustainability strategies at both sites. At VCHC, a new employee anti-stigma orientation training module and on-going staff in-service training have been incorporated into regular practices. These initiatives will ensure that an anti-stigma frame of reference endures. At NYGH, a plan to integrate anti-stigma training as a module in a proposed hospital wide Diversity Framework Training Initiative was developed and implemented. The diversity framework includes six modules (including stigma) which will become competencies measured on staff performance reviews. The implementation of the Diversity Framework Training Initiative will guarantee that both new and long-standing employees are familiar with anti-stigma concepts and that the impact and importance of these concepts are reinforced for all employees on a consistent basis. In addition, a Champion s Committee, formed in the development stage of the project will function on an on-going basis. The work of the committee will ensure that a focus on the importance of addressing stigma continues and that recommendations brought forward by staff are implemented. Central LHIN s Understanding Stigma Phase II program has led to changes at both the individual level and the organizational level. At both sites, elimination of stigma and sustainability of healthcare provider commitment to addressing this issue continues to be reflected in the dedication of leadership and the project posters which serve as a reminder to all. Without this commitment from the leadership, programs and initiatives, although well-intentioned, tend to wane as time progresses. Understanding Stigma Phase II can serve as a leading example for how to implement and sustain effective anti-stigma programming in healthcare settings.

31 References 1) Ucok, A. (2007). "Other people stigmatize...but, what about us? Attitudes of mental health professionals towards patients with schizophrenia." Noropsikiyatri Arsivi/Archives of Neuropsychiatry 44(3): ) Kassam A, Papish A, Modgill G, Patten S (2012). The development and psychometric properties of a new scale to measure mental illness related stigma by health care providers: The Opening Minds Scale for Health Care Providers (OMS-HC). BMC Psychiatry 12 (62).