Ottawa School-based Substance Abuse Program Evaluation Report For the year 2015/16 October 2016
Acknowledgements Participating school boards: Conseil des écoles catholiques du Centre-Est Conseil des écoles publiques de l Ést de l Ontario Ottawa Carleton District School Board Ottawa Catholic District School Board Service providers: Maison Fraternité Rideauwood Addiction and Family Services Program funding partners: Champlain Local Health Integration Network Ottawa Public Health United Way Ottawa s project s.t.e.p. (including funding from the Ottawa Senators Foundation) Conseil des écoles catholiques du Centre-Est Conseil des écoles publiques de l Ést de l Ontario Ottawa Carleton District School Board Ottawa Catholic District School Board ii
Table of Contents 1. Executive Summary 1 2. Background 3 3. Methodology a. Full Collaboration 4 b. Evaluation Plan 4 c. Sample Size 5 d. Statistical Analysis 5 4. Results a. Schools Served 6 b. Students and Families Served 6 c. Client Profile 6 d. Student Engagement 8 e. Student Health and Wellbeing 9 f. Drug and Alcohol Use 11 g. Student Academic Outcomes 12 h. School staff testimonials 13 5. Discussion 14 6. Conclusion 15 iii
1. Executive Summary The Ottawa school-based substance abuse counselling program is a cooperative initiative providing counselling services for all English and French speaking publically funded high school students in the Ottawa region. This unique partnership among the four Ottawa school boards, Maison Fraternité, Rideauwood Addiction and Family Services, Substance Abuse and Youth in School (SAYS) Coalition, the Ottawa Network for Education (ONFE), Ottawa Public Health and the United Way/ Centraide Ottawa, is committed to ensuring that all high school students are provided with easily accessible substance use treatment. Counsellors spend an average of 14 hours a week in each school, although this time varies and is planned for each school according to need. Importantly, this program is able to offer support to students or parents without the delay of a waitlist. Clients of the school based program in 2015/16 were most often aged between 15 to 17 years of age but ranged from 13 to 19 years old. Sixty percent of clients identified as male and 40% as female. Among the clients reporting substance use at some time during the previous year upon entry into the program, the most commonly reported substances of use or problem behaviour included: cannabis (88%), alcohol (82%), nicotine (56%), problematic video game or computer use (38%), ecstasy (37%), hallucinogens (26%), cocaine (22%), and stimulants (21%). Substance use frequently occurs concurrently with other mental health needs. Upon entry into the school-based program, for which 47% of clients were flagged for substance abuse disorder, the percentage of clients flagged for concurrent mental health needs was very high: 60% of clients were flagged for externalizing disorder; 54% for internalizing disorder; 28% for crime or violence; 26% for an eating disorder; 25% for psychosis; and, 21% for post-traumatic stress disorder. In the eight years since this cooperative initiative was begun, the effect of the counselling program on student behaviour, wellness and mental health outcomes has shown very promising results. As with past years, comparisons of 2015/16 pretreatment need with follow-up (at least 3 months after initial assessment) show marked improvement: 31% of clients were no longer flagged for substance disorder at follow-up 22% of clients initially flagged for crime and violence were no longer flagged by comparison 19% of clients were no longer flagged for an eating disorder 30% were no longer flagged for externalizing disorder 1
30% were no longer flagged for internalizing disorder Improved academic achievement is a goal of the school-based counselling initiative but significant results have not typically been obtained in past years. This year a significant improvement in average grade was found for clients of the school based program but not for credit accumulation. Additionally, clients showed good rates of maintained school attendance. It is expected that an effect on academic achievement is most likely to been found in the time following treatment than in the year of treatment, it is therefore recommended that a longitudinal study be conducted to compare the effect of treatment over time. Lastly, sample size continues to be a concern and it is recommended that a revised process be implemented to try to increase the available sample size for evaluation. In summary, the Ottawa school-based substance use counselling initiative appears to be meeting objectives of: providing Ottawa area students with information, education and treatment; decreasing substance use and some concurrent mental health needs; and, improving overall wellness. 2
2. Background The school-based substance use counselling program was developed to provide easily accessible support to Ottawa area high school students struggling with substance use or the use of a family member or friend. By being present on-site in every Ottawa public highschool, services are available to a large number of Ottawa youth in their school. The program is a multi-sector community partnership to promote early prevention, education and treatment services in schools. All four Ottawa area school boards, Conseil des écoles catholiques du Centre-Est, Conseil des écoles publiques de l Ést de l Ontario, Ottawa Carleton District School Board, Ottawa Catholic District School Board, and two service providers, Maison Fraternité and Rideauwood, work in collaboration to offer services in every Ottawa area high school. Program oversight is provided by the Substance Abuse and Youth in School Coalition (SAYS), a coalition with representation from all four Ottawa area school boards, local youth-serving addiction agencies, Ottawa Public Health, United Way/ Centraide Ottawa, enforcement and allied professionals, and facilitated by the Ottawa Network for Education (ONFE). For more information on the SAYS coalition please see www.onfe-rope.ca. ONFE also provides administrative support to the school-based program. In 2008, funding from the four Ottawa school boards (Conseil des écoles catholiques du Centre-Est, Conseil des écoles publiques de l Ést de l Ontario, Ottawa Carleton District School Board, Ottawa Catholic District School Board), Ottawa Public Health, the Champlain LHIN (Local Health Integration Network) and United Way/ Centraide Ottawa s project s.t.e.p. committed a total of one million dollars annually for school-based counselling, education and prevention (funding is proportional to the number of eligible schools within each school board). It was agreed that the outcomes used to measure the effect of the program would focus broadly on improved: health, wellbeing and academic achievement. In the years following, funding has increased and the reach of the counselling program now includes all Ottawa public high schools. As the need for services and number of schools increases, the program has been able to effectively meet these needs, in particular by extending prevention and education programs to a growing number of grade 7 and 8 students, parents and teachers. 3
3. Methodology a. Full Collaboration The annual program evaluation for the school based substance use counselling program in Ottawa was developed with the support of a grant from the Centre of Excellence for Child and Youth Mental Health in 2010 and funded by Health Canada (through project s.t.e.p.) and the Champlain LHIN. All partners and providers agreed to a common framework to facilitate the yearly evaluation of the school based program. Ethical approval for this research was obtained from each school board. This report, and all information reported herein, is completed with the cooperation and contribution of all four Ottawa school boards, Conseil des écoles catholiques du Centre- Est, Conseil des écoles publiques de l Ést de l Ontario, Ottawa Carleton District School Board, Ottawa Catholic District School Board and both service providers, Maison Fraternité and, Rideauwood Addiction and Family Services. Clients of the school-based counselling program are provided with consent for evaluation request forms. Consent must be obtained for individual data to be included in the evaluation. Parents or clients of age of consent (subject to school board policy) are given the opportunity to provide consent, withhold consent or withdraw consent for participation in the program evaluation at any time without consequence. From consenting clients, this evaluation includes the following anonymous and amalgamated information: counsellor assessments, the Global Appraisal of Individual Needs Short Screener (GAIN SS), Behaviour and Symptom Identification Scale (BASIS 32) and the Drug Taking History Questionnaire (DTHQ), credit and grade summaries; and, testimonials provided by parents, students and school staff. Comparison data is obtained at the initiation and completion of treatment or the end of the school year (with a minimum of 3 months between pre and post assessments). b. Evaluation Plan The key outcomes measured by the school-based counselling annual evaluation are: changes in client substance use (frequency and quantity); changes in client mental health and need; changes in client wellbeing and daily living; and, changes in client academic achievement. The design for this evaluation is quasi experimental, comparisons are made between pre and post measures matched by client. Three clinical tools, the GAIN SS, BASIS 32 and DTHQ, are administered as pre and post measures of mental health, wellness and daily living and substance use. Change in academic achievement is assessed by comparison of average grades and credit completion in the current and previous year. Testimonials are solicited from school staff to assess qualitative perception of the program and obtained voluntarily from parents and youth. 4
The tools included in evaluation are those required for screening and assessment by the Ontario Ministry of Health and Long Term Care (MOH/LTC). As these requirements change over time, so too the tools used for assessment and evaluation in the school based program will change. The tools are validated for use with clients aged 12 and older. The modified GAIN SS is a screening tool used to measure and flag substance use and mental health need. The BASIS 32 is a full assessment tool measuring difficulty in the areas of: (1) relation to self and others; (2) daily living and functioning; (3) depression and anxiety; (4) impulsiveness; and, (5) psychosis. The DTHQ tracks reported substance use and engagement in problem behaviour by frequency and quantity. c. Sample Size Sample size is determined by the number consents for evaluation submitted and the completion and submission of clinical assessments and academic records (both pre and post). As such, the sample size varies from n=115 to n=39 depending on the available information (for example, the sample size for change in use of any one substance will vary depending upon the number of clients reporting use for each substance). d. Statistical Analysis Analysis of the results was performed according to the outlined evaluation plan established in 2008. Paired samples, pre and post outcome measures, with sample size permitting were compared with by a paired samples T-test using the open source software RStudio version 0.99.892 2009-2016 RStudio, Inc. 5
4. Results a. Schools Served Students from 57 high schools in Ottawa, including alternate schools, received support from school-based counsellors during the 2015/16 school year. Schools typically receive about 14 hours of service per week but service hours are uniquely planned for each school according to need. Prevention and education sessions for students, parents and teachers were also provided throughout Ottawa high schools and middle schools (grades 7 and 8). About 4,096 high school students, 8,218 grade 7 and 8 students, and 3,015 parents (including parent information evenings) attended education and prevention sessions. b. Students and Families Served Over 1400 students received counselling services from the school-based program in all four Ottawa school boards (reaching French, English, Catholic and Public schools). The school based program does not have a waitlist for service, students are able to access a counsellor when needed. Counselling services are also available to parents of students from all participating schools. Over 200 parents engaged in counselling services in the 2015/16 school year and notably, the waitlist was reduced to less than 10 parents waiting throughout the year. The majority of students, 68%, were referred to a school-based counsellor by an education or training service or program. Twelve percent of students self-referred, 16% were referred by a family member or friend, and 5% were referred by a community agency (including community health centers or treatment agencies, hospitals, social services or legal services). Thirty-five percent of parent clients were self-referred, 42% were referred by family or friends, 8% by an education or training program or service, and 14% by a community agency or service provider (such as a community agency, hospital, mental health practitioner or general practitioner). c. Client Profile In the 2015/16 school year, 60% of clients identified as male and 40% female (n=111). Seventy-eight percent of clients were aged between 15 to 17 years of age. Twelve percent of clients were 14 years of age or younger. Clients ranged in age from 13 to 19 years (see figure 1). 6
% of clients % of students School-based program evaluation 2015/16 50 45 40 35 30 25 20 15 10 5 0 30 24 24 10 9 0 2 1 12 13 14 15 16 17 18 19 Student age at admission Figure 1: In the 2015/16 school year, both the average and median client age was 16 years old and ranged from 13 to 19 years old (n=111). At first assessment in the 2015/16 school year, of those clients who reported at least some substance use in the past year they most frequently reported use of: cannabis (88%), alcohol (82%), nicotine (56%), ecstasy (37%), hallucinogens (26%), cocaine (22%), and stimulants (21%). Problematic video game or computer use in the past year was also frequently reported (38%) (see figure 2). 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Substance/ Behaviour Figure 2: The most commonly reported substances of use or problem behaviour within the year before assessment included: cannabis, alcohol, hallucinogens, nicotine, video games/ computers, and ecstasy (n=78 students reporting at least some substance use or problem behaviour in the past year). 7
% of clients School-based program evaluation 2015/16 In addition to problem substance use and behaviour, a large percentage of clients entered treatment with signs of mental health need. The most commonly flagged disorders by the GAIN SS included: externalizing disorder (60%), internalizing disorder (54%), substance use (28%), and, crime and violence (28%). As measured by the BASIS 32, many clients were also found to be struggling with: daily living and role function (26%), relation to self and others (25%), and impulsive and addictive behaviour (15%). When needed clients were referred to additional services for support with mental health and trauma, or may have been referred to substance abuse counselling after first seeking mental health support. Seventy-two percent of clients entering the school-based program were flagged by two or more screeners on the GAIN SS, showing the great complexity of need common among school-based clients (see figure 3). 60% 50% 40% 30% 20% 10% 0% 24% 24% 18% 16% 11% 5% 3% 0% 0% None One Two Three Four Five Six Seven Eight # of concurrent flags Figure 3: Seventy-two percent of clients in the school based program were flagged upon entry into the program by two or more screeners on the GAIN SS. d. Student Engagement Students may engage with a counsellor by self-referral or at the recommendation of a friend, family member, teacher, guidance counsellor, other school staff member, or by referral from a community program or health care provider. Many students also engage with a counsellor without being formally entered into the program as clients, these individuals may require only information or a few sessions of support, or may be better served by a referral to another organization. Of the students who do enter into programing as a client, the average length of stay was 12 months. 8
e. Student Health and Wellbeing Change in client mental health and wellbeing was assessed by a comparison of need before treatment and at the end of treatment or during treatment at the end of the school year (following at least 3 months of treatment). Need in relation to mental health and substance use as measured by the GAIN SS was alleviated for almost every domain. The number of clients flagged by the GAIN SS decreased by: 31% for substance use; 30% for externalizing disorder and internalizing disorder; 22% for crime and violence; and, 19% for eating disorders (see table 1). Clients flagged at baseline Clients flagged at comparison Change in clients flagged Internalizing Disorder 54% 24% -30% Externalizing Disorder 60% 30% -30% Substance Disorder 47% 16% -31% Crime/Violence 28% 6% -22% Eating Disorder 26% 7% -19% Post-Traumatic Stress Disorder 21% 24% 3% Psychosis 25% 21% -4% Problem Gaming and Internet Usage 14% 14% 0% Gambling 0% 0% 0% Table 1: In the 2015/16 school year, clients were flagged by the GAIN SS most often at baseline for: externalizing disorder (60%); internalizing disorder (54%); substance use (47%); and, crime or violence (28%). At comparison, the percentage of clients flagged by most screeners had decreased: 22% of clients flagged for crime and violence at baseline were no longer flagged at comparison; 19% of clients flagged for an eating disorder at baseline were no longer flagged at comparison; 30% flagged for externalizing disorder and substance use at baseline were no longer flagged at comparison; and, 30% of those flagged for internalizing disorder at baseline were no longer flagged at comparison (n=40). 1 Change in client wellbeing was measured by the BASIS 32 was also measured at the beginning and end of treatment or the end of the year. The average need score decreased (improvement) across all domains: impulsive and addictive behaviour decreased by an average of.3; relation to self and others decreased by an average of.2; and, daily living/ role functioning, depression/ anxiety and psychosis decreased by an average of.1. Additionally, the overall average score decreased by.5 (see figure 4). 1 Comparison based on past 3 months only due to timeline of less than a calendar year between pre and post assessment. 9
Average Change School-based program evaluation 2015/16 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 1.3 1.1 1.4 1.3 1.2 1.1 1 0.7 0.5 0.4 1.1 0.6 Category Figure 4: In the 2015/16 school year, clients showed improved scores on all scales of the BASIS 32, with the greatest improvement on scores for impulsivity and addictive behaviour (decreased by.3 points). Overall average change in score showed an overall decrease of.5 (n=39). The percentage of clients flagged by the BASIS 32 also decreased over the course of treatment for almost every domain. The percentage of clients flagged for relation to self and others and depression and anxiety decreased by 12%. The percentage of clients flagged for impulsive and addictive behaviour decreased by 10% and the percentage of clients flagged for daily living and relation to self and others decreased by 5% (see table 2). Clients flagged at baseline Clients flagged at comparison Change Relation to Self/Others 25% 13% -12% Daily Living/ Role Functioning 26% 21% -5% Depression/ Anxiety 29% 17% -12% Impulsive/ Addictive 15% 5% -10% Psychosis 3% 3% 0% Table 2: In the 2015/16 school year, fewer clients were flagged at comparison on almost all scales of the BASIS 32 when compared with baseline. The greatest percentage change was found for clients flagged for relation to self and others and depression and anxiety which decreased by 12% between baseline and comparison. Impulsive and addictive behaviour decreased by 10% (n=39). 10
f. Drug and Alcohol Use Changes in substance use or engagement in problem behaviour are tracked by the DTHQ at the beginning of treatment and at the end of treatment or school year. Among the most commonly reported substances of use or problem behaviour, most clients are able to reduce or stop their use: 88% of clients reporting use of ecstasy were able to stop their use by comparison; 75% percent of clients were able to stop their use of cocaine or hallucinogens; 64% of clients were able to stop or reduce their use of cannabis; 62% of clients reporting alcohol use were able to stop or reduce their consumption; and, 56% of clients were able to reduce or stop their consumption of nicotine (see table 3). Percentages are provided for the most commonly reported substances of use but given the significant concern in relation to opioid use those results will also herein be reported. The results of four clients reporting prescription opioid use with comparison data showed that: one client was able to stop use; one was able to reduce use; and, two reported increased use. For the two clients reporting over the counter codeine use with follow-up results: one was able to stop use; and, one reported no change in use. Alcohol Change Cannabis Change Nicotine Change Cocaine Change Ecstasy Change Hallucinogen Change Stopped 18% 18% 19% 75% 88% 75% Reduced 44% 46% 37% 0% 0% 0% No change 22% 16% 33% 13% 11% 0% Increased 16% 20% 11% 13% 0% 25% Table 3: Among the most commonly reported substances of use on the DTHQ, high percentages of clients stopped or reduced their use between baseline and comparison: 88% of clients reporting use of ecstasy at baseline stopped their use by comparison; 75% of clients reporting use of cocaine or hallucinogens at baseline stopped their use by comparison; 64% of clients reporting use of cannabis stopped or reduced their use; 62% of clients reporting use of alcohol stopped or reduce their use; and, 56% of clients reporting use of nicotine stopped or reduced their use. This measure only reflects change as reported to the counsellor and not a quantitative amount of change and sample size varies according to reported use by substance. The DTHQ also tracks changes in the average number of days of use and the average consumption in a month. In 2015/16 clients were able to significantly decrease both the average number of days of consumption of alcohol (t=2.54, df= 46, p=0.01) and the average amount of alcohol consumed (t=2.76, df=33, p=0.009). The average number of days of cannabis consumption also decreased significantly (t=2.7, df=53, p=0.009). Average days of use also decreased for: nicotine, cocaine, ecstasy and hallucinogens (see figure 5). 11
Average # of days of use in past 30 days School-based program evaluation 2015/16 30 25 20 17.63 * 23.45 19.2 15 13.45 10 5 0 6.26 * 4.03 2.18 1.59 1.37 0 1.54 0.55 Substance Figure 5: As measured by the DTHQ, average days of use decreased among clients for all substances reported between baseline and comparison. 2 Both the decrease in average days of alcohol use and decrease in average number of drinks were statistically significant. The average number of days of use for cannabis also decreased significantly (sample size varies according to reported use by substance). g. Student Academic Outcomes Academic achievement, as measured by average grade and credit completion, is also assessed for clients of the school based program. Change in average grade following treatment in 2015/16 was found to be significantly improved by 3.06% (t= -2.22, df=106, p=0.029) (see figure 6). The number of credits completed remained essentially unchanged at 5.75 credits in 2014/15 and 5.48 credits in 2015/16. Additionally, almost all students (97%) were able to remain active in school when assessed at follow-up. 2 Some clients report use but not quantity or frequency. 12
Average Grade (%) School-based program evaluation 2015/16 80 70 60 63.74 * 66.80 50 40 30 20 10 0 Past Year Current Year Year Figure 6: According to client credit reports provide by schools, grade average improved among clients from the previous to current year by 3.06% (n=115). h. School staff testimonials In the 2015/16 school year, testimonials were obtained for Maison Fraternité. The following are comments obtained from school staff, parents and youth reflecting on their perception of the school-based program: Personnel scolaire: - L accès rapide et la flexibilité de Maison Fraternité sont appréciés. Le programme est conçu pour aider l élève à s engager en répondant aux besoins distincts des jeunes de chaque école. - La présence du psychothérapeute en milieu scolaire est une excellente ressource qui est bénéfique autant pour l élève que pour l école. - Le travail en équipe, le suivi collaboratif et le partage de stratégies avec le psychothérapeute en milieu scolaire de Maison Fraternité créent une dynamique favorable pour l élève. Parents: - Je suis épatée par les services que je reçois avec la psychothérapeute de Maison Fraternité. Ses interventions font une grande différence dans ma vie et dans notre vie familiale. 13
- Le groupe de parents m a aidé à mieux comprendre mon rôle de parent et aussi comprendre ce que mon jeune passe à travers. Jeunes: - Les rencontres avec l intervenante m a permis de parler avec quelqu un en dehors de la famille et des amis, et de mieux comprendre mes problèmes. - Les conseils que l intervenant m a donnés me font vraiment penser à l extérieur des rencontres pour changer mes habitudes de vie et pour vouloir faire ce qui est mieux pour moi, plutôt que ce que je suis tentée de faire. 5. Discussion Clients of the school based counselling program frequently begin treatment with very complex needs. The effectiveness of treatment is primarily focused on reduced substance use and reduced impact of use on client behaviour and mental health. Reductions in substance use and reductions in harm related to use are the primary objectives of school based counselling and treatment results show a very strong effect in reducing or stopping use of the most commonly reported substances including alcohol, cannabis, nicotine, cocaine, ecstasy and hallucinogens. In comparison with previous annual evaluation reports of the school based program, clients were found to begin treatment with generally higher rates of internalizing disorder, externalizing disorder and crime and violence. It is of interest to examine this change in subsequent years. It is also advised that outgoing referrals be tracked in detail in the upcoming year to assess the effect of coordinated care. The successes reported herein often reflect the work of not only the partners directly credited but many other partner community agencies providing broad support to youth. Average grade and credit completion has rarely shown notable changes in past years but, this year a significant improvement was found for average grade. Given that academic achievement would be expected to show greater improvement in the years following treatment, it is recommended that a longitudinal program evaluation report be considered. Lastly, the recommended tools of screening and assessment have recently been modified and the school based counselling program will be transitioning to the newly recommended suite of tools, including the GAIN Q3/MI. In the upcoming year, both service providers will be 14
integrating both the old and new tools during this transition year and the report for the 2016/17 school year will reflect this change. The upcoming report will be used to determine what new information may be available for evaluation in subsequent years. 6. Conclusion Given that this cooperative initiative, providing school-based substance abuse counsellors to all Ottawa area publically funded high schools, has been ongoing for about seven years and data for evaluation has been gathered for each year it is recommended that a longitudinal evaluation be undertaken to evaluate outcome trends over time and long term effect of the program. The timing of this report would coincide well with the opportunity to gather more information from the new assessment tools (GAIN Q/MI) and could provide direction related to gaps in the current evaluation plan. Additionally, due to the high numbers of students being flagged for complex and concurrent mental health need, it is recommended that referral processes and mental health support be a key focus of programming in 2016/17. Lastly, small sample size has been a recurring concern for this evaluation and should be addressed. It is recommended that a review of the process and plan for improvement be initiated to increase response rate for evaluation. 15