Partnerships for Success II Final Evaluation Report

Transcription

1 Partnerships for Success II Final Evaluation Report Prepared for Maine Office of Substance Abuse and Mental Health Services By Hornby Zeller Associates, Inc. January 2016

2

3 TABLE OF CONTENTS PROJECT BACKGROUND... 1 PURPOSE OF THIS REPORT... 3 METHODOLOGY... 5 DESCRIPTION OF PFS II IN MAINE... 7 EVALUATION FINDINGS Process Results PROJECT OUTCOMES State Epidemiology Outcomes Workgroup Outcomes Monitoring LESSONS LEARNED Successes Challenges CONCLUSIONS AND RECOMMENDATIONS APPENDIX A: Focus Group Protocols APPENDIX B: PFS II Strategy Survey APPENDIX C: Map of Local HMPs Appendix D: PFS II Strategies... 45

4

5 PROJECT BACKGROUND In 2012, the Maine Office of Substance Abuse and Mental Health Services (SAMHS) received funding from the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) to implement the Partnerships for Success II (PFS II) program in Maine. PFS II is designed to address two substance abuse prevention priorities: 1) underage drinking among year olds and 2) prescription drug misuse and abuse among year olds. For Maine s PFS II program, SAMHS added a third priority: the prevention of marijuana use among year olds. Eligibility to apply for federal funds under the PFS II program was restricted to states that were recipients of funding through the Strategic Prevention Framework State Infrastructure Grants (SPF SIG). Maine was one of 15 states or territories to receive PFS II funding and is the only grantee that is using funds at the community level across the entire state. The PFS II grant came to a close on September 30 th, However, work under PFS II will be sustained under the Partnerships for Success 2015 grant which the State was awarded earlier in the year. In the first year of the PFS 2015 grant funds will continue to support prevention strategies implemented at the public health district level in collaboration with local community coalitions. One important change has occurred, however: the process for community strategy selection has been modified from a menu of prevention strategies to a process whereby subgrantees undergo a data-driven needs assessment and subsequently choose strategies that best align with the communities needs. Despite the new process, the core goals of reducing the rates of underage drinking among middle- and high school-aged youth and year olds, reducing the rates of prescription drug use among middle- and high school aged youth and year olds, and reducing the rates of marijuana use among middle- and high school-aged youth and year olds remains the same. Additionally, the PFS 2015 grant will continue to strengthen the work related to building a strong Strategic Prevention Framework in the State. The Strategic Prevention Framework (SPF), 1 which provides the foundation for the PFS II grants, is a prevention approach developed by SAMHSA that embraces and promotes public health data-driven decision-making and effective, culturally appropriate, and sustainable prevention activities. Strategic Prevention Framework theory is based on reducing risk factors and promoting protective factors that have an impact on substance use. Using the SPF model, strategies are developed to address underlying causes and consequences of substance abuse and to build capacity and enhance infrastructure and evidence-based programs and practices. The SPF requires states and communities to undertake the following five steps systematically: 1) Assess their prevention needs based on epidemiological data; 2) Build their prevention capacity; 3) Develop a strategic plan; 4) Implement effective community prevention programs, policies, and practices; and 5) Evaluate their efforts for outcomes. From , Maine used its SPF SIG funds, the first to be granted using the SPF, to develop and expand the state s emerging public health infrastructure, and in particular the state s substance abuse prevention capacity. By 2011, the infrastructure included 27 Healthy Maine 1 PFS II Final Evaluation Page 1

6 Partnerships (HMPs) and nine Public Health Districts (PHDs). SAMHS worked closely with the Maine Centers for Disease Control and Prevention to support the HMPs and provided workforce development and training in the SPF model as well as funds for specific prevention interventions. SPF SIG funds also were used to develop a surveillance network through the creation of the State Epidemiological Outcomes Workgroup (SEOW). The specific focus of Maine s SPF SIG was the prevention of high-risk drinking (underage drinking and binge drinking). Some communities also used funds for preventing prescription drug misuse. Several communities across the state have also received Drug Free Communities (DFC) grants, which provide significant funding to address youth substance abuse, including marijuana use. These major initiatives, as well as other community programs funded by SAMHS, have resulted in measurable success at the local level in reducing youth substance abuse and have helped to lay a solid foundation for Maine s prevention work. Over the same time period separate efforts were undertaken by the State to improve the Maine Youth Drug and Alcohol Use Survey (MYDAUS, now called the Maine Integrated Youth Health Survey, or MIYHS) which is the cornerstone for data collection of youth health behaviors and risk and protective factors. In 2012, when SAMHSA announced the PFS II grant program to grantees that had successfully completed the SPF SIG grant, Maine was well positioned to receive funding. Under the PFS II program, states receive funding and resources to implement the Strategic Prevention Framework across the state and at the community levels. In addition to funding community-level prevention strategies, PFS II supports the work of Maine s SEOW to monitor a wide array of indicators related to rates of substance use in Maine, factors contributing to or mitigating substance use, and related consequences of substance misuse and abuse. The goals of Maine s grant were a five percent reduction in each of the following by September 30, 2015, the end of the federal fiscal year and of the grant itself: Maine s 7 8 th graders who used alcohol, used marijuana, and/or misused prescription drugs in the past 30 days as measured by the Maine Integrated Youth Health Survey (MIYHS); Maine s 9 12 th graders who used alcohol, used marijuana, and/or misused prescription drugs in the past 30 days, as measured by the MIYHS; Maine s year olds who used alcohol in the past 30 days as measured by National Survey for Drug Use and Health (NSDUH); and, Maine s year olds who used marijuana in the past 30 days and who misused nonprescribed pain relievers as measured by NSDUH. A consequence goal was also identified for the grant: Reduction of annual alcohol and/or drug related car crashes among Maine s year olds from 346 to 300 as measured by Maine Bureau of Highway Safety and Maine Department of Transportation (DOT) statistics. PFS II Final Evaluation Page 2

7 PURPOSE OF THIS REPORT SAMHS contracted with Hornby Zeller Associates, Inc. (HZA) to conduct the statewide evaluation of the PFS II grant and to produce yearly reports of process and outcome evaluation findings. HZA is charged with reporting back to the State findings from a variety of quantitative and qualitative sources that are used to describe the grant s prevention activities and their impact on substance use. The Methodology section describes HZA s procedures for collecting evaluation data for both the process and outcome evaluation components. The second section, Description of the PFS II Grant in Maine, describes the State s infrastructure under which the PFS II grant was implemented and the strategies that were selected by community sub-grantees. The Evaluation Findings section includes Process Results, Project Outcomes, and Lessons Learned. Process Results includes findings specifically regarding Healthy Maine Partnership infrastructure and strategy implementation. These findings represent results from the Strategy Survey that was implemented to HMP partners as well as from information gathered during focus groups. Process Results also include findings specific to the Tribal HMP. Project Outcomes represents the SEOW portion of the grant and presents accomplishments related to SEOW, current data on indicators that are directly related to Maine s PFS II objectives, as well as other intersecting consumption, intervening, and consequence indicators. Examining the process and outcome results separately reveals how SAMHS and HMPs have carried out their PFS II strategies and what has been accomplished in achieving the goals laid out above. Examining the two sets of results together indicates the potential impact that PFS II strategies may have had on substance use outcomes. Lessons Learned present general successes and challenges that were encountered at both the state and local levels in the execution of the PFS II grant, and include findings related to both the process and outcome evaluation. The Conclusions and Recommendations section of the report wraps up the evaluation findings of PFS II with recommendations to SAMHS and community sub-grantees for implementation of the PFS 2015 grant. PFS II Final Evaluation Page 3

8 Examining the process and outcome results separately reveals how SAMHS and HMPs have carried out their PFS II strategies and what has been accomplished in achieving project goals. Examining the process and outcome results together indicates the potential impact that PFS II strategies may have had on substance use outcomes. PFS II Final Evaluation Page 4

9 METHODOLOGY Hornby Zeller Associates employs an Action Research Model in its evaluations, which means that the evaluator is actively engaged in the project or grant implementation process by providing technical assistance and guidance on an ongoing basis. As evaluators of the PFS II grant, HZA participates in the HMP grantee meetings, the SAMHS Prevention Advisory Board meetings and the Marijuana Workgroup. The SEOW Coordinator is employed on the project through HZA, providing expertise and technical assistance on data collection and analysis directly to HMPs, developing epidemiological profiles for public health districts and HMPs, and regularly updating indicators of statewide substance use on the web-based Maine SEOW Dashboard. HZA relied on four data sources for the process evaluation: document review, information from the web-based data portal called KIT, focus groups, and a Strategy Survey. For the outcome evaluation, HZA relied on data processed through the SEOW, which uses national and statewide health surveillance systems to collect information on behavioral health in Maine. Documents reviewed included meeting minutes and materials produced by the Prevention Advisory Board and the Marijuana Workgroup, as well as other SAMHS-produced resources that were available to communities. HZA staff also reviewed information that was retrieved from KIT where HMPs report information to SAMHS on prevention activities and benchmarks. Specifically, HZA reviewed process measure counts, HMP work plans, and progress made on benchmarks at the community level. In November of 2015, HZA staff conducted focus groups with prevention staff from all 27 HMPs to understand how sub-grantees implemented PFS II at the community level. Nine focus groups were held, one in each of the nine public health districts, with between one and four prevention staff from each HMP attending. The focus groups were designed to elicit open feedback regarding successes and challenges to implementation, HMP processes for building prevention capacity and community relationships, and level of satisfaction with technical assistance received from the State. Focus group protocols can be found in Appendix A. Additionally, a Strategy Survey was completed by all HMPs wherein strategies were rated on community relevance, feasibility, and ability to have an impact. The Strategy Survey can be found in Appendix B. HZA also held a focus group with individuals directly involved in implementing the grant at a statewide level. Participants included the grant manager, grant coordinators and technical assistance providers, and the Marijuana Workgroup coordinator, all of whom serve on SAMHS Prevention Team. This focus group was designed to gather information about implementation of the project from an administrative perspective, such as provision of technical assistance and workforce development, and to better understand the successes and challenges at the state level. PFS II Final Evaluation Page 5

10 All focus groups gathered information about the grant as a whole ( ) with the understanding that some HMP and State staff may not have been there for its entirety. Additionally, at the time the focus groups were conducted, the PFS 2015 grant was already underway, and so evaluators also sought to collect information on the lessons learned from PFS II that could apply to the new project. For outcomes data presented in the Outcomes Monitoring section, HZA uses data from a variety of state and national data sources, including MIYHS, the Maine Department of Transportation (MDOT), and the National Survey for Drug Use and Health. Data from MIYHS are collected and released bi-annually by the State and are designed to monitor health-related behaviors and attitudes of 5 th through 12 th graders by direct student survey. Data from NSDUH are collected annually by SAMHSA and rolled into two year datasets. NSUDH surveys households throughout the nation on substance use related behaviors and attitudes. Data from MDOT are collected and released annually. These data sources provide the indicators needed to measure the outcomes of this project, and the indicators match across data sets. It is important to note that one NSDUH indicator differs slightly from the corresponding MIYHS indicator: Instead of measuring past 30-day prescription drug use, the NSDUH indicator measures past year misuse of non-prescribed pain relievers. Focus groups were designed to elicit open feedback regarding successes and challenges to implementation, HMP processes for building prevention capacity and community relationships, and level of satisfaction with technical assistance received from the State. PFS II Final Evaluation Page 6

11 DESCRIPTION OF PFS II IN MAINE At the state level, PFS II funds were used to manage and coordinate the project, build statewide infrastructure and capacity by providing resources, professional development opportunities, and technical assistance to HMPs. The Marijuana Workgroup was formed and sustained under PFS II, and the project funded the development of prevention materials that were then disseminated to HMPs. State level funds were also combined with other federal funds to coordinate the State Epidemiology Outcomes Workgroup. At the community level, PFS II funds are used to enhance strategies to prevent alcohol and prescription drug abuse that are already in place at the HMP level and to support new strategies to prevent marijuana use and implement Screening, Brief Intervention and Referral to Treatment (SBIRT). Because SAMHS already funds strategies to support prevention of underage drinking and prevention of prescription drug misuse, HMPs chose the ways in which PFS II funds could enhance and support those existing strategies. In addition, all HMPs were required to implement at least one marijuana strategy and to decide whether they wanted to implement SBIRT. It is important to remember that each HMP selected and implemented different strategies with PFS II funds, that each HMP has a variety of other substance abuse prevention funding and other funding sources, and that each group interprets and implements prevention work within the unique context of local community needs. HMPs are supported by several state funding sources to implement strategies in alcohol and prescription drug abuse prevention: the Substance Abuse Prevention and Treatment Block Grant through SAMHS, the Enforcing Underage Drinking Laws program, and the Fund for a Healthy Maine, as well as federal funding sources such as Drug Free Communities Grants. Most local coalitions receive some combination of these grants, and also leverage other state and federal funding and small foundation and private funding for initiatives specific to their community needs. Because of this complex funding structure to support prevention efforts, it is impossible to attribute any measurable outcome to the PFS II grant alone, although it is clear that it has contributed significantly to those outcomes. Evaluation of PFS II at the coalition level aimed to capture the entire picture of prevention efforts, including the process of coalition building and engagement with community partners, the outcomes and outputs associated with the efforts of the coalition, and the impact on community indicators. Sub-grantees of PFS II funds are the Lead HMPs in each of the nine Public Health Districts (PHDs). They in turn distribute funds to local HMPs (between two and five HMPs per district), where strategies are implemented at the community level. In this report, the unit of analysis is the PHD, with considerable descriptive information provided at the HMP level. A map of PFS II Final Evaluation Page 7

12 Aroostook Central Cumberland Downeast Midcoast Penquis Western York Tribal Healthy Maine Partnerships and their respective public health districts can be found in Appendix C. Throughout this report it is important to remember that each HMP selected and implemented different strategies with PFS II funds, that each HMP has a variety of other substance abuse prevention funding and other funding sources, and that each group interprets and implements prevention work within the unique context of local community needs. To this end, distribution of strategies across PHDs can vary widely. For example, some HMPs selected SBIRT strategies in Year 1 but did not continue those strategies in subsequent years, while other HMPs may have focused heavily on SBIRT strategies in all three years. Table 1 shows which strategies each of the Public Health Districts selected in each of the three years of the grant. Some strategies listed here were not offered in Years Two or Three. Table 1. Public Health District Strategies PUBLIC HEALTH DISTRICTS SUBSTANCE Alcohol STRATEGY Party Smarter, alcoholscreening.org X X X X Table Talk, PWHLTM X X X X X X Sticker Shock X X X RBS X X X X X Task Force X X X Underage Drinking Enforcement X X X X X X X X Alcohol Awareness X X X Marijuana Marijuana Education X X X X X X X X X Marijuana Dissemination X X X X X X X X X PMP Registration X X X X TA for PMP X X X X X Prescription Drug Rx Misuse Awareness X X X X X X X X SBIRT Safe Storage & Disposal Rx Community Intervention SBIRT Healthcare Providers SBIRT Community Organizations X X X X X X X X X X X X X X X X X X X X X X X X X X X X X PFS II Final Evaluation Page 8

13 Table 2 lists the objectives and strategies implemented under PFS II along with a resource link to more information and a brief description of each strategy. The PFS II grant has six major objectives pertaining to the following issues: 1) engagement of liquor licensees to reduce underage access to alcohol; 2) strengthening of the local underage drinking enforcement task force; 3) promotion and dissemination of information regarding high-risk drinking among young adults; 4) increasing awareness of and education on marijuana use; 5) increasing awareness of and education on prescription drug use; and 6) increasing the use of Screening, Brief Intervention, and Referral to Treatment (SBIRT) efforts among healthcare providers and community organizations. Maine was one of 15 states or territories to receive PFS II funding and is the only grantee that is using funds at the community level across the entire state. PFS II Final Evaluation Page 9

14 Table 2. PFS II Objectives and Strategies OBJECTIVE 1: Engage liquor licensees to participate in two or more strategies that lead to responsible retailing as defined by SAMHS guidelines. STRATEGY 1.1 Plan, coordinate, advertise, and/or host in-person Responsible Beverage Server / Service (RBS) trainings. 1.2 Engage liquor licensees to participate in the Card ME program. 1.3 Encourage Law Enforcement to partner with HMP in working with local licensees. DESCRIPTION Develop positive relationships with the liquor licensees that will help to facilitate the likelihood that they will accept an invitation to attend RBS training and CardME program implementation. RBS training provides education on alcohol and Maine Liquor Laws to liquor licensees, managers, clerks, and servers of alcohol. The Card ME Program aims to reduce illegal and/or irresponsible alcohol sales and service by providing communities and liquor licensees with a model and resources to: Make it more difficult for underage and visibly intoxicated persons to obtain alcohol from liquor licensees; Increase a licensee's capacity and motivation in improving their responsible retailing efforts; Build stronger community norms around limiting alcohol availability to underage and visibly intoxicated persons. Foster trust and confidence among liquor licensees and law enforcement agencies and build positive connections between the two. A positive connection is achieved when the two parties understand the other s perspective and respects their work. For instance, licensees can support law enforcement through voluntary compliance, notifying law enforcement of potential furnishing cases (i.e., reporting large purchases of alcohol, or overhears where they re going). LE can support licensees by responding quickly to calls for help, stopping by occasionally, or letting the licensee know when they have given a good tip. OBJECTIVE 2: Engage a local area Underage Drinking Task Force to participate in two or more strategies that lead to the increased perception among youth that they would be caught using alcohol. STRATEGY 2.1 Work with law enforcement agencies to ensure participation on an active Task Force which is representative of key stakeholders in the community. 2.2 Collaborate with law enforcement to increase the enforcement of underage drinking laws. SA 2.3 Work with law enforcement to communicate the increased enforcement efforts and community supports. DESCRIPTION Promote participation of an active Task Force in which members regularly attend meetings and develop and implement plans to prevent underage drinking. Support law enforcement in taking a critical look at what strategic improvements / enhancements can be made to improve their capacity, readiness, and effectiveness around underage drinking enforcement. (See Appendix A for more details) Publicize incidents and penalties of those caught for furnishing alcohol and hosting places to drink. Use media to strategically increase community support for underage drinking enforcement. (See Appendix A for more details) PFS II Final Evaluation Page 10

15 OBJECTIVE 3: Engage partners (e.g., schools, hospitals, provider offices, service agencies, civic organizations, etc.) that promote and disseminate information about the risk of underage alcohol use and high risk drinking in young adults. STRATEGY DESCRIPTION 3.1 Engage business and community organizations to assist with implementing components of Table Talks with parents and Parents Who Host, Lose the Most. 3.2 Engage local public gathering places (i.e. libraries, schools, healthcare facilities) to promote and disseminate SAMHS parent media campaign materials, including the promotion of as a health resource for young adults, and the Party Smarter campaign on campus and offcampus settings. OBJECTIVE 4: Increase communications to the public about the risk and harm of marijuana use. Table Talks are meetings with parents that are held in neutral settings like schools, during sports practices, libraries, or other agreeable places that parents are likely to congregate, and are opportunities for parents and prevention providers to discuss issues and resources related to youth substance use. Promoting the online screening tool can include providing materials with the screening tool info, PSAs, Combine the tool with messaging about the consequences of overdrinking. Party Smarter is a communication campaign to promote safer and smarter drinking among young adults, and includes tips on planning, preparing, and pacing alcohol consumption. STRATEGY DESCRIPTION 4.1 Provide educational Provide educational opportunities such as presentations at teacher in-service days, opportunities using SAMHS lunch and learns, presentations for school based parent organizations, Adult Ed developed training materials to offerings, tables at sporting events, participation in community events, etc. residents including but not limited to, parents, teachers, counselors and community organizations, about the social, legal, health, and community impacts of marijuana use. 4.2 Disseminate information and/or Disseminate materials created by SAMHS and other reputable sources. Develop a materials for residents including, communication plan that includes the use of social media. but not limited to, parents, teachers, counselors, and community organizations about the social, legal, health, and community impacts of marijuana use. OBJECTIVE 5: Increase the implementation of prevention strategies to enhance community engagement in addressing Rx abuse. STRATEGY 5.1 Increase the number of prescribers/dispensers who are registered and actively use the Prescription Monitoring Program DESCRIPTION Presentation of the Prescription Drug Monitoring Program (PMP) PowerPoint vetted through SAMHS through an in-person educational session with prescribers and dispensers. Promote utilization of the PMP and offer technical assistance. PFS II Final Evaluation Page 11

16 5.2 Disseminate materials to increase awareness around the dangers of prescription drug misuse 5.3 Disseminate materials to increase awareness around safe storage and disposal of prescription medication Dissemination of materials in places where prescription drugs misuse awareness information may be relevant: provider offices such as pediatricians and PCPs, dentists offices, pharmacies, child cares, senior centers, area agencies on aging, schools, youth centers such as Boys and Girls clubs, community centers, and faithbased organizations. Dissemination of Up and Away materials in places where safe disposal information may be relevant: provider offices such as pediatricians and PCPs, pharmacies, child cares, senior centers, area agencies on aging, elementary schools, youth centers such as Boys and Girls clubs, or as an addition to healthy homes materials. OBJECTIVE 6: Increase the number of community organizations (e.g., worksites, service agencies, civic organizations) that utilize screening and brief intervention tools as a matter of practice with a special emphasis on health care providers. STRATEGY 6.1 Educate health care providers about how to implement screening, brief intervention, and referral for assessment for individuals with an increased risk of substance abuse problems. DESCRIPTION Increase the number of health care providers that use screening, brief intervention, and referral to assessment/treatment (SBIRT) in an effort to reduce the risk associated with drug and alcohol use before more serious problems occur. Education includes presentation of a PowerPoint training to healthcare providers (provided by SAMHS), that includes information on how to use available screening tools, brief interventions, and the referral process for assessment/treatment. SA 6.2 Educate community organizations to promote the use of screening and brief intervention tools to address individuals with increased risk for substance abuse. Increase the amount of community organizations (e.g., mental health agencies, crisis interventionist, work sites, faith-based organizations etc.) that use screening, brief intervention, and referral to assessment/treatment (SBIRT) in an effort to reduce the risk associated with drug and alcohol use before more serious problems occur. Education includes presentation of a PowerPoint training (provided by SAMHS), that includes information on how to use available screening tools, brief interventions, and the referral process for assessment/treatment. PFS II Objectives 1) Engagement of liquor licensees to reduce underage access to alcohol 2) Strengthening of the local underage drinking enforcement task force 3) Promotion and dissemination of information regarding high-risk drinking among young adults 4) Increasing awareness of and education on marijuana use 5) Increasing awareness of and education on prescription drug use 6) Increasing the use of Screening, Brief Intervention, and Referral to Treatment (SBIRT) efforts among healthcare providers and community organizations. PFS II Final Evaluation Page 12

17 EVALUATION FINDINGS The PFS II evaluation plan has three foci: the first is on infrastructure at the community level; the second is on how marijuana strategies were adapted and implemented; and the third is on ways in which the Tribal coalition employs the Strategic Prevention Framework to implement strategies in its communities. Because implementation of environmental initiatives at the community level is at the core of the prevention infrastructure in Maine, this final report focuses on the prevention strategies implemented under PFS II, including how they were carried out at the community level, which are more effective and why, and ways in which some strategies, including marijuana strategies, are adapted to meet the needs of local communities. Evaluation findings are organized into three sections: Process Results, which includes a section on infrastructure and strategy implementation and a section outlining Tribal HMP-specific results, Project Outcomes, which present accomplishments related to SEOW, quantitative outcomes on substance use, and Lessons Learned for the coming PFS 2015 grant. Process Results Healthy Maine Partnership Infrastructure & Strategy Implementation At each of the focus groups throughout the State, HMPs were asked to score each strategy on three domains: 1) Relevant: The strategy is meaningful and appropriate in the community. 2) Feasible: The strategy is possible to do easily or conveniently, given current resources and capacity. 3) Impact: The strategy has the ability to impact outcomes in the community to a noticeable and significant degree. HMPs scored each strategy on each of the three domains on a scale of 1 5 using the scales shown in Table 3. (See full Strategy Survey in Appendix B.) Table 3. Strategy Scores by Domain Relevant Not meaningful or appropriate. A little meaningful and slightly appropriate. Moderately meaningful and moderately appropriate. Mostly meaningful and mostly appropriate. Very meaningful and very appropriate. PFS II Final Evaluation Page 13

18 Feasible Not possible or convenient to implement. Slightly possible and somewhat convenient. Moderately possible and mostly convenient. Mostly possible and mostly convenient. Very possible and very convenient. Impact No impact on outcomes. Slight impact on outcomes. Moderate impact on outcomes. Substantial impact on outcomes. High impact on outcomes. HMPs were encouraged to respond to each strategy regardless of whether or not that strategy had been most recently implemented in Year 3 of the grant and regardless of whether or not PFS II funds exclusively were used to implement that grant, given that the majority of PFS II funds were used as enhancement dollars to other funding streams. Figure 1 on the following page shows how each strategy was ranked on each of the three domains: Relevant, Feasible, and Impact. The Responsible Beverage Serving (RBS) strategy (strategy number 1.1), Prescription Drug Misuse Awareness (5.2), and Safe Storage and Disposal (5.3) ranked highest on the Relevant domain, meaning that they were strategies that appeared to the respondents as very meaningful and very appropriate to communities. RBS (1.1), Marijuana Information Dissemination (4.4), Prescription Drug Misuse Awareness (5.2), and Safe Storage and Disposal (5.3) ranked highest on the Feasible domain, meaning that they were strategies that were viewed as very possible and very convenient to implement throughout communities. Regarding the Impact domain, RBS (1.1), Increasing Task Force Participation (2.1), and Safe Storage and Disposal (5.3) were ranked the highest, meaning that they were strategies that were perceived to have high impact on outcomes. On the other end of the spectrum, Card ME (1.2) ranked consistently lowest on all three domains, as did both SBIRT strategies (6.1 and 6.2), and Sticker Shock (1.3). PFS II Final Evaluation Page 14

19 Figure 1. Relevance, Feasibity & Impact by Strategy Relevant Feasible Impactful RELEVANT FEASIBLE IMPACT 1.1 RBS 1.2 Card ME 1.3 Sticker Shock 2.1 Task Force 2.2 Underage Drinking Enforcement 2.3 Alcohol Awareness 3.1 Table Talks, PWHLTM 3.2 Party Smarter, alcoholscreening.org 4.1 Marijuana Education 4.2 Marijuana Info Dissemination 5.1 TA for PMP 5.2 Rx Misuse Awareness 5.3 Safe Store & Disposal KEY 6.1 SBIRT Community Orgs 6.2 SBIRT Healthcare Providers PFS II Final Evaluation Page 15

20 Figure 2 shows how each strategy ranked in summation across each of the domains. There are clearly some strategies which are perceived to be more meaningful, possible, and impactful on health outcomes than others. Safe Storage and Disposal ranked highest across each of the three domains, followed by RBS training (1.1), Prescription Drug Misuse Awareness (5.2), and Task Force Participation (2.1). Retail strategies (1.2 & 1.3), SBIRT (6.1 & 6.2), and strategies targeting young adults (3.2) and parents of middle or high school aged students (3.1) ranked lowest on the combined domains. There were no strategies that ranked extremely low or wholly ineffective, indicating that in general HMPs consider PFS II strategies to be at least somewhat relevant, feasible, or impactful. In fact, as HMP staff described their reasoning behind their scores, many of them reported that each strategy had merit, but that some had a considerable barrier or challenge associated with implementation that warranted a lower score. For example, Sticker Shock had been a successful initiative within communities, but the resources needed to engage law enforcement and youth, coupled with the lack of data regarding impact on youth access to alcohol, ranked this strategy relatively low. Figure 2. Relevance, Feasibility and Impact by Strategy 5.3 Safe Store & Disposal 1.1 RBS 5.2 Rx Misuse Awareness 2.1 Task Force 2.2 Underage Drinking Enforcement 4.2 Marijuana Info Dissemination 4.1 Marijuana Education 2.3 Alcohol Awareness 3.1 Table Talks, PWHLTM 5.1 TA for PMP 3.2 Party Smarter, alcoholscreening.org 1.3 Sticker Shock 6.2 SBIRT Healthcare Providers 6.1 SBIRT Community Orgs 1.2 Card ME Relevant Feasible Impactful PFS II Final Evaluation Page 16

21 Lower-Ranking Strategies Most HMPs reported that retail strategies such as Card ME (1.2) and Sticker Shock (1.3) had reached saturation, meaning that most of the retailers in communities had either successfully been reached or were not interested in participating in Card ME or Sticker Shock initiatives. For many HMPs, retail strategies were more effective in earlier years of the PFS II grant but ceased to be effective in later years. HMPs consistently reported that retail strategies were resourceintensive and were difficult to sustain at the local level, especially retail strategies that relied on resources to pay law enforcement for participation. Another major barrier is the time and resources required of retail managers and staff to participate, with HMPs citing that most Mom & Pop stores do not have the needed resources, and the unlikeliness of engaging larger corporate stores. Many HMPs reported that there is a role for the community coalition in building and strengthening the link between community and healthcare systems for the purpose of substance abuse prevention. Both SBIRT strategies and the Technical Assistance for PMP strategy also ranked consistently low, and had similar barriers to implementation. Across the board, HMPs questioned whether coalition-based prevention providers are the right entity to target healthcare systems and healthcare providers. Many HMPs reported that coalitions do not have the skills, knowledge, or expertise needed to spearhead the linkage between community and clinical settings. In fact, the majority of HMPs reported not having strong partners in local healthcare settings; those that did report mild success in these strategies had successfully engaged internal champions within the healthcare system, had contracted with a healthcare professional to execute the work, or shared a healthcare system as a fiscal agent and therefore had inside access to clinical departments. However, many HMPs still indicated that these strategies were relevant, and that there is a role for the community coalition in building and strengthening the link between community and healthcare systems for the purpose of substance abuse prevention. A staff person from HMP who is committed to continue implementation of SBIRT strategies said: [It s] hard to get in providers doors. You might get some clinical staff who are very important partners but they say they need to talk to their provider it might take six months to schedule a training. Even if we offered them small steps to make it happen, they need an internal person to make it more likely to happen. They need ongoing support. I think the biggest challenge with SBIRT is coming up with a game plan and making it work. Also, being flexible and patient. We ve been very flexible with how we ve done it; persistent and flexible. Similar challenges were cited in discussion of the Technical Assistance for PMP strategy (5.1), which had an average ranking. HMPs enjoyed having a role in the PMP, but tended to believe they did not have the tools, knowledge, and resources needed to effectively assist healthcare PFS II Final Evaluation Page 17

22 providers on the PMP. For example, not having access to the PMP itself was a major barrier. As one HMP staff member put it: I think this strategy is really important and relevant and I wish we could figure out a better way for HMPs to work on it. It has done a lot to stop doctor-shopping and pain management, and it is very relevant for Maine prescription use. But, when I talk about the PMP, it s like, come look through this window. It s like heaven in there! I ll do my best to describe it to you. If you need help I can t help you. In addition to not having access to the PMP, another major barrier to providing Technical Assistance for PMP was the issue of not knowing the utilization of the PMP by prescribers in a coalition s catchment area. After mandated registration became effective statewide in March of 2014, the strategy as it was originally written (to increase registration of prescribers and dispensers to the PMP) was no longer applicable. Yet HMPs do not receive local data on PMP utilization or the extent to which high-risk prescribing occurs in the community, despite the fact that this information is crucial to understanding community needs regarding prescription drug availability. HMPs expressed a need for more access and technical assistance from the state regarding use of local PMP data to inform prevention efforts. Party Smarter also ranked lower as a successful strategy. Reasons for this varied, but included the fact that some communities simply do not have universities or colleges or other organizations serving that age group, so strong partnership was lacking. Some HMPs in college towns reported higher success and an increasingly strong partner in higher education administrations, but may have focused more on information dissemination, education, or awareness raising with college aged students rather than the specific Party Smarter campaign. Higher-Ranking Strategies Responsible Beverage Serving (1.1) ranked consistently high across all three domains and was the second-highest ranked strategy overall. HMPs reported success across all three years of the grant on this strategy. Coalitions were pleased with the trainer the state has used throughout the duration of the grant, and in response to his retirement made the decision to shift training responsibilities to HMP staff. Overall, RBS training was identified as a relevant strategy that was not resource-intensive and had a strong perceived impact in the community. All HMPs cited law enforcement as the strongest community partner in implementing prevention strategies. Some HMPs cited increasing success over the duration of the grant as staff from on premise entities returned for training year after year. In general, on-premise trainings were more successful than off-premise ones, due in part to the challenges engaging retail settings mentioned above. HMPs in coastal communities had success engaging seasonal staff of onpremise entities and perceived this to have a strong impact during seasons of high tourism. PFS II Final Evaluation Page 18

23 HMPs reported great success in implementation of increasing awareness of safe storage and disposal of prescription drugs (5.1) and increasing awareness of prescription drug misuse (5.2). There was high enthusiasm for the effectiveness of these strategies and the likelihood of sustainability beyond the PFS II grant. Law enforcement were cited as the strongest partner for all HMPs on these strategies, and some HMPs had used PFS II funds to engage additional partners such as funeral parlors, real estate agencies, and senior living facilities. For most communities, Drug Takeback events and Drug Takeback Boxes were funded at least in part with PFS II funds. HMPs reported higher community buy-in around the issue of prescription drug misuse and strategies related to this substance were perceived to have a higher impact on the community. HMPs reported using the Up & Away materials produced by the Centers for Disease Control and Prevention 2 as well as producing their own materials. These prevention strategies may be more action oriented than other strategies in that materials provide direct guidance, and takeback initiatives are convenient and accessible, which may have contributed to success. One of the most salient successes of the PFS II grant is the increased participation in local Task Forces and efforts to curb underage drinking. Finally, one of the most salient successes of the PFS II grant is the increased participation in local Task Forces and efforts to curb underage drinking. While a few HMPs are currently in restructuring or reimagining their local Task Forces, this work is largely understood to be relevant, feasible, and having an impact on underage drinking rates. HMPs used PFS II funds to leverage existing momentum of the Task Forces by enhancing current strategies and strengthening efforts in the community. Some used PFS II dollars to try something new within their Task Force, such as focus on other substances, or develop a new community initiative. For example, one HMP staffer stated: We have a 27-member task force and we got together and came up with the issues in the community, identified top priorities in the community, and then formed sub-committees to come up with solutions. We used our PFS II money to focus on education. Average-Ranking Strategies Strategies with average scores on the Strategy Survey are also worth discussing. Marijuana strategies, both information dissemination and education (4.1 & 4.2), continue to have high relevance in communities and are relatively feasible. However, HMPs continue to face a multitude of challenges implementing marijuana strategies, including: 1) reaching saturation with information dissemination; 2) difficulty engaging crucial partners such as parents or schools; 3) persistent social norms and perception of harm of use; 4) political climate around 2 Available at PFS II Final Evaluation Page 19

24 legalization of marijuana and pro-marijuana language and momentum; and 5) limited scientific evidence regarding marijuana and health outcomes. HMPs had constructive feedback on the marijuana materials that were developed by the Marijuana Workgroup and approved by SAMHS. Most HMPs distributed SAMHS materials and developed their own materials using core information approved by SAMHS. Many HMPs also used materials from other national or state resources, such as National Institute on Drug Abuse (NIDA) or Community Anti-Drug Coalitions of America (CADCA). All HMPs expressed a need for more materials. In particular, HMPs need more information to give to schools on marijuana, especially other forms of marijuana such as edibles and oils. Many HMPs said that schools have been asking them for more resources. As one HMP staff member stated: We ve had a lot schools approach us about how to talk to kids about it. It s coming back again. They have a problem that they don t want to acknowledge. They re seeing edibles, junior high school kids bringing marijuana into school they re ready for more information. Across all strategies, HMPs believed strongly that they would benefit from a central clearinghouse to share materials between HMPs and the state. Additional requested marijuana resources include: 1) a toolkit for drafting policy regarding local marijuana ordinances; 2) more information regarding what is legal and where; 3) tips on language to use against counter-arguments; and 4) more materials that reach youth and parents in a positive, informative, and eyecatching way. Across all strategies related to information dissemination, awareness building, and education, HMPs believed strongly that they would benefit from a central clearinghouse to share materials between HMPs and the state. Though many were already sharing certain materials, there was an obvious need to streamline the sharing process, especially because material development can be very resource intensive. Another strategy that had average ranking across the domains was the Table Talks and Parents Who Host Lose the Most campaign (3.1) and other underage drinking prevention initiatives such as underage drinking enforcement and alcohol awareness (2.2 & 2.3). In general, parents were not strong partners in this work, and many HMPs reported a need for better materials to engage this group. Some success on these initiatives was met when HMPs capitalized on parent events that were already happening; for example orientation night for parents, mandated classes for parents with children referred to substance abuse interventions such as Student Intervention Reintegration Program (SIRP), and other school or community events. PFS II Final Evaluation Page 20

25 Tribal HMP An important evaluation focus of the PFS II grant was the way in which the Tribal HMP conducted work related to the Strategic Prevention Framework and its success in adapting and implementing strategies. In 2011, the four federally recognized Native American tribes in Maine formed the ninth public health district, the Wabanaki Public Health District. (The Passamaquoddy, Penobscot, Maliseet and Micmac tribes are collectively known as the Wabanaki people or People of the Dawn. ) The four tribes, consisting of five Tribal communities, maintain their own governments, cultural centers, health centers, and schools, and they manage their own land and resources. People belonging to these tribes reside in towns and cities across the state or on Tribal lands. If [tribal youth] don t see a piece of who they are, they aren t interested. It needs to be adapted to their culture. Tribal HMP Staff Wabanaki Public Health District staff are based in Houlton at the Maliseet Health and Wellness Center, which is centrally located to all five of the communities it serves. The district staff work closely with each of the five Tribal health centers listed below, and the five Health Directors serve as an advisory committee to the whole district: Micmac Service Unit, Presque Isle Aroostook County Houlton Band of Maliseet Health Department, Littleton Aroostook County Indian Township Health Center, Indian Township Washington County Penobscot Nation Health Department, Indian Island Penobscot County Pleasant Point Health Center, Sipayik Washington County At the end of Year Two, the PFS II contract with the Tribal Public Health District shifted from the Houlton Band of Maliseet Health Department to the Micmac Service Unit in Presque Isle for reasons related to organizational capacity and staffing. Despite this shift, there are valuable insights from Year Three that capture the overall progress that the Tribal community has made in the implementation of substance abuse prevention strategies. In the Year Two Evaluation Report, findings indicated that the Wabanaki HMP is currently implementing prevention strategies on par with other HMPs throughout the state, indicating significant successes and considerable local capacity. This continued to be true in Year Three; in fact, the Tribal HMP enjoyed unique successes that enabled progress in areas where other HMPs struggled. Culturally-specific Strategies: In the three years of the grant, the Tribal HMP successfully adapted PFS II strategies to meet the needs of the community. For one, SAMHS-approved materials and other resources were adapted to be culturally specific to tribal youth; for example, pictures of local youth were used on materials, and traditional references were incorporated into approved messages. Although this slowed down in Year Three, Tribal PFS II Final Evaluation Page 21

26 HMP staff stressed the importance of doing it at for Tribal communities: If they don t see a piece of who they are, they aren t interested. It needs to be adapted to their culture. Tribal HMP staff are continuing to work with SAMHS to develop culturally-specific materials and have plans to adapt the safe storage for prescription drugs Up & Away materials to be tribally and culturally specific. Youth Engagement & Community Partnerships: More than any other HMP, the Tribal HMP successfully engaged middle- and high school-aged youth in substance abuse prevention strategies. One Tribal HMP staff member said: Our youth are more respectful. Our youth are more involved in everything that happens in the community. The children are always involved. Kids don t stay home in the native communities. They are much more attentive when we re speaking to them. They tend to have a higher level of respect. I think it might be cultural no matter what goes on, you have all the kids running around, everyone knows everybody. You can tell someone else s kid to cut it out [if they misbehave]. It takes a village. the community has grabbed ahold of wanting to get healthy all the way around. They were ready. Tribal HMP Staff As with any community, establishing strong and dedicated partners to engage in substance abuse prevention strategies is crucial, and the Tribal HMP did so with great success. Other HMPs reported difficulty engaging community members in the work, but the Tribal HMP reported differently, citing a shift in community members commitment to action regarding health: the community has grabbed ahold of wanting to get healthy all the way around. They were ready. This cultural shift has also enabled staff to engage with key community stakeholders. Law enforcement, Wellness Court (a tribalspecific substance use intervention), universities, Tribal Counsel, behavioral health providers, elders, and schools were all strong partners. Like other HMPs however, Tribal staff also noted difficulty engaging parents of youth and the year old population. Community Data: The Tribal HMP has undertaken efforts to enhance the availability of community-level data. Health directors from all communities meet quarterly to review and plan around health data on community members. This group was also working on a Tribalspecific youth health assessment and an update to the adult health assessment. Additionally, a survey was developed to administer to healthcare providers to get a sense of PMP use, use frequency, and training needs. While nearly half of the HMPs reported this as a major data gap, the Tribal HMP was the only group who created a survey tool to fill that need. PFS II Final Evaluation Page 22

27 PROJECT OUTCOMES State Epidemiology Outcomes Workgroup Outcomes monitoring for the PFS II grant was conducted through the SEOW. The SEOW is charged with not only monitoring outcomes for PFS II, but also with making data available for substance abuse prevention planning for a range of audiences. Additional SAMHSA funds were made available in 2013 in the form of a supplemental grant, which funded the development of an online data dashboard. Additionally, the SEOW provides epidemiological profile reports by each Public Health District that are used in data-driven decision-making processes at the community level. The following accomplishments were achieved by the SEOW during the project period: Data Dashboard & Epidemiological Profiles: The SEOW successfully launched a webbased data platform offering interactive outcomes monitoring using a variety of data sources and presenting graphs and narrative on data related to the State, Public Health Districts, and counties. Epidemiological profiles highlighting outcomes, intervening, and consequence indicators are released by geographic area and subpopulation, such as LGBT youth, on an annual basis. These profiles were used by HMPs as a tool for informing strategy selection and for monitoring progress made in local communities. Non-Duplication of Efforts: The SEOW merged with the Community Epidemiology Surveillance Network to facilitate broader access to data and to reduce potential duplication. Effective Decision-Making Model: To keep momentum going, the steering group sets the direction and vision for the SEOW, and relies on the expertise of its broader membership to refine project goals and generate new ideas. Promotion and Publicity: Maine s epidemiological profile has been widely cited in media across the State. For many SEOW members, this publicity has been key in justifying continued collection of the data in their localities. Outcomes Monitoring Table 4 describes the local and national data sources and corresponding PFS II indicators. Table 4. Data Sources and Indicators Name Description Indicator Maine Integrated Youth Health Survey (MIYHS) Collects information on student substance use and risk factors related to substance use, as well as consequences, perceptions and social risk factors related to substances. Alcohol, marijuana, and prescription drug use among 7 th -8 th graders and 9 th -12 th graders. PFS II Final Evaluation Page 23

28 National Survey of Drug Use and Health (NSDUH) Maine Bureau of Highway Safety (MBHS), Maine Department of Transportation (MDOT) Provides state-level data on the use of tobacco, alcohol, illicit drugs (including non-medical use of prescription drugs), and mental health. NSDUH is sponsored by SAMHSA. Responsible for tracking all fatalities that occur on Maine's highways and reporting this information through the Fatal Analysis Reporting System. The data represented provides information on highway crashes and fatalities. Alcohol use among youth aged Marijuana and misuse of painkillers among youth aged Annual alcohol and drug related car crashes among youth aged Table 5 shows baseline (2011) data, midway data (2012, 2013, and/or 2014 or ), and final data (2015) as well as target (2015) rates for each of the grant objectives by substance. While the results cannot be attributed to PFS II alone, it appears that the combination of PFS II and other prevention funding contributed to changes in outcomes over the three-year grant period. In this table, targets highlighted in green are PFS II targets that were met by the end of the grant. Targets highlighted in red were not met during the project period. Across all substances, target rates were reached for 7th and 8th graders (middle school) and 9th 12th graders (high school): rates for past 30-day alcohol, marijuana, and prescription drug use decreased during the project period to below the target rate. Targets were not met for youth aged for past 30-day marijuana and alcohol use, but were met for past 30-day prescription drug misuse. The target number for annual number of alcohol and/or drug related car crashes was not reached; in fact, the annual number increased during the project period. PFS II Final Evaluation Page 24

29 Consequence Indicator Consumption Indicators Table 5. PFS II Outcomes Alcohol Use Prescription Drug Misuse Marijuana Use Car Crashes AGE GROUP SOURCE INDICATOR YEAR(S) PERCENTAGE (Baseline) % 7 th 8 th Grade % % MIYHS (Target) % (Baseline) % 9th 12th Grade Past 30 Day Alcohol % Use (Any) % (Target) % Year Olds 7 th 8 th Grade 9th 12th Grade Year Olds 7 th 8 th Grade 9th 12th Grade Year Olds Year Olds NSDUH MIYHS NSDUH MIYHS NSDUH MDOT Past 30 Day Misuse of Prescription Drugs Past Year Misuse of Pain Relievers Past 30 Day Use of Marijuana Number of Annual Alcohol and/or Drug- Related Car Crashes (Baseline) % % % (Target) % (Baseline) % % % (Target) % (Baseline) % % % (Target) % (Baseline) % % % (Target) % (Baseline) % % % (Target) % (Baseline) % % % (Target) % (Baseline) % % % (Target) % (Baseline) (Target) PFS II Final Evaluation Page 25

30 Figure 3 shows the rate of past 30-day alcohol use over the project period by age group. Again, targets met are in green, targets not met are highlighted in red. Targets were met for middleand high school-aged youth but not for youth aged as measured by NSDUH. In fact, the age group actually saw a slight increase by one percentage point of past 30-day alcohol use. Figure 3. Past 30-day Alcohol Use, by Age 30% 25% 28.0% 26.6% 26.0% 23.8% 24.9% 23.9% 23.9% 22.7% 20% 15% 10% 5% 6.3% 6.0% 4.7% 3.9% Target 0% Middle School (MIYHS) High School (MIHYS) 12 to 20 Year Olds (NSDUH) Figure 4 shows the rate of prescription drug use over the project period by age group. Prescription drug use target rates were met for all age groups. Middle- and high school-aged youth rates decreased by 1.0 and 2.3 percentage points respectively, and for youth aged 18 25, the rate decreased by 3.8 percentage points. Again, it is important to note that because the data source for year olds is NSDUH, the indicator for this outcome is slightly different. For this age group, the indicator captures past year use of non-prescribed pain relievers. PFS II Final Evaluation Page 26

31 12% 10% Figure 4. Prescription Drug/ Pain Reliever Misuse, by Age 10.9% 10.4% 8.9% 8% 6% 4% 2% 3.2% 2.6% 2.2% 3.0% 7.1% 5.6% 4.8% 6.8% 7.1% Target 0% Middle School (MIYHS) High School (MIYHS) Year Olds (NSDUH) Figure 5 shows the rate of past 30-day marijuana use over the project period by age group. Targets were met for middle- and high school-aged youth but not for year olds. In fact, marijuana use for the year old population actually increased over the project period by 5.7 percentage points. 30% 25% 20% 15% 10% 5% Figure 5. Past 30-day Marijuana Use, by Age 28.4% 24.7% 25.3% 22.1% 21.6% 22.7% 21.0% 19.6% 4.6% 4.4% 3.8% 4.4% Target 0% Middle School (MIYHS) High School (MIYHS) Year Olds (NSDUH) PFS II Final Evaluation Page 27

32 Finally, Figure 6 shows the annual number of car crashes related to alcohol and/or drugs in Maine. This number increased during the project period and the target number was not met. 400 Figure 6. Alcohol and/or Drug-Related Car Crashes Among Year Olds Target PFS II Final Evaluation Page 28

33 LESSONS LEARNED Successes A major success of the PFS II grant is the impact that the project and other statewide prevention projects have had on substance use among middle and high school aged youth in Maine. Because SAMHS did not make available any other funding sources to target marijuana use among youth other than PFS II funds, the decrease in marijuana use can be attributed more to the work of PFS II and the Marijuana Workgroup than can be for outcomes for alcohol and prescription drugs. For all substances, the project contributed to successful decreases in use by directly targeting risk factors that are associated with use, engaging and educating partners who have impact on the target population, and implementing programs that reduce youth access to substances. Targets for this middle and high school aged youth were not only met but were exceeded, with some rates dropping by more than a third. Healthy Maine Partnerships reported that PFS II funds allowed them to accomplish things that would have otherwise not been possible under other funding sources through SAMHS. In many cases, PFS II dollars allowed HMP staff to strengthen existing partnerships or to forge new relationships. PFS II dollars were used to capitalize on existing community events to provide educational opportunities to the public. Many HMPs held 5k events or health fairs using PFS II dollars. Take Back initiatives were supported in part with Drug Free Community funding and strong foundations were built for the ongoing sustainability of this initiative. Some HMPs used PFS II dollars to create new and locally relevant materials based on SAMHS approved messages. PFS II dollars were also used to cover expenses for projects such as stipends for law enforcement for compliance checks, partnering with champions in healthcare settings, and supporting Task Force members to focus on other substances in addition to alcohol. creating a sub-committee on the Task Force. In the grander scheme of the state of Maine s Strategic Prevention Framework process, the PFS II project as a whole continued the data-driven decision making, capacity building, and implementation principles of the SPF. The strategies implemented under PFS II aligned with distinctive features of the SPF process. 3 1) Outcomes-based Prevention: The project had clear outcome and consequence goals. Each strategy directly targeted at least one outcome. 2) Population-level Change: HMPs implemented a range of strategies and programs and identified risk and protective indicators that are inter-related and influence 3 PFS II Final Evaluation Page 29

34 substance use. Strategies were not designed to reach small groups nor were individual program outcomes a focus. 3) Prevention Across the Lifespan: In addition to middle school aged and high school aged youth, the project also focused on youth ages 18 25, which is an often overlooked yet very critical age group in the prevention of substance use. 4) Data-driven Decision Making: HMPs rely heavily on SEOW epidemiological profiles to assess their community s needs and progress made on substance use indicators. HMPs were able to use SEOW data to understand the severity of a particular problem in their communities and choose prevention strategies accordingly. For example, communities that have higher rates of marijuana use and lower rates of perception of harm among youth focused more heavily on messaging the harms of marijuana to young people. All strategies implemented under PFS II were evidencebased. Although turnover was a major barrier to project management and coordination at the state level, HMPs reported great satisfaction receiving support, resources, and training from the SAMHS project team. The SAMHS team is structured so that each HMP has one SAMHS project officer, but HMPs reported that all team members were always available, and if a team member had more expertise on a subject, they would field a request for technical assistance. As an HMP staff member put it, I feel strongly that the SAMHS staff are really collaborative in working with us I find them to be really helpful. The current SAMHS team rocks! All of the staff are excellent. The SAMHS team itself was also satisfied with the way in which it offered and provided technical assistance, and felt that it had provided the right tools, knowledge, and professional development opportunities necessary for HMP staff to effectively do prevention work. More specifically, the team began an initiative to provide prevention credentialing to substance abuse prevention providers (SAPPs), a process that had been a goal of the department for years. The SAMHS team was able to build the buy-in and support needed to set the process in motion. The team also has a valuable perspective on its relationship with community providers and the importance of empowering communities to be confident and be experts in their own work. As one staff person said, We re trying to have a paradigm shift where we aren t the experts and we are putting it back into their hands. We aren t telling them what to do, we can help facilitate and help them get going. For the PFS 2015 grant, communities are in more control over what and how to implement. Rather than selecting from a limited menu of strategies, sub-grantees conduct data-driven assessment processes to determine their communities gaps, needs, and strengths. This assessment informs an open selection process for evidence-based programs to implement. Subgrantees now choose strategies that fit under certain types of intervention, such as education PFS II Final Evaluation Page 30

35 or information dissemination, and that have the potential to target multiple substances at once. HMPs may now choose any strategy they would like, as long as it is evidence-based and can clearly have an impact on the project goals. This is a departure from the PFS II approach which offered minimal flexibility for HMPs to forgo a strategy that may not have been relevant in a community. Communities and SAMHS itself has expressed early satisfaction with this process, citing it as more egalitarian, more relevant to what communities need, and more likely to succeed. A SAMHS staff person stated, In the past we have been so prescriptive now it s more exploratory and open, and they are asking more questions about whether they can do things. They re trying things that they didn t try before. Finally, a major success of the PFS II grant was the Marijuana Workgroup. Although turnover in leadership in the second year slowed progress, the group benefited from engaged and experienced members and was strong by the end of the third year. The group continued to rely on scientific evidence, examples from other states, and strategic partnerships to move ahead on developing marijuana prevention messaging. In particular, the group was critical in providing to communities the language needed to discuss marijuana with youth, parents, teachers, and other stakeholders. Many HMPs repeated the same strategy of focusing on brain development when discussing marijuana. The Workgroup offered multiple PowerPoint slide banks, rack cards and brochures, radio ads, and key messages for youth that were disseminated to communities. These materials continue to fill a major gap in the prevention of marijuana use. Challenges Outcomes for young adults (aged 18 25) were not as positive as outcomes for younger youth. Past 30-day alcohol and past 30-day marijuana use increased for the and year old populations, respectively. HMPs reported that young adults are a challenging population to reach, and many communities lack strong partners who regularly interact with this age group. Additionally, many HMPs reported that materials and messages often were not targeted towards this age group, and that more creative engagement and social media strategies were needed to have an impact. Across all years of the grant and across the state, marijuana prevention work continues to be a major challenge. Marijuana prevention comes with confusing medical marijuana policies, pervasive social and cultural norms regarding use and the perception of harm, a strong and stalwart counter-argument that has been minimally engaged, and rapidly changing contexts regarding use and forms of the substance. HMPs often find themselves at the difficult intersection of legalities, health, changing local and state policy, safety, and harm reduction, and had one overarching request: more information. HMPs require specific language, tools, and packages with which to discuss marijuana, from policy templates for local recreational use, to information on edibles and emerging forms of use such as vaping with oils. PFS II Final Evaluation Page 31

36 At the state level, the SAMHS team recognizes this as an issue with another point to add: People are often very confused about what they can do: can they lobby, advocate, etc.? Technical assistance on that has been lacking. Another emerging challenge has been the misunderstanding of prevention of alcohol, prescription drugs, and marijuana in light of the heroin epidemic throughout the state. Both HMPs and state staff mentioned this issue and expressed the need for SAMHS to provide resources and language on the importance of primary prevention in light of serious drug issues in the State. In other words, HMPs need resources to be able to justify to community stakeholders, local decision-makers, and the public why they are focusing on alcohol or marijuana as a way to curb substance use initiation rather than heroin use, which may come much later in a person s life after initial alcohol or marijuana use. The issue of reaching high-priority or high-risk groups also was mentioned as a challenge, especially in light of many strategies reaching implementation saturation. For some strategies, general public awareness or education could not go any further, and strategies are needed to focus on high-risk populations in order to move the needle on the general population that much further. Many communities identified lesbian, gay, bisexual, or transgendered (LGBT) youth as a group needing particular focus; others identified lowerincome, non-english speaking, low literacy, and homeless youth as groups experiencing higher burdens of risk factors. Materials were identified as both a success and a challenge. While HMPs were grateful for their resources, many felt that some materials endorsed or produced by SAMHS were out of date and/or inappropriate for the target population. HMPs are eager to use catchy, aesthetically pleasing materials that are very simple and can be used for multiple target populations. Additionally, a central clearinghouse of materials needs to be developed so that communities can share materials either discovered through other organizations or created locally. Even SAMHS staff recognized this as a goal that was not reached during the grant period, despite how beneficial cross pollination of resources would be to communities. PFS II Final Evaluation Page 32

37 CONCLUSIONS AND RECOMMENDATIONS The PFS II grant implemented in Maine overall produced a successful and effective program. The project had a positive impact on measurable outcomes and contributed to an overall reduction in youth substance use and risk. The project helped to build the public health infrastructure at the community level: project funds were used to strengthen relationships and build local partners, increase the use of data-driven decision making in planning and implementation, and enabled HMP staff to work on efforts that could not be addressed under other funding streams. The project also enabled activities at the state level, supporting the Marijuana Workgroup and material development, statewide professional development for substance abuse prevention staff, and ongoing technical assistance and resources to communities. Additionally, the project contributed overall to the State s Strategic Prevention Framework trajectory and set the stage for future prevention programming. In general, SAMHS should continue to build upon successes and lessons learned as the transition from the PFS II grant to the PFS 2015 grant plays out. Already SAMHS is implementing changes to practice and policy that are reflective of lessons learned under PFS II, including conducting community needs assessments, creating workgroups that target high-risk groups, and finding the balance between direction, support, and autonomy while allowing sub-grantees freedom to choose what and how and when to implement prevention strategies. Additionally, some infrastructure changes have begun to happen in the State that have already impacted the PFS 2015 grant, such as the shift of the Prevention Team from SAMHS to the Maine Center for Disease Control. In closing, there are a handful of recommendations to carry forth into the new grant that will additionally strengthen the momentum that SAMHS and its community partners are already enjoying: 1) Sustain momentum with the Marijuana Workgroup. In the first year of the PFS 2015 grant, SAMHS should reassess membership and identify additional champions to serve on the Workgroup or subcommittees while continuing to strike the right balance of state and community leadership. The Workgroup should also consider having early discussions on sustainability beyond the PFS 2015 grant. It may also be useful to develop a guiding document and goals for the Workgroup to keep activities action-oriented and successful. 2) Continue to provide professional development and training opportunities, technical assistance, and resources. With the new grant, communities have more control over strategy selection and implementation, and with that come added responsibility and additional training needs. For example, training on using data to inform program planning, resources on adaptation of evidence-based practices, or language on how to discuss the role of primary prevention might be useful resources for substance use prevention providers. 3) Develop a central clearinghouse of materials. SAMHS should identify a simple and straightforward method for community-level staff to share substance use prevention materials. This forum should be web-based and should allow SAMHS to review and PFS II Final Evaluation Page 33

38 approve messaging if necessary. The forum could be organized by substance or theme and could be sorted in multiple ways. The system could be set up so that only subgrantees had access. 4) Continue environmental strategies while targeting higher-risk groups. While this recommendation is already underway with the creation of the Veterans Workgroup and LGBT Workgroup, as well as additional requirements to target health disparities under the PFS 2015 grant, there are additional ways that SAMHS could target high-priority populations. For example, strategies that target high-risk young adults need to go beyond information dissemination and education towards creative positive environments for high risk young adults that discourage substance use initiation and abuse. 5) Relationship building and capacity development are major facilitators of success. While strategies under PFS 2015 already support ancillary activities that contribute to measurable success, it is important to foster a prevention environment where this principle is upheld. SAMHS can do this by facilitating collaboration between community sub-grantees, putting as much emphasis on process as on measurable benchmarks, and supporting communities to identify creative champions and partners in prevention work. PFS II Final Evaluation Page 34

39 APPENDIX A: Focus Group Protocols PFS II Focus Group Protocol Public Health District Participants & HMP Names Thank you for agreeing to participate in this group today. As you know, the purpose of this focus group is to gather information about the Partnerships for Success (PFS II) grant that was implemented from 2012 to Everyone here was a recipient of PFS II funds and implemented strategies related to preventing alcohol use, prescription drug use, and marijuana use. Our questions are about the effectiveness of strategies, partners that were engaged, your experience engaging with the public, and general lessons learned about the grant. Your answers should reflect on the grant in its entirety. Feel free to answer openly and honestly. There are no right or wrong answers. Information that we gather will appear in aggregate form in our report, and your answers will never be linked to your identity. Any questions before we begin? SECTION 1: Introductions (5 minutes) SECTION II: Strategies (40 minutes) Thank you filling out the Strategy Survey ahead of time. The first part of the focus group will be a discussion of PFS II strategies, specifically which were effective and which strategies may have been more challenging to implement. Please use your Survey answers to guide this discussion. We will collect the Strategy Surveys from you today. The first questions are about strategies related to underage drinking. 1. Thinking about the strategies related to underage drinking, which ones ranked the highest on the three criteria of relevance, feasibility, and ability to have an impact? [Ask participants to be specific and identify strategies by name.] a. Why? What made them more effective than others? b. What helped these strategies to be more effective? (For example, concurrent initiatives targeting the same issue, support from law enforcement, etc.) 2. Which strategies ranked the lowest on relevance, feasibility and ability to have an impact? a. For strategies with low responses, what drives your answer? Are they still worthwhile? b. What, if anything, could anything be done to address this? PFS II Final Evaluation Page 35

40 3. Which strategies had mixed responses? (e.g., high on relevance but low on feasibility.) a. For strategies with mixed responses, are they are still worthwhile? What would have to change to make them rank higher? The next questions are about strategies related to prescription drug use. 4. Thinking about the strategies related to prescription drug use, which strategies ranked the highest on relevance, feasibility, and ability to have an impact? [Ask participants to be specific and identify strategies by name.] a. Why? What made them more effective than others? b. What facilitated these strategies to be more effective? 5. Which strategies ranked the lowest on relevance, feasibility and ability to have an impact? a. For strategies with low responses, what drives your answer? Are they still worthwhile? b. What, if anything, could anything be done to address this? 6. Which strategies had mixed responses? (e.g., high on relevance but low on feasibility.) a. For strategies with mixed responses, are they are still worthwhile? What would have to change to make them rank higher? The next questions are about strategies related to marijuana use. 7. Thinking about the strategies related to marijuana use, which strategies ranked the highest on relevance, feasibility, ability to have an impact? [Ask participants to be specific and identify strategies by name.] a. Why? What made them more effective than others? b. What facilitated these strategies to be more effective 8. Which strategies ranked the lowest on relevance, feasibility and ability to have an impact? a. For strategies with low responses, what drives your answer? Are they still worthwhile? b. What, if anything, could anything be done to address this? 9. Which strategies had mixed responses? (e.g., high on relevance but low on feasibility.) a. For strategies with mixed responses, are they are still worthwhile? What would have to change to make them rank higher? Tribal PHD Only: To what extent did the Tribal HMP staff modify SAMHS strategies to be culturally appropriate to meet the unique cultural needs of the tribal population? o o o If modifications were necessary, please describe them. How were modified strategies effective or not-effective? How were you supported by SAMHS to make these modifications? PFS II Final Evaluation Page 36

41 SECTION III: Engaging Partners and the Community (25 minutes) The next questions are about engaging partners and the public to collaborate on prevention strategies. 10. Who were your strongest community partners in implementing PFS II strategies? a. Why? Is there something unique about your community that facilitated a stronger relationship? b. Which partners were more difficult to engage and collaborate with? Why? 11. What strategies did you use to engage the public? (i.e., parents, college students, working adults, employers.) a. Were you successful? b. Were some groups easier to engage than others? Why? 12. Did success or challenges engaging partners and the community vary by substance? If yes, how and why? (i.e., was it easier to engage partners around prescription drug use than marijuana?) SECTION IV: Communication and Resources (20 minutes) 13. Communication strategies were a major component of the PFS II grant. What was your process for developing communication/messaging? a. Did you develop your own messages or did you use SAMHS produced messages or both? b. Which messages do you think had the most impact? 14. Did you receive resources or technical assistance during the grant? a. Was accessing resources or TA a strength of the grant? b. What additional resources or TA were needed but not met? c. What would you want to see for future prevention grants? 15. SAMHS held multiple trainings throughout the three years. Which were the most useful? Which could have been left out? SECTION V: Wrap-Up and Lessons Learned (20 minutes) 16. What was the greatest success achieved through PFS II funds in your community? a. Did PFS II funds allow your coalition to do something it otherwise couldn t do? If yes, what? 17. How will the past three years experiences inform or support future substance abuse prevention grants? a. Which lessons learned are the big takeaways from the PFS II grant that can inform the new round of funding? b. Substance abuse prevention in Maine more generally? PFS II Final Evaluation Page 37

42 CONCLUSION: (10 minutes) 18. Is there anything else you would like to add? Thank you for participating in this focus group today! If you think of anything else, please feel free to contact us. PFS II Final Evaluation Page 38

43 APPENDIX B: PFS II Strategy Survey PARTNERSHIPS FOR SUCCESS II STRATEGY SURVEY HMP Name: Please fill out and bring to the HZA focus group scheduled for your public health district. Please work with other staff at your HMP to rank each of the following PFS II strategies on whether they are relevant, feasible, and have the ability to make an impact in your communities. If possible, think about the last three years of PFS II work as you answer these questions. Please feel free to respond to a strategy even if funds other than PFS II funds were used to implement. The following definitions will help you to answer these questions. Relevant: The strategy is meaningful and appropriate for your community. 1 Not meaningful or appropriate. 2 A little meaningful and slightly appropriate. 3 Moderately meaningful and moderately appropriate. 4 Mostly meaningful and mostly appropriate. 5 Very meaningful and very appropriate. Feasible: Possible to do easily or conveniently, given your current resources and capacity. 1 Not possible or convenient to implement. 2 Slightly possible and somewhat convenient. 3 Moderately possible and mostly convenient. 4 Mostly possible and mostly convenient. 5 Very possible and very convenient. Impact: Has the ability to impact outcomes in your community to a noticeable or significant degree. 1 No impact on outcomes. 2 Slight impact on outcomes. 3 Moderate impact on outcomes. 4 Substantial impact on outcomes. 5 High impact on outcomes. PFS II Final Evaluation Page 39

44 Circle the number that corresponds to the relevance, feasibility, and impact of each strategy. Objective 1. Engage liquor licensees to participate in two or more strategies that lead to responsible retailing as defined by SAMHS guidelines. SA 1.1 Collaborate with BABLO to ensure availability and utilization of State-Certified Responsible Beverage Server / Service (RBS) trainings in the service area. Least Most Relevant Feasible Impact SA 1.2 Orient liquor licensees to the Card ME program and its materials. Relevant Feasible Impact SA 1.3 Engage retailers to participate in the Sticker Shock Program. Relevant Feasible Impact OBJECTIVE 2: Engage a local area Underage Drinking Task Force to participate in two or more strategies that lead to the increased perception among youth that they would be caught using alcohol. SA 2.1 Work with key community stakeholders including but not limited to, law enforcement, schools, and parents, to ensure participation on an active HMP led, community level Task Force. Least Most Relevant Feasible Impact SA 2.2 Collaborate with law enforcement and liquor licensees to increase the enforcement of underage drinking laws. Relevant Feasible Impact PFS II Final Evaluation Page 40

45 SA 2.3 Work with law enforcement and community stakeholders to increase public awareness around enforcement and youth access to alcohol. Relevant Feasible Impact PFS II Final Evaluation Page 41

46 OBJECTIVE 3: Engage a minimum of x% of partners (e.g., schools, hospitals, provider offices, service agencies, civic organizations) that promote and disseminate information about the risk of underage alcohol use and high risk drinking in young adults. SA 3.1 Engage businesses and community organizations (i.e., libraries, schools, healthcare facilities) to change adult and parental knowledge/attitude/norms related to underage alcohol use through the implementation of programs such as Table Talks, Parents Who Host Lose the Most, parent media materials, and other materials approved by SAMHS. Least Most Relevant Feasible Impact SA 3.2 Engage higher education, businesses, and community organizations that interact with young adults (including but not limited to bars, restaurants, employers) to promote and disseminate the SAMHS Party Smarter campaign or other SAMHS approved alcohol prevention materials and resources. Relevant Feasible Impact OBJECTIVE 4: Increase communications to the public about the risk and harm of marijuana use. SA 4.1: Using SAMHS-developed training materials, provide educational opportunities for residents including, but not limited to parents, teachers, counselors, and community organizations, about the social, legal, health, and community impacts of marijuana use. Least Most Relevant Feasible Impact SA 4.2: Disseminate information and/or materials to residents including, but not limited to, parents, teachers, counselors, and community organizations about the social, legal, health, and community impacts of marijuana use. Relevant Feasible Impact PFS II Final Evaluation Page 42

47 OBJECTIVE 5: Increase the implementation of prevention strategies to enhance community engagement in addressing Rx abuse. SA 5.1 Provide technical assistance to prescribers and dispensers to increase active participation in the Prescription Monitoring Program. Least Most Relevant Feasible Impact SA Increase awareness about the dangers of prescription drug misuse. Relevant Feasible Impact SA Increase awareness about safe storage and disposal of prescription medications. Relevant Feasible Impact OBJECTIVE: Increase the number of community organizations (e.g. worksites, service agencies, civic organizations) that utilize screening and brief intervention tools as a matter of practice with a special emphasis on health care providers. SA 6.1 Educate health care providers about how to implement screening, brief intervention, and referral for assessment for individuals, and the correct billing codes to use. Least Most Relevant Feasible Impact SA 6.2 Educate community organizations to promote the use of screening and brief intervention tools to address individuals with increased risk for substance abuse. Relevant Feasible Impact PFS II Final Evaluation Page 43

48 APPENDIX C: Map of Local HMPs PFS II Final Evaluation Page 44