Program Evaluation for Prevention: Strategic Prevention Framework State Incentive Grant

Transcription

1 Program Evaluation for Prevention: Strategic Prevention Framework State Incentive Grant Annual Report FY2016 October 2016 U.S. Department of Health and Human Services Substance Abuse and Mental Health Services Administration Center for Substance Abuse Prevention SAMHSA logo here after clearance HHS logo here after clearance

2 Acknowledgments This report was prepared for the Substance Abuse and Mental Health Services Administration (SAMHSA) by Lori-Ann Palen, Phillip Graham, and Elvira Elek, RTI International, and Gillian Leichtling, Michelle Hendricks, and Erin Stack, RMC Corporation, under contract number HHSS I/HHSS T, with SAMHSA, U.S. Department of Health and Human Services (HHS). Thomas Clarke served as the Government Project Officer. Disclaimer The views, opinions, and content of this publication are those of the authors and do not necessarily reflect the views, opinions, or policies of SAMHSA or HHS. Public Domain Notice All material appearing in this report is in the public domain and may be reproduced or copied without permission from SAMHSA. Citation of the source is appreciated. However, this publication may not be reproduced or distributed for a fee without the specific, written authorization of the Office of Communications, SAMHSA, HHS. Electronic Access and Printed Copies This publication may be downloaded or ordered at Or call SAMHSA at SAMHSA-7 ( ) (English and Español). Recommended Citation Substance Abuse and Mental Health Services Administration, Program Evaluation for Prevention: SPF SIG Annual Report, FY2016. Rockville, MD: Substance Abuse and Mental Health Services Administration, Originating Office Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration, 5600 Fishers Lane, Rockville, MD

3 Contents Executive Summary... ES-1 1. Introduction and Background Description of the Initiative Cross-Site Evaluation Description Logic Model Evaluation Research Questions Current Report Methods Data Collection Grantee-Level Instrument Community-Level Instrument Grantee Epidemiological Outcomes Analytic Approach Grantee-Level Analytic Approach Community-Level Analytic Approach Grantee Background Grantee-Level Findings Did Grantees Show Change over Time in Infrastructure? (Evaluation Research Question 1A) Were Changes in Grantee Infrastructure Related to Grantee Characteristics? (Evaluation Research Question 1B) Did the Implementation of the SPF SIG Lead to Grantee-Level Improvement on Epidemiological Outcomes? (Evaluation Research Question 2) Past-Month Use of Alcohol among Youth Age Perceived Risk of Harm of Alcohol Use among Youth Age Disapproval of Alcohol Use among Youth Age Binge Drinking among Youth Age Alcohol-Related Traffic Crash Fatalities Alcohol-Related Arrests What Factors Accounted for Variation in Grantee-Level Epidemiological Outcomes across SPF SIG State Grantees? (Evaluation Research Question 3) Community-Level Findings Community Characteristics Targeted Priorities Targeted Populations Community Coalition Status and Partnerships PEP-C Annual Reports October 2016 Contents iii

4 5.1.4 Availability of Community-Level Data for Planning Community-Level Implementation Barriers Did Communities Show Change over Time in Infrastructure, Resources, and Funding? (Evaluation Research Question 4A) Number of Partnerships Number of Non-SPF SIG Funding Sources Prevention Systems Factors Index Capacity-Building Index Number of Sustainability Activity Types Implemented Evaluation and Monitoring Index Was Community-Level Change in Infrastructure, Resources, and Funding Related to Community Characteristics? (Evaluation Research Question 4B) Discussion of Results Grantee-Level Findings Infrastructure Outcomes Community-Level Findings Community Characteristics Infrastructure Implications for Policy and Practice Next Steps References... R-1 Appendices A B C D E F G H Content of the Infrastructure Survey in the Grantee-Level Instrument Content of the Implementation Survey in the Grantee-Level Instrument Statistical Methods Revisions to the Grantee-Level Infrastructure Scale Community-Level Infrastructure and Process Measures Adult Grantee-Level Epidemiological Outcomes Detailed Findings on the Impact of Community Characteristics on Change in Infrastructure, Resources, and Funding over Time Cohort IV/V Grantees PEP-C Annual Reports October 2016 Contents iv

5 List of Exhibits 1. Grantee Cohorts in the Strategic Prevention Framework State Incentive Grant Strategic Prevention Framework State Incentive Grant National Cross-Site Evaluations Logic Model for the National Cross-Site Evaluation of the Strategic Prevention Framework State Incentive Grant Cohort IV and Cohort V Grant Period and Funded Communities Data Collection Frequency and Submission Deadlines for the Strategic Prevention Framework State Incentive Grant Available Grantee and Community Data, Cohorts IV and V Grantee-Level Evaluation Research Questions, Data Sources, and Analytic Approaches Selected Grantee-Level Epidemiological Outcomes, Data Sources, and Baseline and Follow-up Years by Cohort Community-Level Evaluation Research Questions, Data Sources, and Analytic Approaches Cohort IV and Cohort V Grantee Milestones Priorities Targeted by Cohort IV and Cohort V Grantees Cohort IV Grantee Change over Time in Infrastructure Domains (n = 24) Final Model Hierarchical Linear Model Parameter Estimates (Standard Errors) for Change over Time in Grantee Infrastructure Scores Final Model Hierarchical Linear Model Parameter Estimates (Standard Errors) for Grantee Characteristics Predicting the Total Grantee Infrastructure Score Odds Ratio Estimates of Change in Past-Month Alcohol Use among Youth among Cohort IV and Cohort V States PEP-C Annual Reports October 2016 Contents v

6 16. Odds Ratio Estimates of Change in Youths Perceived Risk of Alcohol Use among Cohort IV and Cohort V States Odds Ratio Estimates of Change in Youths Disapproval of Alcohol Use among Cohort IV and Cohort V States Odds Ratio Estimates of Change in Past-Month Binge Drinking by Youth among Cohort IV and Cohort V States Odds Ratio Estimates of Change in Alcohol-Related Traffic Fatalities among Cohort IV and Cohort V States Odds Ratio Estimates of Change in Alcohol-Related Arrests among Cohort IV and Cohort V States Meta-regression Results for Association between Baseline Grantee-Level Infrastructure Scale Scores and Grantee-Level Outcomes Substances Targeted by Communities Consumption Patterns Targeted by Communities Consequences Targeted by Communities Intervening Variables Targeted by Communities Community Coalition Status Types of Agencies Partnered with Communities Types of Data Sources Used by Communities during the Needs and Resources Assessment Types of Implementation Barriers Experienced by Communities Number of Communities with Completed Community-Level Instrument Part I Surveys at Each Reporting Period Descriptive Statistics for Infrastructure, Resources, and Funding Measures Mean Number of Partnerships by Reporting Period Number of Non-SPF SIG Funding Sources by Reporting Period PEP-C Annual Reports October 2016 Contents vi

7 34. Mean Prevention Systems Factors Index Score by Reporting Period Percentage of Communities Endorsing Each Prevention Systems Factor Index Item at First and Last Reporting Period Mean Capacity-Building Index Score by Reporting Period Percentage of Communities Endorsing Each Capacity-Building Index Item at Any Reporting Period Mean Number of Sustainability Activity Types Completed by Reporting Period Percentage of Communities Completing Sustainability Activity Types at First and Last Reporting Period Mean Evaluation and Monitoring Index Score by Reporting Period Evaluation and Monitoring Activities Completed by Communities at the First and Last Reporting Period Final Model Estimates of Cross-Level Interactions for Infrastructure, Resources, and Funding Hierarchical Linear Models PEP-C Annual Reports October 2016 Contents vii

8 Executive Summary The Strategic Prevention Framework State Incentive Grant (SPF SIG) is a national Substance Abuse and Mental Health Services Administration infrastructure grant program. SPF SIG is designed to help grantees and the communities they fund to (1) prevent the onset and reduce the progression of substance abuse, including childhood and underage drinking; (2) reduce substance abuse-related problems in communities; and (3) build prevention capacity and infrastructure. Since 2004, the Center for Substance Abuse Prevention (CSAP) has funded six groups, or cohorts, of state, tribal, and jurisdictional SPF SIG grantees. The Program Evaluation for Prevention Contract (PEP-C) is charged with conducting the final of three SPF SIG cross-site evaluations to assess the overall implementation and outcomes of SPF SIG across all grantees nationwide. The PEP-C team s specific focuses are evaluating SPF SIG Cohort III through Cohort V and analyzing data elements that are common across the first five SPF SIG cohorts. Overall, the PEP-C SPF SIG cross-site evaluation seeks to answer eight evaluation research questions, of which the current report addresses four. Analyses used cumulative grantee- and community-level data from Cohort IV and Cohort V, submitted from the beginning of grant award (July 2009 for Cohort IV and October 2010 for Cohort V) through spring These two cohorts represented 34 grantees and 347 communities that primarily targeted the reduction of underage alcohol use. Grantee-Level Findings Evaluation Research Question 1A: Did grantees show change over time in infrastructure? Cohort IV members experienced significant growth in prevention infrastructure over the course of their grants, both overall and in the specific domains of data systems; evaluation and monitoring; planning; evidence-based programs, practices, and policies; organizational structure; and workforce PEP-C Annual Reports October 2016 ES-1

9 Executive Summary development. This result suggests that CSAP accomplished the SPF SIG goal of building prevention infrastructure at the grantee level. Evaluation Research Question 1B: Were changes in grantee infrastructure related to grantee characteristics (e.g., implementation barriers)? The PEP-C team was unable to identify significant predictors of Cohort IV s infrastructure growth, which may mean that the SPF SIG model was successful at improving infrastructure in a variety of contexts and did not hinge on the existence of additional resources or challenges that varied by grantee. Evaluation Research Question 2: Did the implementation of the SPF SIG lead to grantee-level improvement on epidemiological outcomes? Cohort IV and Cohort V state grantees demonstrated improvement in nearly every outcome we examined, including youth drinking, binge drinking, disapproval of peer alcohol use, alcoholrelated traffic fatalities, and alcohol-related arrests. This finding supports the notion that SPF SIG is effective; however, it should be interpreted with caution given that a number of these outcomes were also improving on a national level during the time period under investigation. Evaluation Research Question 3: What factors accounted for variation in grantee-level epidemiological outcomes across SPF SIG grantees? Grantees baseline infrastructure did not predict change in epidemiological outcomes for state grantees. When follow-up infrastructure data become available, we will be able to examine whether infrastructure improvements may have impacted outcomes. We will also have more power to test models of grantee outcomes when examining Cohort I through Cohort V in future reports. Community-Level Findings The typical community was very collaborative. At the most recent reporting period, nearly three quarters were either partnered with a community coalition or were themselves a community coalition. The average community had about 10 partners PEP-C Annual Reports October 2016 ES-2

10 Executive Summary in its prevention work, with law enforcement, schools, and youth and family stakeholder groups being the most commonly represented partners. Communities reported numerous barriers to their prevention work, with attitudinal and cultural barriers being most common. This finding aligns with the types of intervening variables that communities targeted, with social access and community norms being most common. Evaluation Research Question 4A: Did communities show change over time in infrastructure, resources, and funding? Communities grew in a number of areas related to prevention capacity and infrastructure, including number of partners, prevention systems factors (e.g., working with agencies that manage prevention data, having a documented process for making prevention-related decisions), number of capacity-building activities, sustainability efforts, and evaluation and monitoring. Evaluation Research Question 4B: Was community-level change in infrastructure, resources, and funding related to community characteristics (e.g., implementation barriers)? Being a coalition predicted greater growth in multiple aspects of community infrastructure (e.g., partnerships, capacity-building activities, sustainability activities, and evaluation and monitoring) than did being a lead community organization that neither was a coalition itself nor partnered with one. If this finding is borne out in analyses of complete cross-site data, it may represent an important contribution to the substance use prevention literature. Our findings about infrastructure growth at the grantee and community levels, as well as outcome improvement at the grantee level, suggest that SPF SIG was successful in achieving at least some of its goals. This result supports the merit of continued funding for CSAP s next-generation community substance abuse prevention initiatives, including Partnerships for Success (PFS). Our inability to uncover many significant predictors of change in infrastructure and outcomes also means that, at this time, we are unable to recommend specific changes to the SPF- PEP-C Annual Reports October 2016 ES-3

11 Executive Summary PFS (and future) funding requirements that would improve the likelihood of effectiveness. However, we will continue to investigate the implementation and impact of SPF SIG as remaining cross-site evaluation data are processed and as data from Cohort I through Cohort V are integrated, enabling analyses across a large and diverse set of grantees and communities. PEP-C Annual Reports October 2016 ES-4

12 1. Introduction and Background 1.1 Description of the Initiative The Strategic Prevention Framework State Incentive Grant (SPF SIG) is a national Substance Abuse and Mental Health Services Administration (SAMHSA) infrastructure grant program that supports an array of activities to help grantees and the communities they fund to build a solid foundation for delivering and sustaining effective substance abuse services. SPF SIG is administered by the Center for Substance Abuse Prevention (CSAP) and is designed to (1) prevent the onset and reduce the progression of substance abuse, including childhood and underage drinking; (2) reduce substance abuse-related problems in communities; and (3) build prevention capacity and infrastructure at the grantee and community levels. The SPF is a systemic public health approach to substance use prevention that is data driven, is based on proven theoretical foundation, and involves implementation of evidence-based programs. CSAP provides funding to states, jurisdictions, and tribal entities to implement the five SPF steps. Cultural competence and sustainability are guiding principles that are applied throughout this dynamic process. Step 1: Assess Needs Profile population needs, resources, and readiness to address the problems and gaps in service delivery. Step 2: Build Capacity Mobilize or build capacity to address needs. Step 3: Plan Develop a comprehensive strategic plan. Step 4: Implement Implement evidence-based programs, policies, and practices for prevention and for infrastructure development activities. Step 5: Evaluate Monitor process, evaluate effectiveness, sustain effective programs and activities, and improve or replace those that fail. Grantees are expected to use the SPF framework to guide all prevention activities, whether funded though the SPF SIG or other funding mechanisms. Primary PEP-C Annual Reports October

13 Introduction and Background priorities are determined through the implementation of the SPF steps at the grantee level, and communities are identified to receive SPF SIG funds to implement the SPF at the community level. Exhibit 1 depicts the six SPF SIG grantee cohorts. (Note that one state was funded as Cohort VI; however, it is not included in cross-site evaluation activities.) For the purposes of this document, the term grantee will refer to all funded states, jurisdictions, and tribal entities in Cohort I through Cohort V. 1 Each grantee was funded for up to 5 years 2 to implement the SPF SIG program. Cohorts IV and V are the focus of the current report. Exhibit 1. Grantee Cohorts in the Strategic Prevention Framework State Incentive Grant Cohort Grant Period Total Grantees States Jurisdictions Tribal Entities I FY II FY III FY IV FY * 5 6** V FY VI FY TOTAL Note. *This includes the District of Columbia. **An additional Cohort IV tribal grantee was funded but relinquished the grant. 1.2 Cross-Site Evaluation Description CSAP has funded three national cross-site evaluations of SPF SIG: one focusing on Cohorts I and II; one focusing on Cohorts III, IV, and V; and one continuing to examine Cohorts III V and also integrating data from Cohorts I and II (current evaluation; see Exhibit 2). The Cohorts III V evaluation design builds on the original design used to evaluate Cohorts I and II (Westat, 2006). Although a number of changes in 1 For Cohort I and Cohort II, the term states was used to describe grantees. Grantees became the preferred term when tribal entities were funded in Cohort III. 2 Most grantees were also granted a no-cost extension of up to 1 year. PEP-C Annual Reports October

14 Introduction and Background instrumentation and data collection procedures were made between Cohorts I and II and Cohorts III V, the current evaluation will integrate data elements common to all five cohorts. The Cohorts III V cross-site evaluation was originally conducted under two previous contracts: the Data Analysis Coordination and Consolidation Center (DACCC) and, most recently, the Data Collection, Analysis, and Reporting (DCAR) contract. In September 2014, CSAP exercised the option to transition the evaluation to the Program Evaluation for Prevention Contract (PEP-C), held by RTI International. The PEP-C cross-site evaluation team is continuing with the implementation of a multilevel evaluation design encompassing data collection and analysis at the grantee, community, and participant levels. These data provide information about implementation processes and systems outcomes at the grantee and community levels, as well as a context for analyzing grantee- and community-level epidemiological outcomes and participant-level National Outcomes Measures. Exhibit 2. Strategic Prevention Framework State Incentive Grant National Cross-Site Evaluations Evaluation Contract Contractor Years Focal Cohorts Strategic Prevention Framework State Incentive Grant (SPF SIG) Pacific Institute for Research and Evaluation, Westat I, II Data Analysis Coordination and Consolidation Center (DACCC) Data Collection, Analysis, and Reporting (DCAR) Program Evaluation for Prevention Contract (PEP-C) Human Services Research Institute (HSRI) III, IV, V Westat RTI International 2014 present III, IV, V plus integrative analyses of Cohorts I-V Logic Model Exhibit 3 shows the PEP-C SPF SIG national cross-site evaluation logic model. The logic model provides an overview of the SPF SIG components. In addition, it PEP-C Annual Reports October

15 Introduction and Background provides a framework to address evaluation research questions, which, in turn, drive all other aspects of the evaluation. Exhibit 3. Logic Model for the National Cross-Site Evaluation of the Strategic Prevention Framework State Incentive Grant Inputs Outputs Outcomes Impact SPF SIG funding CAPT TTA Grantee Resources, including non- SPF SIG funding Infrastructure Barriers Subrecipient Resources, including non- SPF SIG funding Infrastructure (coalition, implementers, partners) Barriers Activities Grantee Progress through SPF steps TTA activities Subrecipient Progress through SPF steps Interventions implemented (type, IOM category, combinations, evidence base, curriculum-driven, cost) Participants # and type of subrecipient communities selected Type of TTA received # of collaborators/ implementation partners # reached, demographics Proximal Outcomes Perceived risk/ harm of use Disapproval of substance use Perception of workplace policy Family communication around drug use Exposure to prevention messages Other intervening variables Individual domain Peer-related domain Family domain School domain Community domain Distal Outcomes Consumption 30-day substance use Age of first substance use Consequences Driving while under the influence of alcohol Alcohol-related traffic fatalities Alcohol- and drugrelated arrests Other consumption and consequence variables Daily school attendance Note. CAPT, Center for the Application of Prevention Technologies; IOM, Institute of Medicine; SPF [SIG], Strategic Prevention Framework [State Incentive Grant]; TTA, training and technical assistance Evaluation Research Questions The primary evaluation objectives for the current national cross-site evaluation are (1) to measure infrastructure development at the grantee and community levels and (2) to determine to what extent the SPF has produced changes in targeted community- and participant-level outcomes. These objectives are addressed with eight evaluation research questions at the grantee, community, and participant levels. However, as explained in this section, only a subset of these research questions is examined in the current report. GRANTEE-LEVEL EVALUATION RESEARCH QUESTIONS 1. A. Did grantees show change over time in infrastructure? B. Were changes in grantee infrastructure related to grantee characteristics (e.g., implementation barriers)? PEP-C Annual Reports October

16 Introduction and Background 2. Did the implementation of the SPF SIG lead to grantee-level improvement on epidemiological outcomes? 3. What factors accounted for variation in grantee-level epidemiological outcomes across SPF SIG grantees? Factors that will be examined include grantee infrastructure, resources, and funding across sources; progress through the SPF steps; training and technical assistance to communities; number and type of communities selected; and barriers reported by grantees. COMMUNITY-LEVEL EVALUATION RESEARCH QUESTIONS 4. A. Did communities show change over time in infrastructure, resources, and funding? B. Was community-level change in infrastructure, resources, and funding related to community characteristics (e.g., implementation barriers)? 5. Did the implementation of the SPF SIG lead to community-level improvement on epidemiological outcomes? 6. What factors accounted for variation in community-level epidemiological outcomes across funded communities? Predictors of outcomes may include both grantee- and community-level measures. Factors that will be examined include community resources and infrastructure, including coalition composition and functioning and partner involvement; training and technical assistance provided by the grantee; intervention strategy characteristics, reach, and intervention strategy type mix; funding (across sources) and costs; and barriers reported by communities. The community-level evaluation research questions use data from two different community-level surveys: one that pertains to the community as a whole and one that is specific to the intervention strategies that were implemented. A previous cross-site evaluation contractor delivered to PEP-C an intervention-specific survey database that did not contain a community identification variable. This oversight delayed the process of matching survey responses to communities. Therefore, only general community-level survey data are available for the current report; data specific to communities intervention strategies were unavailable for this report but will be examined in future years. As of November 2014, most Cohort IV grantees were behind in reporting community epidemiological outcomes data, and only one Cohort V grantee had submitted data. For the Year 1 report, the PEP-C team analyzed the available community epidemiological outcome PEP-C Annual Reports October

17 Introduction and Background for Cohort IV and Cohort V, and they concluded that community sample sizes were too small to justify analyses. In 2015, PEP-C researchers requested and received Office of Management and Budget approval to conduct data collection to obtain overdue data from Cohort IV and Cohort V grantees, and the data are currently being processed. Therefore, community epidemiological outcomes data (and, consequently, Evaluation Research Questions 5 and 6) are not analyzed as part of the current report but will be examined in future years. PARTICIPANT-LEVEL EVALUATION RESEARCH QUESTIONS The questions in this section pertain only to individual direct service intervention strategies for which communities administered participant-level pre- and postintervention assessments. 7. Did participation in individual-level SPF SIG intervention strategies lead to participant-level improvement on outcomes? 8. What factors accounted for variation in participant-level outcomes across communities? Predictors of outcomes may include grantee-, community-, and participant-level factors. Factors that will be examined include community characteristics (e.g., community partner involvement, barriers to implementation); intervention strategy characteristics (e.g., evidence based, curriculum driven, adaptations made); intervention strategy funding (across sources) and costs; and participant demographics. PEP-C staff presented preliminary participant-level findings for Cohorts IV and V in the Year 1 SPF SIG report. Only a small amount of participant-level data has been submitted since Year 1 analyses were conducted. Therefore, final analyses of all participant-level data will conducted for future cross-cohort reports; no participantlevel data will be discussed in the current report. 1.3 Current Report This report presents findings from analyses conducted during the second year of SPF SIG cross-site evaluation under PEP-C, with a focus on Cohorts IV and V. Cohort IV consists of 13 states, 5 jurisdictions, and 6 tribal entities; Cohort V consists of 3 states and 7 tribal entities. See Exhibit 4 for information about the grant period and PEP-C Annual Reports October

18 Introduction and Background funded communities. Note that five Cohort IV and five Cohort V tribal grantees were considered single-community grantees and did not fund separate communities. See Appendix H for a complete list of Cohort IV and Cohort V grantees. Exhibit 4. Cohort IV and Cohort V Grant Period and Funded Communities Cohort Grant Period Total Grantees Funded Communities IV FY * 274 V FY Note. Twenty-two Cohort IV grantees received a 1-year no-cost extension, for a total of 6 years of funding, and 1 grantee received a 2-year no-cost extension, for a total of 7 years of funding. Single-community grantees (i.e., grantees that did not fund separate communities) are counted once under Funded Communities. *An additional Cohort IV tribal grantee was funded but relinquished the grant. Note that findings for this report include data from the baseline funding year for these cohorts (FY 2009 and FY 2010 respectively) through spring In the first year of the PEP-C evaluation, for Cohort IV and Cohort V, we presented baseline grantee-level infrastructure and implementation data, preliminary grantee-level outcomes data, and preliminary participant-level data. This year, we examine changes over time in grantee-level infrastructure and implementation for Cohort IV, update Cohort IV and Cohort V grantee-level outcomes analyses, and conduct preliminary analyses of Cohort IV and Cohort V community-level process data. In the coming 2 years, we will present final analyses of Cohort IV and Cohort V data, as well as analyses that integrate data from SPF SIG Cohort I through Cohort V. PEP-C Annual Reports October

19 2. Methods This section discusses data collection (including instruments and outcomes measures) and the analysis plan for the Strategic Prevention Framework State Incentive Grant (SPF SIG) evaluation. 2.1 Data Collection The national cross-site evaluation includes survey and epidemiological data. Exhibit 5 shows the data collection frequency and deadlines for each evaluation data source that is included in the current report. All grantee- and community-level survey data were collected via online data collection systems. Data were submitted through the Data Analysis Coordination and Consolidation Center (DACCC) and Data Collection, Analysis, and Reporting Contract (DCAR) systems from 2009 through November 2014 and through the Program Evaluation for Prevention Contract (PEP-C) Management Reporting Tool from June 2015 through spring Information about available grantee and community data for this report by cohort can be found in Exhibit 6. For states, the PEP-C team obtained grantee-level epidemiological outcomes data from national sources; see Section for additional details. 3 CLI results reflect data submitted through March 16, 2016, and GLI results reflect data submitted through May 1, PEP-C Annual Reports October

20 Methods Exhibit 5. Data Collection Frequency and Submission Deadlines for the Strategic Prevention Framework State Incentive Grant Grantee-Level Instrument (GLI) Community- Level Instrument (CLI) Instrument Frequency Submission Deadlines Infrastructure Twice over grant period First year: February Final year: Before close-out Implementation Twice over grant period At approval of strategic plan Final year: Before close-out Part I Annually November 1 Grantee epidemiological outcomes Annually Not applicable (compiled by PEP-C) Exhibit 6. Available Grantee and Community Data, Cohorts IV and V With Baseline/Initial Data With Follow-Up/Subsequent Wave Data Grantees Communities Grantees Communities Cohort Instrument N (%) N (%) N (%) N (%) IV GLI Infrastructure 24 (100) n/a 22 (92) n/a GLI Implementation 24 (100) n/a 22 (92) n/a CLI Part I 24 (100) 247 (96) 22 (92) 227 (88) V GLI Infrastructure 10 (100) n/a 0 (0) n/a GLI Implementation 10 (100) n/a 0 (0) n/a CLI Part I 10 (100) 73 (100) 8 (80) 70 (96) Note. CLI, Community-Level Instrument; GLI, Grantee-Level Instrument. The denominator for Cohort IV was 24 funded grantees and 258 funded communities. (Only 258 out of 274 funded communities were required to submit CLI Part I, and so 258 was the denominator used to determine the percentage of communities with available data.) The denominator for Cohort V was 10 funded grantees and 73 funded communities Grantee-Level Instrument Two surveys, the Grantee-Level Instrument (GLI) Infrastructure Survey and the GLI Implementation Survey, were developed for assessing the implementation of the SPF SIG process at the grantee level and measuring the development and enhancement of grantee-level prevention systems. They are typically completed by the Project Director and evaluator working in collaboration. Both instruments are modified versions of the interview protocols used in the Cohort I and Cohort II evaluation. PEP-C Annual Reports October

21 Methods GLI INFRASTRUCTURE SURVEY The GLI Infrastructure Survey collects information about the operations of the grantee-level prevention system (i.e., the entire set of agencies, organizations, and persons that contribute to efforts to prevent substance abuse and related problems within the grantee s geographic area). The specific content areas addressed can be found in Appendix A. The baseline survey gathers information about the prevention system at the time the grant was awarded, and the follow-up survey collects information about the prevention system near the completion of the grant (or nocost extension period when applicable). GLI IMPLEMENTATION SURVEY The GLI Implementation Survey collects information about the grantee s execution of the five SPF steps, including items regarding the grantee s capacity-building activities and barriers to implementation of the SPF. These items are used in our analyses as predictors of grantee infrastructure growth (which is measured using the GLI Infrastructure survey). The specific content areas addressed can be found in Appendix B. The baseline survey, designed to be completed after the SPF SIG strategic plan is approved, provides a retrospective picture of the strategic planning period. The follow-up survey, collected near the completion of the grant (or no-cost extension period when applicable), gathers information about grantee implementation of the SPF process during the course of the grant Community-Level Instrument The Community-Level Instrument (CLI) is a two-part survey. The CLI Parts I and II are typically completed by a community representative. 4 After communities complete the CLI Part I or II, a grantee representative reviews and approves the submission. 4 For grantees that do not fund separate communities (i.e., some jurisdiction and tribal grantees), a grantee representative completes the CLI. PEP-C Annual Reports October

22 Methods As mentioned in Section 1.2.2, intervention-specific data (i.e., CLI Part II) are not included in the current report. CLI PART I The CLI Part I collects data about communities resources, infrastructure, and partner involvement, as well as progress through the SPF steps. The CLI Part I is completed annually at the end of each fiscal year Grantee Epidemiological Outcomes To assess epidemiological outcomes at the grantee level, the PEP-C team uses annual state-level substance use data from the National Survey on Drug Use and Health (NSDUH), data on alcohol-related crash fatalities from the Fatality Analysis Reporting System (FARS), and data on substance-related arrests from the Uniform Crime Reports (UCR). Note that these data sources are available only for state grantees; they are not available for tribal organizations or jurisdictions. Grantees are not responsible for collecting or reporting these data; PEP-C staff obtain them directly from SAMHSA, the National Highway Traffic Safety Administration, and the Federal Bureau of Investigation, respectively. 2.2 Analytic Approach This section describes the analytic approach at the grantee and community levels for this year s evaluation report. Technical details regarding the specifics of statistical techniques described in this section are presented in Appendix C Grantee-Level Analytic Approach This section details the approach that the PEP-C team used to address each of the evaluation research questions in Exhibit 7. All grantee-level results are presented in Section 4. PEP-C Annual Reports October

23 Methods Exhibit 7. Grantee-Level Evaluation Research Questions, Data Sources, and Analytic Approaches Evaluation Research Question 1. A. Did grantees show change a over time in infrastructure? B. Were changes in grantee infrastructure related to grantee characteristics (e.g., implementation barriers)? a 2. Did the implementation of the SPF SIG lead to grantee-level improvement on epidemiological outcomes? 3. What factors accounted for variation in grantee-level epidemiological outcomes across SPF SIG grantees? Data Sources GLI Infrastructure Survey GLI Implementation Survey State estimates from the NSDUH Accident reports from the FARS Arrest reports from the UCR State estimates from the NSDUH Accident reports from the FARS Arrest reports from the UCR GLI Infrastructure Survey Analytic Approaches Hierarchical linear models Random effects meta-analysis Meta-regression Note. FARS, Fatality Analysis Reporting System; GLI, Grantee-Level Instrument; NSDUH, National Survey on Drug Use and Health; SPF SIG, Strategic Prevention Framework State Incentive Grant; UCR, Uniform Crime Reports. a This question originally included both resources and funding; however, resources and funding are not assessed adequately in the GLI, so the PEP-C team focused on infrastructure only. EVALUATION RESEARCH QUESTION #1A: DID GRANTEES SHOW CHANGE OVER TIME IN INFRASTRUCTURE? The Grantee-Level Infrastructure Scale (GLIS) is derived from the GLI Infrastructure Survey. The total score is the sum of six domains: organizational structure; planning; data systems; workforce development; evidence-based programs, practices, and policies; and evaluation and monitoring. All scales had Cronbach s alpha coefficients between.70 and.90, indicating good reliability. (See Appendix D for additional details regarding the calculation and reliability of the GLIS score.) To examine grantee change in infrastructure from the beginning to the end of the grant, the PEP- C team estimated hierarchical linear models (HLMs) for both the total GLIS score and each of the six domains. These models were two-level models (time nested within grantee) with time as the only predictor. Both random intercept and slopes were tested to examine grantee variation in infrastructure growth and baseline scores. PEP-C Annual Reports October

24 Methods EVALUATION RESEARCH QUESTION #1B: WAS GRANTEE-LEVEL INFRASTRUCTURE CHANGE RELATED TO GRANTEE CHARACTERISTICS (E.G., IMPLEMENTATION BARRIERS)? To address the relationship between grantee characteristics and infrastructure, the PEP-C team added three additional predictors to the final model for the GLIS score described in evaluation question 1A: the number of barriers to project progress, the number of capacity-building activities completed, and scores on the Resource Index. The capacity-building activities and barriers measures are collected in the GLI Implementation Survey. (See the Grantee-level Modeling section of Appendix C for more details about these measures and how they were calculated.) All three predictors were measured at both time points and were entered at Level 1 as time varying. Furthermore, interactions between these variables and time were also included in the model. We used a top-down model-building strategy, meaning that the initial model included all possible predictors and that the final model was obtained by successfully trimming the initial model, removing fixed or random effects that did not significantly contribute to model fit. Additional technical detail about the modeling approach can be found in Appendix C. EVALUATION RESEARCH QUESTION #2: DID THE IMPLEMENTATION OF THE SPF SIG LEAD TO GRANTEE-LEVEL IMPROVEMENT ON EPIDEMIOLOGICAL OUTCOMES? To address evaluation research question #2, we selected for analysis the grantee epidemiological outcomes most commonly prioritized for SPF SIG initiatives. Grantee-level epidemiological outcomes come from Federal data sources (NSDUH, FARS, UCR), and thus are available only for states and the District of Columbia (16 of 34 Cohort IV and Cohort V grantees). These outcomes are not available for tribal organizations or jurisdictions. The selected epidemiological outcomes, their data sources, and the baseline and follow-up years used for this year s report are shown in Exhibit 8. Each NSDUH outcome was reported for youth age and adults age 18 or older, with the exception of binge drinking (adults only) and disapproval of alcohol use (youth only). FARS and UCR data reflect all ages. PEP-C Annual Reports October

25 Methods Exhibit 8. Selected Grantee-Level Epidemiological Outcomes, Data Sources, and Baseline and Follow-up Years by Cohort Data Source and Measures National Survey on Drug Use and Health (NSDUH) Past-30-day alcohol use Perceived risk of alcohol use Disapproval of alcohol use a Binge drinking b Fatality Analysis Reporting System (FARS) Annual alcohol-related crash fatalities Uniform Crime Reports (UCR) Annual alcohol-related arrests Cohort IV Cohort V Baseline Follow-up Baseline Follow-up 2008/ / / / Note. NSDUH estimates are based on samples pooled across 2 years to ensure adequate sample sizes and to detect year-to-year differences more efficiently (SAMHSA, n.d.). Alcohol-related arrests were calculated by summing arrests for DUI and liquor law license violations. a youth only b adults only The current choice of follow-up years for Cohort IV and Cohort V reflects the availability of data rather than the ideal follow-up date (i.e., time point after all implementation is complete); the latest data available were used for the follow-up year for both cohorts. (As data become available, future reports will use ideal follow-up dates.) The PEP-C team conducted random effects meta-analyses to compute a summary effect across all Cohort IV and Cohort V grantees quantifying change over time; Cohort IV and Cohort V grantees were not reported separately to improve the statistical power of the meta-analysis. For details regarding random effects meta-analysis, see Appendix C. EVALUATION RESEARCH QUESTION #3: WHAT FACTORS ACCOUNTED FOR VARIATION IN GRANTEE-LEVEL EPIDEMIOLOGICAL OUTCOMES ACROSS SPF SIG GRANTEES? Once a summary statistic was calculated using meta-analysis techniques, the PEP-C team conducted meta-regression analyses to examine whether baseline infrastructure scores influenced the change in grantee outcomes from baseline to follow-up. (Once follow-up GLIS data are available for all grantees, we will examine whether change in the GLIS score predicted change in grantee outcomes.) PEP-C Annual Reports October

26 Methods Meta-regression is a technique in which regression analyses are conducted on effect sizes (e.g., odds ratios) rather than on raw data. For details regarding this technique, see Appendix C. Given that only 13 grantees from Cohort IV and 3 grantees from Cohort V had grantee outcomes (because they were state grantees), the PEP-C team conducted the meta-regression on all 16 grantees (rather than separately on Cohort IV and Cohort V) to improve the power of the analysis. The results of these analyses should be considered preliminary, for the following reasons: The most recent grantee outcome data were from 2014 ( for NSDUH). This is important because the dependent variables in our analyses were grantee-level change from baseline to follow-up. To examine shortterm outcomes in the current report, we treated these data as the follow-up years for both cohorts. The follow-up year used in this year s report may have been too early to fully reflect all grantee-level change that occurred as a result of SPF SIG. Appropriate follow-up years for Cohort IV and Cohort V will be used in future reports. 5 Sample sizes for the meta-regression analyses were small, resulting in limited statistical power and lower likelihood of detecting statistical significance Community-Level Analytic Approach Exhibit 9 shows evaluation research questions, data sources, and analytic approaches for community-level analyses. In the summer of 2016, PEP-C completed the collection of overdue data from Cohort IV and Cohort V grantees. Because community-level outcome data for Cohort IV and Cohort V had not yet been processed when this report was drafted, the PEP-C team was not able to address evaluation questions #5 and #6 in this year s annual report. Results for evaluation question #4 are reported in Section 4. 5 Selecting an appropriate follow-up time point is challenging. For example, most Cohort IV grantees funding ended in June 2015, but FARS and UCR data are reported for full calendar years, so either 2015 or 2016 could be seen as appropriate follow-up time points. The PEP-C team will work with CSAP to determine appropriate follow-up points by cohort. PEP-C Annual Reports October

27 Methods Exhibit 9. Community-Level Evaluation Research Questions, Data Sources, and Analytic Approaches Evaluation Research Question Data Sources Analytic Approaches 4. A. Did communities show change over time in infrastructure, resources, and funding? B. Was the change or lack of it related to community characteristics (e.g., implementation barriers)? Note. CLI Part I* Generalized HLM (for noncontinuous outcomes) CLI, Community-Level Instrument; HLM, hierarchical linear models. * The PEP-C team is in the process of matching Cohort IV CLI Part II to communities; therefore, CLI Part II data could not be analyzed for the current report. EVALUATION RESEARCH QUESTION #4A: DID COMMUNITIES SHOW CHANGE OVER TIME IN INFRASTRUCTURE, RESOURCES, AND FUNDING? To examine community-level change over time in infrastructure, resources, and funding, the PEP-C team conducted HLM analyses of the following infrastructurerelated outcomes (for details regarding the choice of measures and how they were constructed, see Appendix E): Number of partnerships; Number of non-spf SIG funding sources; Prevention Systems Factors Index; Capacity-Building Index; Number of sustainability activity types implemented; and Evaluation and Monitoring Index. We tested two-level HLMs (time nested within community) with time as the only predictor. Although the intention was to include random effects for the intercept and slope, the random intercept did not significantly contribute to model fit and the random slope model would not converge, and so no random effects were included in final models. (See Appendix C for modeling details.) PEP-C Annual Reports October

28 Methods EVALUATION RESEARCH QUESTION #4B: WAS COMMUNITY-LEVEL CHANGE IN INFRASTRUCTURE, RESOURCES, AND FUNDING RELATED TO COMMUNITY CHARACTERISTICS? To address the relationship between community characteristics and changes in infrastructure, resources, and funding over time, the PEP-C team added additional predictors to the model for each infrastructure-related outcome described above. The predictors included Number of implementation barriers ever reported Coalition status (i.e., coalition, coalition partner, or other type of organization) Whether or not the community targeted a minority population (defined as a racial/ethnic group other than White non-hispanic) 6 The number of available data sources during needs and resources assessment (serves as a measure of data infrastructure) See Appendix E for details regarding how predictors were constructed. All predictors were time invariant. Interactions between these variables and time were also included in the model. We used a top-down model-building strategy, meaning that the initial model included all possible predictors and that the final model was obtained by successfully trimming the initial model, removing fixed or random effects that did not significantly contribute to model fit. 6 Note that for some SPF SIG communities (e.g., tribal and Pacific/Caribbean jurisdiction communities), the racial/ethnic group targeted is actually the majority population. The term minority is used for brevity only, although it may not accurately reflect the relative size of racial/ethnic populations within all communities. PEP-C Annual Reports October

29 3. Grantee Background This section presents information on Cohort IV and Cohort V grantees milestones (strategic plan approval and community funding) and targeted priorities to provide context for analyses. After the start of the Strategic Prevention Framework State Incentive Grant (SPF SIG) award, grantees needed, on average, 24.6 months to receive approval of their grantee-level strategic plan and approximately 26.5 months (an additional 2 months) to begin funding communities (see Exhibit 10). During the planning period, grantees are expected to work on the first three steps of the SPF process (needs and resources assessment, capacity building, and strategic planning). Exhibit 10. Cohort IV and Cohort V Grantee Milestones Average Minimum Months After Months After Cohort SPF Step N Award Award Received strategic plan approval IV V Began funding communities Received strategic plan approval Began funding communities Maximum Months After Award Note: Six Cohort IV grantees and five Cohort V grantees were considered single-community grantees and did not fund separate communities. SPF, Strategic Prevention Framework. Targeted priorities are reported in grantees approved strategic plans (see Exhibit 11). All 24 Cohort IV grantees and 10 Cohort V grantees had approved strategic plans. Eight grantees identified a single priority, 15 identified 2 priorities, and 11 chose more than 2. Underage use of alcohol was the most frequently targeted priority, followed by binge drinking. Of the 34 grantees with identified priorities, all targeted alcohol-related priorities, 9 in combination with other drug-related priorities. One Cohort V grantee targeted tobacco-related priorities. Of the 29 grantees identifying a targeted age population, 11 (38%) targeted only youth PEP-C Annual Reports October

30 Grantee Background younger than age 18, 3 (10%) targeted only adults age 18 and older, and 15 (52%) targeted both youth and adults. Exhibit 11. Priorities Targeted by Cohort IV and Cohort V Grantees Priority Grantees Targeting Consumption Patterns Underage use of alcohol 27 (79%) Binge drinking 15 (44%) Heavy use of alcohol 6 (18%) Any use of alcohol 6 (18%) Any use of illegal drugs 6 (18%) Drinking and driving 5 (15%) Nonmedical use of prescription drugs 5 (15%) Youth marijuana use 1 (3%) Drinking during pregnancy 1 (3%) Riding with a driver who has been drinking 1 (3%) Tobacco use 1 (3%) Consequences Motor vehicle crashes (including fatalities) 4 (12%) Dependence or abuse 1 (3%) Drug-related morbidity/mortality 1 (3%) Alcohol-related death 1 (3%) Other Priority Reduction of risk factors 1 (3%) Suicide 1 (3%) Note. Percentages will not equal 100% because most grantees selected more than one priority. N = 34. PEP-C Annual Reports October

31 4. Grantee-Level Findings This section will discuss findings related to grantee infrastructure (from the Grantee-Level Instrument [GLI]) for Cohort IV and findings related to the change in grantee-level outcomes from baseline (year funded) to follow-up (2014, the most recent year of available data) for state (but not tribal or jurisdictional) grantees in Cohort IV and Cohort V. 4.1 Did Grantees Show Change over Time in Infrastructure? (Evaluation Research Question 1A) Cohort IV grantees reported significant improvements in infrastructure across each of six domains. The evaluation and monitoring domain saw the greatest improvement. The change in Cohort IV grantee infrastructure scores from the beginning of the Strategic Prevention Framework State Incentive Grant [SPF SIG] to the end of the grant 7 is shown in Exhibit 12. Grantee infrastructure scores in all six domains increased over time. Evaluation and monitoring saw the greatest improvement and workforce development saw the least improvement in scores over time. 7 Year 5 or end of no-cost extension of up to a year. Most grantees received 1-year no-cost extensions. PEP-C Annual Reports October

32 Grantee-Level Findings Exhibit 12. Cohort IV Grantee Change over Time in Infrastructure Domains (n = 24) Cohort IV grantees infrastructure scores increased from the beginning of the grant to the end of the grant in all six domains. Note. Domain scores range from 0 to 1, and higher scores indicate better infrastructure. To examine whether grantee infrastructure change was statistically significant, we estimated hierarchical linear models (HLM) for both the total grantee infrastructure score and each of six grantee infrastructure scale domains shown in Exhibit 12. All models were two-level models (time nested in grantee) with time as the only predictor. Both random intercept and slopes were tested to examine grantee variation in infrastructure growth and baseline scores. Results are shown in Exhibit 13. For the total infrastructure score and all domain scores, the estimate for time (β1) was statistically significant, indicating that grantee infrastructure increased significantly over time, both overall and for each infrastructure domain. PEP-C Annual Reports October

33 Grantee-Level Findings Exhibit 13. Final Model Hierarchical Linear Model Parameter Estimates (Standard Errors) for Change over Time in Grantee Infrastructure Scores Parameters Data Systems Intercept (β0) 0.55*** (0.04) Time (β1) 0.18** (0.05) Evaluation and Monitoring 0.39*** (0.05) 0.32*** (0.07) Planning 0.39*** (0.05) 0.23** (0.06) EBPPPs 0.37*** (0.06) 0.23*** (0.04) Organizational Structure 0.36*** (0.05) 0.31** (0.08) Note. EBPPP, evidence-based programs, policies, and practices. ***p <.001, **p <.01 Workforce Development 0.35*** (0.05) 0.21** (0.05) Total 0.37*** (0.04) 0.26*** (0.04) 4.2 Were Changes in Grantee Infrastructure Related to Grantee Characteristics? (Evaluation Research Question 1B) Degree of grantee infrastructure growth did not have a significant association with either level of resources or numbers of capacity-building activities or implementation barriers. However, grantees with the most resources at baseline tended to also have the greatest levels of infrastructure. Although grantee infrastructure generally improved over time, the evaluation also examined whether infrastructure growth varied on the basis of grantee factors. To examine this evaluation question, we added additional predictors to the final HLM for the total grantee infrastructure score described above. We tested the following potential predictors of infrastructure growth: number of capacity-building activities completed, number of barriers reported, and scores on the resource index. Capacitybuilding activities and barriers are collected in the GLI Implementation Survey. (See Appendix C for details regarding how predictors were calculated.) Results for both the initial and the final model can be seen in Exhibit 14. (Final model estimates were obtained by removing predictors that did not significantly contribute to model fit from the initial model.) Numbers of capacity-building activities and barriers (and their interactions with time) did not contribute to PEP-C Annual Reports October

34 Grantee-Level Findings improved model fit, indicating that these factors were not significantly associated with degree of infrastructure growth, and so were trimmed from the final model. In the final model, time (β1) was retained, again indicating significant change over time in total infrastructure scores. Resource index (β2) was also significant, suggesting that having greater resources (e.g., staff time, data analysis resources) at baseline was associated with a higher grantee infrastructure score at baseline. However, the lack of a time resource index (β5) interaction suggests that change in resources did not predict change in infrastructure. Exhibit 14. Final Model Hierarchical Linear Model Parameter Estimates (Standard Errors) for Grantee Characteristics Predicting the Total Grantee Infrastructure Score Parameters Initial Model Estimates Final Model Estimates Intercept (β0) 0.08 (0.10) 0.13* (0.05) Time (β1) 0.22 (0.16) 0.23*** (0.04) Resource index (β2) 0.04*** (0.01) 0.04*** (0.01) Capacity-building (β3) 0.01 (0.01) Barriers (β4) (0.01) Time resource index (β5) (0.01) Time capacity-building (β6) 0.01 (0.02) Time barriers (β7) (0.02) Note. ***p <.001, *p < Did the Implementation of the SPF SIG Lead to Grantee- Level Improvement on Epidemiological Outcomes? (Evaluation Research Question 2) State grantees reported improvements in rates of past-month drinking among youth, youth binge drinking, and disapproval of peer alcohol use, whereas they saw no improvement in perceived risk of alcohol use. State grantees also reported decreases in alcohol-related traffic fatalities and arrests for all ages. For each outcome, the Program Evaluation for Prevention Contract (PEP-C) team conducted meta-analyses to compute a summary effect of the change from the baseline year (2009 for Cohort IV and 2010 for Cohort V) to the follow-up year PEP-C Annual Reports October

35 Grantee-Level Findings (2014 for both cohorts). 8 All odds ratios (ORs) were calculated such that ORs greater than 1 indicate change in the desired direction (e.g., greater odds of no 30- day use). 9 Note that although the number of alcohol-related arrests is a required National Outcomes Measure for the SPF SIG program, the desired direction of change for this measure may be unclear. At the local level, increased enforcement of existing alcohol laws may result in increases in arrest rates, even when unlawful alcohol use remains unchanged or decreases. We have chosen to code a decrease in alcohol-related arrests as a positive outcome. We present youth results in this section, followed by meta-regression analyses examining whether baseline Grantee-Level Infrastructure Scale (GLIS) scores predicted change in grantee outcomes from baseline to follow-up (see Section 1.2). Very little change in outcomes occurred among adults, an expected finding given SPF SIG s focus on underage use, so results for adults are presented in Appendix F Past-Month Use of Alcohol among Youth Age Exhibit 15 shows ORs representing the change over time in the percentage of youth reporting past-month (30-day) abstinence from alcohol use. The summary OR, representing the summary effect across all 16 Cohort IV and Cohort V state grantees, was 1.41 (confidence interval [CI] = , Z = 7.40, p <.001), indicating that there was a statistically significant decrease in prevalence of past-month alcohol use among youth. 8 For this report, the follow-up year was based on the most recent data available (2014) and may not necessarily be the ideal follow-up year for each cohort. Note that for NSDUH outcomes, estimates are based on samples pooled across two years (e.g., ). 9 This is different from how odds ratios are typically constructed for substance use behavior, with an odds ratio greater than 1 indicating higher likelihood of use. PEP-C Annual Reports October

36 Grantee-Level Findings Exhibit 15. Odds Ratio Estimates of Change in Past-Month Alcohol Use among Youth among Cohort IV and Cohort V States Note. N = 16. (Data from the National Survey on Drug Use and Health were available only for states and the District of Columbia.) Odds ratios greater than 1 indicate change in the desired direction (i.e., greater odds of no 30-day use of alcohol at follow-up than at baseline). Estimates are based on samples pooled across 2 years. For the summary effect, the confidence interval was too small to display Perceived Risk of Harm of Alcohol Use among Youth Age Exhibit 16 shows ORs representing the change over time in the percentage of youth perceiving great risk in having five or more drinks of alcohol once or twice a week. No grantees showed statistically significant change in perceived risk of harm, and the summary OR was not statistically significant (OR = 0.95, CI = , Z = , p =.18). This indicates that there was no notable change in perceived risk of alcohol use from baseline to follow-up, an unexpected finding given previous PEP-C Annual Reports October

37 Grantee-Level Findings research showing perceived risk as an intervening variable mediating substance use rates. Exhibit 16. Odds Ratio Estimates of Change in Youths Perceived Risk of Alcohol Use among Cohort IV and Cohort V States Note. N = 16. (Data from the National Survey on Drug Use and Health were available only for states and the District of Columbia.) Odds ratios greater than 1 indicate change in the desired direction (i.e., greater odds of perceiving great risk of alcohol use at follow-up than at baseline). NSDUH estimates are based on samples pooled across 2 years. For the summary effect, the confidence interval was too small to display Disapproval of Alcohol Use among Youth Age Exhibit 17 shows ORs representing the change over time in the percentage of youth reporting that they strongly disapproved of someone their age having one or two PEP-C Annual Reports October

38 Grantee-Level Findings drinks of an alcoholic beverage every day. The summary OR was 1.25 (CI = , Z = 5.68, p <.001), indicating a statistically significant increase in the proportion of youth who strongly disapproved of daily alcohol use. Exhibit 17. Odds Ratio Estimates of Change in Youths Disapproval of Alcohol Use among Cohort IV and Cohort V States Note. N = 16. (Data from the National Survey on Drug Use and Health were available only for states and the District of Columbia.) Odds ratios greater than 1 indicate change in the desired direction (i.e., greater odds of strong disapproval of daily alcohol use at follow-up than at baseline). NSDUH estimates are based on samples pooled across 2 years. For the summary effect, the confidence interval was too small to display Binge Drinking among Youth Age Exhibit 18 shows ORs representing the change over time in the percentage of youth reporting abstinence from past-month binge drinking (five or more alcoholic PEP-C Annual Reports October

39 Grantee-Level Findings beverages on the same occasion). The summary OR was 1.51 (CI = , Z = 6.17, p <.001), indicating that there was a statistically significant increase in abstinence from past-month binge drinking from baseline to follow-up. Exhibit 18. Odds Ratio Estimates of Change in Past-Month Binge Drinking by Youth among Cohort IV and Cohort V States Note. Note: N = 16. (Data from the National Survey on Drug Use and Health were available only for states and the District of Columbia.) Odds ratios greater than 1 indicate change in the desired direction (i.e., greater odds of no 30-day binge drinking at follow-up than at baseline). NSDUH estimates are based on samples pooled across 2 years. For the summary effect, the confidence interval was too small to display. PEP-C Annual Reports October

40 Grantee-Level Findings Alcohol-Related Traffic Crash Fatalities Exhibit 19 shows ORs representing the change over time in the percentage of all crash fatalities that were alcohol related, as well as the summary effect (derived from the meta-analysis) across grantees in Cohort IV and Cohort V. One grantee was excluded from the analyses because of limited crash fatalities data. Exhibit 19. Odds Ratio Estimates of Change in Alcohol-Related Traffic Fatalities among Cohort IV and Cohort V States Note. N = 15 Data from the Fatality Analysis Reporting System were available only for states and the District of Columbia). One grantee was excluded from this analysis because of limited data. Odds ratios greater than 1 indicate change in the desired direction (i.e., greater odds of fatalities not related to alcohol at follow-up than at baseline). PEP-C Annual Reports October

41 Grantee-Level Findings The summary OR, representing the summary effect across 15 grantees, was 1.11 (CI = , Z = 1.90, p =.06), indicating a marginally significant decrease in the proportion of alcohol-related traffic fatalities from baseline to follow-up Alcohol-Related Arrests Exhibit 20 shows ORs representing the change over time in the percentage of all arrests that were alcohol related. One Cohort IV grantee and one Cohort V grantee were excluded from the analysis because of limited arrest data. The summary effect OR was 1.23 (CI = , Z = 3.20, p <.01), indicating a significant reduction in the proportion of alcohol-related arrests from baseline to follow-up. Exhibit 20. Odds Ratio Estimates of Change in Alcohol-Related Arrests among Cohort IV and Cohort V States Note. N = 14 (Data from Uniform Crime Reports were available only for states and the District of Columbia). Two grantees were excluded from this analysis because of limited data. Odds ratios greater than 1 indicate change in the desired direction (i.e., greater odds of arrests not related to alcohol at follow-up than at baseline). PEP-C Annual Reports October

42 Grantee-Level Findings 4.4 What Factors Accounted for Variation in Grantee-Level Epidemiological Outcomes across SPF SIG State Grantees? (Evaluation Research Question 3) Baseline grantee-level infrastructure did not predict changes in epidemiological outcomes for state grantees. Given that our sample size of Cohort IV and Cohort V state grantees was small (N = 16) and that Cohort V follow-up GLI data were not available, our ability to explore factors that might account for outcomes performance among state grantees was limited to one predictor, baseline grantee infrastructure. For each grantee-level outcome reported in Section 4.3, we used meta-regression to examine whether variation in ORs was related to the baseline GLIS score. The results (Exhibit 21) indicate that the baseline GLIS score did not predict change in any of the granteelevel youth outcomes or consequence outcomes. Exhibit 21. Meta-regression Results for Association between Baseline Grantee-Level Infrastructure Scale Scores and Grantee-Level Outcomes Baseline Grantee-Level Infrastructure Scale Score 95% CI Grantee-Level Outcomes N Β SE Lower Upper Youth past-month alcohol use Youth perceived risk of alcohol use Youth disapproval of daily alcohol use Youth past-month binge drinking Alcohol-related traffic fatalities (all ages) Alcohol-related arrests (all ages) Note. B, standardized regression coefficient; SE, standard error; CI, confidence interval. PEP-C Annual Reports October

43 5. Community-Level Findings 5.1 Community Characteristics The majority of communities targeted underage alcohol consumption patterns. The most commonly targeted consequence was motor vehicle crashes and fatalities, and the most commonly targeted intervening variable was social access. About a quarter of communities were led by coalitions and about half were partnered with a coalition. Almost a third of communities targeted racial or ethnic minority populations. The most commonly reported implementation barrier was easy access to alcohol, followed by community awareness and attitudes barriers Targeted Priorities Communities reported their Strategic Prevention Framework State Incentive Grant (SPF SIG) priorities (consumption patterns, consequences, and intervening variables) in the Community-Level Instrument (CLI) Part I. Two hundred seventysix communities from Cohort IV and Cohort V reported their targeted priorities (see Exhibit 22). Of these, 239 (87%) targeted alcohol-related priorities, either alone (n = 232; 84%) or in combination with drug-related priorities (n = 17; 6%), tobacco-related priorities (n = 1; 0.4%), or with both drug- and tobacco-related priorities (n = 9; 3%). Six percent (n = 17) targeted only drug-related priorities. Exhibit 22. Substances Targeted by Communities PEP-C Annual Reports October

44 Community-Level Findings Communities typically targeted multiple priorities related to substance consumption (mean [M] = 7.4, standard deviation [SD] = 4.4, range = 2 22). Exhibit 23 shows the specific consumption patterns targeted by communities. Most communities targeted underage drinking (n = 247; 90%), and many targeted binge drinking (n = 178; 65%). The average number of consumption patterns targeted was 2.1 (SD = 1.1, range = 1 8). Exhibit 23. Consumption Patterns Targeted by Communities Note. Percentages will not equal 100% because most communities targeted more than one consumption pattern. Consequences targeted by communities are shown in Exhibit 24. Most communities targeted more than one consequence (M = 1.6, SD = 1.6, range = 0 9). The most commonly targeted consequence was alcohol- or drug-related motor vehicle crashes and fatalities (n = 129; 64%). Crime (n = 90; 44%) and alcohol-related morbidity/mortality (n = 85; 42%) were also commonly targeted consequences. Most communities targeted multiple intervening variables (M = 4.6, SD = 2.7, range = 0 12; see Exhibit 25). The most commonly targeted intervening variables were social access to substances (n = 207; 79%), community norms (n = 161; 61%), and access to substances via retail outlets (n = 155; 59%). PEP-C Annual Reports October

45 Community-Level Findings Exhibit 24. Consequences Targeted by Communities Note. Percentages will not equal 100% because most communities targeted more than one consequence. Exhibit 25. Intervening Variables Targeted by Communities Note. Percentages will not equal 100% because most communities targeted more than one intervening variable. PEP-C Annual Reports October

46 Community-Level Findings Targeted Populations Communities also reported specific populations targeted by their SPF SIG initiatives. Two hundred eighty-one Cohort IV and Cohort V communities reported at least one target population. Of those 281 communities, most (n = 197; 70%) targeted 1 4 populations. Most communities targeted underage youth (n = 218; 78%) and about a third targeted one or more racial or ethnic minority groups (n = 88; 31%) Community Coalition Status and Partnerships Communities were asked to report the type of organization that was responsible for implementing the SPF SIG. About half of Cohort IV and Cohort V communities reported that they were an organization partnering with a community coalition, and a quarter of communities were themselves coalitions. The rest were neither a coalition nor a partner to one (e.g., government agencies, school districts, colleges, etc.). SPF SIG-funded community coalitions varied in level of coalition maturity, with an average duration of years (SD = 7.82, range = 3 32 years; see Exhibit 26). Exhibit 26. Community Coalition Status PEP-C Annual Reports October

47 Community-Level Findings Communities reported partnering with a variety of local, grantee-level, and Federal organizations to implement their prevention initiatives. Most communities were involved in partnerships with more than one type of organization (M = 10.6, SD = 4.1, range = 1 17; see Exhibit 27). The most common partnerships were with law enforcement agencies (89%), schools or school districts (86%), and youth groups or representatives (84%). Exhibit 27. Types of Agencies Partnered with Communities Note. Percentages will not equal 100% because communities partnered with more than one agency. N = 313. PEP-C Annual Reports October

48 Community-Level Findings Availability of Community-Level Data for Planning Communities were asked to report the number and types of data used to conduct their community needs and resources assessment. Exhibit 28 shows the types of data used by communities that provided information on their assessment process. Student surveys were the most common data type (used by 75%), followed by law enforcement and public safety data (each used by 73%). The average number of data types used was 6.3 (SD = 3.1, range = 0 12). Exhibit 28. Types of Data Sources Used by Communities during the Needs and Resources Assessment Note. Percentages will not equal 100% because communities used more than one type of data during their needs and resources assessments. N = Community-Level Implementation Barriers Communities were asked to report demographic, environmental, or cultural factors that may have introduced barriers and affected the prevention activities in their PEP-C Annual Reports October

49 Community-Level Findings community. Exhibit 29 shows the reported implementation barriers. On average, communities reported experiencing nearly a dozen community-level implementation barriers (M = 11.5, SD = 4.7, range = 0 20). The most common implementation barrier was easy access to alcohol (reported by 94%), followed by community awareness and attitudes barriers (88% each). The most commonly mentioned other barrier was historical or intergenerational trauma. Exhibit 29. Types of Implementation Barriers Experienced by Communities Note. Percentages will not equal 100% because communities reported more than one barrier. One grantee did not report any barriers. N = 320. PEP-C Annual Reports October

50 Community-Level Findings 5.2 Did Communities Show Change over Time in Infrastructure, Resources, and Funding? (Evaluation Research Question 4A) Communities saw significant improvements over time in most infrastructure-related measures. The most dramatic improvements were in evaluation and monitoring infrastructure. However, communities saw decreases in the number of non-spf SIG funding sources over time, despite increases in activities to leverage other funding sources. The Program Evaluation for Prevention Contract (PEP-C) team used hierarchical linear models (HLMs) to model changes over time in infrastructure, resources, and funding, with time being operationalized as annual reporting period (RP). All HLMs were two-level models with RP (Level 1) nested within community (Level 2). Communities funding began at varying time points after grantee SPF SIG award, so we operationalized time such that First indicates the first RP of CLI Part I data for each community, regardless of the actual date of the first RP. Also, given varying funding dates, the number of annual RPs varies by community. Exhibit 30 shows the number of communities with data available for analysis at each RP. 10 Descriptive statistics that summarize values across all RPs for the chosen measures of infrastructure, resources, and funding are shown in Exhibit 31. See Appendix E for details regarding the measures and how they were scored. 10 Given the drop in completed surveys for communities after RP 3, there was some concern that estimates for later time points would be biased in favor of high-infrastructure communities, who may be more likely to complete surveys at later time points. The PEP-C team explored sub-analytic models that used only communities with four or five RPs to ensure that the observed trends toward higher scores at later time points were due to actual increases on the outcome measures rather than to an artifact of attrition. Trends observed in the sub-analytic models were similar to those in the overall models, suggesting that attrition of communities was not significantly biasing trends over time. PEP-C Annual Reports October

51 Community-Level Findings Exhibit 30. Number of Communities with Completed Community-Level Instrument Part I Surveys at Each Reporting Period Exhibit 31. Descriptive Statistics for Infrastructure, Resources, and Funding Measures Measure (Range) Med M SD Min Max Number of partnerships (0 17) Number of additional non-spf SIG funding sources (0 17) Prevention Systems Factors Index (0 4) Capacity-Building Index (0 5) Number of sustainability activities completed (0 6) Evaluation and Monitoring Index (0 7) Note. Med, Median; M, Mean; SD, Standard Deviation; Min, Minimum; Max, Maximum; SPF SIG, Strategic Prevention Framework State Incentive Grant Number of Partnerships Exhibit 32 shows the mean number of partnerships reported by communities over time. The main effect of RP was significant (log odds = 0.05, t = 4.3, p <.001, OR = 1.05), indicating a slight upward trend in the number of partnerships reported over time. Pairwise comparisons for individual RPs indicated a statistically significant increase in the number of partnerships (p <.001) from the first RP to the PEP-C Annual Reports October

52 Community-Level Findings second RP only, which is unsurprising given the emphasis on building partnerships in the beginning of the grant. Exhibit 32. Mean Number of Partnerships by Reporting Period Note. Blue reporting periods (RPs) represent statistically significant changes compared with the previous RPs. The maximum number of partners that communities could endorse was Number of Non-SPF SIG Funding Sources Exhibit 33 shows communities mean number of non-spf SIG funding sources available for prevention activities over time. The trend over time was significant (log odds = -0.05, t = -2.06, p <.05, OR = 0.96), suggesting a decline in the number of non-spf SIG funding sources over RPs Prevention Systems Factors Index Exhibit 34 shows mean scores over time for the Prevention Systems Factors Index, which measures prevention infrastructure factors such as a documented decisionmaking process and an agency responsible for data management. The main effect of RP was significant (log odds = 0.11, t = 7.3, p <.001, OR = 1.12), indicating prevention system infrastructure growth. Pairwise comparisons of individual RPs indicated a statistically significant increase in the index score from the second RP to the third RP (p <.01). PEP-C Annual Reports October

53 Community-Level Findings Exhibit 33. Number of Non-SPF SIG Funding Sources by Reporting Period Note. There were no statistically significant changes between any pair of adjacent reporting periods; however, the trend over time was significant. The maximum number of funding sources that communities could endorse was 17. SPF SIG, Strategic Prevention Framework State Incentive Grant. Exhibit 34. Mean Prevention Systems Factors Index Score by Reporting Period Note. Blue reporting periods (RPs) represent statistically significant changes compared with the previous RP. The range for the Prevention Systems Factors score is 0 4. Further analyses examined changes in each of the prevention systems factors in the index, from communities first RP to their last (most recent) submitted RP. Exhibit 35 shows the percentage of communities endorsing each prevention PEP-C Annual Reports October

54 Community-Level Findings systems factor at the first and last RPs. Each item showed an increase from first to last RP. The largest increase was in decision-making processes; only 38 percent of communities reported having a written, documented process for making prevention-related decisions at baseline, whereas the majority (57%) had a documented process at the most recent RP. Exhibit 35. Percentage of Communities Endorsing Each Prevention Systems Factor Index Item at First and Last Reporting Period Note. ATOD, alcohol, tobacco, and other drugs Capacity-Building Index Exhibit 36 shows mean Capacity-Building Index scores over time. The main effect of RP was significant (log odds = 0.12, t = 13.9, p <.001, OR = 1.13), indicating an increase in capacity-building activities over time. Pairwise comparisons of individual RPs indicated a statistically significant increase in the index score from the first RP to the second RP (p <.001) and from the second RP to the third RP (p <.001), indicating that the greatest growth occurred in the first 2 years of community funding. PEP-C Annual Reports October

55 Community-Level Findings Exhibit 36. Mean Capacity-Building Index Score by Reporting Period Note. Blue reporting periods (RPs) represent statistically significant changes compared with the previous RP. The range for the Capacity-Building Index score is 0 5. Further analyses examined the percentage of communities implementing each capacity-building activity during the first RP and most recent RP of the SPF SIG (see Exhibit 37). There were large increases in the percentage of communities raising awareness of substance use problems (from 63% to 95%) and the percentage working to sustain prevention intervention activities (from 48% to 90%). PEP-C Annual Reports October

56 Community-Level Findings Exhibit 37. Percentage of Communities Endorsing Each Capacity-Building Index Item at Any Reporting Period Note. SPF SIG, Strategic Prevention Framework State Incentive Grant Number of Sustainability Activity Types Implemented Exhibit 38 shows mean numbers of types of sustainability activities over time. The main effect of RP was significant (log odds = 0.31, t = 16.3, p <.001, OR = 1.36), indicating that communities implemented more types of sustainability activities over time. Pairwise comparisons of individual RPs indicated statistically significant increases in sustainability activities between almost every RP during the course of the grant, suggesting that communities undertook more of these activities as the grant progressed. PEP-C Annual Reports October

57 Community-Level Findings Exhibit 38. Mean Number of Sustainability Activity Types Completed by Reporting Period Note. Blue reporting periods (RPs) represent statistically significant changes compared with the previous RP. The maximum number of sustainability activity types communities could endorse was six. Further analyses examined changes in completion of each sustainability activity type from communities first RP to their last (most recent) submitted RP. Exhibit 39 shows the percentage of communities reporting that they worked on sustainability activity types at the first RP and the most recent RP. Nearly all items showed increases from the first RP to the last. The largest increase was in working to ensure that prevention activities are incorporated in other organizations; only 45 percent of communities reported completing this activity at the first RP, whereas the majority (66%) had completed it at the last RP. Communities also showed increases in efforts to leverage other funding sources, from 31 percent engaging in this activity at the first RP to 48 percent at the last RP. PEP-C Annual Reports October

58 Community-Level Findings Exhibit 39. Percentage of Communities Completing Sustainability Activity Types at First and Last Reporting Period Evaluation and Monitoring Index Exhibit 40 shows mean scores on the Evaluation and Monitoring Index over time. The main effect of RP was significant (log odds = 0.44, t = 21.8, p <.001, OR = 1.54), indicating a strong increase in evaluation and monitoring activities over time. Pairwise comparisons on individual RPs indicated statistically significant increases in evaluation and monitoring activities in almost every RP during the course of the grant, suggesting that communities undertook more evaluation of their substance use prevention activities as the grant progressed. PEP-C Annual Reports October

59 Community-Level Findings Exhibit 40. Mean Evaluation and Monitoring Index Score by Reporting Period Note. Blue reporting periods (RPs) represent statistically significant changes compared with the previous RP. The range for the Evaluation and Monitoring Index score is 0 6. Exhibit 41 shows the percentage of communities reporting that they worked on evaluation and monitoring activities in the first RP and the most recent RP. The biggest increase was in the number of communities that had developed an evaluation plan (from 25% to 82%) Across all RPs, only about 11 percent of communities never said that they developed an evaluation plan in any RP. Of those, the majority (65%) had only one or two RPs of reported data in the CLI Part I, and therefore may not have yet completed an evaluation plan. PEP-C Annual Reports October

60 Community-Level Findings Exhibit 41. Evaluation and Monitoring Activities Completed by Communities at the First and Last Reporting Period Note. SPF SIG, Strategic Prevention Framework State Incentive Grant. 5.3 Was Community-Level Change in Infrastructure, Resources, and Funding Related to Community Characteristics? (Evaluation Research Question 4B) Funded community organizations that were community coalitions tended to see greater growth over time in capacity building, evaluation and monitoring, and sustainability activities than funded community organizations that were neither coalitions nor partnered with coalitions. To examine whether community characteristics influenced community-level outcomes, we added additional (Level 2, time invariant) predictors to the HLMs described in the previous section. Predictors included (1) the number of implementation barriers ever reported; (2) coalition status (coalition, partnered with a coalition, or neither); (3) whether the community targeted a racial/ethnic group other than White non-hispanic (referred to for brevity as minority-serving communities 12 ); and (4) the number of data sources that communities used during 12 Note that for some SPF SIG communities (e.g., tribal and Pacific/Caribbean jurisdiction communities), the racial/ethnic group targeted is actually the majority population. The term minority-serving is used for brevity only, although it may not accurately reflect the relative size of racial/ethnic populations within all communities. PEP-C Annual Reports October