SPF SIG process we CAN change communities if we pay attention to the details! (An initial look at outcomes

Transcription

1 SPF SIG process we CAN change communities if we pay attention to the details! (An initial look at outcomes from Washington s SPF SIG) Scott Waller, Washington State Division of Behavioral Health and Recovery

2 Today s Presentation Review of Washington s SPF SIG evaluation design. Early results. Implications. Details matter! 2

3 Brief Review Washington funded in Fall SPF SIG communities funded in Spring Randomized research design. Prevention priority: Reduce underage drinking and associated problems. 3

4 Research Questions 1 and 2 1. Does implementing the SPF lead to better outcomes? 2. What explains differences in outcomes? Characteristics of the communities. Differences in prevention efforts. 4

5 To answer the 1st question we need to be able to compare SPF SIG sites to sites without the SPF SIG. Therefore: Random selection of sites.

6 Research Question 1 Evaluation Design SPF SIG Communities (n = 12) (N = 5,252) Non urban, low poverty, low minority (n = 4) Non urban, high poverty, high minority (n = 5) Urban (n = 3) Comparison Communities (n = 35) (N = 12,694) Non urban, low poverty, low minority (n = 20) Non urban, high poverty, high minority (n = 9) Urban (n = 6) 6

7 Random selection of sites. Identified eligible sites Clustered into community types Drew from each cluster The rest are comparison sites

8 Geographic distribution of SPF SIG sites and comparison communities Burlington Pt. Angeles Port Gamble S Klallam Tribe Tacoma Seattle (2) Naches Wenatchee Warden Kelso White Swan Asotin

9 Chronology of Washington SPF SIG Activities: How long has it been? Year Year Capacity building (cont d) Implement programs (SPF Step 4) Evaluate Programs (SPF Step 5) Revisit Needs Assessment (SPF Step 1) Capacity building (cont d) Implement programs (cont d) Evaluate Programs (cont d) Capacity building (cont d) Strategic Plans (cont d) Implement Programs (SPF Step 4) Evaluate Programs (SPF Step 5) 5 Year Year State Activity Fiscal approval of strategic plan Community applications Select communities 1 Establish Prevention Priority SEW, JOT, IMT Advisory Council Est. Submit state strategic plan Community Activity Capa city/ coalition building, (SPF Step 2) Needs Assessment (SPF Step 2) Strategic Plans (SPF Step 3) Year Year

10 Logic Model for Washington State SPF SIG Priority: Reducing Underage Drinking Consequences Consumption Intervening Variables Contributing Factors Strategies Mental Health Underage Drinking: Access to Alcohol Feeling sad or depressed Suicide attempts Experimental Social Retail School Coming to school drunk or high Academic failure Underage Drinking: Problem Enforcement of Alcohol-Related Policies School Community Traffic Safety Underage Drinking: Risk Factors Drinking and driving and riding with a drunk driver Heavy Early first use Perceived availability Perceived harm Perceived detection Community laws and norms 10

11 Research Questions 1 and 2 1. Does implementing the SPF lead to better outcomes? 2. What explains differences in outcomes? Characteristics of the communities. Differences in prevention efforts. 11

12 To answer the 2 nd question Compare the SPF SIG sites to each other on important aspects of the project. We have extensive data, measuring many factors. 12

13 How will we measure the outcomes? Cohort or cross sectional? Alcohol use but specifically, What age group?, what level of alcohol use?

14 Cohort and Grade Level (Cross Sectional) Comparisons Over Time Year Grade Cross sectional or grade level. Longitudinal or cohort trend.

15 Defining Underage Drinking: A Composite of 30 Day Alcohol Use & Binge Drinking 15

16 WA SPF SIG Outcome Analysis Designs: Four Forests for Looking at the Trees SPF SIG & Comparison Communities Cross-Sectional Analysis Has the SPF SIG project made a difference in underage drinking and related problems at specific grade levels over time? Cohort Analysis Has the SPF SIG project made a difference in the developmental trajectory of underage drinking and related problems among youth over time? SPF SIG Communities Only What characteristics and activities of SPF SIG communities are related to making a difference in underage drinking and related problems at specific grade levels over time? What characteristics and activities of SPF SIG communities are related to making a difference in the trajectory of underage drinking and related problems among youth over time? 16

17 Data Analytic Approaches Analysis of Variance (ANOVA), representing underage drinking as a single, ordinal outcome variable, ranging from 1 to 4. Multinomial logit and Hierarchical Linear Modeling (HLM) analysis. Nesting students (Level 1) within schools (Level 2). 3 dependent variables: Contrasts of each of the three levels of drinking with the no drinking category as reference. 17

18 Grade 8 Trends in Alcohol Use (Composite): Alcohol Composite State Sample SPF SIG Comparison Year Scale: 1 = no use, 2 = experimental, 3 = problem, 4 = heavy 18

19 Some Possible Reasons Our measurement tools were not sophisticated enough to show changes There is an overall downward trend to alcohol use in Washington so we were coming in with SPF SIG at a time when rates were going down anyway Comparison communities had significant prevention and early intervention work going on so the results between them and SPF SIG communities would be similar 19

20 Analysis of These Trends Differences in trends between SPF SIG and Comparison sites are not statistically significant. Highly significant 3 way interaction: Study Group by Year by Cluster, (i.e., the SPF SIG vs. Comparison sites trends over time are different for the 3 demographic clusters). 20

21 Study Group x Year Interaction for Cluster 1: Non Urban, Low Poverty, Low Minority Sites Alcohol Composite SPF SIG Comparision Year Scale: 1 = no use, 2 = experimental, 3 = problem, 4 = heavy 21

22 Study Group x Year Interaction for Cluster 2: Non Urban, High Poverty, High Minority Sites Alcohol Composite SPF SIG Comparision Year Scale: 1 = no use, 2 = experimental, 3 = problem, 4 = heavy

23 Study Group x Year Interaction for Cluster 3: Urban Sites Alcohol Composite SPF SIG Comparision Year Scale: 1 = no use, 2 = experimental, 3 = problem, 4 = heavy

24 Same Analysis, Different Outcome Measure Constructed a Total Risk and Total Protection scale for use as a more proximal outcome measure. The % of risk (protective) factors on which each youth is deemed at risk (protected), using empirically derived cutoff scores on each risk/protective factor established through research. Similar results to those of the alcohol use (composite) analysis for Total Risk scale, fewer and less significant effects for Total Protection scale. 24

25 Study x Year Trends in Total Risk for Cluster 2: Non Urban, High Poverty, High Minority Sites Risk 40 SPF SIG 38 Comparision Year

26 Study x Year Trends in Total Risk for Cluster 2: Non Urban, High Poverty, High Minority Sites Risk 40 SPF SIG 38 Comparision Year

27 Study x Year Trends in Total Risk for Cluster 3: Urban Sites Risk SPF SIG Comparision Year

28 Key Relationship Between Total Risk, Total Protection and Alcohol Use Alcohol Composite High Medium Low Low Medium High Protection Scale: 1 = no use, 2 = experimental, 3 = problem, 4 = heavy

29 Specific Characteristics for Comparisons Among SPF SIG Sites Strength of Implementation of the SPF model 44 activity rubrics developed by cross state SPF SIG workgroup. Coalition survey results. 9 scales (3 subscales). Community survey results. Permissive attitudes toward youth alcohol use. P/I program. Presence or absence. 29

30 Specific Characteristics for Comparisons Among SPF SIG Sites s.asp?category=23&category_type=publicgeneral 30

31 Specific Characteristics for Comparisons Among SPF SIG Sites 31

32 Specific Characteristics for Comparisons Among SPF SIG Sites Needs assessment management: an authorized entity (e..g., a data management workgroup or individual has been identified and charged with collecting, reviewing and analyzing community level data on substance abuse related (1) consequences, (2) consumption patterns, (3) geographic/target population differences, (4) intervening variables (such as risk and protective factors), (5) prevention resources, and (6) community readiness to address the targeted issue and/or contributing factors. Missing from plan No organization or individual has been identified Weak Fidelity to perform this role. An existing or new group/individual has been tasked with some of these responsibilities but the charge does Moderate Fidelity not include collecting/acquiring, analyzing and The entity has been identified and the charge Strong Fidelity reporting data necessary to address all six of the core clearly incorporates data collection/acquisition, data areas. analysis, and reporting on all six of the core data The entity has been identified and the charge areas, but the project provides limited support clearly incorporates all collect/acquire, analyze, for data acquisition. and report data on the six core data areas. 32

33 Specific Characteristics for Comparisons Among SPF SIG Sites Program/Strategy Penetration Rates. Youth direct services (YDS). P/I selective/indicated services. Family direct services. Parent focused environmental strategies. Enforcement/policy focused environmental strategies. Multi year exposure to YDS. YDS facilitator buy in.

34 SPF SIG Only Comparisons 7 of 13 Factor by Year interactions were statistically significant (p <.01). Three were in theoretically hypothesized direction. Strength of Implementation of SPF model (p <.001). Penetration/Reach of Enforcement/Policy related environmental strategies (p <.001). Strength of Coalition Leadership (p <.001). 34

35 Trends in Alcohol Use (Composite) by Level of SPF SIG Implementation Alcohol Composite State Sample SPF SIG Low (n = 3) SPF SIG Med (n = 5) SPF SIG High (n = 4) Year

36 Using the Fidelity Rubrics SPF Step 1: Profiling Needs, etc. (10 core activities). SPF Step 2: Building Capacity (9). SPF Step 3: Creating Strategic Plan (8). 36

37 Using the Fidelity Rubrics SPF Step 4a: Selecting and Implementing EBPs (6): SPF Step 4b1: Implementing Participant based Interventions (6 core issues). SPF Step 4b2: Implementing Environmental Strategies (4 core issues). SPF Step 5: Evaluation and Monitoring (11). 37

38 Fidelity Rating Scales: An Example SPF Step 1, Key Component: Data Collection. 0 = No data reported for any of the areas (consequences, consumption, etc.). 1 = Some data reported, but inadequate (all areas not addressed, poor reliability/validity of data, no trends over time, etc.). 2 = All areas included, but data access issues preclude multiple estimates for each area. 3 = All areas included with trends over time, comparable data and multiple sources. 38

39 Total Risk by Level of SPF SIG Implementation Total Risk SPF SIG High (n = 4) SPF SIG Med (n = 5) SPF SIG Low (n = 3) Comparision None (n = 35) Year

40 Total Protection by Level of SPF SIG Implementation Total Protection SPF SIG High (n = 4) SPF SIG Med (n = 5) SPF SIG Low (n = 3) 50 Comparision None (n = 35) Year

41 Trends in Alcohol Use by Penetration of Enforcement/Policy Environmental Strategies Alcohol Composite SPF SIG High (n = 4) SPF SIG Low (n = 4) None (n = 4) Comparision (n = 35) Year Scale: 1 = no use, 2 = experimental, 3 = problem, 4 = heavy

42 Trends in Alcohol Use by Strength of Coalition Leadership Alcohol Composite SPF SIG High (n = 4) SPF SIG Med (n = 5) SPF SIG Low (n = 3) Comparision (n = 35) Year Scale: 1 = no use, 2 = experimental, 3 = problem, 4 = heavy

43 Summary There has been a significant reduction in alcohol use among Washington s eighth graders since 2004 at a statewide level, among SPF SIG sites, and among Comparison sites. Overall, there are no significant differences between SPF SIG and Comparison sites over time in key outcome measures. SPF SIG and Comparison sites do differ significantly within demographic clusters. Analyses to date do not include data on prevention activity in study sites. Preliminary not for dissemination. 43

44 Summary (cont.) Among SPF SIG sites, there are statistically significant and theoretically consistent relationships between three key factors and reductions in total risk and alcohol use. Fidelity of Implementation of SPF model. Penetration rate of Enforcement/Policy environmental strategies. Strength of Coalition Leadership. 44

45 Possible Directions for Further Analysis Develop further the Resource Assessment data to sharpen the SPF SIG vs. Comparison sites trends. Probe further into SPF Implementation Fidelity (e.g., which specific SPF steps are most strongly related to reductions in alcohol use and total risk). Probe further into specifics of enforcement/policy environmental strategy implementation. Others??? 45

46 Thank YOU for Your Time! Questions or Comments? Please contact: Scott Waller, Prevention Integration Lead Division of Behavioral Health & Recovery (360)