This plan was created by the Evaluation Work Group which was made up of local and state partners.

Transcription

1 The Wisconsin Tobacco Prevention and Control Program () coordinated local and state partners to form an Evaluation Work Group to create an Evaluation Plan for the statewide movement. This plan covers 2016 through 2020 and focuses on the zoomed out, system-level tobacco prevention and control movement, instead of specific programs that work towards prevention and cessation of tobacco. This plan was created by the Evaluation Work Group which was made up of local and state partners. Name Organization Partner Level Vicki Huntington (DHS) State Randy Glysch (DHS) State Disa Patel (DHS) State Michelle Mercure American Lung Association State Andrea Gromoske Family Health Section (DHS) State Lisette Khalil Wisconsin Women s Health Foundation State Elaine Valliere Great Lakes Inter-Tribal Council Tribal Lorraine Lathen Jump at the Sun, LLC Local Bruce Christiansen UW-Center for Tobacco Research Intervention Academic Karen Palmersheim UW-Center for Urban Initiatives and Research Academic Robin Moskowitz UW-Population Health Sciences Academic John Stevenson UW-Survey Center Academic Elizabeth Hagen Western Wisconsin Working for Tobacco-Free Living Local Last Revision: Aug 2016

2 Evaluation Question Overview Based off the Logic Model (Appendix A), the Evaluation Work Group determined overarching evaluation questions that are answered through the evaluation indicators found in Appendix B. Cessation: Are Wisconsin cessation programs (i.e. Quit Line, First Breath, Freedom from Smoking, Community Programs, WiNTiP, UW-CTRI Outreach) effective in reaching vulnerable populations? Rationale: Increased awareness of services Increased use of services Decreased smoking rate Indicators: 1-11 Disparities: Is the movement effective in reaching populations disparately impacted by tobacco? Rationale: Increased resources created for and on disparate and vulnerable populations Increased activities address disparate populations Decreased disparity between populations using tobacco Indicators: 1-2, Initiation: Is the prevalence of tobacco products decreasing among youth and young adults? Rationale: Decreased access to products Decreased use of new and old products (including e-cigarettes) Decreased prevalence of tobacco initiation and use among youth and young adults Indicators: 1-2, Secondhand Smoke Exposure: Is the prevalence of secondhand smoke exposure decreasing among youth and adults in the home? Rationale: Increased smoke-free housing Increased cessation efforts Decreased exposure to youth and adults Indicators: Intended Use and Users This plan s results will be used to monitor progress of the movement. Results are to be shared with stakeholders (e.g. topic specific work groups, funders, DHS leadership, partners, media, public) through various channels (e.g. press release, fact sheets, reports) as detailed in the Dissemination Plan. Primarily, results will be used by, funded partners, and the statewide coalition. The statewide coalition is going through a reformation process in 2016, and will receive evaluation results from the Evaluation Work Group after its creation. The statewide coalition will identify needs based on recommended focus areas from the Evaluation Work Group. will determine any adjustments to the implementation of activities and strategies (process measures) along with the impact of the program (outcome measures). Additionally, results will be used to ensure effective reporting and shared with partners and the public to 2

3 educate on program activities and outcomes. The statewide coalition will use the evaluation results to determine their policy focus for the upcoming years. Movement Description The statewide (referred to as movement ) is supported through the efforts of the Department of Health Services Tobacco Prevention and Control Program () and is illustrated in the movement s logic model (Appendix A). coordinates efforts from local coalitions, statewide non-profits, academic settings including outreach and researchers, other Department of Health Services programs, and national partners to ensure synergy among activities and outputs. brings guiding plans (e.g. Healthy Wisconsin 2020, Wisconsin Tobacco Prevention and Control State Plan) to the forefront for works groups and the statewide coalition to plan and strategize activities. Additionally, the movement works within environmental factors that can have both positive and negative consequences. These factors include past and current tobacco science and research; local, state, and national policies; and the tobacco industry s marketing and products. A majority of activities are led by local coalitions and state partners. Local coalitions educate stakeholders on tobacco prevention and cessation best practices and serve as content experts to aide in implementing local policies as needed. State partners provide services such as the Quit Line and cessation outreach specific to pregnant women and behavioral health care workforce and population (i.e. First Breath, WiNTiP, UW-CTRI Outreach). Additionally, state partners coordinate activities of statewide youth and young adults efforts (i.e. FACT, N-O-T, Spark, Wisconsin Wins), and of smoke-free housing efforts (i.e. Clear Gains). These activities lead to several outputs which are captured in the movement s logic model (Appendix A) and key process outcomes are highlighted in the Evaluation Question Matrix (Appendix B). The long-term outcomes of the movement are to eliminate tobacco-related disparities, eliminate secondhand smoke exposure, increase cessation, prevent youth initiations, and have an effective, sustainable infrastructure. To reach these, the Evaluation Work Group determined short-term and intermediate outcomes that build to the long-term outcomes. These short-term and intermediate outcomes directly relate to the CDC required outcomes that were laid out in the CDC FOA DP Appendix A has a logic model that overviews the movement s inputs, activities, outputs, and outcomes. The statewide movement is multifaceted and is experiencing a couple stages of development. Overall, the movement is in maintenance as various components have evolved responding to environmental contexts such as the implementation of a statewide smoke-free air law and increased taxes on tobacco products. This has shifted the focus of the movement to educating on new products (i.e. e-cigarettes) and to refocus efforts on addressing tobacco-related disparities, which is one facet of the movement that is in the planning stage. Other facets in the planning stage are the statewide coalition as it is being reformed, and planning work with Counter Tools to gather data systematically on the local level that will tell a statewide story. These efforts are currently being strategized. 3

4 Evaluation Focus The Evaluation Work Group prioritized questions within each long-term outcome. These questions were narrowed using specific criteria and assumptions. Similar questions were then merged and adjusted to be indicators. Additionally, indicators include a statement that demographic make-up will be determined for a majority of the evaluation questions. Although it is assumed looking at demographics will occur as part of assessing indicators, this is specifically stated to ensure assessing disparate populations occurs. These demographics will include: race/ethnicity, sexual orientation, gender identity, income, pregnant women, or education level. Additionally, questions that were not ranked by the group, questions without a viable data source, and questions that were program specific (e.g. FACT participation rate) were removed from the table. An assumption being made is that each review will ask, What are the changes over time? which will help in identifying trends. Overall, thirty evaluation indicators (twelve required by CDC as performance measures) were prioritized. proposed overarching evaluation questions (see Questions Overview) that encompasses and uses the thirty indicators as data to analyze. The thirty indicators are feasible to answer as a majority of the questions on the matrix are already gathered through surveillance. For those indicators that are not gathered through annual/bi-annual surveillance surveys, case studies can be conducted using staff to analyze current reporting on contractors activities (e.g. environmental scans). is confident using the Evaluation Indicators Matrix is an efficient use of time and resources. Furthermore, the creation of an internal tracking system will ensure questions are answered and discussed with the Evaluation Work Group for next steps. Stakeholder Engagement coordinated the Evaluation Work Group by using the CDC s Developing an Effective Evaluation Plan workbook. The Evaluation Work Group comprised of local and state partners who represent diverse organizations and who have experience in evaluation, research, program history, or surveillance. The Work Group convened for two in-person meetings where the group revised a movementwide logic model and brainstormed and prioritized evaluation questions categorized by the National Tobacco Control Program (NTCP) goal areas. proposed a movement-wide logic model that the group revised by adding partners and connections to ensure a logical flow by adding activities, outputs, and outcomes for addressing tobacco-related health disparities, ensuring an effective infrastructure, and other long-term outcomes. Additionally, the group brainstormed questions with no restraints to feasibility as members of the group may have been unaware of available data resources and this facilitated creativity among members. Evaluation questions were to be based on a review of available data and program activities shown through the logic model (Appendix A). A subgroup of the Evaluation Work Group began prioritizing questions and added questions that were missing based on the 4

5 logic model and program description. Afterwards, the entire group prioritized questions within each long-term outcome. then narrowed these questions using specific criteria and assumptions (see Evaluation Focus). The Evaluation Work Group decided to meet every six months with quarterly teleconferences to monitor the program s Evaluation Plan. Monitoring the Evaluation Plan allows to measure program implementation and impact, and identify areas of focus to improve overall program quality. Partners aid in choosing feasible evaluation and performance measures that best support s four priority goals of the State Plan. Furthermore, the Evaluation Work Group will collaborate to determine which results should be shared with stakeholders. Indicators and performance measures Key evaluation questions and methods for answering questions are in Appendix B, Evaluation Question Matrix. These questions include required CDC Performance Measures. The Evaluation Work Group and focused the evaluation questions. After these were focused, shared overarching questions with the Evaluation Work Group that could be answered with the Evaluation Questions Matrix. Data sources has developed and will use data collection systems to collect process outcomes and measure effectiveness of program activities. Program reporting will encompass local and state partners activities, and will monitor implementation of activities such as media, resource development, and funding level. In addition to contractors and s program reporting, the Research Scientist will continue to coordinate state surveillance systems with the Behavioral Risk Factor Surveillance System (BRFSS) and the Youth Tobacco Survey (YTS). Furthermore, the Research Scientist will work with the Maternal and Child Health Program (MCH) and the Division of Health Care Access and Accountability (DHCAA) to analyze birth certificates, Pregnancy Risk Assessment Monitoring System (PRAMS), Women, Infants, and Children (WIC), and Medicaid data. Methods Various methods will be used to assess both process and outcome measures. Majority of the information comes from either annual or bi-annual surveys (e.g. BRFSS, YTS) or from or contractor reporting. This aids in feasibility as a majority of the information has been gathered for several years. Also, the Department of Health Service continues to encourage program integration to aid collaborative efforts. will work with MCH along with DHCAA to answer evaluation questions related to infant exposure to secondhand smoke and Medicaid use of cessation treatment. is currently in the process of creating an internal tracking system to ensure questions are answered and tracked when reported. Part of the criteria for including an evaluation question was the ability to adequately measure the question and the most appropriate method and data exist to answer the question, and if the appropriate resources are available to accomplish this. Quality assurance procedures (e.g. 5

6 ensure adequate sample size, reliability, viability of data sources) will be put into place to ensure that the data used to answer the evaluation questions are reliable and checked for accuracy. Both quantitative and qualitative data collection methods will be used to collect the needed data to answer the evaluation questions, and only data with acceptable confidence intervals and representative samples will be used. Data sources being used are credible, national sources. Roles and responsibilities The Research Scientist will track information when survey data becomes available, and program data will be collected at mid and year-end reports. Questions and data will then be shared with the Evaluation Work Group. The Evaluation Work Group will then determine if any adjustments are needed with program activities to reach project period objectives. Credibility of evaluation information The majority of data sources are state or national surveillance surveys, lending to highly credible information. Other sources are program reporting which is reliable as it is reported on at least a quarterly basis. Creating an internal tracking system for will aid in reliable data as information will be collected regularly. Even with the use of these data sources, there is a need to focus data gathering for disparate populations. will capture more reliable statewide LGBT prevalence data in the 2015 BRFSS, has added an LGBT question to the Quit Line, and will add questions to BRFSS and conduct oversampling as needed to ensure adequate sample size for various population groups. Analysis and Interpretation Plan The Research Scientist will coordinate and collaborate with members of the Evaluation Work Group to analyze data from the Evaluation Questions Matrix. These interim results will be shared with and the Evaluation Work Group through the Evaluation Questions Matrix. The Evaluation Work Group will hold quarterly calls to assess if enough information has been collected to answer any of the overarching questions. If data is gathered during a time the Evaluation Work Group would not be meeting, either an additional call will be scheduled or will propose conclusions to the Evaluation Work Group. The Evaluation Work Group will critique the data gathered and ask additional questions associated with the movement logic model. Additionally, the group will formulate conclusions from the evaluation to be shared with appropriate stakeholders (e.g. topic-specific work groups, funders, DHS). These stakeholders will have the opportunity to provide feedback on conclusions, both positive and negative. The Evaluation Work Group will then use this information to determine if the conclusions have been thoroughly justified, and determine what reports should be shared with external partners. 6

7 Dissemination Plan The Evaluation Work Group will meet in-person twice a year to review data that has been collected. will share information from the Evaluation Question Matrix and the group will focus on the overarching questions that will aid in synthesizing the evaluation findings. The findings will be summarized for a yearly report. The yearly report (20-25 pages) will be created to detail progress made throughout the year and action steps for the coming year. The report will also be summarized (1-3 pages) for distribution to partners. A timeline for the Evaluation Work Group meetings and dissemination is found in Appendix C. Findings will be shared by NTCP goal area as the overarching evaluation questions are similarly categorized. has an infrastructure that includes topic-specific work groups: Youth and Young Adults, Multi-Unit Housing, Coalition Building, Communications, Other Tobacco Products, Sustaining States, and Disparities. These work groups will receive factsheets and/or issue briefs on the evaluation findings and proposed action steps as information is collected. These work groups will be able to use the information to determine appropriate next steps. will continue to coordinate the Evaluation Work Group to ensure effective communication and to facilitate ongoing program improvement. The group will determine results to be shared with stakeholders and update the Evaluation Plan s Dissemination Plan at in-person meetings. The group will inform the design of materials tailored for specific audiences. Target Audience Goals Tools Timeline Evaluation Work Group CDC Detail findings of the evaluation report and identify next steps. Share a success story identified by the evaluation results page Report Annually Local and state partners DHS Topic-specific work group Local and state partners Local and state partners, media, general public Share highlights of year and actions for the coming year Share findings from evaluation report for group to determine action steps Share plan summary (and logic model) with stakeholders to be transparent and ensure synergy for next five years Share tobacco-related data specific to Wisconsin 1-3 page summary Factsheet or Issue Brief Evaluation Plan 1-3 page summary Factsheet Annually 6-month review April 2016 Annually 7

8 Appendix A Movement Logic Model 8

9 Appendix B Evaluation Indicators Matrix Bold = CDC Required Performance Measure # NTCP Goal Area Process/ Outcome Method Indicator Data Source Frequency 1 All Outcome Report Proportion of cigarette smoking among youth and adults BRFSS BRFSS, Birth Birth Certificates Certificates, PRAMS PRAMS Annually YTS YTS Even Years YRBSS YRBSS Odd Years 2 All Outcome Report Proportion of tobacco use among youth and adults BRFSS YTS YRBSS BRFSS Annually YTS Even Years YRBSS Odd Years Responsibility Assumed unless otherwise stated YRBSS DPI YRBSS DPI # NTCP Goal Area Process/ Outcome Method Indicator Data Source Frequency Responsibility 3 Cessation Outcome Report Total quitline call volume by quarter Quit Line Quarterly UW-CTRI NQDW What is the demographic make-up of callers? 4 Cessation Outcome Report Total number of quitline tobacco users who receive a service Quit Line NQDW Quarterly UW-CTRI What is the demographic of users who receive a service? 5 Cessation Outcome Report Quitline reach to persons with low education levels Quit Line 6 months UW-CTRI NQDW 6 Cessation Outcome Survey Proportion of adult smokers who have made a quit BRFSS Annually attempt What is the demographic make-up of those who have attempted to quit? 9

10 NTCP Process/ # Goal Area Outcome Method Indicator Data Source Frequency Responsibility 7 Cessation Outcome Survey Proportion of young smokers who have made a quit attempt What is the demographic make-up of those who have attempted to quit? 8 Cessation Outcome Survey Proportion of adults aware of the Quit Line, First Breath, Community Programs, and Freedom from Smoking YTS YRBSS YTS Even Years YRBSS Odd Years BRFSS Annually YRBSS DPI 9 Cessation Process Survey Proportion of patients asked about their tobacco use by their dentist or doctor BRFSS Annually 10 Cessation Process Report Proportion of Wisconsin Medicaid recipients receiving cessation services DHCAA Medicaid Annually DHCAA 11 Cessation Outcome Report Number of electronic referrals occur with the implementation of Electronic Medical Records UW-CTRI 6 months UW-CTRI # NTCP Goal Area Process/ Outcome Method Indicator Data Source Frequency Responsibility 12 Disparities Process Program Proportion of interventions and strategies implemented address disparate populations Reporting 6 months 13 Disparities Process Program What is the breakdown of interventions by demographics? Number resources created for priority populations (LGBTQ, Mental Health, Low SES) Reporting 6 months # NTCP Goal Area Process/ Outcome Method Indicator Data Source Frequency Responsibility 14 Infrastructure Process Program Percentage of funding (state, CDC, and other) used to meet CDC-recommended funding levels outlined in Best Practices 2014 Reporting Annually 10

11 15 Infrastructure Process Program 16 Infrastructure Process Program 17 Infrastructure S&E 18 Infrastructure S&E 19 Infrastructure Communications 20 Infrastructure Communications Disparities Process Process Process Process Program Program Program Program Number and type of staff positions maintained throughout the entire funding year to support the tobacco control program (e.g., program director, policy coordinator, communications specialist, cessation coordinator, surveillance and evaluation staff, fiscal management systems staff, and administrative staff) Levels of infrastructure increased or maintained as defined by the Component Model of Infrastructure included in Best Practices 2014) Number and type of tobacco-related surveys implemented during the funding year (e.g. BRFSS, YTS) and type of tobacco-related modules implemented (e.g., BRFSS, YRBSS) Number and type of tobacco-related indicators developed and implemented in state surveillance systems during the funding year (e.g. ATS, YTS< BRFSS< YRBSS) Number of monthly speaking opportunities by trained tobacco control spokespersons to educate decision makers, stakeholders, and public Number of paid and earned media efforts that target populations or areas with high concentrations of smoking prevalence, secondhand smoke exposure, and chronic disease Reporting Annually Reporting Annually Reporting Annually Reporting Annually Reporting Annually Reporting Annually 21 Infrastructure Communications Process Program Types of social media activities used to complement traditional paid and earned media efforts (e.g., social media campaign, posting content) and what is the reach of social media activities by social media site used (e.g., Facebook, Twitter, YouTube) Tracker Annually # NTCP Goal Area Process/ Outcome Method Indicator Data Source Frequency Responsibility 22 Initiation Outcome Survey Average age at which young people first smoked a whole cigarette YRBSS YTS BRFSS YRBSS Odd Years YTS Even Years YRBSS DPI Age first tried tobacco products among youth and adults BRFSS Annually 11

12 23 Initiation Outcome Survey Proportion of young people who report never having tried a cigarette 24 Initiation Outcome Survey Proportion of youth able to buy tobacco YRBSS Odd Years WI Wins, SYNAR Annually 25 Initiation Outcome Survey Prevalence of e-cigarette use by youth, young adults, & adults YTS, BRFSS BRFSS Annually YTS Even years 26 Initiation Outcome Survey Locations youth are getting their tobacco products YTS Even Years 27 Initiation Outcome Program Number colleges and universities are smoke/tobacco-free Spark Annually Spark Disparities Reporting (i.e. tech schools, 2- or 4-year schools) 28 Initiation Outcome Reporting Number of cigarettes being sold per capita (adults) Monthly Revenue 6 months Data (i.e. county variance) # NTCP Goal Area Process/ Outcome Method Indicator Data Source Frequency Responsibility 29 SHS Outcome Program Proportion of public housing authorities with 100% smokefree Clear Gains Tracker 6 months Clear Gains Reporting indoor air policies 30 SHS Outcome Reporting Proportion of the population reporting exposure to TUS-CPS Annually secondhand smoke at workplace 31 SHS Outcome Survey Prevalence of SHS exposure in vehicles and homes for youth and adults BRFSS YTS BRFSS Annually YTS Even years 32 SHS Outcome Survey Proportion of households with infants have adult smokers Birth Certificates PRAMS WIC Annually 12

13 Appendix C Evaluation Work Group Timeline Date Activity Topics Channel 2016 March Disseminate Evaluation Plan to partners State and local partners October Evaluation Work Group in-person meeting to December Disseminate findings 2017 April Evaluation Work Group in-person meeting to June Disseminate findings October Evaluation Work Group in-person meeting to December Disseminate findings 2018 April Evaluation Work Group in-person meeting to June Disseminate findings October Evaluation Work Group in-person meeting to December Disseminate findings 2019 April Evaluation Work Group in-person meeting to June Disseminate findings October Evaluation Work Group in-person meeting to December Disseminate findings 2020 April Evaluation Work Group in-person meeting to June Disseminate findings October Evaluation Work Group in-person meeting to December Disseminate findings 13