Creating an Evaluation Plan
Where did we go yesterday? Inputs What are the health issues in your community? Build the case for health issues by using data Activities What are evidence-based programs known to work for your goal? Find an evidence-based program for your program goal Community Engagement
Where are we going today? Inputs What are the health issues in your community? Build the case for health issues by using data Activities What are evidence-based programs known to work for your goal? Find an evidence-based program for your program goal Outcomes What is the goal of my program? Develop an evaluation plan to assess whether you accomplished your goal and reached the intended group Community Engagement
Overview of this Module 1. What are the different types of evaluation and what s in it for me? a. Reach b. Implementation c. Effect 2. Learn about different surveys/tools 3. Lab time
What s in it for me? What can I learn from evaluating my program? How will evaluating my program help me in the future? How will it help my target population?
Evaluation informs what s working and what s not Alamosa Farmer s Market Findings WIC* participants are eating more fruits than vegetables. WIC participants are not taking recipe cards or trying foods as much as seniors. Half of WIC participants said they wouldn t have gone to Farmer s Market if there hadn t been vouchers. * WIC - Special Supplemental Nutrition Program for Women, Infants and Children
Evaluation informs where to put your $$$ next time Summer School Program/Livewell Alamosa Findings Nutrition class led to increased knowledge, more positive attitudes towards veggies, healthier eating Archery and tennis lessons didn t show any changes The following year, Livewell decided to put $$ towards just nutrition classes
Evaluation Tips Always remember the PRE TEST! Consult your evaluation person early and often About 10% of your budget should go towards evaluation. Unless you are trying to establish that your program is evidence-based in which case you need way more than 10%!
Evaluation 101 Evaluation tells you if your program: REACHED your target population was IMPLEMENTED as expected had the EFFECT or outcomes you expected
Increasing Colon Cancer Screening Among Morgan County Latinos Using Promotor Intervention Inputs What are the health issues in your community? Morgan: 33% Latino CO colon cancer screening: Latinos=34% Whites=51% CO colon cancer death rate (2005): Latinos= 28/100k Whites= 19/100k (opposite US trend) Activities What are evidencebased programs for your goal? Promotor will: 1.conduct tailored one-on-one education about screening 2.assist participant in setting appointment 3.accompany participant on appt Outcomes What is the goal of my program? Reach: 200 uninsured Latinos Implementation: Promotor delivers oneon-one education, assist in setting appt, accompany patient Impact: 70% will have valid screenings, 80% increase in knowledge, 90% will advance in stages of change Community Engagement: Community members as navigators, listening to residents for risk factors/ barriers/ issues, go where residents WANT to go, form partnerships (clinics, referrals, transport, media, champions)
Did you reach your target population? How many people participated? What were their ages, gender, income levels? What were their previous behaviors, risk factors, health conditions?
Ways to measure REACH Sign in sheets with basic demographics Internal record keeping forms noting who participated Participant surveys with questions about demographics Refusal surveys with questions about demographics Could compare any of the above 4 to county/district level data.
Promotor Tracking Sheet Use this form to record information about all the people you visited with about getting colorectal screening. Female Male African American Hispanic or Latino Asian or Pacific Islander American Indian or Alaskan Native White Other Ethnicity Unknown Ethnicity Uninsured Insured Never Been Screened Yes, Has Been Screened Tally Total
Using a survey to measure REACH
REACH: Did your program reach the intended audience? N=200 Latino White, Non Hispanic
REACH: Of the 180 Latinos reached, what % were below the poverty level? N=180 Above 200% FPL Below 200% FPL
Did the program get implemented as expected? Which parts of the program were delivered as planned? Which parts weren t delivered as planned and what happened? What components were used more or less?
Ways to measure IMPLEMENTATION Debriefing forms Tracking forms (type of activity, date, description) Check lists (which parts of the program were implemented, by who, quality)
Example of Assessing Implementation Place a if you did the following with this patient: Educate patient about CRC screening? Explore the patient s barriers to screening? Review all available methods about CRC screening? Help schedule the appointment? Remind the patient about the appointment? Review the bowel preparation information? Organize transportation to appointment? Accompany patient to appointment?
Example of Assessing Implementation How many minutes took you to do the following with this patient: _15 Educate patient about CRC screening? _30 Explore the patient s barriers to screening? Review all available methods about CRC screening? Help schedule the appointment? Remind the patient about the appointment? Review the bowel preparation information? Organize transportation to appointment? Accompany patient to appointment?
Promotor: IMPLEMENTATION % of Latino patients this occurred 100 90 80 70 60 50 40 30 20 10 0 Education about CRC Helped schedule appt Accompanied patient to appt
Did you get the effect you were hoping for? Did knowledge, beliefs, behaviors change? Did some participants benefit more than others? Was the amount of participation related to level of outcome? Probably use a pre/post design
Ways to measure EFFECT Pre/post surveys re: knowledge, skills, self-reported behaviors Participants maintain journals to record behaviors Objective measurements (e.g., pedometers) Direct observations Interviews, focus groups
Expected outcomes should be reasonable for the level of intensity of the intervention Single public education sessions are likely to affect knowledge, awareness, and maybe attitudes, but probably not behavior Multi-session interventions may affect skills and behavior Is it reasonable to say your intervention will change body mass index???
Tracking Screening Rates
Resources to Measure Effect Health Maintenance Consortium http://hmcrc.srph.tamhsc.edu/measures/me asuresmain.html Quality of Life Depression and stress Self-Efficacy Social Support Environment Anthropometrics (height, weight, blood pressure) Physical Activity Nutrition Smoking Drug and Alcohol
Health Maintenance Consortium http://hmcrc.srph.tamhsc.edu/measures/measur esmain.html
Resources to Measure Effect Cancer Prevention Research Center http://www.uri.edu/research/cprc/measures.htm Stages of change, self-efficacy Smoking, alcohol, cocaine Mammography Exercise Sun protection Coping and stress Weight control Psychotherapy HIV and Safer Sex Substance Abuse
Resources to Measure Effect Physical Activity & Diet Measures for Kids, Families, and Schools http://www-rohan.sdsu.edu/faculty/sallis/measures.html Child and adolescent physical activity behaviors Dietary fat screening Neighborhood walkability Neighborhood quality of life Also helpful for community assessment physical environment: http://www.activelivingresearch.org/resourcesearch/to olsandmeasures
Resources to Measure Effect Fred Hutchinson Cancer Research Center Diet: http://www.fhcrc.org/science/shared_resour ces/nutrition/index.html Food frequency questionnaires Soy, fat, caffeine questionnaires 24 Hour Dietary Recalls Multiple Day Food Records
Stanford Chronic Disease Self-Management Program Evaluation Tools at http://patienteducation.stanford.edu/research/
scroll down.
Click on Exercise.
Quality of Life tools These sites have validated, free instruments to measure QOL: 1) from the WHO http://www.who.int/substance_abuse/re search_tools/whoqolbref/en/ 2) from the McMaster University and the HUG group: http://www.fhs.mcmaster.ca/hug/
Lab Time: Work on Eval Plan How will you measure effect? What types of changes is the evidencebased intervention you selected known to bring on? What existing tools can you use to measure effect?