Evaluation Strategies for AIAN Community Tobacco Messages, Materials, Intervention (sort of) and Strategies Brenda F. Seals, PhD (Eastern Band Cherokee) Executive Director, NACR Linda Burhansstipanov, DrPH (Cherokee Nation of Oklahoma); Grants Director, NACR Native American Cancer Research (NACR) Headquarters: 393 South Harlan Street, suite 125 Lakewood, CO 80226-3571 Grants Office: 3022 South Nova Road Pine, CO 80470-7830 1-800-537-8295 http://www.natamcancer.org Most common reasons for programs not being re-funded: 1. Insufficient number of AIAN community participants 2. Poor data management 3. Insufficient evaluation strategies and/or not comprehensive (e.g., limited to satisfaction assessments) 2 537-8295 http://www.natamcancer.org 1
How do you measure success? Evaluation Surveys -- self-administered By Mail (typically poor community response, ~<15%) By handout Distributed during Community event at exhibit and PowWow 3 How do you measure success? Typically poor community response, may only get 20%, unless an incentive is included) By CSAQ (Computerized Self- Administered Questionnaires) (without human one-on-one presence still insufficient feedback from community) 4 537-8295 http://www.natamcancer.org 2
Surveys or Questionnaires (administered by trained interviewer) By Telephone (Native interviewer from local area typically gets better response than Native located outside of the area) Audience Response System for group data collection Face-to-Face and/or one-on-one (in home, at work) 5 Central location intercept interviews Survey information collected at Pow-wows, social gatherings, etc. Screening location Market Elders Social gathering In front of casino (with permission and approvals) 6 537-8295 http://www.natamcancer.org 3
Written responses to selected requests (e.g., personal journals, activity logs, debriefing info) Review of existing data (e.g., data collected for the diabetes program) In-depth personal interviews Focus Group (qualitative and quantitative) Working Group / Advisory Group 7 Webinar (with product like Survey Monkey for assessments) Audience Response System (high tech) Theatre Testing (every 5 minute response of how well you like something) Coffee stir sticks and no electricity (no tech) 8 537-8295 http://www.natamcancer.org 4
Observations studies Unobstrusive with Radio Shack nodes for pressure on a mat covering the nodes Teens counted smokers outside the PowWow area Observation checklist Tobacco Friendly Store Checklists (height of tobacco products visible to children) 9 Readability testing (variation of Word Flesch-Kincaid formulas paragraphs vs. page vs. doc) Expert review (Cultural and Scientific Panels) Gatekeeper review (tribal elder / leader) 10 537-8295 http://www.natamcancer.org 5
Media tracking Printed materials (brochures, fliers, posters) How many copies were given out? How do you know if they were read? How do you know if the people understood what they read? How did the message influence the community s participation in your tobacco program? 11 Media tracking Audio (radio, PSA, cassette tape) How many listened? Did they understand the message? How did the message influence the AIAN community s participation in the tobacco program? Audiovisual (videotape, TV, PSA) Same questions as above 12 537-8295 http://www.natamcancer.org 6
Exhibit / Educational Booth (e.g., Pow Wow, Health Fair) Slash marks Red beads / black beads with baskets 13 Evaluating Clinical Services How many times providers implement an intervention (e.g., cessation and success for those who have worked in the clinic less than 18 months) How good is the lab or technician? (e.g., blood cotinine and insufficient samples) 14 537-8295 http://www.natamcancer.org 7
Evaluating Clinical Services Number of participants attending a culturally specific cessation sessions; number who did not keep appointment; Did not attend but called to inform staff of inability to attend. Talk with AIANs before and after the cessation session to see how their experience was. 15 Evaluating Clinical Services Screening services (continued): Ask how provider helped her/her feel comfortable to ask her/him questions. Ask how provider helped her/her feel respected during the session (e.g., respect for ceremonial use). 16 537-8295 http://www.natamcancer.org 8
Evaluation Group #2 The local AI community filmed a public service announcement to be played on TV / Cable stations What are some ways you can learn: a. How often the PSA is playing (other than asking the cable company)? b. When it is playing? c. How well the PSA recruitment is working? 17 Evaluation Group #3 The local urban AI program is collaborating with the State s Tobacco Quitline. What are some ways you can learn: a. How culturally appropriate are the Quitline Operators behaving? b. How accurate and relevant are their statements about AIAN tobacco use? c. How well are they providing results to the local AIAN community? 18 537-8295 http://www.natamcancer.org 9