Government Terminology Partial Translations for AIAN Communities (i.e., Government Terminology Used to Describe Effective / Successful Programs) Objective: By the end of this session, the participant will be able to: 1. Distinguish among government terminologies used to describe best / promising programs and practices Linda Burhansstipanov, MSPH, DrPH Cherokee Nation of Oklahoma 3022 South Nova Road Pine, CO 80470-7830 http://www.natamcancer.org This workshop is dedicated to Eduard Gamito, dear friend and colleague Born 1962 Passed January 22, 2008 Dear Ed, we miss you and we honor you Overview of Government Terminology for Best / Promising Practices 3 Please Don t Shoot the Messenger The federal / state program staff rarely understand Tribal sovereignty Consultation processes as used in Indian Country Historical Trauma and its impact on today s AIAN public health programs AIAN jargon and acronyms (IHS, 93-638, BIA, CHS) Overview of Government Terminology for Best / Promising Practices 4 QUESTION: What do you think of when you hear the term evidence-based (NOTE: tobacco is used as the examples for many of the slides)? An evidence-based program has been: Implemented with a group Evaluated Found to be effective Overview of Government Terminology for Best / Promising Practices 5 Purpose of Taking Time for These Phrases To help AIANs understand when and how miscommunication can occur with funding agency To avoid using phrases that have distinctive meanings for government bodies when AIANs submit applications We KNOW that AIANs use phrases differently than the government Overview of Government Terminology for Best / Promising Practices 6 1-800-537-8295 http://www.natamcancer.org 1
Phrase: Levels of Evidence (LEH-vulz uv EH-vih-dents) A ranking system used to describe the strength of the results measured in a clinical trial or research study. The design of the study The endpoints measured (such as tobacco cessation) affect the strength of the evidence. as modified on 02/17/08 from http://www.cancer.gov/templates/db_alpha.aspx?cdrid=446533 Overview of Government Terminology for Best / Promising Practices 7 Evidence Level 1 A program that is funded by a peerreviewed grant A panel of experts had to approve the intervention and evaluation study design before it received grant funds for planning, implementation, and evaluation. Has findings published in a peer-reviewed journal Overview of Government Terminology for Best / Promising Practices 8 Evidence Level 1 (cont.) Is part of a systematic review. This means that other researchers have tested similar programs. Uses strategies listed in the Guide to Community Preventive Services (http://www.thecommunityguide.org/). This Guide summarizes systematic reviews. It looks for the best practices for effective programs. Evidence Level 2 Funded by peer-reviewed grant Published in peer-reviewed journal Part of systematic review Strategies from systematic reviews but NOT Community Guide Has findings published in a peer-reviewed journal but is not funded by a peerreviewed grant Overview of Government Terminology for Best / Promising Practices 9 Overview of Government Terminology for Best / Promising Practices 10 Evidence Level 2 Is part of a systematic review. This means that other researchers have tested similar programs. Uses strategies listed by the Community Guide (http://www.thecommunityguide.org/) Overview of Government Terminology for Best / Promising Practices 11 Evidence Level 3 Published in a peer-reviewed journal but NOT funded by a peer-reviewed grant Part of a systematic review Strategies from systematic reviews but NOT by Community Guide Uses strategies listed by the Community Guide (http://www.thecommunityguide.org/) Overview of Government Terminology for Best / Promising Practices 12 1-800-537-8295 http://www.natamcancer.org 2
Evidence Level 4 Funded by peer-reviewed grant Published in a peer-reviewed journal Strategies from single study but NOT in systematic review Evidence Level 5 Published in a peerreviewed journal but NOT funded by peerreviewed grant Strategies from single study but NOT part of systematic review Overview of Government Terminology for Best / Promising Practices 13 Overview of Government Terminology for Best / Promising Practices 14 What Is Evidence? Surveillance Data Systematic Reviews of Multiple Intervention Studies An Intervention Research Study Program Evaluation Word of Mouth Personal Experience OBJECTIVE SUBJECTIVE Source: National Cancer Institute, Using What Works: Adapting Evidence- Based Programs to Fit Your Needs: US DHHS, NIH, Washington, DC [NCI Pub. No.06-5874] April 2006. Overview of Government Terminology for Best / Promising Practices 15 Why the Fuss? More Federal funders are requiring program planners to use evidence-based programs. Some consider evidence that is proven through research (explicit). Some consider evidence that is derived from experience or practice (tacit). The best evidence may be a combination of research and practice. Source: National Cancer Institute, Using What Works: Adapting Evidence- Based Programs to Fit Your Needs: US DHHS, NIH, Washington, DC [NCI Pub. No.06-5874] April 2006. Overview of Government Terminology for Best / Promising Practices 16 AIAN Experiences with Evidence-based Tobacco Protocols What has your experience been with evidence-based tobacco programs? What has been helpful about them? What are limitations of them? How would you like to see them changed to be more inclusive and respectful of AIAN communities? Overview of Government Terminology for Best / Promising Practices 17 What are Advantages to Evidence-based Programs? Effective in the study populations Cost effective Shorten the time it takes to develop a program Reduce the time it takes to research a community Help narrow the evaluation Source: National Cancer Institute, Using What Works: Adapting Evidence- Based Programs to Fit Your Needs: US DHHS, NIH, Washington, DC [NCI Pub. No.06-5874] April 2006. Overview of Government Terminology for Best / Promising Practices 18 1-800-537-8295 http://www.natamcancer.org 3
Evidence-Based Program Barriers Using evidence-based programs limits my creativity. Evidence-based programs take too much time and/or money. LB Modified from: National Cancer Institute, Using What Works: Adapting Evidence-Based Programs to Fit Your Needs: US DHHS, NIH, Washington, DC [NCI Pub. No.06-5874] April 2006. Overview of Government Terminology for Best / Promising Practices 19 Evidence-Based Program Barriers, cont. Evidence-based programs are too scientific. My community is unique. An evidence-based program will not be as appropriate as if I developed the program myself. LB Modified from: National Cancer Institute, Using What Works: Adapting Evidence-Based Programs to Fit Your Needs: US DHHS, NIH, Washington, DC [NCI Pub. No.06-5874] April 2006. Overview of Government Terminology for Best / Promising Practices 20 What are issues for AIANs using Tobacco Best Practices that were evaluated in non-native communities? AIAN Tobacco Programs Many AIAN Programs are at evidence levels 4 and 5, but they WORK! This is why the NTTPN Best/Promising Practices Committee is using the phrase, Promising Practices Many AIAN Promising Tobacco Practices are more culturally appropriate and relevant than are non-native best practices Overview of Government Terminology for Best / Promising Practices 22 Promising AIAN Tobacco Practices Note: The only thing really unfortunate for AIAN projects that are at the promising level, is reduced likelihood of future funding. Promising Practices that include comprehensive evaluation remain very competitive for subsequent funding. Overview of Government Terminology for Best / Promising Practices 23 1-800-537-8295 http://www.natamcancer.org 4
QUESTION: Why Should AIAN Tobacco Program Staff Care How the Fed s Use Terminology If we use the same phrases, but with different local connotations, there is miscommunication between federal staff and I/T/U staff They hold our projects to program standards that may be unrealistic to our local communities Overview of Government Terminology for Best / Promising Practices 25 No Community Program Is Now or Ever Will Be Perfect Community programs evolve and change Interventions that work for a while, need to change as the community changes Strategies that work for non-native communities frequently lack inherent characteristics that need to be included in Native programs Overview of Government Terminology for Best / Promising Practices 26 Other Government Terminology (information is for your reference and not necessarily for anything other than skimming today) Scientific Evidence Confusing terminology Rarely does the federal government agree on the definitions used They vary by each federal institution (NIH, CDC, AHCPR, IHS) and within the Divisions and Sections of that Institution This is why it is difficult to find the terminology defined consistently Scientific Evidence, in general Highest level of scientific evidence means the studies: Use the same protocols for every step of the research project Explains the steps and processes involved in a study Overview of Government Terminology for Best / Promising Practices 29 Overview of Government Terminology for Best / Promising Practices 30 1-800-537-8295 http://www.natamcancer.org 5
Protocols include: Who is involved? What is to be done? By whom? To whom? When is it to be done? Where will it occur? Why is it being done? Step-by-step explanation of how the processes are to be done Overview of Government Terminology for Best / Promising Practices 31 Scientific Evidence, in general (cont.) Highest level of scientific evidence means the studies: Use comparable statistical data points and analyses Include populations who are randomized into intervention and control groups Use research models that have been replicated and show similar findings Overview of Government Terminology for Best / Promising Practices 32 Scientific Evidence Always includes quantitative (numbers) assessments and evaluation Evaluation used in Indian Country is Typically qualitative (opinion, feelings, clarifying statements) But may include both quantitative and qualitative Evaluation Poorly regarded program evaluation is satisfaction or happiness assessments How well did you like the program? How would you rank the usefulness of the program? When satisfaction / happiness measures are included with other types of evaluation, they have more credibility Overview of Government Terminology for Best / Promising Practices 33 Overview of Government Terminology for Best / Promising Practices 34 QUESTION: What is the most common reason why AIAN Tobacco programs lose their funding? Insufficient evaluation Lacks quantitative and qualitative evaluation Inconsistent in how evaluation is implemented Evaluation strategies embed biased phrasing that is likely to result in good feedback from participants Definitions for your reference (skimming section) Overview of Government Terminology for Best / Promising Practices 35 1-800-537-8295 http://www.natamcancer.org 6
U.S. Preventive Services Task Force (USPSTF) Centers produce reports, reviews, and technology assessments Findings are based on Rigorous, comprehensive syntheses and analyses Of the scientific literature on topics Evidence-based Practice Centers Overview. November 2007. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/clinic/epc/ USPSTF (cont.) Relevant to Clinical Social / behavioral sciences Economic, And other health care organization and delivery issues Overview of Government Terminology for Best / Promising Practices 37 Overview of Government Terminology for Best / Promising Practices 38 USPSTF (cont.) Emphasize explicit and detailed documentation of methods rationale assumptions. Recommendations may include meta-analyses Cost analyses Evidence-Based Intervention defined: An intervention that has been conducted with a group and shown, through different kinds of evaluation, to be effective. Also referred to as research-tested. Overview of Government Terminology for Best / Promising Practices 39 Overview of Government Terminology for Best / Promising Practices 40 Evidence-based Evidence-based PROGRAMS have been proven to be effective in the populations and settings in which they were studied. Using an evidence-based program Shortens the time it takes to develop a new program Reduces the amount of research needed Helps focus the evaluation Best Practices defined: From a clinical perspective this can mean best screening or treatment options From a health education or communication perspective this can mean best methods for delivering messages to have the greatest impact. Overview of Government Terminology for Best / Promising Practices 41 Overview of Government Terminology for Best / Promising Practices 42 1-800-537-8295 http://www.natamcancer.org 7
Best Practices Programs based on best practices utilize STRATEGIES that have been shown to be effective. Strategies found in the Guide to Community Preventive Services (Community Guide) or the Guide to Clinical Preventive Services (Clinical Guide) provide strategies that can serve as best practices for programs. Overview of Government Terminology for Best / Promising Practices 43 Best Practices The Community Guide can be found at http://www.thecommunityguide.org. An example of best practices being used can be found at: http://www.cdc.gov/tobacco/bestprac.htm Overview of Government Terminology for Best / Promising Practices 44 Formative Evaluation defined: Is carried out during the program planning or delivery Identifies if there are any problems and makes needed corrections Usually focuses on determining whether a program is working as planned Impact Evaluation defined: Assesses the immediate effect of an intervention on the behavior knowledge and attitudes of the target population. Overview of Government Terminology for Best / Promising Practices 45 Overview of Government Terminology for Best / Promising Practices 46 Outcome Evaluation defined: Finds out if the long-term program goals were met. Measures include changes in the Morbidity (burden of disease) Mortality (deaths) Health status of your population Overview of Government Terminology for Best / Promising Practices 47 Process Evaluation defined: Evaluation that looks at activities as they occur during program delivery Provides a way to find problems in the early stages of a program NOTE: NIH refers to process evaluation as occurring during a pilot test Overview of Government Terminology for Best / Promising Practices 48 1-800-537-8295 http://www.natamcancer.org 8
Qualitative Data Collection defined: Uses open-ended questions that bring out discussion rather than one- or twoword responses Usually do not include: Numbers and cannot easily be summarized in a table or graph. Qualitative Data Collection defined (cont.): Usually are studied to find themes, such as with: Focus groups Working groups Informant interviews May include documenting nonverbal cues as well Overview of Government Terminology for Best / Promising Practices 49 Overview of Government Terminology for Best / Promising Practices 50 Quantitative Data Collection defined: Involves numbers or statistics or that can be summarized in a table or graph. Gathered by close-ended questions like multiple choice or those needing only a one- or two-word response. Randomization defined: The process by which people of similar backgrounds or characteristics are broken up into groups. Each is given a certain piece of information or task. Each group has a name, such as experimental or control. Overview of Government Terminology for Best / Promising Practices 51 Overview of Government Terminology for Best / Promising Practices 52 Randomization defined (cont.): The experimental group is usually given a new piece of information or task to test. The control group is not given any new information or tasks. These two groups are compared in order to see the effect the new information or task has on the participants. Research-Tested Intervention Programs defined: These are conducted with an audience and their outcome data have been published in a peer-reviewed journal Overview of Government Terminology for Best / Promising Practices 53 Overview of Government Terminology for Best / Promising Practices 54 1-800-537-8295 http://www.natamcancer.org 9
Theory-based or Theory-driven Theory helps investigate answers to the questions of why, what, and how health problems should be addressed Guides the search for reasons why people engage in certain behaviors. Suggests strategies and identifies which indicators should be monitored and measured during program evaluation. from Theory at a Glance, NCI, Pub # 05-3896, Sept 2005 Theory-based or Theory-driven Theory can provide a road map for Studying problems Developing appropriate interventions Evaluating their successes from Theory at a Glance, NCI, Pub # 05-3896, Sept 2005 Overview of Government Terminology for Best / Promising Practices 55 Overview of Government Terminology for Best / Promising Practices 56 Research-tested Research-tested is a feature of evidencebased practice. It means the program was tested in a peer reviewed and funded research study. A program may not be as effective once it leaves the research setting if there are changes in parts of the program used, the environment, or the population served. Research-tested However, the program serves as a good starting place. Research-tested programs can be found on Step 4 of Cancer Control PLANET (http://cancercontrolplanet.cancer.gov/ Overview of Government Terminology for Best / Promising Practices 57 Overview of Government Terminology for Best / Promising Practices 58 1-800-537-8295 http://www.natamcancer.org 10