Evidence-Based Interventions & Evaluation

Transcription

1 Evidence-Based Interventions & Evaluation Elizabeth Rausch-Phung, M.D., M.P.H. Director, Bureau of Immunization Evidence-Based Public Health 2 Application of the concepts of Evidence-Based Medicine/Practice to the public health domain The development, implementation, and evaluation of effective programs and policies in public health through application of principles of scientific reasoning, including systematic uses of data and information systems, and appropriate use of behavioral science theory and program planning models. Brownson RC, Baker EA, Leet TL and Gillespie KN, Editors. Evidence Based Public Health. New York: Oxford University Press,

2 Evidence-Based Public Health 3 Satterfield JM, Spring B, Brownson RC, Mullen EJ, Newhouse RP, Walker BB et al. Toward a transdisciplinary model of evidence-based practice. Milbank Q 2009; 87(2): Why is EVPH Important? 4 Provides assurance that decision making is based on scientific evidence and effective practices Allows for evaluation to demonstrate what works and doesn t work for a particular public health matter or population Provides assurance that one s time is being used most efficiently and productively 2

3 Key Elements of EVPH Engaging the community in assessment and decision making Using data and information systems systematically Making decisions on the basis of the best available peerreviewed evidence Applying program planning frameworks Conducting sound evaluation Disseminating what is learned Jacobs JA, Jones E, Gabella BA, Spring B, Brownson RC. Tools for Implementing an Evidence-Based Approach in Public Health Practice. Prev. Chronic Dis 2012; 9: DOH: 5 Engage the community in assessment and decision making Meet with your coalition and special population stakeholder(s) Consider o Immunization coverage (where known) NOT limited to special populations with a known baseline Consider alternative proxies to community immunization coverage, such as coverage in school(s) attended by the special population or in clinics serving your population of interest o Barriers/challenges to immunization in your community/selected special population Consult your stakeholders and community members! Interventions should be selected to address these barriers/challenges 6 3

4 Best Available Evidence 7 Not limited to randomized controlled trials! o Continuum of evidence of effectiveness ranging from emerging/promising to well-supported Research evidence is important, but it is not the only standard of evidence that is important in public health Contextual evidence is based on factors that address whether a strategy is useful, feasible to implement, and accepted by a particular community Experiential evidence is based on the professional insight, understanding, skill and expertise that is accumulated over time CDC National Center for Injury Prevention and Control, March 19,

5 The Community Guide a.k.a. The Guide to Community Preventive Services The CDC s preferred source of evidence-based strategies to improve immunization coverage From the CDC s Immunization Program Operations Manual, 2018: Evidence-based strategies can be taken from The Guide to Community Preventive Services ( If a strategy is proposed that is not part of The Guide to Community Preventive Services, please provide the evidence used to support the effectiveness of this strategy 9 The Community Guide 10 Collection of evidence-based findings of the Community Preventive Services Task Force (CPSTF) Based on systematic reviews of effectiveness and economic evidence of public health interventions Designed to answer three questions: 1. What has worked for others and how well? 2. What might this intervention approach cost, and what am I likely to achieve through my investment? 3. What are the evidence gaps? 5

6 Community Guide Recommendations Recommended: The systematic review of available studies provides strong or sufficient evidence that the intervention is effective. The categories of strong and sufficient reflect the degree of confidence the CPSTF has that an intervention has beneficial effects. They do not directly relate to the expected magnitude of benefits. Recommended Against: The systematic review of available studies provides strong or sufficient evidence that the intervention is harmful or not effective Insufficient Evidence: The available studies do not provide sufficient evidence to determine if the intervention is, or is not, effective. This does NOT mean that the intervention does not work. It means that additional research is needed to determine whether or not the intervention is effective 11 CPSTF Findings for Increasing Vaccination May be found online at or Key tips for Community Guide web page: contains 3 pages of findings. To view all, either scroll through the page numbers at the bottom of the page or select Display Results: - All - to view all findings 12 May also sort by recommended interventions at tion_tid=7476&items_per_page=all 6

7 CPSTF Findings for Increasing Vaccination 13 CPSTF Findings for Increasing Vaccination 14 7

8 CPSTF Findings for Increasing Vaccination 15 Evaluation 16 A key part of EBPH Begin planning the evaluation as you are planning the intervention o What barrier/challenge is your intervention designed to address? o What is your desired outcome? Is it measurable? o If your outcome is not measurable, can an alternative outcome or process measure be utilized? o Measure your baseline before implementing the intervention 8

9 Think S.M.A.R.T. 17 Specific o Who, what, and where? Measurable o Is it quantifiable and can WE measure it? Achievable o Can we get it done in the proposed time frame with the resources and support we have available? Relevant o Does this measure relate to the intervention and desired outcome? Time bound o When will this be accomplished? Specific Measures 18 Think about what you want to accomplish, and in whom o Specific population o Age range (if relevant) o Consider targeting individual vaccines, i.e., improving flu vaccination coverage rather than all adult vaccines, or focusing on DTaP as opposed to all childhood vaccines If you are successful, then you can always target a different vaccine in the future o Identify your baseline and a specific target (not simply increase or decrease ) 9

10 Specific Measures 19 Nonspecific: Immunization coverage will improve Specific: 4-dose DTaP vaccination coverage among month old children served by Clinic A will improve to 90% by January 2020 Measurable Measures How will you measure it? o Is there an existing data source (i.e., NYSIIS, school survey data, clinic assessments)? o If no existing data source, can we reasonably collect data? Can YOU measure it? o Consider your internal and community resources o Is it realistic and feasible for your and/or your partners to measure this? If no data source is available nor reasonably attainable, consider alternative measures, proxy measures, or process measures 20 10

11 Consider Process Measures 21 Process measures monitor the completion of critical steps in the process leading to your ultimate outcome May be used to evaluate effectiveness of activities in progress It may not always be possible to measure immunization coverage in certain populations A process measure, relevant to your intervention, may serve as an early indicator of success or of a need to modify or enhance the intervention Process Measure Examples 22 Intervention: Coalition-run event to promote use of standing orders by county Ob/Gyns o Process measure: Proportion of county Ob/Gyns using standing orders, measured prior to the event and quarterly thereafter Intervention: Send reminder/recall notices to children overdue for dose 4 of PCV13 o Process measure: Proportion of children sent reminder/recall notice who received dose 4 of PCV13 11

12 Measurable Measures Not measurable: Improve pneumococcal vaccination coverage among county Hispanic residents aged years living with diabetes to 70% by January 2023 o Problem: county ebrfss data is only available every 3-5 years, and is too small to provide accurate estimates for Hispanic adults with diabetes Measurable: Improve pneumococcal vaccination coverage at Clinic A, which primarily serves a Hispanic population, to 70% or Measurable: By April 2020, pneumococcal vaccine training, with a focus on diabetics and Hispanic populations, will have been provided to 50% of county adult primary care providers 23 Achievable Measures 24 BE REALISTIC about what you can achieve in the proposed time frame and with your internal and community resources o Consider how coalition members, special populations stakeholders and other community members can assist/partner with you o You don t need to do it all by yourself! Don t aim for Healthy People 2020 targets if you re unlikely to achieve them aim for a target that is somewhat challenging but you are likely to achieve as a result of your intervention 12

13 Achievable Measures 25 E.g., County A is concerned about its 50% 2-dose hepatitis A coverage rate, compared with the 85% Healthy People 2020 target Probably not achievable: County month 2-dose hepatitis A vaccine coverage will improve to 85% by 2020 More achievable: County month 2-dose hepatitis A vaccine coverage will improve by 2-3 percentage points per year Relevant Measures Relate to the selected intervention and desired outcome E.g., county X plans a provider education intervention to encourage county Ob/Gyns to administer Tdap vaccine to pregnant patients. County X is aware that they do not have a data source on Tdap vaccination coverage among county pregnant women Too broad to be relevant: Countywide Tdap vaccination coverage among adults aged 18 years and older More relevant: Proportion of county Ob/Gyns that routinely administer Tdap to pregnant patients, measured prior to the initial intervention and quarterly thereafter 26 13

14 Time Bound Measures A specified and reasonable time frame to achieve the outcome Not time bound: Complete HPV vaccine coverage among year old adolescents at Clinic Z will improve to 70% o By when? Time bound: Complete HPV vaccine coverage among year old adolescents at Clinic Z will improve to 70% by January 2023, or Time bound: Complete HPV vaccine coverage among year old adolescents at Clinic Z will improve by 5 percentage points per year QUESTIONS? 14