The Minority AIDS Initiative National Cross-Site Evaluation: An Overview

Transcription

1

2 The Minority AIDS Initiative National Cross-Site Evaluation: An Overview Phillip Graham Nilufer Isvan Program Evaluation for Prevention (PEP-C) Minority AIDS Initiatives New Grantee Conference Bethesda, MD December 10,

3 Agenda Introduction Federal reporting requirements The cross-site logic model & evaluation questions Cross-site evaluation data sources Process & implementation Participant-level outcomes Community-level outcomes Online data systems under development Discussion 4

4 Introducing the SAMHSA Team Contracting Officer s Representative (COR) in charge of PEP-C: Sara Azimi-Bolourian, Ph.D., Public Health Analyst, SAMHSA/CSAP Alternate Contracting Officer s Representative (ACOR) for PEP-C: Thomas Clarke, Ph.D., Social Science Analyst, SAMHSA/CSAP 4

5 Program Evaluation for Prevention Contract (PEP-C) Tasks SAMHSA/CSAP s evaluation contract National cross-site evaluations of four SAMHSA/CSAP grant initiatives Partnerships For Success (PFS) Strategic Prevention Framework State Incentive Grants (SPF SIG) Minority AIDS Initiative (MAI) STOP Act Grants (one-year retrospective study, completed) 5

6 Program Evaluation for Prevention Contract (PEP-C) Leadership Project Director: Phillip W. Graham, Dr.PH Director of the Drugs, Violence, and Delinquency Prevention Research Program in the Center for Justice, Safety, and Resilience, RTI International Deputy Project Director: Elvira Elek, Ph. D. (Social Psychology) Research public health analyst, RTI International 5

7 PEP-C s MAI National Cross-Site Evaluation Team Name & Degree Role on the Team Affiliation Nilufer Isvan Ph.D. in Sociology Mindy Herman Stahl Ph.D. in Human Development Darigg Brown Ph.D. in Biobehavioral Health Melissa Burnett B.A. in Psychology Leena Elsadek B.A. in Global Health and Anthropology 7 Rachael Gerber M.P.H. Lisa Lundquist M.A. in Criminal Justice Team Co-Lead Team Co-Lead HIV/AIDS Prevention Evaluation Specialist Research Analyst, TTA Liaison Research Analyst, TTA Support Research Analyst, Data Manager Research Analyst, Data Manager HSRI RTI International RTI International HSRI RTI International HSRI HSRI

8 Annual Performance Reporting Government Performance and Results Act (GPRA) Originally enacted in 1993 Currently: GPRA Modernization Act of 2010 Federally funded programs required to report performance measures & meet targets Each year s performance used to justify budget requests for following year 8

9 SAMHSA/CSAP s MAI GPRA Measures Measure Target Actual (FY2013) Number of program participants exposed to substance abuse prevention education services (Output) Percent of program participants that rate the risk of harm from substance abuse as great (all ages) (Outcome) Percent of program participants who report no use of alcohol at pre-test who remain non-users at post-test (all ages) (Outcome) 5,734 6, % 96.2% 91.2% 89.2% Percent of participants who report no illicit drug use at pre-test who remain non-users at post-test (all ages) (Outcome) 92.6% 93.9% Number of persons tested for HIV through the Minority AIDS Initiative prevention activities (Outcome) Baseline 36,707 Source: CSAP Accountability Report, Volume XII, FY

10 HHS Core Indicators for HIV 7 Core indicators, mostly measuring treatment and secondary prevention outcomes CSAP reports 3 of the indicators, adapted for primary prevention programs: Numbers tested for HIV Number of positive tests Number of homeless (or unstably housed) individuals tested for HIV 10

11 Cross-Site Evaluation Questions (1) Baseline characteristics of the people served Populations were targeted, strategies implemented, services provided Changes in participants knowledge, attitudes, behaviors, access to healthcare Individual-level factors and grantee characteristics associated with participant outcomes 11

12 Cross-Site Evaluation Questions (2) Changes in the normative environment and in health disparities at the community level Factors associated with changes at the community level Strategies and combinations of strategies associated with participant- and communitylevel changes 12

13 MAI Cross-Site Evaluation Logic Model Inputs Outputs Outcomes MAI funding TTA Activities Needs assessment Capacity building Strategic planning Implementation o # Direct prevention o # Environmental strategies o # HIV/HCV testing o # Referrals and service linkages Evaluation Participants # Served by demog. # Trained in SA, HIV, HCV prevention # Tested for HIV/ HCV, # positive, # counseled, and # linked to care # Tested for the first time # Number with knowledge of test results Proximal (Individual Level) Knowledge Attitudes Risk perceptions Self-efficacy Intentions Social norms Awareness of and access to health services Intermediate (Individual Level) Any alcohol use Binge/heavy drinking Any illicit drug use Unprotected sex Sex while drunk or high Distal (Environmental Level) Changes in community social norms and attitudes around alcohol use and risky sexual behaviors associated with HIV/HCV transmission Individual-Level Moderators Sociodemographics Victimization Discrimination Mental health Criminal justice involvement Social support Grantee-Environment Level Moderators Fidelity Baseline prevalence of HIV/ HCV/STIs Baseline social/economic characteristics 13

14 14 PROCESS & IMPLEMENTATION DATA

15 Quarterly Progress Reports Data collection tool under review by the Office of Management and Budgets (OMB) Online data entry tool under development Future trainings Detailed content after OMB approval Technical training before online system launch 15

16 Structure and Functions of the Quarterly Progress Reports Divided into modules following the Strategic Prevention Framework (SPF) Every quarter, grantees report progress on each step that was worked on Capability for grantees to upload documents Online review/approval capability for GPOs System extracts sent to GPOs for grant monitoring and to PEP-C for analysis 16

17 The Implementation Module will Include Numbers served/reached by gender, ethnicity, race, age Indirect services (environmental strategies and information dissemination) HIV testing Viral Hepatitis (VH) testing VH vaccination Numbers served through direct services by target population 17

18 HIV and VH Testing Reporting Also Includes Numbers tested for the first time Number of positive tests Number of homeless or unstably housed individuals tested 18

19 On Your Flash Drive Additional Documents Additional Documents PEP-C Documents Copies of DRAFT participant-level instruments DRAFT participant-level data collection guide Resources for collecting community-level data 19 Validated SA surveys/items for young adults Validated HIV risk factor surveys/items for young adults

20 EVALUATING DIRECT SERVICES WITH PARTICIPANT-LEVEL DATA 20

21 MAI Participant-Level Instruments The National Minority SA/HIV Prevention Initiative has four standard instruments Youth Questionnaire Adult Questionnaire Group Dosage Form Individual Dosage Form 11

22 Which Participants are Included in this Data Collection Protocol? Requirement to collect data from program participants receiving funded direct service interventions 12

23 Direct Prevention Service Definition Delivered in direct interaction with participants Can be either one-on-one (individual) or group format. 13

24 Direct Services Examples HIV or substance abuse prevention education classes Motivational interviewing Problem identification, referral, and case management services One-on-one or group counseling Refusal skills training HIV testing 24

25 When Not to Administer the Participant-Level Instruments Participant-level data collection protocol does not apply to: Individuals contacted through community outreach or other recruitment efforts only Individuals who only receive testing or vaccination services Data reported in the aggregate in the Quarterly Progress Reports 25

26 Record Management Section Included in all instruments (questionnaires and dosage forms) Filled out ONLY by a staff member with access to the necessary information Must be completed prior to administering questionnaires to participants 26

27 Record Management Fields Grant ID Unique Participant ID Interview type Interview date Intervention(s) received Service duration Intervention vs. comparison group 27

28 Questionnaire Sections SECTION 1: Facts About You Demographic and socioeconomic information Output measures (people served) Disparities in outcomes SECTION 2: Attitudes & Knowledge HIV knowledge, perception of risk, self-efficacy Proximal outcomes (expected to change soon after program) SECTION 3: Behavior & Relationships 28 Substance use, risky sexual behaviors, emotional support May take some time to change (at least 30 days)

29 Participant Burden Reduction Participants with shorter service duration receive: Fewer questions on the questionnaire Fewer survey administrations 23

30 Key Concept Service Duration Length of time between the first and last direct service encounters with the participant Divided into three categories Single Session (does not exceed a single day) Multiple Session Brief (2-29 days) Multiple Session Long (30+ days) 24

31 Which Questionnaire Sections to Administer and When Participant s Service Duration Questionnaire Section to be Administered Data Collection Time Points Single Session (no longer than a single day) Multiple Session Brief (2-29 days) All of Section One and 3 to 5 relevant questions selected from Section Two All of Sections One and Two Exit only Baseline and exit Multiple Session Long (30+ days) Entire questionnaire (All of Sections One, Two, and Three) Baseline, exit, and 3-6- month post-exit followup 25

32 Grant Identification Number Enter the grant identification number assigned by SAMHSA/CSAP If this number is missing or inaccurate, the data record cannot be processed or used in evaluation 28

33 Study Design Group Intervention Group Group receiving services Comparison Group Group NOT receiving any services Comparison groups are NOT required by CSAP. Select Intervention for all of your participants if you are not using a comparison group Records with missing Study Design Group cannot be used in the evaluation. 29

34 Participant Identification Number Unique numbers should be assigned to each program participant by qualified Staff. The same Participant ID number will be used for ALL records associated with the participant (all survey and dosage data) Multiple records sharing identical Participant ID, Survey Administration Date, and Interview Type (e.g. Baseline, Exit, Follow-up) will be flagged and may be eliminated from analysis as this suggests duplicated records or the same Participant ID assigned to more than one participant. 30

35 Date of Survey Administration The 2-digit month, 2-digit day, and 4-digit year should be entered. This should be the date that the questionnaire was administered, not the date the data were entered. Records with missing, incomplete, or inaccurate administration date information cannot be used in evaluation. If administration dates are out of order (e.g. date of exit interview is before the baseline interview), neither record can be used in the evaluation. 31

36 Interview Type Baseline First data collection point, must be prior to program exposure by no more than 30 days Exit Second data collection point, up to 10 days following the final service encounter with the participant. If services lasted a single day ( single session ), administer at the end of the day. Follow-up Third data collection point, three-to-six months after program exit If not accurately filled out, participant may appear to have two interviews at the same time point and it may not be possible to use the data in the evaluation. 32

37 Service Duration Single Session Intervention Total service duration does not exceed one day Multiple Session Brief Intervention Total service duration between 2 and 29 days Multiple Session Long Intervention Total service duration 30 days or longer This field will be used to select the appropriate outcome variables for the participant. If incorrectly assigned, the data may not be included in the relevant outcome analyses. 34

38 Frequently Asked Question Before services begin, we may not know how long the participant will stay in our program. How do we fill out Intervention Duration in the baseline survey? 38

39 Response: Informed Guess Enter best guess at baseline Enter actual duration of services at exit Cross-site team will make the necessary correction before analysis 39

40 Intervention Name(s) Write down the name(s) of the interventions that the participant is (or will be) receiving before administering survey Online data entry system will provide dropdown menu for your convenience 37

41 Sample Scenario An individual participated in Voices/Voces one day; then 10 days later, she enrolled in Protocol-Based HIV Counseling and Testing that lasted 15 days. Service Duration is =26 days select the Multiple Session Brief Intervention option. Administer a baseline and an exit survey Write in Voices/Voces as Intervention Name 1 Write in Protocol-Based HIV Counseling and Testing as Intervention Name 2 41

42 Preparing the Questionnaire Familiarize survey administration staff with the questionnaires (training materials will be provided) Determine the total duration of funded services that the participant is intended to receive Prepare appropriate section(s) of appropriate questionnaire (youth or adult) Determine timing of Baseline, Exit, and Follow-up data collection Complete the record management section before giving to participants 42

43 Administering the Questionnaire Who administers the questionnaires? Qualified staff familiar with the instruments and trained in survey administration Service providers should not administer questionnaires Choose a space that provides sufficient ventilation, lighting, and privacy Budget an appropriate amount of time to complete the questionnaire Single-day services : ~ 5 minutes Services lasting 2 29 days: ~ 30 minutes Services lasting 30+ days: ~ 50 minutes Allow extra time for administrative issues (reading instructions, etc.) Provide services or referrals in the event the questionnaire items about personal issues such as partner abuse cause emotional distress among participants 43

44 44 Questions?

45 Dosage Data 45

46 Dosage Forms Function and Use Used to record the type and duration of direct contact with participants Two different service delivery formats: Group (more than one participant receiving service during the encounter) Individual (one-on-one service delivery) Service codes are provided at the end of the dosage forms A dosage record should be submitted for every service encounter with a participant 50

47 Dosage Forms Contents Record management fields: Encounter Date Grant ID Administration Format Participant ID Number(s) Data fields: Service Code(s) Duration Code(s) (in minutes, rounded up to the nearest 5-min. interval) Dosage records with missing or invalid record management information may be excluded from the evaluation 51

48 Individual Dosage Form Month Day Year Grant ID Grp.Typ. Adm. Frmt. Participant ID # S P #1 #2 Individual Service Code Duration Code a (Closest 5- minute interval) a (Closest 5- minute interval) Example: On March 22, 2010, Marie (ID#65471) participated in a behavioral health intervention. It included an individual HIV education session and an HIV testing counseling session at the Jones Health Center (Grant ID SP00009). The education session lasted 88 minutes and the HIV testing counseling lasted 15 minutes. She decided not to stay for testing on that particular day but may come back. 52

49 Group Dosage Form Encounter Date Group Service Code Duration Code Month Day Year # a Grant ID (To the closest 5- minute interval) S P #2 Grp. Type Adm. Frmt. 1 2 # a a (To the closest 5- minute interval) (To the closest 5- minute interval) Participant ID Numbers: Example: Cityside Prevention Center (Grant ID SP00017) holds a 2-hour group counseling session for adults once a week. On April 12, 2010, 12 people attended. The session went over the normal 2 hour time slot by 4 minutes.

50 EVALUATING INDIRECT STRATEGIES WITH COMMUNITY-LEVEL DATA 50

51 Direct vs. Indirect Strategies Direct (individual-based) Prevention Strategies: Delivered directly to individuals Service provider and participant are in the same location at the same time Indirect (population-based) Prevention Strategies: Aim to change the institutions, policies, norms, and practices of an entire community 51

52 Terminology Alert! Multiple definitions! Different agencies may use terms differently Terminology may differ by academic discipline or journal Terminology we will use: SAMHSA/CSAP s Six Strategies Framework 52

53 Focus of This Webinar Indirect Prevention Strategies Environmental Strategies Information Dissemination 53

54 Environmental Strategies Definition: Aim to change community standards, codes, and practices related to undesirable health behaviors in the general population Subcategories Legal and regulatory initiatives Service, access, systems change, and action-oriented initiatives 54

55 Information Dissemination Definition: Strategies to provide information to a population through one-way communication from source to audience Goals: Inform the public about undesirable health behaviors and their effects healthy lifestyles health services available in the community Change community norms and attitudes 55

56 Terminology Alert! Some definitions classify social norms campaigns and social marketing as environmental strategies The Six Strategies Framework classifies them under information dissemination 56

57 Outcome Assessment Steps Clearly define the targeted community Clearly define the targeted outcomes Select one or more measures that will validly and reliably represent the targeted outcomes Identify data sources for populating the selected outcome measures Obtain baseline data Obtain follow-up data Compare baseline and follow-up values to assess change 57

58 Identifying the Target Community Indirect Strategy Condom distribution on campus Partnering with a school district to implement a Student Prevention Leaders program in every school Running a TV ad with a prevention message Installing billboards on every bus in a community Targeted Community All students enrolled in the institution Students enrolled in the district schools Population group reached by the TV channels on which the ad aired Entire community served by the bus system 58

59 Matching Outcome Measures to Strategies Strategy Goal Strategy Objective Outcome Measure Reduce HIV transmission in County X Reduce HIV risk factors among college students Decrease the rate of new HIV diagnoses in County X by 5% in 3 years Decrease prevalence of unprotected sex on Campus X by 5% by end of grant period Percentage of HIV tests in County X with positive results Percentage of survey respondents on Campus X who report using protection Reduce the negative consequences of problem alcohol use Decrease the rate of traffic accidents due to alcohol in County X Percentage of traffic crashes in County X that are caused by drunk driving 59

60 Possible Sources of Outcome Data (1) Local archival data sources, like Departments of public health, education, and transportation Local police department Local hospitals or Federally qualified health centers (FQHCs) Vital statistics Data collected by college administrations, like Transports from campus to area emergency departments Illicit substances confiscated during dormitory raids 60

61 Possible Sources of Outcome Data (2) Uniform Crime Reports (UCR) published by the Federal Bureau of Investigation Fatality Analysis Reporting System (FARS) maintained by the National Highway Traffic Safety Administration Behavioral Risk Factor Surveillance System (BRFSS) maintained by the Centers for Disease Control and Prevention (CDC) 61

62 Possible Sources of Outcome Data (3) Youth Risk Behavior Surveillance System (YRBSS) maintained by the CDC Other epidemiological data reported by Federal agencies such as the CDC s WONDER databases Epidemiological data maintained by the State s Epidemiological Outcomes Workgroup (SEOW) State or local surveys 62

63 Possible Sources of Outcome Data (4) College surveys such as CORE College health clinic records Dormitory alcohol inspection results 63

64 Measure and Data Factors to Consider (1) Will the data be available through the end of the grant period? How often are the data updated? Is the data source representative of the targeted community? 64

65 Measure and Data Factors to Consider (2) Is the measure sensitive to changes in the desired outcome? Is the measure specific to the desired outcome? 65

66 You have collected your data. Now what? 58

67 Participant-Level Data Submission Completed questionnaires and dosage forms can be entered into digital format for analysis in one of two ways: Online data entry system under development Standard templates and coding manuals under development for preparing and uploading data 60

68 Quarterly Progress Report Submission Online data entry system and user manuals under development Technical trainings will be provided before system launch 68

69 Community-Level Data Submission Online data entry tool under development Will capture information on Data source (e.g. administrative, surveillance, survey, description of the population) Measure description (e.g. rate, percent, average) Calculated value baseline value & annual updates Technical trainings will be provided before system launch 69

70 70 Discussion

71 Future Questions and TA Requests Phone: Your question will be triaged to a Cross-Site Team member best qualified to respond Of course, you can always contact your SAMHSA Project Officer with your questions and TA requests 71