in Non-Profit Organizations Serving

Building Evaluation Capacity in Non-Profit Organizations Serving LGBT and HIV+ Clients Anita Baker, Anita Baker Consulting abaker8722@aol.com Casey Smith-Speirs, AIDS Project Hartford, casp@aphct.org Jamie Bassell, Hartford Gay and Lesbian Health Collective, Jamieb@hghlc.org Yvette Bello, Latino Community Services, ybello@lcs-ct.org AEA 2010 San Antonio Session 868

BEC Background The Hartford Foundation for Public Giving has sponsored the Building Evaluation Capacity (BEC) Initiative since 2006. Objective of BEC is to provide training and technical assistance to support enhanced evaluation for nonprofit organizations. Two classes have completed the training (as of June 2010). Both included organizations that serve the LGBT and HIV+ communities. Session Overview BEC description, Anita Baker. Results of the Medical Case Management Evaluation, Casey Smith-Speirs, AIDS Project Hartford. Lessons learned from the Safety Net Evaluation, Jamie Bassell, Hartford Gay and Lesbian Health Collective. Integrating evaluative thinking throughout Latino Community Services, Yvette Bello, LCS Executive Director. Trainee assessment and next steps, Anita Baker 1

1) Two year commitment BEC Strategies Year 1 = 8 Classroom sessions, HW, 1 Consultation, Conference Year 2 = 5 Classroom sessions, Ind. Project, 4 Consultations, Conference 2) Involve teams including decision-makers 3) Must conduct evaluation projects, write reports, present findings 4) Opportunities for advanced and continued study BEC Content 1) Evaluation Basics: Definitions, Evaluation Terminology, Purposes, Stakeholders, Evaluation Questions, Evaluation Design, Evaluative Thinking 2) Logic Models: Developing and Assessing Logic Models 3) Documenting Program Strategies/Assessing Implementation, 4) Surveys and Interviews: Designing, Coding, Administration Plans, Analysis Plans, Analyzing and Summarizing Findings, Communicating Findings 5) Observations and Record Reviews: Methodological Choices, Designing, Coding, Analyzing, Summarizing 6) Making Data Collection Choices, Data Analysis Basics 7) Designing Evaluations, Level of Effort, Timelines, Cost of Evaluation and Budgeting, Evaluation Reporting 8) RIPPLE, Integrating Evaluative Thinking 2

What Can BEC Trainees Do? Design Evaluations Clarify the Purpose Specify Questions Select Data Collection Methods Specify Timelines and Level of Effort Estimate Cost of Evaluation Commission Evaluation for their Organizations Assess Evaluative Thinking in their Organizations for 15 Different Capacity Areas and Think about Responses/Actions Develop, Assess and Use Logic Models Design Surveys, Identify/Fix Bad Surveys, Determine How many Surveys are Needed, Develop Survey Administration and Analysis Plans, Analyze Data Design and Conduct Interviews, Observations and Record Reviews, Analyze Data Write and Present Evaluation Reports Involve others in Evaluation (RIPPLE) AIDS Project Hartford Medical Case Management Evaluation Case management services were designed to facilitate access to and utilization of the myriad medical, entitlement and social service systems Typical functions include completion of comprehensive intake, needs assessment and service plan for all clients Mid-2000s, Federal Government mandated all Ryan White providers adopt a medical model of case management, but without strict guidelines. APH adapted a model from the Visiting Nurse and Health Services of Connecticut 3

APH MCM APH provides medical case management services to more than 600 HIV-positive clients each year Twelve medical case managers (MCMs) are housed at community health clinics, and CBOs in three counties MCMs are all Bachelor s level professionals who receive in-house training and ongoing g mentoring Supervision provided by a Master s level social worker and a Registered Nurse APH MCM: Stated Goals to maximize clients health outcomes as determined by stable or improved CD4 counts and Viral Loads connect clients to medical care, help maintain care less tangible goal is that participation in the MCM program will enhance the clients overall quality of life 4

MCM Key Strategies Medical-related interview questions Continuous follow-up by MCMs with client medical providers Ongoing training in medical-related issues such as HIV-Related Nephropathy, HIV- Related Cardio-Pulmonary Disease, and the effects of HIV Anti-Retroviral Medications on people with HCV Evaluation Questions Do the APH medical case management clients experience improved health outcomes and improved client compliance with medical treatment and can they be associated with the medical model of case management? Do clients and staff share attitudes about the effectiveness of the APH medical case management model? 5

Data Collection Strategies Record Review. Every client file that was active as of 11/1/09 (n=421) was reviewed. Client medical appointment compliance Client medication adherence rates CD4 lab results* Viral load lab results* Client Surveys to gauge client s attitudes and perceptions about the efficacy of MCM services on their HIV disease progression, medical and medication compliance, and overall health. MCM Surveys to gauge the attitudes of the medical case managers about the efficacy of the medical case management services they provide. Key Findings, Client Survey Percentage of Client Respondents Reporting (N=156) Improved quality of life as a result of MCM 82% Improved health as a result of MCM 67% More likely to attend medical appointments because of MCM 67% More likely to take medications as prescribed because of MCM 65% 6

Key Findings, File Review Figure 2: Client CD4 Range by Year % of Clients 50% 40% 30% 20% 10% 0% 2007 2008 2009 1-200 24% 20% 23% 201-500 43% 43% 46% >500 33% 37% 32% Key Findings, File Review Figure 3: Client Viral Load Range by Year 80% % of Clients 60% 40% 20% 0% 2007 2008 2009 <500 59% 68% 67% 500-10,000 18% 15% 12% 10,000+ 23% 17% 21% 7

Key Findings, MCM Survey MCMs overwhelmingly reported feeling confident and well-prepared in their role. MCMs acknowledged that training needs were being met and that they are receiving and utilizing important medical-related information around HIV-related topics such as nephrology, hepatitis, cardiopulmonary disease and medication-related co-morbidities. Key Findings, MCM Survey Just by asking this set of questions every month gets my clients thinking about medical appointments, medications and their health. They may not think about these things at any other time during the month but they get used to thinking about these things and talking about them when we meet. 8

Key Findings, MCM Survey The Medical Model provides structure, guidance and support to Medical Case Managers. Medical Case Managers have a greater understanding of [clients ] medical conditions. As a result, clients are better educated and are more aware of potential symptoms and of what to look out for. Clients are also more aware of the effects of drug and alcohol use and other medical conditions on their bodies and how these things may affect them differently because of their HIV. Key Findings, MCM Survey Some of our clients have so many medical providers but no one else in their lives. They come to their Case Managers to get away from all of the medical stuff. I try to incorporate the medical and the social support in one visit. 9

Evaluation Summary Clients have adapted d well to the model. MCMs are able to successfully implement the model, but paperwork related to file management is onerous. While it is still too early to conclusively show any links between medical results and medical model implementation, early data were favorable. Action Steps Follow-up with MCMs regarding ongoing training i needs. Research strategies to streamline file maintenance Continue studying the CD4 and viral load Continue studying the CD4 and viral load results and conduct periodic follow-up checkins with clients and MCMs. 10

HGLHC is a non-profit 501(c) (3) organization that provides medical and social services tailored to the LGBTQ community, but serves clients of all genders and gender identities, sexual orientations, ages, racial and ethnic backgrounds. The agency maintains a strong HIV prevention program which serves thousands of clients every year through the provision of HIV and STD counseling, testing, community outreach, and safer sex promotion. Project Background scheduled to operate between January 2010 and December 2010 uses some of the latest technological social networking tools (e.g. text messaging, Facebook & Twtitter) to conduct outreach with safer sex and HIV prevention content prevention messages targeted to MSMs in response to the growing rate of HIV infections among MSM in Connecticut BEC training used to guide evaluation planning 11

Evaluation Questions EQ 1) How much outreach are volunteers doing and what are their demographic characteristics? EQ 2) As a result of their participation in Safety Net, did the program volunteers: demonstrate increased knowledge about HIV, HIV prevention, and access to HIV CTR have an improved positive attitude about safer sex and talking with peers and partners about safer sex and HIV access HIV CTR (either for the first time or during the past 3 months) EQ 3) Did HIV CTR clients at HGLHC, APH, and LCS indicate that they were referred by Safety Net? EQ 4) Did the number and percentage of MSM who access HIV counseling, testing and referral services (HIV CTR) change at the following sites HGLHC, APH, LCS? Data Collection Methods Surveys: Survey Monkey monthly surveys of volunteers; collection of demographic information about volunteers at intake Record Review: Department of Public Health and CDC annual records, intake and testing records, HGLHC, APH, LCS 12

Initial Findings Findings Volunteer diversity was less than desired in terms of age and race/ethnicity HIV Prevention and education messaging has been happening and reaching a wide audience, though reporting has been less than desired. The total number of people reached by the program may be greater than we can accurately measure. 13

Action Steps New HIV incidence data shows that MSM of color are becoming infected at alarming and disproportionate rates, therefore our target population should be geared towards this highrisk group. The majority of people we would like to target will be MSM of color. The lack of diversity among the current volunteer pool should be rectified going forward by recruiting more volunteers of color whose peers will be MSM of color. Action Steps Continue to ensure a large enough pool of volunteers are actively sending the messages and responding to the monthly surveys. Recruit more volunteers, especially: those over 30 years of age people of color. Identify why the response to the surveys is so low and what we can do about it so that data collection can continue. 14

Building Evaluation Capacity at Latino Community Services Years 1-4 Alumni Group BEC Years One & Two Outcomes: - Completed Medical Case Management Evaluation - Formalized evaluation framework for the agency - Increased staff involvement in the evaluation design process - Built internal capacity to do in-house evaluation on two programs: - Healthy Families & Communities (federally funded) - Spiritual Self-Schema Therapy (state funded) 15

Healthy Families & Communities: Evaluation Questions 1. How was the program implemented and to what extent were stakeholders satisfied with program implementation? 2. How and to what extent did this program impact participants in the following areas: programmatic capacity, partnerships and linkages, sustainability, infrastructure, behavior and utilization, systems change, and increasing access? Spiritual Self-Schema Therapy: Evaluation Questions 1. How is 3-S Therapy delivered to clients and how do clients respond to the program overall? 2. How and to what extent have clients experienced a self-schema shift and increased their motivation to reduce their HIV risk behaviors? 16

BEC Year 3 Evaluation of Health Communication/Public Information Health Communication/Public Information (HC/PI) events are health fairs and other similar community education events that are attended by Latino Community Services (LCS) staff, along with presenting agency materials and resources, and/or offering free rapid HIV testing Evaluation Questions 1. How and to what extent does the delivery of HC/PI impact: 1. Staff Learning 2. Number of clients accessing LCS services 3. Agency Publicity 2. What is the return on investment received by LCS for participating in HC/PI events 17

BEC Year 4 Evaluation of Linguistic Services Program The medical interpreter Part A Program at Latino Community Services, Inc. (LCS) is a core service that embodies the client s cultural comfort level and their needs associated with HIV/AIDS. This program gives services providers access to bilingual interpretation services for Ryan White Eligible clients. Clients must be living within Hartford, Tolland and Middlesex counties. Evaluative Questions How and to what extent do medical interpreting ti services impact the lives of clients? Are clients receiving Medical interpreting services more likely to adhere to treatment? 18

Trainee Assessment/ Next Steps All projects were presented to an audience including multiple stakeholders All projects were subjected to multiple reviews. MQP Former REP trainees Foundation Officials BEC Trainer Trainee Assessment/ Next Steps Michael Quinn Patton: described the efforts of BEC trainees as high quality evaluation work clarified that reports developed by BEC participants uniformly demonstrated evaluative thinking. further described BEC as a very impressive demonstration of how local agencies can meaningfully and usefully engage in evaluation thinking, and stated that BEC clearly built evaluation capacity in participating agencies. Alumni study continues 19