End of Project Evaluation of the PANCAP Global Fund Round 9 Project

Transcription

1 End of Project Evaluation of the PANCAP Global Fund Round 9 Project Grant Agreement No. MAC-910-G02-H Conducted by: Ansari Z. Ameen, Ph.D. Rachel Eersel, MD, MPH Ward Schrooten, MD, MSc., Ph.D. November 2015

2 Acknowledgements The consulting team wishes to acknowledge the support from the following institutions/persons: Global Fund PANCAP PACC and RCM The Principal Recipient Sub-Recipients (PCU, CARPHA, COIN, CMLF, UWI, The OECS Secretariat and EDC), and Sub-Sub-Recipients (CVC, CBMP, ILO, CRN+ ) The US Government (CDC, USAID, PEPFAR) UNAIDS PAHO Ministries of Health National AIDS Programmes / Committees Public and Private Laboratories Non-Governmental Organizations Beneficiary Populations Non-institutionally affiliated persons (Various key stakeholders) 2

3 Disclaimer The evaluation questions and methodology were directed at addressing the overall contributions of the Grant to the Regional HIV response and the CRSF rather than evaluating specific project components. This evaluation report is therefore based on data and information collected from existing written reports and input from a wide range of stakeholders. The evaluation team acknowledges the presence of selection bias at several levels and took that into consideration when synthesizing the findings. Findings and recommendations are based on the evaluation team s collective analysis of information and feedback from stakeholders. This report contains the views of the independent external evaluators and does not necessarily reflect the opinions of individual stakeholders or institutions. 3

4 Table of contents Executive Summary... 6 List of acronyms List of tables and figures Overview of the Global Fund Project Objectives of the Grant Sub-recipients Evaluation objectives and methodology Evaluation objectives Evaluation questions Evaluation methodology Evaluation findings Budgetary allocations Value of the Grant on Health Outcomes Promising practices The Caribbean Public Health Agency Caribbean Med Labs Foundation Caribbean Vulnerable Coalition / Centro de Orientacion E Investigacion Nacional Education Development Council Organisation of Eastern Caribbean States PANCAP Coordinating Unit University of the West Indies Branding Sustainability Contribution of the Grant to the CRSF Contribution to the overall objectives of the Grant Lessons learned and implications for the Regional HIV response and projects Grant coordination, management and implementation Non-programmatic accomplishments under the Grant Role of PANCAP in the Regional HIV response Priorities: Recommendations for the new PANCAP Global Fund concept note

5 5 Recommendations for the design of Regional projects Appendix 1: Summary of Key Findings, Challenges and Recommendations Appendix 2: List of Persons Interviewed Appendix 3: Documents Consulted

6 Executive Summary The Global Fund to Fight AIDS, Tuberculosis and Malaria Agreement MAC-910-G02-H entitled Fighting HIV in the Caribbean: a Strategic Regional Approach was signed on behalf of the Community on 5 October The GFATM awarded a total of US$ 28,345,234 to the Pan Caribbean Partnership Against HIV/AIDS (PANCAP) over a period of five years ( ). Phase 1 of the Grant was from January 1, 2011 to December 31, Phase 2 extends from January 1, 2013 to December 31 st, The larger project goals of the Grant are those of the CRSF and are: 1) to reduce the number of new infections; 2) to lower mortality due to HIV; and 3) to lessen the social and economic impact of HIV/AIDS on households, each by 25%. The objectives of the Grant are: - An enabling environment that fosters universal access to HIV services; - Reduced HIV transmission in vulnerable populations; - Lower people living with HIV (PLH) morbidity and mortality in the small islands that make up the Organization of Eastern Caribbean States (OECS); - Improved human and laboratory resources (health systems strengthening); and, - Better information on the epidemic and the response. - Build capacity and promote sustainability (added in Phase II). The project benefits 16 countries including Antigua and Barbuda, Belize, Bahamas, Barbados, Dominica, Dominican Republic, Grenada, Guyana, Haiti, Jamaica, Montserrat, St Kitts & Nevis, Saint Lucia, St Vincent & The Grenadines, Trinidad & Tobago, and Suriname. Populations targeted under the Project include: men who have sex with men (MSM); sex workers (SWs); drug users (DUs); prisoners; youth (aged 15-24); migrant and mobile populations; and, People Living with HIV (PLH). The Project was implemented by seven sub-recipients (SRs) and four sub-sub recipients (SSRs). The SRs and SSRs are: 1. Caribbean Health Research Council (CHRC) (now part of The Caribbean Public Health Agency) (SR); 2. Caribbean Med Labs Foundation (CMLF) (SR); 6

7 3. Centro de Orientacion E Investigacion Nacional (COIN) (SR); a. Caribbean Vulnerable Coalition (CVC) (SSR). 4. Education Development Council (EDC) (SR in Phase I only); 5. Organisation of Eastern Caribbean States (OECS) (SR); 6. PANCAP Coordinating Unit (PCU) (SR); a. The International Labour Organisation (ILO) (SSR); b. Caribbean Broadcast Media Partnership on HIV/AIDS (CBMP) (SSR in Phase I only); c. Caribbean Network of People Living with HIV (CRN+) (SSR); 7. University of the West Indies (UWI) (SR). The Principal Recipient commissioned a retrospective end-of-project evaluation of the Grant. The evaluation questions are: 1. To what extent has the Project contributed to achieving the goals and objectives of the CRSF? 2. To what extent has the Project been of added value to the region in delivering efficiently and effectively on the health outcomes for the targeted populations and countries? 3. What lessons can be derived from the implementation of this Project to strengthen future Regional and National strategies and projects in addressing HIV and AIDS? 4. What promising practices fostered under the Project can be replicated or sustained and how? 5. What are the priorities for follow-up after the end of the Project? 6. What mechanisms were implemented or needed to facilitate sustainability of the Grant activities and Regional HIV response? 7

8 Methodology An Evaluation Oversight Committee, comprised of representatives from the Principal Recipient and each current sub-recipient, provided technical feedback and monitored the evaluation process. The evaluation team addressed feedback from the Oversight Committee on the evaluation sub-questions, evaluation methodology, protocol and data collection instruments. Data sources for the evaluation included project background documents, an analysis of existing secondary data (Grant performance, GARP and PAHO-supplied data), and information from interviews and focus groups with various categories of stakeholders from the 16 beneficiary countries. Among the categories of stakeholders interviewed were Grant beneficiary populations (SW, MSM, Transgender People, Marginalized Youth, Drug Users and ex-inmates, CHLI scholars, DrPh students and CARTTA workshop participants), NGO MiniGrant recipients, sub-recipient and sub-sub-recipient staff, laboratory staff, Ministries of Health and National AIDS Programmes, staff from the Principal Recipient, donors, development partners and Regional Institutions. The evaluation team conducted in-country consultations in nine (9) countries and virtual interviews with stakeholders in the remaining seven (7) beneficiary countries. In total, interviews and focus groups were conducted with over 360 persons. The evaluation findings are based on the evaluator s analysis and synthesis of data from the various sources with efforts to account for selection and other biases. The findings represent the views of the evaluators and may or may not accord with the views of specific individuals and institutions. Key Findings The total value of the Grant was US$ 25,273,865 of which US$ 9,362,649 (37%) was for Phase 1 and US$ 15,911,216 (63%) for Phase II. CVC/COIN implemented the prevention component that targeted vulnerable populations and was allocated the largest part of the budget of US$ 8,718,166 (34%). The PCU received 19% of the Grant resources to work toward creating an enabling environment that fosters universal access to treatment care and support, while the 8

9 OECS component aimed at reducing mortality and morbidity in the OECS countries received 9 % or approximately $2.3 million of the total Grant allocation. CMLF s laboratory strengthening, UWI s training initiatives and CARPHA s M&E components were allocated 8, 6 and 5 percent, respectively. CARICOM Secretariat received US$ 4,192,583 (17% of total budget) for the management of the Project. The evaluation team examined available Health Information data for the 16 beneficiary countries to address evaluation question 2: To what extent has the Project been of added value to the Region in delivering efficiently and effectively on the health outcomes for the targeted populations and countries? The main source of standardized comparable data on HIV is from UNAIDS. A total of 62 indicators were identified that could potentially address progress of the Grant s six objectives. In general, countries reporting on the indicators was inconsistent by indicator, sub-grouping and year. This resulted in insufficient consistent and comparable data across the five year Grant period to permit meaningful monitoring of outcomes of the Grant s objectives. This finding leads to the recommendation that the countries in the Region, under the leadership and guidance of CARPHA, UNAIDS and PAHO should agree to collect a standardized minimum set of data that will permit tracking progress of the epidemic in the Caribbean. Efforts are underway to finalize the M&E framework for the CRSF ( ). While alignment to international reporting requirements is important, attention to ensuring the use of data from countries routine health information is needed in order to generate more consistent data at lower cost. The work of the various sub-recipients and sub-sub recipients under the Grant has generated a number of promising practices (Evaluation Question 3). The Caribbean Public Health Agency worked toward increasing the availability of strategic information on HIV in the Region. Their efforts were focused on assisting countries to develop and implement M&E plans, supporting countries with the analysis of data and the production of data-driven reports, the conduct of national programme evaluations or their components, and M&E capacity building. Despite preexisting and ongoing M&E system development challenges, CARPHA successfully implemented its work programme and support to countries. Overall, the support was well-received and consistent with National needs and strategies. Recommendations for Regional strategic 9

10 information production efforts include: 1) the refinement of the Regional M&E capacity development strategy with an aim toward integration of M&E for HIV into broader Ministry of Health M&E systems, providing technical support to Ministries of Health with M&E system integration, the development of a Regional HIV M&E information system, and increased advocacy for M&E system development and resource allocation at the National and Regional levels. The Caribbean Med Labs Foundation sought to strengthen laboratory and HIV care and treatment support systems in the Region. CMLF provided customized support to countries based on needs assessments. Accomplishments include the development of laboratory strategic plans and policies for 10 and 13 countries, respectively, the development of an electronic laboratory information system for use by labs that cannot afford commercial systems, the conduct of a number of activities aimed at improving the quality of laboratory services including proficiency testing, support with planning the Regional reference laboratory systems and needed support to the CCAS meetings. The laboratory strengthening needs in the Region outstrip the capacity of any single institution. Findings from this evaluation suggests the need for CARPHA to lead the process of developing a laboratory strengthening strategy for the Region in collaboration with CDC, CMLF and other stakeholders. CARPHA should engage CMLF and CDC to coordinate laboratory strengthening efforts in the Region. CMLF staff are well-grounded in the Region and have considerable expertise. This organization can therefore make a significant contribution to lab strengthening in the Region through supporting labs with the implementation of plans and policies, training of laboratory staff, and strengthening lab quality and information systems. CVC/COIN s work toward reducing HIV transmission in vulnerable populations consisted of efforts to strengthen networks of vulnerable populations, facilitating National programming for vulnerable populations, outreach to vulnerable populations, monitoring and evaluation and research. CVC/COIN s efforts accomplishments were noteworthy and included capacity development and strengthening of the networks of sex workers, MSM, Transgender People, and youth. CVC/COIN developed and used a peer educator training manual to train peer educators in six countries. Their mini-grant initiative supported 34 NGOs to conduct outreach to vulnerable populations through the peer educator model. Projects funded by CVC/COIN were developed 10

11 from situation analyses and research conducted under the project. The M&E system and webbased application developed by CVC/COIN and used by the NGOs will be used beyond the life of this Grant. NGOs supported through the initiative were strengthened in many areas, including M&E. An M&E system for monitoring organizational developments by NGOs should be developed and CVC/COIN s model of NGO engagement should be documented and shared. Deliberate engagement and collaboration with National programmes by CVC/COIN will contribute to the sustainability of NGO and their efforts. Data from PAHO through 2013 and from published studies on the OECS show declines in HIV mortality, stabilization in HIV incidence, increased utilization of viral load testing and increased viral suppression. The consistent availability and distribution of ARVs by the OECS PPS contributed to these improvements. The PPS system works well and has resulted in significantly reduced prices for ARVs. That system works well and has contributed to significantly reduced prices for ARVs. There is movement toward using the mechanism for the procurement of laboratory supplies. Despite the availability of ARVs, there were treatment challenges with monitoring patient s viral loads. Several countries failed to absorb the cost of viral loads after the Grant-funded period for viral load support ended thereby resulting in a period of time during which viral loads were unavailable. CDC support remedied the situation by paying for viral loads. The OECS has been working toward developing and implementing a case based surveillance system for HIV. Progress has been made but full implementation will not be achieved under this Grant. There are immediate needs in the OECS to ensure the availability of essential HIV tests and ensure that budgeting for HIV treatment includes not only ARVs but essential tests. Electronic laboratory management information systems are also needed; the CMLF E-logbook could contribute to addressing this need. The PCU advanced the Justice for all initiative through the production of model legislation and policies. Justice for all consultations in 7 countries resulted in the incorporation of elements of the JFA initiative by some National programmes. The slower than expected uptake of the JFA programme may be related to the perception that it is HIV specific, the slow pace of affecting legislative change and the need for a broader MOH response beyond the NAP to address all 11

12 elements of the JFA initiative. The JFA initiative should be continued as per the recommendation of the Special Council of Human and Social Development (COSHOD). The PCU s support to CRN+ was significant and is helping to revitalize that organization. CRN+ can play an important role in Regional advocacy for PLH and strengthen the organizational structures of National networks. The PCU also supported the work of the ILO to train Caribbean Congress of Labour (CCL) affiliate members and implement model programmes in four (4) countries. Continued work by the ILO and its affiliate members has the potential to increase the reach of HIV prevention interventions both for vulnerable groups and the general population. Continued effort to advance this agenda is needed. CBMP s efforts to implement mass media campaigns were successful. Over 200 media broadcasters in 14 countries were trained who in turn engaged in HIV programming at the national level. The live up and Regional testing campaigns were also successful. The efforts by EDC to facilitate implementation of HFLE and skills-based education in schools resulted in draft education policies being developed in 5 countries. Additional effort is needed to engage Ministries of Education and support the use of HFLE curricula in schools. The University of the West Indies capacity building initiatives have all contributed to human resource strengthening in the Region. The CHLI, DrPh and CARTTA programmes are consistent with country s capacity development strategies and needs. The CHLI programme contributed to personal growth and development of the professional who participated. The lack of strategic selection criteria resulted in fewer reported changes at the organizational change than expected. The DrPh programme is academically rigorous but the length of the programme and small number of students will necessarily limit its impact. The CARTTA programme has built capacity in leadership, management and HIV prevention. The closure of CHART and CHLI programmes and the end of the CARTTA programme at the end of the Grant will result in the absence of a capacity building structure in the Region. Capacity development needs continue to exist in relation to clinical care for HIV, scaling up prevention interventions to vulnerable groups, programme management and leadership and laboratory skills and management. 12

13 This evaluation finds that overall there was insufficient sustainability planning under the Grant. Sustainability was largely conceived as securing additional Grants rather than developing internal structures and mechanisms to continue efforts initiated under the Grant. Sustainability planning was initiated late and the work of several organizations will stop with the end of this Grant. The OECS was successful in securing additional resources from the Global Fund. Also, CVC/COIN has been actively engaged in resource mobilization through proposal writing before the end of the Grant. CVC/COIN will continue its work with NGOs through funding from the MAC AIDS foundation. CVC/COIN is also in the process of preparing to submit a Concept Note to the Global Fund. PANCAP has also recently initiated the development of a resource mobilization plan to address challenges with the sustainability of the Regional HIV response. The evaluation team conducted a review of the Caribbean Regional Strategic Framework ( ) and the Project documents to address evaluation question 1: To what extent has the Project contributed to achieving the goals and objectives of the CRSF? The review and analysis involved comparisons of the goals and objectives, alignment of the main Grant components or work with the objectives of the CRSF and alignment of the expected results of the CRSF with the Grant components of work. Findings from this analysis revealed that the Grant was clearly aligned and subsumed under the CRSF, all of the strategic objectives of the CRSF were either clearly or partially addressed for four (4) of the six (6) CRSF priority areas. Over two-thirds (65%) of the expected results of the CRSF were either clearly or partially addressed by the Grant. In conclusion, this evaluation finds clear alignment between the Grant and the CRSF. The Grant covers a small subset of the CRSF. Additional National and Regional initiatives were needed to fully address the CRSF. There are a number of lessons learned from implementation of this Grant that can be used to strengthen future Regional and National initiatives. The successful implementation of this Grant provides clear evidence that the Caribbean s Regional institutions have the ability and capacity to manage large multi-country Grants and meet donor s financial, programmatic and accountability requirements. The performance-based nature of the Global Fund s system was a major contributor to the successful planning and implementation of the Grant. Requirements linking disbursements to programmatic performance resulted in careful work planning and 13

14 budgeting. The Global Fund s oversight mechanisms through the use of the LFA and Global Fund reviews also supported careful programme planning. Despite occasional delays in disbursements, the Principal Recipient and sub-recipients were able to accelerate work plans to meet targets. Internal factors that contributed to the success were the project oversight and management structures including use of the CARICOM Secretariat s Project Management Group, the Priority Area Coordinating Committee and the Regional Coordinating Mechanism. The location of the Principal Recipient at the CARICOM Secretariat resulted in all organs of that institute providing support to the Grant. These included the legal, internal audit, human resources, statistics, finance, and procurement departments. There is widespread consensus that the small Principal Recipient team worked collaboratively with the Global Fund staff and the sub-recipients to implement the Grant. The PR team s supportive supervision and monitoring of the project resulted in excellent working relationships among the sub-recipients. This is evidenced by a number of initially unplanned sub-recipient collaborations. The sub-recipients prior experience working with countries and stakeholders in the Region also facilitated programme implementation. More structured communications and linkages with National programmes and increased joint work planning among sub-recipients were possible. The Grant also resulted in institutional strengthening at several levels. The CARICOM Secretariat itself benefitted as a result of improved ability to manage multi-country Grants and adjustments made to reconcile cash versus accrual accounting systems. Staff from all institutions experienced capacity development in the areas of M&E, project management and financial management. These individuals will make contributions to their institutions and other projects after the end of the Grant. CVC and COIN have experienced the most institutional development from having successfully managed the largest proportion of the Grant s budget. Many NGOs reported increased capacity development that was directly attributable to the support from CVC/CON under this Grant. This evaluation also finds consensus that PANCAP has made an excellent contribution to the Region s HIV response since its creation. While there is a need to revisit the structure, roles and functions of the Partnership given changes in the epidemic, response and landscape, stakeholders 14

15 agreed that PANCAP has a material role to play in the Regional HIV response. The main functions identified for PANCAP are to coordinate implementation of the CRSF, advocacy and resource mobilization. Overall, this evaluation finds that the Caribbean is heavily dependent on external funding. Concerted effort is needed to reduce efforts on external funds though integration of HIV into broader health sector efforts. CARPHA and PAHO can play leading roles in supporting countries in the Region with HIV integration. Efforts are also needed to reinvigorate multi-sectoral responses. Findings from this evaluation in conjunction with reviews of upcoming work planned in the OECS and by CVC/COIN suggest that a new PANCAP Concept Note submission to the Global Fund should address 1) Strategic information, 2) Lab strengthening, 3) Capacity building, 4) Human rights and 5) HIV integration. 15

16 List of acronyms AIDS Acquired Immune Deficiency Syndrome ARV Antiretroviral Treatment BSS Behavioural Surveillance Survey CARICOM Caribbean Community (and common market) CariFLAGS, Caribbean Forum for Lesbian All-Sexuals & Gays CARPHA Caribbean Public Health Agency CARTTA Caribbean Regional Training and Technical Assistance CBMP Caribbean Broadcast Media Partnership on HIV/AIDS CBO Community-Based Organization CBS Case-Based Surveillance CCAS Caribbean Cytometry & Analytical Society CCL Caribbean Congress of Labour CD4 cell T-helper cell (Cluster of differentiation antigen 4) CDC Centers for Disease Control and Prevention CHLI Caribbean Health Leadership Institute CHRC Caribbean Health Research Council CME Continuing Medical Education CMLF Caribbean Med Labs Foundation CMO Chief Medical Officer COHSOD Council for Human and Social Development COIN Centro de Orientacion E Investigacion Nacional CRN+ Caribbean Network of People Living with HIV CROSQ CARICOM Regional Organisation for Standards and Quality CRSF Caribbean Regional Strategic Framework on HIV and AIDS CSO Civil Society Organization CSWC Caribbean Sex Worker Coalition CVC Caribbean Vulnerable Coalition DrPH Doctorate in Public Health DU Drug Users 16

17 EDC Epi-profile GARP GFATM HAPU HFLE HIV ILO IRB ISO JFA Lab Lepto LGBTI LIS LQMS-SIP M&E MAC AIDS MOH MSM MY NAP NCPI NGO OECS OI PACC PAHO PANCAP PCU PLH Education Development Council Epidemiological Profile Global AIDS Response Progress Reporting The Global Fund to Fight AIDS, Tuberculosis and Malaria HIV AIDS Project Unit Health and Family Life Education Human Immunodeficiency Virus International Labour Organisation Institutional Review Board International Organization for Standardization Justice for All Programme Laboratory Leptospirosis Lesbian Gay Bisexual Transgender and Intersex Laboratory Information Systems Laboratory Quality Management Systems - Stepwise Improvement Process Monitoring and Evaluation MAC AIDS Fund Ministry of Health Men Who Have Sex With Men Marginalized Youth National AIDS Program National Commitments and Policies Instrument Non-Governmental Organization Organization of Eastern Caribbean States Opportunistic Infections Priority Area Coordinating Committee Pan American Health Organization Pan Caribbean Partnership Against HIV/AIDS PANCAP Coordinating Unit People Living with HIV 17

18 PLHIV PR PT QMS RCM SR SSR STI SW TB UNAIDS USAID UWI VCT VL WHO People Living with HIV Principal Recipient Proficiency Testing Quality Management System Regional Coordinating Mechanism Sub-Recipients Sub-Sub Recipients Sexually Transmitted Infections Sex Workers Tuberculosis Joint United Nations Programme on HIV/AIDS United States Agency for International Development University of the West Indies Voluntary Counselling and Testing Viral Load World Health Organization 18

19 List of tables and figures Table 1: Monitoring and Evaluation Framework Table 2: Number of persons interviewed Table 3: Number of records for the 16 Beneficiary Countries, by data source Table 4: Number of indicators identified which could potentially report on the progress on each of the 6 PANCAP Grant objectives Table 5: Beneficiary Population outstanding needs Figure 1: Total and percentage by objectives Figure 2: Total budget by SR and SSR Figure 3: Number of different indicators reported (at least once) per Country Figure 4: Total number of individual indicators reported per year by the 16 Beneficiary Countries Figure 5: HIV testing among TB patients Figure 6: Annual number of HIV cases, AIDS cases and deaths from 1984 to 2013 in the six OECS Countries*..55 Figure 7: Twelve month retention on antiretroviral therapy in the six OECS Countries (GARPR data). 55 Figure 8: Viral load measurements of the six OECS countries done through a Regional Laboratory Referral Service*

20 1. Overview of the Global Fund Project The Global Fund to Fight AIDS, Tuberculosis and Malaria GFATM Agreement MAC-910-G02- H entitled Fighting HIV in the Caribbean: a Strategic Regional Approach was signed on behalf of the CARICOM Caribbean Community on 5 October The GFATM awarded a total of US$ 25,273,865 to the Pan Caribbean Partnership Against HIV/AIDS (PANCAP) over a period of five years ( ) which was implemented over two phases. Phase 1 of the Grant was from January 1, 2011 to December 31, Phase 2 of the Grant is currently being implemented. It was initiated on January 1, 2013 and is scheduled to end on December 31 st, Development and implementation of the Grant was guided by the Caribbean Regional Strategic Framework on HIV and AIDS (CRSF) , and the GFATM s mandates to focus on equitable access to services by key affected populations including sexual minorities, to respond directly to the current, documented epidemiological situation, and to use country-oriented and results-driven approaches. The larger project goals are those of the CRSF: 1. To reduce the number of new infections; 2. To lower mortality due to HIV; and 3. Lessen the social and economic impact of HIV/AIDS on households, each by 25%. 1.1 Objectives of the Grant The objectives of the project are: - An enabling environment that fosters universal access to HIV services; - Reduced HIV transmission in vulnerable populations; - Lower people living with HIV (PLH) morbidity and mortality in the small islands that make up the Organization of Eastern Caribbean States (OECS); - Improved human and laboratory resources (health systems strengthening); 20

21 - Better information on the epidemic and the response; and - Build capacity and promote sustainability (added in Phase II). The project benefits 16 countries including Antigua and Barbuda, Belize, Bahamas, Barbados, Dominica, Dominican Republic, Grenada, Guyana, Haiti, Jamaica, Montserrat, St Kitts & Nevis, Saint Lucia, St Vincent & The Grenadines, Trinidad & Tobago, and Suriname. Populations targeted under the Project include: men who have sex with men (MSM); sex workers (SWs); drug users (DUs); prisoners; youth (aged 15-24); migrant and mobile populations; and, People Living with HIV (PLH). 1.2 Sub-recipients The Project Management Unit within the Caribbean Community Secretariat coordinates grant implementation oversight functions with sub-recipients assuming responsibility for grant implementation activities. During Phase 1, the Project was implemented by seven sub-recipients (SRs) and four sub-sub recipients (SSRs). The SRs and SSRs are: 1. Caribbean Health Research Council (CHRC) (SR); 2. Caribbean Med Labs Foundation (CMLF) (SR); 3. Centro de Orientacion E Investigacion Nacional (COIN) (SR); a. Caribbean Vulnerable Coalition (CVC) (SSR). 4. Education Development Council (EDC) (SR); 5. Organisation of Eastern Caribbean States (OECS) (SR); 6. PANCAP Coordinating Unit (PCU) (SR); a. Caribbean Broadcast Media Partnership on HIV/AIDS (CBMP) (SSR); b. Caribbean Network of People Living with HIV (CRN+) (SSR); c. The International Labour Organisation (ILO) (SSR); 7. University of the West Indies (UWI) (SR). 21

22 Phase two of the Grant started on January 1, 2013 and is scheduled to end on December 31 st, Six SRs and two SSRs are implementing activities under Phase 2. The SRs are COIN/CVC, OECS, UWI, CMLF, the Caribbean Public Health Agency (CARPHA) which assumed the functions and staff of CHRC in January 2012, and PCU. Two SSRs - CRN+ and the ILO - continued their work under the PCU. COIN also took on CVC as an SSR and 36 organizations over Phase I and II to strengthen their work with vulnerable populations. 2. Evaluation objectives and methodology In accordance with recommended good practices, the Principal Recipient has made provisions for an end-of-project retrospective evaluation of the Grant, its original assumptions, and progress toward meeting the goals and objectives. Findings from this evaluation are intended to: Determine the extent to which the project has contributed to Regional progress regarding mitigating the consequences of HIV/AIDS; Inform future policies and programmes, advocacy and resource mobilization efforts; Contribute to redefining and adjusting Regional and National strategies; Provide information to inform the development of a new CRSF and Global Fund Grant application; and Identify lessons learned and good practices that can inform future Regional initiatives, including Global Fund projects. 2.1 Evaluation objectives The objectives of this end-of-project evaluation are: To assess the extent to which the Project has contributed to the Grant objectives; To explore the strengths and weaknesses of the regional approach to addressing HIV and AIDs in terms of value for money and health outcomes; To identify and document lessons learnt from Project implementation that can be used to inform future strategies, projects and practices (locally, nationally, regionally and internationally); 22

23 To identify and document the best practices cultivated by the SRs and SSRs under the Project which can be utilized or built on at a local, national or regional level to sustain gains made under the Project and strengthen HIV responses world-wide. To identify gaps, needs and opportunities for sustaining the regional response to the HIV epidemic. 2.2 Evaluation questions The evaluation questions are as follows: 1. To what extent has the Project contributed to achieving the goals and objectives of the CRSF? 2. To what extent has the Project been of added value to the region in delivering efficiently and effectively on the health outcomes for the targeted populations and countries? 3. What lessons can be derived from the implementation of this Project to strengthen future Regional and National strategies and projects in addressing HIV and AIDS? 4. What promising practices fostered under the Project can be replicated or sustained and how? 5. What are the priorities for follow-up after the end of the Project? 6. What mechanisms were implemented or needed to facilitate sustainability of the Grant activities and Regional HIV response? 2.3 Evaluation methodology Monitoring and Evaluation Framework This evaluation employed both primary and secondary data collection methods to collect the information needed to address the evaluation questions and sub-questions. Table 1 summarizes the main evaluation questions, sub-questions and corresponding data sources and collection methodology. 23

24 Table 1: Monitoring and Evaluation Framework Evaluation Question Sub-Questions Data Source Data Collection Method(s) 1. To what extent has the Project contributed to achieving the goals and objectives of the CRSF? 1.1 To what extent are the goals and objectives of the Project in alignment with the CRSF ? CRSF Grant Agreement Document Review and Analysis Compare and align CRSF with Grant Agreement and workplans 1.2 How did the Grant perform against the performance targets in the Grant Agreement? Grant Agreement Performance Framework PUDR reports SR Progress reports for reasons why targets were exceeded or not met SR staff Document Review and Analysis Review of the PF submitted to the GF for each semester. Identification of achievement, over and under performance for each indicator. Creation of a table that summarizes grant indicator performance (i.e., table with semester, # met, over achieved and under achieved). Review of PR and SR reports for explanations for deviations Interviews with SRs and implementing agencies for further explanation and clarifications 2. To what extent has the Project been of added value to the Region in delivering efficiently and effectively on the health outcomes for the targeted populations and countries? 2.1 What services / support / benefits did each country receive via the Grant and to what extent did the services complement and/or strengthen National programmes in beneficiary countries? PR and SR workplans PR and SR progress reports Mission reports National Strategic Plans SR and SSR staff NAP staff Laboratory staff NGOs, CBOs Key populations Document Review and Analysis Abstract and summarize information on the services and support provided to each beneficiary country via the Grant Interviews with SRs and implementing agencies for clarification and verification of services provided to beneficiary countries Interviews with stakeholder groups in beneficiary countries to review services received and assess fit and complementarities with national strategies and needs 24

25 2.2 To what extent were the technical approaches and activities appropriately designed to meet the goals and objectives of the Grant? Grant Agreement PR, SR and SSR workplans Document Review and Analysis Review of the relationships and correspondence between Grant activities and the goals and objectives of the Grant 2.3 To what extent were Regional interventions provided in ways that maximized access/benefit to National programmes and target populations? 2.4 What were the outcomes at the Regional, National, subnational levels and for key populations? 2.5 To what extent can National programmes and target populations identify Grantfunded interventions that contributed to changes at the National level and in beneficiary populations? PR, SR and SSR staff NAP staff Laboratory staff NGOs, CBOs Key populations PR and SR progress reports Mission reports National AIDS Programme M&E and surveillance data SR and SSR M&E data PR, SR and SSR staff NAP staff Laboratory staff NGOs, CBOs Key populations Staff from development agencies NAPs Laboratories NGOs, CBOs Key populations in selected countries Interviews with staff from the PR, SR and SSRs Interviews with stakeholder groups in beneficiary countries Interviews with staff from the PR, SR and SSRs Interviews with stakeholder groups in beneficiary countries Secondary analysis of M&E and surveillance data from NAPs and SR and SSR M&E systems Interviews with staff from development agencies Interviews with staff from the PR, SR and SSRs Interviews with stakeholder groups in beneficiary countries Interviews with stakeholder groups in beneficiary countries 3 What lessons can be derived from the 3.1 What was the overall approach used to develop and design the Project? Development partners Regional Interviews with staff who were involved in the PANCAP Round 9 proposal development process 25

26 implementation of this Project to strengthen future Regional and National strategies and projects in addressing HIV and AIDS? 4 What promising practices fostered under the Project can be replicated or sustained and how? 5 What are the priorities for follow- up after the end of the Project? 3.2 What was the approach used to coordinate, manage and implement the Project? 4.1 What are the lessons learned and promising practices from the Project at the PR, SR, SSR levels in the following areas: Project design; Project coordination, management and implementation; and Intervention development and implementation. 5.1 What are the outstanding needs, priorities and interventions needed for beneficiary populations, Regional support agencies (SRs and SSRs) and, National programmes targeted under the grant? 5.2 What are the main Regional actions that will be required in the upcoming period? What are specific recommendations for the next Global Fund grant? Institutions RCM members NAPs Selected individuals PR, SR and SSR staff RCM PACC PR and SR progress reports Best practice reports Evaluation Reports PR, SR and SSR staff The CARICOM Secretariat Development partners Regional Institutions PR, SR and SSR staff NGOs, CBOs; Key populations in selected countries Development partners Regional Institutions PR, SR and SSR staff NGOs, CBOs; Key populations in selected countries Individual and group interviews Individual and group interviews Individual and group interviews Individual and group interviews 26

27 6 What mechanisms were implemented or needed to facilitate sustainability of the Grant activities and Regional HIV response? 6.1 What was done under the Grant to sustain the critical activities and HIV response? Development partners Regional Institutions PR, SR and SSR staff NGOs, CBOs; Individual and group interviews 6.2 What was done by way of planning for a continued Regional response? 6.3 What actions could be taken to promote and sustain the Regional HIV response? Development partners Regional Institutions PR, SR and SSR staff NGOs, CBOs Development partners Regional Institutions PR, SR and SSR staff NGOs, CBOs Individual and group interviews Individual and group interviews Document Review and Analysis The evaluation team conducted a review of relevant Project documents in order to address the following evaluation questions: To what extent has the Project contributed to achieving the goals and objectives of the CRSF? To what extent has the Project been of added value to the Region in delivering efficiently and effectively on the health outcomes for the targeted populations and countries? What promising practices fostered under the Project can be replicated or sustained and how? 27

28 Secondary Data Analysis The team conducted secondary data analysis to answer the following evaluation questions: To what extent has the Project contributed to achieving the goals and objectives of the CRSF? To what extent has the Project been of added value to the Region in delivering efficiently and effectively on the health outcomes for the targeted populations and countries? The evaluation team reviewed the performance indicators and information from the Project Update and Disbursement Requests to complement the findings from the document analysis in order to develop a comprehensive view of the progress, challenges and accomplishments under the Grant. The evaluation team accessed data from UNAIDS in an effort to determine the extent to which the health outcomes of key populations improved as a result of this Project. The intention was to access available standardized data from the 16 beneficiary countries or subsets of countries on key indicators that were related to the project objectives. The team also received data for the six OECS countries from the Pan American Health Organisation (PAHO). Interviews and focus groups with key stakeholders The evaluation team conducted individual interviews with key stakeholders and focus groups with each beneficiary population to complement information from the document review and analysis of secondary data. Where appropriate, group interviews with key stakeholders were conducted. Information from interviews and focus groups contributed to answering all of the six evaluation questions and related sub-questions. The following categories of stakeholders were interviewed: Regional/International Institutions and Development Partners including: PAHO, UNAIDS, CARICOM, CDC, PEPFAR, USAID, The Global Fund. Grant Management and Implementing Partners including: The Principal Recipient, The Regional Coordinating Mechanism, The OECS Secretariat, OECS PPS, COIN/CVC, CBMP, CMLF, UWI, PCU, CARPHA, EDC, ILO, CRN+, PACC, RCM Institutional Grant Beneficiaries: National AIDS Programmes, public and private laboratories. 28

29 Beneficiary Populations: Representatives of Key Populations (MSM, SW, MY, Drug Users, Transgender People) and CHLI, DrPH and CARTTA scholars. Mini-grant Recipients Non-Institutionally Affiliated Persons Countries were selected for in-country consultations if there were direct services for the specified key populations in country and also if Regional Institutions, Development Partners, Grant Management and Implementing Partners were located in that country. Since there is variability in Member States in the OECS, one additional OECS country besides St. Lucia was included for the in-country consultations. In-country individual and/or group interviews were conducted in nine countries and focus groups with each key population in six of the nine countries where CVC/COIN worked. The countries in which the evaluation team conducted in-country interviews and/or focus groups included all countries in which CVC/COIN worked (The Dominican Republic, Haiti, Jamaica, Suriname, Guyana, and Trinidad), two OECS countries (St. Lucia and Grenada) and Barbados. Virtual interviews (via telephone or Skype) were conducted in seven countries with persons from Antigua and Barbuda, Belize, The Bahamas, Dominica, Montserrat, St Kitts & Nevis, and St. Vincent & the Grenadines. Table 2: Number of persons interviewed Category Description Number of persons National Representatives 16 Countries 105 (MOH, NAP/NAC, Lab) Regional and International USAID, CDC, PAHO, CARICOM 10 Partners Secretariat, UNAIDS, GLOBAL FUND PR and Sub-Recipients 37 CVC COIN MiniGrant Beneficiaries Project Beneficiaries 34 NGOs 50 SW, MY, MSM/TRANS, DU, ex-inmates; 162 CHLI, DrPh & CARTTA Scholars 29

30 Total 364 Includes PACC, RCM and CCM members All interviewees received information on the purpose and nature of the evaluations. They were informed that their participation was voluntary and that the information would not be reported in any way that would allow for attribution to specific individuals. The representatives of the key populations were clearly advised that no names or identifying information was required. In addition, interviews with staff from the NGOs were conducted without the presence of the CVC/COIN National Coordinator; focus groups with key populations were conducted without the presence of NGO staff and focus groups were conducted in different languages (English, Dutch, Spanish and Creole) to allow interviewees to share their views in their own native languages. Evaluation Process An Evaluation Oversight Committee comprised of representatives from the Principal Recipient each current sub-recipients was created to monitor and oversee the evaluation. The evaluation team worked closely and collaboratively with the Oversight Committee throughout the entire evaluation. Following the Consultancy Inception meeting with the Oversight Committee, the consultants submitted an Inception Report detailing the agreed-upon steps and timelines. The evaluation team prepared evaluation sub-questions that were derived from the main evaluation questions. These were revised and finalized through an iterative process with the Oversight Committee. The evaluation team prepared an evaluation protocol that detailed the methods, procedures and data collection instruments. The team addressed and incorporated comments from the Oversight Committee before initiating data collection. Data were collected from July 20, 2015 to November 13, 2015 and involved both in-country consultations in nine countries and virtual interviews in seven countries. The evaluation team held conference calls with the Oversight Committee every two weeks during the course of the 30

31 consultancy. In instances where conference calls were not possible due to the team s travel schedule, the team submitted written reports to the Oversight Committee. In accordance with the evaluation protocol, the evaluation team presented the preliminary findings to the Oversight Committee. This meeting took place in Guyana on October 24, The evaluation team addressed comments and resubmitted a revised evaluation presentation to the Oversight Committee. The team also presented the findings at the Meeting of the National AIDS Programme Managers and Key Partners on October 30, 2015, as well as to the CARICOM Project Advisory Group on November 16,

32 3. Evaluation findings 3.1 Budgetary allocations The total value of the Grant was US$ 25,273,865. Of this US$ 9,362,649 (37%) was for Phase 1 of the Grant from January 1, 2011 to December 13, December 2012 and US$ 15,911,216 (63%) in Phase of the Grant from January 1, 2013 till the end of December 31, The budget was distributed among the six objectives and the programme management. The largest proportion of the budget of US$ 9,789,861 (39%) was allocated to the objective Reduced HIV transmission in vulnerable populations. The budget for programme management was US$ 4,148,524 which was (16%) of total budget. Figure 1 shows this distribution of the budget per objective and programme management. Figure 1: Total and percentage by objectives $1,389,881 ; 5% $526,110 ; 2% $4,192,584 ; 17% $2,938,974 ; 12% $4,148,524 ; 16% $9,780,861 ; 39% An enabling environment which forsters Universal Access to HIV Services Reduced HIV transmission in vulnerable Groups Reduced PLH morbidity and mortality (in the small islands of the OECS) Build Capacity and Promote Sustainability (health systems strengthening) Improved human and laboratory resources Better Information on the epidemic and the response Programme Management $2,296,931 ; 9% 32

33 The Project was implemented by seven SRs and four SSRs; CVC/COIN received the largest budget of US$ 8,718,166 (34% of the total budget), followed by PCU with a budget of US$ 4,819,613 (19% of the total budget). CARICOM Secretariat received US$ 4,192,583 (17% of total budget) for the management of the Project. Further details on the total budget by SR are presented in Fig. 2. Figure 2: Total budget by SR $1,389,881 ; 5% $391,606 ; 2% $2,018,088 ; 8% $1,446,996 ; 6% $2,296,932 ; 9% $4,192,583 ; 17% $4,819,613 ; 19% CARICOM PCU CVC/COIN OECS UWI CLMF CARPHA $8,718,166 ; 34% EDC 33

34 3.2 Value of the Grant on Health Outcomes This section aims to answer to evaluation question 2: To what extent has the Project been of added value to the Region in delivering efficiently and effectively on the health outcomes for the targeted populations and countries? The PANCAP Global Fund Round 9 Grant is a Regional Grant that covers 16 beneficiary countries. The evaluation team therefore reviewed existing available data in order to identify standardized comparable data across the 16 countries or subsets of countries that could contribute to determining outcomes of the six Grant objectives. The principal source of data that could contain standardized, comparably measured data across the beneficiary countries was the Global AIDS Response Progress Reporting (GARPR) data. Methodology The GARPR data were downloaded on October 18, 2015 from the UNAIDS website: The full data set included all reporting countries worldwide. The GARPR dataset combines data from different data sources. Table 3 shows the number of records provided by each data source. A subset was made that retained data for the 16 PANCAP Global Fund Round 9 Grant beneficiary countries. The subset contained records and included all indicators reported per country, year, and sub-group. The indicators relevant for this evaluation are included in the data from the sources UNAIDS_GARPR and WHO_Data (HIV/Tuberculosis reporting), which total 5398 records for the 16 beneficiary countries. 34

35 Table 3: Number of records for the 16 Beneficiary Countries, by data source Source Number of records UNAIDS_GARPR_ 2600 UNAIDS_Spectrum Estimates_ UNAIDS-WHO_Data_ 74 WHO_Data_ 2798 WORLD BANK_WDI_ 69 UNAIDS_Data_ 39 Total This analysis has been done in R version ( ), in a fully reproducible way. This means that each step, from downloading the data to selecting, filtering, analyzing data, and producing tables and figures is written in a syntax file. This provides full transparency of each of the analysis steps and allows the same analysis to be redone, for e.g. for future evaluations. The complete syntax is provided in Appendix 3. Findings In total, the 16 countries reported on 88 indicators. Figure 3 shows the number of different indicators reported (at least once) for each country. All countries have been reporting to GARP, though some variation in the number of indicators reported per country was noted. The variation in number of indicators reported in different years however is more important to consider. Figure 4 shows the total number of individual indicators reported per year by the 16 beneficiary countries. The fluctuations up to 2011 are because GARPR data were initially collected every other year. Thereafter, GARPR data collection was done on a yearly basis, with an increasing number of indicators reported each year. The WHO data include tuberculosis indicators. Part of the tuberculosis data which WHO gathers annually are submitted to the GARPR data. This is a constant set of indicators. For 2014 the tuberculosis data had not been added to GARPR at the time of this evaluation. 35

36 Figure 3: Number of different indicators reported (at least once) per Country Figure 4: Total number of individual indicators reported per year by the 16 Beneficiary Countries The evaluation team reviewed all available indicators and selected subsets which could potentially provide information on Regional progress for each of the six Grant objectives. The 36

37 number of indicators which could potentially report on the progress on each of the six Grant objectives is shown in Table 4. Table 4: Number of indicators identified which could potentially report on the progress on each of the 6 PANCAP Grant objectives OBJECTIVE NUMBER OF INDICATORS An enabling environment that fosters universal access to HIV services 42 Reduced HIV transmission in vulnerable populations 8 Lower people living with HIV (PLH) morbidity and mortality in the small islands that make up the Organization of Eastern Caribbean States (OECS) Improved human and laboratory resources (health systems strengthening) 2 Better information on the epidemic and the response 2 Build capacity and promote sustainability (added in Phase II) 6 2 The evaluation examined the subsets of indicators related to each Grant objective to determine if the data would permit an assessment of progress on each of the six Grant objectives: Objective: An enabling environment that fosters universal access to HIV services In total, 42 indicators have the potential to contribute to evaluating this objective. The majority of these were last reported in As such, they do not correspond to the Grant implementation period and therefore cannot be used for this evaluation. An additional three potentially relevant indicators were first reported in 2014, respectively by 15, 15 and 5 countries. These include: - Laws criminalizing any aspect of sex work; - Laws that criminalize same-sex sexual activities between consenting adults; and - Laws that specifically criminalize HIV non-disclosure, exposure or transmission Since these were only reported for the first time in 2014, they cannot provide information on the evolution under the current grant, but do offer opportunities for future follow-up. 37

38 In conclusion, limited, if any, standardized data are available over the Grant period to allow for measuring progress toward creating an enabling environment that fosters universal access to HIV services. Objective: Reduced HIV Transmission in Vulnerable Populations Eight potentially relevant indicators were identified for this objective. Most of these indicators were reported annually until However, these indicators were reported for different subgroups of the population, resulting in a total of 43 indicator-sub-group combinations. The number of countries reporting for each indicator-subgroup was found to be very low (generally two or three countries) and therefore do not permit aggregation that will permit assessing Regional progress toward reducing HIV transmission in vulnerable populations. In conclusion, insufficient standardized data exist across a sufficient number of countries to allow for measuring progress towards this objective. Objective: Lower People Living With HIV (PLH) Morbidity and Mortality in the Small Islands That Make Up the Organization of Eastern Caribbean States (OECS) Only two potentially relevant indicators were related to this objective: - Twelve month retention on antiretroviral therapy (see figure 7 in section 3.3.5). This indicator is reported annually by the majority of the OECS countries; and - Co-management of tuberculosis and HIV treatment: This indicator is reported annually by eight or nine of the 16 countries, except the OECS countries. In conclusion, standardized, comparable data for the OECS are only available for one indicator. This will not permit a proper assessment of the progress of the Grant toward meeting this objective. 38

39 Objective: Improved Human and Laboratory Resources (Health Systems Strengthening) For this objective, only two potentially relevant indicators were identified: - HIV testing among tuberculosis patients (see figure 5). This indicator was reported annually by all 16 beneficiary countries; and - People living with HIV screened for tuberculosis. Reporting on this indicator was more fragmented. In conclusion, data are available on only one indicator and this will not permit a proper assessment of the progress under the Grant toward meeting this objective. Figure 5: HIV testing among TB patients Objective: Better Information on the Epidemic and the Response Only two potentially relevant indicators were identified for this objective. These indicators capture whether or not countries submit GARPR and NCPI (National Commitments and Policies Instrument) data. Each year, all countries report yes on both indicators, showing that countries are reporting on GARPR and NCPI. Despite the availability of standardized data on two 39

40 indicators, the content of these indicators does not materially contribute to assessing Regional progress towards meeting this objective. Objective: Build capacity and promote sustainability (Added in Phase II) Six indicators were identified that could potentially contribute to evaluating this objective. Two of these were reported for the last time in 2011, therefore not reflecting the period under the Grant. The other four indicators are reported by only a subset of the countries. There is insufficient information available to permit an assessment of the progress on this objective. Overall Summary In summary, despite the fact that all 16 project countries report to GARPR, there are insufficient standardized HIV and Health outcome data available across the 16 Grant beneficiary countries that can contribute to evaluating progress toward meeting the Grant objectives. This is due to a combination of reasons. First, there is inconsistent country reporting on all GARP indicators. Second, although the total number of indicators reported on is high, they do not contain the information needed to provide a complete picture of the Region s progress in HIV. Finally, countries report on different subsets of indicators. Fragmented reporting makes it difficult to create a Regional picture. Recommendations The inability to identify a standardized set of comparable data across countries in the Region, even for a small number of indicators suggests that Global and Regional initiatives that aim to facilitate this objective have not been successful. Selected factors that contribute to this include: modifications to reporting requirements and guidelines stemming from changes in Global initiatives and commitments as the epidemic evolves, modifications to indicator definitions, differing reporting periods and reliance of survey (e.g., BSS and STEPs surveys) data that involve special data collection methods and high costs. Findings from this evaluation led to the recommendation that countries in the Region should agree to collect and report on a minimum set of indicators for HIV that would contribute to Regional progress tracking the HIV epidemic. These indicators should be clearly linked to the 40

41 main programmatic priorities of the CRSF. Caribbean Regional institutions should lead the process of identifying the set of indicators, and provide support in the coordination, data collection, analysis and reporting. The selected indicators should be consistent with international reporting requirements and operational definitions. GARPR indicators should be included. Inconsistent reporting in the past has been linked to the cost and effort associated with special data collection initiatives and surveys. Consequently, as much as possible, the minimum set of indicators should be based on routinely collected data. Indicators such as HIV incidence and mortality are traditionally being collected at National and Regional levels, yet they are not aggregated, reported and made publically available. CARPHA should make these data available in a manner that is similar to that used for GARPR indicators. 3.3 Promising practices Section 3.4 aims to provide an answer to evaluation question 4: What promising practices fostered under the Project can be replicated or sustained and how? The evaluation team reviewed the work of each of the Grant s sub-recipients in order to identify their main accomplishments, challenges, outstanding programmatic needs and to make recommendations for future initiatives. The findings are presented below, in alphabetical order, for each of the individual Sub-recipients and Sub-sub-Recipients The Caribbean Public Health Agency CARPHA s activities under the Project were related to the objective Better Information on the Epidemic and Response in both Phase I and Phase II. CARPHA s approach to increasing the availability and quality of information on the epidemic involved: 1. Building national capacity in monitoring and evaluation CARPHA conducted two Regional M&E workshops in Phase I that targeted M&E staff from beneficiary countries. The institution also conducted M&E workshop for subrecipients and partners to support Grant M&E processes. CARPHA also provided M&E 41

42 support to several beneficiary countries on specific issues. In addition to workshops, they facilitated M&E capacity development by employing a learn-by-doing approach to evaluations of National AIDS Programmes. Selected M&E staff from National and Regional programmes were assigned to join external evaluations of National programmes or components of these in 13 countries. 2. Supporting countries with improving data collection systems CARPHA supported nine countries over the two Grant phases with developing national monitoring and evaluation plans. The institution also supported selected NGOs with developing M&E frameworks in support of their vulnerable group interventions. In both instances, CARPHA also sought to develop capacity within the context of the M&E plan and framework development process. 3. Increasing countries analysis and use of M&E information CARPHA facilitated the analysis and use of information through its work with stakeholders to analyse national data and produce data-driven reports. CARPHA supported 14 of the 16 beneficiary countries in this area. CARPHA also conducted evaluations in 11 beneficiary countries. These evaluations employed a learn by doing methodology and served the dual purpose of evaluating programmes or programme components and building the capacity of both the local evaluation teams and external evaluators. CARPHA s participatory evaluation approach contributed to M&E capacity development of the in-country local evaluation teams. The learn by doing evaluation approach for conducting evaluations was an excellent capacity building tool. This approach allowed selected individuals to bring their own expertise to bear on evaluations while also learning from other team members and area experts. Participants consistently indicated that the exercise increased their knowledge and capacity. The Regional M&E training sessions increased knowledge of participants, although the absence of built-in follow up activities will reduce application. Support to countries 42

43 with developing M&E plans and frameworks resulted in roadmaps to structure and guide their data collection systems to monitor and evaluate implementation of their national responses. The evaluations and data-driven reports were specific to the countries needs. These products and processes resulted in findings and recommendations for strengthening M&E systems and programmes. Overall, countries were satisfied with the support given and indicated that the services were consistent with national priorities and needs. CARPHA encountered a number of challenges related to the implementation of activities and to increasing the availability of information on the HIV epidemic in the Region. Implementation challenges included the timing and availability of countries to receive the support offered under the Grant and delays and difficulty with receiving sign off and approval of final reports in selected countries. Related to the finalization of the reports, is the insufficient use of information and recommendations contained in the reports. A formal examination of the extent to which countries have used the findings and acted on the recommendations is warranted. Other operational challenges included securing cooperation and collaboration between different national agencies and actors, and the need for services to be provided in native languages in Haiti, the Dominican Republic and Suriname. Broader M&E system development challenges that affect the production of strategic information on the epidemic include insufficient institutionalization of M&E in some countries and the need for stronger M&E leadership. Recommendations CARPHA is the principal institution in the Region with the mandate to strengthen M&E systems for health including HIV. As such, this evaluation puts forth a number of recommendations for building systems and structures to increase the availability and quality of information on the HIV epidemic. The recommendations are clustered in four areas. M&E Capacity Building The evaluation recommends that CARPHA conduct a mapping of countries M&E needs. CARPHA can review and update its M&E capacity strategy to incorporate both individual and cross-cutting country needs. The updated strategy could include building the capacity of MOH 43

44 M&E staff to provide support to sub-national partners and NGOs, M&E training with systematic in-country follow-up support, specific M&E training for use with different audiences (i.e., policymakers, programme managers, NAPs, CSOs), mechanisms for providing training in native languages (Spanish, French and Dutch), the use of alternative modalities of training (for example web-based courses and institutions of higher learning) and the use of regional area experts that can support evaluations. National M&E Strategic Planning As the Region s public health agency, CARPHA should provide leadership and guidance to countries with national M&E strategic planning. This support can include supporting Ministries of Health with planning and developing structures for M&E in health, including HIV integration. As part of this it is recommended that CARPHA facilitate collaboration between M&E surveillance and epidemiology departments. CARPHA s work under the PANCAP Grant illustrates the availability of national-level data to inform programmes and policies. Deliberate efforts to increase analysis of existing data and translating data and information into programmatic and policy adjustments are needed. Regional M&E Strategic Planning As illustrated in this evaluation, the Caribbean lacks sufficient strategic information to properly measure progress toward addressing the HIV epidemic. Furthermore, no institution has assumed responsibility for collecting, managing, analyzing and disseminating information from countries in the Region. Findings from the evaluation suggest that CARPHA and UNAIDS should lead the Region s M&E strategic planning processes in collaboration with the M&E technical working group. CARPHA and UNAIDS are encouraged to lead the process for developing the indicators to monitor the implementation of the CSRF with an emphasis on the use of routine health information system data. CARPHA is also in the best position to collect, analyze, manage and disseminate information from the Regional M&E system. 44

45 Advocacy for M&E CARPHA is also uniquely positioned to advocate for M&E systems development in the Region. CARPHA is encouraged to advocate for institutionalizing M&E in MOH structure, for the creation of health M&E units, and for M&E system strengthening at the National and Regional levels. CARPHA can also further sensitize policy makers and programme managers on the importance of M&E and advocate for resources for M&E Caribbean Med Labs Foundation CMLF s activities under the PANCAP Grant fell under the objective Improved Human and Laboratory Resources in Phase I and Build Capacity and Promote Sustainability in Phase II. CMLF s objectives were to strengthen Regional laboratory network to support care and treatment of HIV/AIDS and to establish National laboratory networks to support HIV/AIDS care and treatment. CMLF worked in the following areas: laboratory baseline surveys, laboratory plans and policies, quality management systems, support to CCAS meetings and laboratory information system strengthening (specifically the E-Logbook). Baseline surveys CMLF conducted self-administered baseline surveys and repeated annual laboratory surveys in 16 countries using a survey questionnaire. The surveys collected information on National laboratory infrastructure, quality and diagnostic infrastructures and capacities and readiness to provide HIV laboratory reference services. This information is maintained in an electronic database; this can be used beyond the Grant to monitor progress. Findings from the baseline assessments and annual surveys were used to identify gaps, inform support strategies and customize support to the countries. Existing laboratory baseline and annual survey data are critical to monitoring laboratory progress in the Region. A mechanism to maintain and continue the conduct of these surveys is needed. 45

46 Plans and Policies CMLF s supported countries with developing National strategic plans, policies and network plans based on initial baseline assessments conducted in Phase 1 of the grant. These assessments revealed a universal lack of National policies for laboratory services. Recognition by CMLF of the need for National laboratory policies to impact positively on the sustainability and quality of lab services, was followed by a process of seeking approval from the meeting of CMOs in Year 1 of Phase 2 for a model Regional Policy Framework approach, requesting allocation of funds from Global Fund to develop the model Regional Framework in Year 2, and seeking endorsement from Ministers of Health at COHSOD in 2014 (Year 2 of Phase 2). The development of the plans and policies are intended to contribute to sustainability since these National policies were designed to ensure the access to sustainable quality laboratory services. This framework was used by CMLF to assist countries with developing their own lab policies. Under this Grant, CMLF has provided support to countries in the development of strategic laboratory plans and policies: 13 countries developed lab policies and 10 countries developed strategic laboratory plans. CMLF also supported the development of National laboratory network plans in 10 countries. These plans and policies will enable national representatives to advocate for resources and the critical role of the labs. Country representatives consistently described the process used by CMLF as participatory and all-inclusive. The plan, policy and network development processes included public and private laboratory staff and key governmental departments within and beyond the health sector. Nonlaboratory stakeholders indicated that the process contributed to an increased understanding of the importance of lab functions and the need for strengthening in-country lab services and networks. Despite the development and availability of draft policies and plans, many of them have not yet been approved or endorsed by National authorities. Consequently, they have not been operationalized and implemented. It should be noted that the issues related to approval and implementation reside at the country level and is outside the immediate control of CMLF. 46

47 National approval of the plans and policies should lead to implementation and contribute to improvement of services. CMLF, with support from PANCAP, should therefore advocate for their review and approval by National authorities. Countries will require support with operationalizing the plans and policies and PANCAP and CMLF should advocate for countries to mobilize resources for laboratory strengthening. CMLF should also finalize the draft model legislation for the regulation of labs and engage in consultative processes with countries to customize and develop National laboratory legislation. Laboratory Information Systems (LIS), specifically E-Logbook The baseline assessments conducted by CMLF included assessments of countries laboratory information systems. Findings pointed to the need for electronic laboratory information systems since many laboratories were using paper-based systems. The CMLF-developed E-Logbook seeks to fill gaps in LIS and management, replace manual systems and contribute to surveillance reporting, monitoring turnaround times and other laboratory performance indicators. The E- Logbook started as a simple tool at low cost, but functionality has been added to include elements of an LIS. The E-Logbook system is a useful alternative for labs that cannot afford full commercial LIS. It should be noted that CDC is providing commercial LIS in some countries. The E-Logbook is being piloted and refined with two laboratories in Suriname. The E-Logbook is currently programmed in MS Access which has some technical limitations. Therefore it needs to be reprogrammed in a different program environment. In addition, essential functionality for directly importing test results from laboratory equipment needs to be added. A clear strategy for the completion, roll-out and use of E-Logbooks is needed, including a clear definition of the final end product, users and technical assistance support systems. Currently, development and programming of the E-Logbook is done by a single person. Additional programming and country support staff will be needed to facilitate roll out and use in countries. Funding will also be required for finalization and roll-out in labs. CCAS Meetings 47

48 Under the Grant, CMLF has provided technical and financial support for the CCAS meeting for the last four years. These meetings were a good forum for providing updates on new lab technology and approaches, as well as training and capacity building. The meeting also provided a forum to collect, present and discuss National and Regional network monitoring data. The CCAS meeting brought together lab experts and vendors, which has facilitated improved purchasing arrangements. The CCAS meetings have led to important development in lab systems in the Region such as the Laboratory Declaration and the Barbados testing arrangement for the OECS. The utility of the meeting will be diminished without continued external financial support. CMLF s ability to provide technical support and training at the meetings will also be diminished when the PANCAP Grant ends. CCAS will need to identify additional partners and donors to support the meetings in the future in order to maintain their quality and utility. CCAS meetings should continue with participation of at least two persons from each country. The MOH should strive to provide funding to support the attendance of at least one person. CCAS should identify both public and private sector partners and donors to ensure continuation of the meetings, specifically for the training component. CMLF can build laboratory capacity in the Region through offering CME courses at the CCAS meetings. CMLF could develop training programme using a modular approach and provide certificates and ongoing capacity development. Quality Management Systems Through the CCAS meetings, CMLF developed Regional consensus on the criteria needed for Regional laboratory reference nodes. CMLF also facilitated the development of National Laboratory Network Plans in 10 countries. These plans contributed to the identifications of laboratories that could serve as Regional reference nodes for provision of reference services for 1) Molecular diagnostics Barbados and Jamaica; 2) Validation of HIV test kits Jamaica, and 3) Proficiency testing and Quality Control models Belize and Suriname. CMLF also provided support to selected reference laboratories (Barbados, Belize and Suriname). CMLF has also provided considerable additional support to the MOH in Barbados with their laboratory 48

49 amalgamation process as they move toward merging the National Public Health lab, the Ladymeade Reference and Lepto Laboratories into one Regional Reference Lab. The support included a Human Resource Plan for the new amalgamated lab and development of a National Laboratory Strategic Plan. CMLF is uniquely positioned to provide ongoing support to the Region s reference laboratories as well as to support in the identification and development of additional reference laboratories. CMLF collaborated with PAHO/WHO, CDC, CROSQ and Regional Accreditation Bodies to develop the Laboratory Quality Management Systems - Stepwise Improvement Process (LQMS- SIP) and supported countries with improving their QMS. Specifically, support was provided to countries with monitoring the turnaround times and with Proficiency Testing programme for selected labs (public and private) in areas of HIV, STI and OI testing in 15 countries. Proficiency Testing (PT) panels were also provided by CDC, but laboratories indicated that there was no overlap of the PT panels provided by CDC and CMLF. Laboratory quality management systems need to be fully integrated into lab policies and procedures and maintained consistently over time. Many laboratories have not yet put systems in place to maintain PT and these should be included as part of National laboratories operating budgets and should not be dependent on external funding. CMLF can advocate for countries for laboratories to have dedicated budgets and separate procurement processes and for the inclusion of proficiency testing as a standard part of operating budgets and procedures. Overall Findings CMLF encountered challenges in improving laboratory services in the Region. These include difficulties with establishing linkages with non-english-speaking countries (i.e., Dominican Republic and Haiti). As a result, interventions in these countries began in Phase 2 of the Grant. CDC and CMLF both supported countries in the Region on similar issues (e.g., Strategic Plans, and Proficiency Testing). CMLF collaborated with CROSQ in development of the stepwise approach to accreditation and both supported laboratories with preparing for accreditation. 49

50 Despite the potential for overlap and duplication of support, countries consistently reported that the support from CDC, CMLF and CROSQ were complementary. This evaluation finds that CMLF provided good support to the countries and the Region with improving laboratory services and systems, including the development of plans and policies and advocating for laboratory strengthening. Ministries of Health and laboratories reported high levels of satisfaction with the services and quality of support they received from CMLF. The organization, although relatively small, has been highly productive and has good technical experts who are knowledgeable of the countries and Region s needs. CMLF also contributed to strengthening National and Regional laboratory networking. CMLF used the CCAS meetings, their website and newsletters to increase dissemination of information and communication among laboratories and stakeholders in the Region. CMLF also provided support to the OECS sub-region through its collaboration with OECS HAPU for the improvement of procurement and the pricing of laboratory supplies. Data was collected on the procurement of laboratory supplies from OECS countries, including comparative costs. Support was provided on negotiations which achieved a reduction of 14-30% in HIV rapid test kit prices. CMLF utilized Grant resources to strengthen labs services broadly and not only in the area of HIV. Although considerable effort and progress has been made over the Grant period, there still remain significant deficiencies in lab services in the Region given that the needs for lab strengthening in the Region far outstrip the capacity of any single organization. Overall Recommendations This evaluation confirms the need to both continue and scale-up laboratory strengthening in the Region. The following recommendations have been made for lab strengthening: CARPHA with support from CMLF and CDC and other technical partners should lead on the development of lab strengthening strategy in the Region. All institutions providing lab strengthening support should be guided by this plan. The University of the West Indies Health Economic Unit in collaboration with laboratory technical partners should conduct a cost benefit analysis for lab strengthening. The findings may be useful for advocacy for laboratory initiatives. 50

51 PANCAP, CARPHA and CMLF should advocate for lab strengthening at the Regional and National levels, and specifically advocate for labs to have their own budgets that are separate from general procurement processes. CMLF can play a leading role on laboratory staff capacity building. CMLF can train a cadre of persons in the Region that can support lab strengthening efforts. CMLF should review their approach to strengthening labs and prioritize selected Regional and larger public health labs. All partners should continue supporting labs to improve QMS Caribbean Vulnerable Coalition / Centro de Orientacion E Investigacion Nacional The work of CVC/COIN contributed to two objectives under both phases of the Grant. These are Reduced HIV transmission in vulnerable populations and An enabling environment which fosters universal access to HIV services. The main areas in which CVC/COIN worked were social mobilization and advocacy, partnering with National AIDS programmes to scale-up programming for vulnerable populations, human rights based peer education, HIV prevention outreach and linkage to services for vulnerable groups with community partners, strengthening of Regional networks of key populations such as CARIFLAGS, Caribbean Sex Worker Coalition and YurWorld Marginalised Youth Network, monitoring & evaluation and research. Social mobilization and advocacy CVC/COIN conducted advocacy and social mobilization at several levels. CVC/COIN sought to facilitate government recognition of the added-value and cost-effectiveness of community based organizations to scale-up their work with vulnerabilized groups, and facilitate government ownership of the HIV response. CVC/COIN s approach to engage governments was done by using both Regional meetings with Ministry of Health staff and ministers as well as individual breakfast meetings with senior staff from ministries. Their specific aims were to disseminate information on HIV epidemiological trends, increase understanding of the need to address human rights violations; increase understanding of the importance of community systems strengthening in Regional and National HIV responses; and disseminate tools and good practices 51

52 for engaging and serving vulnerable groups. CVC/COIN s experienced difficulty with engaging the most influential and senior persons in Ministries and efforts to increase national ownership and engagement with NGOs were not completely successful. There remains a need to sensitize senior Ministry officials and increase government s engagement with NGOs and ownership of HIV responses. Alternative approaches for accomplishing this are needed. CVC/COIN s advocacy also sensitized duty bearers (parliamentarians, NAPs, police, immigration) on health and human rights issues. CVC/COIN developed a Training Manual for police and immigration officers. The aim was to sensitize police officers and immigration officers to the human rights issues facing sex workers and to reduce violence and rights abuses against vulnerable populations. The training provided opportunities for sex workers and police to engage in dialogue, discuss issues and challenges faced by police officers working with sex workers, and to develop strategies to support collaboration between sex workers and police. Use of this manual will extend beyond the PANCAP Round 9 Global Fund Grant. The manual can serve as a reference for trainers and should be integrated into the training curricula of police and the military. The integration of key population sensitivity into the training curriculum is one mechanism to fostering structural changes. Stakeholders reported that these trainings were useful, needed and effective. Furthermore, beneficiary populations interviewed confirm the need for developing mechanisms for protecting their basic rights including the training of duty bearers. Efforts toward incorporating training materials for duty bearers into institutions curricula should be continued and training efforts should be continued and widened. These efforts should also be expanded to other countries. CVC/COIN s advocacy work extended to strengthening networks of vulnerabilized groups and increasing their capacity to advocate for themselves and meaningfully contribute to the Regional HIV response. CVC/COIN facilitated Regional coalition meetings with the Caribbean Sex Worker Coalition (CSWC), which resulted in the Montego Bay Declaration issued by the CSWC at their annual meeting in Montego Bay held from August 28 30, This Declaration sets forth the human rights of sex workers, recognizes the discrimination and human rights abuses they face, and calls for Caribbean States to respect and protect these rights and end discriminatory practices against them. 52

53 CVC/COIN s support also resulted in the development of a Regional strategic plan for the Sex Worker network and as well as a board and structure. Overall, the capacity of the Sex Workers network was strengthened and they are better engaged in the Regional HIV response and PANCAP. CVC/COIN engaged in a number of Regional youth network strengthening activities; a sub- Regional advocacy training was held in Jamaica and a bi-national advocacy training in Haiti for young people from key populations of whom many are living with HIV. These training sessions have not only built capacity in advocacy, specifically on the development of an advocacy campaign on access to medicine and intellectual property, but also contributed to strengthening of the youth network. It has brought together young people living with HIV from the sub-region and has provided a safe space to share their experiences. Regional youth advocacy workshops were held and training and capacity building for youth was provided. The strengthening of youth networks has not been as successful as with the other Regional vulnerabilized group networks. This can be attributed to a number of factors including the lack of support for existing Regional youth networks in the Caribbean over the last 10 years as well as the natural maturation and turnover of youth leaders. CVC/COIN supported and strengthened CariFLAGS, the Region s LGBTI advocacy body. CVC/COIN brought community partners together in Montego Bay to develop a strategic framework for strengthening the National responses to HIV for Gay Men, other MSM and trans people in Jamaica, Dominican Republic, Suriname and Guyana. CariFLAGS is now also better engaged in the Regional HIV Response and PANCAP. National network strengthening and capacity development took place: the Guyana Trans organization, for example, has been considerably strengthened as a result of CVC/COIN s efforts. Other notable advocacy efforts include support to the challenge of the buggery laws in Belize, lobbying Congress in The Dominican Republic against Zona de Tolerancia and networking and mobilization around human rights and access to healthcare. CVC/COIN established a fully functioning Human Rights Observatory in The Dominican Republic. The Human Rights Observatory is a mechanism for reporting on human rights 53

54 violations in all areas, including HIV. The Observatory documents human rights violations and provides support to persons as they seek redress. The Observatory offers referral to a network of legal services. This Observatory is being used as a model in Africa and could be used as a model for other Caribbean countries as well. The Observatory is an excellent example of embedding HIV into broader social structures. Overall, CVC/COIN s advocacy efforts under the Grant have been extensive and significant. Mechanism for funding and advancing CVC/COIN s work in this area are needed. Increasing the Contribution of NGOs to National HIV Responses (Mini-Grants) Under the PANCAP Round 9 Global Fund Grant, CVC/COIN worked to strengthen National and Regional responses through greater engagement of civil society organizations by capitalizing on their strengths, including their ability to reach vulnerable groups in a cost-effective ways. CVC/COIN awarded mini-grants to 36 NGOs in the following six countries: Jamaica, Trinidad and Tobago, Guyana, Suriname, Haiti and The Dominican Republic. The projects developed by the NGOs and funded by CVC/COIN were informed by findings from baseline studies, and the interventions were culturally specific and met the local needs. The projects targeted sub populations within broader groups of MSM, Sex Workers, Drug Users, Prisoners, and Marginalized Youth. Peer outreach was the principal method for conducting outreach. In order to support the provision of high-quality interventions, CVC/COIN developed a sexual reproductive health manual for use with youth, sex workers and MSM. The manual includes 14 modules with handouts, slides presentations and teaching tools. Peer educators from the NGOs were trained using the manual. Peer educators who participated in the training consistently describe the training manual as excellent, indicated that they use the material in their own work, and stated that the training significantly enhanced their skills and capacity. The dissemination of the training manual should be increased through the use of a training of trainers model. The manual should be translated in different languages, and simplified materials should be developed for peer 54

55 educators to use with contacts. The manual and peer educator capacity built under the Grant is likely to contribute to outreach efforts well beyond this project. A range of services were provided to the various sub-populations by the NGOs, including tailored prevention interventions and commodities, access to friendly VCT, access to friendly sexual and reproductive health care services including STI and HIV care and treatment, access to safe spaces, empowerment and social mobilization. Peer Educators are an effective method of reaching high risk groups: outreach to vulnerable groups increased through the mini-grants. Although there was selection bias among representatives from beneficiary populations interviewed during this evaluation, all beneficiary populations across all countries reported high levels of satisfaction with services received by the NGOs. Table 5 highlights the outstanding needs identified by the various vulnerable groups. This evaluation finds that the NGOs and staff themselves also benefitted from the CVC/COIN mini-grant scheme. A CVC/COIN capacity development framework was established which guided all capacity development efforts. This capacity development framework consists of four dimensions: organizational capacity assessment, organization-specific capacity development, monitoring and evaluation of capacity development activities and results, and ongoing capacity development support. Guided by this framework, CVC/COIN created individualized capacity development plans that were based on NGO assessments and provided capacity building support in the areas of M&E, finance, advocacy and programming. Most NGOs reported increases in both institutional and individual capacity development, particularly for smaller NGOs. The capacity development, monitoring and supportive supervision resulted in all but two of the NGOs meeting the challenging M&E and accountability requirements of the Global Fund. Selected NGOs also reported securing additional funds from other donors which they attributed to lessons learned and capacity developed through the CVC/COIN mini-grant experience. An M&E system for tracking organizational level changes in mini-grantees should be designed and implemented. CVC/COIN plans to do a thorough post assessment of capacity to evaluate changes in organizational capacity in the NGOs after the end of the Grant. CVC/COIN successfully developed and administered the mini-grant scheme but not without challenges. The administration, management and capacity development of 34 NGOs in six countries required considerable planning and effort. This effort is often compounded by the small 55

56 number of staff and high turnover of many NGOs. The short term nature of the projects also contributed to uncertainty and staff turnover. Some NGO staff reported inconsistencies with guidance from the national coordinator, M&E and finance staff, as well as frequent changes in M&E and finance reporting guidelines. Future mini-grant schemes will benefit from a review and streamlining of the management structure and oversight structures of CVC/COIN. Full documentation of all procedures and forms should be prepared and included in a manual that can be made available to grantees at the start of project. CVC/COIN should also consider awarding larger and longer grants for stronger, well-established NGOs, to the extent that funding permits. Overall, CVC/COIN s approach to NGO engagement and capacity development has been successful and the model employed should be documented and shared with National AIDS Programmes. M&E System/Capacity Development and Research One of the strengths of CVC/COIN s mini-grant scheme was the M&E system developed under the Grant. CVC/COIN supported 34 NGOs with developing and implementing M&E for the CVC/COIN projects. At the core of the M&E data collection system was the Peer Educator data collection forms developed under the project for use with SW, MY, DU, & MSM. These userfriendly forms were able to be used by persons with low literacy levels and captured demographic data while documenting service interventions. Data was captured through a webbased M&E reporting platform that allows for data digitization, analysis, and reporting. The M&E system is fully functional and 34 NGOs provided data. Through this mini-grant scheme, the M&E capacity of many NGOs was increased and there is now increased data on key populations in the six beneficiary countries. Although it is possible for NGOs to use the database for other projects, many NGOs are not currently taking advantage of the opportunity nor are they accessing, analyzing or using the data they report. The web-based M&E system can be maintained and opened up for use by other NGOs. Also a voluntary reporting system for NGOs can be created and capacity in the use of the system and forms to other NGOs should be built. NGOs should be supported with using data and information in the system. CVC/COIN also added to the body of knowledge on vulnerable populations through this project. CVC/COIN conducted baseline studies and situational analyses on the beneficiary populations of 56

57 sex workers, youth, drug users, MSMs and trans sex workers in Trinidad, Jamaica, Dominican Republic, Guyana, Suriname and Haiti. The research findings were disseminated through the website and at meetings. CVC/COIN is actively working with a university to publish some results in peer reviewed journals. Networking and Partnering Many NGOs reported increased networking and improved linkages with NAPs, CCMs, other NGOs and donors. During the course of joint meetings with staff from NGOs in countries, NGOs were sometimes unaware of the range of services offered by one another and identified areas for collaboration. This suggests that a more systematic and structured approach to facilitating networking among NGOs can increase synergies. This evaluation finds variability in the strength of the linkages between CVC/COIN and the National AIDS Programs across countries. Stronger linkages were observed in the Dominican Republic and Jamaica, where CVC/COIN are based. The relationship between CVC/COIN and the National AIDS Program in Jamaica is well-developed and could serve as a good model for NGO and NAP collaboration. Furthermore, there was insufficient sharing of data collected through the mini-grants with National AIDS Programs by both CVC/COIN and the NGOs. This was the case across all six project countries. Mini-grant M&E data should contribute to the national monitoring and evaluation systems of the National AIDS Programs and there should be collaboration with National AIDS Programs to ensure standardization of indicator definitions and forms. More deliberate engagement and collaboration with National AIDS Programs is also needed to ensure sustainability and to guarantee that investments in NGOs are maintained. Table 5: Beneficiary Population outstanding needs Types of support Low threshold drop in sites/safe space Drug users Sex workers MSM/ Transgender People Marginalized Youth Prisoners 57

58 Support with navigating social service systems (Obtaining identification cards, social service benefits, etc.) Mental health services Legal support Employment or skills-based training Job Placement Assistance Basic or remedial education Food Counseling Services Mobile Health Services Sensitive Health Services STI Services Population-specific health information Training for duty-bearers Transitional Housing Education Development Council EDC conducted work aimed at reducing HIV transmission in vulnerable populations, specifically among the youth. EDC activities were aimed at building capacity in 10 countries through the Caribbean Education Sector and AIDS Coordinator Network EduCan and Ministries of Education to build their capacity to develop HIV/AIDS policies in the education sector and 58

59 provide professional development in life skills-based HIV education and Health and Family Life Education (HFLE). EDC s activities took place under Phase I of the Grant. EDC conducted baseline surveys on barriers to implementing life skills-based HIV education in 10 countries that served as the basis for implementing tailored interventions in countries. EDC conducted five in-country policy development workshops (in Barbados, Dominica, Antigua, Grenada and Suriname). EDC also conducted a train-the-trainers workshop in Barbados to prepare participants to assist with rolling out in-country HIV policy development workshops, an essential component of implementing HFLE and skills-based HIV education. Teachers were trained through two Regional and three in-country professional development workshops. In total 7,365 in-school youth were reached with life skills-based HFLE which exceeds the target of 6,000. Collaboration with Ministries of Education took place with the aim of making the activities sustainable. As of the end of Phase I of the Grant, draft HIV/AIDS education policies were made available in five countries. The formal incorporation and use of HFLE curricula by Ministries of Education in all countries in the Region is needed to sustainably educate youth and the general population on HIV and sexual and reproductive health issues. One of the main challenges to the adoption and implementation of HFLE and HIV life-skills based education in the Caribbean is the fact that each of the countries in the Region has its own HFLE curriculum and guidelines. As a consequence, support and advocacy with countries will need to be individualized. The CARICOM Secretariat has an important advocacy role to play in this regard. A mechanism for providing follow-up support to countries with approval and implementation of policies is necessary. At the national level, attrition of HFLE educators and focal points and associated delays with replacement impeded progress Organisation of Eastern Caribbean States OECS s activities under this Grant were related to the objective to Reduce morbidity and mortality of People Living with HIV in the OECS countries. The main activities included: 1) strengthening the OECS pooled procurement and distribution system at Regional and country 59

60 levels, 2) meeting the procurement needs for anti-retroviral drugs, lab supplies, services and other health products, 3) improving prescription and use of antiretroviral drugs through clinical mentoring, updated guidelines and chart reviews, and 4) providing technical assistance to build capacity for scaling-up early detection, care and support. These activities were conducted during both Grant phases. Pooled Procurement and Distribution Systems The OECS pooled procurement system works well, and is a model approach for pooled procurement. Through negotiations during the pool procurement process, prices for antiretroviral drugs have dropped significantly and are currently around $145 USD per patient/year for first line therapy and $348 USD for second line antiretroviral therapy. The procurement system ensures the availability of antiretroviral drugs, and a stock-sharing system is used that allows countries to exchange drug supplies to further prevent local stock-outs. ARVs are available in over 90% of the cases with patients with prescriptions. In instances where drugs are unavailable the stock-sharing system is employed and drugs are shifted from one country to another as needed. The system results in patients receiving the needed drugs within short timeframes and with little expiration and wastage. The Quantimed system for the forecasting of antiretroviral drug needs works well and the OECS PPS built the capacity of country staff to contribute to national forecasting. The OECS PPS staff worked with National teams comprised of clinical care specialists, pharmacy and central medical stores staff to train them on use of the Quantimed forecasting system. The process has resulted in excellent working relationships between the Pharmaceutical Procurement Service and the OECS countries. The Regional forecasting system has evolved in such a way that countries develop National forecasts which are then reviewed and validated by the OECS PPS. In addition to building National capacity to engage in forecasting, the process has improved collaboration and communication among National stakeholders in the care and treatment systems for PLHIV. The Grant also significantly strengthened and systematized the procurement systems at the OECS PPS as a result of adhering to the Global Fund and WHO s procurements requirements for ARVs. The system of shared payments for antiretroviral drugs, coupled with a step-wise decrease in funding from Global Fund per year, facilitated an increased government 60

61 responsibility for incorporating the cost for antiretroviral drugs in the annual budgets. At the time of this evaluation, not all countries have incorporated antiretroviral therapy payments fully into their budgets. The procurement system also procured lab supplies, and viral load tests were made available from the Ladymeade Reference Lab in Barbados at a price of $50 USD, which is below the actual cost price. The Grant funded viral load testing during the initial part of the period, after which the governments were expected to take ownership. This however did not happen in all countries resulting in interruptions in the provision of viral load testing. This was resolved, at least temporarily, when CDC took the responsibility of funding the viral load tests. Going forward, budgeting and funding allocations should include clinical service provision, antiretroviral treatment, opportunistic infection treatments, and laboratory services as a single package. Inventory management control and lab supply forecasting systems need to be strengthened. CMLF should support the Pharmaceutical Procurement Service with developing pooled procurement for general lab supplies in order to ensure consistent availability of lab supplies. Financing mechanisms for procurement of lab supplies need to be revisited and resources for laboratories should be advocated for with Member States, with support from CMLF. Countries need electronic inventory management systems to improve management and procurement of laboratory supplies. Viral load testing was not continuously available, resulting in compromised quality of care as a result of treating without the ability to monitor at individual level of care. There is an immediate need to implement measures to ensure consistent and reliable availability of basic tests for HIV, including CD4, Viral load and basic haematological tests. In addition, indicators for the utilization of viral load testing and for viral suppression should be used to monitor the quality of care and treatment outcome at population level. At Regional level a laboratory needs to be identified to perform HIV drug resistance testing. Improved prescription and use of antiretroviral drugs 61

62 The prescription and use of ARVs was improved through clinical mentoring, updated guidelines and chart reviews. The HIV & STI Clinical Guidelines for the OECS were revised and updated. Training on the clinical guidelines was provided to clinical care personnel in three countries to improve distribution and usage of the guidelines. Partial institutionalization of the new HIV & STI Clinical Guidelines created opportunity to improve the quality of care. Annual audits of the prescribers charts were done, feedback provided and technical assistance and mentoring offered where needed. The OECS Drug formulary was revised with an expanded range of ARV drugs consistent with the new Treatment and Care guidelines. Within this Project, two parallel systems for collecting data on antiretroviral use and adherence exist. On one hand data are being collected by the OECS Secretariat's HIV AIDS Project Unit (HAPU), on the other hand, the Pharmaceutical Procurement Service has been collecting treatment data in terms of procurement of antiretroviral treatment and evaluating outcome. Care and treatment data collection and reporting have improved under this Project. In addition, sub- Regional treatment guidelines have been developed and applied, which further contributed to the improvement of clinical care monitoring. The care and treatment data collection system, which was functional during the Grant period, needs to be maintained in order to provide accurate and timely feedback. Case-Based Surveillance (CBS) HIV case-based surveillance activities were re-initiated under this Grant. Efforts to develop a HIV Case-Based Surveillance in the Region have been ongoing for several years, and previously organizations including PAHO, CDC, and the Caribbean Epidemiology Center (currently CARPHA) have taken initiatives. Experiences from previous initiatives for HIV Case-Based Surveillance and HIV patient monitoring, including previously collected paper-based and/or electronic data should be reviewed in order to facilitate the design and implementation under the current efforts. The goal of HIV Case-Based Surveillance was defined as to provide quality information for evidence-informed HIV and STI programming and policy decision making, thereby minimizing the impact and reducing the spread of HIV/AIDS and other sexually transmissible infections. Specific objectives include to collect, collate, analyse and disseminate data in order to monitor sexual behaviours and practices driving the HIV/AIDS epidemic, assess 62

63 HIV/AIDS/STI trends over time, measure coverage and quality of care of persons living with HIV/AIDS and other STIs and to provide evidence for public health actions. Due to adjustment of the initial plan and administrative delay, the Case-Based Surveillance activities were only started in Phase II. An operational manual for Case-Based Surveillance of HIV, sexually transmitted infections and tuberculosis has been provided and adjusted to the specific context of each country. Field testing and implementation of the manual is the next step in developing the Case-Based Surveillance system. This will be done using a different scenario for the countries who work with an electronic Health Information System and for those who work paper-based. The collection of data for Case-based surveillance should be done in an efficient way, integrated in the daily routine of care providers. Frequent and timely reports should be made available to closely monitor the implementation process and quality of the collected data. Progress toward reducing morbidity and mortality of People Living with HIV in the OECS Data from the OECS Epi-profile 2014, provided by the PAHO, show a decreasing trend in HIV incidence and HIV mortality for the OECS sub-region (see figure 6). The decreasing trend in HIV incidence and mortality started prior to the commencement of the Grant, and cannot be solely attributed to the Grant. At the time of this evaluation, data were only available up to More recent data are needed to evaluate recent evolutions. Currently, there are about 1,020 persons receiving treatment across the OECS with the latest reported one year retention rate approaching 80%. Some challenges in terms of lost to follow-up and treatment adherence remain (see figure 7). A more detailed examination at country level is needed to explain the yearly fluctuations in reported retention rate. The number of viral load tests done in the Region through the Ladymeade Reference Lab has increased, at least up to 2013 as shown in figure 8, and there is an indication that the viral suppression has increased over time (see table 6), at least up to The suggested increase in viral suppression indicates that the continued availability of and adherence to antiretroviral treatment has resulted in a better clinical outcome. More recent data are needed to evaluate the effect on the number of viral load tests done and viral suppression after

64 The OECS Secretariat's HIV AIDS Project Unit (HAPU) and/or Health Desk should provide technical support to the Member Countries in order to produce health/hiv reports at least annually. A standardized set of indicators should be agreed upon, and support provided collecting data and sharing of information. In order to make data processing more efficient, a shift from the current paper-based systems to an electronic system is needed. CARPHA should build the capacity and strengthen OECS HAPU/Health Desk to better meet the M&E needs of Member States. A train-the trainers approach could be used to better leverage the capacity. Figure 6: Annual number of HIV cases, AIDS cases and deaths from 1984 to 2013 in the six OECS Countries* * Data source: OECS Epi-profile 2014, Pan American Health Organisation Figure 7: Twelve month retention on antiretroviral therapy in the six OECS Countries (GARPR data) 64

65 Figure 8: Viral load measurements of the six OECS countries done through a Regional Laboratory Referral Service* Number of participants Number of viral load measurements Table 6: Logged Viral Load and proportion of patients with suppressed VL among 3,216 samples collected between 2009 and 2013 in the six OECS Countries* 65

66 * Data from: Landis RC, Carmichael-Simmons K, Hambleton IR, Best A. HIV Viral Load Trends in Six Eastern Caribbean Countries Utilizing a Regional Laboratory Referral Service: Implications for Treatment as Prevention.PLoS One Apr 29;10(4):e doi: /journal.pone ecollection PANCAP Coordinating Unit PCU s activities under the Project were related to the objective An Enabling Environment That Fosters Universal Access to HIV Services in Phase I and Phase II and Reduced HIV Transmission in Vulnerable Populations which was an objective only for Phase I. PCU s areas of work included the implementation of the Justice for All programme and coordination of activities carried out by the Sub-Sub-Recipients (SSRs): the Caribbean Network of people Living with HIV (CRN+), the Caribbean Broadcast Media Partnership (CBMP) (Phase I), and the International Labour Organisation (ILO). All three SSRs implemented activities in Phase I and CRN+ and ILO continued their activities in Phase II. a. Key findings Justice for All Programme Country consultations on PANCAP s Justice for All Programme (JFA) and Roadmap were held in seven countries and support was provided to countries to develop their own JFA roadmap and activities. Two final consultations are expected before the end of the Grant in December These consultations have led to the adoption of elements of the JFA programme and incorporation in the National Strategic Plans of selected countries. In Belize, agreement has 66