Document of The World Bank IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-75680) ON A LOAN IN THE TOTAL AMOUNT OF US$50 MILLION TO THE

Transcription

1 Document of The World Bank Report No: ICR IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-75680) ON A LOAN IN THE TOTAL AMOUNT OF US$50 MILLION TO THE REPUBLIC OF BOTSWANA FOR A NATIONAL HIV/AIDS PREVENTION SUPPORT PROJECT April 28, 2016 Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Health, Nutrition and Population Global Practice (GHNDR) Africa Region

2 CURRENCY EQUIVALENTS (Exchange Rate Effective July 31, 2015) Currency Unit = Pula US$1.00 = Pula US$1.00=0.72 SDR ABBREVIATIONS AND ACRONYMS AIDS Acquired Immunodeficiency Syndrome MAP Multi-Country HIV/AIDS Program ART Antiretroviral Therapy MCP Multiple Concurrent Partnership ARV Antiretroviral MDGs Millennium Development Goals BAIS Botswana AIDS Impact Survey M&E Monitoring and Evaluation BCC Behavioral Change Communication MFDP Ministry of Finance and Development Planning BHRIMS Botswana HIV/AIDS Response Information MIST Ministry of Infrastructure, Science &Technology Management System BNAPS Botswana National HIV/AIDS Prevention MJDS Ministry of Justice, Defense, and Security Support BNTP Botswana National TB Program MLG Ministry of Local Government CSO Community Society Organization MMR Maternal Mortality Ratio CD4 Cluster of Differentiation 4 MOESD Ministry of Education and Skills Development CMS Central Medical Store MOH Ministry of Health DAC District AIDS Coordinator MTCT Mother to Child Transmission DMSAC District Multisectoral AIDS Committee MTR Mid-Term Review DHS Demographic and Health Survey MYSC Ministry of Youth, Sport and Culture DP Development Partner NAC National AIDS Council GDP Gross Domestic Product NACA National AIDS Coordinating Agency HIV Human Immunodeficiency Virus NCCPP National Cervical Cancer Prevention Program HDI Human Development Index NDP National Development Plan HMIS Health Management Information System NGO Non-governmental Organization IBRD Int l Bank for Reconstruction & Development NSF National Strategic Framework ICR Implementation Completion Results OP Operational Policy IEC Information, Education, Communication OPCS Operations Policy and Country Services IDA International Development Association PAD Project Appraisal Document IDU Injecting Drug Users PBF Performance-based Financing IEC Information, Education, Communication PDO Project Development Objectives IMF International Monetary Fund PLWHA Persons Living with HIV/AIDS IO Intermediate Outcome Indicator PMTCT Prevention of Mother-to-Child Transmission ISN Interim Strategy Note PRRR Pink Ribbon Red Ribbon ISR Implementation Status Report PSO Private Sector Organization EC European Commission RF Results Framework EDF Economic Development Framework SMC Safe Male Circumcision ERV External Results Verification TA Technical Assistance EU European Union TB Tuberculosis FBO Faith-based Organization UN United Nations FM Financial Management UNAIDS United Nations Program on HIV/AIDS GDP Gross Domestic Product USAID United States Agency for Int l Development GFATM Global Fund to Fight AIDS, TB and Malaria VCT Voluntary Counseling and Testing GOB Government of Botswana WHO World Health Organization HNP Director: Olusoji Adeyi Country Director: Guang Zhe Chen Practice Manager: Magnus Lindelow Project Team Leader: Sheila Dutta ICR Team Leader/Author: Edit V. Velenyi ii

3

4 REPUBLIC OF BOTSWANA NATIONAL HIV/AIDS PREVENTION SUPPORT PROJECT Table of Contents A. Basic Information.. v B. Key Dates v C. Ratings Summary.. v D. Sector and Theme Codes. vi E. Bank Staff vi F. Results Framework Analysis... vi G. Ratings of Project Performance in ISRs xv H. Restructuring. xv I. Disbursement Profile xv 1. Project Context, Development Objectives and Design Context at Appraisal Original Project Development Objectives (PDO) and Key Indicators) Revised PDO and Justifications Main beneficiaries Original Components Revised Components Other Significant Changes 5 2. Key Factors Affecting Implementation and Outcomes. 2.1 Project Preparation, Design and Quality at Entry Project Implementation. 2.3 Monitoring and Evaluation Design, Implementation and Utilization Safeguard and Fiduciary Compliance. 2.5 Post-completion Operation/Next Phase.. 3. Assessment of Outcomes Relevance of Objectives, Design and Implementation Achievement of Project Development Objectives Efficiency Justification of Overall Outcome Rating Overarching Themes, Other Outcomes and Impacts Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Assessment of Risk to Development Outcome Assessment of Bank and Borrower Performance Bank Performance Borrower Performance Lessons Learned Comments on Issues Raised by Borrower/Implementing Agencies/Partners. 36 Annex 1. Project Costs and Financing 37 Annex 2. Outputs by Component 40 Annex 3. Economic and Financial Analysis.. 53 Annex 4. Bank Lending and Implementation Support/Supervision Processes.. 68 Annex 5. Implementing Agency Survey Results iii

5 Annex 6. Stakeholder Workshop Report and Results 69 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR.. 70 Annex 8. Comments of Co-financiers and Other Partners/Stakeholders 70 Annex 9. List of Supporting Documents 71 Annex 10. Illustrative Tables and Figures for the Assessment of BNAPS MAP. 81 iv

6 A. Basic Information Country: Botswana Project Name: Botswana National HIV/AIDS Prevention Support Project Project ID: P L/C/TF Number(s): IBRD ICR Date: 04/28/2016 ICR Type: Core ICR Lending Instrument: SIL Borrower: Government of Botswana, Ministry of Finance and Development Planning Original Total Commitment: USD 50.00M Disbursed Amount: USD 50.00M Revised Amount: USD 50.00M Environmental Category: C Implementing Agencies: National AIDS Coordinating Agency (NACA). Cofinanciers and Other External Partners: B. Key Dates Process Date Process Original Date Revised / Actual Date(s) Concept Review: 11/09/2006 Effectiveness: 06/30/ /30/2009 Appraisal: 04/21/2008 Restructuring(s): 12/29/ /15/ /20/2014 Approval: 07/10/2008 Mid-term Review: 06/13/ /13/2011 Closing: 09/30/ /31/2015 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Risk to Development Outcome: Bank Performance: Borrower Performance: Moderately Satisfactory Substantial Moderately Satisfactory Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Moderately Government: Satisfactory Satisfactory Quality of Supervision: Moderately Implementing Moderately Satisfactory Agency/Agencies: Satisfactory Overall Bank Performance: Moderately Satisfactory Overall Borrower Performance: Moderately Satisfactory

7 C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Performance (if any) Rating Potential Problem Quality at Entry Project at any time No (QEA): (Yes/No): None Problem Project at any Quality of Yes time (Yes/No): Supervision (QSA): None DO rating before Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as% of total Bank financing) Health Other social services 7 7 Public administration- Health Theme Code (as% of total Bank financing) HIV/AIDS Participation and Civic Engagement Tuberculosis 4 NCD 2 E. Bank Staff Positions At ICR At Approval Vice President: Makhtar Diop Obiageli Ezekwezili Country Director: Guang Zhe Chen Dirk Reinermann (Acting) Practice Manager/Manager: Magnus Lindelow Christopher J. Thomas Project Team Leader: Sheila Dutta Sheila Dutta ICR Team Leader: Edit V. Velenyi ICR Primary Author: Edit V. Velenyi F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The project development objective is to assist the Government of Botswana to increase the coverage, efficiency, and sustainability of targeted and evidence-based HIV/AIDS interventions through (i) strengthening the National AIDS Coordinating Agency's institutional management and coordination capacity; and (ii) financing strategic and innovative HIV/AIDS-related prevention and mitigation activities. This project development objective will be measured by vi

8 utilizing indicators addressing the longer-term planning, implementation, and effectiveness of national and decentralized coordination efforts. Revised Project Development Objectives (as approved by original approving authority) To assist the Government of Botswana to increase access to prioritized prevention services that reduce the risk of HIV transmission. (Revised on the basis of agreements during the Mid Term Review (MTR) in May 2011 and approved by the World Bank Board on December 29, 2011). Indicators At project approval, 6 Project Outcome Indicators (PDOs) and 13 Intermediate Outcome Indicators (IOs) comprised the project s Results Framework (RF). PDO Indicator #4 was mistakenly reported on as an IO indicator during implementation. This was eventually corrected in March During the ICR process, it was necessary to review and include additional indicators in order to more fully reflect project achievements and measure the impact project activities had on the PDO. As a result, three additional PDO indicators have been included in the datasheet (PDO 7-9): (i) improving TB case detection; and TB treatment success rate; (ii) percentage of population (10-64 years) who have ever received an HIV test; and (iii) percentage of adults (15-49 years) who received an HIV test in the past 12 months and who know their results. (a) PDO Indicator(s) PROJECT DEVELOPMENT OBJECTIVE INDICATORS (PDO) Percentage of youths aged 10 to 14*, 15 to 19 and 20 to 24 who (a) correctly identify three ways of preventing the sexual transmission of HIV and (b) who reject three major misconceptions about HIV transmission. Indicator 1: Value quantitative or Qualitative) This indicator measures the coverage dimension of the original PDO (Phase 1) and access to prioritized prevention to reduce the risk of HIV transmission in the revised PDO (Phase 2). Causal Pathway: improved coverage / access through improved knowledge of HIV prevention and transmission (i.e., increased demand for prevention through IEC to reduce the risk of HIV transmission). Baseline Value years: 4.0* years: years: 40.0 Original Target Values (from Approval Documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years n/a Date achieved 12/31/ /31/2008 3/31/2015 Status: Not achieved. For age group 15-19, the current value represents 34% progress towards the end target value. For age group 20-24, the current value represents 39% progress towards the end target value. Comments (incl. % *Note: The youngest age cohort (10-14 years) was dropped in February, 2009 since this group was not included in the national HIV/AIDS surveys (BAIS III) subsequent to project approval. As a result, no data was collected on this indicator. Hence, the baseline achievement) statistics reported for the age cohort is from BAIS II (PAD Annex 3), while for the other two age groups the data are from BAIS III. Definition to measure youth knowledge on HIV prevention: percentage of young people who both correctly identify ways of preventing the sexual transmission of HIV and who vii

9 reject major misconceptions about HIV transmission. The indicator is measured as a composite of answering "Yes" to all three questions: Can the risk of HIV transmission be reduced by having sex with only one uninfected partner who has no other partners? Can a person reduce the risk of getting HIV by using a condom every time they have sex? Can having no sex at all can prevent HIV infection? and three misconception rejections: Can a healthy-looking person have HIV? Can a person get HIV from mosquito bites? Can a person get HIV by sharing food with someone who is infected? Source: Baseline data from BAIS III (2008) and current data from BAIS IV (2013) Proportion of youth aged 10 to 14*, 15 to 19 and 20 to 24 years reporting either (a) no sexual activity or (b) condom use during the last sexual encounter with a nonregular partner in the past 12 months. Indicator 2: This indicator measures the coverage dimensions of the original PDO (Phase 1) and access to prioritized prevention to reduce the risk of HIV transmission in the revised PDO (Phase 2). Behavioral prevention is also related to the efficiency and sustainability dimensions of the original PDO. Causal Pathway: Improved sexual behavior (abstinence and condom use) -- induced by IEC -- would reduce the risk of HIV transmission. Further, the improved behavioral prevention would reduce the need for treatment, thereby improving disease-specific allocative efficiency and the financial sustainability of responding to HIV. Baseline Value years* No sexual activity: 99.7 Condom use: 68.9 Original Target Values (from Approval Documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years n/a n/a Value quantitative or Qualitative) years No sexual activity: 81.9 Condom use: years No sexual activity:12.5 Condom use: Date achieved 12/31/ /31/2008 3/31/2015 Status: Partially achieved. Targets not achieved for the years age cohort. Targets achieved for the two sub-indicators with respect to the year age cohort. Comments (incl. % achievement) Indicator 3: *Note: The youngest age cohort (10-14 years) was dropped in February, 2009 since this group was not included in the national HIV/AIDS surveys (BAIS III) subsequent to project approval. As a result, no data was collected on this indicator. Hence, the baseline statistics reported for the age cohort is from BAIS II (see also in PAD Annex 3), while for the other two age groups the data are from BAIS III. The baseline and target values for the indicators on abstinence and condom use for the age cohort suggest that the objective was to maintain the results given the complexity of influencing sexual behavioral dynamics. The indicator on condom use is jointly determined by the supply and demand side aspects (e.g. supply chain and availability of condoms and demand for condoms due to IEC campaigns). Percentage of sexually active males/females (M/F) who report having had sex with more than one partner in the past 12 months. viii

10 Value quantitative or Qualitative) This indicator measures the coverage dimensions of the original PDO (Phase 1) and access to prioritized prevention to reduce the risk of HIV transmission in the revised PDO (Phase 2). Behavioral prevention is also related to the efficiency and sustainability dimensions of the original PDO. Causal Pathway: Improved sexual behavior -- induced by IEC -- would reduce the risk of HIV transmission. The HIV cases averted, as a result of reduced MCP, would reduce need for treatment, thereby improving allocative efficiency and financial sustainability. Baseline Value years: 27.7* years: years: 17.7 Original Target Values (from Approval Documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years n/a Date achieved 12/31/ /31/2008 3/31/2015 Status: Not achieved. For the age cohort 15-19, the current value is 5.6% above the baseline, which represents a 32.7% increase during the project period. The observed endline value is 12.7% higher than the target. For the age cohort 20-24, the current value is 2.4% above the baseline, which represents a 13.5% increase during the project period. Comments The observed endline value is 12.1% higher than the target. These trends indicate an (incl. % increasing practice of multiple concurrent sexual partnerships (MCP), implying an achievement) increasing gap between actual behavior and policy ambition. *Note: For reasons noted above, the years age cohort was dropped in February 2009, which focused the indicator on the most programmatically strategic age cohorts (15-19 years, years). The number of male circumcision procedures performed in selected health facilities. Indicator 4: This indicator measures the coverage dimensions of the original PDO (Phase 1) and access to prioritized prevention to reduce the risk of HIV transmission in the revised PDO (Phase 2). As it measures the voluntary uptake of a biomedical intervention, this outcome is jointly determined by the supply- and demand-side inputs. Baseline Value Original Target Values (from Approval Documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Value quantitative or 0 385, ,136 Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Status: Not achieved. The current value represents 39% progress towards the target. Comments (incl. % achievement) Notes: With greater emphasis on interventions that generate demand for safe male circumcision (SMC), the annual number of procedures rose from 1,764 per year in 2012, through 6,824 per year in 2013, to over 18,000 in There were 32,527 SMCs done by all sites between April 2014 and February 2015 against the 50,000 annual target, (i.e. 65% achievement for 2014). Between 2012 and the project s closure in March 2015, over 150,000 procedures were done, which represents a 39% progress towards the target. The initial slow uptake was caused by inadequate supply-demand coordination. When this was resolved, service delivery intensified, showing the potential of SMC. This indicator was first reported on in April 2012, and was originally ix

11 Indicator 5: reported as an IO. In the final ISR (3/31/2015), the indicator was listed among the PDO indicators, its original categorization per the Project Appraisal Document (PAD). Source: Ministry of Health Safe Male Circumcision Report. Health facilities constructed, renovated, and or equipped to be highly active antiretroviral therapy (HAART) compliant. Core sector indicator. This indicator measures the coverage dimensions of the original PDO (Phase 1) and access to prioritized prevention to reduce the risk of HIV transmission in the revised PDO (Phase 2). Baseline Value Original Target Values (from Approval Documents) Formally Revised Target Values Value quantitative or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Indicator 6: Actual Value Achieved at Completion or Target Years Status: Surpassed. This indicator measures the number of primary health clinics constructed/upgraded/renovated. People receiving tuberculosis treatment in accordance with the WHO-recommended Directly Observed Treatment Strategy (DOTS). This indicator measures coverage (original PDO) and access (revised PDO) of TB treatment. While this is not a direct measure of HIV prevention, given the high rate of co-infection and co-morbidity and the integrated TB and HIV/AIDS service delivery (counseling, testing, and treatment), this intervention indirectly contributes to HIV transmission risk and HIV/TB co-morbidity reduction. This is listed as a core sector indicator, although only for IDA countries. Baseline Value Original Target Values (from Approval Documents) Formally Revised Target Values Value quantitative or 9,645 7,000 7,088 Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Actual Value Achieved at Completion or Target Years Status: Achieved. Current value (7,088) represents 96.7% progress toward the end target (7,000) value. Note: The strong referral rates, high treatment initiation rates, and high treatment success rates facilitated progress towards meeting the end target value. Improving (i) TB case detection; and (ii) TB treatment success rate. Indicator 7: This indicator measures coverage and access to preventing TB and HIV/AIDS comorbidity through integrated care. In 2012, 63% of TB program patients had HIV/AIDS coinfection. The indicator is related to the refocused activities during Phase 2 of BNAPS, following the MTR, which placed more emphasis on the TB and HIV/AIDS integration agenda and within that, on prevention of cases and co-morbidities through treatment (timely treatment and compliance with the prescribed treatment regime). x

12 Value quantitative or Qualitative) Date achieved Comments (incl. % achievement) Baseline Value Target Values (Based on Regional and Global Benchmarks) Formally Revised Target Values Actual Value Achieved at Completion or Target Years TB Case Detection: 69% TB Treatment Success: 71% 74.4% 80% 75% 82% 2010 (NTBR-MOH, WHO) 09/11/ (NTBR-MOH) 3/31/2015 Status: Surpassed. Indicator added at ICR to enable measuring the effect of the TB- HIV integration agenda, which gained increased emphasis from 2012, following the MTR and the consequent restructuring. Note: This biomedical indicator is directly related to BNAPS investments in the TB Program and TB-HIV integration agenda. The causal pathways for improvement in case detection is through quality assured bacteriology by training more laboratory staff, and for improving treatment success rate through standardized treatment (e.g., supervised treatment and patient support). Thus, the pathway is through improved provider capacity/skill as a result of staff training and upgrade of laboratory technology. These inputs are listed among the core sector indicators. Indirectly, this biomedical indicator is related to efficiency and sustainability. While these dimensions were only explicit in the original PDO (Phase 1), given that the integration agenda has a strong focus on efficiency and sustainability, such effects should also be assessed for Phase 2. Indicator 8: The target values are based on the National TB Program Report of the MOH (NTBR- MOH 2015) and global/regional benchmarks (WHO 2014 database). Percentage of population (10-64 years) who have ever received an HIV test. This indicator measures coverage (original PDO) and access (revised PDO) to prevention. The causal pathway is through knowledge as a result of voluntary counseling and testing (i.e., knowledge of HIV status and consequent behavioral change as a result of IEC). Baseline Value Original Target Values (from Approval Documents) Formally Revised Target Values Value quantitative or 56% 70% 70.2% Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Status: Achieved. Indicator reintroduced at ICR. Actual Value Achieved at Completion or Target Years Comments (incl. % achievement) Indicator 9: Note: This indicator was originally reported on in June 2014, following increased BNAPS support for MOH programs that were losing support from other development partners. By the March 2015, the indicator was no longer reported on in the ISR. It was reintroduced during the ICR to link the PDO to relevant indicators that can quantify progress related to prevention through activities funded by BNAPS. Sources: BAIS III and BAIS IV Percentage of adults (15-49 years) who received an HIV test in the past 12 months and who know their results. xi

13 Baseline Value Original Target Values (from Approval Documents) Formally Revised Target Values Value quantitative or 41.2% 60% 63.7% Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Status: Surpassed. Indicator reintroduced at ICR. Actual Value Achieved at Completion or Target Years Comments (incl. % achievement) Note: This indicator was introduced in June 2014, following increased BNAPS support for MOH programs that were losing support from other development partners. The indicator measures access to HIV/AIDS prevention. By March 2015, the indicator was no longer reported on in the ISR. It was reintroduced during the ICR to link the PDO to relevant indicators that can quantify progress related to prevention through activities funded by BNAPS. Sources: BAIS III and BAIS IV (b) Intermediate Outcome Indicator(s) Indicator Baseline Value Original Target Values (from approval documents) Formally Revised Target Values Actual Value Achieved at Completion or Target Years Indicator 1 : Annual audit reports for NACA demonstrating transparent and accountable financial management. Value (quantitative 0% 100% 100% or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Indicator 2 : Status: Achieved. Note: This indicator is related to the original PDO, specifically, to measuring the effectiveness of the Project in strengthening the National AIDS Coordinating Agency s (NACA) institutional management and coordination capacity (Component 1). Percentage of proposals originating from the Call for Proposals (CFP) processed within the standard timeframe. Value (quantitative 0% 100% 100% or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Indicator 3 : Status: Achieved. Note: This indicator is related to the original PDO, specifically, to measuring the effectiveness of the Project in strengthening NACA s institutional management and coordination capacity (Component 1). The equivalent indicator for the IBRD buy-down also includes the ratio of rejected and approved and funded proposals as part of performance assessment under this indicator. The proportion of District Multi-Sectoral AIDS Committees (DMSACs) that submit timely quarterly reports on project-supported community grants to NACA. Value (quantitative or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments Status: Surpassed. xii

14 (incl. % achievement) Indicator 4 : Note: This indicator is related to the original PDO, specifically, to measuring the effectiveness of the Project in strengthening NACA s institutional management and coordination capacity (Component 1). Independent Results Verification of civil society component grantees conducted in a representative sample of grants (Yes/No). Value (quantitative No Yes Yes or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Indicator 5 : Status: Achieved. Note: This indicator is related to the original PDO, specifically, to measuring the effectiveness of the Project in strengthening NACA s institutional management and coordination capacity (Component 1). Disbursements from NACA to beneficiaries, public sector as well as Call for Proposals (CFP) grantees. Value (quantitative 0% 100% 100% or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Status: Achieved. Note: This indicator is related to the original PDO, specifically, to measuring the effectiveness of the Project in strengthening NACA s institutional management and coordination capacity (Component 1). % of priority sector ministries that have HIV/AIDS sector policies and programs. Indicator 6 : Value (quantitative 50% 100% 100% or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments Status: Achieved. (incl. % Note: This indicator is related to both coordination and increasing the coverage of HIV achievement) prevention with a focus on assessing line ministries contribution (Component 2). Indicator 7 : Regional TB/HIV coordinators active in two regions. Value (quantitative or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Status: Achieved. Comments Note: This indicator is related to financing strategic and innovative HIV/AIDS related (incl. % prevention delivered under Component 2, specifically through the activities led by the achievement) MoH. Indicator 8 : Three hospitals equipped with isolation wards and functional as TB referral centers. Value (quantitative or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Indicator 9 : Status: Achieved. Note: This indicator is related to financing strategic and innovative HIV/AIDS related prevention delivered under Component 2, specifically through the activities led by the MoH. Number of organizations that have been funded through the Call for Proposals (CFP) mechanism (under Component 3). xiii

15 Value (quantitative or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Indicator 10 : Value (quantitative or Qualitative) Status: Achieved. Note: This indicator is related to increasing the coverage, efficiency and sustainability of targeted and evidence-based HIV/AIDS interventions. Specifically, it measures progress as a result of applying a performance-based platform to contract Civil Society / Private Sector Organization (CSOs/PSOs) under Component 3. NACA developed a mechanism to award multi-year contract to organizations demonstrating a strong track record, with respect to implementing timely reporting and achieving results. This approach critically enabled greater programmatic continuity and reduced transaction costs, but reduced the total number of cumulative grants, than would have been the case if the original annual grant cycle mechanism remained unchanged. % of organizations that have reported (a) baseline data, (b) end-of-project data and (c) both baseline and end-of-project data (under Component 3/Call for Proposals) Baseline data: 0% End-of-project data: 0% Data for both: 0% 95% 85% 75% 100% 100% 100% Date achieved 12/31/ /31/2008 3/31/2015 Status: Surpassed. Comments (incl. % achievement) Note: This indicator is related to strengthening NACA s (grant) management capacity and increasing the coverage and efficiency of targeted and evidence-based HIV/AIDS interventions. Specifically, it measures progress in results monitoring by CSOs/PSOs, which is necessary for performance-based management under Component 3. Indicator 11 : % of organizations funded that have made progress in reaching targets specified in their proposal (Component 3). Value (quantitative 0% 70% 100% or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Status: Surpassed. Comments (incl. % achievement) Indicator 12 : This indicator is related to increasing the coverage of targeted and evidence-based HIV/AIDS interventions. Specifically, it measures progress toward the coverage targets agreed on between the NACA (the fund holder) and the CSOs/PSOs (grantees) to receive the performance-based grants under Component 3. % of organizations funded that have reached targets specified in their proposal (Component 3). Value (quantitative 0% 65% 82% or Qualitative) Date achieved 12/31/ /31/2008 3/31/2015 Comments (incl. % achievement) Status: Surpassed. This indicator is similar to IO 11 but requires full attainment of the specified target as opposed to merely recording progress. xiv

16 G. Ratings of Project Performance in ISRs No. Date ISR DO IP Actual Disbursements Archived (USD millions) 1 2/13/2009 Satisfactory Satisfactory 0 2 6/29/2009 Satisfactory Moderately Satisfactory /23/2009 Moderately Satisfactory Moderately Unsatisfactory 0 4 6/28/2010 Satisfactory Moderately Satisfactory /3/2011 Satisfactory Satisfactory /20/2011 Moderately Satisfactory Moderately Satisfactory /23/2012 Moderately Unsatisfactory Moderately Unsatisfactory /18/2012 Moderately Satisfactory Moderately Satisfactory /24/2013 Satisfactory Satisfactory /15/2013 Satisfactory Satisfactory /14/2014 Satisfactory Satisfactory /12/2014 Satisfactory Satisfactory /31/2015 Moderately Satisfactory Satisfactory 50 H. Restructuring Restructuring Date(s) Board Approved PDO Change ISR Ratings at Restructuring DO IP Amount Disbursed at Restructuring in USD millions* Reason for Restructuring & Key Changes Made 29-Dec-2011 MS MS (26%) Level 1 Restructuring: revise PDO. 15-Jul-2013 S S (64%) Level 2 Restructuring: 12-month closing date extension. 20-Aug-2014 S S (94%) Level 2 restructuring: 6-month extension of closing date and fund reallocation. Note: The disbursed amounts are based on the values reported in the Restructuring Papers. I. Disbursement Profile xv

17

18 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Country Profile, Economic Growth and Governance. During project preparation, Botswana had a population of 1.8 million, about half residing in rural areas. The administrative structure is composed of 9 districts, 1 5 urban districts, and 28 subdistricts. Botswana's postindependence history has been characterized by good governance, democracy, strong macroeconomic policies, an open economy, relatively strong institutions and high revenues from diamond-mining industries. With a gross national income per capita of US$5,750 in 2008, Botswana was classified as an upper middle-income country. The government managed the country's resources prudently and kept its recurrent expenditure within its revenue, allowing for investment in human and physical capital. In 2007, Botswana ranked as Africa's least corrupt country, placing above many European and Asian countries, on Transparency International s ranking. 2. Discordance between Economic and Human Development. The Government of Botswana (GoB) had made a concerted effort to achieve sustainable economic growth. Economic growth was robust, with an average of 5.7% real growth rate between 2002 and Although this growth earned favorable global recognition, it was accompanied by unintended consequences. Growth was largely driven by the mining sector, and it was inequitable, associated with persisting poverty and unemployment. In 2012, Botswana had one of the highest Gini coefficients (60.5) in Africa and globally. The HIV epidemic had a substantial negative impact on fundamental indicators as shown by its declining Human Development Index (HDI) ranking from 71 in 1996 to 124 in Lower life expectancy affected demography and the economy, for example, through higher dependency ratios and an increased share of orphans, representing 20% of all children in These socio-economic constraints were identified as the major factors suppressing growth and became the focus areas of the GoB s economic and development policies. 3. Health Financing, System, and Outcomes. Botswana had dedicated a significant share of its resources to health. At appraisal in 2008, total expenditure on health was 5.3% of GDP. Between 2002 and 2012, allocations from general government expenditures to the health sector were an average 8.9%. In 2008, the per capita health expenditure (at average exchange rate) was US$307, of which approximately 69% were government expenditures. By 2013 per capita health expenditure was US$397, of which 57% was publicly funded (WHO, 2014). At appraisal, Botswana had 24 health districts. At the time of the ICR the MOH listed 27 health districts, and the service delivery infrastructure included 3 referral hospitals, 12 district hospitals, 17 primary hospitals, 222 clinics, 220 health posts, and 740 mobile stops (MOH, 2015). The HIV/AIDS epidemic had a severe impact on the health system, in skewing resource allocation and disease control efforts away from other diseases. For example, compared to the national AIDS treatment program and despite the high prevalence of HIV/TB incidence, there was not an equally strong response to TB. Case detection was 80% in 2006, down from 88% in Treatment success fell from 77% (2000) to 70% (2005). The PAD noted that the implementation of the joint TB- HIV/AIDS strategy needed to be strengthened, particularly at community level where these programs operated in parallel. In sum, at appraisal, key health and human development indicators were not commensurate with the investments in the sector. The low HDI ranking, the increasing infant mortality rate (from 45/1,000 births in 1990 to 85 in 2005), and TB incidence (from 236/100,000 in 1990 to 670 in 2005, and accounting for 38% of AIDS deaths) indicated that for its income and health expenditure levels, Botswana could buy more health for the money by 1 Central, Ghanzi, Kgalagadi, Kgatleng, Kweneng, North East, North West, South East and Southern. 1

19 improving sector efficiency. Given the share of HIV/AIDS in total health expenditures and the rising concerns regarding expenditure inefficiencies, the innovations envisaged to contribute to system level value-for-money objectives included improved prevention efforts, integrated care delivery, and engaging communities and the private sector in healthcare delivery. 4. HIV/AIDS Epidemiology. The first case of HIV infection in Botswana was diagnosed in The number of new infections rose rapidly during the early 1990s, peaking in the mid- 1990s (Figure A10-1 Panel A, Annex 10). Botswana faced the second most severe HIV epidemic in the world (after Swaziland). In 2008, 283,000 adults (over 15 years of age) were living with HIV/AIDS in Botswana (prevalence: 23.8%; NACA, 2008). Rising mortality rates paralleled a maturing epidemic, reaching a peak in AIDS-attributed mortality increased from 4 to 27% of all reported deaths between 1992 and 2003 (Figure A10-1 Panel A, Annex 10). The principal mode of transmission in Botswana was and remains heterosexual. Factors fueling the epidemic include: (i) multiple concurrent sexual partnerships (MCP); (ii) unprotected and intergenerational sex; (iii) vulnerability of women; (iv) persistent inequality and poverty; and (v) high levels of population mobility, including cross-border challenges. These dimensions indicated complex behavioral and socio-economic dynamics, which required a multisectoral response. Geographically, the epidemic had been diverse, with the highest infection rates reported from the northeastern 2 areas of the country. The 2004 Botswana AIDS Impact Survey (BAIS II) estimated the highest prevalence among women years of age (44%). The 2006 Botswana HIV/AIDS Sentinel Surveillance Technical Report corroborated these findings, and showed that almost half of women aged years were HIV positive (Table A10-1, Annex 10). 5. Fiscal Space for HIV/AIDS. In 2001, Botswana was the first African country to provide no-cost antiretroviral therapy (ART) to its citizens. The benefits of the rollout were twofold: preventing 52,000 HIV/AIDS-related deaths by 2007; and slowing the rate of decline in GDP growth from to 0.8%. However, concerns were rising regarding the expenditure implication of HIV/AIDS. The combination of (i) high HIV prevalence rate; (ii) increasing ART coverage; and (iii) increasing life expectancy as a result of treatment success rate (92%) were exerting evergrowing pressure on the sector s budget and, consequently, on general government expenditures. The disease-specific government allocation increased from US$69.8 million in to an average of US$163 million in (Table A10-2, Annex 10). Estimates at the time of the appraisal put the GoB s share of HIV/AIDS-related total expenditures at 57% and the external resource share at 43%, reaching half of HIV/AIDS spending by 2008 (Table A10-2, Annex 10). 3 Yet, even the combined donor 4 and GoB resources could not keep pace with the escalating cost of the response to the epidemic. The fiscal space implication of the epidemic became a concern of the Ministry of Finance and Development Planning (MFDP); specifically, whether the national HIV/AIDS-related investments could increasingly displace other budget priorities. This concern was aggravated by donor flight, with more domestic resources needed to maintain treatment coverage. This meant that sustained commitments to fighting HIV could consume a progressively larger share of the economy. Botswana s HIV/AIDS-related expenditures grew from 2.6% of GDP during appraisal (Table A10-3, Annex 10) to 3.5% in 2010 (Lule and Haacker, 2012). It became clear that the new direction for the national response must incorporate more focus on efficiency and sustainability. 6. Government strategy to fight HIV/AIDS. The increase in the AIDS-attributed mortality rate triggered the emergency expansion of the national HIV/AIDS treatment program, and an increase in the proportion of patients initiating prophylaxis/treatment (from 34.3% in 2001/ BAIS II (2004) reported the highest prevalence in the northeast: Chobe 29.4%, followed by Francistown 24.6%. 3 Currently the GoB finances over 90% of the national response. 4 Global Fund (GFATM), the USG (PEPFAR), other bilateral DPs, the Gates Foundation, and Merck. 2

20 to 89% in 2006/2007). However, the treatment-oriented program resulted in lesser focus on prevention priorities and outcomes. On the demand side, this pattern could be explained by a combination of knowledge gap, behavioral dynamics, and high levels of stigma associated with HIV. Regarding the knowledge gap, although 93% of respondents had heard of HIV/AIDS, the proportion who correctly identified HIV prevention and transmission increased from 36% (2001) to 38% (BAIS II, 2004) against the target of 90% by As to the behavioral aspects, 76% of the cohort had non-regular partner sex in the last 12 months (BAIS 2004). The share reporting unprotected sex increased from 5% (2002) to 14.7% (2007). The share of young people who had multiple concurrent partnership (MCP) in the last 12 months greatly increased from 0.3% (2001) to 17.1% (2005). In 2003, 24% of sexually active men (15-24) reported sex with someone outside their primary relationship. Such behavior had a 72% acceptance rate. Therefore, at appraisal, the most significant challenge for the national response was to strengthen efforts to reduce the annual number of new infections, projected at over 14,000 in Evidence from BAIS II (2004) suggested that to complement the successful rollout of the treatment program (2001), an effective HIV/AIDS response must strengthen prevention, with an emphasis on knowledge and behavior. The findings called for gradually optimizing the balance between the biomedical focus and prevention within the national response. 5 Such a balanced approach required a new delivery platform; closer linkage with communities to influence behavior and deliver services at the lowest levels of the health system. Prior to BNAPS, there was limited progress at community level due to weak community health systems. A required paradigm shift was the introduction of community-centered approaches in HIV/AIDS prevention to ensure broader and more sustainable coverage. The linkages between the national response to the epidemic and higher level development objectives were reinforcing. The National Strategic Framework for HIV/AIDS (NSF I, ) -- which focused on: (i) articulating, disseminating, and educating the public on agreed priorities and strategies within the scope of Vision 2016; (ii) providing clear guidance and a collaborative framework for ministries, districts, CSOs, and the private sector to eliminate the incidence of HIV and reduce the impact of AIDS in Botswana; and (iii) strengthening coordination across NACA, the MOH, and at decentralized levels. -- was linked to the National Development Plan (NDP-9, ). The National Strategy for Poverty Reduction noted HIV/AIDS as both a cause and consequence of poverty and unemployment. 7. Rationale for Bank Assistance. The GoB requested the BNAPS Project a proposed concessional IBRD operation (see more on the innovative instrument, the IBRD buy-down, in paragraph 21 under Section 2.1) in anticipation that the financial and technical resources mobilized through the Bank would play a strategic role in supporting an evidence-based response to the national prevention program. BNAPS was aligned with Botswana s overarching development agenda (NDP-9 and the Poverty Strategy) and the national response to the epidemic, which was defined in the NSF I ( ) and related operational plans, such as the National Operational Plan for Scaling-up HIV Prevention (2008), which focused on prioritizing interventions with the greatest potential impact for preventing new infections. The Project was consistent with relevant sectoral and regional World Bank strategies and policies, 6 and the Interim Strategy Note for Botswana (2008), the first Bank strategy for Botswana, which included HIV/AIDS among its three strategic priorities. Regarding the bigger picture, BNAPS was envisioned to (i) catalyze the transition from an emergency response to a broader, strategic, and sustainable approach; (ii) facilitate complementary response from development partners (DPs); and (iii) create opportunities to advance regional goals. 5 E.g., by using risk mitigation (condom use) and risk elimination (abstinence and delayed sexual debut) measures. 6 Global HNP Strategy (2007); Africa Regional Health Policy (2006); Africa HIV/AIDS Strategy (2008). 3

21 1.2 Original Project Development Objectives (PDO) and Key Indicators 8. The PDO aimed to assist the GoB to increase the coverage, efficiency, and sustainability of targeted and evidence-based HIV/AIDS interventions through (i) strengthening the National AIDS Coordinating Agency's institutional management and coordination capacity; and (ii) financing strategic and innovative HIV/AIDS-related prevention and mitigation activities. The 6 original PDO indicators were: (1) percentage of youths aged 10-14, and who (a) correctly identify three ways of preventing the sexual transmission of HIV and (b) who reject three major misconceptions about HIV transmission; (2) Proportion of youth aged 10-14, and years reporting either (a) no sexual activity or (b) condom use during the last sexual encounter with a non-regular partner in the past 12 months; (3) percentage of sexually active males/females (M/F) who report having had sex with more than one partner in the past 12 months; (4) The number of male circumcision procedures performed in selected health facilities; (5) Health facilities constructed, renovated, and or equipped to be highly active anti-retroviral therapy (HAART) compliant (core sector indicator); and (6) People receiving tuberculosis treatment in accordance with the WHO-recommended Directly Observed Treatment Strategy (DOTS). 1.3 Revised PDO and Justification 9. The Restructuring Paper (November 14, 2011) assessed that, based on a more comprehensive understanding of Botswana s HIV/AIDS epidemic and implementation experience obtained from the May 2011 MTR, efficiency and sustainability would not be easily measurable or attainable in a short time span, even though these goals remained highly relevant. Consequently, the December 2011 level 1 restructuring simplified the PDO, focusing on prevention. The revised PDO is: To assist the Government of Botswana to increase access to prioritized prevention services that reduce the risk of HIV transmission. This change reflects decreased emphasis on the Project s institutional aspect, the strengthening of NACA s coordination and management capacity. The PDO revision was not accompanied by changes in the indicators proposed in the original Results Framework. 1.4 Main Beneficiaries 10. Geographic and Population Coverage. At appraisal, in 2006, approximately eighty percent of the estimated 1.8 million population of Botswana was estimated to be affected by BNAPS. To attain the PDO and to target resources, BNAPS focused on the 15-19, and age groups and vulnerable groups, as defined in the Operations Manual, including children/youth and women. The prioritized and phased implementation indicates that the number of beneficiaries would increase over time. Phase I (years 1-2) was to cover five health districts on the eastern border, including South East, Kweneng East, Francistown, Selebi-Phikwe districts, and Goodhope subdistrict. Phase II (years 3-5) was to attain the nationwide rollout of BNAPS, following satisfying the fiduciary requirements. 1.5 Original Components 11. The design of project components was based on analysis of identified challenges in the national response to the HIV/AIDS epidemic and reflected NACA s key program coordination and implementation areas, as follows: 12. Component 1: Support to NACA (US$7.2 million). This component focused on building NACA s internal capacity to coordinate HIV/AIDS activities effectively. Design features included: (i) coordination of program components by NACA that are decentralized and implemented by the Ministry of Local Government (MLG), the district administration, or NGOs; (ii) NACA, as the financial intermediary, to channel funds from the MFDP to implementing agencies; (iii) NACA to support improving the design and strengthening the National M&E 4

22 Framework, centrally and at district level; (iv) BNAPS to leverage long-term technical assistance (TA)/capacity building to NACA through consultants (see Table A2-2, Annex 2); (v) NACA to finance capacity training for CSOs and annual capacity assessments; and (vi) NACA to commission a social assessment study of the epidemic. 13. Component 2: Public Sector Ministries (US$19.2 million). This component was designed to support public sector ministries focusing on initiatives aligned with the earlier noted three priority areas of the NSF I and following consultation with partners. The component engaged supporting seven ministries (see Table A2-3, Annex 2) to improve HIV-related knowledge, behavior, and health outcomes by delivering prevention and treatment tailored to their target groups. Funding was to be based on Annual Work Plans submitted for review by the Project Steering Committee and the HIV/AIDS Technical Sub-Committee. Upon approval, funding was to be channeled through NACA for disbursement to ministries. M&E activities under this component were coordinated by NACA. 14. Component 3: Civil Society Organizations (CSOs)/Private Sector (US$21.6 million). This component was allocated the highest share of project funds, reflecting the importance of civil society and private sector organizations (CSOs/PSOs) in the achievement of the PDO. This component made community grants available to CSOs/PSOs, focusing on initiatives in line with the NSF I. Performance-based financing (PBF) was applied, involving four interlinked processes: (i) developing calls for proposals (CFP); (ii) evaluating, and awarding proposals by CSOs/PSOs; (iii) monitoring outputs by grantees; and (iv) disbursement linked to achievement of agreed output targets. Further, this component targeted community-level activities that prioritized specific HIV/AIDS prevention results and targeted vulnerable populations, as defined in the Operations Manual. These design features were expected to streamline funding procedures and deliver value for money by improving efficiency and focusing on specific thematic areas. 1.6 Revised Components 15. Components were not formally revised. However, there were some changes made as part of the natural course correction during implementation. This included scaling back BNAPS geographic coverage from nationwide, as described originally in the PAD, down to 11 districts (out of the total 15 nationally) to enable capacity building and avoid spreading resources thinly. Also, allocation of project financing across the three components was slightly modified, increasing resources for prevention activities implemented by CSOs/PSOs under Component 3 (Table A10-5, Annex 10). Lastly, allocations within Component 2 were augmented towards better performing ministries (MOH and the Ministry of Education and Skills Development (MOESD) in line with the MTR s recommendations. Together these ministries accounted for the largest share (69.1%) of project expenditures 7 (Figure A1-2, Annex 1). For example, fund allocations to the MOH increased from 2012, when there was a relative increase in the focus on biomedical prevention, such as safe male circumcision (SMC), and on the integration agenda (TB-HIV/AIDS and cervical cancer-hiv/aids) to boost project performance. 1.7 Other significant changes 16. The BNAPS was extended twice: (i) a year extension, from September 30, 2013 to September 30, 2014, and (ii) a six months extension until March 31, The rationale for the additional 18 months was to provide the necessary time to complete the project, particularly since one year was essentially lost at the start of the project due to the delay in the signing of the Loan Agreement and subsequent delay in project effectiveness. The extension therefore allowed for the Loan to be fully disbursed, as well as to enable the project to realize its full impact (Table A10-5, Annex 10). In addition, the second extension was accompanied by a loan category 7 Data by ministry for was not available, inhibiting a trend analysis for the Project s duration. 5

23 reallocation to reflect the agreed activities in the Implementation Plan over the period of the proposed extension. Allocations for goods and non-consulting services under subprojects were increased from 44.6% to 82.7%. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 17. Project Preparation. Initially, the project was planned to be processed with urgency 8 in response to the GoB s emphasis on the severity of the epidemic. However, the BNAPS took 20 months to prepare given the complexity of the project and the proposed IBRD buy-down mechanism (see more in paragraph 21) which required a high level of coordination with the EC as the co-financier, in addition to Government counterparts, and other key country-level DPs. The Bank may have underestimated the time required for coordination and to design such a novel instrument. Also, it is important to note that during project preparation there was no WBG presence in Botswana. In fact, BNAPS was the first project after a long time, preceding the development of the Country Partnership Strategy (CPS). The WB Country Office was opened during the third year of BNAPS implementation. Hence, throughout preparation (as well as implementation), the Bank team placed an extra emphasis on providing client support in the form of capacity building in Bank policies and procedures. 18. Risks. At appraisal, the overall risk rating was Moderate. While the ratings for the nine risk dimensions presented a largely realistic assessment of the possible challenges that could affect BNAPS, the overall Moderate risk rating was somewhat optimistic; a Substantial rating would have been more accurate (see Table A10-6, Annex 10). The highest risks were associated with the magnitude of the epidemic and the measurability of impact in the short run, the capacity limitations of NACA, limited government experience in collaborating with civil society and the private sector, and potential delays with audits or accounting for fund utilization under Component 3. Most concerns were legitimate but some have been attenuated as a result of technical assistance. 19. Participatory Process and Learning from Previous Operations. The project appraisal and design process was participatory, involving consultations with the GoB, including key ministries, a wide range of local CSOs, the Private Sector, and the main DPs. 9 As there had not been any prior Bank-financed projects in many years in Botswana, the team drew on the experiences of other DPs. For example, the design of fund flows was influenced by the experience of the Global Fund. To mitigate the concern that resources may not reach intended implementers, especially at the community level, BNAPS funds were to be channeled through the District Councils, which were familiar with donor funded project disbursements. 20. Overall, project preparation was moderately effective. It was successful in terms of its emphasis on capacity building and inclusiveness. However, the time required for preparing such a complex an innovative project with a new Borrower was underestimated and the risk assessment was somewhat optimistic. 8 References to urgency and emergency in the PAD (e.g., p. 8) are based on the severity of the epidemic and the importance of combating HIV/AIDS for the GoB. It does not refer to the project preparation process, as defined for Emergency Operations supported by the World Bank, which go through a simplified preparation (i.e., fast track). 9 Major partners: the US government (CDC, PEPFAR, and BOTUSA), the Global Fund (GFATM), ACHAP (partnership between the Bill and Melinda Gates foundation, Merck and the GoB), and key UN agencies (e.g. UNAIDS). Consultations were held with the EC on M&E targets and buy-down triggers to apply a performancebased modality for tranche disbursements. 6

24 21. Project Design. Focus and PDO: The Project was designed to address strategic and implementation gaps identified in the Mid-term Review (MTR) of the NSF I ( ), such as prevention as a national survival strategy, and was developed to support and catalyze the implementation of the National Operational Plan for Scaling-up HIV Prevention (2008). Regarding the broader design aspects, apart from being a Specific Investment Loan (SIL), BNAPS features two novel financing instruments: (i) an IBRD buy-down mechanism; and (ii) performance based financing (PBF), which, respectively, were introduced to reduce the cost of the loan to the Government and introduce more focus on results through output-based development financing. These were both important elements for the efficiency and sustainability of public financing. 22. IBRD Buy-down. BNAPS was recognized as a flagship project because of the IBRD buydown innovation. This was the first IBRD buy-down in Africa and the second globally. Given that its upper middle-income status excludes Botswana from the World Bank s concessional IDA resources, the GoB requested that the proposed operation be financed utilizing an IBRD loan buy-down mechanism, developed to increase the flexibility and concessionality of funding for projects where it is justified by global public good or cross-border externalities. This instrument relies on donor resources to lower the cost of borrowing an IBRD loan. The release of the buydown donor funds depends on project performance against agreed indicators targets. For Botswana, the European Commission (EC) agreed to support the buy-down of BNAPS, approving an additional 14 million (~US$20) grant for sector budget support (SBS) within its four-year Economic Development Framework (EDF10, ). 10 A structural distinction, compared to buy-downs in other countries, was that the EC and the GoB agreed on the flow of funds directly from the EC to the Ministry of Finance and Development Planning (MFDP). By design, the release of the buy-down contribution from the EC was conditional on reaching the targets set for the Intermediate Outcome (IO) indicators of the Project s Results Framework. The release of the first tranche was to be based on the results of the MTR, and of the second/last tranche was to be tied to results at project closing and were planned be discussed with the National Authorizing Office of the MFDP before submission to Brussels. By design, the allocation of the grant across the components favored the CSO/PSO component (50%). The remaining funds were to be equally split between NACA (25%) and participating ministries (25%). 23. Performance-based Financing (PBF). In line with the Project s emphasis in the original PDO on enhancing the efficiency and sustainability of the national response, BNAPS involved a disbursement-linked performance mechanism to strengthen the performance of line ministries and civil society/private sector organizations (CSOs/PSOs) receiving support under the project. The principles of PBF (e.g., active purchasing, contracting, internal and external verification, managerial autonomy, and separation of functions between regulator, fund holder, purchaser, and provider and beneficiaries) were to be applied through four interlinked processes, as discussed earlier. These include: (i) calls for proposals (CFP); (ii) evaluating and awarding proposals by CSOs/PSOs; (iii) monitoring outputs by grantees and verifying results; and (iv) disbursement linked to achievement of agreed output targets. In addition to enhancing efficiency and contributing to the sustainability agenda, the PBF was also designed to encourage an evidencebased approach. For example, the capacity assessment through the CFP mechanism, the monitoring of base and endline values, and the verification of results delivered by CSOs/PSOs were to ensure focused and evidence-based program implementation. 10 The activities under this grants were linked to Improved HIV/AIDS Prevention Indicator 8 of the EDF10. 7

25 24. Component Design: Based on policy documents and extensive consultations, three components were designed to create an enabling environment and a broad-based platform for effective, efficient, and sustainable HIV prevention. A critical challenge to enabling a more technically and financially efficient response was the limited capacity of NACA, as was the case with other countries in the region. Given its coordination role, NACA was designated as the primary coordinating implementing agency for BNAPS. However, at appraisal, NACA was yet to realize its potential due to weak operating systems and HR constraints. 11 Given these capacity constraints, Component 1 focused on improving organizational effectiveness and strengthening NACA s coordination function through capacity building, with a strong emphasis on addressing HR constraints, and strengthening the M&E framework. Component 2 highlighted the importance of the multisectoral response to the epidemic. It was designed in line with the comparative advantage of the involved ministries to encourage evidence-based interventions to reach defined target groups that fall under the mandate and jurisdiction of the given sectoral ministry. Budget reallocation across and within ministries based on performance targets was designed to catalyze more focus on outcomes and efficiency. Under Component 3, a paradigm shift by BNAPS was the introduction of community-centered approaches in HIV/AIDS prevention. This component moved away from the former multi-layered, top-down and supplydriven approach that did not allow for a direct purchaser-provider relationship between the GoB and the CSOs/PSOs, toward a performance-based platform. With the requirement to report baseline, progress and endline data, this component introduced improved accountability and transparency elements. The PBF platform was also expected to contribute to improved efficiency and sustainability. 25. In sum, the design of BNAPS: (i) focused on prevention; (ii) was policy-based and harmonized assistance to the GoB with DPs to operationalize the NSF; (iii) was innovative (e.g., IBRD buy-down, PBF approaches); (iv) used a multisectoral approach through a combination of interventions; (v) fostered the creation of a broad-based platform for the national response that builds on complementarity and harnesses the comparative advantages of the implementing agents (e.g. government at central, district, and local levels, CSOs, and PSOs); (vi) applied a learningby-doing approach, supported by the Bank s Multi-Country AIDS Program (MAP) for Africa; (vii) aimed to be evidence-based by creating systems to store and share evidence to inform choices in design, implementation, and policymaking. Although the design was based on identified shortcomings, it had limitations. First, it was complex, particularly for a first-time operation. Second, while emphasis was placed on necessary institution building, more attention could have been given to identifying the most strategic interventions (e.g., effectiveness, targeting, etc.). Third, the application of two innovative instruments (IBRD buy-down and PBF) with a new Client with a steep learning curve, added to the project s complexity. 26. Quality at Entry. A Quality Enhancement Review (QER) by the Bank carried out in April, 2008 concluded that the PDO was relevant and appropriate and confirmed the importance of the operation moving forward on a priority basis. However, the Panel encouraged the Team to come up with a realistic plan to avoid excessive expectations of what could be achieved in year one given that the Project was complex and engaged a new Borrower. The QER identified three areas for urgent attention: (i) developing a detailed operational plan; (ii) preparing a comprehensive social assessment to inform project design, including intervention targeting and the behavior communication strategy; and (iii) addressing technical issues to achieve improvement in health outcomes. Among technical issues were: integrating the findings of the 11 Assessments by the GoB, the Global Fund and the Bank provided similar findings on NACA s institutional capacity; that it did not possess the requisite internal capacity and processes in order to effectively and efficiently carry out its assigned function. 8

26 economic analysis into the design; articulating the strategies/technical interventions that would be undertaken to improve program efficiency and effectiveness; and discussing the impact on wider health system, STIs programs, and TB/MDR-TB in the southern African context. Regarding the novel instrument, the Panel noted that the IBRD buy-down would likely create stronger interest and ownership by the MFDP. To avoid the perception of a financing instrumentdriven approach, the Panel had recommended that the Team focus its efforts on strengthening some of the technical aspects of the Project to ensure strong ownership by the Ministry of Health. Despite these limitations, the QER Panel supported the timeline proposed by the Team given the urgency in addressing the HIV/AIDS epidemic. In response to the QER Panel s recommendation on the social assessment, the Bank team included among the legal covenants for BNAPS the implementation of an assessment within one year after effectiveness (June 2010). Regarding the finalization of the detailed operational plan within the first six months of implementation, the Team agreed that this was necessary. Addressing the outstanding technical issues identified by the QER Panel received less attention. Overall, the QER Panel s recommendations had led to improved design and also helped identify key action items for Year One supervision, which included: (i) focus on capacity building, including by establishing effective coordinating mechanisms by tapping into platforms used by DPs or through local presence; (ii) develop project-specific indicators (given that the project s objectives were more time-bound than the policy objectives of the NSF to which the Results Framework was linked); and (iii) develop a sound M&E system. Overall, the quality at entry is assessed as Moderately Satisfactory. 2.2 Project Implementation 27. From project inception in 2009 through closure in March 2015, the average rating for implementation progress (IP) was Moderately Satisfactory in Implementation Status Reports (ISRs). Challenges were faced at the start-up phase and at midterm (Table A10-7, Annex 10). 28. Start-up Challenges. There was approximately one year between Approval (July, 2008) and Effectiveness (June, 2009) as a result of a series of country-level delays, including the length of the GoB s Parliamentary review, and subsequent delays in addressing effectiveness conditions 12 due to general public sector challenges, as noted in the Country Partnership Strategy, including HR issues (e.g. civil servant hiring freeze). In December 2009, the Implementation Performance and a number of other performance sub-dimensions were downgraded by the Bank to Moderately Unsatisfactory because a number of covenants were not met, and because of limited staff capacity and lack of adequate progress with the M&E framework among other issues. These delays and the downgrading triggered the need to develop 3-month and 6-month Action Plans to remedy the identified causes of the delays. The implementation support action plan was discussed with Sector and Country Management to ensure effective actions and traction with the Borrower. 29. Mid Term Review (MTR). The MTR (May 31 - June 12, 2011) took place 2 years into implementation and was led by a senior Bank adviser. The MTR team conducted a systematic stock taking of critical issues that had hindered project performance with the objective to formulate recommendations to remedy the sluggish performance and to increase disbursements. At the time of the MTR, only 14.4% of funds were disbursed. Despite the low disbursement, prior to the MTR, the ratings for IP and all sub-dimensions and components, except for Component 2, were Satisfactory (January, 2011; ISR 5). The MTR was hindered by inadequate internal preparation by NACA, specifically, by the absence of timely updates of agreed project monitoring indicators. Given the data gaps, the EU s mid-term assessment was done separately 12 E.g., hiring TAs for FM, procurement, and M&E; clearing audit backlogs; and conducting the social analysis. 9

27 during a mission in October 2011, which focused on validating project performance on the trigger indicators for the IBRD buy-down. The EU confirmed the Bank s finding regarding the lack of basic documentation and management information that would enable the Project s performance assessment. Despite these challenges, based on the intermediate outcome indicators compiled for the EU s report, the assessment found that the level of target achievement was generally very high (Conseil Sante, 2011). While the results related to CSO performance were somewhat lower, most of the targets related to administrative and reporting aspects of the program were met or surpassed as a result of the intensive support provided by BNAPS. The policy trigger for the first tranche of the buy-down was the approval of the NSF II by the GoB, which was attained, triggering full disbursement ( 8 million, Quarter 4/2010) The Bank s MTR identified key implementation challenges as follows: (i) initial unfamiliarity with IBRD procedures; (ii) high expectations in results and ambitious targets; (iii) uneven coordination and management oversight by NACA; (iv) limited NACA HR skills in behavioral science; (v) inadequate oversight of public sector work plans by some implementing ministries, and often limited ownership of work plans, which yielded a supply-driven approach and challenged the ambition of creating integrated, sustainable programs; and (vi) limited familiarity with community-based Call for Proposals (CFP) financing mechanisms. While the overall fiduciary performance of the CSO component improved, substantial additional support was required on technical design, community-level monitoring, and the performance-based approach. Further, there was no independent results verification carried out for the PBF. The MTR also noted the challenges with the cultural context of high-risk sexual behavior. Particularly, MCPs continued to pose an obstacle to accelerating progress in indicators driven by behavioral dynamics and the social context. Indeed, despite the HIV/AIDS-related IEC efforts in Botswana, even when knowledge improved, awareness did not necessarily translate into behavior change The MTR acknowledged that most risks identified at appraisal materialized. In response to these challenges, the MTR recommended that targeted and expeditious response was required from the GoB, with support from the Bank team, to put BNAPS back on track. The MTR proposed to: (i) revise the PDO to enable a more realistic focus; (ii) improve NACA s capacity to coordinate across components through a modified and strengthened TA team, and operational research to strengthen the evidence base on prevention; (iii) focus on well-performing ministries and prioritized interventions; and (iv) strengthen the CSO/PSO component by introducing multiyear grants and providing strong technical and fiduciary TA to ensure the quality of prioritized interventions. One limitation of the MTR was that although it identified high expectations in results and ambitious targets as a challenge, it did not include among its recommendations the revision of targets and indicators in the Results Framework. The EU s October 2011 assessment mission agreed with the conclusion of the Bank s MTR regarding the need to refocus the PDO. Further, the EU recommended that the PDO revision should include the revision of the indicators and targets, including better definition of some indicators and setting more realistic targets that correspond to capacity. Last, it noted that progress in treatment was more robust and given that behavioral change takes a long time, approaches must be urgently discussed and expectations managed. 13 This is more than 50% of the total buy-down, in contrast with the plan for equal disbursements across tranches. 14 The causal pathway between knowledge, behavior, and health outcomes is not straightforward. More evidence is required on which behavior change interventions work and how through encouraging and testing innovative approaches (see Annex 3). Although its impact has not been quantified, the success of the no-cost national treatment program may have inadvertently created a sense of false security, undermining efforts that aim to change risky sexual behavior. The related moral hazard could worsen the fiscal space effect of the epidemic. 10

28 32. In response to MTR s recommendations on NACA s capacity and strengthening the evidence base, BNAPS initiated the procurement of TA to strengthen NACA s human resources for the Project on FM, procurement, and behavior change. However, operational research on prevention did not receive emphasis to inform implementation; for example, exploring which interventions work under Components 2 and 3, what factors determine implementation progress, and how to improve Project effectiveness and efficiency. Documentation of operational experiences was not centralized, despite repeated recommendations, limiting the opportunity to effectively share results and draw on lessons. To address the issue of prioritization, the project correctly sharpened its focus by streamlining activities under Component 2 and focusing on better performing ministries. Further, to improve performance under Component 3, BNAPS introduced a multi-year grant window and invested in improving project management for grantees. However, external results verification for the PBF was not implemented until the cycle. 33. Following the MTR, an Implementation Improvement Plan (through December 2011) was put in place and the Level 1 restructuring was processed, which revised the PDO (see Section 1.3). This was a turning point of the project and marks the beginning of Phase 2 of BNAPS. After this first restructuring, performance initially declined with the PDO and IP ratings downgraded to Moderately Unsatisfactory in April However, by August 2012, both ratings were upgraded to Moderately Satisfactory as a result of improved project management, which gradually intensified activities under all components, and the streamlining of activities to accelerate progress toward the PDO. BNAPS implementation was on an upward trajectory, as witnessed by increased disbursements from April 2012 onwards. 34. Regarding the IBRD buy-down, given the persistent challenges experienced with reporting progress on trigger indicators (particularly updates from line ministries and reporting on NACA s capacity building) and accessing related data for performance assessment, the EU applied a simplified evaluation framework. The disbursement for the second/final tranche was based on 3 sub-indicators with equal weight: (i) satisfactory progress of NACA strengthening; (ii) satisfactory progress of strengthening of key ministries ability to respond to HIV/AIDS; and (iii) satisfactory progress of civil society interventions. As not all targets were met under Component 1 and 2, the disbursement was partial, approximately 67% of the total tranche amount ( 4 million, 1/2013). 15,16 Thus, the total IBRD buy-down disbursed by the EU during the first and second tranches was 12 million, approximately 86% of the total possible buy-down funds. 35. Overall, while implementation had significant shortcomings prior to the restructuring in December 2011 (i.e., a Negligible rating). Corrective actions following the MTR created the environment for improved performance in the second half of project implementation (i.e., a Substantial rating). All components and the majority of sub-dimensions sustained Satisfactory ratings in ISRs and disbursement attained 100% by the project Closing Date. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 36. Country M&E Context for HIV/AIDS. The Botswana HIV/AIDS Response Information Management System (BHRIMS) 17 is the national multi-sectoral response M&E system. At 15 The total for the second tranche was 6 million, of which 2 million was released for attaining the targets under Component 3 and 1 million each for Components 1 and 2 for the partial achievement. 16 CSO performance was difficult to assess, other than relying on the related IOs as the final close-out report for CFP3 was not available for this ICR review. 17 The information management was decentralized under BHRIMS to the district level under the DMSACs. The DACs, who also serve as secretaries to the DMSACs, act as focal persons in data management at district level. The 11

29 assessment, data collection and reporting at national level was found to be advanced. However, routine data for HIV/AIDS were collected and processed at different levels, making coordination and data compilation by NACA a challenge. Monitoring of results at local level, as in most countries, was less well established. Given that changes at community level are critical to alter the course of the epidemic, improvements in the M&E framework and platform were required to inform project-level, programmatic, and policy decisions. 37. BNAPS M&E Objectives and Design by Component. Responding to the national need, BNAPS was designed to support the strengthening of the M&E system, including at decentralized levels. The Project s M&E activities aimed to support: (i) the development of a single national M&E system; (ii) a system for monitoring of community, district, and national activities (at the national level and by enabling districts to monitor and improve performance); (iii) the institutional, HR, and systems development aspects of M&E; and (iv) NACA s role as the lead coordinating agency for M&E activities related to HIV/AIDS between all agencies and donors. Under each component there were shortcomings to address in order to attain an operational M&E system. In order to improve performance monitoring under Component 1, additional HR in M&E, a strengthened M&E framework, and a centralized data repository were suggested. Under Component 2, improved M&E staffing and standardization of metrics/reporting across line ministries were required. Under Component 3, an enhanced community-based M&E framework was needed, which was particularly important given the PBF approach. 38. Design of the Results Framework and Data Sources. The BNAPS Results Framework (RF) was shaped by the Bank s Multi-country HIV/AIDS Program s (MAP) Global Results Scorecard (GRS, 2006). While the MAP was recognized for its flexibility and learning by doing approach, the GRS had acknowledged limitations, specifically some targets were ambitious and broad. The 2007 restructuring of the MAP highlighted the need for customized, stronger and functional M&E systems to demonstrate impact, and enable strategic management of the epidemic and intense supervision to meet the complexity of national responses to HIV/AIDS. Hence, the limitations of the GRS affected the RF of the BNAPS. Second, at the national level, the Project s RF was closely aligned with BHRIMS, the broader M&E system for HIV/AIDS that supported the implementation of the NSF. The benefit of this close alignment was that it ensured ownership and reduced reporting burden. Regarding the data sources for the RF: (i) routine data was to flow from BHRIMS, with particular strengthening under Component 3 as it did not have a CSO/PSO module; (ii) population level data (on prevalence and behavioral change) was to be collected through BAIS; and (iii) efficiency was to be captured through management systems relevant for Project activities. 39. One limitation of the RF -- noted by both the QER Panel and the EU -- was that it did not include enough project-specific SMART 18 indicators, which could lead to implementation challenges in results monitoring. It was noted that some of the PDO indicators were complex (e.g., compound indicators with age groups for which data was not available) or inadequate (e.g., lack of gender breakdown), and that some of the IO indicators were not well defined. Despite the importance of BCC activities carried out by CSOs to reduce risky sexual behavior, there was no indicator to measure the coverage of such interventions, or the efficiency of outreaches. The IBRD-buy down, which worked as output-based budget support, required improved monitoring and reporting. Hence, the EU substantively contributed to the indicator development during its appraisal (April-May, 2008), which were to be used to trigger the disbursement of funds related performance of BHRIMS was demonstrated. Yet, capacity constraints were noted in HR in M&E, and related IT material and software, which delayed the creation of the electronic ebhrims. 18 SMART: Specific, Measurable, Assignable, Realistic, and Time-related. 12

30 to Improved HIV/AIDS Prevention (Indicator 8 of the EDF10). The indicators were largely harmonized; out of the 13 IBRD buy-down trigger indicators, the BNAPS RF included 12 as IOs, one in reduced form (IO2), and one was classified as a PDO indicator. 40. Another limitation of the RF was that the indicators did not fully capture the PDO, and focused more on inputs and outputs than on outcomes. Given the PBF platform and the efficiency agenda, operational research with more specific and standardized measures of costs and effect could have been considered to improve project monitoring. In addition to MAP s learning-bydoing approach, BNAPS could have benefited from a robust evaluation agenda. At the time, a number of countries engaged in programmatic evaluation through (i) the HIV/AIDS Evaluation Program of the Africa Impact Evaluation initiative, started in 2007 (later Development Impact Evaluation Initiative - DIME); and (ii) the Health Results Innovation Trust Fund (HRITF), which launched IEs in While the latter focused on low-income countries and maternal and child health (MCH) programs, there was an emerging wealth of technical resource base in evaluation methodology and implementation to draw on to better inform programmatic and policy decisions. 41. Implementation. The Project s M&E performed weakly in the first three years due to HR constraints. Hiring specialists for M&E, periodic understaffing, and HR planning for M&E delayed defining and calibrating the project monitoring indicators that enable the quantification of progress toward the PDO. Although the EU s assessment mission (2008) contributed to the development of the BNAPS RF, which helped project monitoring, there were delays in developing the overall M&E Framework 19 that covers all components in depth. Following the initial sluggish performance, by January 2011 the M&E rating improved to Satisfactory. By the MTR (May-June 2011), M&E capacity weakened again, especially with respect to support at decentralized levels (Component 3). Delayed and limited M&E updates resulted in a downgrade to Moderately Unsatisfactory in September The EU s mid-term mission (October, 2011), which focused on assessing progress toward the targets for the IBRD buy-down tranche disbursement, also assessed indicators quality. The concerns related to the RF s design continued during implementation. The EU s BNAPS Project MTR Performance report (December, 2011) noted that some of the IO indicators were general, and the variable definitions and reporting were not standardized, undermining comparability and defying robust assessment. For example, it was noted that IO1 did not define the type of audit required (internal or external). IO2 (percentage of proposals originating from the CFP processed within the standard timeframe) was a crude measure of effectiveness. Unlike the EU s IBRD buy-down trigger indicator for this dimension, the BNAPS version for this IO in the RF did not take into account the rejection rate for CSOs applying for a CFPs, which was considerable at the time of the MTR (63%). IO4 did not define the frequency or depth of independent results verification during the project (it was defined as a binary yes/no variable). IO6 did not probe for the relevance and timeliness of line ministry HIV/AIDS policies; only for their existence. 43. The MTR aimed to remedy the noted challenges by specific and time-bound HR recommendations. As a result of hiring a Senior Community M&E specialist (end of 2011), performance gradually improved. The Community M&E TA helped setup baseline indicators and targets for the performance-based community grants, and addressed earlier data gaps. The External Results Verification (ERV) Report (Euro Health Group, 2013), implemented in 9 health districts, noted that indicator definition was an issue. There were a multiplicity of indicators at service delivery points (e.g. different definitions, age groupings, or the number of exposures 19 The M&E Framework accessed by the ERV Team dates December Earlier versions were not obtained. 13

31 required to count as target met ). Some indicators were simple, others were compound, leading to challenges at reporting. The indicators were not defined to enable measuring the efficiency and sustainability dimensions of the original PDO. However, acknowledging the progress made over time, the ERV report assessed that the M&E Framework was comprehensive and useful for all levels from December This valuation did not take into account the implications of the change in the PDO during the level 1 restructuring (December 2011) and the related M&E needs, such as greater emphasis on tracing preventive and innovative interventions. 44. Following the MTR, the need for improving alignment between the project monitoring indicators and the PDO was clear. In addition to the Bank s MTR recommendations, the EU s MTR also underscored the importance of simplifying the PDO and revising the RF to make it more realistic. Yet, despite the revised PDO (as of the December 2011) and the inclusion of the integration agenda, 20 the PDO indicators remained unchanged. As a result of persistent challenges with obtaining information on the progress on trigger indicators, the EU applied a simplified evaluation framework (see paragraph 34) to determine fund release for the second/final tranche. During the ICR process, it became evident that the indicators in the project s RF were not effective at measuring the impact that project activities had on the PDO. In order to more fully reflect project achievements, it was necessary to include additional indicators to measure the Project s effectiveness vis-à-vis the revised PDO and the effect of the integration agenda, which became pronounced during Phase 2 of the BNAPS. 45. Although the RF had deficiencies, due to the strengthened capacity and improved processes, the M&E performance rating remained Satisfactory between August 2012 and the final ISR in March 2015, indicating reliability in the quality and timeliness of reporting and enhanced accountability. 46. Utilization. The Botswana 2013 Global AIDS Response Report documents that the broader health system M&E context in which the Project operated was difficult; characterized by a (i) lack of a central HIV/AIDS database; (ii) lack of HR in M&E and understaffing at district level; and (iii) fragmentation in the M&E platform. While M&E continues to be a system-wide challenge, BNAPS unleashed a momentum to increase transparency and accountability between the GoB and DPs, across the administrative levels, 21 ministries, 22 and between the GoB and the CSOs/PSOs. 23 Despite the project-specific performance fluctuations in M&E prior to the June 2011 MTR, BNAPS support to M&E triggered structural and system-level changes. BNAPS served as a catalyst for strategic harmonization between the in- and outpatient health management information system (HMIS) platforms (PIMS and IPMS) 24 managed by the MOH, 25 and contributed to strengthening the M&E capacity of CSOs. BNAPS substantially contributed to improving M&E at the lower levels using the PBF and decentralized platforms. BNAPS strengthened NACA s M&E capacity and coordinating role. However, the lack of a central repository for data and survey results constrained the potential for evidence-based project and policy management. The impact of BNAPS on the institutional and HR aspect of M&E are also 20 The shifting emphasis from the HIV/AIDS focus toward the integration agenda and system strengthening was motivated by the growing pressure to address inefficiency within the sector (e.g., in the Botswana 2013 Global AIDS Report (NACA, 2014) and the Investment Case (NACA, 2015). 21 DACs, who serve as secretaries to the DMSACs, acted as focal persons in data management at district level. 22 NACA received annual work plans and year-end reports from ministries; used as the basis for resource allocation. 23 CFP, capacity assessment, baseline/close-out reports improved data and strengthened CSO M&E capacity. 24 Patient Information Management System (PIMS II): decentralized patient-level data for outpatient care (280 HIV outpatient sites in 2013). Integrated Patient Management System (IPMS): centralized patient-level data for inpatient care (operational in 11 of the 28 referral and district hospitals in 2013, i.e., 75%) (NACA, 2014). 25 Harmonization started in support of the integrated cervical cancer prevention program. 14

32 less pronounced. For example, M&E staff attrition in line ministries, and challenges with harmonizing reporting and M&E platforms attenuated progress. The Investment Case (NACA, 2015) identifies M&E as a strategic priority to improve the effectiveness and efficiency of the national response to HIV/AIDS. The report notes that across all HIV programs, weaknesses in data collection and M&E systems are major challenges to programmatic success. Shortages of HR for data collection and the lack of integration of various data systems are noted as major obstacles. With the ongoing institutional changes in the sector, it is important to draw on the lessons from BNAPS on the M&E dimension and strengthen the achievements. M&E remains a priority agenda beyond BNAPS. The Investment Case (NACA, 2015) argues for more resources to M&E system strengthening on the grounds that without reliable data, program planning and budgeting are not evidence-based. 2.4 Safeguard and Fiduciary Compliance 47. Safeguards. The Project was classified as Category C for environmental screening purposes, and hence no assessment was required. However, the Government s national medical waste management plan was included in the project file, as per agreement with the Bank s Africa Regional Safeguards unit. Fiduciary Compliance 48. Procurement. Procurement performance was affected by the slow start up of implementation and initial delays with hiring of technical assistance. The initial postprocurement assessment (October 2010) provided evidence of satisfactory performance. However, by April 2012, procurement was downgraded to Unsatisfactory. Due to a vacuum in TA in NACA, procurement stalled during an 8-month period (09/ /2012). From May 2012, the new TA rapidly re-initiated procurement activity, with support and training from the Bank's procurement specialist. The revised procurement plan (April March 2013) was cleared by the Bank. Most procurement activities in the revised plan remained on track, resulting in an upgrade of the procurement rating to Moderately Satisfactory by August Performance further improved with the recruitment of a second position, which improved processing of tenders, leading to a Satisfactory rating in April Financial Management (FM). Given Botswana s good performance on governance and PFM platform, the FM risk was low. Initial supervisions noted no major deficiencies in FM and rated it as Satisfactory. The PMU s computerized accounting system performed well. Internal controls were applied per the Operations Manual. There were no problems with flow of funds and disbursements. Interim Financial Reports (IFRs) were produced on a quarterly basis. By September 2010, the project met all FM conditions, except for: (i) the office of the Auditor General had not cleared all audit backlogs; and (ii) recruitment of finance staff in the public sector. In June 2011, the MTR stated that the project s FM arrangements 26 were acceptable to the Bank. The FM consultants ensured continued compliance with FM requirements. However, the report noted challenges using the government accounting system (GABS) in supporting the preparation of IFRs. Further, as the contracts for the FM TA were coming to an end, 27 the report emphasized the importance of the continuity of FM staff to maintain the adequacy of the FM of the project, especially as it intended to expand its geographic scope in Phase II and under Component 3. During CSO consultations for the MTR, representatives commended the NACA TA team for the training on grants management, financial and procurement management, and implementation support. Beyond the MTR, there was need for NACA to sustain performance in 26 E.g., recording transactions and balances, supporting the preparation of financial statements, safeguarding the entity s assets, and its being subject to auditing arrangements. 27 Respectively, on 31 July 2011 and 30 September

33 these areas. With the planned expansion, NACA management was advised to ensure that technical and fiduciary support to Phase I and II health districts was in place. By April 2012, due to reduced capacity in the FM unit, the FM performance declined. IFR submissions were delayed and quality deteriorated. The audit report for the fiscal year ending in March 31, was submitted five months late. 29 In April 2012, FM performance was downgraded to Moderately Satisfactory. By March 2013, the Bank expressed concerns on delays in disbursement and reporting. To mitigate these, advanced disbursement was proposed to receive funds up to 6 months in advance to pay for eligible expenditures. To improve reporting, the second FM specialist was to be assigned the responsibility of preparing the Bank reports. Given the noted challenges, the MS rating remained for the duration of the project. 2.5 Post-completion Operation/Next Phase 50. Sustainability. The project was designed as an integral part of NACA s work program and participating ministries and aimed to create a strengthened and sustainable 30 platform for multisectoral, coordinated, and broad-based HIV/AIDS prevention with the involvement of civil society. Regarding institutional sustainability, at appraisal it was expected that the social, sectoral, and community capacity aspects continue after project closure. Technical sustainability was assumed given that BNAPS supported proven interventions that had been documented to be effective in other country settings, and by identifying and harnessing complementarities between the public sector and CSOs/PSOs. Further, the PBF approach was to enhance technical and allocative efficiency. As to financial sustainability, the economic analysis argued that Botswana needed a balanced approach of prevention and treatment to ensure that HIV/AIDS expenditures do not crowd out other budgetary priorities and endanger the fiscal position achieved through prudent macroeconomic management. As one avenue, through efficiency gains, the PBF approach was to contribute to improved fiscal space for health. 51. Compared to expectations at appraisal, at closing, the sustainability assessment of the BNAPS was mixed, showing successes and challenges. In terms of institutional sustainability, there were challenges, as reflected in HR retention. The shadowing arrangement for staff capacity building at NACA was not as effective as expected. Staff turnover challenged all three components. However, there were some exceptions, such as the prisons project, managed by the Ministry of Defence, Justice and Security (MDJS), which sustained the staff from BNAPS. The MDJS ensured that the activities, with the requisite staffing needs, are sustained following BNAPS. Another notable success was the effective utilization of districts (DAC/DMSAC) in the context of the decentralized multisectoral management modality. More broadly, the recent restructuring that has merged NACA into the MOH under the Department of AIDS and Preventive Care raises questions and draws attention to important dimensions that need to be carefully managed during the transition phase, which is expected to last for two years (see more in paragraph 94). The financial sustainability aspect is difficult to assess due to data constraints. As discussed, the first restructuring simplified the PDO on the grounds that it was difficult to measure efficiency and sustainability effects during the Project s timeline. There was lack of clarity regarding these targets and the metrics to be used to quantify progress. Beyond BNAPS, lack of micro data to assess efficiency and sustainability has been a limitation in the sector. The micro data limitations were noted during the preparation of the Investment Case (NACA, 2015) This was aligned with the FY cycle of the GoB. 29 The project was included under the internal audit of NACA for the first time for the FY ending in March 31, E.g. through capacity building, under-study/twinning arrangements at various levels of the implementation. 31 The report notes that without additional HIV/AIDS and health funding, the current levels of treatment will not be sustainable Simulations show that filling the HIV/AIDS financing gap will require additional funds, which are significant in relation to the health budget. Relative to the 2014/15 financial years, when the total health budget was P5.69 billion, an increase of 10-20% will be required. 16

34 and by the Bank s missions between 2014 and 2015, which focused on calibrating a sector efficiency analysis to inform policymaking. Going forward, further explorations are required to better understand the financial sustainability aspect of interventions to inform policy decisions. 52. Continued World Bank Support. Building on the lessons from BNAPS, the Botswana health team has continued engagement with the GoB on two tracks: (i) Bank-supported analytic work to explore sector efficiency and related sustainability challenges with respect to health and HIV/AIDS financing; and (ii) Bank-supported project on health systems strengthening, aligned with the broader national agenda, currently under preparation for delivery in Assessment of Outcomes 53. Methodology. Per the Bank s ICR Guidelines a split rating is applied to assess the outcome of the BNAPS Project. The split rating is based on the weighted average of the Project s performance against disbursements before the change in the PDO (the period prior to the level 1 restructuring, ) and following the PDO revision until project closure ( ). These periods are referred to as Phase 1 and Phase Relevance of Objectives, Design and Implementation 54. The analysis below summarizes the relevance aspects across the two project phases, including the relevance of objectives, design and implementation, as follows: Relevance of Objectives 55. Phase 1 (rating=high): The original PDO was aligned with the priorities of NSF I ( ), focusing on prevention, financing innovative and strategic prevention and mitigation approaches, and strengthening the management and coordination of the national response. At the regional and global levels, the PDO was relevant for the MDGs, the MAP for Africa, and the UNAIDS Three Ones Principle. 32 In a country of 2.15 million, where HIV/AIDS remains the number one cause of mortality, with an estimated 319,750 HIV-infected people in 2013, of which 87.26% are on ARTs, and where the estimated number of new HIV infections was at 9,170 in 2013 (NACA, 2014), the relevance of HIV prevention and finding broad-based, innovative, costeffective, and sustainable ways to prevent HIV and reverse the epidemiological trend is unquestionable. Beyond the sustained relevance of preventive efforts, the emphasis on mitigation (e.g., education, livelihood training and employment) was also important as part of a structural and long-term approach to responding to the epidemic. Lastly, the relevance of the PDO with respect to efficiency and sustainability was fully recognized at the highest levels. As presented earlier, the fiscal space implication of the epidemic became a concern of the MFDP. The observed trend of reducing external resources for HIV/AIDS implies that more domestic funds would be required to maintain treatment coverage, and that sustained commitments to fighting HIV could increasingly displace other budget priorities and consume a progressively larger share of the economy. Hence, the focus on efficiency and sustainability of HIV/AIDS financing captures broader development priorities for which sector-level solutions are required. 56. Phase 2 (rating=modest): The PDO was simplified to one broad objective: to increase access to prioritized prevention services that reduce the risk of HIV transmission. While this objective was relevant in general, the revised PDO did not fully capture the gradual shift in focus observed during project implementation, consistent with the parallel move in Government priorities. Starting around the MTR, the Project strategically adjusted to shifting policy needs by incorporating the integration agenda with the objective to test and identify ways to improve the 32 One national authority for HIV/AIDS, one strategic framework, and one M&E system. 17

35 effectiveness and efficiency of service delivery. While this shift was policy-relevant, progressive in its direction, and contributed to setting the agenda for the sector more broadly, it was not reflected in the PDO, nor in the RF, which made project performance assessment difficult. The PDO was not effectively revised to be fully aligned with changing Government priorities or with the planned project activities. Also, the financing context of HIV/AIDS programs during Phase 2 suggests that sustainability remained policy relevant, as testified by a number of reports. The Botswana 2013 Global AIDS Response Report (NACA, 2014) lists decreased funding by DPs, the financial sustainability of no-cost ART, and the financial implications of co-morbidities with NCDs as top challenges. The Investment Case (NACA, 2015) projects that due to an increasing share of patients on second and third-line treatment regimens, the annual estimates for treatment costs per person are expected to increase by about 22% over a 15 year period, by While policymaking increased emphasis on improving the efficiency of HIV/AIDS service delivery and the sustainability of its financing, the revised PDO removed this focus. Therefore, while the revision served to simplify the focus of the project, it actually did not capture the evolving needs and priorities. Hence, the overall rating for relevance of the revised PDO is Modest. Relevance of Design 57. Phase 1 (rating=modest). The design of the project was generally adequate in terms of the nature and type of components and it attempted to bring in innovative instrument, including the buy-down and the PBF. However, the design was overwhelmingly ambitious for a country where the Bank did not have a presence and where this was a first health operation. Also, given the capacity constraints in coordination and management, and the fact that there were a number of weaknesses in the RF, as discussed above, the overall relevance of design is rated as Modest. 58. Specifically, Component 1, BNAPS could have considered (i) supporting the development of a more robust project management platform, particularly given that the Project was a laboratory for PBF; 33 and (ii) undertaking operational research to generate just-in-time information for project implementation and more systematic evidence for policymaking, particularly to inform prioritization decisions (e.g. which interventions to choose and scale up given capacity and/or resource constraints). 34 Under Component 2, enabling direct linkages between line ministries and CSOs/PSOs for activities that required synchronized activities could have been considered. 35 Under Component 3, in line with the MAP s learning-by-doing approach, the PBF aspect of the design was evolving as the project matured. 36 BNAPS could have supported the development and piloting of essential PBF tools and processes to enable informed scale-up. Lastly, regarding the RF, largely, the indicators were relevant but not sufficiently tailored to trace progress for all aspects of the PDO, particularly for efficiency and sustainability. Initially, the targets related to prevention were ambitious as the assessments did not fully account for the complexity of sexual behavioral change. 59. Phase 2 (rating=substantial). Relevance of revised design took into account the limitations raised during the MTR, which led to more focused geographic coverage; more investment for priority combined preventive interventions rather than relying too heavily on 33 The system-wide challenges in information management affected BNAPS. Project management could have benefited from a PBF project dashboard and a central repository of data/studies, as planned for at appraisal/mtr. 34 The economic analysis in the PAD notes the limited understanding of flow of funds, unit costs, cost-effectiveness, and effects. It refers to need for core diagnostics, operational research, or surveys to inform costs and effects. Yet, such analytic efforts could not be supported given that capacity constraints required focus on implementation. 35 Direct link was implemented later between the MOH and CSOs to coordinate supply and demand for SMC. 36 E.g., the selection/qualification criteria was adjusted as more information was available from the first round of CFP. Performance metrics and reporting were improved following hiring the Community-Based M&E TA, and the release of the findings from the ERV ( ). 18

36 behavioral change; increasing access to prevention also through integrated care delivery and, thereby, also improving efficiency; streamlining activities carried out by line ministries; 37 and, upgrading the tools and calibrating the processes for PBF for CSOs/PSOs. The technical edge of BNAPS improved during Phase 2, as reflected in improved ownership by key line ministries (e.g. MOH and MOESD). Sustainability planning, development of a centralized repository of information for more effective coordination and project management, and limited analytical work for evidence-based decisions remained the weak spots of the design. Relevance of Implementation 60. Phase 1 (rating=modest). With respect to the instruments, both the IBRD buy-down and the PBF were novel and were deemed as relevance instruments during preparation in order to enhance performance. Specifically, the PBF was chosen to improve project management, performance standards, and accountability, and the IBRD buy-down was meant to draw attention to developing the monitoring framework for the trigger indicators. In addition, the conditionality for the buy-down grant disbursement on attaining the agreed targets was also synergistic with the PBF approach. However, because of the innovative nature of these instruments, this project may not have been the right one to employ these. Rather, more time was likely needed to initiate the HIV/AIDS operation and then introduce these on a pilot basis. This is particularly so because the project was, as mentioned above, the first Bank-financed project in Botswana in over 20 years. Therefore, more time would have been beneficial to calibrate the design features of these instruments. For example, the PBF design could have been strengthened through pre-piloting before the implementation was scaled up to the initial five districts. Likewise, for the IBRD buydown, while the instrument was aimed to help the Government reduce the cost of borrowing, it required a well-functioning monitoring and evaluation system to be able to meet the triggers. As has been discussed above, the EU had substantial issues in getting the relevant data from Government and had issues with the measurability of the indicators. This led to the EU not being able to fully disburse their buy-down. It is important to note that Phase 1 was essentially spent on trying to figure out how to make these instrument work; while a noble task, this detracted attention from the technical issues at hand. 61. Phase 2 (rating=substantial). During this phase, relevance of implementation modalities began to make more sense. Both instruments, the IBRD buy-down and the PBF, were now more advanced and could be used more effectively. Also, the initial focus on improving capacity and coordination across the various agencies began to pay off. There was now sustained capacity building at all levels: (i) between the Borrower and the Bank, (ii) the Government and CSOs, (iii) NACA s capacity to effectively manage the PBF. The project modality was an effective way to foster stronger local presence. Lastly, and most importantly, Phase 2 allowed for more focus on technical issues and allowed greater focus on M&E which also led to the ability to measure and monitor project implementation as well as the National program. 62. Based on the evidence and support available at the time of the design, and acknowledging the proactive optimization efforts by the team, the proposed rating for Relevance is Modest for Phase 1 and Substantial in Phase Achievement of Project Development Objectives 63. Ten PDO indicators are used to assess project efficacy, of which 6 are original PDO indicators, 3 were added during the ICR review, and 1 is a reclassified IO. Project performance 37 The variation in the strength of the relationship between NACA and line ministries attenuated the Project s impact. This was due to different levels of ownership, commitment, and technical strength, as realized by the MTR. 19

37 is presented by phases, before and after the PDO revision (Phase 1 and 2). 38 For the assessment, the number of PDO indicators changes by phase as not all interventions were applicable/measurable in both phases. 39 Varying degrees of progress were made towards the targets in the original and newly introduced PDO indicators (Table 1). In Phase 1, of the total of 8 indicators, two targets were surpassed ( S : #5 and #9), 2 achieved ( A : #6 and #8), 1 partially achieved ( PA : #2), and 3 not achieved ( NA : #1, #3, and #4). Based on the share of the total PDO indicators classified as Surpassed, Attained, or Partially Attained relative to the total number of PDO indicators relevant for Phase 1 (5/8), 62.5% of the targets were met, which corresponds to a Moderately Unsatisfactory rating (Table 1). In Phase 2, of the total of 10 indicators, 4 targets were surpassed ( S : #5, #7, #9, and IO12), 2 were achieved ( A : #6 and #8), 1 was partially achieved ( PA : #2), and 3 were not achieved ( NA : #1, #3, and #4). The 7/10 ratio translates into a 70% attainment outcome, which corresponds to a Moderately Satisfactory rating (Table 1). 64. BNAPS collected routine data on 12 Intermediate Outcome (IOs) indicators, which were important tracers for the PBF. In Phase 1, out of the 12 IOs, 8 achieved their targets and 4 exceeded their targets, which corresponds to a Highly Satisfactory rating. In Phase 2, the total number of IOs is 11 because IO12 was reclassified during this ICR as PDO. This was motivated by the importance of this indicator to trace progress by CSOs under the PBF, which is not captured at all through the original PDOs. The rating for Phase 2 is also Highly Satisfactory. Table 1: Results for PDO and IO indicators PDO Indicators Phase 1 Phase 2 Surpassed (S) 2 (1 original - core, 1 added at ICR) 4 (1 original - core, 2 added at ICR, 1 reclassified IO) Achieved (A) 2 (1 original, 1 added at ICR) 2 (1 original, 2 added at ICR) Partially Achieved (PA) 1 (original) 1 (original) Not Achieved (NA) 3 (original) 3 (original) Total PDO Indicators 8 10 Total S, A, PA 5 7 Share Attained 62.5% 70% Rating by Phases MU MS IO Indicators Phase 1 Phase 2 Surpassed (S) 4 (original) 3 (original) Achieved (A) 8 (original) 8 (original) Partially Achieved (PA) Not Achieved (NA) Total IO Indicators Total S, A, PA Share Attained 100% 100% Rating by Phases HS HS Rating Scale: HU/U (0-40%); MU (41-64%); MS (65-79%); S (80-94%); HS (95-100%) 38 The split rating approach in this section is challenged by data limitations (e.g., BAIS only provides 2 data points) and changes in data presentation of PDO and IO indicators before and after restructuring. 39 The newly introduced PDO 7 is relevant for Phase 2 as the intervention that is traced by this indicator received funding from Results for the IO 12, which is reclassified here as PDO given its importance to measure progress under the PBF, were not available before the December 2011 restructuring, which marks the start of Phase 2. 20

38 Phase 1 Efficacy Assessment 65. The original PDO focused on increasing the coverage, efficiency, and sustainability of targeted HIV/AIDs interventions, particularly financing strategic and innovative prevention and mitigation activities, and this was to be attained under NACA s improved management and coordination capacity. The attainment of these objectives is discussed below. 66. Coverage. Geographically, as planned, the first phase of BNAPS covered 5 districts, reaching approximately 58% of the Project s target population, including the highest prevalence areas. All original indicators entailed focus on coverage, although for different interventions, including behavior change communication, biomedical interventions, and focused health system/infrastructure strengthening. The indicators are knowledge of HIV transmission (PDO 1); behavior change to alter risky sexual practices and related social norms (PDO 2 and 3); improved biomedical interventions and increased supply side capacity, such as access to quality SMC, to HAART compliant facilities (core indicator), and improved TB care, which is indirectly linked to HIV through the integrated management of comorbidities (PDO 4, 5, and 6). The ICR introduced 2 new indicators, which focus on knowledge of serostatus (PDO 8 and 9). The baseline and end-line values for the PDO indicators, as well as other selected relevant statistics (prevalence, incidence, knowledge and behavior) are presented in Table A10-8 in Annex 10. Briefly, progress on the PDO indicators is as follows: PDO 1 (Not Achieved): IEC coverage increased substantially. Regarding efficiency, there were some results with much greater increase in knowledge than previously, although well below the target. The data shows slow progress on knowledge of HIV transmission and prevention. Between 2008 and 2013 (BAIS III and BAIS IV), with an increase of approximately 7 percentage points from 40 to 47%, more than half of the youth does not have adequate understanding of HIV/AIDS risk. While this indicates moderate effectiveness, relative to the flat trend between 2001 and 2008, BNAPS had an impact, as indicated by the higher rate of progress (19%, compared to 5.6% and 5% for the previous periods, which is approximately a 4-fold growth (Figure A10-4 Panel A, Annex 10). IEC activities under Component 2 and 3 (through mass media, school-based and youth education, community/social communication, and individual counseling) enabled this growth (see Top 10 CSOs budget and activities in Table A1-1 and Box A1-1 of Annex 1, and the detailed activities by the public sector and CSOs/PSOs in Annex 2). PDO 2 (Partially Achieved): The abstinence and condom use targets for the age group were attained. Yet, the trend is flat, measured against targets that merely sustain not improve performance. During and despite BNAPS and other interventions, condom use for the age group regressed relative to their baseline values. The modest effectiveness is influenced by various factors. Apart from the knowledge issues, reports note condom programming 40 and supply chain bottlenecks, 41 which affect physical access to condoms. Further, improving condom use may be challenged by behavioral disinhibition and/or moral hazard. With respect to disinhibition, preventive interventions, such as SMC could cause men to abandon risk-management behaviors, such as condom use. Risky sexual behaviors may also be associated with moral hazard, for example, when condom use is perceived to be less critical given the free access to treatment (ARVs). Hence, the modest effectiveness of BNAPS must 40 Approximately 20,000,000 male condoms were distributed from the CMS in Male condom distribution increased to 22 million (2011). The total need was estimated to be around 33 million. In 2011, the CMS distributed 225,832, female condoms (about 1% of the male condoms). 41 A comprehensive condom situational analysis and survey by CMS identified challenges in the condom logistics system: (i) poor warehouse conditions; (ii) inadequate transportation; (iii) non-adherence to inventory control systems; (iv) few logistics reporting tools at service delivery points (SDPs); (v) lack of central coordination; (vi) limited logistics skills among healthcare workers; and (vii) limited educational materials on condoms at SDPs (Badubi, Endailalu, and Shioso, 2012; USAID, 2013). 21

39 be assessed against this context of system-level inhibiting factors. Unless supply chain and attitudes are addressed, IEC efforts to improve condom use cannot be fully harnessed. PDO 3 (Not Achieved): Despite the extensive IEC efforts by the public sector and the community-based platform, MCP outcomes regressed between base- and endline for both youth cohorts. This raises concerns regarding the effectiveness of IEC, and points to continued challenges with entrenched cultural and social norms with respect to sexual partnership, as well as gender aspects, including economic empowerment. BNAPS could have benefited from a stronger gender dimension to improve resource targeting and effectiveness. Without gradually addressing these structural issues, progress in MCP is expected to be slow. PDO 4 (Not Achieved): Although the SMC target was not met, BNAPS showed significant progress in this area, reaching a total of 150,136 men between 2012 and 2015, and an annual output of 32,527 circumcisions in the last year (Figure A10-3, Panel B). The initial slow uptake was caused by inadequate supply-demand coordination between the MOH and CSOs. When this was resolved, service delivery intensified, showing the potential of SMC which is recognized as a highly effective prevention going forward. PDO 5 (Surpassed): The increase in the number of HAART compliant health facilities from 33 to 566 shows extensive expansion in service delivery capacity and indicates a substantial improvement in target population coverage and access to HIV prevention and treatment services. The magnitude of the expansion suggests that this intervention was highly effective. However, this measure is partial. The PDO s definition is not sufficient to assess the effectiveness of this intervention as there is no data on how the HIV/AIDS-related service delivery volume changed as a results of operating more facilities. PDO 6 (Achieved): The TB treatment with DOTS target was met. Among the contributing factors to this are the improved service delivery infrastructure (PDO 5), training of MOH staff at the various levels on TB diagnosis and care management, and improved equipment, as discussed under PDO PDO 8-9 (Achieved and Surpassed): These indicators are linked to voluntary counseling and testing (VCT), access to and coverage of which have improved during BNAPS under Component 2 and 3 (see Annex 2 and 3). While further improvements are necessary, BNAPS played a role in the observed improvement in testing rates and awareness of one s serostatus. 67. Based on these results, that 62.5% of the PDO indicators were met, and that some of the PDO indicators are not defined adequately to assess effective coverage, the rating for the coverage dimension is Modest. 68. Efficiency and Sustainability of Targeted HIV/AIDS Interventions. The findings on efficiency and sustainability are mixed and incomplete. The IO indicators that are related to efficiency and sustainability focus on operational and procedural efficiency 43 and not on the technical efficiency of the interventions (i.e., the extent to which they achieved their main intended end of reducing transmission). While the targets are met, these measures are limited; they imply positive trends but do not allow quantifying efficiency gains and sustainability. As discussed under the M&E section, the PDO indicators were not defined and selected to enable measuring efficiency and sustainability. Notwithstanding, some progress on efficiency and sustainability can be expected because of the results-oriented instruments, such as the IBRD buydown and the PBF. Phase 1 may also have contributed to improved allocative efficiency, based on the expectation that more prevention would reduce the need for costly treatment (i.e. improved allocative efficiency). However, neither the Project s RF nor the broader monitoring efforts 42 PDO 7 is not presented hare as the funds for the TB intervention that it measured were focused during Phase E.g., timely disbursement and data submission, transparent financial management, number of organizations funded, share of organization funded that have reached targets, etc. 22

40 include an analytical plan for economic evaluation, which would collect effectiveness (e.g. cases treated or averted) and cost data (i.e., unit and incremental cost) to assess efficiency and sustainability based on evidence. There was no trend and counterfactual analysis (e.g., using epidemiological monitoring and project data) to enable an economic assessment of project performance over time, across intervention types, or implementers. This is an omission that has affected and limited decision-making during and beyond BNAPS. Hence, the overall rating for this dimension is Negligible. 69. Strategic & Innovative Prevention and Mitigation. The results suggest that targets related to inputs for prevention (e.g., PDO 5: HAART compliant facilities) and biomedical prevention (e.g. PDO4: SMC) are relatively easier to attain (or manage their progress trajectories). On the other hand, behavioral outcomes for youth cohorts have proven to be sticky and difficult to influence (e.g. PDO 2 and 3, which measure abstinence, condom use, and MCP). At appraisal, the PAD s economic analysis noted three reasons for high HIV prevalence: MCP, lack of SMC, and gender dynamics (Halperin, 2007), which can combine to produce a lethal cocktail that may increase HIV transmission to 30-fold (Wilson, 2007). The modest progress in these areas urges attention and encourages rethinking what interventions and strategies could be used to more effectively address risky sexual behaviors. Further efforts in designing and testing interventions are required to inform prioritization and scale-up decisions. While the policy ambition focused on zero infection by 2016, the experience from BNAPS suggests that aiming for incremental changes is more realistic. Attaining prevention targets requires good data on where and how new infections are occurring. Moreover, attaining targets through combination prevention (behavioral and biomedical) may be more feasible, particularly in the short to medium term. Lastly, the evidence on mitigation interventions is limited. BNAPS was effective in reaching some youth target groups with life skills training. For example, under the MoLG-supported HIV education for Vulnerable Children & Children in Difficult Circumstances, 3,200 children received yearly HIV prevention life skills and 1,280 parents a year were trained on parental skills in psychosocial support to children. Under the CSO component, annually 300 children benefited from youth camps for life skills. Beyond these broad coverage measures, there is no evidence on the impact of these interventions. Strategic multisectoral mitigation interventions (e.g., livelihood skills, employment, etc.) and gender or poverty-based targeting (e.g. through conditional cash transfer to single, deprived mothers, etc.) could make investments more strategic and effects more pronounced. However, more evidence is required to support strategic decision making. The overall performance of BNAPS on this dimension is Modest. 70. Coordination & Management Capacity. BNAPS support to NACA improved its coordination and management capacity, albeit with the delays in TA and HR sustainability limitations noted earlier. Institutionally, NACA had an uneven relationship with line ministries. Coordination, ownership, performance heterogeneity, and staff turnover hindered project outcomes. Yet, substantial progress was attained in capacity building, which laid the foundations for a more effective and efficient operation during Phase 2. The CSO/PSO component was an important extension to build a platform for the long-term response to the epidemic. The CSOs showed agility in implementation and were open to learn and benefit from operating on a PBF platform. NACA had a successful learning curve with the CSOs and the PBF approach to manage grants. Notwithstanding the challenges and the overall low performance during Phase 1, the Project s impact on improving NACA s capacity was Substantial. 71. Based on the results attained under the specific project objectives, described above, the rating for the Efficacy of Phase 1 is Modest. 23

41 Phase 2 Efficacy Assessment 72. The revised PDO focused on access to prioritized prevention to reduce the risk of HIV transmission. 73. Access. Access is closely related to the coverage dimension that appeared in the original PDO. During the second phase, as a result of the geographic expansion relative to Phase 1, which covered 5 districts, BNAPS expanded coverage to the remaining target population reaching a total of 11 districts in the nation, creating access to improved preventive and treatment services for approximately 80% of the total population. 74. Compared to Phase 1, effectiveness improved during Phase 2 due to the sharpened focus on strategic interventions (e.g. improved SMC outcomes, and integrated care delivery), enhanced targeting of project resources, and the improved efficiency of implementation (e.g., due to streamlining Component 2 and strengthening PBF under Component 3). These changes led to improved performance under each component and intensified progress towards the targets in most areas. Progress on PDO indicators 1-6 and 8-9 are not discussed here for the sake of brevity. The context is presented in the previous section. The trends are shown in the Data Sheet, as well as in Table A10-8, Figure A10-3 and Figure A10-4 of Annex 10. Relative to Phase 1, performance improved, even for some prevention interventions. For example, as a result of strengthened supply-demand coordination between the MOH and CSOs, uptake for SMC increased between 2012 and The acceleration during Phase 2 is partially the result of creating the multi-year grant window following the MTR and strengthened processes to screen and monitor CSOs. However, the improvements could not be translated into population level outcomes collected through BAIS IV, which only captures trends until 2013 and not the Project s closure in Hence, it is possible that the behavioral effects are not fully recognized (underestimated). To better capture BNAPS contribution during Phase 2, additional indicators were included during the ICR. One is related to the integration agenda (PDO 7: TB diagnosis and treatment). The other indicator is reclassified IO (IO 12, reclassified as PDO 10). PDO 7 (Surpassed): The ICR included this indicator on the grounds that integration is closely related to the efficiency and sustainability dimensions, two explicit objectives of the original PDO and broader sector objectives. TB case detection and treatment success rate have improved (relative to the baseline and regional benchmarks, respectively shown in Table A2-12 of Annex 2 and Table A3-6 of Annex 3) due to investments by BNAPS in HR (regional TB coordinators and laboratory staff training), procurement of laboratory technology (GeneXpert), infrastructure upgrade (TB isolation wards), and coordination and dissemination meetings. As noted earlier, this indicator is indirectly related to HIV prevention through integrated delivery of preventive activities and managing co-morbidities. Among tuberculosis patients registered in the TB program with known HIV status, the frequency of co-infection was recorded at 63% in Testing for HIV in TB infected patients increased to 87% in 2012 from 68% reported in Cotrimoxazole prophylaxis for TB/HIV co-infected patients reached 90%. While ART coverage was reported at 65.2%, TB and HIV clinical care guidelines were aligned in 2012, making any TB/HIV co-infected patient eligible for life-long ART. The TB and HIV Policy Guidelines were completed in PDO 10 (Surpassed): This indicator, originally IO 12, measured the percentage of organizations funded who reached the target specified in their proposal. The ICR proposes this reclassification because of the importance of CSO performance under the PBF, which is not captured in the original PDOs. This indicator primarily gauges target met and less so efficiency because of the definition challenges noted related to IOs; specifically, that they focus more on outputs than outcomes, and the metrics were not standardized to allow for comparability across implementing CSOs. 24

42 75. Lastly, although efficiency and sustainability are not part of the revised PDO, the contribution of BNAPS to these dimensions must be recognized because of their continued importance on the health policy and development agendas. The efficiency gains generated during Phase 2 through the strengthened PBF and the integration agenda represent important contributions to the overall ambition of BNAPS in supporting the GoB (see more in the economic analysis in Annex 3). 76. As a result of increased access, the strategic refocusing, improved implementation, and strengthened outcomes, the proposed rating for Efficacy in Phase 2 is Substantial. 3.3 Efficiency 77. The PAD included an economic analysis, 44 which provides a rich context and strong economic arguments. The analysis primarily relies on a review of the literature and presents secondary data to build the economic argument for the project. It does not include BNAPSspecific calculations or estimates of net present value (NPV), economic rate of return (ERR), or cost-effectiveness. Hence, it is not possible to directly compare such estimates and expectations between the baseline and the endline (project closure) in this ICR. As in the preceding sections that discuss project performance, efficiency is assessed by phases, before and after the PDO revision. The assessment is guided by an updated review of the literature, which is required not only because new evidence is available since appraisal, but primarily motivated by the fact that the scope of BNAPS changed, including: (i) its geographic scope (scale back; see Section 1.6); and (ii) the thematic scope (vertical to diagonal tendency through the integration agenda). 78. Epidemiology. To contextualize the economic analysis, an overview of HIV epidemiology before and after BNAPS is provided. With respect to final outcomes, a review and analysis of BAIS data between 2004 and 2013 shows mixed results. Cohort specific prevalence data from BAIS II (2004), BAIS III (2008), and BAIS IV (2012) suggests that survival improved (especially among those age 35 and above) prior to 2008, before BNAPS (Figure A3-1). BAIS III (2008) and BAIS IV (2013) data show that incidence marginally declined from 1.45 to 1.35 (0.1 over 5 years), which means that the epidemiological context for BNAPS was largely a stabilized one, as a result of the intensive and effective treatment program from UNAIDS cross-country data (Figure A3-2) corroborates this, showing that the largest drop in incidence is prior to Because BAIS IV has not published age specific incidence estimates, there is no firm conclusion about incidence reduction across the age groups to assess the effectiveness of prevention strategies. Analysis of prevalence data of BAIS from 2004 to 2013 by age-cohorts shows that prevention efforts for youth were largely ineffective since Since then, all three of the youngest cohorts had doubled or tripled their HIV prevalence by 2013 (NACA, 2015). Cohorts in the and year old age groups surpassed the national HIV prevalence at 18.5% in Given the under-resourced prevention context, although the financial contribution from BNAPS was small as a share of the total NSFII budget (6.5%), it represented a significant share of the resources that went toward prevention (65%) (Table A3-2). Overall, there is no strong evidence on the impact of BNAPS through the behavioral channel. However, through some targeted BCC interventions and the biomedical prevention interventions BNAPS played a role in: (i) contributing to stabilizing the epidemic; and (ii) catalyzing system-level changes, required to enable further/more sustainable gains in this area. The assessment of the interventions that contributed to these outcomes are briefly discussed below (see Annex 3). 44 Financing and Delivery of HIV/AIDS Prevention, Treatment, & Social Support Services in Botswana: An Economic Analysis. 25

43 (1) Behavioral Interventions to Prevent HIV Transmission Phase 1 & PDO-level outcomes on condom use and MCP suggest no improvement on these risky behaviors, particularly for the age group. Knowledge of sexual modes of HIV transmission remains low, with little progress detected over time (Table A10-8, Annex 10). Despite improved VCT coverage, further efforts are required to attain a higher coverage and ensure linkage to care. Due to lack of data on program costs and the observed efficacy in terms of cases averted, it is not possible to assess the impact and cost-effectiveness of behavioral change approaches applied in BNAPS. Since one objective of BNAPS was to support highly effective innovative preventive approaches, data limitations affected prioritization and strategic decision making. The status of these indicators and the data challenges motivate a Negligible rating for the efficiency of behavior change interventions for both phases. (2) Biomedical Prevention of HIV Transmission through SMC Phase 1 & In line with the combination prevention method, BNAPS ramped up its prevention efforts in Phase 2 by adding increased emphasis on SMC, leading to a total of 150,136 males circumcised by March 2015 (i.e., 39% progress toward the target set by the end of the project). Although the target was not attained, the coverage enabled by BNAPS is considerable; equivalent to approximately 35% of the male population covered under BNAPS, and 18% of the total male population. Although further efforts in demand generation are required to attain the coverage target, the MOH has been strategic in its targeting (see Box A3-1) and the improved coordination with CSOs has also contributed to improved efficiency. With regards to effectiveness and costeffectiveness, evidence from a randomized control trial in South Africa concludes that in settings with high or moderate HIV prevalence among the general population, adult SMC is likely to be a cost-effective HIV prevention strategy, even when it has a low coverage (see more in paragraphs and Box A3-1, Annex 3). 82. These findings suggest that the increased emphasis on SMC under BNAPS was an effective shift in the approach mix and led to improved outcomes in prevention. Given the slow uptake of SMC during Phase 1, the efficiency of this intervention was Negligible. Following improved effectiveness during Phase 2, the efficiency is rated as Substantial, particularly taking into account long-term effects and secondary transmission to women. (3) Biomedical Intervention & Integration Agenda: TB-HIV/AIDS Phase 1 & Apart from high efficacy, the estimates on economic returns indicate good value for money. The rating on efficiency for Phase 1 is Substantial and High for Phase 2, during which BNAPS-supported activities intensified and the integration agenda became more pronounced. (4) Biomedical Intervention & Integration Agenda: Cervical Cancer-HIV Phase Beyond the biomedical effects and the consequent economic effects, the cervical cancer program served as a model and had a catalytic role in promoting and operationalizing the integration agenda, leveraging the support of the MFDP (in the form of resource allocation to EPI), which led to a national rollout of HPV. The efficiency rating for cervical cancer prevention and integrated care delivery model is High, only applies to Phase 2 given its launch in (5) Broader Effects 85. In addition to the disease-specific health and economic impact, BNAPS had broader implication on the health system through catalyzing policy change and harmonization of the Health Management Information System (HMIS). These broader implications underscore the relevance and importance of BNAPS and justify a Modest rating for Phase 1, which provided lessons and the context for project design revision, including system-level aspects, and a Substantial rating for Phase 2, which prompted broader changes. 26

44 Efficiency of Implementation and Overall Efficiency Rating 86. The sluggish project start-up and initial low effectiveness indicates that the efficiency of implementation was Negligible during Phase 1. Following the restructuring, the efficiency of implementation significantly improved during Phase 2, including accelerated disbursements, improved outcomes for the investment, which merits a Substantial rating. Given the evident discontinuity in performance for the above discussed interventions before and after the December 2011 restructuring, the proposed overall rating for efficiency during Phase 1 is Modest and for Phase 2 it is Substantial. Overall, while Phase 2 demonstrated good performance, the development effectiveness potential of BNAPS was weakened by the low implementation efficiency prior to the restructuring and the inadequate progress on the behavioral dimension. Table 2: Summary of efficiency sub-ratings Efficiency Sub-Ratings Phase 1 Phase 2 Behavioral 1 1 SMC 1 3 TB 3 4 CC 4 Broader Effects 2 3 Efficiency of Implementation 1 3 Average Overall Efficiency Modest Substantial Scale: 1=Negligible 2=Modest 3=Substantial 4=High 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 87. The proposed overall outcome rating for BNAPS is Moderately Satisfactory. The table below summarizes the proposed ratings for each phase (before and after the level 1 restructuring in December 2011, which revised the PDO) and provides the calculation for the overall outcome rating for BNAPS (Table 3). The numerical value is 4.23, which corresponds to the final MS rating. Table 3: Overview table for overall outcome rating Original PDO Phase 1 ( ) Revised PDO Phase 2 ( ) Relevance Modest Substantial Objective High Modest Design Modest Substantial Implementation Modest Substantial Efficacy Modest Substantial Efficiency Modest Substantial Outcome Rating U S Numeric Value of Rating 2 5 Weight (Disbursement Share) 26% 74% Weighted Value Overall Project Outcome Rating 4.23 = MS Note: The overall project rating is based on Table 1, Appendix J, ICR Guidelines (OPCS, 2006). The calculation is based on the weighted average of project performance for Phase 1 and 2, using the disbursement shares. Scale: 1=HU, 2=U, 3=MU, 4=MS, 5=S, and 6=HS. 27

45 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 88. BNAPS is aligned with the GoB s National Strategy for Poverty Reduction, which emphasized the role of HIV/AIDS as both a cause and consequence of poverty and unemployment. BNAPS aimed to contribute to breaking this vicious circle by targeting districts that not only suffer from high HIV/AIDS prevalence but (given the high correlation) also show high levels of poverty. 45 Beyond the broad geographic targeting (focusing on high prevalence districts/areas), BNAPS is expected to have a favorable poverty effect as a results of the age cohorts (14-19 and 20-24) at its focus. Poverty in Botswana has a young face, with children less than 15 years of age representing 46.2% of the poor in 2009/10 (World Bank, 2015). In Botswana, poverty is found to be highly correlated with family size, which in turn, is correlated with HIV prevalence Strong gender disparity in HIV prevalence appears throughout all age groups in BAIS IV (2013). The overall HIV prevalence amongst women was 21% as compared to 14% among men. The vulnerability of women is related to gender based violence and poverty. In 2012, the Gender Based Violence Indicator Study reported that 67% of all women in Botswana had experienced some form of gender related violence in their lifetimes, increasing the risk of HIV acquisition. Poverty and power discrepancies in intimate relationships particularly drive women to expose themselves to HIV and sexual violence. The 2011 ANC Sentinel Surveillance Survey showed that 80% of all mothers were single, and 61% were unemployed with only 25% reporting full time employment. Through its preventive interventions (e.g. BCC messages targeted at youth women using the school-based platform under the MoESD and TEC, reinforced by CSOs; communication via Women Sector Committees implemented under the MIST; and female condom distribution managed by the MYSC, etc.; see Table A2-4, Table A2-8 and Table A2-9, Annex 2) BNAPS support to vulnerable women has contributed to reducing the gender disparity. As to the level of impact, given that gender-specific data collection was introduced in BAIS IV, it is not feasible to compare results at base and endline. Going forward, to further deepen the impact on vulnerable women, multisectoral mitigation interventions (e.g., livelihood skills, employment, etc.) and gender or poverty-based targeting (e.g. through conditional cash transfer) could be harnessed. The Bank s Botswana portfolio is active in both education and social protection. School-based interventions, targeted CCTs that reach vulnerable groups could reinforce health-sector efforts. (b) Institutional Change/Strengthening 90. BNAPS had substantial impact on institutional strengthening for NACA, including its management capacity through improved tools and processes, and the application of the PBF platform. Beyond strengthening the coordinating body, the Project had a broader effect on the participating line ministries at the central and decentralized levels. While many of the changes had catalytic effects, as discussed in Section 2.5 and Section 4, there are elements that face sustainability risks. Post-completion operations and next steps could focus on consolidating impact and minimizing institutional risk. (c) Other Unintended Outcomes and Impacts (positive or negative) N/A 45 The majority of the poor live in the South-East and North-East. Acute levels of poverty are found in more remote areas, prominently in the far North-West (Ngamiland) and in the west and southwest. Pockets of very high poverty are also found in the South-East, particularly in Kweneng and parts of the Central district (World Bank, 2015). 46 Larger households with more children have higher poverty rates. Families with both parents have lower poverty rates than single-parent families. With the world s 2 nd highest HIV/AIDS prevalence, there is a large number of AIDS deaths and orphans, and a sizeable number of incomplete families, which are more likely to be poor. 28

46 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 91. A BNAPS ICR Survey collected responses from representatives of the implementing agencies to provide an opportunity to share their views regarding the relevance, effectiveness, efficiency, and sustainability of the project. The survey was designed and collected through Survey Monkey, and represented a small sample size (22% of 105 targeted individuals). The distribution list for the participants was provided by NACA. The survey was not intended to be a representative survey with full coverage, given the time and budget limitations to implement the ICR. The objective of the survey was to offer an anonymous platform and an opportunity to share views. This was considered important due to the large number of implementers, spread across the country, and the multi-sectoral and decentralized implementation platform, which did not enable consultations with all implementing agencies at the various levels for this ICR review. The survey respondents had a largely favorable view of the BNAPS but noted some implementation challenges. Regarding effectiveness, 75% of respondents expressed satisfaction with reaching the agreed performance targets (see Annex 5 for more details). 4. Assessment of Risk to Development Outcome Rating: Substantial 92. Overall, the PAD s risk assessment was somewhat optimistic. A number of the high risk items materialized. The trend in the ratings indicates that although some of the risks were underestimated, particularly with respect to implementation during Phase 1, progress was made during Phase 2 (Table A10-6, Annex 10). Assuming that these improvements can be retained and form a strong foundation for future undertakings, the risk rating would be Moderate. However, to determine the final risk rating, two critical sustainability factors need to be taken into account. Given the expectations that (i) financial sustainability and (ii) the fluidity of institutional context could have significant bearing on the sustained and long-term development effectiveness of the Project, the proposed rating for development risk is Substantial. 93. Financial/Fiscal Sustainability. A number of documents and analyses (e.g., NDP11 and the Investment Case by NACA, 2015) emphasize the sustainability aspect of the national response. Addressing HIV/AIDS requires long-term financial commitments, at a time when tax revenues are expected to be under pressure due to slowing economic growth and diminishing fiscal contribution from diamond mining. Expectations of reduced donor funding and diminishing domestic fiscal resources will lead to more intense competition between the various claims on government spending, and hence prioritization becomes important. As a result of these broader macroeconomic and financial trends, combined with the increasing costs 47 of HIV/AIDS programs, a financing gap will emerge over the period to The lower bound is estimated to be approximately 500 million Pula (around 10% of the current health budget). 48 To boost the national response to HIV/AIDS in this context of increasing resource constraints, the strategic investment areas have been identified. 49 While there is a defined policy direction and a firm commitment to finding more efficient ways to operate, during the implementation of BNAPS there were some inefficiencies. The MOH and NACA both provided national funding to districts 47 The annual per person treatment costs at the end of 2014 were estimated at 4,320 Pula (US$480), expected to increase to 5,400 Pula (US$600) by 2030 due to increasing patients on 2 nd and 3 rd -line regimens (NACA, 2015). 48 1,000 million Pula is estimated annually to maintain current treatment eligibility and programs or the proposed expanded ART access and improved HIV prevention scenarios. 49 The 2014 Investment Case prioritized the following five areas as critical to realizing efficiencies, improving patients outcomes and ensuring the long-term financial sustainability of the HIV Response: (i) expanding HIV prevention with focus on adolescent, youth and key populations; (ii) improving M&E (including procurement and supply chain management); (iii) integrating health services to maintain clinical care excellence; (iv) expanding HIV testing, tracking and engagement in care; and (v) Community System Strengthening within the National HIV Response. BNAPS strategically contributed to these priority areas. 29

47 for programming needs and research. DPs may provide funds directly to either or both institutions. To ensure optimal use of resources, given the sustained multisectoral nature of the challenge, coordination of responsibilities and activities and transparent budget planning processes are important to avoid duplications, for example, by agreements on which institution is to receive, allocate and monitor particular funding of the HIV response. 94. Institutional Context and Sustainability. A critical factor that may affect the longevity and sustainability of development effects is the fluidity of the institutional context, specifically the possible effect of the ongoing institutional transformation within the health sector. As the national response has successfully managed the nation out of the emergency response mode -- particularly in terms of reduced mortality as a results of accessible and effective treatment -- the power relationship between NACA and the MOH started to change. This was first marked by the changes to the resource flow from 2010, the majority of which were channeled to the MOH leaving a residue although financially substantial but in relative terms a lighter share for NACA. The pressure for efficiency gains, partly through improved integration and coordination, further accentuated this trend. In 2014, the MOH began the implementation of the national health care system restructuring. Propelled by the efficiency agenda, a taskforce conducted a rationalization exercise to inform the restructuring. The integration of NACA into the MOH, now located in the Department of AIDS and Preventive Care, was recently confirmed (early 2016) and the transition is projected to take place over 2 years. The institutional change was motivated by the fact that NACA had fulfilled its mandate; the HIV/AIDS epidemic had been stabilized. Setting up NACA as a standalone entity was justified at the time of its establishment given the specific challenges facing the country then. As the MOH shifts toward a more integrated approach, bringing the institution under the MOH can be justified. However, there are concerns regarding the institutional transformation which can pose a threat to sustaining progress and thus, require attention as part of the change management strategy. First, the consolidation of the organizational structure and the transition management of NACA staff are critical. The government has committed to addressing the issue of duplication of roles. Further, strategic relocation would need to ensure the transfer of skills that had been built under Component 1 to benefit future HIV/AIDS programs. Second, while NACA was acknowledged for its responsiveness, ministries, in general, are known to be slower. The reorganization provides an opportunity to ensure that agility continues to be embraced. Third, the dissolution of NACA has raised questions regarding the coordination of the multisectoral response. Although the MOH has considerable comparative advantage in sector-level coordination and integration, the importance of cross-sectoral coordination cannot be underestimated, nor the necessity to connect to the district level multisectoral platform. HIV/AIDS remains a multisectoral challenge and, hence, attaining and sustaining development objectives can only be guaranteed on such a platform. The absence of a clear institutional link between a well-functioning and integrated health care platform and the multi-sectoral platform for HIV/AIDS response could jeopardize gains and lead to increased costs, including through the pathway of transaction costs, delayed or inadequate response. Given these arguments, it was raised that the integration of NACA into the MOH should be done in a way that contributes to health system strengthening; yet without losing sight of critical multisectoral contributors to HIV/AIDS outcomes, such as education interventions and efforts at the grassroots through CSOs, both of which performed well under BNAPS. Lastly, CSOs expressed reservations regarding the change and noted the lack of formal communication on the integration, as well as funding challenges. The reorganization offers an entry point for the MOH to take on the engagement model with civil society. However, this requires continued resource flows to support such engagement, which could be considered during the budgeting process. The reorganization creates a strategic opening to address these issues to ensure an environment for continued progress in responding to HIV/AIDS. 30

48 95. In sum, despite the operational and institutional challenges, BNAPS has left an important mark on HIV/AIDS control, as well as on the health sector. With respect to the sustainability of outcomes, BNAPS demonstrated its relevance and showed its potential in effectiveness. Although the restructuring created space for significant economic impact; the full benefits of the interventions are yet to be seen given the long-term impact trajectories of some of the interventions supported by the project (e.g. behavior change, care integration, and cervical cancer screening). Regarding financial sustainability, the results-oriented PBF and IBRD buy-down instruments catalyzed more efficient resource utilization. Furthermore, the buy-down which leveraged additional resources from the EU directly contributed to financial sustainability by lowering the cost of the IBRD loan. Lastly, BNAPS created a robust implementation platform for future engagements, including through the performance-based management platform used with implementers in the public sector and civil society. Building on this strong foundation, it is expected that the take-off for future operations would be significantly shorter, which would further improve development effectiveness and efficiency (i.e., value-for-money) ratings. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 96. As the first project in the Bank s Botswana portfolio and because of the IBRD buy-down innovation, BNAPS was a flagship project. The Bank closely worked with the Government to prepare a highly relevant operation. Project preparation was effective with respect to assessing the context and developing a design through a participatory approach that would contribute to strengthening management and coordination capacity of the national response. Given the unfamiliarity of the GoB with Bank operations and procedures at the time and the novelty of the financing instruments, capacity building was central to BNAPS success. The design was improved through QER recommendations. 97. The PAD was evidence-based and in-depth, reflecting robust technical capacity and pragmatism. However, there were some shortcomings, which in retrospect could have raised early warning flags. Despite the emergency modality, the preparatory time (between Concept Review and Appraisal) was 17 months, followed by a 3-month period from Appraisal to Approval. While the project was assessed as ready for implementation at approval, the long take-off path indicated otherwise. The pressure created by the context that this was a debut operation with Botswana and for the sector, the GoB was in an emergency mode with respect to the national response, and the IBRD buy-down was a novel mechanism hurried readiness. Readiness was affected by higher level systemic and structural impediments to effectiveness. Notwithstanding, BNAPS could have benefited from developing a sharper and more realistic PDO, a robust and timely RF, and laying out the details of the operational design and requisite implementation aspects for key performance drivers (e.g. PBF) to manage the components more effectively and efficiently from the start. 98. Lastly, although capacity constraints urged the team to prioritize, BNAPS may have benefited from an evaluation component as an integral part of the Project (or through complementary ASA) 50 to reinforce the importance of data management and utilization to inform design and implementation, as well as to improve accountability and boost the establishment of an evidence- and performance-based culture. 50 The QER recommended that BNAPS could integrate the findings of the economic analysis into the project, including a robust agenda of economic and sector work (e.g., resource tracking, cost-effectiveness analysis, and IE). 31

49 99. Overall, the Bank and the GoB designed an innovative project to respond to the situation in Botswana. Based on these strengths and limitations, the proposed rating for quality at entry is Moderately Satisfactory. (b) Quality of Supervision Rating: Moderately Satisfactory 100. Throughout supervision, the Bank team was proactive in identifying bottlenecks and proposing/finding pragmatic solutions. Given that the Bank did not have an active portfolio in the country and had not been active in Botswana for 20 years, the team dedicated a great deal of time to thoroughly explore the country and sector context to build the foundations for a solid relationship. BNAPS was formative in its relationship building as it laid the foundations for the engagement between Botswana and the Bank more broadly The missions were focused and systematic in their approach to understanding the strength, weaknesses, threats and opportunities. The project design and Year One plans suggest enthusiasm, ambition, and dedication. Following effectiveness, the team focused on the most critical sources of delays, such as issues related to the hiring of the requisite TAs, fine-tuning the M&E framework, and meeting the legal and technical conditions that were set to commence and intensify project disbursement. The ISRs were focused and reflected the nature of challenges accurately and in a problem-solving manner, including constructive comments from Bank Sector Management. The ISRs PDO ratings could have been more consistently assessed throughout project implementation as some of the ratings were not warranted based on the outcomes at the time. This would have required that the December 2011 restructuring involved a revision to the RF to introduce relevant new indicators to better link activities to outcomes. The Aide Memoires from the missions were well-written and actionable. The Bank team closely monitored activities and held frequent (weekly) teleconferences to engage in status updates. The team proactively responded to TA needs, including preparing frameworks, drafting TORs, and other documents that facilitated project management in a practical manner The team prepared an excellent MTR Issues Paper, which guided discussions during the MTR mission and the subsequent the Level-1 restructuring. While the restructuring led to greater absorptive capacity and improved results on the ground, it failed to align the PDO with the activities that received increased emphasis and the revised PDO lacked specificity. Despite the MTR s and EU s recommendations, the restructuring did introduce updated indicators of progress in the RF. Further, strengthened evidence-based for strategic decision-making was also lacking. In retrospect, it may have been worthwhile for the team to explore the feasibility and benefit of revision to the indicators and strengthening the evaluation agenda to better inform implementation, scale up and resource allocation decisions. Overall, the correction path between the MTR s findings and the level one restructuring took half a year, which delayed refocusing resources and accelerating progress The Borrower s feedback on the TA from the Bank was positive. However, there were concerns regarding processes, specifically delays with No Objections for procurements due to the Bank s arduous procurement requirements. The Bank had a number of quality enhancing recommendations for the CSO component, including the condition to contract a firm to implement external results verification to assess the effectiveness of the performance-based mechanism and, in general, accountability, measured through the frequency and accuracy of data flows on expenditures and outputs/outcomes. The TA that was provided to CSOs on management, hiring the community-based M&E expert for the CSO/PSO component, and other 51 WB Gaborone Office did not exist until the third year of project implementation and the first Country Partnership Strategy was presented to Board after the BNAPS Project was presented to the Board. 32

50 recommendations contributed to the timeliness and quality of implementation. As a result of the hands-on Bank support, by December 2013, implementation of project activities intensified, reaching a total disbursement of US$35 million. Performance was turned around from low to satisfactory and was maintained throughout. The Bank team watched out for occasional red flags, such as delays with audit and deterioration in the quality of financial reports, to sustain performance. The Government s request (July 2013) for a Closing Date extension and Loan expenditure category reallocation was approved, giving BNAPS another year to complete the cycle and meet the PDO by September The Bank team offered a number of practical tools (e.g., Gantt charts for project management) and had good recommendations to facilitate data flows, including the creation of a central data repository. Yet, due to capacity constraints and consequent prioritization pressures, the project could not fully harness the potential of data that could have accrued to BNAPS and the Government The Bank had a catalytic effect on broader outcomes, such as the integration agenda and HMIS harmonization. Further, with respect to client engagement, the Bank team built strong links and established good rapport with the Government. While during the implementation stage communication was frequent and effective, local presence of Bank staff in Botswana would have better facilitated project start-up, monitoring, and daily coordination (there was not a Bank office in Botswana at the time the project was designed and approved). Lastly, while the Bank team pursued the possibility of extending BNAPS through an AF mechanism to further strengthen well-performing initiatives supported by the project, this did not materialize In sum, although the corrective actions came with a delay, the successes during Phase 2 demonstrate the adaptability of the Bank team to address challenges in a practical and performance-oriented manner and their openness to policy-relevant changes to implement a complex and innovative operation. BNAPS fully disbursed and delivered more on a number of programmatic activities relative to the commitments outlined in the PAD, thereby generating additional benefits that were originally not foreseen. Thus, the proposed rating for the World Bank supervision is Moderately Satisfactory. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 107. The combined rating for the Bank, based on its performance on quality at entry and implementation supervision is Moderately Satisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 108. The GoB requested the BNAPS Project a proposed concessional IBRD operation in anticipation that the financial and technical resources mobilized through the World Bank would play a strategically significant role in supporting a more evidence-based response to the national prevention program. The Government, particularly NACA, the MOH, and the MFDP have been vital in shaping the content and design of BNAPS and the related processes to activate the loan. Although the GoB in general was committed and supportive to combating HIV/AIDS, as a means to improve health and economic outcomes, there were bottlenecks that contributed to the initial delays and the challenges during implementation. These were related to broader political processes (e.g., delayed Parliamentary approval) and structural issues (e.g., labor market context and HR constraints that arose from the hiring freeze of civil servants). Notwithstanding these shortcomings, BNAPS was moderately successfully delivered with important development impact, which span beyond its project horizon. Full disbursement was possible thanks to the two 33

51 closing date extensions, requested by the GoB, which demonstrated the Borrower s commitment and interest in the operation to harness its development potential. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 109. BNAPS had a complex implementation platform, with one primary implementing agency (NACA); seven line ministries and the Tertiary Education Council under the public sector component (Component 3); and 54 CSOs and private sector entities under Component 2. The rating is a composite, which reflects the effectiveness of NACA as the coordinator and also the dynamics with line ministries and the CSOs/PS The Government was open regarding the institutional limitations of NACA. This proactive self-critical assessment laid the foundations to design Component 1 of BNAPS, which focused on improving NACA s management and coordination capacity. A core recommendation and condition for the success of the project (provision of adequate HR to manage the project and to build capacity in the agency beyond BNAPS) was not met in a timely manner because of a combination of broader HR challenges beyond NACA as discussed above and HR planning, which led to implementation delays. TA quality standards at times were not met. The twinning arrangement, which was to attain longer term capacity building, did not materialize for the reasons discussed in the section on design (Section 3.1). Beyond periodic short-staffing on FM, M&E, procurement, and behavioral science, NACA management changed twice during the project, which caused discontinuity in approaches and implementation Institutionally, NACA had an uneven relationship with line ministries, as noted in the External Results Verification (ERV) report, which can be seen in lack of a standardized results monitoring framework and irregular reporting. The slow takeoff was driven by inadequate ownership (i.e., variability in the level of interest and commitment). Additionally, heterogeneity in the technical capacity of focal points, and the rate of attrition of BNAPS-affiliated staff contributed to differentials in outcomes. The intensity of implementation varied by component and increased in the later years of implementation. Performance-based reallocations within and across ministries aimed to contribute to optimizing resource use. However, the lack of consistent reporting and metrics made reallocation processes difficult. Lastly, despite the ambition of sustainability, BNAPS supported programs/positons that faced the risk of discontinuation, except for a few notable success stories, such as the national rollout of the HPV Vaccine on the EPI platform funded by the MFDP, or the prisons program that was scaled up and incorporated into core work plan of the MoDJS, including its HR and budget requirements. Other line ministries noted reassignment or loss of personnel. In sum, coordination, ownership, performance heterogeneity, and sustainability challenges were noted. However, the nature of these challenges suggests that NACA s performance was intertwined with the broader public sector performance context, and affected by economy or sector-level structural rigidities The civil society component was an important extension to build a platform for the longterm and sustainable response to the epidemic. The CSOs showed agility in implementation and were open to learn and benefit from operating on a PBF platform. NACA had a successful learning curve with the CSOs and the performance-based financing approach to manage grants. The management TA provided by NACA to CSOs was highly valued and served as an incubator for CSOs. Under BNAPS, CSOs gained considerable training in technical aspects related to designing HIV prevention interventions. There were shortcomings, however. While, in general, it was noted that while BNAPS had a strong impact on CSOs, they noted that there was limited involvement of umbrella organizations to more effectively manage the broad-based platform, 34

52 ensure standards, and represent interests. 52 Notwithstanding the sustainability challenges, which were also noted in the public sector, the engagement with civil society was ground breaking, including the scaled application of PBF, which introduced improved management and strengthened accountability. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 113. Overall, the GoB demonstrated openness to innovation and strategic shifts and implemented a complex operation which has long-term potentials, including through the engagement with civil society and the private sector. The proposed performance rating for the Borrower is Moderately Satisfactory. 6. Lessons Learned 114. The following lessons are drawn from the preparation and implementation of BNAPS The importance of intensive supervision in a country with limited prior WBG operational engagement and where there is no local staff must be underscored. Following a challenging early implementation period, the Project moved out of problem status as a result of the Team s intensive implementation support strategy, key elements of which were: (i) just-intime/additional missions to support implementation, in combination with regular supervision missions; (ii) frequent, problem-solving oriented engagement with GoB counterparts; and, (iii) following the MTR, close joint tracking of implementation progress with counterparts, using the improved Project management tools and External Results Verification to strengthen the robustness and reliability of self-reported data Innovative financing approaches require close coordination with the Client and cofinanciers, they can involve a trade-off, and require focus on results. The BNAPS team worked closely with the GoB and the EU to introduce the IBRD-buy-down innovation, agree on a monitoring framework and triggers for disbursement, and in monitoring the targets. Hence, beyond the financial benefits the buy-down also contributed to results monitoring. However, the project also shown that coordination requires substantial time and effort, particularly when novel instruments are introduced. When project preparation is urgent, project teams must assess the trade-off between being innovative and being timely Operations of this scale and with a number of innovative interventions should incorporate a robust evaluation agenda to better inform design, course correction, and scale-up decisions in a timely and operationally relevant manner. As the economic analysis illustrates, it was difficult to estimate effects and, particularly, efficiency, which are critical for resource allocation decisions and policy formulation Slow progress in behavioral prevention questions the efficacy and targeting of BCC intervention, and it may be an unexpected adverse side effect of Botswana s exemplary success in treatment. As sexual behavior is difficult to change, first, it is important to carefully choose and target preventive interventions. Second, investing in structural approaches remain critical to address the root causes of HIV transmission, in coordination with other line ministries, to attain medium to long-run disease control and development targets. The Bank s Botswana portfolio is engaged in the social protection and education sectors. Targeted CCTs that reach 52 Umbrella organizations did not play a major role in BNAPS given the results from capacity assessment. 35

53 vulnerable groups (e.g., gender-based targeting) and school-based interventions with proven efficacy could further reinforce health-sector efforts In order to prioritize and concentrate allocations on cost-effective interventions for HIV prevention, it is critical to collect data. Data scarcity or inadequate use of data undermines strategic change and innovation. Given the multiplicity of interventions that may not lead to expected changes in behavior required to arrest the spread of the HIV, as argued above, more focus on interventions with proven effectiveness and cost-effectiveness would ensure improved allocative efficiency and resource use and, therefore, contribute to the sustainability agenda. The importance of the effectiveness and cost-effectiveness criteria for intervention rollout and resource allocation decisions underscores the relevance and role of further data and evidence generation (economic evaluation) when testing alternative prevention and service delivery modalities. For evaluation, a key condition is good M&E design that provides data on a routine basis for rapid assessment and action Through the integration pilots, BNAPS illustrates that there is added value in capitalizing on the shared challenges, drivers, and potential solutions across the different disease categories (CDs and NCDs), with broader health-systems strengthening measures and outcomes. The key lesson from the TB and HIV/AIDS and cervical cancer and HIV/AIDS integration pilots is that given resource constraints and some shared determinants, there is scope for an integrated approach, focusing on functions (prevention, treatment, and care) rather than on disease categories. The experience with the integration of cervical cancer control within the HIV/AIDS service delivery platform illustrates the opportunities and feasibility to integrate NCD prevention and treatment into existing services and programs for CDs. There are not only many links between these diseases, but shared underlying social conditions, interacting co-morbidities, as well as common solutions such as vaccination, standardized syndromic protocols, and care models. In resource-limited countries, leveraging existing HIV/AIDS platforms and investments to expand services for other priority, and often unmet health needs, is doable and necessary to maximize the utilization of existing resources and improve the effectiveness of service delivery. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 121. The Government of Botswana provided specific comments on the ICR draft which are summarized in Annex 7. The comments that were received from the Ministry of Finance and Development Planning were addressed. Two out of the three comments received from the Ministry of Health were also addressed (comments 1 and 3). The comment regarding the HAART indicator is noted. However, this remained unchanged since this is how Indicator 5 appeared in the Results Framework of the Project Appraisal Document (PAD) and traced and reported on during the project cycle. The NACA endorsed the content of the report. (b) Co-financiers 122. The EU Delegation to Botswana reviewed the draft ICR and provided important information for this ICR Report as well as valuable exchanges that enabled the World Bank team to reflect the EU s view, particularly with respect to the IBRD buy-down. 36

54 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Estimate Actual Expenditures Project Cost by Component and/or Activity Projected Total (US$) Projected Shares (%) Actual/Latest Estimate (US$) (%) of Appraisal Component 1: NACA $7,200,000 15% $6,390, % Component 2: Public Sector $19,200,000 40% $16,790, % Component 3: CSO & Priv. S. $21,600,000 45% $25,049, %.7 Total Baseline Cost $48,000,000 $48,230,561 Physical Contingencies $2,000,000 Price Contingencies Total Project Costs 1 $50,000,000 $50,000, % Total Financing Required 1 Note: Including taxes of US$ 1.7 million (3.5% of total financing). (b) Financing Source of Funds Type of Co-financing Appraisal Estimate (US$ millions) Actual/Latest Estimate (US$ millions) Borrower International Bank for Reconstruction and Development Percentage of Appraisal Figure A1-1: BNAPS expenditure by component including direct payment (FY ) Millions BNAPS Expenditures by Component (FY09 15) FY09 10 FY10 11 FY11 12 FY12 13 FY13 14 FY14 15 C1 NACA C2 Public Sector C3 CSO TEC Total (inc. DP) Source: Project financial reports, including Gantt Charts. 37

55 Figure A1-2. Panel A: BNAPS Component 2 expenditure by line ministry in US$ (FY ). Panel B: Same in expenditure share term (%) Source: BNAPS Project Data (Project Management and Monitoring Gantt Charts) Table A1-1: CSO allocations Top 10 grant recipients Rank Health_District CSO Q1_2013/14 through Q2_2014/15 US$ Equivalent (BNAPS EXR) US$ Equivalent (Midpoint EXR ) Share of Total Cumulative Share 1 Kweneng East Makgabaneng 14,073,237 $ 1,761,356 $ 1,599, % 14.0% 2 South East DEE ZONE 10,580,760 $ 1,324,250 $ 1,202, % 24.6% 3 Selebi Phikwe Humana 5,000,933 $ 625,899 $ 568, % 29.6% 4 Palapye BBCA 4,829,144 $ 604,398 $ 548, % 34.4% 5 South East AMEST 2,839,531 $ 355,386 $ 322, % 37.3% 6 Palapye BCC 2,479,148 $ 310,281 $ 281, % 39.7% 7 Kweneng East Tse Dikgolo 2,110,362 $ 264,125 $ 239, % 41.8% 8 Goodhope Baylor 2,020,364 $ 252,862 $ 229, % 43.9% 9 South East OTSE HBC 1,982,533 $ 248,127 $ 225, % 45.8% 10 Goodhope Stepping Stones 1,978,917 $ 247,674 $ 224, % 47.8% Box A1-1: Top 10 Grant Recipients CSO & Private Sector (>50% of total funding under Component 3) 1. Makgabaneng, Kweneng East (Abstinence, Condom, MCP & SMC). Makgabaneng brought years of experience in the use of media in HIV prevention and behavioral change through radio serial drama. They used TV drama to communicate HIV prevention messages and published teen magazines to engage children and youth in school clubs for BC. The strength of this program was its potential to reach the entire country. 2. DeeZone, South East (Abstinence, Condom, MCP & SMC). DeeZone developed docu-drama TV series. The comparative advantage of DeeZone was of the viewership of the entire country. Their messages were reinforced in communities in the 9 districts and selected communities in the country. 3. Humana People to People, Selebi-Phikwe (Abstinence, Condom, MCP & SMC). Humana had extensive experience in community mobilization using the house to house campaigns. The same strength was utilized in 7 out of the 9 BNAPS districts to mobilize the target groups on Abstinence, CCC, MCP and SMC. 38

56 4. BBCA, Palapye (Abstinence, Condom, MCP & SMC). BBCA mobilized the private sector in Botswana to mainstream HIV/ AIDS in the work place. They targeted two important districts (Palapye and Francistown) which had seen rapid population expansion because of industrialization. 5. African Methodist Episcopal Services Trust (AMEST), South East (Abstinence & MCP). AMEST mobilized communities to address MCP and targeted local schools in South East for abstinence. They have over the years developed and harnessed effective methodologies relevant for the local communities. 6. Botswana Council of Churches (BCC), Palapye (Abstinence & MCP). BCC brought the faith based element into BNAPS. They had a very large membership of churches around the country including in BNAPS districts. They explored the same to reach their membership in Palapye with abstinence and MCP with over 50 churches reached. 7. Tse Dikgolo, Kweneng East (Abstinence & MCP). Tse Dikgolo used theatre and follow-up reinforcement activities in Mankgodi. Their strength was Community Theater for behavior change. 8. Botswana Baylor Children's Clinical Centre of Excellence, Goodhope (Abstinence, Condom & MCP). Teen Club targeted unique and sensitive sub-group-hiv positive adolescents with abstinence and MCP messages. The only group with a sustainable intervention for the vertical transmission of HIV among youth. Baylor also implemented the in- and out-reach program, with a focus on improving adherence in homes and enhancing the capacity of medical practitioners in hospitals on HIV. 9. Otse Home Based Care, South East (Abstinence & MCP). Otse HBC had experience in palliative care. They integrated HIV prevention to their program. The organization boasts of highly qualified and experienced staff, Peer Educators who were equally highly trained and experienced in community mobilization as well as a good working partnership with the local community built over the years. 10. Stepping Stones International, Goodhope (Abstinence, Condom, MCP & SMC). SSI s strength was life skills building amongst in-school youth who reached out to their peers. They used innovative strategies ( aunties and uncles ) who are not paid; potentially sustainable beyond BNAPS funding. 39

57 Annex 2. Outputs by Component COMPONENT 1: NACA CAPACITY BUILDING Table A2-1: NACA Capacity Building ITEM PLANNED/INPUT ACHIEVEMENT/OUTPUT Team Leader 1 3 Behavior Scientist 1 1 Financial Management (FM) 2 3 Consultants Procurement Consultants 2 3 Monitoring and Evaluation (M&E) 2 4 Consultants NGO TA Management Consultant 1 Achieved World University Services of Canada (WUSC) provided the training to CSOs NGO TA technical 1 Not Achieved External Results Verification (ERV) 7 cycles of verification 4 cycles in 2012/13 3 cycles in 2013/14 Achieved Database Development Database Manager Not Achieved Table A2-2: NACA TA No. BNAPS TA BNAPS TA Team Leader Team Leader 2 Senior Capacity Building Consultant Senior Capacity Building Consultant 3 Senior M&E Consultant Senior M&E Consultant 4 Senior Procurement Consultant I Senior Procurement Consultant I 5 Senior Procurement Consultant II Senior Procurement Consultant II 6 Senior Financial Management Consultant I Senior Financial Management Consultant I 7 Senior Financial Management Consultant II Senior Financial Management Consultant II No. BNAPS TA BNAPS TA Team Leader Team Leader/Behavior Change Specialist 2 Senior M&E Consultant Senior M&E Consultant 3 M&E Specialist Community-Based M&E Specialist Community-Based 4 M&E Officer M&E Officer 5 Senior Procurement Consultant II Senior Procurement Consultant 6 Senior Procurement Consultant 7 Senior FM Consultant II Senior Financial Management Consultant 8 Senior Financial Management Consultant No. BNAPS TA 2014 October March Team Leader/Behavior Change Specialist 2 Senior Financial Management Consultant 3 Senior Procurement Consultant 4 Senior M&E Consultant 53 The PAD specified 8 TA positions: Senior Management and Implementation Specialist; 2 Senior Financial Management Specialists; 2 Senior Procurement Specialists; Senior Monitoring and Evaluation specialist; Senior Capacity Building Specialist; Senior Strategic Planning and Partnerships Specialist 40

58 COMPONENT 2: PUBLIC SECTOR OUTCOME SUMMARY Ministries focused on behavioral change activities targeting their staff and clients (e.g., MLG: community level clientele and out-of-school youths; MLHA: prisons staff and prisoners; MOE: teachers and in-school youths; MYSC: staff and out-of-school youths; MWT: staff and the mobile populations at high-risk; MOH: HIV+ patients receiving ART and on providing TA to participating ministries). Table A2-3: Public sector projects, areas of focus PROJECT NAME AREAS OF FOCUS/KEY STRATEGIES KEY ACHIEVEMENTS 1 Ministry of Education and Skills Development (MOESD) Including Tertiary Education Council (TEC) In-school Youth (primary, secondary and tertiary levels), Teachers /counselling on air, communities and on campuses/ training of teachers in paediatric HIV. All teachers in the country acquired paediatric HIV training. On average 2,600 youth a month nationwide received counselling on life issues. 43 institutions had campus-based Counsellors and counselling reaching about 1,294 males a month and 1,173 females a month. 2 Ministry of Defence, Justice and Security (MDJS) 3 Ministry of Transport and Communication (MTC) 4 Ministry of Infrastructure, Science and Technology (MIST) HIV education and communication, counselling and testing, ART, and SMC at 2 prisons; Gaborone and Francistown Truck drivers at highway stops, public transport drivers stations, government drivers on out-of station assignments, communities and internal MTC staff Mobile population at construction sites in 7 districts, localities and communities within 10km radius of target population and internal MIST staff HIV Educational campaigns and Condom distribution 5 Ministry of Youth, Sport and Culture (MYSC) via sport and edutainment 6 Ministry of Health (MOH) SMC, TB/HIV, cervical cancer prevention among girls, research (STEPS Study), HMIS through networking equipment, refurbishing of drug warehouses and a hospital, TB wards, provision of diagnostic equipment and mobile vans for SMC, funding of Pharmacy students 7 Ministry of Local Government and Rural Development (MLGRD) HIV psychosocial support to children and skills for parents, community leaders/community conversation. M&E and CCEP Officers Availability of HIV education, counselling and testing services to inmates and surrounding neighbours reaching about 638 people a quarter. 13 SMCs a month on site. Expansion of BNAPS model to other prisons Capacity for mainstreaming BNAPS model nationwide. Capacity for Wellness committees, men and women sector groups to sustain BNAPS model Capacity to replicate BNAPS model in all districts 5 Mobile trucks for SRH/HIV integration, improved accessibility for SMC, improved diagnostics for TB and NCDs, HPV vaccination of 2,000 girls, trained 36 pharmacy undergraduates, drug availability, improved forecasting and order turnaround time from one month to 3 days. 72 functional IDCCs, approximately 41,105 patients reached Capacity to replicate BNAPS model in all districts 41

59 Table A2-4: Inputs, Outputs, and Outcomes - Ministry of Education and Skills Development (MOESD) PROJECT/ACTIVITY PLANNED PURPOSE/ INPUTS ACHIEVEMENT/OUTPUT MOESD Projects YOUTH COUNSELING ON 8 counsellors, AIR (YOCA) 4 sets of call center communication 12 hours Toll free access by all equipment, youth (in-school and out-ofschool), to HIV counselling on Marketing of YOCA services on local media Hiring of private premises for the project; air. (TV, Newspapers, Radio, Billboards); Procured laptop and printers and multimedia projector for YOCA & and M & E Officer; Call Centre Management Course for two officers in South Africa; Teacher Training HIV Pediatric The Voice Study Multidisciplinary Counseling Centers (MDCC) To provide education on pediatric HIV/AIDS to school teachers who have a direct contact with children in all primary and secondary schools by June 2014 IN- REACH AND OUT- REACH To capture the voice of vertically transmitted HIV positive youth through research Consultancy to Baylor children clinical center of excellence to train teachers to provide psychosocial support to HIV positive children in the classroom, and support adherence to medication. Baylor s Mentorship and didactic training for local service providers to improve ART in outreach sites and homes OUTCOMES On average 2600 youth counselled on air per month. About 87% call center performance. Major topics were relationships, adolescent and intergenerational sexual unions, multiple and concurrent partnerships, alcohol and drug use. Country-wide training of all teachers in primary and secondary schools in pediatric HIV schools were targeted and 808 were reached. 1,016 teachers were targeted to be trained but 834 were trained; 21 difficult to reach out reach sites were covered and 70 home visits a month; Trained primary and secondary school teachers on Pediatric HIV and AIDS Education and Care; Trained project personnel in ICT/SPSS and EPI-info Printed Voice Study material; Disseminated the VOICE Study results to key stakeholders and schools (MOESD and Baylor) 4 MDCC; MDCCs in 4 Community Junior secondary schools; Tsienyane, Lentsweletau, Palapye and Mabutsane. Youth Camps Life skills based Counsellors, Boot camp facilities Coordinator, Research Officer, Psychologist, Counsellor and Nurse for each center No. counselled a month 2 camps a year : 150 children per session from around the country 42

60 Department of Technical Vocational Education and Training (DTVET) Campus Based HIV Education and Counseling Silent Shout Tertiary Education Council (TEC) To provide knowledge and skills on HIV and AIDS prevention among students and staff in 45 DTVET institutions by June 2014, Targeting 8000 students and 1400 staff population SILENT SHOUT TV production campus-based HIV education and Counselling targeting students from 49 institutions 32 DTVET institutions ( 7 technical Colleges and 26 Brigades) had campusbased counsellors, HIV specialist, 32 counsellors; Purchase and distribute IEC materials (Condom models) to 26 DTVET institutions; Train 26 Counsellors on HIV&AIDS Peer Education; Educational campaigns in 6 DTVET institutions to promote HIV Testing, SMC, Abstinence, alcohol and drug abuse, condom use and partner reduction among students and staff Youth Forums (young people affected by HIV & AIDS and related challenges); once a week airing of Teachers, children and community views on HIV; Conduct 2 quarterly focus group discussions as a follow up of issues o 1,011 schools targeted o 1,016 teachers targeted 10 counsellors, 300 Peer Educators; sensitization workshops for campus health focal persons (50 people); Procured 10 Porta-cabins for NEW Campus Health Centers, furniture and accessories for CHCs (desk, computers, 2 chairs, office chair, filing cabinet, book shelf and camera) On average, 1,234 students and staff were reached per month 32 qualified counsellors, one per each institution. 808 schools reached 834 teachers reached 10 clusters of TEC institutions received counselling on campuses. On average, 13,451 are reached in a quarter for SMC demand creation, CCC, Abstinence, MCP and counselling. Highest no. of condoms distributed was 37,736 in a quarter. 43

61 Table A2-5: Inputs, Outputs, and Outcomes - Ministry of Health (MOH) PROJECTS PLANNED ACHIEVED OUTCOMES TB/HIV 2 coordinators, 13 wards to 2 Coordinators, and computers; Patient defaulter rate reduced from 4.4% to 2% in most be renovated. Community Two M&E training meetings per year (co-funded districts; level by MOH & PEPFAR) Enhanced care detection (2010= 71%; 2013= 82%); TB medication adherence 2 wards renovated Notification rate (2010= 503/100,000; in all districts Jubilee Ward fully refurbished and equipped 2013=331/100,000); (construction co-funded by Global Fund) Treat success (2010=81%; 2013= 82%); 70,552 health workers trained in DOTS & TB-HIV Co-infection rate (2010= 64%; 2013= 60%); co-infection treatment; Increased integration in DOTS and ART uptake for 5,268 community health workers and non-health HIV-TB co-infections: volunteers trained to support TB patients in 17 o ART uptake (2010=53%; 2013=75%); districts to adhere to medication and avert opportunistic infections. o TB integration = 670 health facilities (including health posts); o ART integration = 500 health facilities. Diagnostic equipment 20 Gene Xpert and tissue processing machines Testing (2010= 81%; 2013=92%) operating in District Hospitals and Clinics; Reduced turnaround time from 72 hours to 3 hours 2 Lab techs trained in South Africa; MDR test reduced turnaround time from 8 weeks to 3 Trained laboratory staff for quality assurance; hours Patient lost reduced due to early death averted, or before patient moved Infection Disease Control Clinics (IDCC s) Safe Male Circumcision (SMC) 72 IDCC out of functional IDCCs. Acquisition of PCs, peripherals, networking equipment for 60 IDCCs (Cabling & Networking of the Greater Francistown, North East,Bobirwa, DHMTs). Procurement of furniture, refrigerators and medical equipment for 40 ARV Clinics 3 mobile vans 5 mobile vans were purchased to improve accessibility of clients for SMC Procurement of male circumcision kits and outdoor advertising materials SMC TV advertising campaign Staffing: 1 physician and 8 nurses Improved patient tracking, and reduced non-compliant ARV rates Improved HIV, TB, SRH, STIs, Malaria, ARV and other registries Rollout of IDCC and coverage Three mobile clinics in Gaborone and two mobiles providing outreach in other health districts (March-Dec 2014 = 10,000) Demand creation (from 12% to 24% increase BAIS- IV) 2014 Target = 50,000 44

62 National Cervical Cancer Prevention Program (NCCPP) PROJECTS HPV- CERVICAL CANCER Pilot Target = 2000 girls Training in 3 districts Demand creation support with CSOs Component 3- BNAPS Logistics for HPV vaccine demonstration project Consultancy on HPV Vaccine national rollout plan In February 2015 MOH launched the national HPV vaccination roll out with the first dose. The second dose is planned for September Achieved = 32,527 (65%) Integration of outreach with mobile units shared between SMC and Cervical Cancer and other SRH Programs; SMCs done since inception = 150,136 Achieved = 1,967 from 2,488 eligible girls 9 years and older (79%) HPV Vaccine Demonstration transitioned to EPI to allow for budgeting and sustainability. NCD Programs Pharmacist Training Support See and Treat piloting in 9 Clinics Networking equipment to link see and treat clinics to IMPS Hospital Wide Area Network Purchase of automated histology equipment to address human resource capacity challenges at the national lab. STEPS SURVEY & CANCER REGISTRY 42 Pharmacists trained and graduated; Addition of two more See and Treat sites at Kanye, Molepolole as well as mobile sites at Kang and Kasane. Baseline training in Zambia for 15 additional See and Treat providers Support for IT infrastructure at the nine See and Treat sites to enhance program monitoring and evaluation Computerization and networking of greater Gaborone and Francistown clinics to facilitate access to IPMS and improve medical information storage and retrieval Purchase and installation of laboratory equipment to enhance histology services 150 health workers trained with WHO support CanReg-4 being updated to CanReg-5 Formal training in CanReg-5 and IPMS for Cancer Registry staff 36 remained, 6 dropped off due to medical and academic reasons; 9 See and Treat clinics operational in 5 Health Districts to be expanded to 16 clinics in 8 new Districts by the end of 2015 Integration of outreach with mobile units shared between SMC and NCCPP NCCPP now fully funded by MOH Lab results available to clinics from District hospitals through IPMS with reduced turnaround time; IPMS available in clinics by July, 2015 will reduce lab reports turnaround from 3 weeks to 2 weeks for negative specimens) Smart automation has reduced Pap smears turnaround time of specimen from 6 months to 6 weeks (2 weeks for negative specimens). 6 weeks delay in turnaround times due to lack of pathologist to read the slides with positive results. Target = 6,400 age Achieved = 63% Report in process CanReg-5 in process graduated; o 2 from University of Western Cape (UWC), 45

63 Drug Warehouse refurbishing 3 Drug Warehouse refurbishing will graduate; UWC (4), NMMU (2) Rhodes (1) 3 refurbished improved drug stock forecasting and management o 12 from Nelson Mandela Metro University (NMMU) graduated ; UWC (1), NMMU (5), National University of Lesotho (10) 30 Pharmacists placed at drug stores around the country to improve quality dispensing Efficiency gain: Reduction in overall cost due to averting drugs expiration Table A2-6: Inputs, Outputs, and Outcomes - Ministry of Local Government and Rural Development (MoLGRD) PROJECTS PLANNED ACHIEVED OUTCOMES M&E and Community Conversation Enhancement 30 8 M&E Officers and 9 CCEP Officers supporting the national response MOH staff at the DHMTs have assumed M&E functions from Local Government; Program (CCEP) Officers Support Improved data recording on national response; Vulnerable Children & 960 a quarter 3,200 children a year: 3,200 children received yearly HIV Children in Difficult Develop and distribute child friendly HIV/AIDS prevention life skills Circumstances - HIV IEC materials (Comic Books); 1,280 parents a year trained on parental Education Conduct health talk sessions on HIV/AIDS, abstinence, correct use of condoms, SMC, gender based violence and intergenerational sex; skills in psychosocial support to children Conduct life skills camps to promote healthy life choices and make referrals to appropriate service providers; Benchmarking trip to RSA to learn about the best practices in HIV/AIDS programs for children living in difficult circumstances 46

64 Table A2-7: Inputs, Outputs, and Outcomes - Ministry of Transport and Communications (MTC) PROJECTS PLANNED ACHIEVED OUTCOMES HIV education, counseling and testing. Provision of condoms to mobile population in 3 locations: Kasane, Francistown and Palapye 3 project coordinators, 3 drivers, service provision by Tebelopele (counselling and testing and SMC referral, condom usage demonstration and distribution) Voluntary counselling and testing (VCT) and education by contracted CSOs (Includes MIST support); Formulated, designed and distributed MT&C in-house IEC material (designed and produced); Procured Porta cabin for Kazungula and connect utilities Mobile Population Project (MPP); Trained 3 coordinators VCT, Project Management, Health and Safety, Data Collection and Management Processes (SPSS) Table A2-8: Inputs, Outputs, and Outcomes - Ministry of Infrastructure Science and Technology (MIST) PROJECTS PLANNED ACHIEVED OUTCOMES Reaching internal and external MIST audience with HIV education. Communication and These were achieved in coordination with MTC. HIV Service delivery was provided by Tebelopele (with MTC) services via Wellness Conduct 3 awareness campaigns on key prevention Committees and Men and strategies for construction workers Women Sector Committees Construction sites in Mahalapye, Bobonong, Lobatse Old Naledi, Francistown Ghantsi, Molepolole, and Palapye. Tebelopele to provide counselling and testing, referral services and condoms Table A2-9: Inputs, Outputs, and Outcomes - Ministry of Youth Sport and Culture (MYSC) PROJECTS PLANNED ACHIEVED OUTCOMES Condom Distribution 5,000 a quarter for male and 1,000 for female No. of Sport Events for HIV Education (AIDS SWAGGAs) Sector coordinators trained in Project Management 500,000 sector specific branded condoms designed and procured; Developed and distributed IEC packages: 20,000 Pocket media, Book folders 20,000 and 20,000 Key rings; Conduct Youth Dialogues, Sport festivals, Arts performances; 2 regions 861 is the cumulative highest number of young people reached in a quarter NGO Tebelopele contracted to provide VCT and other HIV services - highest no. of condoms per quarter 10,960 male and 1,030 female; Increased VCT & condom distribution in mobile population; Wellness groups and sector committees were trained and have capacity to sustain the model 3,665 male condoms monthly average Outcomes in BAIS-IV 47

65 Table A2-10: Inputs, Outputs, and Outcomes - Ministry of Defense Justice and Security (MDJS) PROJECTS PLANNED ACHIEVED OUTCOMES Village Prison 1 Infectious disease control 1 IDCC fully operational Staff absorbed by government. SMC clinic IDCC Procurement of additional medical Replication of BNAPS model in other HIV Education and 16 (2 doctors, 2 nurses, 2 Equipment and furniture prisons Counseling counsellors, 2 Health care Procure fencing material All qualified patients under ARV and Technical Staff Assistants, 2 data clerks, 2 Full complement of staff achieved. PMTCT (about 60 patients) accessing Others drivers, 2 cleaners, 2 Training of Prison staff on M&E & ARV therapy Pharmacists dispensing, and HIV counselling VCT and HIV services provided extends to 25 SMCs targeted 300 inmates reached for HIV education. people living outside the prison location Maximum of 13 SMCs in a month. Francistown Prison 1 Infectious disease control 1 IDCC fully operational Project Staff absorbed by government. SMC clinic IDCC Procurement of additional medical Replication of BNAPS model in other HIV Education and 16 (2 doctors, 2 nurses, 2 Equipment and furniture prisons in the country Counseling counsellors, 2 Health care Procure fencing material All qualified patients under ARV and Technical Staff Assistants, 2 data clerks, 2 Training of Prison staff on M&E & ARV PMTCT (about 83 patients) accessing Counseling & Testing drivers, 2 cleaners, 2 dispensing, and HIV counselling therapy Pharmacists Quarterly target of 25 SMCs Full complement of staff achieved. VCT and HIV services provided extends to 338 inmates reached for HIV education. people living outside the prison location. Maximum of 13 SMCs in a month. Inmates have services within the walls of the prison. 48

66 Table A2-11: BNAPS supported activities for TB/HIV care integration BNTP Strategic Objectives Enhance and expand high quality DOTS services: - Improving TB case notification rates and DOTS coverage. - Improving case detection through quality assured bacteriology. - Improving standardized treatment through supervised treatment and patient support - Monitoring and Evaluation. Addressing TB/HIV, Drug Resistant TB and other challenges: - Strengthening the health care system to respond effectively to the TB/HIV epidemic. - Decrease the burden of TB among people living with HIV. - Implement TB infection control in all settings - Decrease the burden of HIV among people with TB BNTP Challenges Supported 1. Inadequate Human Resource capacity for the TB control. 2. Inadequate formal TB/HIV collaboration at national and district level. 3. Inadequate role of Laboratory in diagnosis and followup of treatment deficit. 4. Inadequate supervision. 5. Inadequate Monitoring (poor recording and reporting). BNAPS Indicators Funds Used (BWP) Funds Used (USD) Payment of salaries and gratuity for the two Regional TB/HIV Coordinators 436,112 $ 54,582 3% Conduct quarterly Mentoring and Support Supervision visits to the DHMT s 168,000 $ 21,026 1% Bi- annual national TB/HIV review workshops. 386,856 $ 48,418 2% Procurement of ICT (2 laptops, 2 printers and 2 LCD Projectors) for TB/HIV Coordinators. 84,560 $ 10,583 1% Complete Phase 2 Renovations of Jubilee MDR TB Isolation Ward (Co- funded with Global Fund Grant) 4,000,000 $ 500,626 24% Procurement of equipment for Jubilee MDR TB Isolation Ward. 5,000,000 $ 625,782 30% Training of Health Care Workers (HCW) on TB/HIV case management 1,177,340 $ 147,352 7% Training Lay Counsellors / Health Education Assistants (HEA) on Community TB /HIV care. 800,000 $ 100,125 5% Training of non-hcw on Community TB/HIV care. 878,295 $ 109,924 5% Train two laboratory Scientists per year on second line culture and DST in RSA 58,744 $ 7,352 0% Procurement of 20 GeneXpert equipment and accessories (TB rapid test) equipment, installation and consumables. 3,336,589 $ 417,596 20% Minor renovations on 12 Health Facilities to strengthen TB infection control. 173,858 $ 21,759 1% Training of 15 laboratory staff per quarter on External Quality Assurance (EQA), for LED Microscopy. 313,37 $ 39,220 2% Share (%) $ 2,104, % 49

67 Table A2-12: Output and results for TB program under the MOH Component 2 STRATEGIC AIM AND RESPECTIVE ACTIVITIES TB/HIV PROJECTS SUPPORTED BY BNAPS Aims Indicator 1. Enhance and expand high quality DOTS services Improving TB case notification rates and DOTS coverage by intensifying case finding, enhancing infection control, scaling up access to isoniazid preventive therapy and ARVs, as well as involving communities in TB care. Improving case detection through quality assured bacteriology by training more laboratory staff, and reducing turnaround time for diagnosis and initiating treatment of TB (including MDR-TB). Improving standardized treatment through supervised treatment and patient support: this was achieved by improving available resources for TB control as well as training, mentoring and supporting healthcare workers and support staff. Improving the quality of monitoring and evaluation by promoting better recording and reporting of cases, and organizing numerous TB/HIV meetings and workshops. 2. Addressing TB/HIV, drug resistant TB and other challenges TB notification rate Measure Of Success Baseline value (2007/2008) 536 per 100,000 population End of project value (2013) 337 per 100,000 population TB case detection rate 69% 75% (2012) Comment 37% decline following universal access to ARV. 3 percentage point increase MDR TB cases that started treatment % increase MDR-TB cases that died before treatment initiation % decrease Patients confirmed as MDR-TB cases but lost to follow-up % decrease Patients treated as MDR-TB suspects % decrease Specimen (sputum) result turnaround 72 hours 3 hours time 3 weeks 24 hours 21-fold decrease Time to diagnose drug resistance 5 patients over 17 patients over 5 months 5 months 240% increase Time to initiate MDR-TB treatment At least 8 Dramatic 3 hours weeks Decrease Treatment success among smear positive TB cases - 71% 82% (2012) 15.5% increase. 50

68 Strengthening the health care system to respond effectively to the TB/HIV epidemic by improving and expanding coordination and collaborations for TB/HIV activities. Decreasing the burden of TB among people living with HIV by screening them for TB, regularly assessing the capacity of healthcare facilities for TB control, and renovating clinics that focus on TB and MDR-TB treatment. Implementing TB infection control in all settings by social mobilization, community sensitization, promoting behavioural change, and scaling up community based TB care. Decreasing the burden of HIV among people with TB by increasing their access to condoms, HIV testing, ARV and CPT, addressing the needs of vulnerable populations (including healthcare workers), engaging people with TB and sensitizing work places to TB and HIV. - - Community TB care enrollment 9% (2009) 70% 672% increase Known HIV status among TB cases 68% 92% 35% increase HIV/TB rates 64% 60% 6% decrease ARV uptake 20% 75% 275% increase CPT uptake 32% 95% 197% increase 51

69 COMPONENT 3: CSOs Table A2-13: Summary of key CSO outputs with scaled coverage PROJECT/ACTIVITY PLANNED PURPOSE/ INPUTS ACHIEVEMENT/OUTPUT Youth Counseling on Air (YOCA) 12 hours toll free access by all youth (in-school and out-of-school), to HIV counselling on air. 8 counsellors, 4 sets of call center communication equipment, hiring of private premises for the project. On average 2,600 youth counselled on air per month. About 87% call center performance. Major topics were relationships, adolescent and intergenerational sexual unions, multiple and concurrent partnerships, alcohol and drug use. Teacher Training HIV Pediatric In-reach & Out-reach Consultancy to Baylor children clinical center of excellence to train teachers to provide psychosocial support to HIV positive children in the classroom, and support adherence to medication. Baylor s Mentorship and didactic training for local service providers to improve ART in outreach sites and homes. Country-wide training of all teachers in primary and secondary schools in pediatric HIV schools were targeted and 808 were reached teachers were targeted to be trained but 834 were trained. 21 difficult-to-reach out reach sites were covered and 70 home visits a month The Voice Study To capture the voice of vertically transmitted HIV positive youth Produced and disseminated through research. Multidisciplinary Counseling Centers (MDCC) 4 MDCCs, Coordinator, Research Officer, Psychologist, Counsellor and Nurse for each center. MDCCs in 4 Community Junior secondary schools; Tsienyane, Lentsweletau, Palapye and Mabutsane. Youth Camps: Life skills based Counsellors, camp facilities 2 camps a year : 150 children per session from around the country Department of Technical Vocational Education and Training (DTVET) Campus-based HIV Education and Counselling 45 DTVET institutions, 45 qualified counsellors one per each institution. Targeting 8000 students and 1400 staff population. 32 DTVET institutions (7 technical Colleges and 26 Brigades) had campus-based counsellors, HIV specialist, 32 counsellors, On average, 1,234 students and staff were reached per month. Silent Shout Tertiary Education Council (TEC) - Campus-based HIV Education and Counselling TV production 1011 schools targeted 1016 teachers targeted 10 counsellors, 300 peer educators targeting 51,000 students from 49 institutions. Once a week airing of teachers, children and community views on HIV. 808 schools and 834 teachers reached. 10 clusters of TEC institutions received counselling on campuses. On average, 13,451 are reached in a quarter for SMC demand creation, CCC, Abstinence, MCP and counselling. Highest number of condoms distributed was 37,736 in a quarter. 52

70 Annex 3. Economic and Financial Analysis 121. Section 3.2 rated BNAPS with respect to efficacy, based on how it performed on the PDO indicators. Beyond effectiveness, the development impact of the project depends on efficiency, which requires an assessment whether the resources were used to invest in interventions with positive economic returns, and whether the allocations were optimal. Efficiency is important because of its positive association with sustainability Subject to data limitations, this Annex explores the efficiency dimension and summarizes the key findings of the economic analysis related to the behavioral and biomedical interventions supported by BNAPS, the integrated care delivery models tested, and the broader effects. As in the preceding sections that discuss project performance, efficiency is assessed by phases, before and after the PDO revision. The presented economic analysis covers BNAPS-specific observations based on the available data and documents. Where data is not available/sufficient, the assessment is guided by an updated review of the literature, which is required not only because new evidence is available since appraisal, but primarily motivated by the fact that the scope of BNAPS considerably changed, including: (i) its geographic scope (scale back); and (ii) the thematic scope (vertical to diagonal tendency through the integration agenda) To rate efficiency, the review is organized around the key outcomes of BNAPS, classified into the following main groups (see Table A3-1): 54,55 (1) behavioral interventions to prevent HIV transmission (HIV education and communication, counselling) delivered through schoolbased platforms, including the primary, vocational, and tertiary levels, as well as life skills, and via mass media campaigns, communities, or phone; (2) biomedical interventions, with a focus on HIV prevention through SMC; (3-4) biomedical interventions that ride on the integration agenda (TB-HIV/AIDS care integration and cervical cancer (CC) and HIV/AIDS care integration); and last, (5) broader effects on the health system, health policy, institutional change, and sustainability Epidemiology. To contextualize the economic analysis, an overview of HIV epidemiology before and after BNAPS is provided. 56 With respect to final outcomes, a review and analysis of BAIS data between 2004 and 2013 shows mixed results. Cohort specific prevalence data from BAIS II (2004), BAIS III (2008), and BAIS IV (2012) suggests that survival improved (especially among those age 35 and above) prior to 2008, before BNAPS (Figure A3-1). BAIS III (2008) and BAIS IV (2013) data show that incidence marginally declined from 1.45 to 1.35 (0.1 over 5 years), 57,58 which means that the epidemiological context for BNAPS was largely a stabilized one, as a result of the intensive and effective treatment program from The main focus when BNAPS started was on: (i) sustaining the positive momentum of containing the epidemic; and (ii) finding innovative mechanisms to further reduce incidence in a costeffective manner. The question was not whether treatment is effective in increasing life 54 Annex 2 provides a detailed summary of outcomes attributable to BNAPS, including delivered by CSOs. 55 The analysis does not address the impact of inputs with system-wide implication due to attribution issues. 56 Before & after analysis, based on: BAIS III, 2008; BAIS IV, 2013; UNAIDS, 2015; NTBP, 2015; MOH, UNAIDS cross-country data (Figure A3-2) corroborates this, showing that the largest drop in incidence is prior to Comparing the change in HIV incidence in Botswana to other countries between 2012 and 2014, there is insignificant reduction against Lesotho, Namibia, and Zambia and no change relative to South Africa and Zimbabwe. 58 Because BAIS IV has not published age specific incidence estimates, there is no firm conclusions about incidence reduction across the age groups to assess the effectiveness of prevention strategies. Analysis of prevalence data of BAIS from 2004 to 2013 by age-cohorts shows that prevention efforts for youth have been largely ineffective since Since then, all three of the youngest cohorts had doubled or tripled their HIV prevalence by 2013 (NACA, 2015). Cohorts in the and year old age groups surpassed the national HIV prevalence at 18.5% in

71 expectancy (reducing mortality) and averting new infections (i.e., treatment as prevention) but whether there are more efficient alternatives to avert new cases (e.g., behavioral and combination prevention), to contribute to attaining the ambitious target of zero new infection (NSF II). Table A3-1: Overview of selected key inputs and outcomes attributed to BNAPS Intervention Type Intervention/Input Selected Main Outputs / Outcomes Infrastructure & Equipment Behavior HEC & BCC Behavior HEC & BCC Behavior HEC & BCC Behavior HEC & BCC Building, Reconstruction, Procurements Counseling on Air School-Based Youth Education (Primary) School-Based HEC. Department of Vocational Education & Training (DTVET) (Secondary) School-Based HEC. Tertiary Education Council (TEC) (1) Jubilee Hospital's TB Ward. (2) 566 HAART compliant facilities. (3) 72 functional IDCCs (out of 230), approximately 41,105 patients reached. (4) 5 mobile trucks for SMC shared with CC. 2,600 youth (monthly nationwide) 808 Schools, 834 Teachers 7 technical Colleges and 26 Brigades 10 clusters of TEC institutions received counselling on campuses. On average, 13,451 are reached in a quarter for SMC demand creation, CCC, Abstinence, MCP and counselling. Highest number of condoms distributed was 37,736 in a quarter. Behavior HEC & Youth Education - Life BCC Skills 2 camps/year (*150 children) Biomedical HIV Prevention - SMC 150,136 Biomedical Integrated Care - TB & HIV/AIDS Biomedical Broader Effect - System Level (Improved Care Quality/Efficiency) Broader Effect (Sustainability) Broader Effect (Policy Change) Integrated Care - CC & HIV/AIDS HRH, Supply Chain, HMIS Sustainable Scale-up Policy & Institutional Change (1) Patient defaulter rate reduced from 4.4% to 2% in most districts. (2) Enhanced case detection (2008= 69%; 2012= 75%). (3) Notification rate (2010= 503/100,000; 2013=331/100,000). (4) Treat success (2008=71%; 2013= 82%). (5) Co-infection rate (2010= 64%; 2013= 60%). (6) Increased integration in DOTS and ART uptake for HIV- TB co-infections: ART uptake (2010=53%; 2013=75%). (7) TB integration = 670 health facilities (including health posts). (8) ART integration = 500 health facilities. (1) HPV Vaccine: Achieved = 1,967 from 2,488 eligible girls 9 years and older (79%). (2) 9 See & Treat clinics operational in 5 Health Districts to be expanded to 16 clinics in 8 new Districts by the end of 2015 (1) Lab Staff for CC and TB. (2) Drug Availability - Reduced turnaround from 72 days to 3. (3) Reduced turnaround time for TB and CC test. (4) Catalyzed harmonization effort between IPMS and PIMS, specifically under CC patient management (CC registry). 72 functional IDCCs, approximately 41,105 patients reached. (1) Prisons Program scaled up and fully funded by MoDJS. (2) NCCPP fully funded by MoH (1) Integration Agenda. (2) HIV/TB Integration, including the MoH Departments - Public Health and HIV/AIDS. (3) NCCPP Strategy Sources: Based on BNAPS reports, monitoring Gantt charts, BNAPS output table, and program reports by implementing agencies. Note: ART = Anti-Retroviral Therapy. BCC = Behavior Change Communication. CC = Cervical Cancer. DOTS = Directly Observed Treatment. HEC HIV/AIDS Education and Communication. HMIS = Health Management Information System. HRH = Human Resources for Health. IEC = Information, Education, Communication. NCCPP = National Cervical Cancer Prevention Program. TB = Tuberculosis. 54

72 Figure A3-1: Age-specific HIV prevalence Sources: BAIS II (2002), III (2008) and IV (2013) Figure A3-2: Incidence of HIV in Botswana and its neighboring countries Source: UNAIDS (2015) 125. Policy Priorities and Resource Allocation. Considering resource allocations across prevention and treatment, the stagnation in these final outcomes may not come as a surprise. The Investment Case (NACA, 2015) highlights that for the entire period of NSFII ( ), prevention activities were allocated 10% of the total budget, 73% of which went solely to PMTCT. While this share is well-justified given the high prevalence among pregnant women, other preventive interventions receive a small proportion of the funds. Budget allocation toward IEC campaigns accounted for approximately 4% of the total budget, and allocations toward HIV counseling and testing were minimal for the entire period of NSFII. 59 Given such under-resourced prevention context, although the financial contribution from BNAPS was small as a share of the total NSFII budget (6.5%), 60 it represented a significant share of the resources that went toward 59 NACA, UNAIDS, Botswana Partnership Mapping Report, BNAPS resources were $US50 million between 2009 and 2015, which is approximately 6.5% of the total NSFII budget (2009 and 2016). This statistics assumes that the majority of BNAPS resources were used for prevention. 55

73 prevention (65%) (Table A3-2). 61 Against this backdrop, despite its relative small weight, 62 through some targeted BCC interventions and the biomedical prevention interventions, BNAPS played a role in (i) contributing to stabilizing the epidemic; and (ii) catalyzing system-level changes, which are required to enable further and more sustainable gains in this area. The interventions that contributed to these outcomes are discussed below in depth. Table A3-2: NSFII Budget and BNAPS share of NSFII Strategic Priorities Total NSFII Total NSFII Allocation Budget (Pula) Budget ($US) (%) Priority 1: Prevention of New HIV Infections 620,726,144 $ 76,908,208 10% Priority 2: Systems Strengthening 570,568,434 $ 70,693,648 9% Priority 3: Strategic Information Management 68,716,249 $ 8,513,970 1% Priority 4: Treatment, Care and Support 4,966,259,000 $ 615,321,398 80% Grand-Total (All 4 Priority Areas) 6,226,269,827 $ 771,437, % BNAPS Share of Total NSFII Budget ( ) 403,556,665 $ 50,000, % Source: NACA, NSFII (NACA, 2015) 63 (1) Behavioral Interventions to Prevent HIV Transmission - Phase 1 & In support of the Government s objectives, as outlined in the NSF ( ), the BNAPS not only shifted the focus to prevention from the formerly treatment-centered national response but placed a heavier emphasis on behavior change. In terms of the options for prevention, whereas structural strategies seek to change the context that contributes to vulnerability and risk, and biomedical interventions block infection or decrease infectiousness, behavioral strategies attempt to motivate behavior change within individuals or social units by use of a range of educational, communication, motivational, peer group, skill building, and community 64 normative approaches The extraordinary social and economic cost of the HIV/AIDS epidemic suggests that prevention can be far cheaper than treatment, thus motivating continued search for innovative prevention approaches that are more effective and cost-effectives. There is ample evidence that prevention is cost-effective. However, cost-effectiveness is sensitive to the epidemiological (e.g. generalized, vs. localized) and economic (e.g. low vs. high-income) context. For example, an epidemiological model-based cost-effectiveness analysis of approaches to combat HIV in developing countries in Sub-Saharan Africa and South Asia shows that reducing HIV transmission could be done most efficiently through mass media campaigns, interventions for sex workers and treatment of sexually transmitted infections where resources are most scarce (Hogan et al., 2005). 65 Further, prevention of mother to child transmission, voluntary counselling and testing, and school based education would yield further health gains at higher budget levels and would be regarded as cost effective or highly cost effective based on standard international benchmarks. 61 A smaller share of BNAPS resources contributed to system and information management strengthening (e.g., making facilities IDCC compliant, IT support for See & Treat sites, facilitating access to IPMS and improve medical information storage and retrieval. If the denominator includes systems strengthening (9% of NSFII), then BNAPS resources account for 34.2% of the NSFII funds that were allocated to prevention and systems strengthening. 62 Using target population estimates and assuming a 5 year project duration, annual BNAPS funding per beneficiary was expected to be approximately US$6. To put this in context, the per capita health expenditure and government health expenditure are, respectively, US$307 and US$214 (2008, at exchange rate). 63 The budget figures presented in the table are drawn from the Investment Case paper (NACA, 2015) 64 Individuals cannot change their behavior in a vacuum, but are heavily influenced by their social networks and group norms. Their perceptions of risk are ordered and nurtured by the peer group and social context within which they operate. Behaviors have to be supported and reinforced by the value system of the society. (MAP, 2007). 65 This depends strongly on the nature of the epidemic, and how good the particular campaigns are. 56

74 128. Among the preventive interventions assessed in the paper with respect to costeffectiveness, BNAPS engaged in the following under the umbrella of behavior change approaches: (i) school-based education; (ii) voluntary counseling and testing (VCT); 66 and (iii) mass media. Region-wide experience with effectiveness and cost-effectiveness suggests that among the three interventions applied, the effectiveness in terms of DALYs averted is comparable for mass media and VCT, however the former is significantly more cost-effective with a much lower average cost-effectiveness ratio. However, results are sensitive to the specific intervention design and, hence, country and project-specific investigations are critical to inform operations and policies. Among all interventions presented, school-based education is the least effective and has a relatively high price tag. However, evidence from 2011 by UNESCO and UNAIDS suggests that sexuality education programs can be highly cost-effective, especially when compulsory, adapted from existing models and integrated into the mainstream school curriculum In contrast with the results from the region-wide modeling, Dupas (2009) uses a randomized experiment to test whether information can change sexual behavior among teenagers in Kenya. The paper is motivated by the debate over whether scalable school-based HIV/AIDS education programs can be effective in limiting the spread of HIV/AIDS among youths, and over what should be the content of these programs. Many sub-saharan African countries have incorporated HIV/AIDS education in their school curriculum, but the great majority of those curricula are limited to risk avoidance information: they aim at completely eliminating premarital sex, by promoting abstinence until marriage. They omit to provide risk reduction information, for example that condom use reduces the risk of HIV transmission. The study finds that content of the curriculum and framing the message matters. Providing information on the relative risk of HIV infection by partner's age led to a 28% decrease in teen pregnancy, an objective proxy for the incidence of unprotected sex. Self-reported sexual behavior data suggests substitution away from older (riskier) partners and towards protected sex with same-age partners. In contrast, the national abstinence-only HIV education curriculum had no impact on teen pregnancy. These results suggest that teenagers are responsive to risk information but their sexual behavior is more elastic on the intensive than on the extensive margin. The implication of these findings is to place more emphasis on risk education, which is more realistic then expecting prolonged abstinence Based on the evidence summarized above, the conclusion for BNAPS is that mass media and counselling interventions appear to be between low to medium efficacy and efficient, while the effectiveness and cost-effectiveness of school-based BCC approaches depends on the content and messaging. It must be noted that cost-effectiveness is a relative concept as it depends on the level of government resources and income (GDP per capita) so if the intervention proves to be effective relatively higher unit costs per case or life averted can be absorbed as these are justified on the grounds of rate of return for public resources Although it is not possible to make a conclusive judgment on the impact and costeffectiveness of behavioral change approaches applied in BNAPS due to lack of data on intervention or program costs and on the observed efficacy in terms of cases averted the PDOlevel outcomes on condom use and MCP suggest that there has been no improvement on these 66 In January 2004, the GoB launched a Routine HIV Testing (RHT) policy at all health facilities, following an explicit opt-out principle. The goal of this approach was to normalize HIV/AIDS-related disease perceptions and expand access to all available (and appropriate) services in a timely manner. Despite the opt-out testing, rates were low at the baseline in 2008 (BAIS III) but improved by 2013 (BAIS IV) (see Table A10-8). 57

75 risky behaviors 67 (particularly for the age group) (see more on MCP in Box A3-1). Knowledge of sexual modes of HIV transmission remains low, with less than half of the youth target groups correctly identifying ways of prevention and little progress detected over time Despite improved coverage in VCT during the timeline of BNAPS, further efforts are required to attain a higher coverage rate and ensure linkage to care. The status of these indicators motivates a Negligible rating for the efficiency of behavior change interventions for both phases. Low efficacy rating means that the economic effects of these interventions have fallen behind expectations (i.e., low value for money or low return on investment). This rating does not mean that behavioral interventions are not good investments. It points to the importance of further explorations in this area. First, testing innovative approaches to induce behavior change (e.g., CCT and lottery, as piloted in Tanzania and Kenya) is encouraged. Second, it is important to better understand the role and effect of intervention by the public sector, CSOs, and the private sector and obtain data that enables such assessment. The M&E framework of BNAPS was not designed to enable evaluating the relative effectiveness (and cost-effectiveness) of interventions through these channels/agents, or the extent of complementarity and supplementary between them. As Component 3 absorbed approximately half of the resources, and CSOs had a big role to play in behavioral strategies, lack of data on effectiveness and efficiency is unfortunate as this prevents evidence-based program adjustments. The documentation obtained for this ICR reveals that CSOs primarily engaged in BCC related to abstinence, condom use and MCP, and inducing demand for SMC (Box A1-1, Table A1-1). However, it is not possible to isolate expenditures for these interventions and the consequent outputs/outcomes. Future projects could incorporate a component to collect data on effectiveness and costs to enable a robust economic analysis to inform prioritization decisions. Box A3-1: The lethal cocktail of HIV transmission SMC, MCP, and gender aspects in Botswana SMC: BAIS IV estimated the National SMC rate by 2013 at 24.3%, well below the National target set at 80%. In that same year, the Ministry of Health launched a program targeting neonatal circumcision, and in early 2014 based on its experience and achievements, re-defined its targets expecting to achieve an overall SMC coverage of 61% by Special focus was directed at year olds with targets set at 80% and year old males with targets set at 40% and by June 2014 the SMC program had reached 65% of its proposed target for adolescent boys. During 2013 the National SMC program completed 46,793 circumcisions. By March 2015, a total of 150,136 males had been circumcised. BNAPS has contributed to these outcomes in SMC, particularly through the increased fund allocation to the MOH to enable service delivery, as well as to CSOs to generate demand for SMC. Continued emphasis on demand creation is essential to increase the ratio of men that benefit from SMC. Social norms on SMC can change very quickly For example, in Madagascar which has a much more favorable context given its localized epidemic there is a widely spread practice of SMC, with approximately 95% of males circumcised (World Bank, 2015). MCP: The lack of progress (and reversal) on this behavioral indicator urges exploring alternative and innovative mechanisms to influence sexual partnership practices. Approaches, such as conditional cash transfer (CCT) to reward safe sexual behavior (de Walque et al. 2012; de Walque, Dow, and Nathan, 2014) and lotteries (Bjorkman Nyqvist, 2015) to alter risk preferences have now been piloted in 67 Risky behaviors may also be associated with moral hazard, which can be triggered by free access to care (ART). It can be assumed that with no-cost treatment, the economic incentive for changing risky behavior is low. 68 The BAIS II (2004) found that 93% of the respondents had heard of HIV/AIDS. Yet, the proportion of respondents years who correctly identify ways of preventing the sexual transmission of HIV and reject major misconceptions about HIV transmission/prevention increased from 36% in 2001 to 38% in 2004, against the GoB s target of 90% by See trend in Figure A10-3. Several prevention targets in the NSF were not met. 58

76 Tanzania and Lesotho, 69 including follow-up rounds to assess medium-term effects and the heterogeneity of gender response over time. These innovative approaches differ from the more mainstream socio-behavioral intervention in a fundamental way. They apply hard, economic incentives vis-a-vis the softer incentives through behavioral communication and social norming, which take substantively longer time to induce lasting gains. 70 Gender: The gender dimension has been a critical but under-monitored and under-researched area within the documentation provided and reviewed for this ICR. The recent studies that explored the effectiveness of innovative interventions to induce behavior change specifically to reduce MCP and increase safe condom use found that while the interventions had lasting impact on men, who were observed to have adopted safer sexual behavior practices as a result of the pilot programs, the effect on women dissipated following the discontinuation of cash transfers. There is a clear economic argument behind the observed gender gradient. Women, who are economically less empowered, are more exposed to risky sexual behavior because of their economic dependency. In other words, there is a stronger price and income effect for women, which is associated with increased elasticity of risky sexual behavior in response to cash, the functional form of which is likely non-linear; meaning that poor and vulnerable women are exposed to higher risk probability. Targeted interventions to these vulnerable subgroups may be effective in reducing the gender gradient. (2) Biomedical Prevention of HIV Transmission through SMC Phase 1 & Cognizant of the slow project disbursement trajectory and concerned by the stickiness of behavioral indicators, BNAPS increased the focus on some biomedical interventions known to be effective in reducing HIV transmission without the need for continuous reminders or followup to ensure compliance, as required in BCC or preventive treatment A series of papers on HIV prevention in The Lancet emphasize that highly active HIV prevention inevitably must be combination prevention (Coates, Richter, and Caceres, 2008). Advances in biomedical HIV prevention, as in the case of safe male circumcision (SMC) or the potential of antiretroviral therapies for prevention, provide substantial opportunities to reinvigorate behavioral approaches to HIV prevention and challenge governments to advance structural approaches so that these approaches can get to those who need them the most. All these prevention approaches contribute to effective HIV prevention within communities, and thus behavioral strategies need to be used in combination with biomedical and structural approaches that are combined strategically to address epidemics In line with the combination prevention method, BNAPS ramped up its prevention efforts by adding increased emphasis on SMC, leading to a total of 150,136 males circumcised by March 2015 (i.e., 39-% progress toward the target set by the end of the project). Although the target was not attained, the coverage enabled by BNAPS is considerable; equivalent to approximately 35% of the male population covered under BNAPS, and 18% of the total male population. 71 Although further efforts in demand generation are required to attain the coverage target, the MOH has been strategic in its targeting as described in Box A3-1 and the improved coordination 72 with CSOs has also contributed to improved efficiency. 69 The economic foundation of these experiments rests in neoclassical economics, which predicts that incentives will alter behavior in part via price effects. Conditioning a monetary value on safe sexual behavior increases the shadow price of risky sex. Income effects also play a role, particularly with increasing value of cumulative repeated rewards, which act as an income substitute. The studies use objective, biomedical measures, such as the incidence of STIs, not reported measures for this outcome, which have proved to suffer from reporting bias. 70 Exploring the feasibility and desirability of adopting such approaches to the Botswana context is encouraged. 71 BAIS IV (2013) shows a 24.3% target population coverage of SMC; increased from 11% in 2008 (BAIS III). 72 This indicator is promising given that the outcome was affected by initial coordination challenges. Improved supply-demand coordination between the MOH and CSOs led to higher effectiveness and efficiency over time. 59

77 136. With regards to effectiveness and cost-effectiveness, evidence from a randomized control trial 73 in South Africa concludes that in settings with high or moderate HIV prevalence among the general population, adult SMC is likely to be a cost-effective HIV prevention strategy, even when it has a low coverage. SMC generates large net savings after adjustment for averted HIV medical costs (Kahn, Marseille, and Auvert, 2006). Based on the cost-effectiveness estimates from Kahn et al. (2006), the effect of the SMC intervention in 2013 alone, when 46,793 circumcisions were carried out, could be around 14,412 HIV infections averted (HIA) over 20 years (80% CI 8,840 and 20,027). Imputing the cost of treatment averted 74 for the HIA translates into a savings of approximately $113.2 million (80% CI 69.5 million to million) during the 20-year horizon These findings suggest that the increased emphasis on SMC under BNAPS was an effective shift in the approach mix and led to improved outcomes in prevention. Given the slow uptake of SMC during Phase 1, the efficiency of this intervention was Negligible. Following improved effectiveness during Phase 2, 75 the efficiency is rated as Substantial, particularly taking into account long-term effects and secondary transmission to women. (3) Biomedical Intervention & Integration Agenda: TB-HIV/AIDS Phase 1 & Epidemiological Context. The GoB has a strong commitment towards tuberculosis (TB) control. The Bank came on board with a five year plan on the realization that Botswana, despite being an upper middle income economy, still grappled with high burden of TB, which remains a leading cause of death among people living with HIV. 76 At appraisal, Botswana ranked sixth in the world in terms of the number of the TB patients per capita. Historically, the notification trend had increased dramatically from 202/100,000 population in 1989 to a peak of 623/100,000 population in The 536/100,000 population reported in 2008 formed the baseline data for BNAPS. TB continues to be a major public health problem in the country with the 2009 Botswana National TB Program (BNTP) Review showing a TB/HIV co-infection rate of 63% Costs and Benefits of TB and HIV/AIDS Integration in Botswana. From the list of challenges stated by the 2009 BNTP Review document, BNAPS mainly supported hospital infrastructure renovation, equipment purchase, training for various HR, and to facilitate knowledge sharing (Table A2-11, Annex 2). The BNAPS TB/HIV final report (MOH, 2015) assessed program performance with respect to expenditures and outcomes. Although the report 73 Cost-effectiveness was modeled for 1,000 SMCs done within a general adult male population. Intervention costs included performing SMC and treatment of adverse events. HIV prevalence was estimated from published estimates and incidence among susceptible subjects calculated assuming a steady-state epidemic. Effectiveness was defined as the number of HIV infections averted (HIA); estimated by dynamically projecting over 20 years the reduction in HIV incidence observed in the trial, including secondary transmission to women. Net savings were calculated with adjustment for the averted lifetime duration cost of HIV treatment. Sensitivity analyses examined the effects of input uncertainty and program coverage. Results were discounted to the present at 3% per year. For Gauteng Province, assuming full coverage of the MC intervention, with a 2005 adult male prevalence of 25.6%, 1,000 circumcisions would avert an estimated 308 (80% CI ) infections over 20 years. 74 This is based on the value presented in the Investment Case (NACA, 2015), which estimates the treatment cost averted to be around $7,852 for a 15-year period between 2015 and While this is not identical with the 20 years used for the calculations above, it can be used as a lower bound estimate. The report provides annual estimates for treatment cost per person, which starts from $450 per capita in $US 2015 prices (2015) and reaches around $550 by 2030 (i.e., a 22% increase over 15 years). 75 Improved MOH-CSO coordination for demand generation IEC addressed the earlier implementation bottlenecks. 76 The number one of cause of death is HIV (35%), followed by cancer (6%), and stroke (5%). TB ranks fourth on the causes-of-death list, accounting for nearly 5% of all deaths. The Top 10 diseases account for 70% of total death. Source: 60

78 notes that expenditure reporting by activity was difficult, 77 it provides aggregated data by programmatic categories (Table A2-11, Annex 2). Regarding outcomes, the report describes the trends in TB and HIV/AIDS care management (Table A2-12, Annex 2). Selected key outcomes, supported and measured by BNAPS, are highlighted below. People Receiving Treatment in Accordance with WHO-recommended DOTS: The hallmark of TB control is early case detection and treatment, promoted by directly observed treatment, short course (DOTS). 78 Improving Case Detection through Quality Assured Bacteriology: The BNAPS resources that were dedicated to laboratory staff training translated into improved case detection (from 69% to 75%) and treatment success rate (from 71% to 82%) between 2008 and Lab Efficiency Reduced Sputum Turnaround Time due to Gene Xpert: To further support rapid and accurate diagnosis of TB, BNAPS procured 20 Gene Xpert machines 80 and accessories. The machines shorten the time period between diagnosis and initiation of treatment for both TB and drug resistant TB as shown in Table A3-3. Table A3-3: Impact of diagnostic equipment (Gene Xpert) on specimen turnaround time ACTIVITY BEFORE XPERT AFTER XPERT 1 Specimen result turnaround time 72hrs 3 weeks 3 hours 24 hours 2 Initiating MDR-TB Treatment 8 weeks or more 3 hours 3 Drug resistance Diagnosis About 5 patients over 5 months About 17 patients over 5 months Source: TB Program, MOH (2015) 140. Estimates of the Economic Returns on Investment in TB Diagnosis. UNAIDS reports suggests that annual TB-related deaths in PLWs dropped by nearly 50% (from 1,800 to 930) between 2008 and 2013 (Table A3-4). To illustrate the economic value of TB mortality averted among PLWHs, we focus on the 170 lives saved between 2012 and 2013, when BNAPS contributed significantly to improving TB case management through investing in HR, lab equipment, and infrastructure. 81 Imputing the average per capita GDP, the economic value of the 170 lives saved in one year is approximately US$1.21 million. Projecting the gains through 2020, assuming an average growth rate of 4.3% and a discount rate of 3%, the total economic value of the lives saved is approximately US$10.1 million. While this rudimentary estimation does not take into account the mortality dynamics during the 8 years of the projection and it can also be argued that not all of these lives saved during this period can be attributed to BNAPS, the economic returns on the investment (US$2.1 million) 82 are clear, even after adjusting the assumptions for the number of lives saved and the discount rate parameters Due to irregular reporting by districts, partially because of ambiguous reporting lines in a decentralized implementation platform, which may result in data by-passing the TB Program at the HQ of the MOH. Districts ran trainings without the HQ as they had trained ToTs and accessed funds through GABS, the GoB s budgeting system. 78 DOTS: a 6-month regimen of 4 first-line anti-tb drugs: isoniazid, rifampicin, ethambutol, and pyrazidnamide. 79 This success is important for TB epidemiology and economically because multi-drug resistant (MDR) TB is becoming an increasing threat to health and development gains are placed in hospital laboratories in the country, 1 at the National Laboratory, and 5 in hard-to-reach areas. 81 E.g., BNAPS invested in the training of lab technicians, financed the salary of 2 regional TB coordinators, procured the GeneXpert, and the renovation of the Jubilee hospital s TB ward (Table A2-11, Annex 2). 82 Table A2-11, Annex 2 shows the funds for TB from BNAPS (US$2.1 million) by key investment line items. 83 With a cumulative investment of $2.1 million to support TB diagnostics, if 170 lives were saved between 2012 and 2013 as a result of the program, this would mean that the return for every dollar invested is US$4.8. For a more conservative estimate, using a 5% discount rate, the return for every dollar invested is US$4.5. To address potential concerns regarding the attribution of mortality aversion due to the program, if only half of the lives are assumed to be saved by the program (85), the economic return remains positive (US$2.4 at the 3% discount rate and US$2.2 at the 5% discount rate). In sum, the investment value ranges between double and nearly five-fold of the original value. 61

79 Table A3-4: TB-related death among people living with HIV (PLWHs) Change Botswana 1,800 1,600 1,400 1,300 1, % Lesotho 7,200 6,700 6,100 5,600 5,400 5,200 28% Mozambique 38,000 39,000 40,000 40,000 40,000 38,000 0% Malawi 8,100 7,000 5,900 5,100 4,100 3,400 58% Swaziland 3,300 3,500 4,000 4,100 4,400 5,000 52% Tanzania 8,400 8,200 7,700 7,300 6,800 6,100 27% South Africa 89,000 88,000 85,000 77,000 67,000 64,000 28% Zambia 8,700 8,800 9,400 9,200 8,600 8,600 1% Zimbabwe 29,000 27,000 26,000 24,000 22,000 22,000 24% Source: UNAIDS. Accessed on: 11/05/ The contribution of Gene Xpert to effectiveness and economic impact in terms of opportunity and treatment costs, and DALY is significant. Before Gene Xpert (2013), there were many patients treated as MDR-TB suspects. A significant share of patients diagnosed as MDR-TB by the National TB Referral Lab died before treatment as a results of loss to follow up. The numbers of MDR-TB patients who died before initiation of treatment were very high as compared to at present. Much success has been recorded through BNAPS in this area as shown in the outcomes in Figure A3-3 (see Table A2-12 in Annex 2). Figure A3-3: Impact of improved TB diagnosis since using GeneXpert Treated as MDR TB Suspects Confirmed as MDR TB Loss to Follow Up Died before initiation of RX Starting MDR TB RX Source: TB Program, MOH (2015) 142. Additional benefits from BNAPS-supported TB management: Infection Control: BNAPS funded Infection Control (IC) activities including the Health Facility Assessments that were done quarterly as part of the MSS. Community-Based TB Care (CTBC): Enrolment under CTBC to increase, from as low as 9% in 2009 to 69.5% in 2013, against a target of 75%. Table A3-5 shows the trends in CTBC uptake. The observed gains in scale-up are largely attributed to contributions from CSOs, through funding and technical support from Global Fund, BNAPS and ACHAP. 62

80 Table A3-5: Trends in community-based TB control (CTBC) TARGET GROUP COMMUNITY ACTIVITY TOTAL TRAINED* Community Leaders TB/HIV Sensitization 2,350 Workplaces TB/HIV Sensitization 4,532 Congregate Settings TB/HIV Sensitization 3,306 Community Members TB/HIV & CTBC Sensitization 51,522 TB patients and their families TB/HIV & CTBC Sensitization 15,033 CBOs/FBOs/NGOs TB/HIV & CTBC Training 2,497 Note: The numbers reported are far below the actual figures because recording and reporting remains a challenge. Policy Change: During the period of BNAPS support, the Botswana National Policy on HIV/AIDS was reviewed. Among the many changes, TB was included realizing that TB could not be divorced from HIV and that TB was since recognized as a public health emergency, needing an intersectoral approach just like HIV. This national policy is formulated as a strategic intervention to guide the conduct of the employment relationship to promote mutual understanding of all, including government, employers and employees in managing HIV/AIDS and TB issues in organizations. Its application is intended to result in a safe, healthy and productive workforce, by fostering the elimination of discrimination, stigma, ignorance and prejudice in the workplace. The national policy is part of the country s response to the HIV/ AIDS & TB epidemics, providing guidance to efforts to prevent the spread and manage the impact of HIV/AIDS and TB (see more on TB-HIV integration in Box A3-2). Box A3-2: Evidence on TB and HIV/AIDS Integration TB is a major cause of morbidity and mortality in people with HIV and about a quarter of HIV-related deaths are attributed to tuberculosis. A systematic review of the effectiveness and cost-effectiveness of eight integrated strategies recommended by WHO that represent coordinated delivery of HIV and tuberculosis services finds the followings: Evidence supports concurrent screening for tuberculosis and HIV, and provision of care 84 during routine HIV care and at voluntary counselling and testing centers. Although integration of antiretroviral therapy into tuberculosis care has shown promise for improving health outcomes for patients, evidence is insufficient to make conclusive claims. Evidence is also insufficient on the accessibility of condoms at tuberculosis facilities, the benefits of risk reduction counselling in patients with tuberculosis, and the effectiveness of TB infection control in HIV healthcare settings. The vertical response to the tuberculosis and HIV epidemics is ineffective and inefficient. Implications for policy makers and funders include further investments in implementing integrated tuberculosis and HIV programs with known effectiveness, preferably in a way that strengthens health systems; evaluative research that identifies barriers to integration; and research on integrated strategies for which the effectiveness, efficiency, and affordability aspects are not well established (Uyei et al., 2011). Sources: Uyei et al. (2011); Legido-Quigley (2013) 143. In sum, apart from high efficacy, 85,86 the estimates on economic returns indicate good value for money, which merits a Substantial rating on efficiency for Phase 1, during which DOTS coverage started to increase, and a High rating for Phase 2, during which BNAPSsupported activities intensified and the integration agenda became more pronounced. In addition 84 Treatment can be either co-trimoxazole during routine TB care or isoniazid. 85 The review of activities and outcomes managed under the TB Program of the MOH indicate considerable impact during BNAPS as shown in Table A2-12 in Annex The NTBP Report notes that attribution of impacts to the BNAPS is not always feasible as the TB Program benefited from resources and technical assistance from other DPs. Some of the achievements were collectively brought about by DPs (e.g., the Global Fund, ACHAP, CDC Botswana, I-TECH, UPenn Botswana, KNCV, URC, PRR, and WHO), though there is no doubt that BNAPS had been among the partners who provided more support. 63

81 to the progress over time, Botswana also compares favourably on TB related outcomes to its regional peers (Table A3-6). Table A3-6: Cross country comparison of TB outcomes in selected regional peers Country Incidence (per 100,000) Estimated # of Cases per Year TB-related Mortality of PLWHs (2013) TB-related Mortality of PLWHs (2008) TB Detection Rate (%) TB Treatment Success Rate (%) Population (millions) GNI Current, Atlas Method Botswana , , $7,710 Lesotho ,000 5,200 7, $572 Malawi ,000 3,400 8, $292 Mozambique ,000 38,000 38, $275 South Africa ,000 64,000 89, $2,950 Swaziland 1,382 17,000 5,000 3, $1,819 Tanzania ,000 6,100 8, $378 Zambia ,000 8,600 8, $553 Zimbabwe ,000 22,000 29, $597 AFRICA 255 2,300, ,111 $2,443 GLOBAL 122 8,600, ,162 $12,781 Sources: WHO, World TB Report 2014; Regional TB Project CN (2015); UNAIDS; 90 WHO NHA 2015; TB Program, MOH (4) Biomedical Intervention & Integration Agenda: Cervical Cancer-HIV Phase Cervical cancer a preventable condition that usually results from a viral infection by HPV that is generally sexually transmitted is one of the leading causes of premature death and ill health among women in Sub-Saharan Africa. 92 This is due to minimal screening services, resulting in a significant number of patients diagnosed with advanced-stage disease. A key problem in most cases is the limited health-system capacity In Botswana the HIV/AIDS epidemic overstretched the capacity of the system. In response to this, the MOH strategically shifted focus to the integration agenda, as in the case of TB-HIV/AIDS care management. The Department of Public Health developed the National Cervical Cancer Prevention Program (NCCPP) Comprehensive Strategy ( ). To operationalize the NCCPP, the GoB, with the Bank and the George W. Bush Institute s Pink Ribbon Red Ribbon (PRRR) initiative, 93 rolled out a Cervical Cancer Control Program with key elements financed by BNAPS, including: (i) cervical screening (i.e., secondary prevention); and (ii) HPV vaccination. These drew on the HIV diagnostic and treatment platforms established across Botswana over the last decade. The results of investing in these interventions are impressive Secondary Prevention: With support from BNAPS, See and Treat clinics are operational in 5 health districts and planned to be expanded to 16 clinics in 8 new districts by the end of 15. The results from a pilot program among HIV-positive women at a community-based clinic in Gaborone indicate that the low-cost see and treat approach for the prevention of cervical cancer is a feasible and efficient alternative, especially for reaching women living in 87 WHO reports 414 per 100,000 population. The table is based on the TB Program s Report of the MoH (2015). 88 The statistics reported in the table is for 2012, based on the TB Program s Report of the MoH (2015). The improvement is 4 percentage points between 2011 and 2012 (from 71 to 75%). WHO Global Database reports 82%. 89 More recent data confirmed Malawi s incidence rate is almost double the rate reported here Section draws on: Marquez and Farrington (2013); Report of the NCCPP of the MOH of Botswana (MOH, 2015). 92 In Eastern & Southern Africa, it is compounded by high prevalence of HIV. HIV+ women are 4-5 times more likely to develop cervical cancer. 93 PRRR mobilizes the coordinated participation of institutions such as US CDC, US PEPFAR, USAID, UNAIDS, Susan G. Komen Foundation, the Gates Foundation, CARIS Foundation, pharmaceutical companies such as Merck, GlaxoSmithKline, and Bristol-Myers Squibb, and Becton, Dickinson and Company, IBM, and QIAGEN. 64

82 distant and/or underserved regions. The approach is synergistic with other interventions introduced by BNAPS, generating a compounded effect and, expectedly, reducing costs Demonstration Projects: The MOH launched the Introduction of the HPV Vaccine Demonstration Project in March There were 2,488 eligible girls from 23 public and private schools in Molepolole village, with a minimum enrolment age of 9 years and older (grades 4, 5 and 6). Of the target cohort, 1,967 (79%) girls have received all three doses. Informed by the success of the trial, a phased national roll-out plan was proposed and the HPV Vaccine Demonstration transitioned to EPI to allow for sustainability (Table A3-7). In February 2015, the GoB launched the national rollout of HPV Vaccine with a target of 69,582, and attained coverage of 68,304 girls/women (a 98.2% progress toward the target; Table A3-8). The effects of BNAPS, in collaboration with PRRR, in support of the NCCPP objectives are significant. Beyond the biomedical effects and the consequent economic effects, 94 the program served as a model and had a formidable catalytic role in promoting and operationalizing the integration agenda, leveraging the support of the MFDP, expressed in the form of resource allocation to EPI, which led to a national rollout of HPV. The efficiency rating for cervical cancer prevention and the integrated care delivery model is High, and only applies to Phase 2 given its launch in Table A3-7: Phase 2 of HPV rollout using the EPI platform District Target HPV 1 HPV 2 HPV 3 % Coverage (Given) (Given) (Given) HPV1 HPV2 HPV3 Average Kweneng West 1,771 1,784 1,760 1, % 98.6% 96.9% 99.7% Kweneng East 3,305 3,279 3,206 3, % 97.0% 95.7% 97.9% Selibe/Phikwe 1,587 1,583 1,555 1, % 97.9% 94.4% 97.3% 6,663 6,646 6,521 6, % 97.83% 95.67% 98.0% Source: NCCPP, MOH (2015) Table A3-8: Target and attained coverage of national rollout of HPV Vaccine No. of Girls Vaccinated by Age No. of Girls Vaccinated, Total No. Of Girls Vaccinated Group Junior Schools by Age by Age Age Target Vaccinated Age Target Vaccinated Age Target Vaccinated 9yrs 3,496 9yrs 7 9yrs 3,503 10yrs 11,996 10yrs 2 10yrs 11,998 11yrs 16,244 11yrs 74 11yrs 16,318 12yrs 15,053 12yrs 1,683 12yrs 16,736 13yrs 8,347 13yrs 7,618 13yrs 15,965 14yrs 2,564 14yrs 53 14yrs 2, yrs 1,167 15yrs 15yrs 1,167 Total 61,442 58,867 Total 8,140 9, yrs 69,582 68,304 Source: NCCPP, MOH (2015) 94 In a model-based cost effectiveness of cervical cancer prevention in Sub-Saharan Africa, for Botswana Kim et al. (2013) estimate 11 cases averted per 1,000 vaccination. For the pilot supported by BNAPS, this translates to averting 22 cases. Beyond BNAPS, supported by GoB funds, the model-based estimate for cases averted through the scaleup that covered 68,304 girls is approximately 751 cases averted. 65

83 Box A3-3: Cost-Effectiveness of Cervical-Cancer Prevention There are a number of strategies available for cervical cancer prevention and the balance of vaccination, screening, and treatment needs to be according to country context. For Sub-Saharan African countries characterized by low income, high mortality and low treatment levels, increased coverage of treatment with or without screening would be cost-effective, as would one-off PAP or VIA screening at 40 years of age or vaccinations (at $0.60 per dose). The VIA method does not require laboratory facilities and can enable treatment of pre-cancerous lesions. Where resources are limited, cost-effectiveness could be improved with targeted screening and by directing vaccinations towards people infected with HIV, since HIV infection is associated with increased risk of cervical cancer. Given the present high cost of HPV vaccines, countries need to decide the best strategies for their context, based on the evidence, in order to allocate resources efficiently and equitably. While the GAVI Alliance has recently decided to support the introduction of HPV vaccines, and public-private partnerships exist to make breast and cervical screening as well as HPV vaccination more available and affordable in SSA, countries still need to consider the longer-term budgetary implications for sustainable programs, and ensure that effective treatment is available for detected lesions. Early detection and screening programs could achieve down-staging of the targeted cancers within five years, and could reduce mortality within ten years. Cost-Effectiveness of Cervical-Cancer Screening in Five Developing Countries In five developing countries, a cross-country analysis examined clinically effective and cost-effective strategies that enhanced the linkage between screening and treatment, through either a reduced number of visits or improved follow-up, and that relied on less laboratory infrastructure than did conventional cytologic methods. The screening of women with one-visit or two-visit visual inspection or HPV DNA testing at about 35 years of age would reduce the lifetime risk of cervical cancer by 25 to 36%. Two screenings in a lifetime would provide a relative increase in the lifetime reduction of risk of cancer of approximately 40%, although the incremental benefits of three screenings are much smaller. The lifetime costs associated with alternative screening approaches vary among countries, owing to differences in the costs associated with labor and nontradable goods and the relative proportion of direct medical, time, and transportation costs. For similar reasons, cost-effectiveness ratios vary as well. For example, the cost per year of life saved for screening twice in a lifetime with a one-visit visualinspection strategy is $91 in India and $319 in Kenya; this same strategy with the use of HPV DNA testing costs $310 per year of life saved in Thailand, $453 in Peru, and $1,093 in South Africa. Despite considerable differences among absolute cost-effectiveness ratios, the policy implications for these countries are similar once their relative resources (e.g., per capita GDP) are considered. There is no universal criterion that defines a threshold cost-effectiveness ratio, above which an intervention would not be considered cost-effective. The study used guidelines specifically intended for international comparisons, as proposed by the Commission on Macroeconomics and Health, which defines interventions with a cost-effectiveness ratio that is less than the per capita GDP as very costeffective. Expressed in international dollars, the per capita GDP ranges from $1,005 in Kenya to $9,486 in South Africa, suggesting that screening for cervical cancer twice in a lifetime in Kenya and three times in South Africa, Peru, Thailand, and India would be considered very cost-effective. Source: Goldie et al. (2005) (5) Broader Effects 148. The broader effects include policy and institutional change and HMIS. Catalyzing or Contributing to Policy & Institutional Change. During BNAPS, the MOH began designing an Integrated Health Sector Plan, which became a ten-year strategic plan to guide the country in tackling current priority problems, and preparing for future health needs. The MOH engaged in restructuring, merging the Departments of HIV/AIDS and of Public Health, thereby creating a more streamlined approach to planning, care, and evaluation. As discussed, the MOH also developed its integrated care strategies for TB-HIV/AIDS and the NCCPP. 66

84 Harmonization of HMIS: Botswana has been making significant efforts in integrating its multiple, disparate, disease-focused, standalone health information systems. The MOH completed the Health Information Management System Strategic Plan in April 2012 that provides an ICT road map in line with the MOH-adopted e-health Strategy (2015), to integrate the current standalone systems into the existing Integrated Patient Management System (IPMS). IPMS is a centralized, electronic medical record system focused on patient care and treatment in clinic, and hospital settings. It stores data on various health services, including ART, prevention of mother-to child transmission, laboratory, and pharmacy, and in the near future will support the clinical case-management of Safe Male Circumcision and the National Cervical Cancer Prevention Program. It is being implemented in 11 hospitals covering close to 75% of total MOH hospital beds, their labs, and satellite clinics and plans to expand to at least 30 hospitals across Botswana The economic impact of improved policies that respond to national priorities and challenges requires no explanation. The economic impact of improved HMIS, through the pathways of improved efficiency and quality of care management is also evident. These broader changes that have system-wide bearing affect national level outcomes both in terms of effectiveness and efficiency in a profound way. These broader implications underscore the relevance and importance of BNAPS and justify a Modest rating for Phase 1, which provided lessons and the context for project design revision, including system-level aspects, and a Substantial rating for Phase 2, which prompted broader changes This assessment takes into account that consolidating the broader effects will require sustained efforts and investments from the GoB/DPs, and that the financial contribution of BNAPS to these effects was relatively modest. 67

85 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team Members Names Title Unit Responsibility/ Specialty Lending Amie E. Batson Asst. to the Managing Director SEM Slaheddine Ben-Halima Senior Procurement Specialist AFCS2 Donald A. P. Bundy Lead Specialist AFTHE - HIS Sheila Dutta Senior Health Specialist GHNDR TTL Elizabeth Laura Lule Consultant GHNDR Eugenia M. Marinova Senior Country Officer LCC7A Jonathan Nyamukapa Sr. Financial Management Specialist AFTME - HIS Albertus Voetberg Lead Health Specialist GHNDR Christopher D. Walker Lead Specialist AFTHE - HIS Supervision/ICR Modupe A. Adebowale Sr. Financial Management Specialist AFTME - HIS Dirk Bronselaer Senior Procurement Specialist AFTPE - HIS Simon B. Chenjerani Chirwa Senior Procurement Specialist GGODR Sheila Dutta Senior Health Specialist GHNDR TTL Daniel Erim Consultant GHNDR ICR Team Member Peter A. Gaius-Obaseki Research Analyst HDNHE - HIS Task Team Member Patricio Marquez Lead Health Specialist GHNDR Overall Technical Advice and Integration Agenda Tandile Gugu Zizile Msiwa Financial Management Specialist GGODR Ronald Upenyu Mutasa Senior Health Specialist GHNDR Task Team Member Component 3 Adyline Waafas Ofosu- Amaah Regional Coordinator LLIOP Chitambala John Sikazwe Senior Procurement Specialist GGODR Edit V. Velenyi Economist GHNDR ICR Team Leader/Author Christopher D. Walker Lead Specialist AFTHE - HIS (b) Staff Time and Cost Stage of Project Cycle Lending Supervision/ICR Staff Time and Cost (Bank Budget Only) USD Thousands (including No. of staff weeks travel and consultant costs) Total: Total: ,306.5 Total 342 1,

86 Annex 5. Implementing Agency Survey Results 148. The BNAPS ICR Survey was designed 96 and collected through Survey Monkey. The distribution list for the participants was provided by NACA. The survey was not intended to be a representative 97 survey with full coverage, given the time and budget limitations to implement the ICR. The objective of the survey was to offer an anonymous platform and an opportunity to share views. The survey covers 29 questions, including: (i) respondent characteristics (age, gender, education, job title, years working on BNAPS); (ii) implementing agency characteristics (public, CSO, private, central or district level, location of office, geographic coverage under BNAPS, age of organization/entity); and, (iii) views on relevance, implementation, effectiveness, efficiency, and sustainability (quantitative and open-ended responses). Data collection started on July 30, 2015 and ran for 4 weeks. Given the low response rate (22% of 105 targeted individuals), the survey is not representative. The key findings have been shared with the GoB to help drawing lessons and inform interventions and policies. A brief summary is provided in Box A5-1 below. Box A5-1: Key Findings from the BNAPS ICR Survey Relevance: The majority of respondents considered increasing access to HIV prevention fully relevant. There is more variability regarding the relevance of capacity building for NACA. Implementation: 76.2% of respondents noted challenges occasionally and 19% frequently. The biggest bottlenecks included staffing and staff skills, procurement and other implementation aspects. Financial delays were considered very important by more than half of the respondents, followed by general project management and oversight issues. When asked about when these problems occurred, CSO respondents highlighted two years: 2013 When we experienced delay of disbursements to our account which not only affected output but triggered exodus of staff. and 2014 when respondents noted challenges with end of project closure activities and outstanding activities. Effectiveness: With respect to the question How do you think your program performed under the BNAPS relative to the targets that your organization set? ¾ of respondents expressed satisfaction. The remaining ¼ indicated falling below expectations in reaching targets. Efficiency: As to the question What is your opinion of the efficiency of your program's performance (i.e. the amount of money spent to produce outcomes)? -- the majority of respondents expressed satisfaction. The total share is less than stated for effectiveness, indicating that the efficiency dimension was more challenging. Sustainability: In response to What is your opinion about the continuation and sustainability of the program(s) supported by BNAPS following the closing of the project? -- the majority of respondents expressed concern ( Close Down 28.6% and Face Serious Challenges 28.6%). 4.8% of the projects was reported to continue and 38% to partially continue (scale back). Overall Performance: Survey respondents have a largely favorable view of the BNAPS project ( Satisfactory % and Moderately Satisfactory 42.9%). Annex 6. Stakeholder Workshop Report and Results 96 The survey was developed based on a Focus Group Discussion with CSOs organized by NACA during the ICR mission (May, 2015). The CSOs expressed appreciation both to NACA and the Bank for the opportunity to share their views. The ICR Team thanks NACA for availing the distribution list and reviewing the draft instrument, and the representatives from each component who helped testing the instrument and provided valuable feedback. 97 The lack of a scientific sampling strategy is important when interpreting the results. The survey can provide insights for qualitative assessment but should not be quoted as a representative study on BNAPS s performance. Notwithstanding this limitation, driven by time and budget constraints, the survey offers entry points for discussion. 69

87 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR The following comments were received on the draft ICR from the Government of Botswana: Ministry of Finance and Development Planning We do not have any substantive comments. We suggest that under Basic Information, the Borrower is the Government of Botswana represented by the Ministry of Finance and Development Planning. The other point is under Male circumcision (page x) - report does not indicate the difficulties encountered. This should be a relatively easier target to achieve and is a good preventive measure. Ministry of Health The BNAPS project was under the auspices of World Bank from , with the aim of increasing coverage, efficiency, and sustainability of targeted and evidence based HIV/AIDS interventions. The document captured the implementation status as it was but minor observations were made as follows: 1. Indicator 2- page ix, baseline column between the ages of 10-14years-its written No activity (can we specify to abstinence or no sexual activity ). 2. On page xi (indicator 4) it is better to mention ART rather than HAART to just comply with the terms used internationally. Besides, HAART does not appear among the documents. 3. Page 55, table A3-1 under biomedical interventions-align statement that start with patient defaulter rate to Integrated Care TB & HIV/AIDS not under SMC. NACA We find the ICR very comprehensive and factually representative in many of the variables that were indicated for assessment and, therefore, concur with the report and endorse its recommendations, especially on the ratings. Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders The following comments were received on the draft ICR from the Delegation of the European Union to Botswana and SADC: We have gone through the report, which is well written and reflects mostly that the project was successful. As we had sorted out queries earlier, we have not much more to say at this stage other than to thank you for the great cooperation and your hard work. 70

88 Annex 9. List of Supporting Documents 1. Botswana HIV/AIDS Project. Quality Enhancement Review (QER). Final Panel Report. April 28, Project Appraisal Document, June 13, Loan Agreement (Loan Number 7568-BW), January 29, Implementation Status Reports (ISRs 1-13, ) 5. Aide Memoires ( ) 6. BNAPS MTR Issues Paper. Revised. May, 24, Restructuring Papers (211, 2013, and 2014). 8. Close-out Reports CFP1-2 (2011, 2012). 9. NACA National HIV/AIDS Strategic Framework Government of Botswana. 10. NACA (National AIDS Coordinating Agency) The Mid-Term Review of the Botswana National Strategic Framework for HIV/AIDS : On the Road to Vision Gaborone: Government of Botswana. 11. NACA The Second National Strategic Framework for HIV and AIDS: Ministry of State President, Government of Botswana. 12. University of Pretoria. Social Assessment NACA. Government of Botswana. 13. NACA, Euro Health Group Botswana National HIV/AIDS Prevention Support (BNAPS) Project FINAL REPORT. External Results Verification for the Financial Year (April March 2013). Government of Botswana. 14. Baylor, MOESD The Voice of the HIV Infected and Affected School Age Children in Botswana: A Cross-Sectional Psychosocial Survey. June, Identification Mission for the HIV/AIDS Component of the Botswana 10th EDF HRD Programme, Specific contract no 2008/ July Conseil Sante Assessment of the Botswana National HIV/AIDS Prevention Support (BNAPS) Project MTR Performance EC ref N o : 2011/ Draft Final Report. December European Union. Report prepared by: Conseil Sante Consortium. 17. Botswana 2013 Global AIDS Response Report. Progress Report of the National Response to the 2011 Declaration of Commitments on HIV and AIDS. Republic of Botswana. 18. UNAIDS Global Report. UNAIDS report on the global AIDS epidemic NACA Progress Report of the National Response to the 2011 Declaration of Commitments on HIV and AIDS. Reporting Period: Republic of Botswana. 20. NACA Investment Case. 21. Lule and Haacker The Fiscal Dimensions of HIV/AIDS in Botswana, South Africa, Swaziland, and Uganda. 22. Marquez and Farrington The Challenge of Non-Communicable Diseases and Road Traffic Injuries in Sub-Saharan Africa. An Overview. Washington, DC. The World Bank. 23. WHO, Database World Bank International Development Association Proposed Umbrella Restructuring and Amendment of the Financing Agreements for the Projects under Multi-Country HIV/AIDS Program for Africa (MAP). Report No.: AFR. May 29,

89 Annex 10. Illustrative Figures and Tables for the Assessment of BNAPS Table A10-1: Administrative districts: population characteristics & HIV prevalence District Name District Code Capital A (km²) Pop Pop Share Pop Density (pop/km 2 ) Botswana BWA Gaborone 570,162 2,024, % 2,141, Notes: Prevalence 98 rates for 2006 are for pregnant women between years of age. LB = lower bound. UB = upper bound. Prevalence for 2014 are from serosurvey. The 2 sources are not comparable given the difference in the sample populations. The last row shows the population weighted average based on the prevalence rate statistics shown for the districts. Figure A10-1: HIV/AIDS Epidemiology: Panel A and Panel B Annual Growth (%) HIV/AIDS Prev LB-UB HIV/AIDS Prev LB-UB Central 1 CEN Serowe 142, ,604 32% 675, Ghanzi GHA Ghanzi 117,910 43,355 2% 45, Kgalagadi 6 KGA Tsabong 105,200 50,492 2% 53, Kgatleng KGL Mochudi 7,960 91,660 5% 96, Kweneng 2 KWE Molepolole 31, ,549 15% 322, North East 3 NEA Masunga 5, ,225 8% 168, North West NWE Maun 129, ,631 9% 185, Sout h East 4 SEA Ramotswa / Gaborone 1, ,613 17% 365, Sout hern 5 SOU Kanye 28, ,775 11% 228, Source: PAD. Panel A NACA Panel B BAIS Panel A: New infections and mortality. Panel B: Prevalence by gender and age. 98 HIV prevalence is heterogeneous, with the highest prevalence recorded in Selebi-Phikwe and North East. The literature notes the lack of age-and location-adjusted prevalence maps that could be used for targeting HIV educational programs and efficient allocation of resources to higher risk groups (Kandale et al., 2012). 72

90 Table A10-2: HIV/AIDS Expenditures, external & government sources ( ) 99 HIV/AIDS Expenditures: External & GoB Sources ($US) HIV/AIDS Expenditures: External & GoB (% ) USG $ 76.0 $ 93.0 $ 90.0 $ % 70.6% 58.3% 55.9% ACHAP $ 13.0 $ 26.0 $ 56.5 $ % 19.7% 36.6% 35.1% GF Round 2 $ 9.6 $ - $ - $ - 8.4% 0.0% 0.0% 0.0% UNICEF $ 0.8 $ % 0.6% UNFPA $ 1.2 $ % 0.4% UNDP $ 2.0 $ % 1.5% DFID $ 2.3 $ 0.3 $ 0.3 $ % 0.2% 0.2% 0.1% Botswana Harvar $ 4.2 $ 4.2 $ 2.8 $ % 3.2% 1.8% 0.6% EC $ 1.3 $ 1.3 $ % 1.0% 0.8% WB Loan $ - $ - $ - $ % PSI $ 3.5 $ 3.5 $ 3.5 $ % 2.7% 2.3% 2.2% Total External $ $ $ $ % 46.7% 50.7% 51.8% GoB $ $ $ $ % 53.3% 49.3% 48.2% Total HIV/AIDS $ $ $ $ Table A10-3: HIV/AIDS prevalence and expenditures in selected countries in Africa Source: UNGASS country report for 2010 for HIV spending ( Studies for Swaziland and Zambia. IMF (2010) for GDP. 99 Global Fund's Round II grant to Botswana was cancelled. The GoB s application for Round VII was not approved. The total request for 5 years was US$36.9 million or, on average, US$7.4 million a year. 73

91 Table A10-4A: BNAPS coverage by public sector, CSOs/PSO No. BNAPS Coverage No. CSO District Coverage 1 South-East District 1 Bobirwa 2 Francistown 2 Francistown 3 Selebi-Phikwe 3 Goodhope 4 Goodhope 4 Kweneng East 5 Kweneng East 5 Kweneng West 6 Kweneng West 6 Palapye 7 Central Serowe-Palapye 7 Selebi Phikwe 8 Kgalagadi South 8 Serowe 9 Ngamiland East 9 South East 10 Ngamiland West 11 Central Bobonong Notes: Column BNAPS Coverage shows the 11 districts that include Phase I and Phase II districts. Column CSO District Coverage Shows the 9 districts covered by CSOs/PSOs under Component 3 through community grants. Table A10-4B: Overview of beneficiary groups by main intervention types Target Intervention Type Intervention Introduced Group Male Female HIV Education and Communication & Behavior Change Communication (BCC) Preventative Treatment HIV Education and Communication & Behavior Change Communication (BCC) Knowledge of HIV/AIDS Transmission & Prevention Abstinence / Late Debut (Risk Elimination) Consistent Condom Use (Risk Reduction) Multiple Concurrent Partnership (MCP) Safe Male Circumcision (SMC) Knowledge of HIV/AIDS Transmission & Prevention Abstinence / Late Debut (Risk Elimination) Consistent Condom Use (Risk Reduction) Multiple Concurrent Partnership (MCP) Age Discordant Partnership Original, PAD 2008 Original, PAD Added emphasis Post MTR, 2012 Original PAD, Male Same as for Male Cohort Original, PAD Same as for Male Cohort Same as for Female Cohort Original, PAD Same as for Female Cohort Female Male Same as for Other Male Same as for other male cohorts but late Original PAD, Cohorts debut is dropped as age increases Female Same as for Other Female Cohorts Same as for other female cohorts but late debut is dropped as age increases and age discordance is less emphatic as age increases. Original PAD, 2008 Pregnant Original PAD, Preventative Treatment PMTCT Women 2008 Infants Preventative Treatment PMTCT Original PAD, 2008 Teachers & Students HIV Education & Communication Training on Pediatric HIV/AIDS Counselling Original PAD, 2008 School Girls Preventative Treatment Cervical Cancer Screening (HPV Vaccin.) Post MTR, 2012 TB Patients Treatment DOTS Original,

92 Prisoners Mobile Populations Truck & Public Transport Drivers TB-HIV/AIDS Integrated Care HIV Education & Communication Preventative Treatment Treatment Prevention, Preventative Treatment, and Treatment Prevention, Preventative Treatment, and Treatment Improved Diagnostics & Treatment Post MTR, 2012 Counselling & Testing SMC (Male) & PMTCT (Female) ART - HIV IEC and BCC interventions per above - SMC - HIV IEC and BCC interventions per above - SMC Original PAD, 2008 Original PAD, 2008 Original PAD, 2008 Original PAD, 2008 Original PAD, 2008 Table A10-5: Key changes in design, structure, and implementation At Appraisal Following MTR & Restructuring Project 5 years (4 effectiveness) 6.5 years (5.5 effectiveness) Duration Geographic Scope & Phases Phase I: 5 Districts (Eastern) Phase II: Nationwide Coverage Phase I: 5 Districts (Eastern) Phase II: Additional 6 Districts (Total 11). CSO Coverage focuses on 9 Districts BNAPS Resource Allocation 100 Public Sector (C2) Resource Allocation CSO Component C1-NACA: 15% C2-Public Sector: 40% C3-CSO & Private Sector: 45% Weak link between allocation and performance. Prior to BNAPS no purchaserprovider split and weak communitybased service delivery. BNAPS introduces competitive process through Call for Proposal (CFP) to select community grant beneficiary CSOs. The CFP approach was to ensure a focused and evidence-based approach. BNAPS introduces performance based contracts, primarily linked to output metrics. Termination of nonperforming CSO by NACA, unprecedented in Botswana. Single-year grant window. C1-NACA: 13.3% C2-Public Sector: 34.8% C3-CSO & Private Sector: 51.9% Within Component 2, budget allocations were linked to performance (output targets). Increased allocations to better performing line ministries (i.e., performance streamlining ), for example, to the MOH. 101 Improved selection criteria and metrics (Capacity Assessment Tool CAT) for the CFP process to select community grant beneficiary CSOs. The CFP was refined based on lessons from implementation. The first two cycles provided the Government lessons which helped to further refine the mechanism. TA from NACA s IEC/BCC to foster the development of effective and innovative interventions to influence behavior. Improved performance based contract management by applying External Results Verification (ERV), implemented by an independent third party. - Multi-year grant window to reduce transaction cost and ensure program continuity for wellperforming CSOs. 100 Allocations across components are based on the audited NACA Budget & Expenditure Data ( ). 101 No trend data are available. The ISRs and Aide Memoires suggest increased allocations toward better performing ministries, with particular emphasis toward the end of the project when the integrated care agenda increased the weight of MOH-managed and implemented interventions. The project monitoring tool was instrumental in improving project management but it had limitations. For example, reporting for for Component 2 by ministry is missing, making the assessment of resource allocations within the component difficult. Yet, it offers insights into share of expenditures by ministries for the period of , and these financial records are more detailed than what is available in standard financial statements. 75

93 Decentralized Management Intervention Focus Results Framework & M&E Platform District Multisectoral AIDS Committees (DMSACs) play an important role in the coordination of activities in the public sector and CSOs. District AIDS Coordinator s (DAC) Office are multisectoral with representations from line ministries and implementing CSOs. DACs monitor and report on progress. District Treasury is used to channel funds. Pure focus on HIV/AIDS prevention, in line with the National Strategic Framework, to balance the formerly treatment-heavy national response. Increased focus on the behavioral dynamics, following the findings from the Social Assessment Survey. Project Specific M&E EU mission contributes to RF (buydown triggers) Delayed and weak M&E framework, with primarily process and output indicators (standard at the time). Lack of standardized M&E tools for BNAPs under the public sector component (C2). Initial lack of M&E tools and information systems to enable performance monitoring for resultsbased contracts with CSOs. General M&E Context RF draws on the MAP s Global Results Scorecard, which had limitations. 102 This affected many countries. Indicators adopted from the NSF s RF, which streamlined reporting but challenged adding project-specific indicators with closer alignment in focus and delivery timeline (based on the principle of using country systems/frameworks). Decentralized management of interventions empowers districts (e.g. TB Training of Trainers) and contributes to strengthened capacity and quality. However, to avoid challenges in management, planning, monitoring, and expenditure tracking, record keeping is essential on inputs and outputs across the administrative and delivery levels. The system established for managing community grants performed well, particularly the use of District Treasuries for financial disbursement and tracking overall expenditures at the district level, implicating that this mechanism could be scaled up. Maintained importance of reducing the propensity to engage in risky sexual behavior (e.g. MCP, inconsistent condom use, etc.). Relative increased in attention to inducing outcomes through biomedical interventions (e.g. SMC), coupled with demand side behavioral communication strategy by CSOs. Broadened focus, which emphasizes the interlinkage between CDs and NCDs and rides on the care delivery integration agenda (e.g. HIV/AIDS and cervical cancer; HIV/AIDS & TB), driven by sector efficiency, value-for-money, and financial/fiscal sustainability arguments. No revision to the Results Framework, despite PDO revision and thematic scope expansion. Improved M&E framework, tools developed for the CSO component to facilitate monitoring of inputs and outputs (use of Gantt chart), and encouraging collecting outcome indicators where possible. In-depth annual review of CSO performance by NACA which was summarized in Close-Out Reports for each CFP cycle. Established a basic customized system for tracking CSO performance. EU s MTR mission assesses indicators and progress External results verification contributes to M&E assessment and strengthening. Efforts to streamline the M&E platform within the MOH, to pilot electronic record keeping for cervical cancer patients (CC Registry), and enable tracing patients between levels of care (referral tracking) to assess the continuum of care delivery. 102 The challenges and modifications related to the MAP s Global Results Scorecard are discussed in the World Bank, Report No.: AFR (2007). The paper notes the need for stronger and functional M&E systems to demonstrate impact, and enable the strategic management of the epidemic and intense supervision to meet the complexity of the national responses to HIV/AIDS. 76

94 Table A10-6: Risks at appraisal, assessed for Phase 1 and Phase 2, and for the future Main Risk Phase Phase PAD Original Risk Dimensions Categories 1 2 Future Establishing World Bank- Client Engagement Limited experience of the Client to work with the Bank and vice versa (M). H M L, Good rapport established and robust engagement platform is built Epidemiology Social Context Institutional Capacity of Coordinating Agency Implementation Platform Financial Management The magnitude of the epidemic may limit the measurability of effectiveness in the short term (H). Continued denial and stigma, and limited government experience in reaching highly vulnerable population groups (M). Coordination, accountability, and capacity limitations within NACA (H). Limited government experience in collaborating directly and channeling significant resources to civil society and the private sector (H). Overly participatory approach could lead to delays and accountability challenges in project implementation (M). Audit of the project financial statements may be delayed beyond 6 months after the end of the government financial year. (M) H H M, Use robust evaluation design and improve HMIS to mitigate M M M, Use targeting mechanism H M H, Due to ongoing institutional change which can affect NACA s role/position. Clarify institutional relationship. H M L, Draw on BNAPS CSO/PS experience and enhanced PBF approach. Use DAC/DMSAC, MoLG. H M M, In addition to the above, use ERV and monitoring tools to ensure accountability. M S N/A Delay in Financial Statements from the District Council level (M). M M M, Use PBF approach with districts and condition on reports. Use resource tracking. Delay in accounting for utilization of funds at the CSO and CBO levels (S). S M M, Use improved expenditure reporting and resource tracking in future. Note: H (High Risk), S (Substantial Risk), M (Modest Risk), L (Low Risk) N (Negligible Risk) 77

95 Table A10-7: BNAPS ISR ratings (ISR 1-13) ISR No MTR Average Date 2/13/09 6/29/09 12/23/09 6/28/10 1/3/11 9/20/11 4/23/12 8/18/12 4/24/13 12/15/13 6/14/14 12/12/14 3/31/15 na Disbursed (% of Total) 0% 0% 0% 4.75% 6% 20% 28% 40% 50% 71% 79% 96% 96% na Disbursed (US$ Million) $ - $ - $ - $ 2.38 $ 3.06 $ 9.97 $14.14 $20.13 $ $ $39.48 $ $48.14 na PDO IP Project Management Financial Management Procurement M&E Counterpart Funding C1 (NACA) C2 (Line Ministries) C3 (CSOs & PSOs) Note: Scale is between 1 and 6. 1 represents the lowest rate and 6 the highest possible performance score. The scores are based on the standard rating definitions used in ISRs. 1 = Highly Unsatisfactory (HU). 2 = Unsatisfactory (U). 3 = Moderately Unsatisfactory (MU). 4 = Moderately Satisfactory (MS). 5 = Satisfactory (S). 6 = Highly Satisfactory (HS). The definitions for the ratings are provided in the ICR Guidelines. Figure A10-2: BNAPS disbursement trend 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Q1-FY-2009 Q2-FY-2009 Q3-FY-2009 Q4-FY-2009 Q1-FY-2010 Q2-FY-2010 Q3-FY-2010 Q4-FY % 6% 6% 8% Q1-FY-2011 Q2-FY-2011 Q3-FY-2011 MTR: May 31 -June 12, % 20% 26% Q4-FY-2011 Q1-FY-2012 Q2-FY % Q3-FY % 40% Q4-FY-2012 Q1-FY % Q2-FY % 66% 63% 50% Q3-FY-2013 Q4-FY-2013 Q1-FY-2014 Q2-FY % Q3-FY % Q4-FY % 100% 95% Q1-FY-2015 Q2-FY-2015 Q3-FY-2015

96 Table A10-8: Comparison of selected indicators from BAIS III (2008) & BAIS IV (2013) 2008 (BAIS III) 2013 (BAIS IV) CHANGE HIV Epidemiology Incidence (%) Prevalence (%) Prevalence in Towns Prevalence in Cities HIV Prevention Testing At Least Once (10-64) (%) Testing Last 12 months (15-49) and Who Know their Results (%) Safe Male Circumcision (SMC) (%) HIV Knowledge & Behavior Correctly Identify 3 Ways of HIV Prevention & Reject Misconception about HIV Transmission (%) Age Group Age Group Early Debut Condom Use during Last Sexual Intercourse with NRP (% ) Age Group Age Group Age Group Condom Use Every Time with Non-Regular Partner (NRP) (%) Multiple Concurrent Partnership (MCP) (% ) Age Group Age Group Age Group Tuberculosis (TB) TB Incidence (WHO 2015 and NTBP Report Data) per 100,000 population TB Mortality in PLWHs 1, TB Case Detection (%) TB Treatment Success Rate (%) Sources: Primary: BAIS III (2008), BAIS IV (2013). Secondary: Where BAIS does not provide estimates, WHO (2015), UNAIDS (2015), and NTBP-MOH (2015) statistics are presented. 79

97 Figure A10-3: Panel A: Knowledge of prevention. Panel B: SMC uptake , , , , , , , , , , Baseline Endline 2 SMC MOH Clinical Data (No. Annual) PDO4 SMC MOH Clinical Data (No. Cumulative) Sources: BAIS II-IV for Panel A. MOH for Panel B. Note: The right vertical axis in Panel A and the grey bars shows the percentage change (%) between the survey data points. The left axis is the scale for the level of knowledge measured in terms of share of the target population (%). In Panel B the dashed line shows the cumulative outcome, the solid line shows the annual uptake. Notes for Panel A: The proportion of respondents years who correctly identify ways of preventing the sexual transmission of HIV and reject major misconceptions about HIV transmission/prevention increased from 36% in 2001 to 38% in 2004, an increase of 2 percentage points over 3 years, against the GoB s target of 90% by For the same cohort, the increase between 2004 (BAIS II) and 2008 (BAIS III) is 2 percentage points, from 38% to 40%. Between 2008 (BAIS III) and 2013 (BAIS IV) the increase is 7.7 percentage points, from 40% to 47.7%. While the progress was larger in this last round, in absolute terms it remains insufficient. Figure A10-4: Selected BNAPS PDO indicators at base- and endline PDO1 Prevention & Rejection (%) PDO1 Prevention & Rejection (%) PDO2a No Sexual Activity (%) PDO2b CondomPDO7a TB Case Use Last Detection (%) Encounter NRP 12mnts (%) 82 PDO7b TB Treatment Success Rate (%) PDO8 Testing at Least Once (10-64) (%) PDO9 Testing Last 12 mnts (15-49) & Know Results (%) Baseline Endline 80

98 81