Ex Post-Evaluation Brief BURUNDI: Health Sectoral Programme II

Transcription

1 Ex Post-Evaluation Brief BURUNDI: Health Sectoral Programme II Sector Basic health infrastructure Health sector programme, Phase II - Programme/Client BMZ No * incl. accompanying measure, AM - BMZ No Programme executing agency Burundian Ministry of Health Year of sample/ex post evaluation report: 2013/2013 Appraisal (planned) Ex post-evaluation (actual) Investment costs (total) EUR million EUR million Own contribution EUR 0.05 million EUR 0.05 million Funding, of which budget funds (BMZ) EUR million EUR million * random sample 2013, ** of which EUR 1.5 million for AM EUR million** EUR million Short description: The sectoral programme comprised two components: component 1 (approx. 2/3 of the Financial Cooperation funds) to promote sexual and reproductive health through the government programme for reproductive health (Programme National de Santé Reproductive PNSR) and component 2 (approx. 1/3 of the Financial Cooperation funds), a social marketing programme through a nongovernmental organisation to prevent the spread of HIV. Both components supported campaigns to raise awareness, change behaviour and provide high-quality contraceptives and condoms but they focused on different issues and target groups. Objectives: The programme aimed to improve the reproductive and sexual health of the Burundian population and reduce the population's strain on the country's limited resources (overall development objective). The programme objectives were to change the population's attitudes and behaviour in relation to family planning and HIV/AIDS prevention as well as to increase the use of modern contraceptives (including condoms). The educational measures carried out in the programme aimed to stimulate demand for condoms and other contraceptives. Improving the availability of condoms and other contraceptives was intended to meet the increase in demand that, in turn, aimed to lower new HIV/AIDS cases and increase the prevalence of contraception. Target group: of the programme was the entire population of Burundi of reproductive age between 15 and 49 years, particularly women (component 1) and young people (component 2). Overall rating: 2 The developmental effectiveness of the programme is rated good overall. Points to note: The programme successfully established a system to distribute contraceptives in Burundi in a post-conflict situation. This system ensures that all healthcare facilities in the public sector have a reliable supply of all contraceptives on the market. A contribution was made to changing central indicators of reproductive health through educational measures and raising awareness. This contribution was highly significant because almost two-thirds of the national Burundian programme was financed by Financial Cooperation. Sustainability Efficiency Rating by DAC criteria Overall assessment 1 Project Development impact Average rating for sector (from 2007) Relevance Effectiveness Average rating for region (from 2007)

2 GENERAL CONDITIONS ANS STATUS OF THE PROGRAMME The civil war in Burundi between 1993 and 2003 impeded the country's economic and social development for more than a decade. At the end of the civil war, the infrastructure was destroyed, skilled personnel had left the country and Burundi was at the lower end of the list of the poorest countries in Africa based on its economic and social data. After a project to improve the health-care infrastructure in several provinces, which was originally planned together with the World Bank in 1995, had to be cancelled shortly after starting (costs of EUR 0.4 million had been incurred by that time), an attempt was made right after the civil war to redesign this programme and direct it towards the country's most pressing health and demographic problems. This ex-post evaluation con-centrates on the assessment of the programme implemented after the war (approx. 96% of the Financial Cooperation funds). EVALUATION SUMMARY Overall rating Overall, the overall development impact of the project was just enough for a rating of good. Rating: 2 Relevance When the programme resumed in 2004, the population of 7.5 million at that time was growing by 3.4% p.a., which, in light of the country's high population density of almost 300 inhabitants per square kilometre, was not manageable. This further intensified the country's conflict situation due to its extreme lack of natural resources. Average life expectancy was only 43.1 years in 2004 due to, among other things, a high rate of maternal mortality of 1,000/100,000 births and a child mortality rate of 190/1,000 children up to the age of 5. The relatively high prevalence of HIV/AIDS of 2.3% was also a cause for concern. Given this situation, it made sense to realign the programme following the conclusion of the conflict the priorities in healthcare had undergone a major change as a result of the conflict. Burundi's very unsatisfactory health statistics at the beginning of the programme and particularly the fertility rate, which is much too high for the existing population density, are evidence of the high developmental relevance of a programme to improve sexual and reproductive health and reduce the population's strain on natural resources. The programme has a somewhat complicated implementation structure with its strategy, on the one hand, to promote family planning and mother-child health through the public sector (component 1) and, on the other, to target young people, who are particularly difficult to reach via the public sector, through social marketing, particularly about HIV/AIDS prevention (component 2). This implementation structure, however, was the right choice because it was the only way to reach the widely varying target groups. Moreover, both programme components mutually benefited from the measures offered in both programme components to raise awareness and change behaviour. 2

3 While the component (1) to promote sexual and reproductive health were easy to coordinate with the activities of other bilateral and multilateral donors through the government PNSR for the entire duration of the programme, the coordination of the activities of the social marketing component with other donor-funded HIV/AIDS prevention and treatment measures via the national AIDS programme was insufficient. The programme's relevance is also evidenced by the fact that its objectives were completely consistent with the priorities of the Burundian government as they are defined, e.g., in the three priorities of the National Health Plans or (Plan Nationale de Développement Sanitaire PNDS) or in the National Strategy Plan to Fight HIV/AIDS (Plan National de Lutte contre le VIH/SIDA). Furthermore, the programme s focus corresponds to the priorities of German development policy and three of the 8 millennium goals relating to improving mothers' health, lowering child mortality and fighting the HIV/AIDS epidemic. As a result, the programme s focus is consistent with the priorities of national and global development policy. Sub-Rating: 1 Effectiveness: The programme objectives were to change the population's attitudes and behaviour in relation to family planning and HIV/AIDS prevention as well as in-crease the use of modern contraceptives (including condoms). The original plan was to use 10 indicators to determine whether these programme objectives had been achieved. These indicators relate to outcome as well as output levels. Consistent with the current state-of-the-art, the outcome indicators are primarily used in the ex-post evaluation (shaded in grey in the following overview). The other established indicators on the lower output level are used as additional information. Indicators Starting value 2006 Target 2010 Outcome June 2013 Component 1 PNSR 1 PNSR 2 PNSR 3 PNSR 4 Number of employees in public healthcare service trained in the application of family planning Percentage of government healthcare facilities that do not have contraceptives in stock Prevalence rate of contraceptives (modern methods without condoms) Number of regular users of modern contraceptives n.a n.a. < 5% < 1.5% 5.8% 10% 23%** 114,

4 PNSR 5 Component 2 PSI 1 PSI 2 PSI 3 PSI 4 PSI 5 Number of women of childbearing age who are familiar with modern birth control methods* Percentage of young people (15-24 years) who know that they can be infected with HIV if they don't use condoms Percentage of young people who can use a condom correctly* Percentage of young people who used a condom during their last sexual contact Percentage of young people who know at least one sales location for condoms in their area Country coverage with condoms from the social marketing programme that are distributed through wholesalers n.a. 15% 98,7% 81% 86% 86% 25% 35% 35% 54% 60% 54% 41% 46% 22% 47% 52% 41% * Important "preliminary stage" for the outcome indicator; the significant difference between the target value and the actual outcome suggests that the target value was too conservative in view of insufficient initial data. **Estimate Sources: for the values for 2010: final reports of Phase II by GFA and PSI. PNSR figures for June 2013 are based on information from GFA. The PSI data for 2011 and 2013 are identical and based on the last available PSI TRAC study from Due to lack of an updated study, no more-current data are available here. The positive changes to all indicators relating to component (1) are evidence of the good results of the education measures for the staff working in reproductive health in government healthcare facilities, of the extremely efficient logistics for the large number of contraceptives provided for free and of the change in the knowledge about and use of modern contraceptive methods, which was far be-yond expectations. Visits to the field have completely verified these results. In contrast, the results of component (2) are less positive. The planned changes were made for the first two indicators that relate to changes in the knowledge of young people. No change occurred in the only outcome indicator, a particularly important indicator, namely the use of condoms for sexual con-tact. The last two indicators relate to country coverage with social marketing condoms and their availability to young people. The values here are even lower than at the outset. A part of this deterioration is likely due to a change in the social marketing agency's distribution policy, which was intended to increase efficiency by using more private wholesalers instead of its own staff for nation-wide distribution. These wholesalers are less motivated to supply remote parts of the country. As a result, the measures to boost the efficiency of the distribution system negatively affected market penetration. Moreover, during the period assessed here, considerable quantities of free condoms from the World Bank and the Global Fund flooded the condom market and covered around 80% of the market. Around 13% of the overall condom market was supplied through the social marketing component and approx. 7% was distributed through component (1) for family planning. The 4

5 high percentage of free condoms leads to the conclusion that the condom supply was highly "oversubsidised". Overall, positive results can be attributed to component (2) in terms of changes in awareness, but the results were unsatisfactory with respect to ensuring market coverage with condoms and, most importantly, with regard to changes in behaviour. The effectiveness of the components to promote sexual and reproductive health can thus be rated very good, but those of the social marketing component are rated not satisfactory. Overall, the programme's effectiveness is rated satisfactory. Sub-Rating: 3 Efficiency The efficiency of such a complex programme to improve reproductive and sexual health is difficult to capture in quantitative terms because the results/benefits of the programme cannot be evaluated in monetary terms, and because there are no standard costs for comparable projects showing whether the programme costs fall into a normal range. The two components are once again rated very differently in terms of efficiency. Component (1) to promote sexual and reproductive health through the PNSR achieved considerable results with relatively limited funding contributions overall. In contrast, the social marketing component, which accounted for approximately 30% of the programme costs, showed only limited impact. Component (1) of the programme included establishing a logistics system for contraceptives for public health facilities, training of staff in the promotion and use of modern contraceptives and creating adapted advertising materials. It is considered very positive that despite these considerable initial investments, the programme was able to accomplish so much over an extended period of almost seven years with limited financial resources overall. There were probably no alternatives to the considerable investments needed to set up the government supply structure. The limited impact of the social marketing component can be explained in part by the highly "oversubsidised" market, i.e. through the approx. 80% market share that is covered by free condoms. In addition, sales activities were increasingly limited to urban areas over the course of the implementation of the programme under Financial Cooperation. This was a result of the shift in the distribution network to private dealers, which was intended to improve the efficiency of component (2). The majority of the country's population, however, lives in rural areas and was thus only reached to a limited extent. Here there was a trade-off between broadscale coverage and more efficiency. But particularly for young people, there are no affordable commercial condoms in Burundi outside of the capital, nor is there any access to free condoms distributed through government channels. This target group can only be reached using a parallel social marketing strategy. 5

6 While the efficiency of component (1) was relatively high, the efficiency of the social marketing component (2) was rated not satisfactory. Overall, the efficiency of the programme is thus rated satisfactory. Sub-Rating: 3 Impact After the programme was redesigned in 2004, the overall objective was set of making a contribution to improving the sexual and reproductive health of the Burundian population as well as limiting the population's strain on natural resources. Indicators to measure whether the overall objective was reached were not defined at the start of the programme. From today's perspective, one would use the maternal mortality rate, the child mortality rate, the prevalence rate of HIV/AIDS (no data is available on incidence) and the fertility rate as indicators to measure the overall objective - all indicators that correspond to the priorities of the Burundian government and also, to a great extent, the millennium goals. For this reason, they are used to assess the overall developmental impact within the ex-post evaluation. Because German development cooperation during the implementation period provided approximately two-thirds of the total funds necessary for the national programme for reproductive health, the improvements in the health indicators can be attributed, at least in part, to the programme. Population growth has slowed and is currently 2.6% p.a. (programme appraisal: 3.5%) a figure, however, that is still very high. Between 2004 and 2012 the fertility rate declined significantly from 6.8 to 4.1 births per woman. Life expectancy in this limited period rose by 7.8 years to 50.9 years. The maternal mortality rate improved to 800/100,000 live births and the child mortality rate (children under the age of 5) to 142/1,000. All of these figures are still not satisfactory compared to other countries in the region, but the significant improvements in the post-conflict period are evidence of a positive trend that is likely to continue. Another positive aspect is the decline in the prevalence of HIV/AIDS from 2.3% to the current figure of 1.3%. However, the social marketing component of the Financial Cooperation probably made only a very minor contribution to this improvement because the indicators for these components yielded inadequate results or changes in terms of outcomes. Furthermore, financing under Financial Cooperation in this area is rather low compared to the contributions of other donors such as the Global Fund, the World Bank and USAID. Due to the programme's targeted activities and the considerable importance of the German Development Cooperation's contribution to finance the programme to improve reproductive and sexual health in Burundi, a clear link can be established between the programme measures and the significant improvement in the central indicators mentioned above over the course of the programme. Little was accomplished in the prevalence rate of HIV, which has to be seen in light of the considerable contribution of component (1). In the ex-post evaluation, we therefore assume that the Health Sectoral Programme II made key contributions to attain- 6

7 ing the overall objective with component (1), but the programme s overall development impact is rated only good due to the only satisfactory rating of component (2). Sub-Rating: 2 Sustainability With respect to the sustainability of the programme, a distinction should be made between three different types of sustainability. It is difficult to answer at the moment whether the partner (Burundian Ministry of Health) is capable of assuming financing of the activities provided by the programme in the future because these activities have been funded so far in follow-up projects of German Development Cooperation. At the time of the ex-post evaluation, it looked as if the Dutch Development Cooperation would provide follow-on financing for both components of the programme under Financial Cooperation. At the moment, Burundi's limited budget for the healthcare sector is certainly not adequate to assume the costs of the activities carried out here. There are, however, a number of indications that the Burundian government will be able to carry out the measures of the component to promote sexual and reproductive health through PNSR with contributions from other donors and, in the medium term, also with increased budget contributions and (planned) insurance systems. In contrast, the social marketing component can only be continued if long-term donors are found that continue to support this component due to the considerable need for subsidies. At the moment, this component is also being continued by the Dutch Development Cooperation for the time being. Overall, the financial sustainability is thus good. The question as to whether the employees of the government executing agency, the PNSR, and the healthcare facilities trained under the programme are able to continue the programme services, i.e. guaranteeing institutional sustainability, can be answered with yes based on the results of the field visits. In addition, the social marketing activities can be continued by the qualified personnel of the locally-based social marketing agency, but only if these personnel can be retained via support from external donors. The answer to the question whether the impact achieved by the programme can be assured over the long run, i.e. its sustained effectiveness, is also yes, at least with respect to a change in attitude. Overall, the programme s sustainability is therefore rated good. Sub-Rating: 2 7

8 Notes on the methods used to evaluate project success (project rating) Projects (and programmes) are evaluated on a six-point scale, the criteria being relevance, effectiveness, efficiency and overarching developmental impact. The ratings are also used to arrive at a final assessment of a project s overall developmental efficacy. The scale is as follows: 1 Very good result that clearly exceeds expectations 2 Good result, fully in line with expectations and without any significant shortcomings 3 Satisfactory result project falls short of expectations but the positive results dominate 4 Unsatisfactory result significantly below expectations, with negative results dominating despite discernible positive results 5 Clearly inadequate result despite some positive partial results, the negative results clearly dominate 6 The project has no impact or the situation has actually deteriorated Ratings 1-3 denote a positive or successful assessment while ratings 4-6 denote a not positive or unsuccessful assessment Sustainability is evaluated according to the following four-point scale: Sustainability level 1 (very good sustainability): The developmental efficacy of the project (positive to date) is very likely to continue undiminished or even increase. Sustainability level 2 (good sustainability): The developmental efficacy of the project (positive to date) is very likely to decline only minimally but remain positive overall. (This is what can normally be expected). Sustainability level 3 (satisfactory sustainability): The developmental efficacy of the project (positive to date) is very likely to decline significantly but remain positive overall. This rating is also assigned if the sustainability of a project is considered inadequate up to the time of the ex post evaluation but is very likely to evolve positively so that the project will ultimately achieve positive developmental efficacy. Sustainability level 4 (inadequate sustainability): The developmental efficacy of the project is inadequate up to the time of the ex post evaluation and is very unlikely to improve. This rating is also assigned if the sustainability that has been positively evaluated to date is very likely to deteriorate severely and no longer meet the level 3 criteria. The overall rating on the six-point scale is compiled from a weighting of all five individual criteria as appropriate to the project in question. Ratings 1-3 of the overall rating denote a "successful" project while ratings 4-6 denote an "unsuccessful" project. It should be noted that a project can generally be considered developmentally successful only if the achievement of the project objective ( effectiveness ), the impact on the overall objective ( overarching developmental impact ) and the sustainability are rated at least satisfactory (rating 3). 8