Final Report The Retrospective Evaluation of ACSD: Benin

Transcription

1 ACCELERATING CHILD SURVIVAL AND DEVELOPMENT (ACSD) Final Report The Retrospective Evaluation of ACSD: Benin Submitted to UNICEF on 7 October 2008 Institute for International Programs Johns Hopkins University Bloomberg School of Public Health Baltimore, MD

2 Disclaimer: This report was prepared by IIP-JHU under contract with UNICEF. All photos were taken by members of the IIP-JHU evaluation team after requesting permission from those who were photographed. All text, data, photos and graphs should be cited with permission from the authors and UNICEF. ii

3 Introduction Summary UNICEF implemented the Accelerated Child Survival and Development (ACSD) project between 2001 and 2005 in 11 countries in Africa with support from Canadian CIDA. The aim of ACSD was to reduce mortality among children less than five years of age by working with governments and other partners to increase coverage with a set of proven interventions. In the high-impact countries of Benin, Ghana, Mali and Senegal, a total of 16 districts worked to deliver the full set of interventions grouped into three packages: EPI+ including vaccinations, vitamin A supplementation and the use of insecticide-treated nets (ITNs) for the prevention of malaria; IMCI+ including promotion of exclusive breastfeeding for six months, timely complementary feeding, use of iodized salt and improved and integrated management at the health facility and community levels of children suffering from pneumonia, malaria and diarrhea, including home-based ORS use, treatment of malaria, and treatment of pneumonia with antibiotics; and ANC+ including intermittent preventive treatment of malaria with SP (Fansidar) for pregnant women (IPTp), tetanus immunization during pregnancy to prevent maternal and neonatal tetanus and supplementation with iron/folic acid during pregnancy and with vitamin A post-partum. An internal evaluation by UNICEF estimated through modeling that the levels of coverage achieved through ACSD were associated with about a 20 percent reduction in all-cause under-five mortality relative to comparison districts in participating districts in four high-impact countries. This retrospective evaluation was commissioned by UNICEF to confirm these findings and provide additional information that could be used in planning effective programs to reduce child mortality and achieve the 4 th Millennium Development Goal (MDG-4) in poor countries in Africa. The IIP evaluation team worked with ACSD managers at international and national levels to develop a generic ACSD framework that defined the pathways through which ACSD activities were expected to lead to reductions in child mortality and improvements in child nutritional status. The generic framework served as the backbone of the evaluation design. The country-specific evaluations also addressed equity across socioeconomic and ethnic groups, for urban-rural residence and for girl and boy children. At the request of UNICEF, the evaluation does not include an economic evaluation or a full assessment of the effects of ACSD on national policy. Aim of the independent retrospective evaluation in Benin The aim of the evaluation was to provide valid and timely evidence to child health planners and policy makers about the effectiveness of ACSD in reducing child mortality and improving child nutritional status in Benin, as a part of the larger retrospective evaluation designed to inform future programs intended to reduce child mortality and accelerate progress toward MDG-4. Equity was also assessed. Two questions served as a guide to the analysis and reporting of the evaluation findings: a. Was ACSD implementation associated with improvements in coverage, nutrition and mortality over time? b. If so, was progress in the ACSD zones faster than that observed for the national comparison area? ACSD implementation in Benin In March 2002, US$1.5 million in support from Canadian CIDA was transferred to UNICEF-Benin for support of ACSD activities. UNICEF staff reported that ACSD activities have been supported since 2003 with funding from Belgium and UNICEF general resources. The Benin ACSD project aimed to build on and complement existing child health activities in the country, with a specific focus on two health zones Djidja/Abomey/Agbangniizoun (DAA) in the department of Zou and Pobè/Adja-Oere/Kétou in the department of Plateau. These six communes are referred to as high-impact zones or HIZs. Together the two HIZs represented an estimated 482,838 people, or 7% of the population of Benin. ACSD i

4 implementation began in late 2002 in both HIZs. ACSD-Benin inputs and activities in the HIZs focused on: 1) Providing essential drugs, supplies and equipment. ACSD-Benin a) provided 19 motorcycles and four vehicles to the HIZs over the course of the project for outreach and supervision activities; b) equipped health facilities with basic medical equipment including delivery kits in 2002 and a kit for newborn care in 2004, refrigerators for the cold chain, and computers for monitoring activities; and c) supplied commodities including vaccines, vitamin A supplements and ITNs for the prevention of malaria. 2) Training and supervising facility-based workers. ACSD-Benin trained 57 health workers in 2002 and 84 health workers in in EPI provision, monitoring and surveillance. Thirty-one facility-based health workers were trained in standard IMCI and their facilities were reported to have received three supervision visits per year between 2003 and Over 200 health workers were trained in healthy child visits in 2003 and As part of the ANC+ component, facility-based workers were trained on IPTp, focused ANC care, management of the neonate and active management of the third stage of labor. There was some additional training on the prevention of mother-to-child-transmission of HIV for facility-based staff in Supervision of workers in provision of ANC was reported to be sporadic and no quantitative data on the frequency or quality of supervision are available. 3) Training, equipping and supervising community health workers. ACSD-Benin provided support for the training of approximately 400 community-based health workers (CHWs) in 200 villages to deliver key messages concerning family practices and to coordinate ITN distribution and retreatment in 2003 and CHWs were supervised annually thereafter. These 200 villages in the HIZs received health kits containing chloroquine, antipyretics, antihelminthic drugs for deworming, ORS, and iron to be distributed trained CHWs. As part of a pilot study in 2005, additional training was provided to 40 of these community-based workers in community management of pneumonia using cotrimoxazole. ASCD also provided support to train community-based women s groups in the promotion of ITNs. 4) Supporting outreach and campaign activities. ACSD-Benin provided support for national and local vaccination campaigns, catch-up vaccination activities, and bednet retreatment campaigns. The major barriers to implementation of ACSD-Benin were reported to be delays in receipt of funds from UNICEF headquarters, insufficient amounts of resources for implementation, and the absence of sufficient incentives (whether monetary and non-monetary) to motivate facility- and community-based health workers to apply the skills learned through training, especially after initial program funds were exhausted. Additionally, MOH and UNICEF program managers reported that the detection of wild poliovirus from Nigeria in 2003 meant that time and resources were diverted to organize an intensive series of national immunization campaigns. Evaluation design and methods The IIP evaluation team worked with UNICEF-Benin, the Government of Benin and other partners to adapt the generic ACSD evaluation design to local implementation characteristics. Relevant data were identified and assessed based on minimum quality criteria. Baseline data on coverage and nutritional status were obtained from the 2001 demographic and health survey (DHS) representative at the provincial level. Endline data on coverage, nutritional status and mortality were obtained from the 2006 DHS. To ensure sufficient sample size in the HIZs, additional households were sampled in early 2007 using methods and procedures as similar to the 2006 DHS as possible. Information on ACSD implementation and on contextual factors that could have affected the evaluation results or their measurement was collected by independent IIP investigators. Methods included review of documents and administrative reports and personal interviews. Inequities by socioeconomic status were determined by comparing priority indicators across quintiles of household assets; results were also stratified by sex of the child, urban/rural residence and ethnic group affiliation. ii

5 The initial analyses described levels and time trends in intervention coverage, nutritional status and under-five mortality in the HIZs. Next, these results were related to levels and trends in a national comparison area, which included the remainder of the Benin, except metropolitan Cotonou and the HIZs. Additional analyses were conducted to explore alternative explanations for the results. All results and interpretations were reviewed with representatives of the Government of Benin and UNICEF-Benin on several occasions. Results Vitamin A supplementation for children 6-59 months and puerpera, and ITN utilization among children and pregnant women increased significantly in both the HIZs and the comparison area. ITN use among pregnant women increased more in the HIZs than in the comparison area. Neither the HIZs nor the national comparison area showed significant improvements in coverage for vaccination, antenatal care, delivery care, case-management or infant feeding over the study period. For postnatal visits, coverage declined in the HIZs and remained stable in the comparison area. For coverage, the answers to the two primary evaluation questions were: (a) ACSD implementation was associated with increases in vitamin A supplementation and ITN utilization among women and children. Most indicators of coverage and practices did not improve over time in the HIZs, and coverage levels at the end of the implementation period were well below the ACSD targets. (b) Overall, there was no consistent increase in coverage in the HIZs relative to the rest of the country. The evaluation results on nutritional status and mortality were consistent with the lack of an effect on most coverage indicators. For nutrition, the findings were: a. The HIZs showed a reduction over time in underweight and wasting prevalence, but not in stunting. b. The lack of change in stunting and the reduction in wasting prevalence over the course of the study period were similar for the HIZs and the comparison area. The reduction in underweight in the HIZs was not observed in the rest of the country, but lack of progress in the national prevalence of underweight was due to a food crisis in the north of the country; once this region is excluded from the comparison area, time trends in underweight reflect what was observed in the HIZs. For mortality, the findings were: (a) There was a non-significant reduction of 13% in under-five mortality in the HIZs between 1999 and 2006, half of the ACSD target of a 25% reduction. (b) In the comparison area, the U5MR declined by 25% during the same period. Analyses of mortality rates by age subgroups within the 0-59 month range also found no evidence that rates fell more rapidly in the HIZs than in the rest of the country. Analyses of inequalities in coverage in showed that poor populations are consistently being underserved, both in the ACSD and comparison area. It was not possible to examine how inequalities changed over time due to small sample sizes at baseline. Inequalities were present even for interventions delivered through community-based outreach and/or campaigns, such as vitamin A supplementation and ITNs. No coverage inequalities were found in by the sex of the child. As observed globally, rural populations were systematically worse off than urban populations. There was some evidence that this gap was smaller in the HIZs than in the comparison area. Differences in coverage and impact between the two major ethnic groups Fon and Yoruba were not consistent and could not be interpreted as systematic disparities. iii

6 Discussion and interpretation In Benin, there were improvements in two important coverage indicators vitamin A and ITNs for children but other important indicators were either stagnant or declined. There was no consistent evidence of an improvement in nutritional status, and mortality declined by 13%, about half of the proposed target. There were no difference in coverage, nutrition or mortality trends in the HIZs relative to the rest of the country. We examined the possibility that external factors might explain the apparent lack of an impact of ACSD through extensive reviews of existing data and interviews with key informants, and were unable to identify any contextual factors that might account for the lack of impact. The retrospective nature of the evaluation imposed important limitations. First, information on ACSD implementation and contextual factors had to be reconstructed from available reports and the subjective recall of program implementers. There is no way to confirm the validity of the results or the extent or direction of possible bias in the documentation of activities. Second, data quality issues limited the evaluation because some of the available data did not meet basic quality criteria. The evaluation team did its best to overcome these limitations by working closely with the in-country implementation team and reviewing all summaries and results with them on successive occasions. How can the finding of no effect of the ACSD-Benin project be explained? Possible explanations generated in collaboration with the in-country team included the following: 1) Weaknesses in program design. Given the cause-of-death profile in Benin, accelerated mortality reduction is dependent on the effective prevention of deaths from malaria, pneumonia, diarrhea and neonatal causes. The potential effect of ACSD-Benin was limited by three basic design weaknesses. First, messages designed to improve infant feeding practices and careseeking for suspected pneumonia were only included in late Only 32% of children with suspected pneumonia (the second largest cause of child deaths in Benin) were taken to a trained provider for care, and this did not change over time. Second, no effective antimalarial treatment was available in the HIZs. National policy recommended artemisinin combination therapy (ACT) as the first-line treatment for fever (presumed malaria). However, ACTs were not widely available during the study period, and both facility and community workers prescribed chloroquine long after resistance levels had reduced the drug s antimalarial effectiveness Finally, no monetary incentives were provided to the community health workers who were expected to deliver lifesaving interventions to mothers and children. ACSD program implementers highlighted this as a major barrier to ACSD effectiveness. 2) The intensity of implementation was lower than needed to achieve coverage or change family practices. The ACSD program in Benin received US$ 1.5 million in initial implementation funds, transferred approximately one year after funds were received by the three other ACSD high-impact countries. This represents approximately US$15 per child less than five years of age in the HIZs over the life of the project. National implementers believe that this delay, in combination with the low level of investment, did not allow coverage to increase to the levels necessary for achieving an impact on mortality. For example, ITNs are a key component of ACSD, but increased by only 20 percentage points in the HIZs, compared to 23 percentage points in the rest of the country. Deaths due to diarrhea are also unlikely to have changed as a result of ACSD implementation, because correct home treatment practices also failed to increase. 3) The implementation strategies failed to reach the poorest, among whom the impact would have been greatest. The evaluation findings indicate that coverage of interventions, even those delivered through campaigns or community-based delivery strategies, remained highly inequitable in The lack of a differential effect of implementation on coverage in the HIZs must be interpreted in light of numerous concurrent initiatives in the rest of the country. These included initiatives to which ACSD contributed directly or indirectly, particularly UNICEF-supported programs for promoting vitamin A supplementation and ITN use by children under five. Despite these combined efforts, however, coverage levels for most of these high-impact interventions remained below 60% in both the HIZs and the rest of the country in The acceleration effects expected by ACSD planners did not occur. iv

7 Despite high levels of commitment and effort by the MOH, UNICEF, and other country partners, the resources available for acceleration under ACSD were too little and too late to result in a major acceleration in coverage for child survival interventions in the HIZs. The evaluation results on mortality, undernutrition and equity are consistent with the lack of an effect on intervention coverage and family practices. In comparison to the rest of the country, where several other initiatives were also being promoted, ACSD was unable to accelerate progress towards child survival in Benin. v

8 Table of Contents 1. The external retrospective evaluation of ACSD in four countries Evaluation methods Characteristics of high-impact and comparison area ACSD as implemented in Benin Coverage and family practices Nutrition Mortality Equity of coverage, nutrition and mortality Conclusions References Appendices A. Description of Benin and high-impact zones B. Methodology for documentation of implementation activities and contextual factors C. Documentation of ACSD implementation in high-impact zones D. Definition of priority indicators E. Comparison of survey questions used for priority coverage indicator calculation F. Methodologies of surveys and other data in Benin G. Tables presenting priority coverage indicators over time for ACSD high-impact zones H. Tables presenting comparisons of priority coverage indicators over time in ACSD high-impact zones and the comparison area I. Tables presenting results for key indicators in the ACSD high-impact zones by sociodemographic characteristics of the population J. Summary of contextual factors possibly associated with coverage outcomes K. Description of methodological challenges L. Tables presenting additional nutritional analyses M. Tables presenting additional equity analyses N. References for the appendices O. Annotated list of documents reviewed in the ACSD evaluation (file available upon request) vi

9 Acknowledgements This evaluation could not have been conducted without full participation of the representatives from the Ministry of Health, the national statistics offices, and UNICEF-Benin, who formed the Benin ACSD evaluation team. We thank them for their commitment to child survival, as reflected in their willingness to share their time, as well as information and their personal opinions about the contributions and limitations of the ACSD project. We specifically would like to thank Alban Quenum and Gilbert Vissoh from the Ministry of Health who provided insights throughout the evaluation. The national statistics office (Institute Nationale de la Statistique) carried out surveys integral to this evaluation; we especially thank Elise Ahovey who collaborated throughout the evaluation process. UNICEF-Benin staff were responsible for working with governments and partners to implement the ACSD project and collaborate in activities related to the independent retrospective evaluation and we thank them for their commitment to child survival and to the evaluation process as a means of improving program effectiveness. Andrée Cossi was an essential resource throughout the evaluation, we are truly grateful for her dedication to this evaluation process and her strong commitment to using data to improve programs. We would also like to express our appreciation to Souleymane Diallo, Philippe Duamelle, Marianne Clark-Hattingh, Paul Adovohekpe, Hortense Kossou, Dominique Robez-Masson, Arnaud Houndeganme and Loukmane Agbo-Ola. UNICEF-Benin also provided financial support for the supplemental survey and advanced technical assistance from Macro, International. This support was essential, as without it there would have been few data to analyze. We would also like to thank UNICEF staff at regional and global levels for their efforts to provide us with documentation about ACSD and the values and conceptual frameworks that guided its implementation. Additionally, we would like to thank the members of the IIP-JHU for their insights and help throughout the evaluation, as well as Macro International and Trevor Croft for technical assistance. Suzanne van Hulle provided valuable assistance in the documentation of ACSD implementation and contextual factors. Finally, we thank the leadership of UNICEF and CIDA, for their continuing commitment to the importance of independent evaluations and their efforts to see that this work was completed. vii

10 Acronyms ACSD ACT ANC ANC+ BASICS Accelerated Child Survival & Development Project Artemisinin combination therapy for use in treating fever/malaria. Antenatal care One of the ACSD intervention packages, consisting of antenatal care and the intermittent prevention of malaria during pregnancy (IPTp) Basic Support for Institutionalizing Child Survival, a project supported by the United States Agency for International Development. CFA Central and West African Francs, the currency used in Benin, Mali and Senegal. On 7 January 2007, Forex quoted the exchange rate as USD1 = CFA 504. CDC CHW CIDA C-IMCI DAA DHS DPT EPI EPI+ F-IMCI Hib HIZs IEC IIP IMCI US Centers for Disease Control and Prevention Community health worker Canadian International Development Agency Community component of Integrated Management of Childhood Illness Djidja/Abomey/Agbangniizoun health zone (DAA) in the Zou Collines region of Benin; one of the ACSD high-impact zones Demographic and Health Surveys (DHS), supported by USAID. Diphtheria, Pertussis, Tetanus immunization Expanded Program on Immunization One of the ACSD intervention packages, consisting of the full EPI schedule as well as the provision of vitamin A and deworming twice each year for children aged six to 59 months, and the provision of insecticide-treated nets for the prevention of malaria. Facility component of Integrated Management of Childhood Illness, which includes improving the skills of facility-based health workers as well as strengthening aspects of the health system needed to provide appropriate care for children less than five years of age. Haemophilus influenzae type b immunization High-Impact Zones of ACSD implementation, including Djidja/Abomey/Agbangniizoun health zone (DAA) in the Zou Collines region and Pobè/Adja-Oere/Kétou (PAK) health zone in the Ouémé-Plateau region of Benin Information, Education and Communication The Institute for International Programs at JHU Integrated Management of Childhood Illness viii

11 INSAE IPTi IPTp ITN JHSPH MBB MDG MDG-4 MICS MOH NID NGO ORS ORT PAK PBT PMTCT PNLP pp PROLIPO PSI RHF SP TT U5MR Institute Nationale de la Statistique Intermittent preventative treatment for malaria in infancy Intermittent preventative treatment for malaria in pregnancy Insecticide-treated net The Johns Hopkins University Bloomberg School of Public Health Marginal Budgeting for Bottlenecks, a tool developed by UNICEF and the World Bank to support results-based planning for maternal, newborn and child survival in developing countries. Millennium Development Goal The fourth millennium development goal, which aims to reduce mortality among children less than five years of age by two-thirds from levels in Multiple Indicator Cluster Survey designed by UNICEF Ministry of Health National Immunization Days Non-governmental organization Oral Rehydration Salts, usually pre-packaged in a sachet Oral Rehydration Therapy, can be either pre-packaged in a sachet or prepared in the home Pobè/Adja-Oere/Kétou health zone in the Ouémé-Plateau region of Benin; one of the ACSD high-impact zones Preceding birth technique a simplified approach to estimating young child mortality Prevention of mother-to-child transmission of HIV Programme National de Lutte contre le Paludisme; National malaria control program percentage points Projet de Lutte Integré contre le paludisme dans l OuéméPlateau (MOH-led malaria control project in OuéméPlateau region in Benin, implemented by CDC with support from USAID. Population Services International Recommended home fluids, for the treatment of childhood diarrhea A combination of two drugs, sulfadoxine and pyrimethamine. This drug combination is commonly known as Fansidar. Tetanus toxoid vaccination Under five mortality rate ix

12 UN UNICEF USAID United Nations United Nations Children s Fund United States Agency for International Development x

13 1. The external retrospective evaluation of ACSD in four countries UNICEF implemented the Accelerated Child Survival and Development (ACSD) project between 2002 and 2005 in 11 countries in Africa with support from Canadian CIDA and other partners. The main objective was to use results-based planning techniques to increase coverage with three packages of high-impact interventions known to reduce child mortality (see Box 1). In Benin, Ghana, Mali and Senegal, 16 high-impact districts worked to deliver all three packages; in the remaining countries, the focus was on the EPI+ package that included Box 1: vaccination, Vitamin A and insecticide-treated nets (ITNs) for the prevention of malaria. Internal UNICEF evaluations in 2003 and 2004 showed increases in coverage for the EPI+ package in all countries; UNICEF modeled the associated reductions in mortality using the "Marginal Budgeting for Bottlenecks " (MBB) tool and estimated an overall mortality reduction of 20% in the high-impact districts in the four countries, relative to comparison districts. 1 UNICEF and the evaluation team recognized the limitations of a retrospective evaluation, including the difficulties associated with reconstructing project assumptions and activities on a post hoc basis, and making the best possible use of available data and information despite their shortcomings. Readers are reminded to treat the results with caution. 2 The aim of the evaluation is to provide valid and timely evidence to child health planners and policy makers about the effectiveness of ACSD Phase I in reducing child mortality and improving child nutritional status. The specific objectives are: 1. To evaluate the impact of ACSD on mortality and nutritional status among children under five. 2. To document the process and intermediate outcomes of ACSD and results-based planning as a basis for improved planning and implementation of child health programs. 3. To document the contextual factors necessary for effective implementation of efforts to reduce child mortality in order to be able to extrapolate evaluation findings to other settings. ACSD High-Impact Implementation Packages* Immunization plus (EPI+) Routine immunization and periodic measles catch-up and mop-up Vitamin A supplementation bi-annually Distribution and promotion of Insecticide Treated Nets for all children who are fully vaccinated as well as pregnant women, and re-dipping of bednets every six months Improved management of pneumonia, malaria and diarrhea (IMCI+) Promotion of exclusive breastfeeding for six months, timely complementary feeding Improved and integrated management (at the health facility, community and family levels) of children suffering from ARI, malaria and diarrhea, including home-based ORS use, treatment of malaria with anti-malarial blisters, and treatment of ARI with antibiotic blisters Promotion of household consumption of iodized salt Antenatal Care (ANC+) Intermittent preventive treatment (IPT) of malaria with SP (Fansidar) for pregnant women Tetanus immunization during pregnancy to prevent maternal & neonatal tetanus Supplementation with iron/folic acid during pregnancy and with vitamin A post-partum. * UNICEF grouped these interventions into paragraphs in different ways at various points during the project; we have adopted the grouping used in the final report from UNICEF to CIDA for the ACSD project in To assess the process, outcomes and impact of ACSD and results-based planning on socioeconomic, ethnic and gender inequities. 1

14 Achievement of these objectives should also expand regional and global capacity for large-scale effectiveness evaluations of strategies, programs and interventions designed to improve child health in low-income countries. 1.1 Evaluation design Geographic focus. The global retrospective evaluation covers the four countries within which UNICEF defined selected districts or zones as high-impact for the ACSD project. Within each country, the evaluation focuses on these high-impact zones (HIZs). Development of a generic impact model for ACSD. The first step in any evaluation is to define what those implementing the project expect to happen as a consequence of project activities. The evaluation team developed an impact model that specifies the pathways through which UNICEF and implementing countries expected ACSD activities to result in reductions in child mortality. 3 Figure one presents the generic ACSD impact model in two parts. Figure 1A shows the top of the framework describing expected ACSD inputs and processes from the point of introduction at national level in a country through the definition of the three packages of interventions recommended for accelerated implementation (see Box one for a description of the three packages). We derived the top of the framework from ACSD documents e.g.,4 and discussions with ACSD implementers at all levels. Figure 1B shows the bottom of the framework, defining the pathways through which each of the three packages was expected to result in reductions in under-five mortality and improvements in the nutritional status of infants and young children. ACSD documents did not describe the pathways in the bottom of the model in detail, but made reference to other sources where the effects of the interventions are defined and quantified. 5,6 For the internal evaluation, 1 UNICEF utilized the estimates of effectiveness published in these sources and changes in intervention coverage as the basis for modeling the impact of ACSD on child mortality. A central tenet of the evaluation is that the coverage, family practices and impact reflected in the bottom of the framework cannot be attributed to ACSD alone. UNICEF and country partners designed ACSD to reinforce existing activities in child survival by the government of each country and its partners. Therefore, increases or decreases in coverage and mortality must be understood as the result of a combined implementation effort, tempered by contextual factors. A key challenge for the current evaluation is to arrive at a qualitative assessment of ACSD s role as a part of this overall effort. Quantified attribution of the results to ACSD alone is not warranted given the implementation approach. Definition of priority indicators for coverage and family practices. Priority coverage indicators address the prevalence of key family practices and intervention coverage for each of the elements defined in the bottom of the framework. Although some of these indicators reflect behaviors such as exclusive breastfeeding and complementary feeding rather than intervention coverage, these will be referred to as coverage indicators throughout the text. Priority indicators of coverage utilized in the evaluation are defined in Appendix D. Whenever possible, the ACSD priority coverage indicators are consistent with those supported by a consensus of United Nations (UN) agencies and multi- and bi-lateral partners for tracking progress toward MDG-4. 7,8 Where no international consensus indicator exists, the evaluation team contacted technical experts in the topical area to obtain advice on selection of a valid coverage indicator that could be calculated using the data available in Benin. Definition of priority indicators of impact (nutrition and mortality). The main objective of the ACSD project was to reduce mortality among children less than five years of age. The primary impact indicator is the under-five mortality rate, expressed as the probability of dying between birth and exact age five years. Additional priority indicators include neonatal and infant mortality. Some ACSD project documents described expected improvements in child nutritional status, reflecting 2

15 the synergy between undernutrition and infectious disease. 9 Thus, priority impact indicators include prevalence of stunting, wasting and underweight. Appendix D presents the definitions of the priority indicators for mortality and nutritional status. Equity. As part of the evaluation, we examine inequity in coverage and impact indicators, including socioeconomic status, sex of the child, place of residence (urban or rural) and ethnic groups. Documenting contextual factors. Contextual factors are defined as variables that can confound the association between the delivery of interventions and their health impact, or modify the effects of the approach. 10 We documented contextual indicators in the HIZs and comparison area, including: (1) indicators of implementation-related contextual factors such as characteristics of the health system (e.g., utilization rates), child health policy, drug policy, and availability of drugs; and (2) indicators of impact-related contextual factors including baseline levels and patterns of child morbidity and mortality that can affect the potential magnitude of program impact. 10 Economic evaluation. At the request of UNICEF, the evaluation does not include an economic component. 3

16 Figure 1A ACSD impact model: Top model showing inputs and processes Results-based approach Selection of effective interventions Choice of delivery channels Building upon what exists Establishing partnerships Procurement of supplies HW training and supervision (facilities) CHW training and supervision IEC activities Immunization + Antenatal care + IMCI + Bottom model showing interventions to impact ACSD impact model: Bottom model showing interventions to impact Figure 1B Top model showing inputs and processes Immunization + Antenatal care + DPT, Hib, measles vaccines Vitamin A supplementation High attendance at facilities/outreach sessions; deployment at community level Increased coverage Insecticide treated nets IPT for malaria Iron/folic acid Tetanus toxoid High attendance at facilities/outreach sessions Increased coverage Post-partum Vitamin A Pneumonia Measles Meningitis / sepsis Diarrhea Malaria Preterm delivery Neural tube defects Neonatal tetanus????? Spillover effect (co-morbidity) Improved nutrition Reduced mortality IMCI + Reduced mortality? Improved nutrition? Malaria treatment ORT Pneumonia treatment Breastfeeding promotion Deployment of interventions at community level Increased coverage Malaria Diarrhea Pneumonia Other infections Spillover effect (co-morbidity) Reduced mortality Improved nutrition 4

17 2. Evaluation Methods 2.1 Evaluation design Overall design. The overall design was retrospective, drawing on existing population-based surveys with over-sampling in the two high-impact zones (HIZs), commissioned for the purpose of this evaluation. We re-analyzed data sets whenever possible to ensure that the indicator definitions were correct and consistent. Preliminary results were reviewed in meetings of the evaluation team with representatives of the Benin Ministry of Health, the National Statistics Unit and the UNICEF country office in Cotonou, Benin in August 2007, Dakar, Senegal in October 2007 and Bamako, Mali in June Coverage and family practice indicators. We reanalyzed existing household surveys to calculate the ACSD priority coverage and family practice indicators. As described above, these indicators are consistent with those used internationally for monitoring progress toward the Millennium Development Goals 7,8 and are presented in Appendix D. Appendix E provides the specific survey questions utilized for indicator calculations. Nutrition and mortality indicators. We reanalyzed existing household surveys to calculate the priority nutrition indicators using the 2006 WHO Growth Standards. 11 Appendix L and section 6 present more details on these methods. For calculation of priority mortality indicators, the evaluation team analyzed mortality retrospectively, using direct child mortality estimates based on full birth histories. Intervention areas. The two health zones selected for ACSD high-impact implementation served as the intervention areas. They are: 1) Djidja/Abomey/Agbangniizoun (DAA) in the Zou department and 2) Pobè/Adja-Ouere/Kétou (PAK) in the Plateau department. Throughout the body of the report we refer to these zones as highimpact zones (HIZs). Comparison groups. The main comparison group is the remainder of Benin excluding Cotonou. We have excluded Cotonou because access to services and living conditions in this metropolitan area differs considerably from the mostly rural HIZs. Intervention activities. We documented the timing and scale of intervention activities using information collected from key informant interviews and document review, such as administrative and supervision reports and monitoring data. (See Appendix B for details). Equity. To examine inequities, we performed analyses of selected intervention coverage indicators and impact measures stratified by sub-groups of the population, including household assets (expressed in quintiles), sex of the child, place of residence (urban/rural) and ethnic group. 5

18 Contextual factors. We collected standard information on contextual factors, defined above, in order to assist in interpretation of the results and the potential contributions of ACSD. Certain elements, such as economic status, ethnicity and access to clean water were reanalyzed for HIZs and comparison area using existing household survey data. Field visits to the HIZs, key informant interviews and document review provided contextual information not available in existing surveys. Appendix B provides further details on the methods used to collect contextual factors. 2.2 Data sources and methods Tables 1a and 1b summarize the different types of information used in the evaluation. The primary data sources for estimates of intervention coverage and nutritional status were the demographic and health surveys (DHS) conducted in 2001 and 2006, and a supplemental survey linked to the 2006 DHS that over-sampled in the two ACSD high-impact zones (table 1a). These surveys used highly comparable methodologies with data quality controlled by Macro International, although the sample size in 2001 was much smaller than that in We used the DHS to estimate child mortality both before and after ACSD implementation. The full-birth history method used to collect mortality data allows the calculation of period estimates of mortality ranging from the previous 12 months to 10 or more years in the past. The oversampling of the DHS allowed for more precise child mortality estimates. Section 7 describes the mortality analysis methods in detail. Other survey data were available, but given lesser prominence in the analyses, because they did not fully meet the quality criteria established for the evaluation. These criteria were: 1) full data sets and documentation, including sampling weights, available to the evaluation team so that the data could be reanalyzed using the standard definitions for priority indicators; and 2) no more than 5% missing values on key socio-demographic variables (e.g., child age) or the variables needed to construct the priority indicators. We did not use the data from the Benin Cooperation Coverage 2005 and CDC-ACSD 2003 surveys in the primary analyses because they did not fulfill these criteria; data from the 2003 and 2005 surveys are presented to explore time trends between 2001 and Appendix F provides descriptions of the methodology and conduct of all surveys presented in the evaluation, as well as overviews of other data not utilized in the evaluation. Full documentation of 2003 ACSD-CDC survey data quality issues is available upon request from JHU evaluation team. Table 1b presents sources of information used in the documentation of intervention activities and contextual factors. We collected information through: 1) key informant interviews; 2) review of documents, including administrative and monitoring reports; and 3) searches and review of published and grey literature. Available information regarding ACSD expenditures from 2002 to 2004 was utilized to confirm the documentation of ACSD implementation. Technical staff at UNICEF-Benin provided input and revisions throughout the process of documentation. Due to the retrospective nature of the evaluation, it was difficult to collect complete and standardized information on ACSD implementation and other health activities in the HIZs. Furthermore, the collaborative nature of ACSD made it difficult to distinguish which activities were: 1) carried out as part of the ACSD program; 2) carried out with only partial technical and/or financial support from the ACSD program; or 3) carried out by ACSD partners, but independent of the ACSD program. Documents providing full descriptions of the ACSD activities were not available for all activities; in many instances, we relied on summative reports and presentations for this information. Sometimes, although not often, information in one document conflicted with information found in other sources. 6

19 Table 1a: Data sources for the independent retrospective evaluation of ACSD in Benin - populationbased surveys. TYPE OF DATA DESCRIPTION USE IN EVALUATION Population-based surveys that met inclusion criteria DHS 2001: Nationally representative household survey conducted from June to December Used to establish baseline levels of priority coverage and nutrition indicators Other populationbased surveys DHS 2006: Nationally representative household survey conducted from August to December Supplemental survey in DAA and PAK zones: Additional 1540 households in HIZs surveyed using DHS methods in May CDC-ACSD 2003: Household survey of 2610 households in HIZs carried out from August to September Benin Cooperation Coverage Survey 2005: Household survey in Ouémé-Plateau & Zou-Collines departments, including 1097 households in HIZs, carried out in January Used to estimate coverage, nutrition and mortality indicators in comparison area in Used in combination with data collected in DAA and PAK zones by DHS 2006 to estimate coverage, nutrition and mortality indicators in HIZs Reported, but given limited weight in analysis due to concerns about data quality. Reported, but given limited weight in analysis due to concerns about comparability. 7

20 Table 1b: Data sources for the independent retrospective evaluation of ACSD in Benin - routine data, administrative reports and key informant interviews. TYPE OF DATA DESCRIPTION USE IN EVALUATION Routine health information system data Routine data collected through health facilities pertaining to intervention coverage, compiled at local, regional and national levels Documentation of MOH and ACSD activities Administrative reports Job aids and tools Summary reports & presentations Survey reports, maps & other documents Key informant interviews Working discussions Training and workshop reports: Over 35 summative reports pertaining to training of trainers, training of health providers and community workers, and workshops to develop strategies, materials and capacity. Supervision and monitoring reports: Over 20 summative reports describing supervision and monitoring activities and findings. Administrative and routine activity reports: Over 50 documents pertaining to ACSD and MOH planning and activities, including notes from routine meetings, ACSD consultant activity reports, outbreak investigations, etc. Compiled ACSD expenditure data : List of UNICEF-Benin ACSD expenditures compiled as part of the internal evaluation. Job aids and tools, such as visual aids and register books, used in the implementation of ACSD were collected and reviewed where possible Over 30 reports and presentations compiled by UNICEF and partners summarizing the activities, results and challenges of ACSD and other child survival activities Over 30 survey reports, maps, and other documents pertaining to contextual factors (e.g., 2002 Census report 16 ) and child survival activities in Benin were collected during field visits and through literature searches Over 15 interviews and focus groups during field visits conducted with key informants in PAK, DAA and Cotonou during field visits and missions; see list of informants in Appendix B. Collaborative discussions in Cotonou (Aug 2007), Dakar (Oct 2007), and Bamako (June 2008) to review preliminary results and refine analyses with UNICEF-Benin staff, MOH officials, and Benin statistical agency (INSAE) staff. Documentation of ACSD and partners activities. Documentation of ACSD and partners activities. Documentation of ACSD activities. Documentation of contextual factors. Documentation of ACSD activities and contextual factors. Discussion and documentation of ACSD activities and contextual factors; interpretation of results. 8

21 2.3 Analysis We have employed the Habicht et al. framework 17 for real-life evaluations. Starting with an adequacy evaluation, we assessed whether trends in coverage, nutrition and mortality indicators moved in the expected direction within the ACSD areas, and whether goals were met. Next, we carried out a plausibility evaluation, in this case a controlled, non-randomized study that assessed whether observed impact could be attributed to program implementation. ACSD in Benin was a combination of separate interventions vaccines, mosquito nets, vitamin A supplementation, etc. that are highly efficacious if delivered at optimal coverage. This evaluation does not assess the efficacy of these interventions, but instead focuses on their impact when delivered under routine conditions. We worked with national counterparts to conduct the analysis of coverage and nutrition in four steps. i Each step is explained below. Step 1: Generating indicator levels for each survey in the analysis. Objective: To describe levels of priority indicators for coverage and nutrition in all surveys included in the analysis, overall and for specific subsets of children defined by age, sex, geographic location of the household, mothers education and socioeconomic status, where sample sizes permit. We applied standard indicator definitions to the reanalysis of all datasets to ensure the comparability of indicators over different surveys. For each indicator, only data for women and children with known responses were included in the analyses; cases with missing or unknown data were excluded. The point estimates of indicators presented here may therefore differ slightly from those calculated using standard DHS and MICS tabulation programs, which do not exclude missing cases from the analysis. Step 2: Comparing rates of change over time within each ACSD zone ( time trends ). Objective: To determine whether there are statistically significant differences in indicator levels within HIZs from before ACSD was implemented to after ACSD was implemented in ACSD areas, with a midpoint during the process of implementation where adequate data are available, overall and for specific subsets of children. This step refers to the adequacy evaluation. Step 3: Comparing rates of change between ACSD and non-acsd zones within each country ( time trend with comparison ). Objective: To determine whether there were statistically significant differences in the rates of change for indicator levels between the HIZs and a defined comparison area where ACSD was not implemented (the comparison area is comprised of the rest of Benin, excluding Cotonou and the HIZs), overall and for specific subsets of children. Step 4: Attributing improvements to ACSD and related child survival activities at country level. Objective: To determine whether any statistically significant changes in indicator levels can be attributed to ACSD activities, including activities implemented by others in collaboration with ACSD and the national child survival plan, overall and for specific subsets of children. Steps 3 and 4 refer to the plausibility evaluation, assessing whether progress was greater in the HIZs than in the comparison area, and whether or not external factors can account for these differences. For all comparisons of coverage and prevalence of undernutrition across time and geography, we initially calculated a simple chi-square statistic of difference. The simple chi-square statistic does not take into account the design effect of the survey, thus it under-estimates the variance. If no statistical differences were observed using the simple chi-square statistic, we assumed that none would be observed after the design effect was taken into consideration (adding to the variance) and that the groups were therefore not statistically different from one another. For comparisons with a significant chi-square, we calculated standard errors and 95% confidence intervals that take into account the survey design effect, using the Taylor Linearized Variance method. We used a difference-in-differences approach to compare whether i Section 7 explains the analysis of mortality in more detail. 9

22 the change in each indicator over time differed significantly between the HIZs and comparison area for the plausibility analyses. 10

23 3. Characteristics of the high-impact zones and comparison area This section presents pertinent characteristics of Benin as a whole and the HIZs and comparison area. We emphasize differences between the HIZs and comparison area, as well as factors that have changed over the evaluation period to help guide the interpretation of evaluation results. Some of the quantitative results (table 2) presented here are based on our reanalyzes of available survey data, because these provide the most recent information disaggregated by the HIZs and comparison area. Appendices A and J present additional information on the geographic, socio-demographic, economic, health and health service factors in Benin and the HIZs. Figure 2: Map of Benin and its 3.1 The Benin context neighbors The Republic of Benin has an estimated population of about seven million, divided into 12 departments and 77 communes, 16 with an estimated under-five mortality rate of 125 and an infant mortality rate of 67 in the five year period preceding the 2006 DHS survey. 18 Benin (then called Dahomey) became a territory of France in 1946, and declared independence in A succession of military leaders ruled Benin until Major Mathieu Kerekou seized power in 1972 and gradually restored civilian rule. Political newcomer Thomas Boni Yayi, former head of the Togo-based West African Development Bank, won the second round of presidential elections in March 2006, gaining more than 74% of the vote. There have been no major armed conflicts involving Benin since independence. To the north, there have been sporadic clashes along Benin's border with Burkina Faso primarily due to land disputes between rival communities on either side of the border. Box 2: Overview of child health in Benin Injuries 2% HIV/AIDS 2% Measles 5% Causes of under-five deaths in Benin* Pneumonia 21% Neonatal 25% 1990 Malaria 27% Diarrhea 17% 2006 Thousands of Togolese refugees fled to Benin in 2005 following political unrest in their homeland. Benin requested and received international aid to help shelter and feed the exiles. Although Benin s economy has grown over the past few years and it is now one of Africa's largest cotton producers, it ranks among the world's poorest countries. The economy relies heavily on trade with its eastern neighbor, Nigeria. Benin is a member of the African Financial Community, and exports cotton and palm oil. The World Bank estimated the GNI per capita in 2006 as US$ Appendix A provides more details about Benin and the HIZs where ACSD was implemented. 3.2 Child health in Benin The population of children under age five in Benin Mortality rates (per 1000 live births)** was estimated at about 1.4 million in 2000, and had Under-five increased to 1.5 million by The under-five Infant 88 mortality rate has decreased from 185 deaths per Prevalence of undernutrition*** 1000 live births in 1990 to 148 deaths per 1000 live Stunting (% mod + severe) 44 births in 2006, a reduction of 20%. This rate of Underweight (% mod + severe) 20 reduction falls short of that needed for Benin to Sources: *WHO, ; **SOWC 20 ; ***DHS achieve the two-thirds reduction from 1990 levels defined by the fourth Millennium Goal (62 per 1000 live births). Box two shows the major causes of under-five deaths in Benin in 2003, as reported by 11

24 WHO. 19 The major causes are malaria (27%), pneumonia (21%) and diarrhea (17%), with relatively low proportions of deaths due to measles (5%) and HIV/AIDS (2%). One-quarter of all under-five deaths occur in the neonatal period, Among these deaths, infections account for approximately one-third (34%) with the remainder attributed to preterm births (28%), asphyxia (19%) and other causes representing less than one in ten neonatal deaths. Child undernutrition is also a major problem in Benin. 22 Estimates from 2006 using the new WHO growth standards indicate that 44% of children are either moderately or severely stunted, 9% are wasted, and 20% are underweight. Appendix A includes the full profile of maternal, newborn and child health in Benin from the Countdown to 2015: 2008 report Selection of the ACSD high-impact zones in Benin Two health zones (zones sanitaires), equivalent to health districts in other countries, were selected for high-impact implementation of ACSD: Djidja/Abomey/Agbangniizoun (DAA) health zone in the Zou department and Pobè/Adja-Ouere/Kétou (PAK) health zone in the Plateau department. Each of these health zones is comprised of three communes. Thus, the high-impact zones (HIZs) cover six communes out of 77 communes at the national level in Benin and in 2004 represented an estimated 515,515 people or 7% of the total population of Benin. 23 Figure 3 presents the HIZs and comparison area. UNICEF-Benin reports that the two HIZs were selected due to high levels of underfive mortality and poor access to health services. UNICEF was already supporting activities in two communes (Kétou and Djidja) in these zones, which was another consideration. Figure 3: Map of Benin showing high-impact zones and geographic comparison area. 12

25 3.4 Socio-economic and demographic factors Figure 4 presents the incidence of non-monetary poverty in the country as a whole and in the departments which include the HIZs, as measured in Benin in the 2002 census. 24 Within the department of Zou, Djidja and Agbangniizoun are among the poorest communes in the country, while the more urbanized Abomey is much wealthier (shown in the call-out box). The Plateau department is better off compared to the rest of the country; however, the PAK zone is poorer than the department as a whole, with Adja-Ouere commune among the poorest communes in the country. Figure 4: Map of non-monetary poverty in Benin showing high-impact zones and geographic comparison area. Source: Vodounou et al. Carte de pauvreté non monétaire au Benin Table 2 presents selected characteristics of these areas at two points in time: 1) in 2001, before the implementation of the ACSD project had begun; and 2) in , after the implementation of the project. Additional details on the characteristics of the HIZs are available in Appendix A. Appendix table D2 presents the definitions and calculation of selected contextual variables presented in table 2. The results show differences between the HIZs and the geographic comparison area that need to be considered in the analysis and interpretation of the results. Households in the HIZs tended to be less poor than in the comparison area based on their household assets, although this difference was not statistically significant. A higher proportion of households in the HIZs were of the Fon ethnic group. A secondary analysis of the characteristics of the Fon relative to the other ethnic groups in the 2001 sample showed that they are significantly more likely to fall into the least poor quintile of the population. This is consistent with the overall pattern of results presented in Table two, suggesting that households in the HIZs had somewhat more resources than those in the comparison area. Education and literacy among women was similarly low in both the HIZs and comparison area. Approximately one-third of women reported any education in 2001 and and less than one-fourth of women reported literacy during this same period. 13

26 The HIZs and comparison area have similar proportions of rural residences, with about 70% of households rural and 30% urban (table 2). Households in the HIZs tend to have better water supplies (p=0.05) and sanitation facilities (p>0.05), which is likely due to Guinea worm projects that have focused on improved water sources and hygiene in these areas. Table 2: Selected characteristics of high-impact zones and comparison area (all other areas of the country excluding the HIZs and Cotonou) as measured in the DHS 2001 and the DHS and supplemental surveys, Benin DHS 2006/7 DHS HIGH IMPACT COMPARISON HIGH IMPACT COMPARISON ZONES AREA ZONES AREA INDICATORS n* % n* % P n* % n* % P Ethnicity Adja Fon < Yoa & Lokpa <0.01 Yoruba Other Wealth quintiles Poorest Poorer Poor > >0.10 Less poor Least poor Education among women None > Primary school Secondary school >0.10 Literacy among women Hygiene ** > >0.10 Improved water source <0.01 Improved sanitation > <0.01 Rural residence Excluding Cotonou and High Impact districts *Weighted > > Environmental characteristics Table 3a presents selected environmental characteristics of the HIZs and comparison area; appendix A provide further details. The HIZs are located in the central transitional region of Benin, falling between the tropical, wet climate of the south and the semi-arid tropical and Sahelian regions to the north. The HIZs experience similar rainfall as the rest of Benin, and malaria transmission is seasonal eight to nine months of the year, similar to the rest of the country, except in the drier northern regions. 25 Before and during ACSD implementation, resistance of malaria parasites to chloroquine grew; Appendix table J4 provides full details on antimalarial resistance trends. Table 3a: Selected environmental characteristics of high-impact zones, PAK and DAA, and comparison area, Benin. 14

27 CHARACTERISTICS Climate Main Geographic Characteristics HIZS DAA PAK Transitional between tropical Semi-arid tropical wet & semi-arid tropical Savanna & the Palm plantations semi-deciduous Grasslands forest COMPARISON AREA Tropical wet, Semi-arid tropical Palm plantations Grasslands; Savanna & the semi-deciduous forest; Sahel Annual rainfall (mm) Months of malaria transmission months/year 8-9 months/year 5-9 months/year, with shortening transmission season in north Rest of Benin, minus Cotonou and HIZs Source: INSAE, Our investigations did not reveal any natural disasters, famines or other emergencies in the HIZs over the primary evaluation period. Key informants reported that Djida commune in the DAA zone experienced sporadic periods of instability. In 2005, the regions of Aribori and Atakora in the north of Benin did experienced food insecurity, associated with the famine in neighboring Niger Baseline health conditions Section 3.2 presents a profile of child health in Benin as a whole, including the cause of death profile. Cause of death information is not available disaggregated by HIZs and comparison area. We present and consider baseline levels of undernutrition in section six and under-five mortality in section seven. 3.7 Health service characteristics Availability of health services. Table 3b summarizes available information on health facilities in the HIZs and comparison area as reported by the Benin Ministry of Health in Appendix J presents more details on the coverage of health facilities over time. On average and over time, DAA health zone had higher per capita coverage of health facilities than the comparison area, while PAK zone has fewer health facilities than DAA or the comparison area. Table 3b: Health system characteristics in PAK and DAA zones and comparison area in 2006, Benin HIZS CHARACTERISTICS DAA PAK COMPARISON AREA Total health facilities Average # of health centers per commune Population per facility Private or NGO health facilities Hospitals Rest of Benin, minus Cotonou and HIZs Source: Annuaire des statistiques sanitaires Changes in health policies. 15

28 A number changes in national policies, implemented in both the HIZs and comparison area, influencing child health took place between 2001 and 2006 (see Appendix J for further details), including: Introduction of Hib vaccination into national policy and inclusion in routine EPI vaccination schedules in June 2005; Change in first-line antimalarial policy from chloroquine to ACTs in 2004, with no generalized availability of ACTs until 2008; Importation of polio from Nigeria to Benin, with two cases in late 2003,and six cases in early necessitating the organization of national immunization days; Distribution of Vitamin A supplementation twice a year nationally, coupled with polio campaigns when they were organized, starting in 2002; Change in the national pricing policy of insecticide-treated nets (ITNs) from approximately US$7 in 2002 for all nets to US$1 for targeted populations in Other projects that may impact child health Health and development projects in the HIZs. In addition to the ACSD program and routine government services, other health and development projects supported activities related to child health during the period under study in the HIZs. The Benin Malaria Control Program, local health services, Africare and CDC implemented activities to prevent and treat malaria and other childhood illness through the African Integrated Malaria Initiative (AIMI, also known as PROLIPO in French) with support from USAID. These activities took place from 2001 to 2005 in Ouémé and Plateau departments, which include the PAK health zone. Other projects taking place in 2002 to 2007 focused on clinical improvements, especially in maternal and neonatal health, including support and care for AIDS orphans and vulnerable children, as well as prevention of mother-to-child transmission of HIV (PMTCT). Local and international NGOs in DAA also carried out nutritional rehabilitation and education in selected communes. Appendix J provides further details about activities, geographic coverage and timing of other health and development programs in the HIZs. Other external partners and donors in whole of Benin. A multitude of other donors and external partners provided support for activities in the rest of Benin, which is our comparison area excluding Cotonou. USAID supported health activities throughout Benin, with annual budgets for child survival and infectious disease ranging between US$3 and US$4 million in fiscal years 2004 to The Global Fund to Fight AIDS, Tuberculosis and Malaria supported national-level malaria control activities through a US$2.4 million grant issued in 2003, of which a large proportion was utilized for ITN distribution and treatment activities. Other large-scale health and development partners in the rest of Benin included various local and international NGOs, WHO, UNFPA, France, Denmark, Germany, Belgium, The Netherlands, Canada, Switzerland, the World Bank, the European Union, and the African Development Fund. 16

29 4. ACSD as implemented in Benin This section provides an overview of the ACSD activities in the HIZs. Funding, adaptation of the generic ACSD package, and the timeline of activities are considered; the activities and inputs for each ACSD component are then briefly described. Appendix C includes further textual description on implementation and detailed timelines for ACSD activities Funding UNICEF headquarters transferred US$1.5 million to UNICEF-Benin in March 2002 to support ACSD activities in Benin. 34 This was about one year later than the transfer of funds to the other three highimpact countries. UNICEF staff report that ACSD activities have been co-funded and sustained since 2003 through support from Belgium, UNICEF-Benin general resources, and general program funds Adaptation of the generic ACSD intervention package and approach The generic ACSD strategy was adapted at the national level, starting with a planning workshop held in Bohicon in January In the Abomey commune, community IMCI (C-IMCI) communication messages and materials were adapted for use in the DAA zone at a two-week workshop held with community leaders such as teachers, community health workers (CHWs), and health staff in November Participatory community situation assessments and communication plan development workshops utilized materials developed for use in all ACSD countries. A similar workshop took place in February 2003 in Pobè to adapt C-IMCI materials for use in the PAK zone Results-based planning ACSD implementers at the international level chose the package of interventions to be implemented in the four high-impact countries based on evidence and cost-effectiveness, using the Marginal Budgeting for Bottlenecks (MBB) tool. The ACSD strategy set specific targets for each package and UNICEF monitored results at the zone and commune levels. We did not find evidence that ACSD in Benin included performance contracts or other innovations linking results to specific incentives. 4.4 Timeline for ACSD activities in the high-impact zones UNICEF officially launched the ACSD project in February 2002 in collaboration with the Government of Benin. A national steering committee, put in place by the Ministry of Health (MOH) in early 2002, held its first meeting in April The remainder of 2002 was devoted to strategic planning, development of tools and materials, and logistic preparations with the MOH and partners. During this time, UNICEF hired two consultants to assist with implementation of ACSD in each health zone. Implementation of most ACSD-supported activities began in Figure 5 presents a summary timeline for the acceleration or implementation of the ACSD intervention packages, as well as for household surveys conducted in the two HIZs. These timelines are based on information obtained from key informants and document reviews. Table 4 provides additional information about the timing of specific activities. Appendix C includes more details about the timing and content of ACSD activities. 17

30 Figure 5: Timeline for implementation of ACSD packages of interventions and surveys conducted to evaluate intervention coverage, Activities Surveys EPI+ ITN IPTp IMCI Facility Community Facility Community 2001 May DHS Nov 02 July 03 Dec 02 July Jan ACSD-CDC 2005 No ITNs Available DHS Supplemental Survey Figure 5 Key: First bar represents PAK zone and second bar represents DAA zone Grey bars represent implementation before acceleration through ACSD 18

31 Table 4: Start times for accelerated implementation of ACSD intervention packages in the ACSD high-impact zones of Benin. INTERVENTION PACKAGE ~ START IN PAK ~ START IN DAA EPI ITNs IPTp * Routine EPI on-going prior to ACSD; acceleration in 4 th quarter 2002 Late 2002 Facility July Community October 2003 (Training of Trainers) Routine EPI on-going prior to ACSD; acceleration in 4 th quarter 2002 November 2002-Facility July Community October 2003 (Training of Trainers) Facility IMCI June 2001 December 2002 C-IMCI: management of illness, EPI defaulter tracing and ITN re-treatment, promotion of exclusive breastfeeding, complementary feeding, vitamin A, etc. 3 rd quarter 2003 Mid rd quarter 2003 Mid 2004 Strengthening of nutrition & feeding elements Mid 2005 Mid 2005 ITN stock-outs 3 rd quarter 2005 to mid 2007 * IPTp began in PAK and DAA with the training of trainers in October 2003, followed immediately by training of service providers. Administration of SP to pregnant women presenting to health facilities began in ACSD activities in the high-impact zones EPI+. Vaccination and vitamin A supplementation Vaccination activities and the provision of vitamin A supplements to children 6-59 months of age were already in place through the MOH system prior to the introduction of ACSD. The health system in Benin delivers vaccines to children in three principal ways: 1) routine, facility-based vaccination; 2) routine outreach activities; and 3) vaccination campaigns. Routine facility and outreach vaccination activities were supported throughout the evaluation period. Measles campaigns took place in early 2003 and late After importation of wild poliovirus from Nigeria was detected in Benin in 2003, 33 the quality and quantity of national-level supplemental immunization days (NIDs) for polio were reinforced. Primarily, the NIDs and other isolated campaigns delivered vitamin A twice a year, starting in 2002 with support from UNICEF. Table 5 summarizes available information about ACSD inputs intended to reinforce EPI+ activities extracted from administrative and summary reports. Appendix table C2 presents further description of EPI+ activities and details on exact timing. 19

32 Table 5: Description of inputs related to vaccination and vitamin A supplementation in the ACSD highimpact zones of Benin. DESCRIPTION OF TIMING ACTIVITY Provision of basic medical materials, refrigerators for cold chain, computers for monitoring and data collection activities and commodities Provision of new motorcycles for outreach activities GEOGRAPHIC AREA PAK, DAA and expansion area of Zou-Collines and Ouémé-Plateau INTENSITY OF ACTIVITY 2002 DAA 4 motorcycles PAK Quantitative data incomplete 15 motorcycles COVERAGE ESTIMATE 4 motorcycles for 29 arrondissements 15 motorcycles for 17 arrondissements Provision of 4x4 vehicle for supervision Training of facility-based health workers in EPI provision, monitoring and surveillance Ouémé department DAA PAK DAA 4 vehicles 57 health workers trained 1.3 trained workers per 1000 children U5* Quantitative data incomplete 84 health workers trained 1.8 trained workers per 1000 children U5* Support for NIDs, local vaccination campaigns, catch-up vaccination, supervision and monitoring activities Provision of vitamin A capsules PAK and DAA Other Quantitative data incomplete 2004 DAA PAK, DAA, Zou- Collines, Ouémé- Plateau, nationally 4 rounds catch-up EPI & 2 tetanus campaigns; Detailed quantitative data presented in appendix C *Population of children under-five extracted from 2002 census, Cahier des villages; (population estimated at 56,609 children U5 in PAK and 45,624 children U5 in DAA) 20

33 Insecticide-treated nets (ITNs). ACSD in Benin utilized different strategies for the provision and promotion of utilization of ITNs in Ouémé- Plateau and Zou-Collines, including the PAK and DAA health zones, respectively. In the Ouémé-Plateau departments, including the PAK zone, bednets were sold and their use promoted by women s groups and in maternity centers through an agreement with Africare, an international NGO. The CHWs promoted, distributed and re-treated ITNs starting in mid In the departments of Zou-Collines, including the DAA health zone, bednets were sold through social marketing techniques at town markets, as well as at health centers; they were also sold and promoted by CHWs in villages. Population Services International (PSI) was a key partner and implementer of this activity. Mosquito nets were retreated through periodic community-based treatment campaigns in the four departments, including the PAK and DAA zones. UNICEF and other partners, such as the national malaria control program (PNLP) supported provision of insecticide treatment at no cost. Table 6 summarizes available information about ACSD contributions to the promotion, distribution and treatment of mosquito nets extracted from administrative and summary reports. To provide rough guidance on the potential coverage of these activities, we present several of the indicators as ratios per 1,000 children under-five years, even though it is recognized that the ITNs were also targeted to pregnant women and may have been used by non-targeted members of the population. As a result, the coverage estimates below are likely overestimated. Appendix table C3 presents more details about ITN distribution, promotion and treatment. 21

34 Table 6: Description of inputs related to the distribution, promotion and treatment of bednets in the ACSD high-impact zones of Benin. DESCRIPTION OF ACTIVITY TIMING GEOGRAPHIC AREA INTENSITY OF ACTIVITY COVERAGE ESTIMATE Training of women s groups in ITN promotion 2002 PAK 270 women trained in 90 villages 5.5 trained women per 1000 children under 5* Training and deployment of CHWs to promote, distribute and treat bednets 2003 PAK DAA 200 CHWs trained 200 CHWs trained 4.1 trained CHWs per 1000 children under 5* 5.0 trained CHWs per 1000 children under 5* 2002 DAA 1900 bednets distributed 42 bednets per 1000 children under 5* Distribution of bednets 2003 PAK DAA 9765 bednets sold at subsidized price 22,575 bednets distributed 199 bednets sold per 1000 children under 5* 495 bednets per 1000 children under 5* ,375 bednets distributed 184 bednets per 1000 children under 5* 2005 DAA ii 7,150 bednets distributed 157 bednets per 1000 children under 5* ,575 bednets distributed 93 bednets per 1000 children under 5* PAK Quantitative data incomplete Community campaigns for treatment of bednets Early 2003 Late 2003 PAK DAA PAK DAA 2004 PAK 2005 PAK 2005 DAA 9,330 bednets treated 2,892 bednets treated 10,883 bednets treated 4,509 bednets treated 9,295 bednets treated 19,795 bednets treated 55,378 bednets treated 164 bednets treated per 1000 children under 5* 65 bednets treated per 1000 children under 5* 192 bednets treated per 1000 children under 5* 101 bednets treated per 1000 children under 5* 164 bednets treated per 1000 children under 5* 350 bednets treated per 1000 children under 5* 1213 bednets treated per 1000 children under 5* *Population of children under-five extracted from 2002 census, Cahier des villages; (population estimated at 56,609 children U5 in PAK and 45,624 children U5 in DAA) ii Summary presentations given by UNICEF staff indicate that 44,250 ITNs were distributed at a subsided price between 2002 and 2006 in the DAA health zone. 22

35 IMCI+. Integrated case management of child illness (IMCI) and promotion of improved feeding practices were carried out in both facilities and the community through the ACSD strategy. Standard 11-day training of facility-based health workers had already taken place in the PAK zone in June UNICEF and ACSD supported the MOH to carry out IMCI training in DAA in late Periodic supervisory visits and review of monitoring data collected from IMCI-compatible health registers were carried out to reinforce IMCI implementation. In addition to IMCI, facility-based health workers in PAK and DAA received training in healthy child consultations and the minimum package of nutrition activities, which focused on micronutrient supplementation and infant feeding practices. The PAK and DAA health zones were pilot zones for community IMCI in Benin. As explained above, UNICEF supported a series of workshops and community situation analysis exercises with MOH officials, local health zone staff and community members and leaders to assist in planning of community IMCI strategies. In mid-2003, 404 community health workers (CHWs) were selected in 202 remote villages in PAK and DAA (102 in PAK and 100 in DAA), chosen by local leaders. CHWs and community leaders received a series of initial and refresher trainings addressing: Vaccination promotion and defaulter tracing; Treatment of bed nets; Home management of malaria and diarrhea and medicine management; Promotion of infant feeding practices, vitamin A and appropriate management of child illness; Hygiene; and Recording of data, including birth registration. CHWs were issued bednets and a medicine box with ORS, chloroquine, paracetamol, mebendazole, and iron in 2003 and visual aids for health promotion in The medications were to be sold to the families of sick children in the villages at reasonable prices; a small margin of benefit for the CHWs and community committees was planned. In 2004, a UNICEF-supported operational research project trained 40 CHWs in the Kétou commune in PAK and 40 CHWs in the Djidja commune in DAA to manage pneumonia with cotrimoxazole at the community level. Table 7 summarizes available information about ACSD contributions to facility and community IMCI extracted from administrative and summary reports. Again, we present selected indicators as ratios per 1,000 children less than five years of age to better assess potential coverage. Appendix table C4 further describes IMCI+ activities and exact timing. 23

36 Table 7: Description of inputs related to the implementation of the IMCI+ intervention package in the ACSD high-impact zones of Benin. DESCRIPTION OF ACTIVITY TIMING GEOGRAPHIC AREA INTENSITY OF ACTIVITY COVERAGE ESTIMATE Standard facility IMCI training On-going IMCI supervision and monitoring Health child visit / minimum package of nutrition training 2003 DAA 2004 PAK PAK 2004 DAA PAK and DAA 24 health workers trained; denominator of health workers not available 7 health workers trained for a cumulative total of 38 health workers out of 42 4 supervisions per year 44 health workers trained 184 health workers trained 0.8 health worker per 1000 children under 5* 4.0 health worker per 1000 children under 5* Training of CHWs and community leaders in promotion of EPI, ITNs, infant feeding practices, hygiene and correct management of ARI, malaria, diarrhea 2003 PAK DAA 200 CHWs trained 40 arrondissement leaders trained 200 CHWs trained 20 arrondissement leaders trained 3.5 CHWs per 1000 children under 5* 40 leaders of 29 arrondissements 4.5 CHWs per 1000 children under 5* 20 leaders of 17 arrondissements Deployment of village drug kits (chloroquine, antipyretics, deworming, ORS, and iron) managed by CHWs Training of CHWs for community pneumonia management using cotrimoxazole Supervision and monitoring of CHW activities through field visits and meetings 2003 PAK 2004 DAA Kétou (PAK) Djidja (DAA) PAK and DAA 102 villages with drug kits 100 villages with drug kits 40 CHWs trained 39 CHWs trained Throughout ACSD period, at least yearly 102 villages / 128 villages in PAK 100 villages / 155 villages in DAA 1.8 CHWs per 1000 children under 5* in Kétou 2.0 CHWs per 1000 children under 5* in Djidja See Appendix table C4 for details and timing *Population of children under-five extracted from 2002 census, Cahier des villages Visual Aids for promotion of correct illness management and feeding practices were distributed to CHWs in

37 ANC+. Antenatal care interventions supported under the ACSD approach of Focused ANC+ in Benin included: 1) focused antenatal care iii ; 2) utilization of ITNs; 3) intermittent preventive treatment for malaria for pregnant women (IPTp) with a combination of sulfadoxine-pyrimethamine (SP); 4) prevention of motherto-child transmission of HIV (PMTCT); 5) deworming; and 6) supplementation in iron and folic-acid. In late 2005, an ANC kit was introduced that contained a bednet, iron/folic acid supplements, SP for IPT of malaria, and mebendazole for de-worming. iv In the DAA zone, this ANC kit was provided in a special sachet to facilitate distribution; the different elements are sold separately to pregnant women in PAK. A radio communication system was put into operation with UNICEF support in the DAA zone in 2004 to facilitate evacuation of obstetrical emergencies. UNICEF reported 95% coverage of health facilities and maternities in Full installation of the radio system was complete in PAK in Table 8 summarizes available information about ACSD inputs related to ANC activities extracted from administrative and summary reports. Appendix C and table C5 provide further description of ANC+ activities and timing. iii Focused ANC reorients ANC care to treat all pregnancies as at risk. Starting in the first ANC visit, this strategy is intended to encourage: 1) women to plan for the delivery; 2) planning logistically and financially for evacuation in the case of complications; and 3) husbands to assist at least one ANC visit to help with this planning. Increasing the decision power of pregnant women is at the heart of this strategy. iv The ANC kit is sold for CFA 1000 (~ USD 2.00) to a woman at her first antenatal visit to a community health center or maternity. According to health officials, the total cost of the kit contents to the health center is more than the price of CFA Health centers lose money with the sale of kits at the subsidized price, and cannot recover their costs. Thus, strong disincentives exist to promote or make kits available and issues remain with re-supplying the commodities included in the kits. 25

38 Table 8: Description of inputs related to the implementation of the ANC+ intervention package in the ACSD high-impact zones of Benin. DESCRIPTION OF ACTIVITY Provision of medical kits and supplies TIMING GEOGRAPHIC AREA 2002 PAK and DAA INTENSITY OF ACTIVITY COVERAGE ESTIMATE All health centers equipped with basic kit 2004 PAK and DAA Kits for newborn care Introduction of IPTp and focused ANC 2004 PAK DAA Quantitative data incomplete 37 nurses/ midwives trained 37 trained nurses for 21 maternities Training of maternity personnel in management of neonate ( 2 sessions) 2004 PAK DAA Quantitative data incomplete 32 nurses/ midwives trained 32 trained nurses for 21 maternities Training of nurses and midwives in Active Management of the Third Stage of Labor (2 sessions) DAA 30 nurses and midwives trained 30 trained nurses for 21 maternities Training of Drs, midwives, nurses, lab techs and social workers in PMTCT 2005 PAK DAA Quantitative data incomplete 61 personnel trained Supervision and monitoring activities for ANC activities, including IPTp, PMTCT, neonatal care PAK and DAA Sporadically throughout ACSD period See Appendix table C5 for details and timing 26

39 5. Coverage and family practices This section of the report presents the results and interpretation of priority coverage and family practices indicators. Section 2 describes the methodology for the analysis of coverage and family practices, with priority indicators defined in appendix D. We present results in graphical form for selected priority coverage indicators within each intervention package. Two graphs are presented for each package. The first shows time-trends in indicator levels in the HIZs. We present data from surveys conducted in 2003 and 2005 in shades of grey and without confidence limits because, as explained in the methods section, these estimates are of lesser quality and should be interpreted with caution. The second graph for each intervention component presents indicator levels in 2001 (baseline) and (endline) in both HIZs and the comparison area. The bars in these graphs represent the 95% confidence limits. We carried out differences-in-differences statistical tests for these comparisons and they are presented in the text. Appendices G and H present the full results for HIZs and comparison groups; further detail on coverage levels by zone, urban-rural residence, education of the mother and wealth quintiles in the survey, as well as other descriptive tables, are presented in Appendix I. For certain indicators and sub-populations, the results should be interpreted with caution due to the small sample sizes for some cells. 5.1 Results EPI+. Vaccinations and vitamin A supplementation. Figure 6 shows time trends in measles and DPT vaccination coverage and in vitamin A supplementation in the ACSD HIZs. There is no evidence of an increase in measles or DPT between 2001 and Coverage levels for vitamin A supplementation increased dramatically in this period; the ACSD coverage survey results suggest that the greatest part of this increase occurred between 2001 and Not shown in graphical form are results on coverage of Hib vaccine. Hib vaccine was introduced to children less than one year of age in June The survey results indicate that only 17% of children aged months had been vaccinated with the recommended three doses of Hib by their first birthday in , but coverage increases to 25% when examined among the cohort of children born after June 2005, when the vaccination was officially introduced. Trends in EPI+ coverage were similar in PAK and DAA health zones (Appendix table G2). 27

40 Figure 6: Coverage levels for measles and DPT3 vaccination and the receipt of one vitamin A supplement in the preceding six months in the ACSD high-impact zones as measured in DHS surveys in 2001 and , as well as coverage reported in the CDC-ACSD survey in 2003, Benin DHS 2001 ACSD 2003 DHS Coverage (%) * Measles DPT3 Vitamin A *Vitamin A coverage data available only for children aged 6-32 months in the 2003 ACSD survey Note: Measles and DPT3 indicators are calculated based on MICS protocol, where the distribution of children with card confirmed vaccination before 12 months is applied to all other children reported as vaccinated. Appendix tables I2 and I3 provide further information on coverage levels for vaccinations and vitamin A supplementation in the HIZs in Vaccine coverage tended to be higher in DAA than PAK, with a slightly greater differential in DPT3 coverage (p<0.001) than for measles vaccination (p=0.03). Vaccination coverage also tended to be slightly higher in urban areas, although this difference was not statistically significant. Children of more educated mothers were more likely to have been vaccinated for measles (p=0.04) and to have received three doses of DPT (p=0.01) before their first birthday. A greater proportion of boys than girls were vaccinated against DPT3 (p=0.05). Children residing in the poorest quintile of households were significantly less likely to be vaccinated for measles (p<0.001) or DPT3 (p<0.001) than children residing in less poor households. For example, reported measles vaccination was 29% in the poorest quintile and 65% in the least poor quintile; coverage for DPT3 was 37% and 72% for these two quintiles, respectively. For vitamin A coverage, again DAA performed better than PAK (p<0.001), and higher levels of maternal education were associated with higher levels of coverage (p<0.001). Despite delivery through campaigns, often thought to promote equity, coverage for vitamin A supplementation was relatively inequitable in the HIZs (p<0.001): it reached only 45% of children 6-59 months in the poorest quintile of households compared to 70% among the least poor. There were no urban/rural or sex differentials in vitamin A supplementation. 28

41 Figure 7: Coverage levels for measles and DPT3 vaccination and receipt of one vitamin A supplement in the preceding six months in ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin. Absolute change in percentage points +6-3 High-impact zones Comparison area Coverage (%) Measles DPT3 Vitamin A Figure 7 shows coverage levels for vaccinations and vitamin A supplementation in the HIZs and the comparison area in 2001 and Measles vaccine coverage remained stable between 2001 and in HIZs, but there was a non-significant increase in comparison area. The difference between time trends in the HIZs and the comparison area was not statistically significant. DPT3 coverage decreased slightly in both the HIZs and the comparison area; neither these changes nor the difference-indifferences between the HIZs and the comparison area were statistically significant over time. Vitamin A supplementation increased by 51 percentage points (pp) in the HIZs and 46 pp in the comparison area. Although increases over time in both the HIZs and comparison area were statistically significant (p<0.001), the difference between the two rates of increase was not statistically significant. 29

42 Insecticide-treated nets (ITNs). Figure 8 shows time trends in the use of ITNs in the HIZs. There were increases in the proportion of children sleeping under an ITN, but only one in four children was protected in Results for pregnant women are more difficult to interpret because the baseline survey did not assess whether the nets used by pregnant women were impregnated with insecticide. There does appear to be an increase in coverage for any net use. However, in only one out of every four to five pregnant woman reported using an ITN the night before the survey. Figure 8: Coverage levels for insecticide-treated nets in the ACSD high-impact zones as measured in DHS surveys in 2001 and , as well as coverage reported in the CDC-ACSD survey in 2003 and a coverage survey in 2005, Benin DHS 2001 ACSD 2003 Coverage survey 2005* DHS Coverage (%) Child slept under ITN Data not available 34 Pregnant woman slept under ANY net* Data not available 27 Data not available 22 Pregnant woman slept under ITN *Data concerning insecticide treatment for women s nets not collected in 2001; any net used as a proxy NOTE: All measurements of ITNs in ACSD survey 2003 for nets treated in previous 6 months (as compared to previous 12 months in DHS and coverage surveys) Appendix tables I4 and I5 provide further information on coverage levels for ITNs in by health zone, commune, urban/rural residence and child s age or women s education. Coverage among pregnant women and young children was very similar in the DAA and PAK health zones. Levels of ITN use among children were slightly higher in urban versus rural areas, although this trend was not significant. Younger children also had higher proportions of reported utilization of ITNs (p=0.03) than older children. The proportion of children sleeping under an ITN the night previous to the survey ranged from only 14% in the lowest wealth quintile to 41% in the highest (p<0.001). Use of an ITN among pregnant women was also positively associated with wealth (p=0.02). 30

43 Figure 9: Coverage levels and absolute change in percentage points for insecticide-treated nets in ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin. Absolute change in percentage points High-impact zones Comparison area Comparison ~+17 area ~ Coverage (%) Child slept under ITN Pregnant woman slept under ANY net ~ 0 ~ 0 Pregnant woman slept under ITN Figure 9 shows reported ITN use in the HIZs and comparison area in 2001 and The proportion of children sleeping under an ITN increased by 20 pp in the comparison area and 23 in the HIZs (both p<0.001). The rates of increase over time did not differ significantly between the HIZs and comparison area. Women sleeping under any net used as a proxy due to lack of data in 2001 increased by 20 pp in the HIZs and by 11 pp in the comparison area; the difference in percentage point change was not significant. If we assume that ITN use was zero in 2001, coverage among pregnant women by would have increased 22 pp in the HIZs and 17 points in comparison area (p< 0.01). IMCI+. The ACSD IMCI+ strategy implemented in Benin included two packages of interventions, to be delivered by trained health workers. The first package is comprised of treatments for childhood illnesses, the second package contains interventions designed to improve child feeding. We report coverage levels for these interventions in this section. Case management of childhood illnesses. Figure 10 shows time trends in the case management of childhood fever (presumed to be malaria in this highly endemic country), appropriate careseeking for suspected pneumonia and oral rehydration therapy with continued feeding for diarrhea. These indicators rely on mothers reports for children with these symptoms in the two weeks prior to the survey. Appendices G and I provide additional data. The results show no improvement in the case management of childhood illness between 2001 and , although the power of the before and after comparison is affected by the small sample sizes at baseline. About two-thirds of children with reported fever received an antimalarial, but over 90% of children received only chloroquine (Appendix table I7). This drug is no longer effective in Benin, where the national policy recommends ACT 29 (see Appendix table J4 for trends in antimalarial resistance and policy). In both

44 and , approximately one-third of children with probable pneumonia were taken to a health facility, and a similar proportion of those with diarrhea received oral rehydration and continued feeding. Further details on the management of diarrhea (Appendix table I8) and care seeking for pneumonia (Appendix table I9) over time are provided in Appendix I, but should be interpreted with caution due to limited sample sizes in some cells. Figure 10: Coverage levels for case management indicators in the ACSD high-impact zones as measured in DHS surveys in 2001 and , as well as coverage reported in the CDC- ACSD survey in 2003 and a coverage survey in 2005, Benin. 100 DHS 2001 ACSD 2003 Proportion of ill children managed for illness (%) Coverage survey 2005 DHS Fever* Careseeking for suspected pneumonia Diarrhea *Fever management is with any antimalarial, regardless of policy 32

45 Appendix table I6 provides a breakdown of case-management indicator levels estimated from the survey by health zone, sex, commune, urban/rural residence and child s age and mother s education. A slightly higher proportion of children of more educated women received an antimalarial (p=0.08) than children of women with less education. Pneumonia care seeking was marginally more common for boys (p=0.13) than girls, and better in PAK than in DAA (p=0.13). Older children tended to receive better diarrhea management (p=0.07) than younger children. Children residing in better off households were more likely to receive any antimalarial (p=0.04) than children in poorer households. There were no significant associations between household wealth and pneumonia care seeking or diarrhea management, but it should be noted that the sample sizes for these indicators, within each wealth quintile, were very small. Figure 11: Coverage levels and absolute change in percentage points for the proportion of sick children correctly managed at home in ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin. Absolute change in percentage High-impact zones Comparison area Comparison -1 area Coverage (%) Fever Careseeking for suspected pneumonia Diarrhea NOTE: Fever management is with any antimalarial, regardless of national policy Figure 11 shows coverage levels for case management in the HIZs and the comparison area in 2001 and Levels of treatment with any antimalarial for fever decreased slightly in both the HIZs and the comparison area; the difference-in-differences test was not statistically significant. However, if we define the indicator as treatment of fever with an effective and nationally recommended antimalarial there was a precipitous drop in coverage in both HIZs and national comparison area (Appendix table I7). Care seeking for pneumonia and correct home management practices for diarrhea remained stable, with no statistical differences over time or between rates of change in the HIZs and comparison area. 33

46 Feeding, including breastfeeding. IMCI+ as recommended by ACSD also included promotion of appropriate infant and young child feeding practices (Box 1). Figure 12 shows the prevalence of selected feeding behaviors reported by mothers of children less than one year of age at the time of the survey. Breastfeeding behaviors tend to be stable over time, so the apparent fluctuations should be interpreted with caution because they may reflect differences in how the questions were posed or the answers recorded. Additional data are available in Appendix G, H and I. There is no evidence of improvement over time among the indicators assessed. Exclusive breastfeeding for infants less than six months of age is least adequate, with only 27% of mothers reporting this practice in the survey. Figure 12: Prevalence of infant feeding behaviors as reported by mothers in the ACSD high-impact zones as measured in DHS surveys in 2001 and , as well as coverage reported in the CDC- ACSD survey in 2003 and a coverage survey in 2005, Benin. Prevalence of behavior as reported by mothers (%) Data not available 0 47 Initiation of breastfeeding within one hour of birth Exclusive breastfeeding to six months of age DHS 2001 ACSD 2003 Coverage survey 2005 DHS Timely and appropriate complementary feeding Appendix table I10 presents the breakdown of infant feeding practices in by socio-demographic characteristics. Reported exclusive breastfeeding levels were marginally higher in DAA than in PAK (p=0.09); other breastfeeding practices were not significantly different in PAK and DAA. Mothers with higher levels of education were more likely to report complementary feeding at ages 6 to 9 months than mothers with less education (p=0.14). Breastfeeding of children at months was significantly more common in rural (76%) than in urban areas (42%); (p<0.001). Infant feeding practices were not significantly different among children residing in households of different socio-economic status, as measured by wealth quintiles. 34

47 Figure 13: Coverage levels and absolute change in percentage points for infant feeding behaviors as reported by mothers in ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin. Prevalence of behavior as reported by mothers (%) Absolute change in percentage points ? Initiation of breastfeeding Exclusive breastfeeding to six Timely and appropriate within one hour of birth months of age complementary feeding High-impact zones Comparison area Comparison area Figure 13 shows the prevalence of selected infant feeding behaviors in the HIZs and the comparison area in 2001 and Initiation of breastfeeding within one hour of birth did not change significantly in the HIZs; in the comparison area it increased by seven pp (p<0.01). Exclusive breastfeeding up to six months of age showed a non-significant decrease in HIZs, while increasing slightly in the comparison area. Complementary feeding from six to nine months of age remained relatively stable, again with slight increases in the comparison area. Changes over time in the three feeding indicators were not significantly different between HIZs and comparison area. ANC+. The ANC+ package as implemented in Benin included interventions in both the antenatal and perinatal periods. We present coverage levels for antenatal interventions and then coverage with interventions designed to improve maternal and neonatal health during delivery and the post-natal period in this section of the report. Antenatal care. Figure 14 shows the time trends in coverage of antenatal care in the HIZs. Appendices G and I present further details. A high proportion of mothers reported three or more ANC attendances, but there was no evidence of an increase across the study period. Intermittent presumptive treatment (IPTp) for malaria during pregnancy increased slightly. Reported coverage with SP during pregnancy was 28% in 2001; however, this was before the IPTp intervention was available in Benin. Technical staff at the statistical agency implementing the survey (INSAE), as well as health implementation staff, thought this was a measurement error. Nonetheless, only seven percent of pregnant women reported IPTp with SP in Tetanus toxoid (TT) vaccination during pregnancy increased slightly in this period, but this trend was not significant. The use of iron supplements during pregnancy increased from 33% to 55% during the study period (p<0.001) (Appendix table G4). 35

48 Figure 14: Coverage levels for antenatal indicators in the ACSD high-impact zones as measured in DHS surveys in 2001 and , as well as coverage reported in the CDC-ACSD survey in 2003 and a coverage survey in 2005, Benin. 100 DHS 2001 ACSD 2003 Coverage survey 2005 DHS Coverage (%) Data not available ~ Data not available antenatal visits IPTp with SP* 2 TT doses *Any dose of SP during pregnancy (not limited to two doses, due to data limitations) NB: IPTp with SP incorrectly measured as 28% in 2001 in the HIZs, before IPTp with SP was available Appendix tables I11 and I12 provide further information on coverage levels of antenatal care in the HIZs in Coverage was significantly higher for women residing in the DAA zone, in urban residences and among those with higher levels of education. In the DAA zone, 74% of women reported three or more antenatal visits during their previous pregnancy versus 52% in PAK (p<0.001). IPTp and TT2 vaccination, closely linked to ANC visits, were also significantly more frequent in DAA (p<0.001 for both comparisons). Women reporting a birth in the 6 to 11 months preceding the survey were significantly more likely to report three or more antenatal care visits (p=0.01) than those with a more recent birth. Higher levels of women s education were also associated with three or more ANC visits (p<0.01), TT2 (p=0.05), IPTp (p<0.001) and iron supplementation (p<0.001). Coverage of ANC interventions was highly inequitable in in the HIZs. More than double the proportion of women in the wealthiest households reported three or more ANC visits (90%) as compared to those in the poorest households (40%); (p<0.001). Similar inequities were observed for IPTp (p<0.001), TT2 vaccination (p<0.001) and iron supplementation (p<0.001)). 36

49 Figure 15: Coverage levels and absolute change in percentage points for antenatal indicators in ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin. Absolute change in percentage points +2? High-impact zones Comparison area Comparison +44 area Coverage (%) ~0 ~ antenatal visits IPTp with SP* 2 TT doses *Any dose of SP during pregnancy (not limited to two doses, due to data limitations) NB: IPTp with SP incorrectly measured as 28% in the HIZs and 6% in the comparison area in 2001 DHS, before IPTp with SP was available Figure 15 shows reported antenatal care in the HIZs and the comparison area in 2001 and There were few differences between the rates of change in the HIZs and comparison area between 2001 and Changes over time did not differ significantly between HIZs and comparison area for three or more ANC visits, IPTp, TT2 vaccination or iron supplementation. 37

50 Assisted delivery and postnatal care. Figure 16 shows trends in assisted deliveries and postnatal care as reported by women having a birth within 12 months before the survey. Assisted deliveries include those attended by a doctor, nurse or midwife, but not those assisted by assistant nurses or auxiliary mid-wives. Additional data concerning these indicators are available in Appendices G, H and I. Sample sizes are limited in 2001 with only 88 women having a live birth within the previous 12 months. Assisted delivery and postnatal visits were quite high at baseline and there is no evidence of a consistent improvement. Supplementation with vitamin A within 40 days after birth improved significantly over the period from 2001 to (p< 0.001). Figure 16: Coverage levels for assisted deliveries and postnatal care in the ACSD high-impact zones as measured in DHS surveys in 2001 and , as well as coverage reported in the CDC- ACSD survey in 2003 and a coverage survey in 2005, Benin DHS ACSD DHS Coverage (%) Data not available Assisted delivery* Postnatal visit Postnatal Vitamin A *Includes only deliveries assisted by doctor, nurse or midwife Appendix table I13 shows the breakdown of delivery and postnatal care in in the HIZs by sociodemographic characteristics. Women in the DAA zone had significantly higher proportions of assistance during delivery (p<0.001), postnatal visits (p<0.001) and postnatal supplementation with vitamin A (p<0.001). Urban dwellers had higher proportions of assisted deliveries than those living in rural areas (p<0.01), but not higher coverage levels for postnatal care or vitamin A supplementation. Mothers with higher levels of education were also significantly more likely than mothers with less education to report assisted delivery (p<0.01) and postnatal care (p<0.01). Assisted delivery and postnatal care were much less frequent in the poorest household than in less poor household. Women in the highest wealth quintile were almost twice as likely to have a delivery assisted by a skilled provider (96%) than women in the poorest households (48%); (p<0.001). Coverage results for postnatal visits within three days (p<0.001) and postnatal vitamin A supplementation (p<0.01) also showed significant inequities. 38

51 Figure 17: Coverage levels and absolute change in percentage points for assisted delivery and postnatal care indicators in ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin. 100 Absolute change in percentage points ? High-impact zones Comparison area Comparison area Coverage (%) Assisted delivery* Postnatal visit Postnatal vitamin A *Includes only deliveries assisted by doctor, nurse or midwife Figure 17 shows reported coverage of assisted deliveries and postnatal care in the HIZs and the comparison area in Deliveries assisted by skilled providers were relatively common at baseline in the HIZs (76%) and remained stable over time. In the comparison area, women reporting an assisted delivery increased from 63% to 75% in 2006; the difference in differences test was not statistically significant (p=0.10). Postnatal visits were also very high at baseline, and decreased somewhat in the HIZs, while remaining stable in the comparison area. The change in pp over time were significantly different between the HIZs and the comparison area (p=0.04). Supplementation with vitamin A within 60 days of birth increased significantly in the HIZs, and this increase was significantly greater in HIZs than the in the comparison area (p=0.03). 39

52 Indicators in the expansion area. The departments of Ouèmè, Plateau, Zou and Collines were part of the ACSD expansion phase and are included in our national comparison area, excluding the HIZs. These departments represented 38% of the comparison area sample in the 2006 survey. Figure 18 presents key coverage indicators for three groups: 1) the HIZs, 2) the ACSD expansion area, excluding the HIZs, and 3) the rest of the country excluding the ACSD expansion area, HIZs and Cotonou). For all indicators except breastfeeding, coverage in the HIZs was similar to that in the comparison area after excluding the expansion area. In fact, coverage in expansion area was slightly higher than the HIZs for ITNs (p<0.001), DPT3 (p=0.03), skilled delivery (p<0.001) and exclusive breastfeeding (p=0.03). In short, removing the expansion zones from the national comparison area did not change the conclusion that the HIZs did not perform better than the rest of the country in terms of the outcomes under study. Appendix M provides further coverage results for the expansion area. Figure 18: Coverage levels for selected indicators in ACSD high-impact zones, ACSD expansion area, and national comparison (excluding HIZs, expansion area and Cotonou) as measured in DHS , Benin. HID 100 Expansion areas (excluding HIZ) Nat'l Comparison (excluding HIZs, Exp. Areas & Cotonou) 86 Coverage (%) Vitamin A for children 6-59m ITN for children 0-59m DPT3 for children 12-23m Skilled delivery for births in previous 12m Exclusive breastfeeding in infants 0-5m 40

53 5.3 Summary and interpretation of results Table 9 summarizes the main results of the analyses of time trends in coverage. Most indicators did not improve in either the HIZs or comparison area. Indicators showing no significant improvement included vaccination, antenatal care, delivery care, case-management and infant feeding. IPTp with SP for women increased only slightly in the HIZs and comparison area. Two sets of indicators improved rapidly in both HIZs and comparison area: vitamin A supplementation among post-partum women and children 6-59 months and ITN utilization among children and pregnant women. Significantly different trends were found between HIZs and the comparison area only for ITN use, postnatal care and postnatal vitamin A coverage. For ITN use, coverage for pregnant women increased significantly in both HIZs and comparison area, with significantly greater gains in the HIZs For postnatal visits, coverage declined in the HIZ areas and remained stable in the comparison area. For postnatal vitamin A, coverage increased significantly faster in the ACSD than in the comparison area. The findings for postnatal visits and postnatal vitamin A seem likely to be due to chance, because the level of statistical significance was borderline (p=0.04 and 0.03, respectively). These results, taken together, suggest that ACSD as implemented in the HIZs in Benin did not have an effect on coverage levels for the interventions targeted for accelerated implementation. A technical team from Benin reviewed and discussed these preliminary results in October Team members included those directly involved either in ACSD implementation or in the collection and analysis of the data used in the evaluation. The interpretation presented below is largely based on these discussions and the review of implementation and contextual documentation. EPI+. In , coverage levels for measles and DPT3 were around 50 to 60%, well below the stated ACSD target of 80% EPI coverage and universal child immunization goals. The stagnation in vaccination coverage in both HIZs and nationally is compatible with the information provided by local officials. According to child health program managers, routine vaccination activities such as monitoring and supervision have received less priority nationwide in recent years due to the time and human resources needed to organize national immunization days, specifically for polio. Between 2003 and 2007, Benin has carried-out 13 national polio campaigns (Appendix table C2) and this may have contributed to the stagnation in routine vaccination services. Administrative reports and key informants also indicated that vaccination outreach activities have faltered, possibly due to the general deterioration of transportation resources and changes in how motorcycles and per diems are managed for health workers. v This may have resulted in low health worker motivation and irregular outreach activities. Coverage of vitamin A supplementation among children increased markedly between 2001 and in both ACSD and comparison area. Vitamin A distribution was linked to national vaccination campaigns starting in late The frequency of national vaccination campaigns, especially polio, is likely to have contributed to this marked increase. UNICEF support and inputs of vitamin A at the national level may explain increases in the comparison area. MOH officials reported that there have not been substantive differences in vitamin supplementation activities between ACSD zones and the rest of the country. UNICEF reported having supported these nationwide efforts through funding from the Canadian government. UNICEF and the MOH noted in key informant interviews that increases in coverage with vitamin A supplementation are considered one of the main achievements of ACSD. To our knowledge, there have been no activities supporting vitamin A supplementation in the HIZs other than those supported by UNICEF. v Before 2001, the motorcycles given for outreach activities became the property of the health workers after 3 years of use. The motorcycles now become the property of the health center, which may have implications for transportation maintenance and health worker motivation. 41

54 Table 9: Summary of ACSD coverage results in ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin. COVERAGE INDICATOR AREA BASELINE VALUE IN 2001 (%) ABSOLUTE CHANGE (% POINTS) DIFFERENCE IN DIFFERENCES TEST (p LEVEL) Measles vaccine HIZ 51-2 Comparison >0.10 DPT HIZ 63-3 Comparison 67-3 >0.10 Vitamin A to child HIZ Comparison >0.10 ITN for child HIZ Comparison >0.10 Net for pregnant woman HIZ Comparison >0.10 ITN for woman HIZ ~0 +22 Comparison ~0 +17 <0.001* Any antimalarial for fever HIZ 70-3 Comparison 66-8 >0.10 Careseeking for pneumonia HIZ 32-2 Comparison >0.10 Oral rehydration for HIZ 38-4 diarrhea Comparison 43-1 >0.10 Breastfeeding initiation HIZ Comparison >0.10 Exclusive breastfeeding HIZ Comparison Complementary feeding HIZ 80-2 Comparison >0.10 Antenatal care (3+ visits) HIZ 71-7 Comparison >0.10 IPTp with SP HIZ ~0 +7 Comparison ~0 +7 >0.10 Tetanus toxoid in HIZ pregnancy Comparison >0.10 Skilled delivery HIZ 76-2 Comparison >0.10 Postnatal visit HIZ Comparison Postnatal vitamin A HIZ Comparison * Difference in end-line estimates only, assuming 0% coverage with ITNs at baseline in HIZs and comparison area. ACSD in Benin had an objective of achieving 80% coverage for all EPI+ interventions. In spite of the significant increases, vitamin A levels remain at around 60% in all areas, and two in every five children are not currently covered. 42

55 The promotion, distribution and re-treatment of ITNs were large components of the ACSD strategy in Benin and elsewhere. Increases in coverage with ITNs were observed between 2001 and in HIZs and nation-wide. However, these increases were much less than the expected objective of 60% coverage among children and pregnant women in the HIZs. Only one quarter of all children under age five reported sleeping under a treated net the previous night in The relatively modest effect on coverage is consistent with the report of widespread stock-outs in nets, starting in late 2005 and persisting until the end-line survey, and the delay in re-treatment campaigns in the HIZs (carried out in November 2006, just after the DHS data collection). Issues in retreating bednets, such as the onerous logistics and problems with meeting the required periodicity, prompted the government to opt for longlasting bednets in In fact, the 2003 and 2005 levels of ITN use were observed to be higher than those in Although there are comparability problems with the 2003 and 2005 surveys, this trend is likely reflective of changes in pricing structures vi and subsequent ITN stock-outs. This finding may have important implications for sustainability of such interventions. The fact that one out of four children slept under a net the night before the survey suggests that the despite the stock-outs there were still some nets in the zone that were re-treated and utilized. IMCI+. Efforts were made to introduce and strengthen facility-based and community IMCI in the HIZs. UNICEF and other partners developed supervision and communication materials, and deployed, trained and supported CHWs in 2004 and 2005 to try to strengthen this ACSD component. However, in , case management practices for common child illnesses remained unchanged in the HIZs and the comparison area. Administrative and summary reports noted deficiencies in stock management, mobilization at the community level, and lack of motivation of CHWs in both PAK and DAA zones throughout ACSD implementation. Although the indicator concerning treatment of fever presented here appears to have remained relatively stable, this does not necessarily represent effective management of fever because there are high levels of resistance to chloroquine 29 in southern and central Benin. Despite the first-line policy change in 2004, the more effective artemisinin combination therapy for use in treating fever/malaria is still unavailable in Benin except in 12 communes in Mono and Couffo departments receiving support from Africare through the Global Fund and the PNLP. 35 No child with fever received an ACT in in the HIZs, and less than one percent did so in comparison area. Most children are still treated with chloroquine (Appendix table I7), which is no longer an effective treatment for malaria in Benin and could not have contributed significantly to reductions in child mortality. Only one-third of children were taken for care to an appropriate facility for presumptive pneumonia throughout the period from 2001 to Another third did not receive any care and approximately one-fifth received inappropriate treatments (médicaments par terre) from shops or ambulatory vendors. The community case management of ARI is in a pilot phase and does not cover many villages; therefore, it is not surprising that almost no child was reported to receive care from a community health worker for pneumonia. In terms of home management practices for diarrhea, there were no improvements either in the HIZs or comparison area over the period under evaluation, and current levels mean that fewer than one half of children with diarrhea are being properly managed. There was no evidence of improved infant and young child feeding practices in the HIZs over the course of the study period. In fact, there were slight decreases in the proportion of women reporting vi When ACSD was first implemented, the official cost of ITNs was CFA 3500 (~ USD 7.00) for all. With the advent of ACSD, the official price was reduced to CFA 1500 (~ USD 3.00) for pregnant women and children less than five years of age. The price was further reduced in 2005 to CFA 500 (~ USD 1.00) for targeted populations. Program managers report that this most recent price reduction led to a large increase in demand for bednets, which combined with problems in the financing of the program, created widespread ITN stock-outs beginning in late 2005 to early-mid

56 recommended practices in the HIZs, despite other small-scale projects with nutrition components reported as operating in the HIZs. In the comparison area, there were small gains in breastfeeding and feeding indicators between 2001 and Training of health providers in the minimum package of activities for nutrition - including the promotion of exclusive breastfeeding and appropriate feeding practices - took place in PAK and DAA in 2003 and IMCI also includes a focus on infant and young child feeding at both facility and community levels. However, community-based promotion of exclusive breastfeeding and appropriate feeding practices supported by UNICEF started intensively only in 2005, after the dissemination of results from a household survey highlighted the low prevalence of appropriate infant feeding practices in the HIZs, especially PAK. Approximately 400 community-base workers received training in promotion of case management, breastfeeding and feeding practices; UNICEF provided visual aids to assist in their work. Of the three breastfeeding practices assessed, early initiation appears to have responded to the interventions within the study period, but exclusive breastfeeding and continued feeding remained stable and have been found in other settings to require intensive behavior change interventions extending over longer periods. It is therefore possible that more time is required to measure the impact of the efforts started in 2005 on these two practices. The stagnation/decline in feeding practices may also reflect the intensive early emphasis by ACSD-Benin on more vertical interventions such as vitamin A and ITNs relative to case management and feeding practices. ANC+. There was no evidence of significant increases in coverage of ANC interventions in the HIZ over the course of the study period. The ACSD program in Benin had a stated objective of 80% ANC coverage; observed coverage of ANC interventions fell short of this objective. ANC intervention indicators tended to be better in the DAA health zone before (2001) and during ACSD (2006-7). According to administrative reports, ANC kits have been promoted in DAA since late 2005; however as noted above, issues associated with recovery of health center costs associated with the ANC kits remain. Long-standing stock-outs of ITNs have also meant that there have been difficulties in providing the complete set of materials needed for a kit. IPTp was introduced in 2004 in the HIZs and 2005 nationally. vii However, the end-line estimates indicated coverage of less than 10 percent in both HIZs and in the comparison area. Assisted deliveries and postnatal visits were relatively high at baseline and did not improve significantly in the HIZs, although higher coverage was observed in the DAA than the PAK health zone for both interventions. Postnatal supplementation with vitamin A was introduced in the HIZs between 2001 to , and coverage improved in response. Contextual Factors. The contextual factors considered in the evaluation were based on those proposed by Victora et al 10 for child survival programs. Section 3 and appendices A and J provide a more comprehensive description of contextual factors. Given that the findings on coverage do not suggest that ACSD had a significant effect beyond what was happening in the rest of the country, the analysis of contextual factors here examines two questions to better interpret the data: 2. Were there any major disruptions in the HIZs that could explain why ACSD did not lead to a more marked effect on coverage levels? 3. Why did the interventions for which coverage increased substantially in the ACSD areas also improve markedly in the comparison area? vii Baseline coverage estimate assumed to be zero for IPTp in 2001 in HIZs and comparison area; measured coverage was 28% in HIZs and 6% in comparison area, although this was before introduction of the intervention and likely to do survey measurement error. 44

57 To our knowledge, no major events occurred in the HIZs that could have disrupted the deployment or effects of ACSD. We have noted in the text above the implementation factors, such as changes in delivery strategies, pricing and national policies, which may have affected the coverage of certain interventions. We describe other development partner activities in the HIZs in section three and appendix J; however, we would expect these to have a positive effect on coverage. A potential problem affecting the HIZs only was that the introduction of health worker support (per diems and transport costs) for activities such as outreach as part of ACSD, followed by their withdrawal, reportedly led to reduced health worker motivation. However, this managerial decision must be regarded as part of the ACSD intervention, and not as an external factor. In short, there is no evidence of any marked disruption affecting exclusively the HIZs that could have offset a positive effect of ACSD on coverage. Vitamin A supplementation and utilization of ITNs improved significantly in the HIZs as well as in the national comparison area. As noted above, supplementation with vitamin A for children was coupled with polio campaigns nationally starting in UNICEF, through ACSD and other programs, supported these efforts at a national scale in Benin. The distribution, promotion and retreatment of bednets were supported by national programs and other externally funded programs outside the ACSD areas. For example, USAID supported similar ITN programs in Borgou-Alibori and Ouémé-Plateau departments and the Global Fund to Fight AIDS, Tuberculosis and Malaria supported national-level malaria control activities through a US$2.4 million grant issued in 2003, of which a large proportion was utilized for ITN distribution and treatment activities. Finally, changes in policies concerning the price of ITNs, discussed above, may have affected the financial accessibility of ITNs over time in both HIZs and nationally. These activities in the country as a whole may help explain why the observed improvements in vitamin A and ITNs were observed not only in the HIZs where they were delivered through ACSD but also in the comparison area where other agencies promoted their delivery. Methodological Challenges. Here we present a very brief overview of the methodological challenges, noting how they may affect the evaluation results of ACSD coverage. Complementing this section, Appendix L provides a more thorough review of methodological challenges, Appendix F provides descriptions of surveys included in the evaluation, appendices D and E provide indicator definitions and a list of the questions utilized for indicator for each survey and Appendix Q compiles the questionnaires from each survey. The retrospective nature of the evaluation, which necessitates relying on existing even if imperfect data and information presented methodological challenges to evaluation team. 2 The 2001 DHS survey had limited sample sizes for calculation of baseline coverage indicators in the HIZs, especially those indicators measured among limited subgroups. These small sample sizes affect the precision of point estimates and the statistical power to detect small differences over time, even though they are representative of the HIZs. The 2001 and DHS, used in both the adequacy and plausibility comparisons, were very similar in methodology and conduct. Appendices F and K provide a review of differences in the surveys and the associated methodological challenges; however, these differences were minimal and we would not expect them to greatly affect the findings. The data available in the 2001 DHS survey did always allow for calculation of the preferred indicators used for monitoring progress toward the Millennium Development Goals (MDG). 7,8 In the 2001 DHS, the treatment status of bednets used by pregnant women was not collected, bed net use by children was only assessed through the women s questionnaire, the timing of antimalarial administration for febrile children was not available and women who reported a facility-based delivered were not asked about a postnatal visit. viii For the evaluation of time trends between 2001 and , we utilized indicator definitions that could be calculated from the 2001 data to ensure comparability with indicator estimates in (see Appendices D and E). These proxy indicator definitions were less stringent than the preferred indicator in viii It was assumed that women who delivered in a facility received a postnatal visit. In the DHS, all women were questioned about postnatal consultation, regardless of place of delivery. 45

58 all cases; coverage estimates from using the more stringent, MDG preferred coverage indicators are presented in Appendices G, H and I. Taken together, these methodological issues are not likely to influence the endline comparisons between the HIZs and comparison area. Differences in the conduct of the survey, the DHS questionnaires and interviewers style of asking questions may have introduced some bias into the comparison of coverage levels between 2001 and However, these methodological challenges are not likely to change the main evaluation findings or conclusions in any substantial way. 46

59 6. Nutrition In this section, we describe the differences in nutritional status of young children between the ACSD highimpact zones (HIZs) and comparison area; including the rest of the country with the exception of Cotonou and the HIZs. As described in section 2, data from the comparison area were collected in the 2006 DHS. Results from the HIZs represent a combination of the 2006 DHS results and those from a supplemental survey carried out in 2007 using the same methodology, aimed at increasing the sample size in HIZs. About half of the HIZ sample available for analyses comes from each of the two surveys. Three indicators of undernutrition prevalence were calculated from the baseline (2001) and endline (2006-7) surveys: prevalence of stunting (low length for age for children below 24 months; low height for children months of age), wasting (low weight for length/height), and underweight (low weight for age). Based on the 2006 WHO Growth Standards, 11 a cutoff of minus two z-scores was used to define moderate or severe undernutrition and a cutoff of minus three z-scores was used to define severe undernutrition. Mean z scores of the three indices were also calculated. Appendix L presents a schematic of the inclusion and exclusion criteria for children included in the analysis. Results are presented for all children less than five years of age. For stunting, results are also presented for children aged months, the age group with the highest prevalence of this condition. 36 Wasting results are described for children aged less than 24 months. Presentation of the results follows the approach used in the section on coverage indicators. First, the adequacy findings are discussed (time trends in the HIZs), followed by the plausibility results (comparison between HIZs and the rest of the country). Appendix L presents full nutrition results for sub-groups in both areas. 6.1 Results Figure 19 shows that there was little change in the prevalence of stunting over time in the HIZs. The prevalence of underweight declined from 26 to 20% (p=0.02), and the prevalence of wasting from 19 to 13% (p=0.04). 47

60 Figure 19: Time trends in stunting (children months), wasting (children 0-23 months) and underweight (children 0-59 months) in the ACSD high-impact zones as measured in DHS surveys in 2001 and , Benin DHS 2001 DHS Prevalence (%) Stunting Wasting Underweight (children 24-59m) (children 0-23m) (children 0-59m) Table 10 and Figure 20 show results for the HIZs and comparison area at baseline and endline. There was an increase in stunting over time of about two percentage points (pp) for children months in the HIZs, compared to a five pp increase in the comparison area. The increase in overall stunting (moderate or severe) in both areas was due to a rise in the prevalence of severe stunting. Mean height/length for age worsened in both areas. The increases are also present in analyses of all children less than five years of age. There was no significant difference between the HIZs and the comparison area. There were important declines in wasting over time: six and five pp in the HIZs and comparison area, respectively, among children less than 24 months of age. Most of the change can be attributed to declines in moderate rather than severe wasting. Similar results were observed for all children under five, and the analyses of mean weight for length/height were consistent with the prevalence results. There were no significant differences by area. There was a decline in the prevalence of underweight over time: six pp in the HIZ for all children under five years of age, compared to stable levels in the comparison area. Most of the decline was due to severe underweight, and the difference between HIZs and the comparison area was statistically significant. However, there was no decline in mean weight for age, suggesting that the lower tail of the distribution was affected but not the overall weight for age curve. 48

61 Table 10: Summary of anthropometry results in ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin DHS 2006/7 DHS P VALUE NUTRITIONAL INDICATOR Stunting (height for age) P VALUE P VALUE HIGH IMPACT (BASELINE- ENDLINE) GEOGRAPHIC HIGH IMPACT GEOGRAPHIC (BASELINE- ZONES COMPARISON ZONES COMPARISON ENDLINE) n % n % n % n % HIZ NC DIFFERENCE IN DIFFERENCE S months % stunted (< -2 SD) > > % severely stunted (< -3 SD) >0.10 <0.001 >0.10 mean (sd) -2.0 (1.3) -1.9 (1.3) -2.2 (1.4) -2.1 (1.5) > months % stunted (< -2 SD) >0.10 <0.001 > % severely stunted (< -3 SD) >0.10 < mean (sd) -1.7 (1.4) -1.5 (1.5) -1.8 (1.5) -1.9 (1.6) 0.02 Wasting (weight for height) 0-23 months % wasted (< -2 SD) <0.001 >0.10 % severely wasted (< -3 SD) > >0.10 mean (sd) -0.7 (1.4) -0.6 (1.4) -0.3 (1.5) 0.01 (1.6) > months % wasted (< -2 SD) >0.10 > % severely wasted(< -3 SD) >0.10 >0.10 >0.10 mean (sd) -0.3 (1.28) -0.3 (1.2) -0.1(1.4) 0.06 (1.5) >0.10 Underweight (weight for age) 0-59 months % underweight (< -2 SD) > % severely underweight (< -3 SD) <0.001 > mean (sd) -1.2 (1.3) -1.1 (1.2) -1.1 (1.2) -1.0 (1.3) >

62 Figure 20: Prevalence of stunting, underweight and wasting and absolute change in percentage points in the ACSD high-impact zones and comparison area as measured in DHS surveys in 2001 and , Benin. 100 Absolute change in percentage points -8 +2? High-impact zones Comparison area Comparison area 80 Prevalence (%) Stunting Wasting Underweight (children 24-59m) (children 0-23m) (children 0-59m) 5.2 Summary and interpretation of results The results suggest that there was no differential impact of ACSD on nutritional status. Factors considered in this interpretation are presented below. Stunting and wasting. Stunting increased slightly over the period of 2001 to in both the HIZs and the comparison area. This is compatible with the national trends reported by the DHS 2006 for all children under five years. 13 The active process of stunting, or growth faltering, occurs up to the age of 24 months, after which children tend to grow parallel to the growth standards and prevalence remains constant up to five years of age. The most sensitive indicator, therefore, is the prevalence of stunting among children age months. However, for an intervention such as ACSD to have an impact on stunting, children should be exposed to it during their first two years of life when active faltering is occurring. For this reason, analyses were repeated for children who were born in 2004 (when ACSD was fully implemented) and therefore spent their first two years with fully implemented ACSD. Stunting prevalence was 53.5% in the HIZ and 53.4% in the comparison area, confirming the absence of an impact. Stunting is primarily influenced by dietary quality and quantity, as well as by the incidence and severity of infections. Because data on implementation and coverage did not suggest differences between the HIZs and the comparison area in terms of dietary, preventive or case-management interventions, the lack of impact in stunting is not surprising. 50

63 Because analyses of the endline survey (see section three) showed that the HIZs children tended to be slightly poorer than children in the comparison area, we investigated whether these socioeconomic differences might have affected our findings. We used direct standardization techniques to estimate the prevalence of stunting among HIZs children, had their socioeconomic distribution been similar to that observed in the comparison area. The standardized stunting prevalence in HIZ was equal to 51.7%, virtually identical to that in the comparison area. Wasting, on the other hand, decreased similarly in the HIZs and the comparison area, consistent with what was reported by DHS for children under five years in the country as a whole. 13 Underweight. The finding of a decline in severe underweight prevalence in the HIZs, in the absence of a similar decline in the comparison area, was not compatible with the lack of differences in time trends of coverage indicators. Because ACSD implementation was unlikely to explain the observed impact, we sought alternative explanations for the findings. These included: Presence of other nutritional interventions or programs in the HIZs. Although our analyses of contextual factors (see annex J) showed that a number of programs were active in the HIZs, their coverage relative to the total HIZs populations was limited, and it is unlikely that these programs could have led to a generalized impact on the area as a whole. Seasonality and differences between the main DHS and supplemental survey. The main DHS was carried out in the HIZs and comparison area simultaneously, but the supplemental survey - restricted to the HIZs - was carried out in a different season, about six to eight months later than the main survey. When the HIZs results were stratified into the two data collection periods, the endline prevalence of 20% in the HIZs was shown to be a combination of a prevalence of 17.4% in the main DHS and of 23.5% in the supplemental survey. This implied that the observed decline in the HIZs would be even greater had the analyses been restricted to the main DHS survey. In addition, the effect of seasonality should be most marked for wasting (weight for length/height) - a better proxy for acute undernutrition than weight for age - but this was not the case (Table 10). For these reasons, neither seasonality nor differences in the two surveys can explain our findings. Contextual factors. Documentation of contextual factors showed that the Djidja and Pobè zones make up the cereal product granaries of their respective departments, with intense agriculatural production (two rainy seasons). At the national level, there had been a food crisis in some Northern zones in the comparison area between 2005 and This observation prompted a reanalysis of the anthropometric data, stratifying the national comparison area into regions (Table 11). The stagnation in underweight prevalence in the comparison area over time was shown to result from the combination of a sharp increase in the Northern zones (from 25.1 to 33.0% in the baseline and endline surveys, respectively), a sharp decline in the Central zones (from 25.4 to 19.4%) and near stagnation in the Southern region (17.5 and 16.4%). The increase in the North is compatible with the reported food crisis that was limited to this part of the country. Although the HIZs belong to the Southern region, they are contiguous to the Central region (see map in section four) where underweight declined by six pp in the period, the same magnitude of reduction observed in the HIZs. 51

64 Table 11: Prevalence of underweight among children under five years of age, by region of the country, as measured in DHS surveys in 2001 and , Benin. DHS 2001 DHS 2006/7 % % SEVERELY UNDERWEIGHT UNDERWEIGHT (<-2 SD) (< -3 SD) n % SEVERELY UNDERWEIGH T (< -3 SD) DIFFERENCE IN ENDLINE - BASELINE % SEVERELY UNDERWEIGHT (< -3 SD) % UNDERWEIGHT % UNDERWEIGHT GEOGRAPHIC AREA (<-2 SD) n (<-2 SD) ACSD "High Impact" Zones National Comparison area National Comparison stratified by region Northern* Central** Southern*** *Alibori, Atakora **Borgou, Donga, & Collines ***Couffo, Mono, Atlantique, Plateau, Oeume, Zou The examination of alternative explanations suggests that the differences in time trends between HIZs and the national comparison area are due to the food crisis in the Northern zones, which masked a declining trend in the country as a whole. Summing up, there was no evidence of an impact of ACSD on any of the three nutritional indicators studied. 52

65 7. Mortality This section reports on changes in child mortality in the ACSD high-impact zones (HIZs) and in the national comparison area, the latter having been defined earlier in this document. The comparisons in this section differ from those presented previously because the full birth history data collected in the 2006 DHS and its 2007 extension as a supplemental survey to ensure adequate sample sizes in the HIZs, considered here as a single survey, are used to estimate child mortality both before and after ACSD became operational. There are two reasons why we elected to use the survey as the basis for estimating mortality throughout the evaluation period. First, the use of a full birth history allows the calculation of period estimates of mortality from the previous year to 10 or more years in the past. The use of the same survey greatly reduces non-sampling error, although use of this method for short periods may lead to large sampling errors. The second reason for using a single survey to estimate mortality for the two time periods before and after ACSD implementation - is that both estimates are based on the same sample of households. This reduces the sampling error of the difference in mortality between the two periods, and hence enables smaller differences to be measured more precisely. Our main comparison refers to two periods of about three years each, before and after ACSD implementation in the HIZs. As shown in Figure 21, based on the documentation of ACSD implementation, we defined the baseline period as July 1999 to June 2002, and the full implementation period as January 2004 to December 2006, with a phase-in period between. Figure 21: ACSD implementation time periods in Benin for the retrospective mortality analysis using full-birth history data, based on documentation of ACSD implementation. YEAR (from full birth history) Time periods used in mortality analysis A. BASELINE: before implementation ACSD Jun 1999-Jun B. PHASE-IN: start of ACSD interventions Jul 2002-Dec 2003 Compare U5MR C. ENDLINE- Full implementation ACSD Jan 2004-Dec A. BASELINE: No ACSD implementation: start of period chosen for symmetry with period C B. PHASE-IN: Start reinforcement of EPI & cold chain; donation of vehicles, motos & refrigerators; outreach C. ENDLINE = EPI + ITNs; Vitamin A; CHW training & deployment; IPTp introduced 53

66 The under-five mortality rate (U5MR) is our priority indicator for measuring changes in mortality, because the goal of the ACSD project was to reduce it by 25% by the end of One benefit of using U5MR relative to other measures of child mortality (see Box 3) is that it provides the largest sample size, and is less sensitive to age heaping than infant or neonatal mortality. Although we present findings for specific age groups within 0 to 59 months, we have considered U5MR as the primary indicator of mortality impact. Box 3: Measures of child mortality (Expressed as deaths per 1,000 live births) Neonatal mortality (NN) Post-neonatal mortality Infant mortality (IMR) The probability of dying between birth and the first month of life The probability of dying between the exact age of one month and the exact age of one year The probability of dying between birth and exact age one year 7.1 Results Figure 22 presents annual estimates of U5MR in the HIZs and comparison area from 1997 to Mortality over the last 10 years is declining in both areas, and no statistically significant difference in the rate of decline between the HIZs and the comparison area given the large sampling error for the yearly estimates. Figure 22: Annual estimates of under-five mortality rates in the ACSD high-impact zones and comparison area, , Benin. 250 Child mortality (CMR) Under-five mortality (U5MR) The probability of dying between exact ages one and five years The probability of dying between birth and exact age five years 200 U5MR (deaths per 1000 births) High-impact National comparison 95% confidence bounds 95% confidence bounds Year 54

67 Figure 23: Under-five, infant and child mortality rates and absolute change in the ACSD high-impact zones and comparison area, Benin. Under-five mortality rate Absolute change in mortality July June ? Jan Dec July June Jan Dec High-impact zones Comparison area Comparison area July June 2002 Jan Dec Baseline Endline Baseline Endline Baseline Endline Under-five Infant Child Figure 23 presents changes in under-five, infant and child mortality rates for the HIZs and the comparison area in the periods before and after full ACSD implementation. The numbers of births on which these estimates are based are 1,445 in the HIZs at baseline and 1,624 at endline; the corresponding numbers are 7,887 and 7,805 in the comparison area. In every age range examined, mortality appears to have declined faster in the comparison area than in the HIZs. Table 12 presents age-specific mortality rates in the baseline and end line periods for the HIZs and the comparison area, as well as reductions over time and 95% confidence limits for these estimates. Postneonatal mortality was the only indicator that did not seem to have fallen faster in comparison area than in the HIZs. There was a reduction in the U5MR of 18 deaths per thousand, or 13%, between baseline and endline for the HIZs; this reduction was not statistically significant at the 95% confidence level as the limits include zero going from a lower bound of minus five to an upper bound of 41. The national comparison area had double the reduction (36 deaths per thousand, or 25%) of the HIZs, and this reduction was statistically significant at both the 95% and 99% confidence levels. The difference in the rate of under-five mortality decline in the national comparison area relative to the decline in the HIZs was not statistically significant, nor were there statistical differences in the differences in declines for agespecific mortality indicators. 55

68 Table 12: Priority and age-specific mortality rates in high-impact zones and comparison area before and after ACSD implementation, Benin. MORTALITY MEASURES HIGH IMPACT ZONES JUL JUN 2002 JAN DEC 2006 COMPARISON AREA JUL JUN 2002 JAN DEC 2006 HIGH IMPACT ZONES Difference, Baseline (A) - Endline (C) COMPARISON AREA Difference, Baseline (A) - Endline (C) ABSOLUTE p-value ABSOLUTE p-value DIFFERENCES IN DIFFERENCES (P VALUE) Priority Indicator Under-five mortality (5q0) > <0.001 > % CI ( ) ( ) ( ) ( ) ( ) ( ) Age-specific indicators ACSD Phase-in period Neonatal mortality (NN) > <0.001 > % CI ( ) ( ) ( ) ( ) ( ) ( ) Postneonatal mortality (PNN) > > % CI ( ) ( ) ( ) ( ) ( ) ( ) Infant mortality (1q0) > <0.001 > % CI ( ) ( ) ( ) ( ) ( ) ( ) Child mortality (4q1) > < % CI ( ) ( ) ( ) ( ) ( ) ( ) ACSD Phase-in period 7.2 Summary and interpretation of results Based on these findings, the U5MR in Benin has declined by 13% in the period from 1999 to 2006 from 141 to 123 per 1,000 live births about half of the ACSD goal of 25%. However, U5MR declined at almost double this speed in the comparison area where ACSD was not implemented. This finding held true across all age subgroups with the exception of post-neonatal mortality, which declined at about the same rate in both the ACSD HIZs and the comparison area. Contextual Factors We considered the possibility that contextual factors might have offset the impact of ACSD. Our extensive review led to the identification of two contextual factors that might have distorted the findings: socioeconomic status and ethnic composition. In both instances, we carried out additional analyses to explore this possibility. Both the HIZs and the comparison area had similar U5MR at baseline, but important differences at the endline. There was evidence that the HIZs may have worsened over time in socioeconomic terms relative to the rest of the country (section 4), and mortality levels tend to be higher among the poor (section 9). We used direct standardization to adjust mortality for endline differences in wealth, applying quintile-specific mortality rates in the HIZ to the socio-economic distribution of the population in the comparison area. The standardized U5MR in the HIZs was 122 per thousand, still substantially higher than in the comparison area. This suggests that socioeconomic characteristics cannot account for the observed differences in mortality. The next section on equity examines further mortality impact by sex of the child, place of residence (urban/rural) and socio-economic status. 56

69 Methodological Challenges. Three important methodological issues may have affected the results of the retrospective estimation of the effect of ACSD on under-five mortality. The first was the potential effect of data quality issues on the estimates, and specifically whether differences in survey procedures might have affected the comparability of the data collected through the original DHS in 2006 and the supplemental survey conducted in 2007 in the HIZs. We attempted to document survey procedures independently for both parts of the survey, interviewing survey planners and surveyors and even participating in the training of surveyors and early period of data collection for the supplemental survey (Appendices F and K). We did document differences in survey procedures that may have affected data quality and biased the reported estimates, but further analyses suggested that the overall trends and their interpretation were not affected. Appendix K presents a more detailed discussion of these issues and how they were addressed in analysis. A second methodological challenge was the definition of the before and after periods of ACSD implementation. Documentation of implementation is difficult in a retrospective evaluation, and is based by necessity on records maintained for other purposes and the subjective recall of project implementers. The two, three-year periods defined for the purpose of this evaluation were discussed and agreed to with in-country teams composed of ACSD implementers and national counterparts, and we believe that they do distinguish between times before ACSD was implemented and times during which ACSD was fully implemented in the views of those responsible. A third issue is the extent to which ACSD activities may have affected mortality in the remainder of Benin (excluding Cotonou), either directly or indirectly. We address this in the conclusions of the report, which begin on page

70 58

71 8. Equity in coverage, nutrition and mortality In addition to evaluating the impact of ACSD implementation on indicators of coverage, undernutrition and mortality, it is also important to assess whether or not the strategy helped reduce inequities in health. In this section, we describe within-population inequalities for the ACSD HIZs and the comparison area across socioeconomic levels, sex of the child, place of residence and ethnic group. Socioeconomic level was analyzed by wealth quintiles, obtained from an index based on ownership of household assets and housing characteristics (Appendix D). The definition of urban or rural residence was based on the sampling frame from the 2002 national census; this categorization was used as a basis for sampling the 2006 DHS. Ethnic groups included Fon, Yoruba and Adja; all other groups were pooled, as each comprised less than five percent of the study population. Our analyses document how inequalities differ between the HIZs and the national comparison area in 2006/07, after implementation of ACSD, because the sample for the 2001 DHS was too small to support subgroup analyses. Here we present results for both the HIZs and comparison area, but focus on six coverage indicators representing the three ACSD components. For the EPI+ component, we present coverage for measles vaccine and vitamin A supplementation for children. For the IMCI+ component, we present coverage for correct treatment of child diarrhea (ORS, RHF or increased fluids plus continued feeding) and the use of an insecticidetreated bednet by children less than five years of age. For the ANC+ component, we present coverage for three or more antenatal visits and the reported presence of a skilled attendant at delivery. We also assessed equity for the two primary indicators of impact in the evaluation: stunting among children aged months and under-five mortality. Appendix I presents the breakdown of all coverage indicators by sex and by wealth quintiles within the HIZs For the examination of inequities by socioeconomic status, we present measures of inequality using three methods. First, we examine levels of selected indicators by wealth quintile, and present the results in graphs. Second, we calculated two summary measures of inequality. The slope index of inequality shows the absolute difference between the top and bottom of the wealth scale, based on a regression approach that uses data from all quintiles rather than just the two extreme groups. Concentration indices show the extent to which the outcome is equitably distributed across all wealth groups, as reflected in a value between minus one and one. A value of zero indicates that the outcome is equitably distributed across all wealth groups. A negative value indicates disproportionate concentration of the health variable among the poor, for example in the case of disease or malnutrition, where the poor are more likely to be affected. A positive value indicates that the poor are getting less than would be expected had the distribution been equitable, as often occurs for preventive and curative interventions. ix 8.1 Results Socioeconomic inequalities. Figures 24a-h and Table 13 summarize the equity results based on a subset of indicators; appendix M presents the full results. Marked socioeconomic inequities were documented for most coverage indicators in both the HIZs and the comparison area, with children in the poorest group having lower levels of coverage than their better-off peers. For example, a slope index of 48 for measles coverage in the high-impact zones indicates the difference in percentage points (pp) in coverage between the poorest and least poor children. The size of the gap between the poorest and least poor is about pp for most indicators, the exception being diarrhea management in the comparison area for which inequality is negligible. The concentration indices for coverage indicators show similar trends, taking values between 0.06 and 0.25, again with the exclusion of ORT that was remarkably equitable in the comparison area. ix For more information see: ( Techniques/health_eq_tn07.pdf) 59

72 Table 13: Summary indices of socioeconomic inequalities for selected indicators in high-impact zones and comparison area, Benin, INDICATOR SLOPE INDEX OF INEQUALITY* CONCENTRATION INDEX ACSD COMPARISON ACSD COMPARISON Measles coverage Vitamin A (children) ITNs (children) Diarrhea management ANC (3 visits) Skilled delivery Stunting Under-five mortality In the ACSD zones, vaccination coverage and Vitamin A supplementation are weak, even in wealthy households. While prenatal visits (at least 3) figure e and birthing assistance figure f, the levels were near 90% for women from wealthy households. It is not surprising that indicators that require contact with functional health services, such as antenatal or delivery care (Figures 24e-f), show the greatest inequities, as this has been reported for several countries. 37 It is harder to explain why care is so inequitable for interventions that are delivered primarily through campaigns, outreach or community approaches including measles vaccination, ITNs or vitamin A (Figures 24a-c) are so inequitable. Contrary to intervention coverage indicators, the slopes for stunting and mortality (Figures 24 g-h) are in the opposite direction than for coverage, that is, higher levels among the poor than the rich. The summary indices (slope index of inequality and concentration index, table 13) take a negative sign under these conditions. In summary, there were no consistent patterns of greater or lesser inequities in coverage or impact in the HIZs than comparison area, with the exception of diarrhea management, which is marginally less equitable in the HIZs (p=0.03). 60

73 Figure 24a-h: Socioeconomic inequalities, showing breakdown by wealth quintiles of selected indicators in ACSD high-impact zones and the comparison area, Benin,

74 Inequalities by sex of the child. There is no evidence of preferential treatment for boys or girls, either in the HIZs or in the comparison area (table 14). Sex inequalities were not analyzed for antenatal or delivery care, when the sex of the baby was not yet known (assuming a low frequency of pregnancy ultrasound). For the impact indicators, boys were more likely to be stunted than girls (p=0.01) but mortality rates were similar; these findings were consistent for HIZs and the comparison area. Table 14: Selected coverage and impact indicators by child s sex in high-impact zones and comparison area, Benin, COVERAGE or NUTRITIONAL INDICATOR Any measles Innoculation (12-23m) ITN use for under five children Vitamin A supplementation of children (6-59m) Diarrhea management Moderate & severe stunting (24-59m) 2006 DHS + DHS supplemental Male Female AREA TOTAL % n % n HIZs Comparison HIZs Comparison HIZs Comparison HIZs Comparison* HIZs* Comparison* MORTALITY AREA U5MR U5MR Births U5MR Births Under-five mortality HIZs Comparison * p < 0.01 comparison of coverage by child's sex within area Urban/rural inequalities. About one quarter of the sample was urban in the HIZs, and one third in the comparison area. Urban children show higher coverage than rural children for all interventions, in both areas, although not all differences are statistically significant. Urban/rural differentials tended to be less marked in the HIZs for most coverage indicators: measles vaccine, ITNs, vitamin A and antenatal care (table 15). Coverage of diarrhea management showed small urban/rural differentials, and skilled birth attendant showed large differentials in both areas. These findings suggest that urban/rural differentials in coverage, as a whole, were less marked in the HIZs. However, in terms of the impact indicators stunting and mortality the magnitude of the advantage of urban children was similar in both HIZs and the comparison area. 62

75 Table 15: Selected coverage and impact indicators by place of residence in high-impact zones and comparison area, Benin, COVERAGE or NUTRITIONAL INDICATOR Any measles Innoculation (12-23m) ITN use for under five children Vitamin A supplementation of children (6-59m) Diarrhea management Skilled birth attendant 3+ visits ANC care Moderate & severe stunting (24-59m) 2006 DHS + DHS supplemental URBAN RURAL AREA TOTAL % n % n HIZs Comparison* HIZs Comparison* HIZs Comparison* HIZs Comparison HIZs* Comparison* HIZs Comparison* HIZs* Comparison* MORTALITY AREA U5MR U5MR Births U5MR Births Under-five mortality HIZs Comparison * p < 0.01 comparison of coverage by urban-rural within area Ethnic group inequalities. Table 16 presents analyses of ethnic group inequalities. Because the Adja are not numerous in the HIZs, the main comparisons are between the Fon and Yoruba. For most indicators studied, the Yoruba have lower coverage than the Fon in the HIZs, but higher or similar coverage in the national comparison area. For stunting, similar levels are observed among the two groups in the HIZs, but in the comparison area the Yoruba children are significantly less stunted than the Fon. In contrast to coverage, in both the HIZs and the comparison area, U5MR is lower among Yoruba children than Fon children (p<0.01). 63

76 Table 16: Selected coverage and impact indicators according to ethnic group in high-impact zones and comparison area, Benin, INDICATOR Any measles Innoculation (12-23m) ITN use for under five children 2006 DHS + DHS supplemental Adja Fon Yoruba Other AREA TOTAL % n % n % n % n HIZs** Comparison* HIZs Comparison* Vitamin A HIZs** supplementation of Comparison** Diarrhea management Skilled birth attendant 3+ visits ANC care HIZs HIZs** HIZs** Comparison** Comparison** Comparison Moderate & Severe HIZs Stunting (24-59m) Comparison** MORTALITY AREA U5MR U5MR Births U5MR Births U5MR Births U5MR Births Under-five mortality HIZs 123 ± ± Comparison 109 ± ± * p < 0.05 comparison of coverage by Yoruba-Fon within area ** p < 0.01 comparison of coverage by Yoruba-Fon within area Sample size too small for estimate (< 25 children); ± Adja enthnicity combined with "Other" due to small sample size 64

77 9. Conclusions In this section, we summarize the findings of the evaluation, addressing two separate questions: a. Was ACSD implementation associated with improvements in coverage, nutrition and mortality over time? b. If so, was progress in the ACSD zones faster than observed for the rest of the country? Figure 25 summarizes coverage trends in the HIZs and comparison area during the study period. The horizontal axis shows the change in coverage in the HIZs and the vertical axis the corresponding changes in the comparison area. When the indicator increased or decreased to a similar extent in both areas, the points are close to the diagonal. Indicators that are above the diagonal showed better performance in HIZs than in comparison area. The reverse is true for those below the diagonal. Only four indicators two for vitamin A and two for ITNs increased by more than 15 percentage points (pp) in the HIZs all others showed lower increase or even declined. The same four indicators also increased more than 15 pp in the comparison area. Figure 25: Summary of changes between 2001 and in coverage and family practices in highimpact zones and comparison area, Benin. Absolute percentage change in coverage HIZs Absolute percentage change in coverage Comparison area Vitamin A 40 (child) AM for fever DPT3 ORT & feeding TT2 Measles ITN (women) Vitamin A (postnatal) IPTp BF within 1 hr Skilled delivery ITN (child) Complementary feeding Careseeking pneumonia ANC3+ Postnatal visit EBF +51 HIZs +46 comp Key: EPI+ interventions Case management Infant feeding ANC+ interventions Relative to the two questions posed at the beginning of the chapter, the answers for coverage indicators are: a. Most indicators of coverage and behaviors did not improve over time in the HIZs, and even if they did - levels at the end of the implementation period were well below the ACSD targets - all of which were set between 50% and 80%; b. As a whole, the HIZs did not perform better than the comparison area in terms of improvements in coverage. 65

78 Turning to nutrition again referring to our two basic questions, the conclusions are: c. The HIZs showed a reduction over time in underweight and wasting prevalence, but not in stunting. d. The lack of change in stunting and the reduction in wasting prevalence over the course of the study period were similar for the HIZs and the comparison area. The reduction in underweight in the HIZs was not observed in the rest of the country, but lack of progress in the national prevalence level was due to a food crisis in the north of the country; once this region is excluded from the comparison area, time trends in underweight reflect what was observed in the HIZs. Reducing under-five mortality by 25% by 2006 was the primary goal of the ACSD strategy. Our analyses showed that: a. There was a non-significant reduction of 13% in under-five mortality in the HIZs, half of the stated goal of 25%. b. In the comparison area, the U5MR declined by 25% during the same period. Analyses of mortality rates by age subgroups within the 0-59 month range also found no evidence that rates fell more rapidly in the HIZs than in the rest of the country. Analyses of inequalities in coverage indicators showed were limited to comparisons at the end of the study period. Due to small sample sizes in the HIZs at baseline, it was not possible to study trends over time. Our conclusions are: a. Important socioeconomic and urban-rural inequities were present in the HIZs after ACSD implementation, even for interventions distributed through strategies found to promote equity in other contexts such as community-based outreach and/or campaigns for vitamin A supplementation or ITNs. No coverage inequalities were found in by the sex of the child; both boy and girl children in HIZs were equally likely to have received the target interventions. Differences in coverage and impact between the two major ethnic groups Fon and Yoruba were not consistent. b. Levels of inequities in coverage, nutrition and impact indications did not vary significantly between the HIZ and comparison area. Explanations for the negative findings in the Benin ACSD evaluation include the following, alone or in combination: 1) Despite effective strategies, the intensity (effort) of program implementation was insufficient; 2) The quality of implementation was lower than needed for changing practices this could be particularly true for case-management and feeding counseling; and/or 3) The program was implemented as planned, but the strategies and interventions did not work well in this context. The retrospective nature of the evaluation and incomplete documentation does not allow us to answer these questions definitively; however, we briefly consider these explanations here. The ACSD program in Benin received US$ 1.5 million in initial implementation funds; thus, ACSD was somewhat limited in the financial inputs compared to other high-impact countries or even other programs operating in Benin, such as the Global Fund which provided US$ 2.3 million during the period in 2003, or USAID-supported programs with estimated inputs of approximately US$ 3-4 million annually throughout the ACSD period. Delays and insufficient levels or resources were commonly cited in ACSD program reports as barriers to full implementation, especially the absence of incentives (whether monetary or non-monetary) to motivate facility- and community-based health workers. The retrospective evaluation documented that the strongest implementation efforts were directed to vitamin A, ITNs and community IMCI the latter mostly as a strategy for distributing and treating bednets. Other components of ACSD, including vaccination, correct case-management of infections, nutrition and antenatal care, were less strongly supported. These findings are reflected in the coverage results presented here: vitamin A and ITNs increased, but 66

79 interventions and practices promoted through community health workers such as community case management of illness and infant feeding practices stagnated or declined. ACSD program documents and respondents identified the need for future programmatic efforts to increase community mobilization, and provide better support, supervision and incentives to community health workers. Box 4 presents key lessons learned from the ACSD experience and recommendations for future child health programming, according to counterparts in Benin. The lack of a differential positive effect on coverage in HIZs relative to what was observed in the comparison area must be interpreted in light of numerous concurrent initiatives - including those to which ACSD contributed - aimed at improving coverage for proven maternal and child health interventions throughout the whole country. This is particularly true for initiatives seeking rapid increases in coverage for vitamin A supplementation and ITN use by children under five. Despite these combined efforts, however, coverage levels for most interventions were still below 60% in both the ACSD Box 4: The way forward: Lessons learned in ACSD according to Benin counterparts 1. There was too little focus on interventions to improve undernutrition and diarrhea management practices. Promotion of infant feeding practices and promotion of key family practices were not well integrated until 2005; these efforts should be continued and strengthened. 2. CHWs were a promising delivery strategy to reach hard-to-access populations, although CHW motivation and supervision were an on-going challenge, especially in scaling up. CHW systems should be reinforced, with particular emphasis on the five key family practices; In 2005, supervision systems were adapted to re-group CHWs at health facilities for more frequent supervision; this strategy should be continued and strengthened. 3. Limited and discontinued funding may have hindered results. Adequate resources need to be provided to carry-out activities, especially in support of outreach activities and motivation of health staff and volunteers; Commodity security, such as ensuring no stock-outs of ITNs, will be essential in future programming; Program managers need to plan for the discontinuation of funds, so that there is not a gap in funding for activities. 4. ACSD contributed to the uptake of effective interventions, such as ITNs and vitamin A, at the national level; as well as introducing promising strategies in Benin, such as ANC packages. The revision of national child survival policies should continue, with special attention to nutrition; The experiences from ACSD in Benin should continue to contribute to the policy dialogue. 5. The retrospective nature of the evaluation across the four high impact countries posed important challenges ACSD in Benin was implemented later than in other countries, with only 3 yrs of implementation; Future evaluations should measure impact only after sufficient time for continued intensive implementation; Prospective evaluations will be needed that can plan to use the same methods throughout the evaluation process HIZs and the rest of the country in The acceleration effects expected by ACSD planners did not occur in Benin. We examined the possibility that external factors might explain the apparent lack of an impact of ACSD, through extensive reviews of existing data and interviews with key informants. We were unable to identify any contextual factors that might account for the lack of impact. Results on mortality, undernutrition and equity are consistent with the lack of an effect of ACSD on levels of intervention coverage and family practices. The ACSD strategy was implemented with suboptimal resource levels and in comparison to the rest of the country where several other initiatives were also being promoted, ACSD was unable to accelerate progress towards child survival. 67

80 References 1. UNICEF. Accelerating child survival and development: A results-based approach in high under-5 mortality areas. Final Report to CIDA. New York, Bryce, J., Gilroy, K., Black, R.E., Jones, G. & Victora, C.G. A Retrospective Evaluation of the Accelerated Child Survival and Development Project in West Africa; Inception Report. Baltimore, MD, Johns Hopkins University Institute for International Programs, Bryce, J., Victora, C.G., Habicht, J.P., Black, R.E. & Scherpbier, R.W. Programmatic pathways to child survival: results of a multi-country evaluation of Integrated Management of Childhood Illness. Health Policy Plan. 20 Suppl 1: i5-i17 (2005). 4. UNICEF. Accelerating early child survival and development in high under-five mortality areas in the context of health reform and poverty reduction: a results-based approach. UNICEF proposal to Canadian CIDA. New York, Darmstadt, G.L., Bhutta, Z.A., Cousens, S., Adam, T., Walker, N. & de Bernis, L. Evidencebased, cost-effective interventions: how many newborn babies can we save? Lancet. 365 (9463): (2005). 6. Jones, G., Steketee, R.W., Black, R.E., Bhutta, Z.A. & Morris, S.S. How many child deaths can we prevent this year? Lancet. 362 (9377): (2003). 7. Bryce, J., Terreri, N., Victora, C.G., Mason, E., Daelmans, B., Bhutta, Z.A. et al. Countdown to 2015: tracking intervention coverage for child survival. Lancet. 368 (9541): (2006). 8. UNICEF. UNICEF/WHO Meeting on Child Survival Survey-based Indicators. Summary List of Child Survival Indicators. New York, Available at 9. Bryce, J., Boschi-Pinto, C., Shibuya, K. & Black, R.E. WHO estimates of the causes of death in children. Lancet. 365 (9465): (2005). 10. Victora, C.G., Schellenberg, J.A., Huicho, L., Amaral, J., El Arifeen, S., Pariyo, G. et al. Context matters: interpreting impact findings in child survival evaluations. Health Policy Plan. 20 Suppl 1: i18-i31 (2005). 11. W H O Multicentre Growth Reference Study Group & de Onis, M. WHO Child Growth Standards based on length/height, weight and age. Acta Paediatr. 95 (s450): (2006). 12. INSAE & ORC_Macro. Enquête Démographique et de Santé au Benin Cotonou, Benin & Calverton, MD, Institut National de la Statistique et de l Analyse Économique (INSAE) & ORC Macro, INSAE & MEASURE_DHS. Enquête Démographique et de Santé, Benin Rapport Finale. Cotonou, Benin et Calverton, MD, Institut National de la Statistique et de l Analyse Économique (INSAE) et ORC Macro, CERTI. Enquêtes de couverture de la Stratégie Accélérée pour la Survie et le Développement de Enfant [Benin] Cotonou, Benin, CERTI, CDC & UNICEF, CEFORP. Etude de base dans les zones d'intervention du programme de coopération Benin- UNICEF Cotonou, Benin, Centre d'etudes, de Formation et de Recherches sur la Population (CEFORP), République du Benin & UNICEF,

81 16. République_du_Bénin. Troisième recensement général de la population et l habitation (RGPH-3), Cotonou, Benin, Institut National de la Statistique et de l Analyse Economique, Ministère Chargé du Plan, de La Prospective et du Développement, Habicht, J.P., Victora, C.G. & Vaughan, J.P. Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. Int J Epidemiol. 28 (1): 10-8 (1999). 18. INSAE & MEASURE_DHS. Enquête Démographique et de Santé, Benin Rapport Préliminaire. Cotonou, Benin, Institut National de la Statistique et de l Analyse Économique (INSAE), WHO. World Health Statistics. Geneva, UNICEF. State of the World's Children New York, UNICEF, Mills, A., Brugha, R., Hanson, K. & McPake, B. What can be done about the private health sector in low-income countries? Bull World Health Organ. 38 (3): 24-30, 41-4 (2002). 22. Bryce, J., Requejo, J. & 2008_Countdown_Working_Group. Tracking progress in maternal, newborn, and child survival: the 2008 report. (avilable at New York, UNICEF, MOH_Benin. Annuaires des statistiques sanitaires Cotonou, Benin, Ministère de la Santé Publique, République du Bénin, Vodounou, C., Ahovey, E. & Hounkpodote, E. Carte de pauvreté non monétaire au Benin. Cotonou, Benin, Institut National de la Statistique et de l Analyse Économique (INSAE), Appawu, M., Owusu-Agyei, S., Dadzie, S., Asoala, V., Anto, F., Koram, K. et al. Malaria transmission dynamics at a site in northern Ghana proposed for testing malaria vaccines. Trop Med Int Health. 9 (1): (2004). 26. PNLP. Situation de la chloroquine-resistance au Benin, presentation at the ProLIPO Partners Meeting in January 2002 by Dr. Dorothée KINDE-GAZARD (Coordinator of the National Malaria Control Program in Benin) World_Bank. Project Appraisal Document for the Booster Program in Benin, PNLP. Politique et stratégies nationales de lutte contre le paludisme, presentation at CIEVRA in July 2006 made by Dr. Hortense KOSSOU (Coordinator of the National Malaria Control Program in Benin) Aubouy, A., Fievet, N., Bertin, G., Sagbo, J.C., Kossou, H., Kinde-Gazard, D. et al. Dramatically decreased therapeutic efficacy of chloroquine and sulfadoxine-pyrimethamine, but not mefloquine, in southern Benin. Trop Med Int Health. 12 (7): (2007). 30. MARA_ARMA. Benin: Duration of the Malaria Transmission Season, available at: World_Food_Program. Where we work - Benin. Information obtained from: accessed 15 February MOH_Benin. Annuaires des statistiques sanitaires Cotonou, Benin, Ministère de la Santé Publique, République du Bénin,

82 33. Wild poliovirus importations--west and Central Africa, January 2003-March MMWR Morb Mortal Wkly Rep. 53 (20): (2004). 34. UNICEF. Accelerated Child Survival and Development in Benin. Cotonou, Benin, Unpublished report, Global_Fund_to_Fight AIDS, T., _and_malaria. Grant Performance Report: Support for the fight against malaria in the Mono and Couffo departments, Africare. Obtained from: ountryid=ben, accessed 17 November 2007, Shrimpton, R., Victora, C.G., Onis, M., Lima, R.C., Blossner, M. & Clugston, G. World wide timing of growth faltering: implications for nutritional interventions. Pediatrics. 107 (5): E75 (2001). 37. Victora, C.G., Wagstaff, A., Schellenberg, J.A., Gwatkin, D., Claeson, M. & Habicht, J.P. Applying an equity lens to child health and mortality: more of the same is not enough. Lancet. 362 (9379): (2003). 70

83 ACCELERATING CHILD SURVIVAL AND DEVELOPMENT (ACSD) Final Report The Retrospective Evaluation of ACSD: Benin APPENDICES Submitted to UNICEF Headquarters on 7 October 2008 Institute for International Programs Johns Hopkins Bloomberg School of Public Health Baltimore, MD A. Description of Benin and high-impact zones B. Methodology for documentation of implementation activities and contextual factors C. Documentation of ACSD implementation in high-impact zones D. Definition of priority indicators E. Comparison of survey questions used for priority coverage indicator calculation F. Methodologies of surveys and other data in Benin G. Tables presenting priority indicators over time for ACSD high-impact zones H. Tables presenting comparisons of priority indicators over time in ACSD high-impact zones and the comparison area I. Tables presenting results for key indicators in the ACSD high-impact zones by sociodemographic characteristics of the population J. Summary of contextual factors K. Description of methodological challenges L. Tables presenting additional nutrition analyses M. Tables presenting additional equity analyses N. References for the appendices O. Annotated list of documents reviewed in the ACSD evaluation (file available upon request)

84 APPENDIX A Description of Benin and high-impact health zones Located in West Africa, the Benin Republic covers a surface of 114,763 square kilometers. It is bordered in the north by Burkina Faso and Niger, in the east by Nigeria, in the west by Togo and in the south by the Atlantic Ocean with a coastline spanning 120 kilometers. Geographically, Benin is made Figure A1: Map of regions, Benin BENIN REPARTITION POPULATION 2002 PAR DEPARTEMENT BURKINA FASO Légende Océan Atlantique Alibori Atacora Littorral Borgou Col lines Couffo Donga Littoral Mono Oueme Plateau Zo u Atacora TOGO Donga Collines Borgou Plateau Zou # Couffo # Oueme # # Atlantique # Mono Littoral Alibori NIGERIA NIGER N OCEAN ATLANTIQUE Kilometers up of a sandy coastal band in the south, with the two plateau zones of the Atacora in the north where all of Benin s rivers originate. Administratively, Benin is made up of 12 departments established since January 15, 1999: Atacora, Donga, Borgou, Alibori, Atlantic, Littoral, Mono, Couffo, Oueme, Plateau, Zou and Collines. These departments are divided into 77 communes, including 3 with particular status: Cotonou, Porto- Novo and Parakou. These communes are subdivided in 546 arrondissements comprised of villages and neighborhoods in towns. The Beninese population is characterized by a plurality of ethnic groups and languages, and contains about fifty ethnic groups. The informal sector continues to develop in the country: according to the third general Population and Housing Census of February 2002 (RGPH3), 95% of people are employed in the informal sector. The secondary sector contributes for 13% to the Gross Domestic Product (GDP) with 35% for the primary sector and 52% for the tertiary sector. The process of administrative decentralization, aimed at promoting community lead development is currently underway. In the last twenty three years, the population of Benin has doubled in size, growing from 3,331,210 in 1979 to 6,769,914 inhabitants in 2002; a population growth rate of 3.25% between the 1992 and 2002 censuses 1. This growth rate is very high compared to the average growth rate of other A2

85 similarly developed countries. The population is 51.4% female, with 46% of women in reproductive age (15-49 years), with 3.5% of the population under twelve moths and 17.4% under five years of age. With 46.8% of the population under 15 years of age, the Beninese population is very young. Figure 1 depicts the population distribution by department in Cultural and educational aspects The population of Benin is characterized by a plurality of ethnic groups and languages; approximately fifty ethnics groups exist, but French remains the working language. For most of Benin, access to school has notably progressed during the past ten years, however recently it has become relatively stagnated. From to , the gross schooling in primary education changed from 96.4%, with boys at 108% and girls 84.3%, to 95.6%, with boys at 104.8% and girls 86.1 %. The rate of completion of primary education has increased from 37% in , with boys at 51% and girls 24%, to 54% in , with girls at 42%. The proportion of school aged children aged 6-14 is currently 56.2% as of Health situation in Benin Since 1996, the population has been characterized with a high fertility rate, at approximately 6.3 children per woman. Fertility has since decreased, but remains relatively high, estimated at 5.6 children per woman in 2001 and 5.7 children per woman in Like other developing countries, Benin is characterized by high child mortality rates; however, the estimated IMR decreased from 83 per 1000 live births in to 67 per 1000 live births in , and the U5MR decreased from 151 per 1000 population to 125 per 1000 population for the same period. 2 The maternal mortality ratio, estimated at 498 per live births in 1996 has remained stable and was estimated at 474 per live births in According to the Expanded Programme of Immunization (EPI), all children should receive all vaccinations before their first birthday. In 1996, 49% of the month old children were fully immunized; however 15% of children in this age group did not receive any vaccinations. In 2001, 59% of month old children were fully vaccinated and 7% of children did not receive any vaccinations. In 2006, the rates were 47% were fully immunized and 7% did not receive any vaccinations. According to the MOH 2005 routine health information system data, the most frequent child consultations were for malaria (41%), acute respiratory infections ARI (20%) and diarrheal diseases (8%). Case fatality rates for malaria are relatively high among children. Figure A2 presents the overall profile of child, neonatal and maternal health from the most recent Countdown to 2015 report. 3 A3

86 Figure A2: Benin country profile of maternal, newborn and child survival extracted from Countdown to 2015, 2008 Report. 3 A4