IMPACT AUDIT. Development Media International (D.M.I.) Child Survival Program. ImpactMatters

Transcription

1 IMPACT AUDIT Development Media International (D.M.I.) Child Survival Program ImpactMatters Published November 27, ImpactMatters

2 FINDINGS Child Survival Program MISSION PROBLEM To save the lives of children and newborns in countries with high child mortality rates. Families lack awareness of healthy behaviors. INTERVENTION D.M.I. designs, develops and runs media campaigns that provide repeated exposure to information on how to improve child health. ENGAGEMENT Appropriate careseeking, antenatal care, health-center births IMPACT Reduction in mortality among children under 5 IMPACT AND COST Approximately $1,700 per child life saved IMPACT AND COST CALCULATION We estimated the impact and cost of a nationwide scaleup of D.M.I. s child survival program in Burkina Faso. To do so, we use data from a randomized controlled trial D.M.I. conducted on a regional media campaign in Burkina Faso from The trial collected survey data and health facility data. The survey data did not find statistically significant declines in child mortality. However, the health facility data found substantial increases in health visits in regions where D.M.I. broadcast the campaign. As the survey data were ambiguous, D.M.I. modeled changes in child mortality using health facility data and a statistical model called the Lives Saved Tool. We use an adjusted version of D.M.I. s methodology to calculate campaign impact. We then compare impact to D.M.I. s and patients total costs to calculate a cost/benefit ratio of $1,700 per child life saved. Considering only D.M.I. s direct costs, the cost/benefit ratio falls to $400 per child life saved. QUALITY OF EVIDENCE QUALITY OF EVIDENCE ASSESSMENT The impact and cost model uses two sources of high-quality evidence: a well-conducted randomized controlled trial of D.M.I. s regional campaign and the Lives Saved Tool, an evidence-based statistical model for estimating the effectiveness of health interventions.

3 FINDINGS Organizational Effectiveness GEOGRAPHY STAGE AGE SIZE Burkina Faso, Democratic Republic of the Congo, India and Kenya Scale Current model in operation for 6 years Approximately 2.9 million women of reproductive age reached in a national campaign in Burkina Faso QUALITY OF MONITORING SYSTEMS D.M.I. excels at using data to make informed decisions about campaign design and implementation and transparently describes the methods it uses to track outcomes. We find two shortfalls: (1) D.M.I. lacks a clear methodology for complementing survey data with administrative data from health facilities; and (2) D.M.I. has begun collecting timeseries data without detailed research objectives and an analysis plan. Criteria Credible Actionable Responsible Transportable ACTIVITY DATA TARGETING DATA ENGAGEMENT DATA FEEDBACK DATA OUTCOMES DATA LEARNING AND ITERATION D.M.I. considered four changes to the design of its program over the last three years. Three of these changes were appropriately tested and systematically implemented. The fourth change a contract for D.M.I. to produce medical training videos was directed by a partner and did not follow an appropriate process for iterative learning. Criteria ITERATION IS BASED ON DATA DATA ARE OF HIGH QUALITY ITERATION IS SYSTEMATIC AND PERIODIC Finding Yes Yes Yes

4 Table of Contents Executive Summary 6 Nonprofit Comment 10 Nonprofit Program Description 12 Mission 12 Theory of Change 12 Problem 12 Activities 13 Assumptions 14 Risks 15 Measures of Engagement 15 Measures of Impact 15 Program Details 16 Geography 16 Stage 16 Age and Scale 16 Other Programs 16 Funding 17 Impact And Cost 18 Findings 19 Impact 20 Cost 26 Sensitivity Analysis 28 Displacement and Other Effects 30 Evidence from other mass media campaigns 37 Quality of Monitoring Systems 39 Rating 40 Activities 42 Targeting 42 Engagement 43 Feedback 44 Outcomes 45 Learning and Iteration 46 Rating 46 Iterations 48 Viral videos 48 Social Media 48 Health Worker Call-In Show 49 Training Videos for Health Workers 49 Back Matter 50 Metadata 50 Monitoring Systems Scoring 51 Glossary 52 Reference List 57 Quality of Evidence 32 Internal Evidence

5 ImpactMatters Broadway New York, New York Some rights reserved. Rights and Permissions This work is available under the Creative Commons Attribution- ShareAlike 4.0 International license: Under the Creative Commons Attribution-ShareAlike license, you are free to copy and redistribute the material in any medium or format and adapt this work, for any purpose, including commercial purposes, under the following conditions: Attribution: Please cite the work as follows: ImpactMatters. Impact Audit of Development Media International License: Creative Commons Attribution-ShareAlike CC BY SA 4.0. ShareAlike: If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. Feedback We welcome your feedback at 5

6 EXECUTIVE SUMMARY Program Description and Top-Level Findings Development Media International s (D.M.I.) child survival program reduces newborn and early-childhood deaths by running media campaigns that encourage behaviors that could save the lives of children. D.M.I. runs these campaigns in countries with high rates of child mortality and measures success by the number of child lives saved. This audit analyzes the impact of a radio campaign D.M.I. ran nationwide in Burkina Faso between 2015 and During D.M.I. s nationwide campaign, it cost an average of $1,700 to save the life of a child. This figure includes costs borne by D.M.I. and patients. Patients pay for about three quarters of the cost per life saved, mostly through drug purchases. The campaign in Burkina Faso saved 6,200 lives in 2015 and That translates to a reduction in child mortality of between 4 and 6 percent of the mortality rate. The national campaign began in 2015 and will end in 2017, but our benefit-cost analysis is restricted to because cost data are not yet available for By the time the three-year campaign ends, we predict it will save about 8,700 lives. For the campaign, the evidence of impact is strong. D.M.I. and the London School of Hygiene and Tropical Medicine conducted a randomized controlled trial (R.C.T.) to directly measure the campaign s impact on child mortality in Burkina Faso in We base our estimate of the impact of the campaign on a statistical model fit by D.M.I. (described below). Impact and cost D.M.I. s nationwide campaign saved child lives at an average cost of $1,700 per life saved. The evidence for the program s impact comes from a randomized controlled trial conducted between 2012 and The investigators defined regions of Burkina Faso where they could isolate exposure to radio campaigns and randomly assigned those 6

7 regions to treatment and control groups. D.M.I. broadcast a radio campaign in the treatment regions for three years. To estimate changes in child mortality and parents behavior, the study gathered data on child health outcomes from two sources: household surveys and records of patient visits to health clinics. The survey data on child mortality were inconclusive. Although child mortality declined in the treatment zone, it also declined in the control zone. The trial s survey data on child mortality did not prove that the media campaign caused the observed decline in mortality. i Confronted with the failure of the survey data to show impacts child mortality, the investigators instead used administrative data from health facilities to demonstrate the impact of the media campaign. During the R.C.T., caregivers and pregnant women in the treatment zone increased their use of medical care faster than those in the control zone. Pregnant women attended routine antenatal visits and chose to deliver babies at health facilities. Parents brought children who were sick with pneumonia, malaria and diarrhea to health facilities. Those changes in parents behavior led to life-saving medical treatments for their children. D.M.I. used the Lives Saved Tool (LiST), a widely accepted statistical model, to estimate the decline in child mortality that resulted from the observed increase in care-seeking. D.M.I. calculated a decline of 4 to 6 percent in child deaths attributable to its radio campaign. We estimate that the national campaign during saved 6,200 lives. Although the radio campaign continued to broadcast in 2017, we exclude these impacts from our analysis because our cost data only cover This impact was achieved at a cost of $10.4 million. Of that total cost, 23 percent was incurred by D.M.I. in executing the campaigns. Patients paid the remaining 77 percent in health care expenses. Although the radio campaign in Burkina Faso increased the number of patients who visited health clinics, the best econometric study available concludes that the increase in visits would not lead to higher public health expenditures. i The statistical power of the trial was limited by the nature of the media campaign. Media campaigns are broadcast over a large area, which means that investigators can choose fewer treatment and control sites. Due to the way that clustering affects statistical power, having fewer treatment and control sites in the design reduces overall statistical power to detect an intervention effect. 7

8 Quality of Evidence The quality of evidence for the impact of D.M.I. s child survival program is high. Mass media campaigns have been used for decades to promote child survival. Most studies of their impact have been weak, relying on respondents to report whether or not they heard the broadcasts. The D.M.I. study in Burkina Faso is exceptionally well-designed among such studies, in that it randomized exposure to the radio broadcasts. Data collected by surveying a sample of households in could not demonstrate the impact of the broadcasts on lives saved. However, data collected from health facilities during the trial show that the campaign led parents to seek medical care more often. In the treatment arm of the study, visits to health clinics for sick children rose by anywhere from 16 to 35 percent more than the control zone. Clinics also reported that uptake of antenatal care and births at facilities increased by 6 to 9 percent. To show how many lives could have been saved by parents decisions to seek medical treatment for sick children, D.M.I. used LiST statistical model to estimate how many lives can be saved by particular health interventions. The LiST model relies on multiple peerreviewed studies to show how mortality responds to all the major risks of child mortality. Estimates from the LiST tool are calibrated to local health risks, such as the prevalence of certain diseases and the coverage of meaning how much of the population has access to care such as vaccinations. D.M.I. s R.C.T. showed that parents who heard the radio campaign brought more children into health facilities for treatment. The LiST model estimated how many lives could be saved by treatments for pneumonia, diarrhea and malaria provided at those visits. Quality of Monitoring Systems D.M.I. has outstanding monitoring systems in the many countries where it runs mass media campaigns. Every media campaign begins with formative research on the media and health landscapes of the country where it will be broadcast. Staff research what kinds of media are widely consumed in the country. Staff also identify major health risks that the media campaign can address and the barriers to populations seeking and receiving care for those health risks. D.M.I. distills the research into a message brief that articulates target behaviors for parents and how specific obstacles to those behaviors should be addressed. 8

9 D.M.I. takes steps to ensure campaign broadcasts run smoothly. Radio spots are pretested with focus groups before broadcast. Broadcasting is monitored by independent listeners, tracking software planted at radio stations and self-reporting by radio stations. Once a campaign goes on air, D.M.I. collects feedback from listeners to identify any needed changes to the design of the campaign. Script writers incorporate feedback from listeners into their new scripts. D.M.I. has begun to gather even more data in Burkina Faso in the hopes of using time series analysis to show that media broadcasts are responsible for behavior change. D.M.I. has yet to set in place a clear plan for analyzing this data, potentially leading to wasted resources. D.M.I. should develop detailed research objectives and an analysis plan for the time series analysis prior to data collection. Learning and Iteration D.M.I. s process for considering changes to its interventions is excellent. Program modifications are subjected to high-quality pilot tests before they are expanded. Senior management meet regularly to consider which concepts for program modifications have sufficient evidence to proceed with pilot tests. A unit called D.M.I. Labs is responsible for managing a portfolio of ideas under development. D.M.I. Labs tests ideas in stages, beginning with small proof-of-concept studies before proceeding with larger trials. D.M.I. has tested new ways to reach its audiences. A pilot test in Kinshasa showed that social media can be effectively combined with radio and television broadcasts. Another pilot test in Burkina Faso showed that videos exchanged by phone memory cards rather than by broadcast are effective in reaching people that do not listen to radio or watch television. In the past three years, D.M.I. only undertook one change to its model without adequate testing, a project to produce a series of short films to train health workers for the University of the West Indies. The impact of the videos was insufficiently researched and the pilot was not evaluated. This project was a one-off and not representative of the typical high level of care D.M.I. takes in planning campaign changes. 9

10 NONPROFIT COMMENT DMI delivers mass media campaigns to improve health and save lives in low-income countries. We ran the first randomised controlled trial to show that radio can change lifesaving behaviours. We have two priorities: to continue to generate ground-breaking research, while taking proven strategies to the largest possible scale, saving as many lives as possible. DMI thanks ImpactMatters for the rigour and thoroughness of this impact audit. We feel that the results of this audit reflect the emphasis that DMI places on robustly evaluating the impact of our work. We would also like to thank the London School of Hygiene and Tropical Medicine, our evaluation partner for the randomised controlled trial that is referenced in this report as well as our broadcast partners in Burkina Faso without whom we would not have been able to save these lives. We recognise and appreciate the efforts ImpactMatters have made to include the full costs of saving lives, including costs to patients and governments. It s worth emphasising that of the $1700 required to save a life, just $400 is DMI s cost. The remainder, as ImpactMatters make clear, is largely the cost of drugs and is born by the patients. Readers should note that the cost per cycle of anti-malarial or antibiotic treatment is in the $1-2 range, which is affordable for most households. We are grateful to ImpactMatters for highlighting that our impact estimates do not include our impact on morbidity and therefore understate the impact of the program on improved child health. Inspired by this report, we intend to carry out some research that will allow us to estimate our impact on morbidity as well as mortality. In response to particular critiques: Methodology for complementing survey data with administrative data from health facilities DMI s methodology for interpreting facility and survey data (and any differences between the two) is outlined in the RCT results paper (publication pending). 10

11 In general we believe that facility and survey data measure two different things. Facility data allows us to collect statistically robust data on behaviours. We are not able to collect large enough samples during a single survey to conclusively measure responses to malaria, diarrhoea and pneumonia, for example, whereas facility data, with tens or even hundreds of thousands of cases, is far more powerful. Survey data is much more useful for measuring other indicators, such as media exposure, knowledge and intentions. The results will be stronger, of course, if knowledge, intentions and actual behaviours all point in the same direction. Time Series Data We actually think that the time-series approach is a more cost-effective way of collecting data than bespoke surveys, which require multiple (often underpowered) baseline and endline surveys for each and every campaign that DMI needs to evaluate. The time series platform was not designed to test a single, specific hypothesis but rather provides continuous data that DMI uses to evaluate and adapt the design of multiple campaigns. For any specific campaign (such as our scale-up funded by Comic Relief) we detail objectives beforehand. In the case of Comic Relief, this was detailed in the research protocol that received ethical approval from the Ministry of Health s Institutional Review Board. The sample sizes collected are calculated to be appropriate for measuring the multiple indicators in which we are interested. It should be noted that the annual cost of a time-series survey is just $75k thanks to a highly efficient staff of 7 data collectors armed with a motorbike and smart phone (whereas a single survey with a sample size of 5000 in this region can cost over $100k). As described in this report, DMI is committed to transparency, measuring the impact of our work, and continuous improvement. We welcome any questions or comments that readers may have and will continue to share the findings from our research in order to achieve greater collective impact. 11

12 NONPROFIT PROGRAM DESCRIPTION This section summarizes D.M.I. s mission and constructs a theory of change that describes the problem, D.M.I. s intervention and the appropriate measures of engagement and impact. Mission To save the lives of children and newborns in countries with high child mortality rates. Theory of Change PROBLEM Poor and rural populations in developing countries suffer extremely high rates of child mortality. Parents of young children do not always seek medical care or use appropriate home remedies for a wide range of treatable diseases. Under-treatment is partially due to a lack of accurate knowledge about treatments or an incorrect belief that modern medicines are inferior to traditional remedies. Lack of knowledge or inaccurate beliefs are not the only barriers to care for the rural poor. Children do not receive care for myriad reasons: a lack of health facilities and health workers, high cost of care, counterfeit medicine, bad roads, the need for caregivers to work, the stigma of a diagnosis and so on. However, there is scope for mass media to improve knowledge or change beliefs and thus lead to behaviors that promote health. Mass media can correct misinformation about safe and affordable medical care, reinforce the value of preventative care, teach simple rules of thumb about nutrition and hygiene, and reduce stigma associated with seeking reproductive health care and care for certain diseases such as tuberculosis. 12

13 Development Media International (D.M.I.) runs media campaigns to increase knowledge of the availability, quality, cost and benefits of medical care. People do not seek care for a wide range of diseases and other health conditions. D.M.I. has launched campaigns to reduce child mortality, improve reproductive and maternal-child health, increase family planning, increase tuberculosis treatment and improve responses to public health emergencies such as disease outbreaks. These diseases and health conditions have severe consequences: preventable deaths of infants and children under the age of 5; high morbidity and mortality in childbirth; postpartum depression and the attendant consequences for children; unplanned pregnancies; lung disease and intra-household infections of tuberculosis. ACTIVITIES D.M.I. media campaigns use radio, television and social media to disseminate accurate information about appropriate health behaviors to poor and often rural populations. D.M.I. produces several different types of media: short spots for radio and television, callin talk shows for radio and short videos. FORMATIVE RESEARCH D.M.I. uses formative research to determine the strategy of each media campaign. D.M.I. investigates the causes of health problems with a combination of desk research and qualitative field research. D.M.I. researches behaviors that reduce these health problems and the main barriers preventing members of the targeted population from changing behaviors. D.M.I. conducts qualitative field research to validate that behavior change is possible. Based on formative research, D.M.I. writes a message brief that articulates how health messages can change the public s knowledge and attitudes and thus promote behaviors. The brief explains the rationale for why a particular health behavior will improve health for members of the target audience or their children. The brief also lists the major obstacles to behavior change along with factors that may increase the likelihood of behavior change. MEDIA PRODUCTION AND BROADCAST D.M.I. s method for changing behavior, termed Saturation+, calls for repeatedly exposing the target audience to short radio or TV spots, sometimes in combination with longformat radio or TV dramas. D.M.I. conducts market research to select channels to broadcast spots and ensure the right messages reach the target audience frequently. 13

14 In Burkina Faso, scripts were designed for short radio spots and long-format radio dramas. D.M.I. s script writers created stories that dramatize the barriers that prevent people from changing health behaviors. The protagonists in the story face difficult decisions about health care and manage to overcome those obstacles. At the climax of long-format radio dramas, the presenters pause the drama and request listeners call-in and share what they think the characters should do. During call-ins presenters correct callers mistaken beliefs. TOPICS OF THE MEDIA CAMPAIGN D.M.I. s campaign in Burkina Faso covered behaviors that promote child survival. D.M.I. runs campaigns elsewhere on other topics, such as family planning and early child development. ASSUMPTIONS D.M.I. makes reasonable assumptions about the populations it serves, broadcasters and the health care sector. The most important partners by far in the child survival campaign are the national ministry of health and the parents of young children. Media campaigns for public health assume that health clinics have the capacity to provide care. Women must be able to obtain antenatal care from trained health workers and skilled birth attendants. Medicines to treat the leading causes of child mortality, such as antimalarials, oral rehydration salts and antibiotics, must be readily available. Counterfeit and stock-outs have been a recurrent problem in some regions of Africa. Radio and television stations must be able to deliver on their commitments to broadcast spots and long-format dramas. D.M.I. places its radio spots and long-format dramas in the hands of broadcasters. Broadcasters in Burkina Faso include many community radio stations as well as national broadcasters. In the case of Burkina Faso, several aspects of radio stations capacity were at issue. Radio stations needed solar electrical power during frequent electrical outages. D.M.I. s target audience must be able to afford care, including antenatal visits, neonatal care and medicines. Care can be unaffordable because prices of medicine and materials are too high; or because of the way that prices are negotiated in health care; or because insurance markets and financial services fail to facilitate patients access to care. Either on the private market or through a government-sponsored health system, the target audience must be able to afford the care that is promoted by the media campaign. 14

15 Finally, media campaigns assume that the public regularly listens to radio and watches television. Over time, if the public consumes less mass media and more social media, traditional media campaigns will reach less of the public. Similarly, if markets or government support weaken broadcasters over time, radio and television campaigns may have reduced reach. On the other hand, new media channels such as social media and peer-to-peer transmissions offer alternatives to traditional mass media, which could bring new opportunities. Indeed, D.M.I. has successfully incorporated social media into its campaigns, where appropriate. RISKS D.M.I. uses software and paid listeners to monitor radio broadcasts for its campaigns to ensure messages are accurately transmitted. D.M.I. requires on-air presenters to field calls from the public about a radio drama. D.M.I. uses its own research staff and script writers to check that radio drama broadcasts accurately convey health messages. MEASURES OF ENGAGEMENT D.M.I. promotes healthy and health-seeking behaviors among the populations it targets for campaigns. For D.M.I. s child survival campaign, the main measures of audience engagement are: the number of caregivers seeking medical care for their children under age 5 when they display symptoms of diarrhea, pneumonia or malaria; the number of pregnant women obtaining antenatal care; and the number of pregnant women delivering babies in health facilities. D.M.I. focuses on diarrhea, pneumonia and malaria as they are three of the leading causes of child mortality in most low-income countries. D.M.I. tracks other health behaviors in the time-series, such as: breastfeeding initiation within one hour after birth; use of insecticide-treated bed-nets for expecting mothers and children under 5; and handwashing. MEASURES OF IMPACT The impact of the child survival program is the reduction in under-5 child mortality that can be attributed to D.M.I. s campaign. 15

16 Program Details GEOGRAPHY D.M.I. has run child survival media campaigns in four countries: Burkina Faso, Democratic Republic of the Congo (D.R.C.), India and Kenya. D.M.I. is planning a campaign in Mozambique. This audit focuses on the nationwide campaign in Burkina Faso. STAGE In Burkina Faso, D.M.I. has scaled up its child survival program. D.M.I. has articulated a clear methodology for designing mass media campaigns and is now searching for new ways to apply its methodology to public health problems and in other countries. 1 AGE AND SCALE D.M.I. s child survival radio campaign in Burkina Faso is intended to reach 2.9 million mothers and pregnant women between 2015 and ,3,4 OTHER PROGRAMS D.M.I. is running campaigns to promote family planning in Burkina Faso and D.R.C. The campaigns aim to reach 3.5 million and 2.2 million people, respectively, with the objective of increasing use of modern contraceptives. D.M.I. intends to complete the campaign in D.R.C. in 2017 and the campaign in Burkina Faso in ,6 In the past, D.M.I. has worked with partners to create short films promoting early child development. In the Caribbean, D.M.I. created short films aimed at mothers. D.M.I. screened the films in waiting areas of primary health centers. 7 In Jamaica, Peru and Bangladesh, D.M.I. created short films to train health workers on how to properly conduct parent-child home visits. 12 Videos and home visits for parents are intended to change parenting behavior, improving parents understanding of how to foster early child development. D.M.I. is not actively pursuing these programs because they do not follow its cornerstone methodology, Saturation+. D.M.I. is now seeking opportunities to run an early child development campaign using its methodology. D.M.I. has launched media campaigns to promote nutrition for young children and women of childbearing age in two countries. D.M.I. s Tanzania campaign will be completed by 2019 and its Mozambique campaign by ,10 These campaigns have not yet been evaluated for impact. 16

17 In 2006, D.M.I. ran a tuberculosis campaign in Brazil. 11 The campaign encouraged people to get tested for tuberculosis if they had a cough lasting longer than three weeks. The campaign included three TV spots and 10 radio spots broadcast over a 30-day period. D.M.I. intends to run future TB campaigns in Mozambique. D.M.I. is currently developing a program, called D.M.I. Reach, to rapidly create mass media campaigns across sub-saharan Africa in response to public health emergencies. 12 The program is designed to disseminate accurate knowledge about how to prevent transmission of an infectious disease and seek appropriate care. D.M.I. is also developing an Advisory Services arm, which will provide technical assistance to other organizations seeking to run their own media campaigns. FUNDING D.M.I. s expenses on all programs in 2014, 2015 and 2016 totaled $2.5 million, $4.2 million and $3.9 million, respectively (2016 U.S.D.). The child survival program accounted for nearly all of D.M.I. s expenses in In 2015, 59 percent of total expenses went to child survival, 27 percent to child nutrition and 14 percent to family planning. In 2016, child survival and family planning accounted for about a quarter of total expenses each, with 41 percent of total expenses dedicated to child nutrition, 3 percent to early child development and 1 percent to D.M.I. Reach. The budget figures include direct and indirect expenses for each program. ImpactMatters allocated indirect expenses to the five different programs based on a mix of D.M.I. s grant budgets and interviews about how staff time was spent. We are uncertain how well the allocations reflect the actual resource use of each program. Figure 1. Program Shares of Annual D.M.I. Budget 17

18 IMPACT AND COST WHY WE ESTIMATE Impact audits estimate the philanthropic impact and cost of a nonprofit s programmatic interventions. We base those estimates on best available evidence, however imperfect, drawn from the auditee (internal evidence) and research literature (external evidence). As such, our estimates are the best possible evidence-based gauge of philanthropic success. HOW WE ESTIMATE First, we identify outcomes that best capture the auditee s mission. Next, we settle upon ways to measure progress against those outcomes, relying on the tools of modern social science. Second, we report our estimate of impact, the change in outcomes that can be attributed to the auditee s intervention over a designated period of time. We take explicit account of counterfactual success the change in outcomes that would have occurred without the program. And whenever possible, we take explicit account of third-party effects, especially unintended harm to vulnerable individuals because of the auditee s intervention. For benefits that accrue over time for example, the increased earnings from high school graduation we discount these future benefits at a 5 percent discount rate. The length of time over which benefits are assumed to accrue is based on available internal and external data. Third, we report total costs. Total costs include marginal costs (direct program delivery) and fixed costs (for example, administrative overhead) whether those costs are borne by the nonprofit, public authorities, private funders or participants. For programs that generate commercial revenue, the revenue is treated as a subtraction of costs. Fourth, we report the ratio of impact to cost (a benefit/cost ratio). 18

19 Finally, we analyze key factors for example, stage of development, whether the nonprofit be in pilot phase or expansion phase relevant for understanding the audit findings. Typically, impact is estimated on a single outcome. However, if an auditee s intervention affects several outcomes, we report impacts on distinct outcomes separately. Concretely, a program might seek to raise incomes and improve health status. We do not, as yet, attempt to combine the impact on multiple outcomes into a single aggregate outcome. Concretely, by combining the value of the income effects and health-status effects. To aggregate, we would need weights: the relative value of one outcomes weights that would reflect the nonprofit s values (not those of ImpactMatters as auditor). Findings Development Media International s (D.M.I.) nationwide campaign in Burkina Faso to promote child survival is projected to save the lives of 8,700 children. The three-year campaign, launched in 2015, is still underway. Complete benefit and cost data are available for 2015 and Over those two years, the campaign saved an estimated 6,200 lives at a cost of $10.4 million an average of $1,700 per life saved. Of the cost to save a life, about 23 percent is borne by D.M.I. in running the program and 77 percent is borne by patients. Although data for 2017 is not yet available, $1,700 is a reasonable estimate of the cost per life saved achieved by the entire three-year intervention. The impact of the campaign in 2017 is modeled in the same way as 2015 and 2016 and the costs are likely similar to 2016 costs. If anything, excluding 2017 data from the analysis overestimates the cost per life saved of the three-year intervention. 19

20 Table 1. Impact and cost of national child survival campaign in Specification TARGET POPULATION Total 2.9 million UNDER-5 LIVES SAVED 8,700 COST PER LIFE SAVED (ALL STAKEHOLDERS) a $1,700 COST PER LIFE SAVED (D.M.I. S COST ONLY) a $400 Impact We measure the impact of D.M.I. s nationwide campaign in Burkina Faso as the number of lives saved of children under 5 years of age. To save children s lives, D.M.I. conducts radio campaigns designed to reach caregivers of young children especially pregnant women and mothers with health messages. The messages encourage behaviors that could lead to lower child mortality, such as attending regular antenatal visits, delivering babies in a health facility, and seeking treatment for childhood pneumonia, malaria and diarrhea. The nationwide campaign under investigation in this audit is a successor to a regional campaign D.M.I. ran in Burkina Faso between 2012 and In partnership with London School of Hygiene and Tropical Medicine (L.S.H.T.M.), D.M.I. ran a randomized controlled trial (R.C.T.) to measure the effectiveness of the regional campaign. Survey data collected during the R.C.T. did not show a statistically significant change in child mortality or most behaviors. However, data from medical facilities over the same time period showed a statistically significant increase in patient visits. To estimate the impact of the regional campaign and in consultation with L.S.H.T.M., D.M.I. used the facility data and a complex statistical model, the Lives Saved Tool (LiST), to predict reductions in child mortality that should result from more visits. D.M.I. also used data from the regional campaign to predict the impact of a hypothetical nationwide scale-up of the campaign occurring between 2017 and D.M.I. subsequently launched an effort to scale the program nationwide between 2015 and To estimate the impact of this scale-up, we use the estimates D.M.I. created for the hypothetical campaign, adjusting for differences in background levels of mortality. 20

21 D.M.I. S RANDOMIZED TRIAL In partnership with the L.S.H.T.M., D.M.I. conducted an R.C.T. to measure the impact of D.M.I. s regional radio campaign in Burkina Faso on child mortality. Study staff collected data from two sources: household surveys and records routinely produced by medical facilities. The household survey was conducted on clusters composed of villages within the broadcast range of a single radio station. Half of the clusters were randomly selected to receive the radio campaign; the other half constituted the control group. In both treatment and control clusters, a sample of households with women of reproductive age and households with at least one child under age 5 were interviewed before and after the campaign. With just 14 clusters available for randomization, the survey lacked the statistical power to detect large impacts from the media campaign. The survey analysis found no statistically significant effects on child mortality and 19 out of 20 indicators of parental behavior change. Over the same time frame, researchers collected data from health facilities across treatment and control clusters. Compared to surveys, far more data was available on facility visits, enabling researchers to detect statistically significant changes. Health facility data show an increase in the number of health consultations for children under age 5, antenatal check-ups for pregnant women, and births in health facilities in treatment clusters relative to control clusters. MODELING IMPACT WITH THE LIST TOOL In 2017, D.M.I. and L.S.H.T.M. predicted the change in child mortality from a hypothetical nationwide campaign in Burkina Faso using LiST, a complex statistical model that uses scientific studies of the effectiveness of various health interventions to predict the number of lives that will be saved from increased coverage of those interventions. The tool can be calibrated to produce more accurate results based on a country s demography, health status (measured by dozens of indicators such as the percent of children with various vitamin deficiencies), rates of death due to various causes and baseline coverage of health interventions. LiST is further described in quality of evidence. We use D.M.I. and L.S.H.T.M. s LiST modeling as the basis for our own benefit/cost analysis of the national campaign on child survival. We agree with D.M.I. that the health-facility data have greater statistical power to detect an effect than the household survey data. The healthy facility data includes records of tens of thousands of visits, covering 12 percent of the intervention and control populations, while the household survey data only includes approximately one thousand sick children in each of the intervention and control 21

22 arms. We also choose to use the results from D.M.I. and L.S.H.T.M. s LiST modeling because LiST is the best model of its kind for predicting child mortality based on the peerreviewed evidence behind specific public health interventions. D.M.I. and L.S.H.T.M. predicted the Burkina Faso nationwide campaign, which they assumed would run from 2017 to 19, would decrease mortality 5.6 percent in year one, 3.5 percent in year two and 3.7 percent in year three. 13 D.M.I. launched the national campaign two years earlier, in We use the treatment effects modeled by D.M.I. and L.S.H.T.M. for the hypothetical campaign to estimate the impact of the campaign, adjusting for background rates of mortality in the earlier period. IMPORTANT ASSUMPTIONS USED IN MODELING IMPACT D.M.I. and L.S.H.T.M. s LiST model relied on several assumptions. Below, we describe four important assumptions the modeling team made: 1. In the LiST model of a hypothetical nationwide campaign in , the baseline levels of coverage of interventions promoted by D.M.I. s campaign were set based on data from health facilities. 2. The coverage of all other health interventions not promoted by the media campaign was held at the same level during the campaign. 3. Estimates of the effectiveness of health interventions promoted by the media campaign were derived from scientific studies. Those scientific studies are the default inputs to the LiST model and D.M.I. and L.S.H.T.M. assumed the studies were the best sources of information on intervention effectiveness. 4. The number of visits to a health facility spurred on by campaign messages is defined as a set proportion, about 121 percent, of the number of caregivers that hear the campaign messages. Our estimate of impact borrows from D.M.I. and L.S.H.T.M. s LiST estimates and therefore relies on these assumptions. In addition, we assume the campaign continues to cause caregivers to seek medical care for their children, and therefore continues to save lives, even after it goes off the air. We assume this residual effect is small and lasts one year. Coverage Levels of Promoted Interventions To estimate change in child mortality, the LiST model requires an estimate of the baseline level of coverage in Burkina Faso of health interventions that affect child and maternal health. As data on coverage was not yet available for , the D.M.I. and L.S.H.T.M. modeling team set the increased coverage of interventions promoted by the media 22

23 campaign to the levels of increased consultations reported in administrative data from health facilities between 2012 and In our benefit/cost ratio, we assume coverage levels are the same in as they were in Coverage Levels of Other Interventions D.M.I. and L.S.H.T.M. assumed the coverage levels of all health interventions not promoted by the media campaign stayed the same during the campaign. This assumption allowed them to isolate the effects of the media campaign. However, it might have caused them to understate the impact of the campaign. Children whose lives are saved by one health intervention still face other mortality risks. As a result, saving lives by treating malaria (as an example) might lead to additional deaths from diarrhea and pneumonia. Data from W.H.O. show other diseases not addressed by D.M.I. are generally on the decline in Burkina Faso. 14 This decline in other diseases means additional children were alive during the campaign and could have been saved from illnesses targeted by D.M.I. s campaign. By not adjusting LiST for this declining mortality, the D.M.I. team may be slightly understating the impact of the campaign. Studies used to estimate effectiveness of interventions In the LiST model, the effectiveness of a health intervention is defined as the percent of deaths due to a specific cause that are reduced by the intervention. For example, the effectiveness of oral rehydration solution (O.R.S.) is the percent of deaths caused by diarrhea that can be reduced by O.R.S. The modeling team chose not to set their own parameters for effectiveness for all interventions except antimalarials for which a discounted effectiveness figure was used in order to be more conservative than the default LiST parameters. For the other interventions, they used the default LiST parameters for the effectiveness of the health behaviors promoted in the campaign. Those parameters are set by the LiST team based on their analysis of scientific reviews and studies. 13 This is a reasonable decision. The LiST team is expert in analyzing the effectiveness of health interventions and D.M.I. would likely not have improved the parameters substantially by replicating its work. Proportion of caregivers who act on radio messages The modeling team assumed that the proportion of caregivers of young children (the target population for radio and TV spots) who act on campaign messages is the same during the hypothetical nationwide scale-up as during the regional campaign 23

24 after adjusting for the fact that a smaller proportion of women across the country listen to the radio compared to women in the regional campaign zones. Residual effect after campaign ends We assume there is a small residual effect of the media campaign the year after broadcasts end. Routinely collected data from health facilities show that rates of health care usage remained higher in treatment clusters 10 months after D.M.I. s regional campaign broadcasts went off the air in January We assume messages broadcast during the active years of the campaign ( ) lead to a similar residual effect in All three years of the campaign contribute to this effect and so we credit one-third of the residual effect to each year. LIVES SAVED BY D.M.I. S NATIONWIDE CAMPAIGN The number of lives saved from D.M.I. s nationwide child survival campaign is the sum of lives that were saved during the three active years of the campaign and the lives that were saved in the year after broadcasts ceased as a result of lingering effects of the campaign. Because there is a time lag between campaign activities and resulting lives saved in 2018, we convert lives saved in 2018 to present value using a social discount rate of 5 percent. Calculations are in the accompanying spreadsheet. 24

25 Equation 1. Number of lives saved by the nationwide campaign Number of lives saved = A B + C D + E F + G H Symbol Variable Value A Lives saved in 2015 (percentage of counterfactual mortality) a 5.6% B Counterfactual mortality in 2015 (number of deaths) b 63,000 C Lives saved in 2016 (percentage of counterfactual mortality) a 3.5% D Counterfactual mortality in 2016 (number of deaths) b 63,000 E Lives saved in 2017 (percentage of counterfactual mortality) a 3.7% F Counterfactual mortality in 2017 (number of deaths) c 60,000 G Residual lives saved in 2018 (percentage of counterfactual mortality) d 1.4% H Counterfactual mortality in 2018 (number of deaths) c 63,000 a Modeled by D.M.I. and L.S.H.T.M. using LiST b Estimated by ImpactMatters using a linear model of known counterfactual mortality rates in and c Inferred from D.M.I. and L.S.H.T.M. s LiST modeling, which shows 3,358 lives saved in 2017, or 5.6 percent of counterfactual mortality d Estimated by ImpactMatters using a log-linear model of lives saved in years one through three of the campaign In total, the three-year national campaign will save approximately 8,700 lives. To calculate a benefit/cost ratio, we restrict our analysis to 2015 and 2016, the years for which we have financial data from D.M.I. To estimate impact for 2015 and 2016, we use the following equation: Equation 2. Number of lives saved by nationwide campaign activities in 2015 and 2016 Number of lives saved = A B (G H) + C D + 1 (G H)

26 Cost The campaign saved 6,200 lives in 2015 and 2016 at a cost of $10.4 million, or $1,700 per life saved. D.M.I. s cost to run the media campaign accounted for 23 percent of the total social cost to achieve the campaign s impact. Patients paid the remaining cost in fees, price of drugs and travel costs. D.M.I. costs were based on annual expense reports. The costs of patient care were divided between patients and the government of Burkina Faso. During the time period when the national campaign was running, patients paid for antenatal care visits and deliveries at a health facility, including consultation fees, the price of drugs and travel costs. The same was true for child-health visits until mid-2016, when the government of Burkina Faso eliminated consultation fees for child-health visits. From then on, patients paid only for drugs and transportation to the health facility. Even though patients no longer paid consultation fees, we assume that the government did not then have to spend additional money to make up for the shortfall in fees. A 2004 study in the Kossi province of Burkina Faso found that the governmental budget would hardly rise even with a more than twofold increase in utilization of health services because of the slack capacity of existing personnel and facilities. 16 Additional visits to health facilities would not raise the fixed costs of salaries and facilities, only the variable costs of drugs, which patients pay. We assume consultation fees for child-health visits can simply be removed from the cost side of the ledger after mid We acknowledge that this assumption is a strong one and is based on data more than 10 years old. However, we have found other sources that substantiate it, at least in part. Our interview with a member of D.M.I. s advisory board, a public health specialist and former senior advisor to the Burkina Faso Minister of Health, broadly corroborated that the government believes the health system had sufficient capacity to meet the additional demand generated by D.M.I. In addition, the W.H.O. s Service Availability and Readiness Assessment (SARA) of Burkina Faso in 2014 suggests considerable slack capacity in the health system to absorb a surge in demand. A key detail: The slack capacity is not due to stellar availability of medical services; in fact, Burkina Faso lags far behind the W.H.O. s recommended number of health professionals and hospital beds per 10,000 people. 17 Rather, we think the excess capacity is due to extremely low uptake of health services by the population. The number of outpatient consultations registered across the country in 2014 was 15 million, according to the W.H.O. s SARA report. 17 Meanwhile, our back-of-the-envelope 26

27 calculation indicates the health system actually had the capacity to provide about 77 million consultations that year. Our calculation is based on the SARA report, the 2004 study in the Kossi province, and research literature on the duration of consultations. 18 If our estimate is correct, the health system should have had the capacity to absorb 62 million more consultations at negligible cost to the government, and should easily have absorbed the fewer than 4 million patient visits spurred by D.M.I. s campaign. Our analysis assumes bribes or side payments are not required to obtain medical care in Burkina Faso. If this assumption is wrong, our calculation of costs for the public sector will likely be too low. To calculate patients costs, we use administrative data from health facilities in the same area D.M.I. conducted its regional media campaign. The data show a sizable increase in treatments over the intervention period. We extrapolate these findings to the population reached by the nationwide campaign in order to estimate that an additional 2.8 million people sought treatment due to the campaign activities in 2015 and Those patients paid $7.9 million for that care. Figure 2. Costs of national child survival campaign, shares by payer D.M.I. pays radio stations for the air-time it uses, but often negotiates for prices under the going rate. However, D.M.I. often compensates radio stations by providing solar panels and staff training. We assume that the combination of air-time fees and in-kind donations fully compensates radio stations for their services. The value of this compensation is captured in D.M.I. s expenses and therefore accounted for in the benefit/cost analysis. 27

28 Sensitivity Analysis To estimate impact, we rely on assumptions that are made using the best available evidence. We analyze how sensitive our findings are to these assumptions by varying the value of the assumption and looking at the change in the impact and cost estimate. For the benefit/cost analysis of D.M.I. s nationwide campaign, we test two assumptions: Discount rate. In our impact and cost estimate, we assume that future benefits and costs are discounted to present value at a rate of 5 percent a year. We test how that estimate changes to discount rates ranging from 0%, very long-term minded, to 50%, very short-term minded. Residual effects. In our impact and cost estimate, we assume that there are residual effects in the year after the campaign concludes. We re-calculate the estimate assuming there are no residual effects. SENSITIVITY TO DISCOUNT RATE D.M.I. will broadcast messages for three years during its nationwide campaign. If messages stick with listeners even after broadcasts cease, listeners will continue seeking more health care than they would have otherwise sought for some time after the conclusion of the campaign. Patients seeking treatment in the future adds to both impact (lives saved) and costs (patients pay to access health care). In our impact and cost estimate, we convert these future impacts and costs to present value using a discount rate of 5 percent and estimate the cost per life saved is $1,700. To test sensitivity of our estimate to discount rate, we re-calculate the estimate using discount rates that range from 0 percent to 50 percent. The cost per life saved remains very stable across this wide range of discount rates, suggesting our choice of discount rate is not an important factor driving our conclusions. Table 2. Sensitivity to discount rate Discount Rate 0% 5% a 10% 15% 50% COST OF IMPACT (ALL STAKEHOLDERS) COST OF IMPACT (D.M.I. S COST ONLY) $1,700 $1,700 $1,700 $1,700 $1,700 $400 $400 $400 $400 $

29 IMPACT (LIVES SAVED) 6,300 6,200 6,200 6,100 5,900 NONPROFIT COST $2.4M $2.4M $2.4M $2.4M $2.4M PARTNER COST BENEFICIARY COST $8M $7.9M $7.9M $7.9M $7.8M TOTAL COST $10.4M $10.4M $10.4M $10.3M $10.2M a We use a 5% discount rate in our impact and cost estimate. SENSITIVITY TO INCLUSION OF RESIDUAL EFFECTS In our impact and cost estimate, we assume that after the nationwide campaign ends there is some residual effect, leading more people to seek treatment for up to a year afterwards. However, if listeners new knowledge, attitudes and practices don t stick after radio messages cease, there will be no further lives saved from the campaign in 2018 and beyond. This scenario is mathematically synonymous with an extremely high discount rate, where all future impacts and costs are worth zero in the present day. Excluding residual effects results in an estimate of 5,700 lives saved instead of 6,200, at a cost of $10.1 million instead of $10.4 million. If there are no residual effects, the cost per life saved is $1,800, compared to $1,700 in our base case. Table 3. Sensitivity of inclusion of residual effects Inclusion of Residual Effects No Residual Effects Residual Effects a COST OF IMPACT (ALL STAKEHOLDERS) COST OF IMPACT (D.M.I. S COST ONLY) $1,800 $1,700 $400 $400 IMPACT (LIVES SAVED) 5,700 6,200 NONPROFIT COST $2.4M $2.4M PARTNER COST - - BENEFICIARY COST $7.7M $7.9M TOTAL COST $10.1M $10.4M a We include residual effects in our impact and cost estimate. 29

30 Displacement and Other Effects This section discusses other effects of the nationwide campaign both positive and negative that are not captured in the impact and cost estimate of the program. MATERNAL LIVES SAVED EFFECT: POSITIVE CONSEQUENCE: HIGH Behaviors promoted by D.M.I. s nationwide campaign (primarily more antenatal visits and deliveries at a health facility) likely reduce maternal mortality. D.M.I. s R.C.T. found that its campaign saved the lives of 38 mothers. D.M.I. predicts that a nationwide campaign would save 250 maternal lives. 13 IMPROVED CHILD HEALTH EFFECT: POSITIVE IMPORTANCE: HIGH Measuring impact as the number of fatally ill children whose lives are saved effectively ignores all but the most severe cases. The behaviors promoted by D.M.I. s campaign also improve the health of children who are not at risk of death. Children may suffer less severe or fewer symptoms of pneumonia, malaria and diarrhea for a shorter period of time. They may be saved from long-term developmental impairment and disability. Only counting the number of child lives saved therefore understates the impact of the program. DISPLACEMENT OF OTHER ADVERTISERS EFFECT: AMBIGUOUS IMPORTANCE: LOW D.M.I. directly competes with other purchasers of limited radio and TV air-time and therefore likely takes ad time from other advertisers. This may be a concern because D.M.I. sometimes secures spots at below the market rate (though D.M.I. does compensate radio stations in other ways that may make up for this shortfall). However, there is no reason to think D.M.I. spots are displacing other spots with high social value, lessening concern about pushing out other advertisers. 30

31 OVERBURDENED HEALTH CLINICS EFFECT: NEGATIVE IMPORTANCE: LOW D.M.I. s campaigns teach listeners to recognize their symptoms and seek the appropriate medical care. But its campaigns will be vastly less effective if the health system cannot handle an increase in demand for services. D.M.I. therefore carries out desk or field research prior to launching a campaign to ensure that the supply of health services and commodities in a country or region is sufficient to absorb an increase in demand brought on by a campaign. A 2004 study in the Kossi province of Burkina Faso 16 and W.H.O. data on the capacity and utilization of health services in the country 17 suggest the existing health infrastructure ought to be able to accommodate a large increase in patient visits. While we cannot draw a conclusion, we cautiously assume, based on those sources, that the 35 percent increase in child health consultations spurred by D.M.I. s campaign would not strain the health system. 31

32 QUALITY OF EVIDENCE WHY WE RATE Quality of evidence captures confidence in the impact and cost estimates. For programs with high-quality evidence, the impact and cost estimates are more likely to accurately reflect the effectiveness of the program. HOW WE RATE Quality of evidence is rated using an adaptation of the GRADE methodology, a systematic approach to judging evidence. Initially, studies are ranked by whether they are observational, quasi-experimental or experimental. Then, each study is assessed against quality criteria: risk of bias, inconsistency of results, indirectness of evidence, imprecision, risk of publication bias, magnitude of effect, evidence of a dose-response relationship and attenuation bias. In the ideal case, data from the program are solely used to estimate the impact of the program. However, external data can be used for additional parameters or to link behavior change to outcomes. When the analysis is substantively based on data from multiple sources, the quality of each is assessed. If only very-low-quality internal data is available, high-quality external data may be substituted. In addition, external evidence can serve to confirm or contradict internal evidence. Star Rating Quality of Evidence Quality rating is Very Low ; or Quality rating is Low but high-quality external evidence contradicts its findings Quality rating is Low ; or Quality rating is Medium but high-quality external evidence contradicts its findings; or Quality rating is Very Low but high-quality external evidence corroborates its findings 32

33 Quality rating is Medium ; or Quality rating is High but high-quality external evidence contradicts its findings; or Quality rating is Low but high-quality external evidence corroborates its findings Quality rating is High ; or Quality rating is Medium but high-quality external evidence corroborates its findings Rating The impact and cost model for D.M.I. s nationwide campaign in Burkina Faso uses internal evidence that is of high quality. The impact and cost model for the campaign relies on two sources of evidence: a randomized controlled trial (R.C.T.) of D.M.I. s regional campaign in Burkina Faso and the Lives Saved Tool (LiST), a statistical model for estimating the effectiveness of health interventions. D.M.I. s randomized trial was conducted well. The study surveyed households and did not find an impact on child mortality or 19 out of 20 health behaviors. However, routine health facility data, with greater statistical power, collected over the same time period showed marked improvement in child health consultations. Clinical evidence from elsewhere links child health consultations to changes in child mortality. To model the impact of health consultations on child mortality, D.M.I. uses LiST, the most respected method for estimating how many lives can be saved by public health interventions. In a simulation of the Burkina Faso regional radio campaign, changes in health behaviors caused a decline in mortality of 4-6 percent. 15 Evidence on other similar mass media campaigns is of low quality, although some studies have suggested a strong impact. Three systematic reviews published in the past decade examine the impact of mass media on child survival. These reviews find some links between media campaigns and behavior change, but no strong statistical evidence that those campaigns cause a decline in mortality. Most of the media campaigns studied had a different methodology than D.M.I. s campaigns, which seek to fully saturate listeners with many repeated messages. As a result, research on these programs are a poor guide to D.M.I. s impact. 33