2015-2016 IMPACT AUDIT Development Media International (D.M.I.) Child Survival Program Findings and Executive Summary ImpactMatters Published November 27, 2017 www.impactm.org/a/dmi ImpactMatters
FINDINGS Child Survival Program MISSION PROBLEM To save the lives of children and newborns in countries with high child mortality rates. Families lack awareness of healthy behaviors. INTERVENTION D.M.I. designs, develops and runs media campaigns that provide repeated exposure to information on how to improve child health. ENGAGEMENT Appropriate careseeking, antenatal care, health-center births IMPACT Reduction in mortality among children under 5 IMPACT AND COST Approximately $1,700 per child life saved IMPACT AND COST CALCULATION We estimated the impact and cost of a nationwide scaleup of D.M.I. s child survival program in Burkina Faso. To do so, we use data from a randomized controlled trial D.M.I. conducted on a regional media campaign in Burkina Faso from 2012-14. The trial collected survey data and health facility data. The survey data did not find statistically significant declines in child mortality. However, the health facility data found substantial increases in health visits in regions where D.M.I. broadcast the campaign. As the survey data were ambiguous, D.M.I. modeled changes in child mortality using health facility data and a statistical model called the Lives Saved Tool. We use an adjusted version of D.M.I. s methodology to calculate campaign impact. We then compare impact to D.M.I. s and patients total costs to calculate a cost/benefit ratio of $1,700 per child life saved. Considering only D.M.I. s direct costs, the cost/benefit ratio falls to $400 per child life saved. QUALITY OF EVIDENCE QUALITY OF EVIDENCE ASSESSMENT The impact and cost model uses two sources of high-quality evidence: a well-conducted randomized controlled trial of D.M.I. s regional campaign and the Lives Saved Tool, an evidence-based statistical model for estimating the effectiveness of health interventions.
FINDINGS Organizational Effectiveness GEOGRAPHY STAGE AGE SIZE Burkina Faso, Democratic Republic of the Congo, India and Kenya Scale Current model in operation for 6 years Approximately 2.9 million women of reproductive age reached in a national campaign in Burkina Faso QUALITY OF MONITORING SYSTEMS D.M.I. excels at using data to make informed decisions about campaign design and implementation and transparently describes the methods it uses to track outcomes. We find two shortfalls: (1) D.M.I. lacks a clear methodology for complementing survey data with administrative data from health facilities; and (2) D.M.I. has begun collecting timeseries data without detailed research objectives and an analysis plan. Criteria Credible Actionable Responsible Transportable ACTIVITY DATA TARGETING DATA ENGAGEMENT DATA FEEDBACK DATA OUTCOMES DATA LEARNING AND ITERATION D.M.I. considered four changes to the design of its program over the last three years. Three of these changes were appropriately tested and systematically implemented. The fourth change a contract for D.M.I. to produce medical training videos was directed by a partner and did not follow an appropriate process for iterative learning. Criteria ITERATION IS BASED ON DATA DATA ARE OF HIGH QUALITY ITERATION IS SYSTEMATIC AND PERIODIC Finding Yes Yes Yes
ImpactMatters 2017 1460 Broadway New York, New York 10036 www.impactm.org info@impactm.org Some rights reserved. Rights and Permissions This work is available under the Creative Commons Attribution- ShareAlike 4.0 International license: https://creativecommons.org/licenses/by-sa/4.0/legalcode. Under the Creative Commons Attribution-ShareAlike license, you are free to copy and redistribute the material in any medium or format and adapt this work, for any purpose, including commercial purposes, under the following conditions: Attribution: Please cite the work as follows: ImpactMatters. Impact Audit of Development Media International. 2017. License: Creative Commons Attribution-ShareAlike CC BY SA 4.0. ShareAlike: If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original. Feedback We welcome your feedback at www.impactm.org/feedback. www.impactm.org 4
EXECUTIVE SUMMARY Program Description and Top-Level Findings Development Media International s (D.M.I.) child survival program reduces newborn and early-childhood deaths by running media campaigns that encourage behaviors that could save the lives of children. D.M.I. runs these campaigns in countries with high rates of child mortality and measures success by the number of child lives saved. This audit analyzes the impact of a radio campaign D.M.I. ran nationwide in Burkina Faso between 2015 and 2017. During D.M.I. s 2015-2017 nationwide campaign, it cost an average of $1,700 to save the life of a child. This figure includes costs borne by D.M.I. and patients. Patients pay for about three quarters of the cost per life saved, mostly through drug purchases. The campaign in Burkina Faso saved 6,200 lives in 2015 and 2016. That translates to a reduction in child mortality of between 4 and 6 percent of the mortality rate. The national campaign began in 2015 and will end in 2017, but our benefit-cost analysis is restricted to 2015-16 because cost data are not yet available for 2017. By the time the three-year campaign ends, we predict it will save about 8,700 lives. For the campaign, the evidence of impact is strong. D.M.I. and the London School of Hygiene and Tropical Medicine conducted a randomized controlled trial (R.C.T.) to directly measure the campaign s impact on child mortality in Burkina Faso in 2012-14. We base our estimate of the impact of the 2015-17 campaign on a statistical model fit by D.M.I. (described below). Impact and cost D.M.I. s nationwide campaign saved child lives at an average cost of $1,700 per life saved. The evidence for the program s impact comes from a randomized controlled trial conducted between 2012 and 2014. The investigators defined regions of Burkina Faso where they could isolate exposure to radio campaigns and randomly assigned those www.impactm.org 5
regions to treatment and control groups. D.M.I. broadcast a radio campaign in the treatment regions for three years. To estimate changes in child mortality and parents behavior, the study gathered data on child health outcomes from two sources: household surveys and records of patient visits to health clinics. The survey data on child mortality were inconclusive. Although child mortality declined in the treatment zone, it also declined in the control zone. The trial s survey data on child mortality did not prove that the media campaign caused the observed decline in mortality. i Confronted with the failure of the survey data to show impacts child mortality, the investigators instead used administrative data from health facilities to demonstrate the impact of the media campaign. During the R.C.T., caregivers and pregnant women in the treatment zone increased their use of medical care faster than those in the control zone. Pregnant women attended routine antenatal visits and chose to deliver babies at health facilities. Parents brought children who were sick with pneumonia, malaria and diarrhea to health facilities. Those changes in parents behavior led to life-saving medical treatments for their children. D.M.I. used the Lives Saved Tool (LiST), a widely accepted statistical model, to estimate the decline in child mortality that resulted from the observed increase in care-seeking. D.M.I. calculated a decline of 4 to 6 percent in child deaths attributable to its radio campaign. We estimate that the national campaign during 2015-16 saved 6,200 lives. Although the radio campaign continued to broadcast in 2017, we exclude these impacts from our analysis because our cost data only cover 2015-16. This impact was achieved at a cost of $10.4 million. Of that total cost, 23 percent was incurred by D.M.I. in executing the campaigns. Patients paid the remaining 77 percent in health care expenses. Although the radio campaign in Burkina Faso increased the number of patients who visited health clinics, the best econometric study available concludes that the increase in visits would not lead to higher public health expenditures. i The statistical power of the trial was limited by the nature of the media campaign. Media campaigns are broadcast over a large area, which means that investigators can choose fewer treatment and control sites. Due to the way that clustering affects statistical power, having fewer treatment and control sites in the design reduces overall statistical power to detect an intervention effect. www.impactm.org 6
Quality of Evidence The quality of evidence for the impact of D.M.I. s child survival program is high. Mass media campaigns have been used for decades to promote child survival. Most studies of their impact have been weak, relying on respondents to report whether or not they heard the broadcasts. The D.M.I. study in Burkina Faso is exceptionally well-designed among such studies, in that it randomized exposure to the radio broadcasts. Data collected by surveying a sample of households in could not demonstrate the impact of the broadcasts on lives saved. However, data collected from health facilities during the trial show that the campaign led parents to seek medical care more often. In the treatment arm of the study, visits to health clinics for sick children rose by anywhere from 16 to 35 percent more than the control zone. Clinics also reported that uptake of antenatal care and births at facilities increased by 6 to 9 percent. To show how many lives could have been saved by parents decisions to seek medical treatment for sick children, D.M.I. used LiST statistical model to estimate how many lives can be saved by particular health interventions. The LiST model relies on multiple peerreviewed studies to show how mortality responds to all the major risks of child mortality. Estimates from the LiST tool are calibrated to local health risks, such as the prevalence of certain diseases and the coverage of meaning how much of the population has access to care such as vaccinations. D.M.I. s R.C.T. showed that parents who heard the radio campaign brought more children into health facilities for treatment. The LiST model estimated how many lives could be saved by treatments for pneumonia, diarrhea and malaria provided at those visits. Quality of Monitoring Systems D.M.I. has outstanding monitoring systems in the many countries where it runs mass media campaigns. Every media campaign begins with formative research on the media and health landscapes of the country where it will be broadcast. Staff research what kinds of media are widely consumed in the country. Staff also identify major health risks that the media campaign can address and the barriers to populations seeking and receiving care for those health risks. D.M.I. distills the research into a message brief that articulates target behaviors for parents and how specific obstacles to those behaviors should be addressed. www.impactm.org 7
D.M.I. takes steps to ensure campaign broadcasts run smoothly. Radio spots are pretested with focus groups before broadcast. Broadcasting is monitored by independent listeners, tracking software planted at radio stations and self-reporting by radio stations. Once a campaign goes on air, D.M.I. collects feedback from listeners to identify any needed changes to the design of the campaign. Script writers incorporate feedback from listeners into their new scripts. D.M.I. has begun to gather even more data in Burkina Faso in the hopes of using time series analysis to show that media broadcasts are responsible for behavior change. D.M.I. has yet to set in place a clear plan for analyzing this data, potentially leading to wasted resources. D.M.I. should develop detailed research objectives and an analysis plan for the time series analysis prior to data collection. Learning and Iteration D.M.I. s process for considering changes to its interventions is excellent. Program modifications are subjected to high-quality pilot tests before they are expanded. Senior management meet regularly to consider which concepts for program modifications have sufficient evidence to proceed with pilot tests. A unit called D.M.I. Labs is responsible for managing a portfolio of ideas under development. D.M.I. Labs tests ideas in stages, beginning with small proof-of-concept studies before proceeding with larger trials. D.M.I. has tested new ways to reach its audiences. A pilot test in Kinshasa showed that social media can be effectively combined with radio and television broadcasts. Another pilot test in Burkina Faso showed that videos exchanged by phone memory cards rather than by broadcast are effective in reaching people that do not listen to radio or watch television. In the past three years, D.M.I. only undertook one change to its model without adequate testing, a project to produce a series of short films to train health workers for the University of the West Indies. The impact of the videos was insufficiently researched and the pilot was not evaluated. This project was a one-off and not representative of the typical high level of care D.M.I. takes in planning campaign changes. www.impactm.org 8
NONPROFIT COMMENT DMI delivers mass media campaigns to improve health and save lives in low-income countries. We ran the first randomised controlled trial to show that radio can change lifesaving behaviours. We have two priorities: to continue to generate ground-breaking research, while taking proven strategies to the largest possible scale, saving as many lives as possible. DMI thanks ImpactMatters for the rigour and thoroughness of this impact audit. We feel that the results of this audit reflect the emphasis that DMI places on robustly evaluating the impact of our work. We would also like to thank the London School of Hygiene and Tropical Medicine, our evaluation partner for the randomised controlled trial that is referenced in this report as well as our broadcast partners in Burkina Faso without whom we would not have been able to save these lives. We recognise and appreciate the efforts ImpactMatters have made to include the full costs of saving lives, including costs to patients and governments. It s worth emphasising that of the $1700 required to save a life, just $400 is DMI s cost. The remainder, as ImpactMatters make clear, is largely the cost of drugs and is born by the patients. Readers should note that the cost per cycle of anti-malarial or antibiotic treatment is in the $1-2 range, which is affordable for most households. We are grateful to ImpactMatters for highlighting that our impact estimates do not include our impact on morbidity and therefore understate the impact of the program on improved child health. Inspired by this report, we intend to carry out some research that will allow us to estimate our impact on morbidity as well as mortality. In response to particular critiques: Methodology for complementing survey data with administrative data from health facilities DMI s methodology for interpreting facility and survey data (and any differences between the two) is outlined in the RCT results paper (publication pending). www.impactm.org 9
In general we believe that facility and survey data measure two different things. Facility data allows us to collect statistically robust data on behaviours. We are not able to collect large enough samples during a single survey to conclusively measure responses to malaria, diarrhoea and pneumonia, for example, whereas facility data, with tens or even hundreds of thousands of cases, is far more powerful. Survey data is much more useful for measuring other indicators, such as media exposure, knowledge and intentions. The results will be stronger, of course, if knowledge, intentions and actual behaviours all point in the same direction. Time Series Data We actually think that the time-series approach is a more cost-effective way of collecting data than bespoke surveys, which require multiple (often underpowered) baseline and endline surveys for each and every campaign that DMI needs to evaluate. The time series platform was not designed to test a single, specific hypothesis but rather provides continuous data that DMI uses to evaluate and adapt the design of multiple campaigns. For any specific campaign (such as our scale-up funded by Comic Relief) we detail objectives beforehand. In the case of Comic Relief, this was detailed in the research protocol that received ethical approval from the Ministry of Health s Institutional Review Board. The sample sizes collected are calculated to be appropriate for measuring the multiple indicators in which we are interested. It should be noted that the annual cost of a time-series survey is just $75k thanks to a highly efficient staff of 7 data collectors armed with a motorbike and smart phone (whereas a single survey with a sample size of 5000 in this region can cost over $100k). As described in this report, DMI is committed to transparency, measuring the impact of our work, and continuous improvement. We welcome any questions or comments that readers may have and will continue to share the findings from our research in order to achieve greater collective impact. www.impactm.org 10
ImpactMatters, a 501(c)3, conducts impact audits of nonprofits to rigorously estimate their philanthropic impact, compelling them and their funders to make evidence-based decisions. Learn more at www.impactm.org. ImpactMatters