Finding True Program Impacts Through Randomization

Transcription

1 David Evans (World Bank) Finding True Program Impacts Through Randomization Impact Opportunities: Evidence and Action to Save Lives in Nigeria Uyo, Nigeria, May 7-10, 2013

2 Objective Evaluate the causal impact of a program or an intervention on some outcome Examples How much did free distribution of bednets decrease malaria incidence? Which of two supply chain models was most effective at eliminating drug shortages? 2

3 Counterfactual Criteria Treated & comparison groups Have identical average characteristics (observed & unobserved) The only difference is the treatment Therefore the only reason for any difference in outcomes is the treatment Key question: What would participant look like if she hadn t received the program? 3

4 Perfect Experiment 1. Identify target beneficiaries 2. Clone them! Identical on the outside (observable) We re both fiveyear-old puppets Kami We both love to take up new health interventions! Tami Identical on the inside (unobservable) Muppet photos copyright Sesame Workshop 4

5 Perfect Experiment Give the intervention to one set of clones Kami Tami 5

6 Perfect Experiment Observe some time later Kami Tami Because the groups are identical (inside & out), the difference is due to the bednets! 6

7 Back to Reality What would Tami look like if she didn t receive the bednet???? Room For Improvement Control Groups Before After Participants Non Participants 7

8 RFI: Before-After BEFORE BEDNETS 6 malaria episodes in 6 months AFTER BEDNETS 2 malaria episodes in 6 months What else might be going on besides the bednets? Seasonal differences Rising incomes: Households invest in other measures Too many other factors! Impact of bednets =???

9 RFI: Participants vs Non-Participants Compare recipients of a program to People who were not eligible for the program People who chose not to enroll in the program Example: Complications in childbirth Impact of clinic births? Home births Clinic births What else might explain the difference? 9

10 RFI: Participants vs Non-Participants Kami Observable differences Income Education Grover Unobservable differences Heard rumor about hospitals Neighbor available to care for other children Muppet photos copyright Sesame Workshop 10

11 RFI: Participants vs Non-Participants How much of difference is because of clinic? Example: Complications in childbirth Other factors! Impact of clinic births Home births Clinic births Impact of clinic births =??? 11

12 Selection bias People who choose to join the program are different! If we cannot account completely for those differences in our data We never can How do you capture attitudes toward health systems? Initiative? then our comparison will not show the true impact of the program 12

13 What should we do? Gold standard: Randomized experimental design

14 Randomized Experimental Design Randomly assign potential beneficiaries to be in the treatment or comparison group Treatment and comparison have the same characteristics (observed and unobserved), on average, so Any difference in outcomes is due to treatment 14

15 Why Randomization Works Randomization with two doesn t work! Treatment Comparison But differences average out in a big sample Treatment Comparison On average, same number of Kamis and Grovers Observable AND unobservable Result: Measure true impact of program 15

16 Can we randomize? Randomization does not mean denying people the benefits of the project Usually existing constraints in project roll-out allow randomization Randomization often the fairest way to allocate treatment 16

17 Use Staggered Roll-out Roll-out to 200 clinics Roll-out to 400 more clinics Roll-out to 200 more clinics Jan 2014 July 2014 Jan 2015 Randomize the order in which clinics receive program Compare Jan 2014 group to Jan 2015 group at end of first year Example: Mexico parent training staggered roll-out among vulnerable communities 17

18 Vary treatment INTENSITY OF TREATMENT Malaria Information Campaign 100 villages Malaria Information Campaign + SMS Reminders 100 villages NATURE OF TREATMENT Radio campaign 100 villages Newspaper campaign 100 villages 18

19 What if randomization is impossible? Think again: It often is possible on some level, and it s the best way to get a clear measure of impact Always begin the IE with imagining what the ideal would look like With a national policy Use randomization to test implementation 19

20 Key takeaway #1 The single best way to evaluate the true average impact of an activity is by randomizing treatment. 20

21 Key takeaway #2 Randomization is more flexible than you think: It does not require withholding of benefits. It can take advantage of necessary staggered roll-out. It can test different reforms or packages of services across groups at the same time (so all receive at least some package). 21

22 Key takeaway #3 It is more ethical to test programs rigorously before universally implementing them than it is to use scarce public resources to implement a universal program with uncertain benefits. 22

23 Thank you! 23

24 BONUS SLIDES

25 25