What can go wrong.and how to fix it! Megha Pradhan Policy and Training Manager, J-PAL South Asia Kathmandu, Nepal 29 March 2017
Introduction Conception phase is important and allows to design an evaluation enabling to answer the research questions But the implementation phase of the evaluation is also extremely important: many things can go wrong J-PAL POST-DESIGN CHALLENGES 2
Objectives To be able to identify the main threats to validity during the implementation phase of the evaluation To define strategies to prevent each of these threats To know some of the methods that can be used during analysis phase J-PAL POST-DESIGN CHALLENGES 3
Lecture Overview Attrition Unexpected Spillovers Partial Compliance Intention to Treat (ITT) & Treatment on Treated (ToT) J-PAL POST-DESIGN CHALLENGES 4
Lecture Overview Attrition Spillovers Partial Compliance Intention to Treat (ITT) & Treatment on Treated (ToT) J-PAL POST-DESIGN CHALLENGES 5
Attrition Attrition occurs when not all of the individuals we intend to survey for the evaluation are available to be surveyed. Why is there attrition? People migrate and can t be tracked People refuse to get surveyed Businesses shut down Children don t come to school Why is attrition a problem?
Attrition Problem Attrition is a threat when it significantly reduces the sample size and power to detect significant impacts. Treatment Control No longer detect smallest effect size = motivated, entrepreneurial type
Attrition Problem Attrition is a threat when the treatment and control groups are no longer comparable. Treatment Control Started / expanded businesses; too busy to be interviewed Understating impact? = motivated, entrepreneurial type
Attrition Problem: An Example Problem: Some children are malnourished and sick because they don t get enough food Intervention: You start a school feeding program where free lunch is provided to children Evaluation: You want to measure children s weight as the outcome. You assign schools to randomized treatment and a control groups, and you weigh everyone who is in school on a given day Impacts: 1. Weight of children: Children who receive school meals gain more weight 2. Increased enrollment: Weak, stunted children start going to school more if they are enrolled in a treatment school
Before Treatment After Treament T C T C 20 20 22 20 25 25 27 25 30 30 32 30 Ave. Difference Difference J-PAL THREATS AND ANALYSIS 10
Before Treatment After Treament T C T C 20 20 22 20 25 25 27 25 30 30 32 30 Ave. 25 25 27 25 Difference 0 Difference 2 J-PAL THREATS AND ANALYSIS 11
What What if if only children children < 21 > 21 Kg Kg are come less to likely school? to come to school? J-PAL THREATS AND ANALYSIS 12
What if children < 21 Kg are less likely to come to school? Before Treatment A. Will you underestimate the impact? B. Will you overestimate the impact? C. Neither After Treament T C T C 20 20 22 20 25 25 27 25 30 30 32 30 D. Ambiguous E. Don t know 0% 0% 0% 0% 0% A. B. C. D. E. J-PAL THREATS AND ANALYSIS 13
What if children < 21 Kg are less likely to What if only children > 21 Kg come to school? come to school? Before Treatment After Treament T C T C [absent] [absent] 22 [absent] 25 25 27 25 30 30 32 30 Ave. 27.5 27.5 27 27.5 Difference 0 Difference -0.5 J-PAL THREATS AND ANALYSIS 14
Limiting Attrition: Partial Solutions Devote resources to tracking participants after they leave the program If there is still attrition, do some checks: Does it seem as if the missing people were different to begin with than those who remained in the study? Try to bound the extent of the bias Suppose everyone who dropped out from the treatment got the lowest score that anyone got Suppose everyone who dropped out of control got the highest score that anyone got.
Lecture Overview Attrition Spillovers Partial Compliance Intention to Treat (ITT) & Treatment on Treated (ToT) J-PAL POST-DESIGN CHALLENGES 16
Spillover Problem Spillovers are a problem when the outcomes of the control group reflect indirect program effects and not what would have happened in the absence of the program (i.e., the counterfactual). Evaluation Sample Random Assignment Treatment Group Control Group If positive If negative understate impact overstate impact
Spillovers: Solutions Two strategies seen during design phase: avoid them or measure them 1. Design the level of randomization so that it encompasses the spillovers If we expect spillovers that are all within school, randomization at the level of the school allows for estimation of the overall effect 2. It can be good to measure spillovers as they can have important implications for policy design Compare spillover group to pure control to measure spillover effect
Lecture Overview Attrition Unexpected Spillovers Partial Compliance Intention to Treat (ITT) & Treatment on Treated (ToT) J-PAL POST-DESIGN CHALLENGES 19
Partial Compliance Individuals assigned to comparison group could move into treatment group Alternatively, individuals allocated to treatment group may not receive treatment Can be due to project implementers or to participants themselves J-PAL POST-DESIGN CHALLENGES 20
Non compliers Target Population Not in evaluation What can you do? Can you switch them? Evaluation Sample Random Assignment Treatment group Control group Participants No-Shows Non- Participants Cross-overs J-PAL POST-DESIGN CHALLENGES 21
Non compliers Target Population Not in evaluation What can you do? Can you drop them? Evaluation Sample Random Assignment Treatment group Control group Participants No-Shows Non- Participants Cross-overs J-PAL POST-DESIGN CHALLENGES 22
Sample Selection Bias Sample selection bias could arise if factors other than random assignment influence program allocation Even if intended allocation of program was random, the actual allocation may not be J-PAL POST-DESIGN CHALLENGES 23
Non compliers Not in evaluation Target Population You can compare the original groups Evaluation Sample Random Assignment Treatment group Control group Participants No-Shows Non- Participants Cross-overs J-PAL POST-DESIGN CHALLENGES 24
What can be done? Ideally: prevent it during design or implementation phase => cannot always be done Monitor it during implementation phase => important to be aware that it happens Interpret it during analysis phase J-PAL POST-DESIGN CHALLENGES 25
Lecture Overview Attrition Unexpected Spillovers Partial Compliance Intention to Treat (ITT) & Treatment on Treated (ToT) J-PAL POST-DESIGN CHALLENGES 26
A school feeding program Let s take the example of a school feeding program Some schools receive the program, some don t (random allocation) But allocation is imperfectly respected J-PAL POST-DESIGN CHALLENGES 27
Compliance is imperfect School 1 Intention to treat? Treated? Pupil 1 Yes Yes Pupil 2 Yes Yes Pupil 3 Yes Yes Pupil 4 Yes No Pupil 5 Yes Yes Pupil 6 Yes No Pupil 7 Yes No Pupil 8 Yes Yes Pupil 9 Yes Yes Pupil 10 Yes No School 2 Intention to Treat? Treated? Pupil 1 No No Pupil 2 No No Pupil 3 No Yes Pupil 4 No No Pupil 5 No No Pupil 6 No Yes Pupil 7 No No Pupil 8 No No Pupil 9 No No Pupil 10 No No J-PAL POST-DESIGN CHALLENGES 28
Compliance is imperfect School 1 Intention to treat? Treated? Observed Change in weight Pupil 1 Yes Yes 4 Pupil 2 Yes Yes 4 Pupil 3 Yes Yes 4 Pupil 4 Yes No 0 Pupil 5 Yes Yes 4 Pupil 6 Yes No 2 Pupil 7 Yes No 0 Pupil 8 Yes Yes 6 Pupil 9 Yes Yes 6 Pupil 10 Yes No 0 Avg. Change among Treated A = 4.67 School 1: Average of treated in School 1 School 2: Average of treated in School 2 (A) 4.67 (B) 0.5 A-B 4.17 School 2 Pupil 1 No No 2 Pupil 2 No No 1 Pupil 3 No Yes 3 Pupil 4 No No 0 Pupil 5 No No 0 Pupil 6 No Yes 3 Pupil 7 No No 0 Pupil 8 No No 0 Pupil 9 No No 1 Pupil 10 No No 0 Avg. Change among Not-Treated B = 0.5
Intention to Treat (ITT) Easiest way to deal with partial compliance: Calculate the Intent to Treat (ITT): The difference between the average outcome of the group that was randomly assigned to treatment and the group that was randomly assigned to control, regardless of whether they actually received the treatment. What does intention to treat measure? What happened to the average child who is in a treated school in this population?
Intention To Treat School 1 Intention to treat? Treated? Observed Change in weight Pupil 1 Yes Yes 4 Pupil 2 Yes Yes 4 Pupil 3 Yes Yes 4 Pupil 4 Yes No 0 Pupil 5 Yes Yes 4 Pupil 6 Yes No 2 Pupil 7 Yes No 0 Pupil 8 Yes Yes 6 Pupil 9 Yes Yes 6 Pupil 10 Yes No 0 Avg. Change among Treated A = School 1: Avg. Change among Treated School 2: Avg. Change among Not-Treated A-B (A) (B) School 2 Pupil 1 No No 2 Pupil 2 No No 1 Pupil 3 No Yes 3 Pupil 4 No No 0 Pupil 5 No No 0 Pupil 6 No Yes 3 Pupil 7 No No 0 Pupil 8 No No 0 Pupil 9 No No 1 Pupil 10 No No 0 Avg. Change among Not-Treated B =
Intention To Treat School 1 Intention to treat? Treated? Observed Change in weight Pupil 1 Yes Yes 4 Pupil 2 Yes Yes 4 Pupil 3 Yes Yes 4 Pupil 4 Yes No 0 Pupil 5 Yes Yes 4 Pupil 6 Yes No 2 Pupil 7 Yes No 0 Pupil 8 Yes Yes 6 Pupil 9 Yes Yes 6 Pupil 10 Yes No 0 Avg. Change among Treated A = 3 School 1: Avg. Change among Treated School 2: Avg. Change among Not-Treated (A) 3 (B) 1 A-B 2 School 2 Pupil 1 No No 2 Pupil 2 No No 1 Pupil 3 No Yes 3 Pupil 4 No No 0 Pupil 5 No No 0 Pupil 6 No Yes 3 Pupil 7 No No 0 Pupil 8 No No 0 Pupil 9 No No 1 Pupil 10 No No 0 Avg. Change among Not-Treated B = 1
ITT / ToT Intention To Treat Treatment on Treated What happened to the average child who is in a treated school in this population? What happened to a child that actually received the treatment? Measuring the impact of launching the program Measuring the impact of the program itself - ITT and ToT are two different ways to analyze the data - ITT may relate more to actual programs, especially if imperfect compliance is likely to happen => Let s now see how we do it J-PAL POST-DESIGN CHALLENGES 33
From ITT to ToT We conceptually divide our treatment and control groups into three categories: 1/ The always takers, who will get the meals no matter if they are in the treatment or the control group 2/ The never takers, who won t get the meals no matter if they are in the treatment or the control group 3/ The compliers, who will behave according to the group they have been assigned to J-PAL POST-DESIGN CHALLENGES 34
A situation of imperfect compliance Treatment Group Control Group
Division into the three categories Treatment Group Control Group Always-takers Compliers Never-takers As the assignation was done randomly, the proportion of each category should be similar in Treatment and Control
Comparing the compliers Treatment Group Control Group Always-takers Compliers Never-takers To measure the impact of receiving the treatment, we compare compliers from Treatment and Control This measure of the impact is local : it is only valid for compliers. It can have a different impact for always-takers or never-takers.
ToT Estimator What values do we need? Y(T) Y(C) Prob[treated T] Prob[treated C] Y T Y C Prob treated T Prob[treated C] J-PAL POST-DESIGN CHALLENGES 38
ToT estimator School 1 Intention to treat? Treated? Observed Change in weight A = Gain if Treated B = Gain if not Treated Pupil 1 Yes Yes 4 Pupil 2 Yes Yes 4 Pupil 3 Yes Yes 4 Pupil 4 Yes No 0 Pupil 5 Yes Yes 4 Pupil 6 Yes No 2 Pupil 7 Yes No 0 Pupil 8 Yes Yes 6 Pupil 9 Yes Yes 6 Pupil 10 Yes No 0 Avg. Change Y(T) = School 2 Pupil 1 No No 2 Pupil 2 No No 1 Pupil 3 No Yes 3 Pupil 4 No No 0 Pupil 5 No No 0 Pupil 6 No Yes 3 Pupil 7 No No 0 Pupil 8 No No 0 Pupil 9 No No 0 Pupil 10 No No 0 Avg. Change Y(C) = ToT Estimator: A-B A-B = Y(T)-Y(C) Prob(Treated T)-Prob(Treated C) Y(T) Y(C) Prob(Treated T) Prob(Treated C) Y(T)-Y(C) Prob(Treated T)-Prob(Treated C) A-B
ToT estimator School 1 Intention to treat? Treated? Observed Change in weight Pupil 1 Yes Yes 4 Pupil 2 Yes Yes 4 Pupil 3 Yes Yes 4 Pupil 4 Yes No 0 Pupil 5 Yes Yes 4 Pupil 6 Yes No 2 Pupil 7 Yes No 0 Pupil 8 Yes Yes 6 Pupil 9 Yes Yes 6 Pupil 10 Yes No 0 School 2 Avg. Change Y(T) = 3 Pupil 1 No No 2 Pupil 2 No No 1 Pupil 3 No Yes 3 Pupil 4 No No 0 Pupil 5 No No 0 Pupil 6 No Yes 3 Pupil 7 No No 0 Pupil 8 No No 0 Pupil 9 No No 1 Pupil 10 No No 0 Avg. Change Y(C) = 1 A = Gain if Treated B = Gain if not Treated ToT Estimator: A-B A-B = Y(T)-Y(C) Prob(Treated T)-Prob(Treated C) Y(T) 3 Y(C) 1 Prob(Treated T) 60% Prob(Treated C) 20% Y(T)-Y(C) 2. Prob(Treated T)-Prob(Treated C) 40% A-B 540
ToT estimator The intuitive idea: Let s say the ITT effect of afterschool classes is a 2 point test score difference between treatment and control schools. But only 40% of the children in the treatment schools actually went to the classes (for simplicity let s assume no children in control schools got the classes). If the effect of 40% take-up is to increase scores by 2 points, then we can say that if everyone were to take the classes, the effect would be 1 2 0.4 5 points J-PAL POST-DESIGN CHALLENGES 41
The ITT estimate will always be smaller (e.g., closer to zero) than the ToT estimate A. True B. False C. Don t Know 0% 0% 0% A. B. C. J-PAL THREATS AND ANALYSIS 42
ITT / ToT: Conclusions Both ITT and ToT can provide valuable information to decision-makers ToT gives the effect of the intervention on the ones that take-up the programme ITT gives the overall effect of the intervention, admitting that partial compliance can happen (which is inherent to any policy) J-PAL POST-DESIGN CHALLENGES 43
Conclusions Internal validity is the great strenghth of Randomized Evaluations so everything undermining it must be carefully considered Design phase and power calculation are important but so is the ability to face challenges during implementation phase Distinguish well between attrition, spillovers and partial compliance J-PAL POST-DESIGN CHALLENGES 44
Thank you mpradhan@povertyactionlab.org Follow us on Twitter @JPAL_SA Explore our website www.povertyactionlab.org