Experimental Methods. Policy Track

Transcription

1 Experimental Methods Policy Track East Asia Regional Impact Evaluation Workshop Seoul, South Korea Nazmul Chaudhury, World Bank

2 Reference Spanish Version & French Version also available, Portuguese soon. Free download at

3 Impact Evaluation Road Map Logical Framework Measuring Impact How the program works in theory Methods to identify impact Operational Plan Resources

4 Our Objective Estimate the causal effect (impact) of program (P) on outcome (Y). Impact= Y T - Y C P = Program or Treatment Y = Indicator, Measure of Success Y T = Outcome with the program Y C = Outcome without the program (control)

5 Example: CCT Progresa National anti-poverty program in Mexico Cash Transfers conditional on school and health care attendance. Targeting Eligibility based on a poverty index Timing: Started 1997 Phased Roll-out, 5 million beneficiaries by 2004

6 Research Question What is the Impact of. a conditional cash transfer (P) on household consumption (Y)

7 Challenge No counterfactual Impact= Y T - Y C We do not observe what would have happened to the pupils if they did not receive any cash transfer (the counterfactual )?

8 CLONE PERFECT COUNTERFACTUAL

9 Perfect Experiment First, identify the target beneficiaries

10 Perfect Experiment - Clones and then clone the target beneficiaries

11 Perfect Experiment - Clones Give the cash transfer to one set of the clones

12 Perfect Experiment - Clones and compare their consumption some time later Because the people who received the cash transfer are exactly the same as those who did not receive the cash transfer, we can truly attribute the difference to the program

13 Before & After Comparing those enrolled with those not BAD COUNTERFACTUALS

14 Case 1: Before & After (1) Observe only beneficiaries Y 268 A (2) Two observations in time: Consumption in 1997 Consumption in t=1997 t=1998 B $35 Time ESTIMATE OF IMPACT =A-B= $35

15 Case 1: Before & After Consumption (Y) Consumption after program start (treatment) Consumption before program start (control (counterfactual)) Estimate of Impact 35.3*** Estimated Impact on Consumption (Y) Linear Regression 35.27** Multivariate Linear Regression 34.28** Note: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

16 Case 1: Problem: we don t know what would have happened without the program Economic Boom: o Real Impact=A-C o A-B over-estimates impact Economic Recession: o Real Impact=A-D o A-B underestimates impact Y T=0 T=1 A C? B D? Impact? α = $35 Impact? Time

17 Case 2: Some people enroll, others not Ineligibles (Non-Poor) Eligibles (Poor = Target Population) Not Enrolled Enrolled

18 Case 2: Some people enroll, others not TREATMENT GROUP CONTROL GROUP

19 Case 2: Problem of Selection Bias What if those who chose not to enrol are different?

20 Case 2: Problem of Selection Bias And, what if we cannot observe (control for) these differences

21 Case 2: Problem of Selection Bias And, these difference influence outcomes? Are the factors that determine enrolment correlated with consumption?

22 Case 2: Some people enroll, others not Consumption (Y) Consumption with program (enrolled - treatment) Consumption without program (not enrolled - control) Estimate of Impact -22** Estimated Impact on Consumption (Y) Linear Regression -22** Multivariate Linear Regression Note: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

23 Progresa Policy Recommendation? Impact on Consumption (Y) Case 1: Before & After Case 2: Enrolled & Not Enrolled Linear Regression 35.27** Multivariate Linear Regression 34.28** Linear Regression -22** Multivariate Linear Regression Note: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

24 Keep in Mind Before-After Comparison Problem: other factors that matter also change over time. Compare those who enroll with those who don t Problem: Selection bias. The enrolled may be different, and we don t observe these differences. Both comparison groups may lead to biased estimates of the program impact.

25 RANDOMIZATION GOOD COUNTERFACTUAL

26 Randomization: creating similar groups With large sample, two groups have very similar characteristics ON AVERAGE

27 External and Internal Validity Random Assignment Target Population Random Sample External Validity Internal Validity

28 Drawing a random sample from two groups does not make them comparable. Random Sample is NOT the Same as Random Assignment! Participants Non Participants

29 Case 3: Randomized Assignment in Progresa How do we know we have good clones of the treatment group? In the absence of Progresa, treatment and comparisons should be identical Let s compare their characteristics at baseline (t=0)

30 Case 3: Balanced characteristics at baseline Consumption ($ monthly per capita) Head s age (years) Spouse s age (years) Head s education (years) Spouse s education (years) Randomized Assignment Treatment Comparison T-stat ** Note: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

31 Case 3: Balanced characteristics at baseline Randomized Assignment Treatment Comparison T-stat Head is female= Indigenous= Number of household members Bathroom= Hectares of Land Distance to Hospital (km) Note: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

32 create the correct counterfactual Y 268 A True Impact 239 B 233 t=1997 t=1998 Measure of Impact=A-B= $29

33 Note: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**). Case 3: Randomized Assignment Treatment Group (Randomized to treatment) Counterfactual (Randomized to Comparison) Impact Baseline (t=0) Consumption (Y) Follow-up (t=1) Consumption (Y) ** Estimated Impact on Consumption (Y) Linear Regression 29.25** Multivariate Linear Regression 29.75**

34 Progresa Policy Recommendation? Impact of Progresa on Consumption (Y) Case 1: Before & After Case 2: Enrolled & Not Enrolled Case 3: Randomized Assignment Multivariate Linear Regression 34.28** Linear Regression -22** Multivariate Linear Regression Multivariate Linear Regression 29.75** Note: If the effect is statistically significant at the 1% significance level, we label the estimated impact with 2 stars (**).

35 PROGRESA EXAMPLE RANDOMIZATION IN PRACTICE

36 Program Operates at Community Level

37 Community-level Randomization Randomly assign communities to receive program 320 treatment 506 communities 186 control

38 Program was randomly phased in Before During After Time evaluation period

39 Keep in Mind Randomized Assignment In Randomized Assignment, large enough samples, produces two statistically equivalent groups. We have identified the perfect clone!

40 Pop Quiz!

41 What is the problem when comparing those who enroll with those who choose not to? A. Those not enrolled could be very different to those enrolled, which might be correlated to the outcome of interest. B. Those not enrolled will also want to receive the program. C. Those not enrolled could be very different to those enrolled, but which is NOT correlated to the outcome of interest.

42 What is the biggest problem of doing a before/after comparison? A. Sample size might not be large enough B. Any difference in outcome we observe before-after could be due to other factors that changed. C. You might not find the people you surveyed in the beginning

43 What is a counter-factual? A. An argument B. It is what would have happened if the program did not occur. C. A type of vaccination. D. Contrary to the facts

44 Why can randomization create the correct counter-factual? A. It is fair because everyone has an equal chance of receiving the program. B. It clones each person. C. Those who receive the program are, on average, the same as those who did not receive the program.