THREATS TO VALIDITY Presentation by: Raul Sanchez de la Sierra
What I want you to avoid
Captain, sorry, I forgot the spillovers!
Lecture Overview External Validity Attrition Spillovers Partial Compliance Fishing for results
Lecture Overview External Validity Attrition Spillovers Partial Compliance Fishing for results
Threat to External Validity: Are these results applicable in a different context?
Generalizability of Results Depend: Sample: is it representative? Sensitivity: would a similar, but slightly different program, have same impact?
Lecture Overview External Validity Attrition Spillovers Partial Compliance Fishing for results
Attrition: the problem Is it a problem if some of the people in the experiment vanish before you collect your data? How about if mostly the treated people disappear? Why is it a problem? Should we expect this to happen?
Attrition: the problem April 06/07 Tests Tests Aug 05 Pay for performance Initial Test Fixed wage Jun 05 11
Attrition: the problem School nutrition program What if only children > 21 Kg come to school? A. Will you underestimate the impact? B. Will you overestimate the impact? C. Neither D. Ambiguous E. Don t know
Before Treatment After Treament T C T C 20 20 22 20 25 25 27 25 30 30 32 30 Ave. 25 25 27 25 Difference 0 Difference 2
Before Treatment After Treament T C T C [absent] [absent] 22 [absent] 25 25 27 25 30 30 32 30 Ave. 27.5 27.5 27 27.5 Difference 0 Difference -0.5
Attrition Bias: are we hopeless? There are solutions!
Attrition Bias: Solutions Implementation: Track participants Analysis: Check attrition by treatment status Check attrition by observables Bound the bias Suppose that dropped participants are extremes
Bound the bias: example April 06/07 Tests Tests Aug 05 Pay for performance Initial Test Fixed wage Jun 05 17
Lecture Overview External Validity Attrition Spillovers Partial Compliance Fishing for results
Spillovers: the problem Not in evaluation Total Population Target Population Evaluation Sample Random Assignment Treatment Group Control Group
Spillovers: the problem Not in evaluation Total Population Target Population Treatment à Evaluation Sample Random Assignment Treatment Group Control Group
Spillovers: the problem Not in evaluation Total Population Target Population Treatment à Evaluation Sample Random Assignment Treatment Group Control Group
Spatial spillovers example
Information spillovers example
Spillovers: the problem Example: Suppose you randomize vaccinations within schools What problems does this create for evaluation? How can we measure total impact?
Vaccines in school A:
No vaccines in school B:
Spillovers: the solution Design What unit of randomization? Design the randomization to estimate spillovers
Spillovers: designed based solution
Lecture Overview External Validity Attrition Spillovers Partial Compliance Fishing for results
Non Compliers: the problem Not in evaluation What can you do? Can you switch them? Target Population No! Evaluation Sample Random Assignment Treatment group Control group Participants No-Shows Non- Participants Cross-overs
Non Compliers : the problem Not in evaluation What can you do? Can you drop them? Target Population No! Evaluation Sample Random Assignment Treatment group Control group Participants No-Shows Non- Participants Cross-overs
Non Compliers : the solution Target Population Not in evaluation You can compare the original groups Evaluation Sample Random Assignment Treatment group Control group Participants No-Shows Non- Participants Cross-overs
Intention to Treat (ITT) Intention to Treat What happened to the average child who is in a treated school in this population? What does this measure mean?
When is ITT Useful? Impact of a vaccination program vs. Impact of a vaccination Which one is relevant to you?
Non Compliers: a general problem Movement across groups Example: School feeding program. Parents could attempt to move their children from comparison school to treatment school
Non Compliers: a better solution Always takers Never takers Compliers Defiers
Never takers TREAT! NOT TREAT!
Always takers TREAT! NOT TREAT!
Compliers TREAT! NOT TREAT!
Defiers TREAT! NOT TREAT!
Non Compliers: a better solution TREAT! NOT TREAT! TAKE PILL NOT TAKE PILL
Non Compliers: a better solution TREAT! NOT TREAT! TAKE PILL Compliers, Always takers NOT TAKE PILL
Non Compliers: a better solution TAKE PILL NOT TAKE PILL TREAT! Compliers, Always takers Never takers, Defiers NOT TREAT!
Non Compliers: a better solution TAKE PILL NOT TAKE PILL TREAT! Compliers, Always takers Never takers, Defiers NOT TREAT! Never takers, Defiers
Non Compliers: a better solution TAKE PILL NOT TAKE PILL TREAT! Compliers, Always takers Never takers, Defiers NOT TREAT! Never takers, Defiers Compliers, Never takers
Non Compliers: a better solution If there are no defiers: We can estimate perfectly the impact of the project among the compliers.
From ITT to LATE Local Average Treatment Effect (LATE) Local: only for those who obey the treatment (compliers) What is the impact of the vaccine amongst people who would take it if told to, and not if not told to? Is this population relevant? 47
Lecture Overview External Validity Attrition Spillovers Partial Compliance Fishing for results
Fishing for results: the problem Let s just measure everything: something may improve Problem: The more outcomes you look at, the higher the chance you find at least one significantly affected by the program
Fishing for results: the solution Solution: Pre-specify outcomes of interest Report results on all measured outcomes, even null results Correct statistical tests (Bonferroni)
Theory of Change GOOD EVALUATIONS! Pay for performance Matatus How to randomize Spillovers Why randomize Power And if you have any doubt: Call us! Sample