Program Evaluations and Randomization Lecture 5 HSE, 10.11.2014 Dagmara Celik Katreniak
Overview Treatment Effect under perfect conditions Average Treatment Effect Treatment effect under imperfect conditions Intention To Treat (ITT) Treatment On the Treated (TOT) Local Average Treatment Effect (LATE) Other Externality Effect
Average Treatment Effect (ATE) SUTVA assumption Non-interference Being in the treatment or control group does not influence the outcome of anyone No variation in treatment All subjects are subject to the same treatment Unconfoundedness/Ignorability Assignment to treatment not connected with outcome
Average Treatment Effect (ATE) Randomly assign subjects to T or C Calculate ATE ATE = measures the effect of treatment on a randomly selected person Randomization not conditional on observables Perfect compliance No attrition SUTVA and unconfoundedness
Example Blimpo (AEJ: Applied Microeconomics, 2014) All takeovers, no attrition Randomized at school level T1= Individual target group T2= Team target group T3= Team Tournament group C = Control group sssss i,s = β 0 + 3 k=1 β k T i,s k + ε i,s
Randomization balance
The conditional probability of selection Stratify your sample By stratas, randomize into T/C The probability to be picked to Treatment depends on observables (stratification) The allocation of T unequal across stratas Possible for example if the number of people in T and C not equal but fixed number of awards in T
Stratified sample Checking randomization balance Stratum level But then no overall balance check Problem of too many tests at once Possibly low power cause of small strata size Overall/general level But may not be balanced by stratas F-test by Firpo, Foguel and Borges Jales (conference paper) What if imbalances exist?
Example Voucher program in Colombia Students picked to schools by lottery Given number of winners in each city The ratio of winners differs by town Lottery random within each city conditional on whether households have an access to a phone Causal effect of the program on voucher s applicants with access to a telephone in surveyed cities (Angrist et al., 2001)
The conditional probability of selection Weighted average over all strata Weight = proportion of treated subjects within strata Regression Control for all strata variables Include all dummies plus their interactions Be careful about the degrees of freedom
The conditional probability of selection Conditional Randomization implies: E Y i C X, T E Y i C X, C = 0 E x E Y T i X, T E Y C i X, T = E Y T i x, T E Y C i x, C P X = x T dd If discrete comparison of means using proportion of treatment within cells as weights
(Im)Perfect Compliance Subjects from Treatment/Control group without/with treatment Spillover effects Low take up rate Switching of subjects during the experiment What is the question of interest? The effect of the intervention itself? The effect of a diet rich in iron The effect of the instrument? Common for policy relevant research Do we need all participants to participate or we want to see the effect on those who choose to participate?
(Im)Perfect Compliance Perfect compliance of interest Prepare design accordingly Example: Thomas, Frankenberg, Friedman, Habicht, & Al (2003) on iron supplementation Partial compliance of interest Esp. encouragement designs Duflo and Saez (2003) Partial compliance as the only option Deworming program (Miguel & Glewwe, 2004) Mostly because tracking not possible
TOT=? ITT=?
Intention to Treat Estimate (ITT) The effect of offered intervention Differ from the effect of the treatment in case of imperfect compliance Based on the initial division into treatment and control group Perfect compliance ITT=ATE
Average Treatment Effect on Treated Estimate (TOT) Y(T) = A PPPPPPPPPPP TTTTTTT T) + B (1 PPPPPPPPPPP TTTTTTT T)) Y C = A PPPPPPPPPPP TTTTTTT C) + B (1 PPPPPPPPPPP TTTTTTT C)) TTT = Y T Y C PPPP TTTTTTT T) PPPP TTTTTTT C)
Scenario 1: Example Researcher chooses 100 people into treatment group (he gives them $100) and 100 people into control group (gives them 0) Collects information on rewards All people in T got money No person from C got money ITT=? TOT=? ITT=TOT=$100
Example Scenario 2: Researcher tells 100 students if they come to his office, he gives them $100 (= treatment), he says nothing to the control group Cca 20% of students do not trust him/lazy to come Collects information on rewards On average treatment group received $80, control $0 ITT=?, TOT=? ITT=$80 TOT=$100
Example Scenario 3: Researcher tells subjects to come at midnight to his office to pick up $100 (treatment), he says nothing to the control group Only 50% of treatment group show up Some 10% of control group students show up Collects information on rewards ITT=?, TOT=? ITT=$40 TOT=$100
Example Scenario 4: Researcher gives $25, $50, $75, $100 to four groups of students (each 1/4 in size), $0 to control 30% of treated pick up money, no control Collects information on rewards ITT=?, TOT=? ITT=$21 If everyone came, TOT=$62.5 When 30% came, TOT=$21/0.3=$70
ITT versus ATE Two identification assumptions (Imbens and Angrist, 1994): Independence Outcomes are not directly affected by the instrument (Duflo, Glennerster & Kremer, 2007, p.52) Y i C, Y i T, T i 1, T i 0 Z Monotonicity The monotonicity assumption requires that the instrument makes every person either weakly more or less likely to actually participate in the treatment (Duflo, Glennerster & Kremer, 2007, p.52) T i 1 T i 0 fff aaa i or T i 1 T i 0 fff aaa i
ITT versus ATE Wald estimate local average treatment effect (LATE) on compliers, i.e. on those whose treatment status was affected by the instrument IV Estimate of β (using Z as an instrument) can be interpreted as the ATE for a well-defined group of individuals, namely those who are induced by the instrument Z to take advantage of the treatment (Duflo, Glennerster & Kremer, 2007, p.53)
ITT versus ATE β W = E Y i Z i =1 E Y i Z i =0 E T i Z i =1 E T i Z i =0 β W = E Y i T Y i C T i 1 T i 0 = 1
IV estimate (LATE) No person from control group got treatment LATE = TOT All people from treatment group got treatment Oversubscription design Interpretation! Causal effect for compliers Not necessarily the effect for the entire population
Example Voucher program in Colombia PACES = governmental program, scholarships for private secondary schools assigned by lottery Lottery winners and losers picked from PACES program applicants Stratification by localities, conditional to access to a phone What effects do they measure? The effect of winning the lottery The effect of scholarship
Angrist et al. (2002)
Estimates of the effects The effect of winning the lottery on scholarship use, school choice and schooling 6-7% more likely to begin the 6 th grade 15-16% more likely to be in private school Decision to be in private school seems to be sensitive to price, decision to attend the school not Lower repetition rate for winners Higher likelihood to be in 8 th grade for winners Angrist et al. (2002)
Estimates of the effects The effect of winning the lottery on test scores Subsample of children invited by phone to be tested + by letter More than one test opportunity if failed Refreshments, after each test other rewards picked by lottery (e.g.bicycle) Travel costs covered NO SIGNIFICANT relationship between being tested and voucher status Angrist et al. (2002)
Angrist et al. (2002)
Estimates of the effects The effect of receiving scholarship Using voucher win/loss status as IV Initial randomization = IV versus Actual treatment 6% (24%) of lottery users used voucher 90% of winners used scholarship Two stage least squares estimation The 2SLS estimates based on this difference are necessarily larger than the reduced form effects of winning the lottery since winning the lottery is only imperfectly correlated with receiving a scholarship. Angrist et al. (2002)
Spillovers How would you measure them? Vary the exposure to a treatment Duflo and Saez (2003) Use variation in exposure across groups that arises from randomization Duflo, Kremer and Robinson (2006) Random assignment to peer groups
Attrition May or may not be a problem Problem if there is correlation between attrition and treatment Biases results Random attrition Lowers statistical power Non-random attrition No independence of outcomes Try to limit ex-ante
Attrition Parametric techniques Heckman correction Survival model and attrition Non-parametric techniques Manski-Lee bounds Matching matching lottery winners and losers Angrist et al (2006) Imputation methods
Case Study Please, work on the case study
Summary Randomization Experimental Designs Sample size and power calculations Budget, CBA and CEA Data Analysis under perfect and imperfect randomization *** Further use of randomization Topics Presentations