Instrumental Variables I (cont.)

Transcription

1 Review Instrumental Variables Observational Studies Cross Sectional Regressions Omitted Variables, Reverse causation Randomized Control Trials Difference in Difference Time invariant omitted variables are no longer a problem Regression Discontinuity Treatment and control groups very similar, exploit discontinuity to look for an effect Instrumental Variables I Start with Ordinary Least Squares Y = βx + ε If COV (X,ε) = 0 then β is said to be unbiased RCT and Regression discontinuity guarantee that the above condition is satisfied Instrumental Variables I (cont.) An Instrument Has to satisfy Two conditions A variable Z that is correlated with X Cov (Z,X) not equal to 0 A variable that is uncorrelated with Error term Cov (Z,ε) = 0 Can you think of an instrument that satisfies these two conditions? we use them all the time in clinical i l research Consider that in the Randomized control trial setting the Consider that in the Randomized control trial setting, the randomization procedure is the instrument

2 Instrumental Variables I (cont.) In Most Cases the instruments we find in the real world are not perfectly correlated with the In an RCT, the randomization tool predicts with a probability of 1 or 0 whether someone is offered the Most instruments ts simply increase the probability of getting by some amount We will come back to this Instrumental Variables I (cont.) Assume you found an instrument that satisfies both conditions How do you operationalize IV? When X and Z are categorical (0,1) variables: Wald Estimator [E(Y z=1 ) E(Y z=0 )] / [E(X z=1 ) E(X z=0 )] What is the Wald estimator in the RCT case: The analog is [E(Y T ) and E(Y C )] / (1-0) = [E(Y T )-E(Y C )] Instrumental Variables I (cont.) In regression: First run the following regression X = α + βz + ε Predict X using estimates from regression - X Then run the following regression using the predicted values of X Y = α + βx + ε Using only the variation in X that is due to the instrument Z Instrumental Variables II Where do instruments come from? Easy to satisfy first condition Cov (X,Z) ~= 0 Easy to find variables that are correlated with X For ex. one potential instrument for education Parental education How do you satisfy the second condition? Want to show that Cov (Z,ε)=0ε No obvious way to test whether this condition is satisfied Does parental education satisfy second condition?

3 Instrumental Variables II (cont.) You can test whether adding measurable covariates to your regression model changes your Instrumental t Variables estimate t (they shouldn t) But this is just a weak test t Doesn t get at whether instrumental variable is correlated with omitted/hard to measure variables Note that this test is similar to what we did in the Regression discontinuity and Diff. in Diff. cases Instrumental Variables II (cont.) Ultimately, whether the second condition is satisfied can only be decided a-priori based on an understanding of the institutional details Instrumental Variables III Ex. 1 - Effect of police on crime? (Levitt,1997) Causality hard to show in cross section, WHY? Level of crime determines level of police force Likely to find a positive relationship Instrument Whether or not it s an mayoral or gubernatorial election year Politicians up for re-election want to show that they are tough on crime Instrumental Variables III (cont.)» First condition is satisfied: Election year is correlated with size of police force Second condition is likely l satisfied» No a-priori reason to think election years are related to other characteristics that might affect crime rates

4 Instrumental Variables III (cont.) Ex. II Does more intensive of MI reduce mortality (McClellan et al. 1994)? Does Catheterization and subsequent revascularization procedure affect MI outcomes? Cross-sectional data If use of catheterization is correlated positively with initial iti sickness then underestimate effectiveness of catheterization Instrumental Variables III (cont.) If use of catheterization is correlated negatively with initial sickness then overestimate effectiveness of catheterization Without some type of randomization it would be difficult to tell a-priori which direction the bias might run Descriptive Statistics No Cath 90 Days Cath 90 Days Age Female 53.8% 41% Walks indep. 68% 89.5% CVA/Stroke 18% 9.8% Kilip Class=1 56% 73% 90 day Cabg 0 31% 90 day Ptca 0 38% Mortality 30,90 29%,48% 6.8%,14.9% Instrumental Variables III (cont.) What s the Instrument? Instrument Differential Distance [Distance to Nearest cath. Hospital Distance to nearest non. Cath. Hospital] The lower this term is the more likely that one would expect to receive a cath Instrument 2- Is the closest hospital a cath. Hospital? Hospital does 5 or more procedures.

5 Descriptive Statistics Diff. Dist. <=-.5 Diff. Dist. > -.5 Age Female 48.2% 47.5% Walks independently 78% 78% CVA/Stroke 14.4% 13.9% Kilip Class=1 63% 64% 90 day Cath 49.7% 43.3% 90 day 14.7%,19.2% 13.8%,16.1% cabg,ptca Mortality 30,1yr 18.6%,32.2% 19.1%,32.8% Instrumental Variables III Does the Differential Distance measure satisfy two conditions? 90 Day cath. Rate is 49% vs 43% for < -.5vs vs. >-.5 Looks balanced on observable covariates But not totally clear, If sicker individuals decide to live closer to cath. hospitals, the second condition is not satisfied Using instrument will still underestimate effect Does closest hospital measure satisfy first condition? 26% rate of cath. if closest hospital is cath. Hospital 20% rate of cath. If closest hospital is non. Cath. hospital Two samples also look balanced on observables Instrumental Variables III Ideally we want our instrument t to predict perfectly as in RCT (0,1) For our first instrument t probability bilit of t t increased from 43% to 49% For our second instrument t probability bilit of increased from 20% to 26% Instrument t is only informative for individuals id on this margin Similar il to regression discontinuity, it estimates t only informative for individuals near the threshold Instrumental Variables III (cont.) - RESULTS IV estimates are about 30% lower than OLS estimates that adjust for a variety of comorbidities and other health care use (beta- blockers) This means in the cross section, healthier individuals are much more likely to get catheterized

6 Final Example Effect of Schooling on Income? (Angrist/Krueger,1992) OLS estimates consistently show a positive relationship Many possible omitted variables that are hard to measure Ability Motivation - Effort Family Connections Health Education coefficient likely biased upwards Final Example (cont.) Instrument Quarter of Birth Quarter of birth is correlated with amount of education. Men born in early quarters of the year tend to have lower average schooling Can you guess Why? Those born early in the year and those born later in the year enter school at the same time Children born in the 4 th quarter enter school at age 5.75, while those born in the first quarter enter school at age 6.75 Final Example (cont.) Compulsory school laws require that students remain in the school until their 16 th birthdays This means kids born in different quarters are forced to attend school for different lengths of time Final Example (cont.) Date of Birth Percent Enrolled April Born Jan 1 Mar (age 16) 87.6% Born April 1 Dec (age 15) 92.1% Difference -5.5%

7 Quarter of Birth and Education Level Final Example (cont.) ation Y ears o o f Educ Line 1 Is quarter of birth uncorrelated with error term? Very likely - quarter of birth unlikely to be unrelated to ability, motivation, health, family connections No a-priori reason to think otherwise But as we said before no real way to test for this Year and Quarter of Birth Log Weekly Ea arnings Graphical Results Quarter of Birth and Earnings Year and Quarter of Birth 1 Final Example (cont.) Result - Wald Estimator Ratio of Change in Earnings and Change in Education Born in Quarter 1 vs. Quarters vs difference years of education vs difference in Log (weekly wages) Ratio = ( )/ ( ) =.072» An extra year s worth of education increases Log(weekly wages) by 7%

8 Final Example (cont.) Some Specification Checks Look at whether education of parents varies systematically for individuals born in different quarters of birth Look to see if quarter of birth affects the level of education of college graduates Alternatively just look at individuals with education levels of high school or less More Examples Look at effect of Insurance on health Instrument 1 Job characteristics and other common demographics Instrument 2 Self-Employment status Both of these instruments fail the second condition Look at Effect of BMI on Occupational Attainment Instrument 1 - Area level mean BMI Instrument 2 Area level overweight Both of these instruments also fail the second condition Instrumental Variables III What does IV measure? In RCT,,you get a measure of the average effect In IV, you are getting a measure of the marginal effect That is you are only getting a effect for those individuals for whom the instrument makes a difference in getting g Instrumental Variables III (cont.)- Never Taker Complier Always taker Instrument No Effect, Gets more No Effect, never takes Always up because of instrument takes up

9 Instrumental Variables III (cont.) Apply to MI/Catheterization Example In this example even having a cath. Hospital as the first hospital implies that an individual does not get cath. 74% of the time NEVER TAKER In this example even having a NON Cath. Hospital as the first hospital implies that an individual does get a cath. 20% - ALWAYS TAKER IV in this example only affects 6% of the population Instrumental Variables IV (cont.) We said first condition was that Instrument, Z, is correlated with X How strong does this correlation have to be? Stock and Staiger recommend 1-st stage F- statistic is > 10 F-statistic = [Model Mean Square / Error Mean Square] From simulations Stock/Staiger show that when the F-statistic is below 10 the IV estimates biased towards OLS estimates Other Applications for IV, RCT Randomized Control Trials Individuals in group don t always comply Not everyone takes medications Alternatively, individuals offered vouchers don t always take them up If there a lot of non-compliers, then you are really getting estimates of what? being offered the (which is also interesting) This is called intent-to-treat analysis Other IV Applications, RCT (cont.) Is there a way to recover something about effect instead of the offer effect? One Solution is to use the randomization as an instrumental variable Note that randomization tool satisfies both conditions of valid instrument Correlated with Uncorrelated with omitted variables This is called the Treatment on Treated (TOT) Average effect of the on those in the group who actually get

10 Other Applications, RCT (cont.) Instrument Accept Don t Accept Offer Compliers Don t Comply Get Don t get Don t Offer No No Other Applications, RCT (cont.) TOT EFFECT is the WALD Estimator Z is the randomization tool-the offer of X is whether or not someone accepts Y is your outcome [E(Y Z=1)-E(Y Z=0)] / [Pr(X=1 Z=1) P(X Pr(X=1 Z=0)] Other Applications, RCT (cont.) What is Pr (x=1 z=1)? Share of people who are accept given offer: Compliance What is Pr (x=1 z=0)? Share of people who get when not offered: Zero What is E(Y z=1)? Mean of Outcome variable for those offered What is E(Y z=0)? Mean of outcome variable for those not offered Treatment Other Applications, RCT cont. This means that TOT is simply: ITT/(Share of people p who COMPLY) Intuitively this should make sense, Why? Because as the share of people who comply Because as the share of people who comply approaches 1 then TOT approaches ITT

11 Propensity Score - Briefly A complicated Matching Procedure T=1 get treated, T=0 no Propensity Score: Pr (T=1 Set of Covariates) When is this useful? When you have a group and you have to sample from the population to find an appropriate control group Easy to do if just one covariate, harder if many covariates This is also useful if you have a and control group but and control groups look very different Propensity Score I (cont.) - Intuitively it tells us the probability that someone would be treated using the covariates that you have identified How do you operationalize Run a regression procedure (OLS, Logistic) where dependent variable is whether or not someone got Using the coefficients from the regression predict the probability of getting Propensity Score (cont.) Once you have a propensity score How do you actually do the matching? Should you match with replacement or without replacement? What is your definition of close when matching on propensity score values? Propensity Score Method 1 Pick propensity score strata and compare outcomes for treated and untreated individuals in the strata How fine should the strata be?» If strata are too tight then you might not find matches» If strata are too broad then your control and treated groups are more likely to be different» This is a tradeoff between precision and bias

12 Propensity Score (cont.) - Method 2 Nearest Available matching Treated and Control groups are randomly ordered First treated individual matched with first control individual with closest propensity score (again what is close) Method 3 - Rank High to Low on propensity score for the and control groups Match highest ranked unit first and remove matched comparison unit Keep going Propensity Score II Variety of methods When there is a lot of overlap between the treated and control groups in propensity scores, different methods produce same results When there is not a lot of overlap Either need to use techniques where there is replacement (observations reused) Or just have to throw away data Different techniques might have dramatically different results Propensity Score III Notice that the Propensity Score technique does NOT solve omitted variables problem How can it? There is no new information! Final Thoughts The majority of papers you read will be from observational studies with cross-sectional data Always useful to ask yourself, if I had to do this paper in a RCT framework how would I do it? Because I can t do an RCT, what are the omitted variables the model does not capture? Are these omitted variables likely to be important? Regression discontinuity is underused, quite a few examples of this in the real world but very few papers Valid instrumental variables are difficult to find Quasi-Experiments will be the primary tool for causal analysis in the social sciences

13 References Identification of causal effects using instrumental variables, Angrist, Imbens, Rubin, JASA, 1996, Does compulsory school attendance affect School and Earnings, Angrist, Krueger, QJE, Does more intensive of AMI reduce mortality, McClellan, Newhouse, McNeil, JAMA, Using electoral cycles in police hiring to estimate the effect of police on crime, Levitt, AER, 1997