For Objec*ve Causal Inference, Design Trumps Analysis. Donald B. Rubin Department of Sta*s*cs Harvard University 16 March 2012

Transcription

1 For Objec*ve Causal Inference, Design Trumps Analysis Donald B. Rubin Department of Sta*s*cs Harvard University 16 March

2 Prologue to Objec*ve Causal Inference in Observa*onal Studies My Introduc*on Physics Wheeler 1961 Experimental Design Cochran 1968 Clear Separa*on Between Science = object of inference: DEFINE QUESTION FIRST What is done to learn about the science Intervene to measure aspects at a point in *me Same nota*on/representa*on of science no maxer how we try to learn about or measure Missing data always exist Cannot go back in *me 2

3 Poten*al Outcomes Approach to Causal Inference Simplest SeZng Units 1... N Y(1) Y(0) 3

4 Poten*al Outcomes Approach to Causal Inference Simplest SeZng Fundamental problem of causal inference For each i, only Y i (1) or Y i (0) can be observed Y(1) Y(0) T 1? 1.? 1 Units.? 1.? 0.? 0 N? 0 Random assignment of ac*ve versus control representa*ve sample of Y i (1) will be compared to representa*ve sample of Y i (0) 4

5 Poten*al Outcomes Approach to Causal Inference Simplest SeZng with Covariates Units 1... N X Y(1) Y(0) Same as before, except includes pretreatment covariates, e.g., age, sex, background educa*on Randomiza*on s*ll works for females 5

6 Randomized Trials That Are Designed Oden Using Covariates Randomized blocks (e.g., males, females) Forces balance on blocking variables Probability of treatment versus control can depend on covariates values, some*mes in complicated ways This is the template for the design and analysis of nonrandomized (e.g., observa*onal) data Assignment- based approaches use assignment mechanism for inference Fisherian & Neymanian Predic*ve approach predicts from observed values Bayesian posterior predic*ve (Rubin, 1978) 6

7 Design Observa-onal Studies to Approximate Randomized Trials 1. Hide outcome data un*l the design phase is complete 2. Think very carefully about decision makers and the key covariates that were used to make treatment decisions 3. If key covariates are not observed or very noisy, usually best to give up and seek bexer data source 4. Find subgroups (subclasses or matched pairs) in which the treatment and control groups have balance essen*ally the same distribu*on of observed covariates Not always possible to achieve balance Inferences are limited to subgroups where balance is achieved 5. Protocol specified analysis #1 - #5 combine to create an objec*ve design that approximates a randomized trial in each subclass that is balanced with respect to observed covariates 7

8 Illustra*ve Example with One Key Covariate (Cochran, 1968) Popula*on: Male smokers in U.S. Treatment = cigar/pipe smoking Control = cigarexe smoking Outcome = death rate/1000 person years Decision maker is the individual male smoker Reason for a smoking male to choose cigarexes versus cigar/pipe? Age is a key covariate for selec*on of smoking type for males 8

9 Subclassifica*on to Balance Age To achieve balance on age, compare: young cigar/pipe smokers with young cigarexe smokers old cigar/pipe smokers with old cigarexe smokers Or bexer, compare: Young, middle aged, old Even more age subclasses Design phase, no outcome data, objec*ve: Approximates a randomized trial within subclasses Now look at outcome data Reference: Rubin DB. The Design Versus the Analysis of Observa*onal Studies for Causal Effects: Parallels With The Design of Randomized Trials. Sta*s*cs in Medicine

10 Comparison of Mortality Rates for Two Variable Smoking Groups in U.S. Mortality Rates per 1000 person- years, % Adjusted Mortality Rates using subclasses, % CigareXe Smokers Cigar/Pipe Smokers age subclasses age subclasses age subclasses Source: Cochran WG. The effec*veness of adjustment of subclassifica*on in removing bias in observa*onal studies. Biometrics 1968; 24: Note: 20 four- level covariates over million million subclasses 10

11 Propensity Score Methods Rosenbaum and Rubin. The Central Role of the Propensity Score in Observa*onal Studies. Biometrika Observa*onal study analogue of randomiza*on The propensity score is the probability of treatment versus control as a func*on of observed covariates Model the reasons for treatment versus control at the level of the decision makers For example, logis*c regression model to predict cigarexe versus cigar/pipe smoking with age, educa*on, income, etc. as predictors Then subclassify (or match) on the propensity score as if it were the only covariate, e.g., 5-10 subclasses If correctly done, this creates balance within each subclass on ALL covariates used in es*ma*ng the propensity score Using diagnos*cs to assess and to document balance is cri*cal 11

12 Example: GAO Study of Breast Conserva*on versus Mastectomy Six large and expensive randomized clinical trials had been completed showing lixle difference for the type of women randomized in the trials and par*cipa*ng clinics Ques*on: Same results in general prac*ce? Observa*onal data available SEER Database: covariates, treatments, post- surgery outcomes Design phase Hide outcomes Balance covariates between treatment and control Reasons for mastectomy versus breast conserva*on Age, marital status, region of country, urbaniza*on, race, size of tumor, etc. Reference: Rubin DB. Es*mated Causal Effects from Large Datasets Using Propensity Scores. Annals of Internal Medicine 1997; 127, 8(II):

13 Es*mated 5- year Survival Rates for Node- nega*ve Pa*ents in Six Randomized Clinical Trials Study Women Breast Conserva*on (BC) Es*mated Survival Rate for Women Es*mated Causal Effect Mastectomy (Mas) BC Mas BC Mas n n % % % US- NCI Milanese French Danish EORTC US- NSABP Single- center trial; Mul*center trial Reference: Rubin DB. Es*mated Causal Effects from Large Datasets Using Propensity Scores. Annals of Internal Medicine 1997; 127, 8(II):

14 Propensity Score Analysis Approach Es*mate propensity scores Then subclassify (or match) on propensity score as if the only covariate, e.g., 5-10 subclasses Why does this work? Creates balance in each subclass on ALL covariates used in es*ma*ng the propensity score This balance will be achieved in large samples just like the balance that will be achieved in a large randomized clinical trial 14

15 Propensity Score Subclass Es*mated 5- year Survival Rates for Node- Nega*ve Pa*ents in the SEER Database within Each of Five Propensity Score Subclasses Women Breast Conserva*on (BC) Es*mated Survival Rate for Women Es*mated Causal Effect Mastectom y (Mas) BC Mas BC Mas n n % % % Averages Across Five Subclasses Reference: Rubin DB. Es*mated Causal Effects from Large Datasets Using Propensity Scores. Annals of Internal Medicine 1997; 127, 8(II):

16 Diagnos*cs for Accessing Balance Assessing balance simpler in large samples, just as with randomized experiments To illustrate diagnos*cs, use a marke*ng applica*on that involved a weight loss drug Units = doctors Treatment = sales rep visits doctor to discuss Control = no visit Decision- makers = sales reps Key covariates = prior Rxs, medical specialty, years in prac*ce, size of prac*ce, etc. 16

17 Histograms for background variable: Prior Rx Score (0-100) at Baseline Source: Rubin DB and Waterman RP. Es*ma*ng Causal Effects of Marke*ng Interven*ons Using Propensity Score Methodology. Sta*s*cal Science 2006; 21(2):

18 Histograms for background variable: Specialty Source: Rubin DB and Waterman RP. Es*ma*ng Causal Effects of Marke*ng Interven*ons Using Propensity Score Methodology. Sta*s*cal Science 2006; 21(2):

19 Histograms for summarized background variables: Linear Propensity Score Source: Rubin DB and Waterman RP. Es*ma*ng Causal Effects of Marke*ng Interven*ons Using Propensity Score Methodology. Sta*s*cal Science 2006; 21(2):

20 Histograms for a variable in a subclass of propensity scores: Prior Rx Score Source: Rubin DB and Waterman RP. Es*ma*ng Causal Effects of Marke*ng Interven*ons Using Propensity Score Methodology. Sta*s*cal Science 2006; 21(2):

21 Histograms for a variable in a subclass of propensity scores: Specialty Source: Rubin DB and Waterman RP. Es*ma*ng Causal Effects of Marke*ng Interven*ons Using Propensity Score Methodology. Sta*s*cal Science 2006; 21(2):

22 Marke*ng Example: Achieved Balance Within each narrow subclass of propensity scores, the treatment and control groups will be as balanced as if randomly divided Claim: This holds for all subclasses in which there are both treated and control subjects, and holds for all covariates that were used to es*mate the propensity score Works best when the propensity score subclasses have large sample sizes and are rela*vely narrow Five to ten propensity score subclasses oden fully adequate to balance all covariates No outcome data used in the design stage 22

23 * * 23 23

24 24 24

25 25

26 Simple Noncompliance, Instrumental Variables, and Bayesian Generaliza*ons Template for other observa*onal studies involves more complex randomized experiment Illustrate with completely randomized experiment with noncompliance with assigned treatment Return later to combined analysis with observa*onal study design

27 Sommer and Zeger Vitamin A Data Row True Compliance Type Treatment Assignment Treatment Received Y obs Number of Children 1? ? N N C C Reference: Sommer and Zeger (1991). On Estimating Efficacy from Clinical Trials. Statistics in Medicine.

28 Results of Three Standard MoM Analyses Method Es-mate Calcula-on Row Comparison ITT , 4, 5, & 6 vs. 1 & 2 As- treated & 6 vs. 1, 2, 3, &4 Per protocol & 6 vs. 1 & 2 Reference: Sommer and Zeger (1991). On Estimating Efficacy from Clinical Trials. Statistics in Medicine.

29 MoM CACE Analysis ACE = p N. NACE + p C. CACE = 0.2. NACE CACE = 0.8. CACE CACE = /0.8 =

30 Bayesian Analysis of Sommer & Zeger Data Imbens G.W. and Rubin D.B. (1997) Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. Annals of Statistics 25(1):

31 Bayesian Analysis of Sommer & Zeger Data, Marginal Posterior Distribu*ons with and without Exclusion Restric*on Es*mand Exclusion restric*on Mean Standard devia*on Median 5 th percen*le 95 th percen*le CACE No ITT Y (n) No CACE Yes Imbens G.W. and Rubin D.B. (1997) Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. Annals of Statistics 25(1):

32 Bayesian Analysis of Sommer & Zeger Data Imbens G.W. and Rubin D.B. (1997) Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. Annals of Statistics 25(1):

33 Bayesian Analysis of Sommer & Zeger Data Imbens G.W. and Rubin D.B. (1997) Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. Annals of Statistics 25(1):

34 Bayesian Analysis of Sommer & Zeger Data Imbens G.W. and Rubin D.B. (1997) Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. Annals of Statistics 25(1):

35 Hypothe*cal Example Illustra*ng Frequen*st Superiority of Bayes over IVE (MoM) and MLE, Popula*on Parameters with Exclusion Restric*ons and Monotonicity T P(C i = t π) D i (0) D i (1) Y i C i = t, Z i = 0, π Y i C i = t, Z i = 0, π c N(0.1, 0.16) N(0.9, 0.49) n N(1.0, 0.25) N(1.0, 0.25) a N(0.0, 0.36) N(0.0, 0.36) Imbens G.W. and Rubin D.B. (1997) Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. Annals of Statistics 25(1):

36 Hypothe*cal Example Illustra*ng Frequen*st Superiority of Bayes over IVE (MoM) and MLE, One Sample Imbens G.W. and Rubin D.B. (1997) Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. Annals of Statistics 25(1):

37 Hypothe*cal Example Illustra*ng Frequen*st Superiority of Bayes over IVE (MoM) and MLE, Frequen*st Evalua*on under Monotonicity and Exclusion Restric*ons Es*mator Posterior mean Posterior median Mean bias Median bias Root mean squared error Median absolute error Coverage rate 90% interval Median width MLE IVE Imbens G.W. and Rubin D.B. (1997) Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance. Annals of Statistics 25(1):

38

39 Jin and Rubin (JASA, 2008) - JR JR reanalyzed data from EF using principal stra*fica*on (Frangakis & Rubin, Biometrics, 2002) Randomized Treatment (ac*ve drug) versus Control (placebo) and meaured: Compliance (pill count) - intermediate outcome Cholesterol reduc*on - primary outcome Cri*cal axributes of Analysis Here: Principal stra*fica*on and the various problems that it can address Formula*on using hypothe*cal experiment Bayesian method of analysis, which easily incorporates scien*fic understanding 39

40 The EF Data 164 men were randomized to the treatment group and assigned the drug, Z i = T 171 men were randomized to the control group and assigned placebo, Z i = C For each pa*ent, cholesterol levels were measured before and ader taking the drug or placebo The outcome variable, Y i (T) or Y i (C), was the decrease in cholesterol level: the only variable used by EF or JR, besides treatment assigned and dose taken 40

41 Complica*ons with the EF Data Par*al and Extended Noncompliance Most pa*ents in the treatment group took only a propor*on of the assigned drug: D i (T) [0,1] Most pa*ents in the control group took only a propor*on of the assigned placebo: d i (C) [0,1] By design, D i (C ) = 0 and d i (T) = 0 Thanks to Brad Efron for sharing data 41

42 Rela*onship Between Observed Cholesterol Reduc*on and Observed Compliance Figures from Efron and Feldman,

43 How to es*mate dose- response? Observed dose- response in both arms Somehow, subtract Y(C) versus d(c) plot from Y(T) versus D(T) plot EF axempted this, but wrixen discussion (including by DBR) of ar*cle indicated debatable success Objec*ve in JR was to do this subtrac*on correctly under explicit assump*ons Here, highlight principal stra*fica*on, hypothe*cal experiment for dose- response, and Bayesian approach to analysis 43

44 Standard Assump*ons Stable Unit Treatment Value Assump*on (SUTVA): One pa*ent s treatment assignment does not affect other pa*ents poten*al outcomes; For each pa*ent, no hidden versions of treatment and no hidden versions of control Ignorable Treatment Assignment of T versus C True for randomized experiment These are accepted by both EF and JR. 44

45 Histograms of Observed Compliance Figures from Jin and Rubin,

46 Q- Q Plot of Observed Drug and Observed Placebo Compliance Figure from Jin and Rubin,

47 Possible Assump*ons at the Individual Level Perfect Blind: D i (T) = d i (C); obviously wrong Equipercen*le Equatable Compliances Align percen*les of D i (T) and d i (C), as in Q- Q So both known for all men: D i (T) = F(d i (C)) EF assume this, which is true in expecta*on Side- Effect Monotonicity Nega*ve: D i (T) < d i (C) Posi*ve: D i (T) > d i (C) JR assume nega*ve side effects; plausible 47

48 Meaning of d i and D i d i : compliance to placebo indicates pa*ent i s psychological compliance status, a covariate that is missing for men assigned drug D i : compliance to drug reflects both pa*ents i s psychological compliance status and his tolerance to nega*ve side effects of the drug, etc. But D i hints at possibility of es*ma*ng dose- response Similar comments in EF, but JR allow D i (T) F(d i (C)) 48

49 Es*ma*ng a Dose- Response Rela*onship within the Rubin Causal Model (Holland, 1986) To es*mate a dose- response rela*onship Need a hypothe*cal experiment where different doses of drug are randomly assigned and enforced Principal stra*fica*on framework (Frangakis and Rubin, 2002) vast generaliza*on of IVE The intermediate outcome d i (C) is unaffected by treatment assignment Therefore is a par*ally observed covariate For each stratum of pa*ents with the same d i (C), the assignment of dose is stochas*c and latent ignorable (Frangakis and Rubin, 1999) 49

50 Specific Hypothe*cal Experiment Measure d i* = baseline compliance for each pa*ent when assigned full placebo dose Randomly divide pa*ents into Treatment and Control In treatment group, stochas*cally assign dose Z Di < d i * according to a Beta random variable In control group, assign full placebo and measure d i We no*ce d i (C) = d i * in the control group, then toss d i * in the control group and in the treatment group Thus, nonignorable assignment of Z Di, but latent ignorable given d i * Also, forget the rule for the assignment of Z Di 50

51 Principal Stra*fica*on Framework for Dose- Response with d i (C) Defining Strata and Z Di (T) Defining Dose 51

52 Treatment Assignment Mechanism Nota*on θ = All parameters Assume d i (C) = d i * for everyone, denoted d i Actual Randomiza*on of Z=T versus Z=C [Z d i, Y i (C ), {Y i (Z Di )}, θ] = [Z θ] ~ constant Hypothe*cal Randomiza*on of Dose Z D given Z=T [Z Di d i,y i (C ),{Y i (Z Di )},Z=T,θ] = [Z Di d i, Z=T, θ] ~ d i. Beta(α 1, α 2 ) Latently ignorable given par*ally observed variable d i 52

53 JR s Parametric Model Given θ Covariate Distribu*on [d i θ] ~ Beta(α 3, α 4 ) Poten*al Outcomes Joint Distribu*on Close to EF s [Y i (C) d i, θ] ~ N(β 0 + β d i, σ C2 ) [Y i (Z Di ) Y i (C), d i, θ] ~ N[Y i (C) + γ 1 Z Di + γ 2 Z Di2 + γ 3 Z Di d i,σ 2 T. C ], Mutually condi*onally independent across the Z Di, - - plausible? And γ 1 >0, γ 2 >0, γ 1 + γ 3 >0 When Z Di = 0, expecta*on of Y i (Z Di ) - Y i (C) = 0 Dose- response is monotonely increasing for this range of doses 53

54 Prior Distribu*on on θ Prior distribu*on on parameters of Betas is specified by adding six fake observa*ons with both Z Di and d i observed on the equal percen*le equa*ng line and nothing else observed These are the minimum, 25 th percen*le, median, 75 th percen*le, maximum Purpose of these observa*ons is simply to stabilize computa*on and has lixle influence on inference (n fake =6 versus n real >300) and are accurate in expecta*on because of the randomiza*on Prior distribu*on on the rest of θ is independent and is the standard noninforma*ve prior propor*onal to 1/(σ C σ T. C ) 54

55 Figures from Jin and Rubin,

56 JR s Computa*on Missing data problem, which is addressed using MCMC to draw Bayesian inferences Parameters are θ Key missing data are d i for those assigned treatment and Z Di for those assigned control Given θ, draw key missing data; given key missing data, draw θ; iterate un*l approximate convergence Vast number of such draws approximates posterior distribu*on of dose- response as a func*on of principal strata defined by d i (C) 56

57 Diagnos*c Checks for JR s Model One Posterior Draw of Key Missing Data Figures from Jin and Rubin,

58 Dose- Response Results for Principal Strata Maximum d, 75 th d, median d, 25 th d Figures from Jin and Rubin,

59 Discussion of the Dose Response Conclusions Under EF s assump*ons, dose- response at each d i (C ) is a point because D i (T) = F(d i (C)) JR s dose- response results are causal under debatable assump*on Is Nature s randomiza*on of dose given placebo compliance (i.e., the crucial latent ignorability assump*on) plausible? Or do we need to condi*on further on background medical characteris*cs related to possible side effects of the drug? Such sensi*vity analysis is future work 59

60 60

61

62 Are Job- Training Programs Effec*ve? Donald B. Rubin Harvard University Presentation based on joint work with Fabrizia Mealli, Paolo Frumento, and Barbara Pacini. 62

63 The Na*onal Job Corps Study A randomized study to evaluate the effects of a training program on employment and wages Randomiza*on assures fair comparison, in expecta*on, between treatment groups Sampled youths (n=15,386) were assigned randomly to a job training program group or a control group Only those assigned to the job training program group were able to enroll in Job Corps Post- treatment complica*ons Noncompliance (only 73% axended the offered training) Trunca*on of wages for the unemployed Missing outcomes due to nonresponse 63

64 Poten*al Outcomes Approach to Causal Inference Simplest SeZng Units 1... N Y(1) Y(0) 64

65 Poten*al Outcomes Approach to Causal Inference Simplest SeZng Fundamental problem of causal inference For each i, only Y i (1) or Y i (0) can be observed Y(1) Y(0) T 1? 1.? 1 Units.? 1.? 0.? 0 N? 0 Random assignment of ac*ve versus control representa*ve sample of Y i (1) will be compared to representa*ve sample of Y i (0) 65

66 Poten*al Outcomes Approach to Causal Inference Simplest SeZng with Covariates Units 1... N X Y(1) Y(0) Same as before, except includes pretreatment covariates, e.g., age, sex, background educa*on Randomiza*on s*ll works for females 66

67 Poten*al Outcomes Approach to Causal Inference Simple Noncompliance with Ac*ve Treatment Units D(1) D(0) Y(1) Y(0) N 0 0 compliers noncompliers Randomiza*on s*ll works for compliers 67

68 Poten*al Outcomes Approach to Causal Inference Simple Noncompliance with Ac*ve Treatment: Observed Data D(1) D(0) Y(1) Y(0) T 1 1 0? 1 complier status observed. 1 0? 1 Units.? 0? 0.? 0? 0 complier status missing. 0 0? 1 noncomplier status observed N? 0? 0 noncomplier status missing Compliers For individuals assigned treatment (T=1), D(1)=1 & D(0)=0 For individuals assigned control (T=0), D(1)=? because true compliance under treatment is unknown & D(0)=0 Noncompliers For individuals assigned treatment (T=1), D(1)=0 & D(0)=0 For individuals assigned control (T=0), D(1)=? because true compliance under treatment is unknown & D(0)=0 Randomiza-on s-ll works for compliers 68

69 Key Idea: Principal Stra*fica*on (Frangakis and Rubin, 2002) Stra*fy on values of post- treatment intermediate outcome Convert D i (1), D i (0) into stra*fica*on variable True complier c if D i (1)=1 Noncomplier n if D i (1)=0 Idea works more generally 69

70 Intermediate Outcome - Employment Employed (yes, no) at a given *me post- treatment is an important outcome, but is also needed to define principal strata for final outcomes, Y, describing axributes of possible employment, such as wages, re*rement plan benefits, etc., which are not well- defined if unemployed Principal strata are defined by employment status EE = employed whether assigned to training or not EU = employed if trained, unemployed if not trained UE = unemployed if trained, employed if not trained UU = unemployed whether assigned to training or not Causal effects of training on Y only well- defined for EE UE empty? Reserva*on wage issue 70

71 Causal Effects of Training within Principal Strata Principal strata are defined by compliance with assignment to job training and by employment status c&ee, c&eu, c&ue, c&uu n&ee, n&eu, n&ue, n&uu By assump*on (exclusion restric*on on employment), we rule out n&eu and n&ue If assignment does not affect entry into training, assignment cannot affect employment status Also assume exclusion for axributes of employment, Y Causal effects of T on Y are only well- defined for c&ee and n&ee principal strata (no effect on Y in n&ee by exclusion restric*on) 71

72 Not Done Yet with Needed Principal Strata Indicators for response to survey items asking about employment status and wages, etc. R(1) and R(0), each indica*ng respondent or not Do not make exclusion restric*on here e.g., males could have R i (1) respond if assigned training, but R i (0) not respond if assigned control But do assume missing at random (MAR) A nuisance, not of scien*fic interest 72

73 Causal Effects Assignment to be trained on being job- trained Pr(c) = propor*on compliers Assignment to be trained on being employed Pr(c&EU) Pr(c&UE) Assignment to be trained on being employed for compliers [Pr(c&EU) Pr(c&UE)]/Pr(c) Rela*ve sizes of principal strata c&ee, c&eu, c&ue, c&uu, n&ee, n&uu Distribu*ons of X within principal strata 73

74 Causal Effects on Wages For the always employed Ave[Y i (1) - Y i (0) c&ee or n&ee] For the always employed compliers Ave[Y i (1) - Y i (0) c&ee] By exclusion, for the always employed noncompliers Ave[Y i (1) Y i (0) n&ee] = 0 74

75 Method of Analysis Direct likelihood at each of three post- treatment points in *me Search for parsimonious model to help guide policy Needs scien*fic judgement 75

76 Es*mated Means of Covariates within Principal Strata Week 52 Principal Stratum c&ee c&eu c&ue c&uu n&ee n&uu Percent in Stratum Female Age at baseline White With a Partner Has children Educa*on Ever arrested Mother s educa*on Father s educa*on Household income > $ Person income > $ Have job Had job, previous year Months in Job, previous year Earnings, previous year

77 Es*mated Means of Covariates within Principal Strata Week 130 Principal Stratum c&ee c&eu c&ue c&uu n&ee n&uu Percent in Stratum Female Age at baseline White With a Partner Has children Educa*on Ever arrested Mother s educa*on Father s educa*on Household income > $ Person income > $ Have job Had job, previous year Months in Job, previous year Earnings, previous year

78 Es*mated Means of Covariates within Principal Strata Week 208 Principal Stratum c&ee c&eu c&ue c&uu n&ee n&uu Percent in Stratum Female Age at baseline White With a Partner Has children Educa*on Ever arrested Mother s educa*on Father s educa*on Household income > $ Person income > $ Have job Had job, previous year Months in Job, previous year Earnings, previous year

79 Percent within Principal Strata by Time Period Principal Stratum c&ee c&eu c&ue c&uu n&ee n&uu Week Week Week For compliers, % EE increases in *me, and % UU decreases For noncompliers, EE remains fairly stable Causal effect of training slightly increases in *me, i.e., the difference between propor*ons in c&eu and c&ue appears to increase in *me Economists lock- in effect during the period of training 79

80 Es*mated Average Hourly Wages for Those Employed in Dollars within Principal Strata by Time Period Principal Stratum c&ee(1) c&ee(0) c&eu(1) c&ue(0) n&ee Week Week Week Es*mated causal effect on wages for always employed compliers is approximately 0.2 for all *me periods Always employed compliers, whether trained or not, have the lower hourly wages than the some*mes employed (c&eu or c&ue) or n&ee Wages tend to increase in *me 80

81 Final Conclusions for This Job Training Program In long run, for compliers, minor posi*ve effect on employment status For always employed compliers, minor posi*ve effect on wages at all *me periods Background characteris*cs of individuals differ across principal strata Suggests need for more targeted programs Even if evalua*on is based on randomized experiment, difficult to analyze correctly 81

82 Observed Values Covariates, X i Treatment received, T i Observed outcome: Cri*cal to understand the assignment mechanism Reasons for T i = 1 versus T i = 0 82