ALMALAUREA WORKING PAPERS no. 9

Transcription

1 Snce 1994 Inter-Unversty Consortum Connectng Unverstes, the Labour Market and Professonals AlmaLaurea Workng Papers ISSN ALMALAUREA WORKING PAPERS no. 9 September 211 Propensty Score Methods for Causal Inference: On the Relatve Importance of Covarate Selecton, Relable Measurement, and Choce of Propensty Score Technque by Peter M. Stener Unversty of Wsconsn Madson Ths paper can be downloaded at: AlmaLaurea Workng Papers seres Also avalable at: REsearch Papers n Economcs (RePEC)

2 The AlmaLaurea workng paper seres s desgned to make avalable to a wde readershp selected works by AlmaLaurea staff or by outsde, generally avalable n Englsh or Italan. The seres focuses on the study of the relatonshp between educatonal systems, socety and economy, the qualty of educatonal process, the demand and supply of educaton, the human captal accumulaton, the structure and workng of the labour markets, the assessment of educatonal polces. Comments on ths seres are welcome and should be sent to pubblcazon@almalaurea.t. AlmaLaurea s a publc consortum of Italan unverstes whch, wth the support of the Mnstry of Educaton, meets the nformaton needs of graduates, unverstes and the busness communty. AlmaLaurea has been set up n 1994 followng an ntatve of the Statstcal Observatory of the Unversty of Bologna. It supples relable and tmely data on the effectveness and effcency of the hgher educaton system to member unverstes governng bodes, assessment unts and commttees responsble for teachng actvtes and career gudance. AlmaLaurea: facltates and mproves the hrng of young graduates n the labour markets both at the natonal and nternatonal level; smplfes companes' search for personnel, reducng the gap between the demand for and supply of qualfed labour ( makes avalable onlne more than 1.5 mllon currcula (n Italan and Englsh) of graduates, ncludng those wth a plurannual work experence ( ensures the optmzaton of human resources utlzaton through a steady updatng of data on the careers of students holdng a degree ( Each year AlmaLaurea plans two man conferences ( n whch the results of the annual surveys on Graduates Employment Condtons and Graduates Profle are presented. AlmaLaurea Inter-Unversty Consortum vale Masn Bologna (Italy) Webste: E-mal: pubblcazon@almalaurea.t The opnons expressed n the papers ssued n ths seres do not necessarly reflect the poston of AlmaLaurea AlmaLaurea 211 Applcatons for permsson to reproduce or translate all or part of ths materal should be made to: AlmaLaurea Inter-Unversty Consortum emal: pubblcazon@almalaurea.t fax phone

3 Internatonal Conference on Human Captal and Employment n the European and Medterranean Area Bologna, 1-11 March 211 Propensty Score Methods for Causal Inference: On the Relatve Importance of Covarate Selecton, Relable Measurement, and Choce of Propensty Score Technque by Peter M. Stener * The author was supported n part by grant R35D133 from the Insttute of Educatonal Scences, U.S. Department of Educaton. I thank Thomas D. Cook, Kelly Hallberg, Steff Pohl, and Wllam R. Shadsh for helpful dscussons of studes revewed n ths paper. Abstract: The popularty of propensty score (PS) methods for estmatng causal treatment effects from observatonal studes has ncreased durng the past decades. However, the success of these methods n removng selecton bas manly rests on strong assumptons, lke the strong gnorablty assumpton, and the competent mplementaton of a specfc propensty score technque. After gvng a bref ntroducton to the Rubn Causal Model and dfferent types of propensty score technques, the paper assess the relatve mportance of three factors n removng selecton bas n practce: () The avalablty of covarates that are related to both the selecton process and the outcome under nvestgaton; () The relablty of the covarates measurements; And () the choce of a specfc analytc method for estmatng the treatment effect ether a specfc propensty score technque (PS matchng, PS stratfcaton, nverse-propensty weghtng, and PS regresson adjustment) or standard regresson approaches. The mportance of these three factors s nvestgated by revewng dfferent wthn-study comparsons and meta-analyses. Wthn-study comparsons enable an emprcal assessment of PS methods performance n removng selecton bas snce they contrast the estmated treatment effect from an observatonal study wth an estmate from a correspondng randomzed experment. The emprcal evdence ndcates that the selecton of covarates counts most n reducng selecton bas, ther relable measurement next most, and the mode of data analyss ether a specfc propensty score technque or standard regresson s of least mportance. Addtonal evdence suggests that the crucal strong gnorablty assumpton s most lkely met f pretest measures of the outcome or constructs that drectly determne the selecton process are avalable and relably measured. * Unversty of Wsconsn Madson e-mal: pstener@wsc.edu 1

4 1. Introducton The attenton pad to causal nference n general and to propensty score methods for estmatng causal effects n partcular has been consderably ncreasng over the last three decades. Though randomzed experments are frequently consdered as the gold standard for causal nference, randomzaton s frequently not possble due to ethcal, admnstratve, or budgetary reasons. In such cases quas-expermental methods lke regresson-dscontnuty desgns, nterrupted tme seres analyss, nstrumental varable approaches, or non-equvalent control group desgns mght be employed to estmate the treatment effect of an nterventon or program (Shadsh, Cook & Campbell, 22). In comparson to a randomzed experment, quas-expermental methods do not nvolve randomzaton but are nstead confronted wth dfferent types of selecton processes. The selecton process nvolved mght be known as for regresson-dscontnuty desgns, where subjects get assgned to a treatment and control condton based on a contnuous assgnment varable and a strct cutoff (Lee & Lemeux, 29); Or, t mght be unknown as t s the case wth non-equvalent control group desgns where subjects select themselves or are assgned by admnstrators or thrd persons nto a treatment or control condton (Rosenbaum, 22, 29). Studes nvolvng nonequvalent control groups or nstrumental varables are also referred to as observatonal studes. In observatonal studes, the problem assocated wth dfferental selecton nto treatment and control condtons s that unadjusted treatment effects are very lkely based. For nstance, f unemployed persons most promsng for gettng a job n the near future are assgned nto a labor market program ther average employment rate after program partcpaton s very lkely hgher than the average employment rate of the non-partcpatng unemployed persons not necessarly because of the treatment effect but because of the better ntal poston partcpants were n (ths effect s frequently called creamng ). In such a stuaton we can estmate unbased treatment effects only f we are able to adequately model the selecton procedure (e.g., Heckman, 1974, 1979), statstcally control for observed selecton dfferences (e.g., Cochran & Rubn, 1973; Rubn, 26), or dentfy a relable source of exogenous varaton (.e., an nstrument; see Angrst, Imbens & Rubn, 1996) otherwse bas due to dfferental selecton remans. The assumptons requred for an unbased estmaton of causal treatment effects are well known and formulated n statstcal theores about causal nference (e.g., Angrst & Pschke, 29; Rosenbaum, 22; Rubn, 26; Steyer, 25; Steyer et al., 2a, 2b) but also n structural causal modelng approaches (e.g., Heckman, 25; Pearl, 29). However, the crucal queston n appled research s whether these assumptons requred for an unbased estmaton of the treatment effect are actually met for an observatonal dataset n hand. If the assumptons are not met the estmated treatment effect s very lkely based and the causal clams drawn from the observatonal study mght be nvald. Whle some of the assumptons nvolved n a causal nference from observatonal data are testable lke assumptons about the statstcal model (e.g., normalty or homoscedastcty of the error term) many desgn assumptons lke strong gnorablty or exogenety are not. Unfortunately, effect estmates are frequently much more senstve to volatons of untestable assumptons than testable ones. Thus, volatons of untestable assumptons are especally a potental threat to the valdty of causal nferences, unless we can convncngly rule them out by carefully chosen desgn elements lke non-equvalent outcome measures or multple comparson groups (for a dscusson of desgn elements see Shadsh, Cook & Campbell, 22). In ths paper we exclusvely focus on propensty score (PS) methods for removng selecton bas from observatonal studes. We nvestgate emprcal evdence about whether PS methods actually work n practce and under whch condtons they lkely succeed or fal n reducng selecton bas. Accordng to theory, PS methods succeed n removng selecton bas from observatonal data f all confoundng covarates are relably measured, the propensty score model s correctly specfed, and an approprated PS technque s chosen. But do these condtons hold n practce? And are some of these condtons more mportant than others? Usng results from wthn-study comparsons, whch 2

5 compare effects estmates from PS analyses to comparable benchmark estmates from randomzed experments, the prmary objectve of ths paper s to present emprcal evdence wth regard to the relatve mportance of the selecton of covarates, ther relable measurement, and the choce of a specfc PS method. A secondary objectve s to dscuss emprcal condtons, lke the avalablty of pretest measures of the outcome, for an (almost) complete reducton of selecton bas. Usng metaanalytcal results, we also address the queston whether PS methods do better n practce than covarance adjustments va standard regresson analyss. The paper s organzed as follows. The next secton ntroduces the potental outcomes notaton of the Rubn Causal Model and presents the theoretcal assumptons underlyng the estmaton of causal treatment effects usng PS or standard regresson technques. The thrd secton brefly outlnes the most frequently used types of PS methods. The fourth secton dscusses the desgn and ratonal of wthn-study comparsons for evaluatng PS methods success or falure n estmatng unbased causal treatment effects n practce. The revew secton then present results from dfferent types of wthn-study comparsons and meta-analyses. Fnally, the last secton summarzes the fndngs and concludes wth a dscusson of the generalzablty of the fndngs. 2. The Rubn Causal Model & Defnton of the Treatment Effect The Rubn Causal Model (Rubn, 1974, 1979; Holland, 1986) wth ts potental outcomes notaton provdes a convenent way for formulatng the crtcal assumptons underlyng causal nferences from randomzed experments or observatonal studes (alternatve formalzatons of causal models can be found n Pearl, 29, or Steyer et al., 2a, 2b). In ts smplest formulaton wth one treatment (Z = 1) and one control condton (Z = ), the Rubn Causal Model postulates two potental outcomes for each subject = 1,..., N: a potental control outcome Y whch s observed f subject receves the control condton (Z = ), and a potental treatment outcome Y 1 whch s observed f subject receves the treatment condton (Z = 1). Y 1 and Y are called potental outcomes because these outcomes are unknown but fxed pror to treatment selecton. Whch of the two potental outcomes then actually realzes depends on subject s selecton or assgnment to the treatment or control condton. Gven the par of potental outcomes ( Y, Y 1 ), dfferent causal estmands of nterest can be defned. For llustratve purposes we only defne the average treatment effect for the overall target populaton (ATE) and do not dscuss condtonal treatment effects lke the average treatment effect for the treated (ATT). Usng the potental outcomes notaton, we can defne the average treatment effect (ATE) as the expected dfference between potental treatment and control outcomes: τ = E( Y 1 Y ) = E( Y 1 ) E( Y ) However, snce we rarely observe both potental outcomes ( Y, Y 1 ) smultaneously for each sngle subject we cannot drectly estmate the average treatment effect. Ths s known a as the fundamental problem of causal nference (Holland, 1986). Dependng on the treatment status Z, the observed outcome s ether the potental control or potental treatment outcome: 1 Y = Y ( 1 Z ) + Y Z (Rubn, 1974). Snce we observe the potental treatment outcomes exclusvely for the treatment subjects we can only nfer the condtonal expectaton of treatment outcomes, 1 E ( Y Z = 1) = E( Y Z = 1), nstead of the uncondtonal expectaton E ( Y 1 ) whch was used n defnng the treatment effect n equaton (1). Analogously, we only obtan the condtonal expectaton of control outcomes, E ( Y Z = ) = E( Y Z = ), for the control group. Snce these condtonal expectatons dffer n general from the uncondtonal expectatons, ( 1 E ) and E ( ), Y (1) Y 3

6 the prma face effect, that s the unadjusted dfference between the condtonal expectatons, s n general not equal to the causal estmand defned n (1): 1 1 E ( Y Z = 1) E( Y Z = ) E( Y ) E( Y ) =τ. (2) The prma face effect s typcally based due to dfferental selecton of subjects nto the treatment and control condton. An unbased treatment effect can be estmated only f we can demonstrate that the selecton mechansm (or treatment assgnment) that gave rse to the observed data s gnorable (Rosenbaum & Rubn, 1983). Whether a selecton mechansm can be consdered as gnorable or not manly depends on the desgn of the study under nvestgaton. In a randomzed experment, where subjects are randomly assgned nto a treatment and control condton, the assgnment mechansm s gnorable. Ths s so because randomzaton balances the treatment and control group s characterstcs the two groups are statstcally equvalent and, thus, comparable. More specfcally, randomzaton also balances the potental treatment and control outcomes between the treatment and control group. That s, potental 1 outcomes are ndependent of treatment assgnment Z, whch we can denoted as ( Y, Y ) Z. Under ths ndependence assumpton (.e., gnorablty of treatment assgnment) t s easy to show that the condtonal expectatons of the potental outcomes are dentcal to the uncondtonal expectatons: 1 1 E ( Y Z = 1) = E( Y Z = 1) = E( Y ) and E ( Y Z = ) = E( Y Z = 1) = E( Y ). Consequently, the average treatment effect (ATE) can be obtaned from the estmable condtonal expectatons, τ = E( Y Z = 1) E( Y Z = ), whch s dentcal to ATE n equaton (1). However, snce randomzaton s frequently not possble n practce we need to rely on observatonal data whch typcally exhbt selecton bas due to dfferental selecton nto the treatment and control groups. Consequently, the prma face effect (the unadjusted mean dfference between treatment and control group subjects) s n general based. Fortunately, we can show that the average treatment effect can be estmated wthout bas f we are able to adequately measure the selecton process such that t s gnorable. Ths condton s formulated by the crucal strong gnorablty assumpton whch s also called condtonal ndependence or unconfoundedness assumpton, or selecton on observables (Imbens, 24; Rosenbaum & Rubn, 1983; see Steyer et al., 2a and 2b, for a dscusson of dfferent and also weaker assumptons). The strong gnorablty assumptons states the followng: If we have a set of p observed covarates X = ( X1, K, X p)' such that potental outcomes ( Y, Y 1 ) are ndependent of treatment selecton condtonal on X, that s, 1 ( Y, Y ) Z X, (3) and f treatment probabltes are strctly between zero and one, < P ( Z = 1 X) < 1, then the selecton mechansm s sad to be strongly gnorable (Rosenbaum & Rubn, 1983). If the strong gnorablty assumpton holds we can show that the average treatment effect (ATE) s the dfference n the observable condtonal expectatons of treatment and control group s outcomes: τ = EX { E( Y Z = 1, X)} EX{ E( Y Z =, X)}, whch s equvalent to the treatment effect defned n equaton (1). The nner expectatons refer to the expected potental outcomes for a gven group and set of values X, the outer expectatons average the expected potental outcomes across the dstrbuton of covarates X. From a practcal pont of vew, the strong gnorablty assumpton requres the relable measurement of all constructs X that are smultaneously assocated wth both treatment status Z and potental outcomes ( Y, Y 1 ). Gven that the strong gnorablty assumpton s met, we can estmate unbased treatment effects usng multvarate matchng, multvarate stratfcaton, or covarance adjustments va standard regresson analyss (Cochran & Rubn, 1973). Note that unbased effects only result f the method 4

7 specfc assumptons are also met (e.g., the correct specfcaton of the functonal form n a regresson analyss). The dsadvantage of these methods, partcularly of multvarate matchng and stratfcaton, s that they may be rather neffcent when the number of covarates s large: The more covarates the less lkely t s to fnd treatment and control cases that are dentcal (or very smlar) on all observed covarates. In order to avod ths curse of dmensonalty Rosenbaum and Rubn (1983) suggested to use a composte score created from the observed covarates: the propensty score (PS). The propensty score s the condtonal probablty of belongng to the treatment group gven the observed covarates X,.e., e(x) = P(Z = 1 X). Rosenbaum & Rubn (1983) also proved that f the selecton mechansm s gnorable gven X (see equaton 3), then t s also gnorable gven the propensty score e(x): 1 ( Y, Y ) Z e( X). (4) Ths means that nstead of drectly usng a set of observed covarates X t s suffcent to use the correspondng unvarate propensty score for establshng a strongly gnorable selecton mechansm. 3. Propensty Score Estmators If the strong gnorablty assumpton holds statstcal methods that approprately control for the confoundng covarates are at least potentally able to remove all the bas. In the followng secton we brefly outlne four dfferent types of propensty score estmators for removng selecton bas. All PS technques requre frst the estmaton of the unknown propensty score e(x). Gven the observed data, we may estmate the PS wth bnomal regresson models, lke logt or probt models, but we can also use nonparametrc approaches lke boosted regresson (Berk, 28; McCaffrey, Rdgeway & Morral, 24). However, there s not yet enough evdence that nonparametrc methods do on average better than parametrc bnomal regresson models (Lee, Lessler, & Stuart, 29; Setoguch, Schneewess, Brookhart, Glynn, & Cook, 28). In specfyng a PS model, the goal s to balance pretreatment group dfferences on observed covarates, where balance refers to the equvalence of the jont covarate dstrbutons n the treatment and control group. Achevng balance n observed covarates va PS adjustments bascally tres to mmc the balance as establshed va random assgnment randomzaton guarantees that the treatment and control groups are statstcally equvalent wth regard to the observed but also unobserved covarates dstrbuton. However, n strong contrast to random assgnment, a PS can only be modeled wth respect to balance n observed covarates there s no guarantee that a PS also balances unobserved covarates. Gven a set of covarates, we can assess the balancng property of the estmated propensty scores wth descrptve statstcs lke the standardzed mean dfference n covarates (e.g., Rubn, 21) or sgnfcance tests lke the t-test whch tests the mean dfferences between the treatment and control group (e.g., Ho, Ima, Kng & Stuart, 27, wth a crtcal dscusson of balancng tests). Once we estmated a balancng propensty score we may used t n bascally four dfferent approaches for removng selecton bas from the outcome of nterest: () PS matchng: Indvdual treatment and control subjects are matched on the bass of the estmated PS, that s, for each treatment subject we try to fnd a control subject wth an dentcal or at least very smlar PS; () PS stratfcaton: Treatment and control subjects are stratfed on the estmated PS nto a rather small number of homogeneous subclasses (frequently between 5 and 1 strata are used); () Inversepropensty weghtng: In analogy to Horvtz and Thompson s nverse probablty weghtng (1952), each subject gets a weght derved from the estmated PS whch can then be used n a weghted least squares regresson, for nstance; (v) Regresson estmaton usng PS related predctors: Dummy varables of PS strata or the cubc polynomal of the PS logt, for nstance, mght be used as predctors n an outcome regresson. Moreover, we may combne all these four basc propensty 5

8 score technques wth an addtonal covarance adjustment wthn the regresson framework. For nstance we can combne nverse-propensty weghtng and covarance adjustment n a weghted least squares regresson; Or, run a regular outcome regresson on matched data. The underlyng ratonal for such mxed methods whch are also called doubly robust methods s that the correct specfcaton ether of the PS model or the outcome regresson model s suffcent for a consstent estmaton of the causal treatment effect (Robns & Rotntzy, 1995; Rubn, 1973; Rubn; 1979). However, t s mportant to note that f both models are msspecfed whch s almost always true n practce mxed methods mght actually perform worse than smple PS analyses wthout an addtonal covarance adjustment (Kang & Schafer, 27). Though ths s only a very bref descrpton of PS technques, t s suffcent for the purpose of ths paper. We refer readers who are nterested n a more detaled descrpton and dscusson of dfferent propensty score technques, ncludng mxed methods approaches, to Imbens (24), Lunceford & Davdan (24), Morgan & Wnshp (27), Rubn (26), Schafer & Kang (28), or Stener & D. Cook (n prnt). The only addtonal topc to be brefly addressed here are the dfferences n fnte and large sample propertes of dfferent PS methods. For nstance, PS stratfcaton removes on average only 9% of the selecton bas f fve strata based on quntles are used, that s, 1% of the ntal selecton bas remans (Cochran, 1968; Rosenbaum & Rubn, 1983); PS matchng and nverse-propensty matchng suffer from fnte sample bas due to the lack of exact matches or the lack of suffcent overlap, respectvely (for a dscusson see Busso, DNardo & McCrary, 29); Or, as a fnal example, regresson estmaton usng PS-related predctors reles on the correct specfcaton of the functonal form between the outcome and the PS (or ts logt). In addton to the estmators (un)basedness and consstency propertes they also dffer wth regard to ther effcency whch we do not dscussng here (e.g., Busso, DNardo & McCrary, 29; Imbens, 24; Schafer & Kang, 28). Though some estmators have better large sample propertes than others there s no unque best estmator for fnte samples. Thus, the queston of nterest addressed n ths paper s whether n practce some PS technques remove regularly more bas than others. An addtonal queston s whether PS methods outperform classcal regresson analyss. We nvestgate both questons by revewng correspondng fndngs from wthn-study comparsons and metaanalyses. 4. Strateges for Emprcally Evaluatng Propensty Score Technques As dscussed above, the success of propensty score technques n removng selecton bas depends on testable and untestable assumptons. Whle testable assumptons can be drectly probed wth the data n hand, untestable assumptons cannot. In an emprcal evaluaton of PS technques we are prmarly nterested n the effect of volatng untestable assumptons because these are the most crucal ones but we are also nterested n the relatve mportance of bases due to the choce of a specfc method (e.g., fnte sample bas). In order to nvestgate the effect of volated assumptons we may run smulaton studes where the data generatng model ncludng the true treatment effect s under the nvestgator s control. Though such smulaton studes are very nstructve wth respect to a method s senstvty to devatons from requred assumptons, they are not very realstc snce the complexty of the smulated data generatng mechansm rarely matches up wth the complexty of real world processes. Moreover, smulaton studes cannot ndcate whether assumptons are lkely to be met n practce or not. For ths reason, an emprcal evaluaton of PS technques or other quas-expermental methods requres a real world settng wth real subjects, selecton processes, treatments, covarates, and outcomes. However, the problem wth such settngs s that we rarely know the true treatment effect, thus, we lack a benchmark to whch a PS-adjusted treatment effect can be compared. So, where can we get a relable causal benchmark from? From a randomzed experment whch s frequently consdered as the best desgn for a vald causal nference, gven ts relable 6

9 mplementaton, of course. In an deal settng, the randomzed experment and the quas-experment are conducted wthn one sngle study such that all thrd-varable confounds lke dfferent target populatons or dfferences n treatment mplementaton and measurement can be ruled out. We call such studes that compare quas-expermental methods to a randomzed experment wthn-study comparsons. As Cook, Shadsh & Wong (28) dscuss, the qualty and valdty of a wthn-study comparson depends on several condtons: () The randomzed experment whch provdes the causal benchmark for the comparson need to be of hgh qualty otherwse t cannot serve as a relable benchmark for evaluatng a quas-expermental method s performance. () The mplementaton of the quas-experment has to be of hgh qualty, too. There s no pont n comparng a poorly mplemented quas-experment to a perfectly mplemented randomzed experment. () The quasexperment and the randomzed experment need to be comparable. Ths means that the two experments should only dffer n ther selecton process (self- or thrd-person selecton vs. random assgnment) but everythng else s held constant such that thrd-varable confounds can be ruled out. (v) Both studes need to estmate the same causal quantty. If the randomzed experment estmates an ntent-to-treat effect (ITT) but the quas-experment an average treatment effect (ATE) a comparson of the effect estmates s useless. (v) The analysts of the quas-experment should be blnded from the randomzed experment s results. Blndng avods that the PS analysts try to specfy ther model wth an eye on the randomzed benchmark to be replcated. (v) The crtera for comparng the quas-experments and randomzed experments result need to be clearly defned. That s, when do we consder the results to be statstcally equvalent? If these sx crtera are only poorly met vald conclusons about a quas-experments performance n removng selecton bas can hardly be made. The hstory of wthn-study comparsons reveals that ther qualty has consderably ncreased over the last decades (see Cook, Shadsh & Wong, 28). The frst wthn-study comparsons were conducted n the tradton of LaLonde (1986) who had a randomzed treatment and control group and a non-equvalent comparson group. In ths desgn, the outcome mean dfference between the randomzed treatment group and the randomzed control group yelds the causal benchmark, whereas the mean dfference between the randomzed treatment group and the adjusted non-equvalent control group yelds the quas-expermental estmate. We call such studes three-arm wthn-study comparsons snce they employ three groups n ther comparson (Fgure 1). In contrast, four-arm wthn-study comparsons are based on four groups: a randomzed treatment and control group, and a non-equvalent treatment and comparson group from an observatonal study (Fgure 2). Thus, four-arm wthn-study comparsons consst of two separate studes wth ther own treatment group. In order to guarantee comparablty of the randomzed experment and the observatonal study n a four-arm desgn another randomzaton step s requred: partcpatng subjects need to be randomzed nto the randomzed experment and the observatonal study n order to ensure that the underlyng target populatons are exactly the same. In comparson to three-arm desgns, four-arm wthn-study comparsons have the advantage of holdng everythng constant except for the assgnment mechansm. In partcular, thrd-varable confounds lke local dfferences n target populatons or dfferences n nstrumentaton (measurement of covarates and outcomes) can be ruled out n a four-arm desgn. 5. Revew of Wthn-Study Comparsons and Meta-Analyses 5.1. Four-Arm Wthn-Study Comparsons Gven the hgher valdty of two recent four-arm wthn-study comparsons by Shadsh, Clark & Stener (28) and Pohl et al. (29) we descrbe these two studes n more detal and provde addtonal evdence from two revews of wthn-study comparsons (Glazerman, Levy & Myers, 23, Cook, Shadsh & Wong, 28). The two four-arm wthn-study comparsons have also the advantage that they where reanalyzed especally wth respect to the relatve mportance of covarate 7

10 selecton, ther relable measurement and the choce of a specfc analytc method (Stener et al., 21, 211; Cook & Stener, 21; Cook, Stener & Pohl, 29). Shadsh et al. (28) used a four-arm desgn n order to nvestgate whether PS methods can reproduce the results from a randomzed experment. Accordng to Fgure 2, they frst randomly assgned 445 undergraduate psychology students to a randomzed experment (N = 235) or a quasexperment wth self-selecton (N = 21). In the randomzed arm, students were randomly assgned to a mathematcs or vocabulary tranng whle students assgned to the quas-experment were allowed to choose between one of the two tranngs. Not surprsngly, consderably more students choose the vocabulary (N = 131) n order to avod the mathematcs tranng (N = 79). The short tranngs conssted ether of learnng transformng exponental equatons or learnng the meanng of specal vocables. In estmatng the mathematcs treatment effect on the mathematcs outcome the mathematcs group was the treatment group and the vocabulary group served as a comparsons group and vce versa for the effect of vocabulary tranng on the vocabulary outcome. In order to control for the selecton bas n the quas-experment Shadsh et al. carefully measured 23 constructs based on 156 questonnare tems. The 23 constructs belong to fve broader construct domans: demographcs, proxy-pretests, pror academc achevement, topc preference, and psychologcal predsposton. The proxy-pretest doman ncluded pretests of the mathematcs and vocabulary outcome. These pretests are consdered as proxy-pretests because they were measured on a dfferent content and scale. The topc preference doman, whch later turned out to contan the most relevant covarates for explanng the selecton process and removng selecton bas (Stener et al., 21), conssted of constructs representng students lkng and preference of mathematcs and vocabulary but also students math anxety. The pror academc achevement doman covered hgh school and college scores. The bg fve personalty factors and the Beck depresson scale formed the psychologcal predsposton doman. Gven the 23 constructs Shadsh et al. measured, all the propensty score methods they appled to the quas-expermental data yelded a treatment effect comparable to the one obtaned from the randomzed experment. They also found that consderable selecton bas would have remaned f demographcal covarates would have been the sole measures avalable. In a re-analyses of the Shadsh et al. data, Stener et al. (21) demonstrated the mportance of selectng a set of covarates that establshes strong gnorablty. For both tranngs and outcomes (mathematcs and vocabulary), they showed that proxy-pretests and drect measures of the selecton process (.e., constructs from the topc preference doman) are very effectve n reducng selecton bas whle the other domans were not (demographcs, pror academc achevement, psychologcal predsposton). Indeed, two sngle constructs (out of 23) would have been suffcent for removng almost all the selecton bas. These constructs ether drectly determned the selecton mechansm (lkng mathematcs or preferrng math over lterature) or the outcome of nterest (proxy-pretests n math and vocabulary). If one would have known these covarates n advance the measurement of two constructs would have been suffcent for removng almost all the selecton bas. However, n practce we rarely know for sure whch covarates are the crucal ones. Thus, Stener et al. (21) also nvestgated how successful bas reducton would have been f the two most effectve constructs ndentfed for each of the two outcomes would not have been measured. It turned out that the remanng 21 covarates together (wthout the two most effectve ones) would have also removed almost all of the selecton bas. Ths suggests that measurng a broad set of covarates that covers dfferent construct domans wth multple measures s a potental strategy for establshng a strongly gnorable selecton mechansm. Such a strategy can be pursued especally when no relable theores or knowledge about the selecton process and the outcome model are avalable. However, note that even havng a multtude of construct domans and multple measures wthn domans does not necessarly guarantee that all or most of the selecton bas s removed. The results of Stener et al. (21) also ndcate that the selecton of covarates s much more mportant than the choce of a specfc PS method. The authors analyzed the quas-expermental 8

11 data wth three PS technques (PS stratfcaton, nverse-propensty weghtng, PS regresson estmaton) and a standard regresson approach (OLS wth all 23 covarates) and found that none of the methods they compared performed unformly best or worst. Moreover, the varaton n bas reducton due to the analytc method chosen was much less than the varaton nduced by selectng dfferent sets of covarates. Ther results clearly show that the selecton of covarates s much more mportant for an unbased estmaton of the treatment effect than the choce of an analytc method (gven ts competent mplementaton, of course). In a second study, Stener et al. (211) nvestgated the effect of unrelable covarate measurement on bas reducton usng PS methods. For ths purpose, they set up a smulaton study based on the Shadsh et al. data assumng that all 23 covarates where measured wthout bas (relablty ρ = 1) and that all relably measured covarates together remove all the selecton bas. Then they smulated unrelablty n covarate measurements by systematcally addng measurement error to each covarate (except for demographcal covarates) such the ndvdual covarate relabltes where stepwse reduced by.1 decrements to ρ {.9,.8,.7,.6,.5}. The results of the smulaton study can be summarzed as follows: Frst, f selecton and the outcome are determned by latent constructs then measurement error n covarates attenuates a covarates potental to remove bas. Second, the relable measurement of effectve constructs (.e., constructs that have a hgh potental to remove selecton bas) s more mportant than the relable measurement of neffectve constructs (whch have no or only a small potental of removng selecton bas). There s no need for relably measurng constructs that do not reduce any bas even f perfectly measured. Thrd, multple measures from dfferent construct domans are able to partally compensate for each other s unrelable measurement. Thus, havng multple measures reduces the rsk of remanng selecton bas not only due to unobserved constructs but also due to mperfect measurements (whch can be seen as partally unobserved constructs). The smulaton of Stener et al. (211) also allows an assessment of the relatve mportance of covarate selecton, ther relable measurement, and the choce of an analytc method snce they run the smulatons wth three dfferent PS methods (PS stratfcaton, nverse-propensty weghtng, PS regresson estmaton) and a standard regresson analyss. Not surprsngly, measurng the constructs that are smultaneously related to both treatment selecton and potental outcomes s more mportant than ther relable measurement. In partcular, unrelably measured effectve constructs typcally remove more bas than perfectly measured neffectve constructs. Then, the relable measurement of constructs s more mportant than the choce of a specfc analytc method. Whle measurement error systematcally attenuates a covarates potental to remove selecton bas, the choce of a specfc analytc method dd not systematcally affect bas reducton. No PS method performed unformly better or worse than all other methods (PS and standard regresson). Gven that all these results rely on a sngle data set, though wth two ndependent treatments and outcomes, they cannot drectly be generalzed to other settngs. However, Pohl et al. (29) set out to replcate the Shadsh et al. study n Berln (Germany). They bascally used the same desgn but nstead of the vocabulary tranng they had an Englsh tranng. In the German study 22 students were randomly assgned nto a randomzed experment or a quas-experment wth self-selecton. Those n the randomzed experment then got randomly assgned to the mathematcs or Englsh tranng whle students n the quas-experment chose ther preferred tranng. It s nterestng to note that the selecton mechansm n ther study dffered from the Shadsh et al. (28). Instead of avodng the mathematcs tranng, students actvely chose the tranng accordng to ther needs. Students tended to chose the topc where they thought they could need more tranng n order to mprove ther sklls. Moreover, t turned out that the selecton process dd not result n a selectonbased mathematcs outcome the unadjusted treatment effect of the quas-experment was almost the same as the treatment effect from the randomzed experment. Thus, there was bascally no bas to be removed. Nonetheless, Pohl et al. (29) showed that PS and standard regresson adjustments dd not nduce bas n the mathematcs treatment effect. For the vocabulary outcome, Pohl et al. 9

12 bascally replcated the fndngs by Shadsh et al. (28). Controllng for all covarates, ether n a PS or standard regresson approach, removed all the bas n the vocabulary outcome. The choce of method dd not play an mportant role but the selecton of covarates and ther relable measurement dd, as demonstrated n Cook, Stener & Pohl (21). Thus, the German wthn-study comparson replcated the fndngs of the analyses conducted wth the Shadsh et al. data: The selecton of constructs s the most mportant factor for a successful removal of selecton bas, the relable measurement of constructs s the next most mportant factor, and the choce of a specfc PS method or regresson analyss s of least mportance, gven ts competent mplementaton. Ths does, of course, not mean that the choce of a partcular method s unmportant. For a gven dataset, the choce of an analytc method mght actually matter one method mght ndcate a sgnfcant treatment effect, whle another method mght suggest an nsgnfcant effect Revews of Three-Arm Wthn-Study Comparsons Though the results derved from the two four-arm wthn-study comparsons gve valuable nsghts regardng the relatve mportance of covarate selecton, relable measurement, and choce of method, they are not drectly generalzable to more realstc settngs lke evaluatons n the context of labor market programs whch typcally face more complex selecton mechansms and outcome generatng models. However, three-arm wthn-study comparsons n the tradton of LaLonde (1986) offer some further evdence for the fndngs derved from the four-arm studes. Glazerman, Levy & Myers (23) revewed 12 wthn-study comparsons on labor market programs and Cook, Shadsh & Wong (28) 12 studes from the other socal scences. The wthn-study comparsons that Glazerman et al. (23) nvestgated are all on job tranng programs and ther causal effect on subsequent earnngs. Not all of these comparsons nvolved PS technques they also used multvarate matchng or standard regresson approaches. It s nterestng that none of the quas-expermental methods was able to reproduce the treatment effect obtaned from the correspondng randomzed experments. However, the averages of the quasexpermental results and the randomzed experments results were rather smlar. Though the quasexpermental estmates dffered from ther expermental benchmark estmates, Glazerman et al. (23) state that statstcal adjustments, n general, reduced bas, but the bas reducton assocated wth the most common methods regresson, propensty score matchng, or other forms of matchng dd not dffer substantally. Thus, also n these 12 revewed three-arm wthn-study comparsons the choce of a specfc analytc method was of least mportance. Wth regard to the mportance of covarate selecton for removng selecton bas, the author concluded that bas was lower when researchers used measures of preprogram earnngs and other detaled background measures to control for ndvdual dfferences (p. 86) and that baselne measures of the outcome are mportant for bas reducton (p. 8). Other than Glazerman et al. (23), Cook et al. (28) revewed a rather heterogeneous set of wthn-study comparsons comng from dfferent felds of socal scences. Ther revew of wthnstudy comparsons not only ncluded non-equvalent control group desgns but also other quasexpermental desgns lke regresson dscontnuty desgns. Cook at al. found that eght of the comparsons produced observatonal study results that are reasonably close to those of ther yoked experment, and two obtaned a close correspondence n some analyses but not others. Only two studes clamed dfferent fndngs n the experment and observatonal study, each nvolvng a partcularly weak observatonal study. Taken as a whole, then, the strong but stll mperfect correspondence n causal fndngs reported here contradcts the monolthc pessmsm emergng from past revews of the wthn-study comparson lterature (p. 745). The authors concluded that completely knowng and measurng the selecton procedure s one way for establshng an gnorable selecton mechansm. It s mportant to note that ths does not only refer to regresson dscontnuty desgns but also to some propensty score studes (e.g., Daz & Handa, 26). Cook et al. also 1

13 ponted out that havng geographcally close matches of ntact groups (e.g., schools or labor markets), whch mnmzes the ntal bas before ndvdual subjects get matched, helps n reducng selecton bas. Further, statstcally adjusted treatment effects from observatonal studes are less successful (1) when comparson cases are selected from natonal data sets that dffer from the nterventon group n populaton, settngs, and measurement; (2) when sample szes n the control or comparson condtons are modest; and (3) when only demographc varables are avalable as covarates (p. 746). Cook et al. also menton the mportance of pretest measures of the outcome n combnaton wth locally matched ntact groups: We can also trust estmates from observatonal studes that match ntact treatment and comparson groups on [ ] pretest measures of outcome (p. 745). Though, the authors dd not drectly nvestgate whether the choce of a specfc analytc method matters, there s no ndcaton that PS methods dd better or worse than standard regresson analyses (Cook & Stener, 21) Meta-Analyss Comparng PS Methods wth Standard Regresson Methods The fndng that PS and regresson methods do not systematcally dffer s also supported by two meta-analyses n epdemology. These meta-analyses nvestgated studes that estmated the treatment effect of nterest usng both PS and regresson methods. Shah et al. (25) compared test results of 78 pars of PS and regresson estmates from 43 studes and found that only eght out of 78 (1%) dffered sgnfcantly. Moreover, they nvestgated 54 dfferences n effect szes (.e., dfferences n the natural logarthm of odds or hazard ratos) and found that, on average, propensty score methods gave an odds or hazard rato approxmately 6.4% closer to unty than tradtonal regresson methods. Shah et al. concluded that the two methods usually dd not dffer n the strength or statstcal sgnfcance of assocatons between exposures and outcomes (p. 552). The second study by Stürmer et al. (26) meta-analyzed 69 studes n epdemology and found that only 13% of all PS estmates had an effect estmate that dffered by more than 2% from conventonal regresson estmates. Lke Shah et al. they found no emprcal evdence [ ] that PS analyses controlled confoundng more effectvely than dd conventonal outcome modelng (p. 44). Though both meta-analyses do not ndcate sgnfcant dfferences between PS methods and regresson analyses on average, one has to be cautous n concludng that they do not dffer n general. It mght be that PS methods do better or worse under certan condtons, e.g., dependng on the treatment and control groups sample sze, the ntal smlarty of the groups, or the presumed complexty of the selecton and outcome model. Nether the meta-analyses nor the wthn-study comparson dscussed n ths artcle dd systematcally nvestgate condtons under whch one method could outperform the other. 6. Dscusson In revewng two four-arm wthn-study comparsons, two revews of 24 three-arm wthn-study comparsons, and two meta-analyses we demonstrated that, n practce, the selecton of covarates s the most mportant factor for removng selecton bas from observatonal studes. If we fal n measurng constructs that are smultaneously related to treatment selecton and potental outcomes, gven the other measured covarates, the strong gnorablty assumpton wll not hold and estmates wll be based. The second most mportant factor for a successful bas reducton s the relable measurement of constructs. If confoundng constructs are unrelably measured bas reducton wll be attenuated. It s mportant to note that relablty only matters f latent constructs nstead of observed covarates determne the selecton mechansm and the potental outcomes. Fnally, the choce of an analytc method turned out to be of least mportance n removng selecton bas. The wthn-study comparsons dd not ndcate any systematc dfferences between dfferent types of PS 11

14 methods nor dd they reveal dfferences between PS and standard regresson methods. The latter fndng was also supported by two meta-analyses from epdemology. Though the emprcal evdence gathered so far does not suggest that PS methods are superor to standard regresson approaches, they have some desgn advantages (Rubn, 27, 28). Frst, PS technques enable the nvestgator to balance pretreatment group dfferences wthout usng the outcome. Ths allows the nvestgator to be blnded from the outcome measure untl he freezes the fnal PS. The separaton of PS estmaton and outcome analyss also helps n preservng the type-1 error rate for testng the treatment effect. Second, PS methods offer a natural way for checkng the heterogenety of the treatment and comparson group by comparng the groups dstrbuton of the PS. When the treatment and comparson group overlap only partally on the propensty score, PS methods typcally delete non-overlappng subjects and, thereby, avod relyng on extrapolatng treatment effects. Gven the crucal mportance of measurng a set of covarates that establshes a strongly gnorable treatment assgnment, the queston s whether we can dentfy types of covarates that are n general more successful n reducng selecton bas than other types. Ths revew of wthn-study comparsons ndcates that two types of covarates do frequently better than others. Frst, covarates that drectly ndex the selecton mechansm very lkely remove a consderable part f not all the selecton bas. In the case of admnstrator assgnment, measures on whch admnstrators base ther selecton of subjects need to be collected (e.g., Daz & Hand, 26). If subjects select themselves nto the treatment or control condton motvatonal measures for choosng or avodng the treatment mght successfully remove bas (e.g., Stener et al., 21). The second group of covarates conssts of pretest measures of the outcome. They frequently reduce a major part of the selecton bas because of ther presumably hgh correlaton wth the outcome of nterest. In partcular, t s hard to magne a selecton mechansm that strongly affects the potental outcomes but s only weakly related to the correspondng pretest measures. Wthn the group of pretest measures, we can further dstngush between two types: true pretest measures and proxy-pretest measures. Whle the former are measured on exactly the same content and scale, the latter typcally dffer n content but also n ther scale of measurement. In general, true pretest measures are preferable to proxy-pretest measures snce true pretests are more lkely hgher correlated wth the outcome than proxy-pretests. However, a methodologcal study by Hallberg, Stener & Cook (211) on the effect of retanng nstead of promotng poorly performng Kndergarteners demonstrates that proxy-pretest measures can do as well as true pretest measures. In ther study, two repeated measures ether of the true pretest (achevement score on the same content and scale as the outcome) or the proxy-pretest (teacher assessment) removed almost all selecton bas. That s, both the true or proxy-pretests were able to remove nearly as much bas on ther own as compared to the case where both types of pretests were combned wth the more than 2 avalable covarates. The study also demonstrates that even f the true and proxy-pretest measures would have not been avalable the rch set of more than 2 covarates would have removed almost the same amount of selecton bas as all covarates and pretest measures together. As already noted n the dscusson of the two four-arm wthn-study comparsons, ths fndng suggests that a potentally successful strategy for removng selecton bas from observatonal studes s to measure multple construct domans and multple constructs wthn domans. Nonetheless, an nformed measurement of selecton- and outcome-relevant constructs combned wth the addtonal measurement of a heterogeneous set of constructs s preferable to an unnformed and haphazard collecton of data. Despte the ncreasng number of wthn-study comparsons that emprcally evaluate the performance of PS methods n removng selecton bas one needs to be cautous n generalzng the fndngs dscussed above. The generalzablty of the results of the four-arm wthn-study comparsons s restrcted due to ther laboratory-lke settng (short treatments and not very complex selecton processes). The valdty of some three-arm wthn-study comparsons s threatened by thrd varable confounds lke the selecton of treatment and comparson groups from dfferent local 12