Experimental methods in population studies

Transcription

1 Expermental methods n populaton studes Duncan Thomas Duke Unversty Elzabeth Frankenberg Duke Unversty August 2014 Abstract Randomzed controlled trals (RTs) have been proftably used to dentfy causal effects n populaton research. However, the desgn and mplementaton of socal experments s not straghtforward and t s not clear that t s ether feasble or desrable to attempt to answer some questons n populaton usng only the so-called gold standard doubleblnd RT. It seems lkely that the ntegraton of the creatve use of theory wth the advantages of both RTs and non-expermental study desgns has the greatest hope of advancng scentfc knowledge about populaton behavors and processes. Prepared for the Internatonal Encyclopeda of the Socal and Behavoral Scences, 2 nd edton (Area 3).

2 1. Introducton A goal of populaton research s to solate causal effects of, for example, polces, programs, crcumstances and behavoral choces. For example, what s the causal effect of mproved access to famly plannng servces on the health and economc status of women? What s the mpact of marrage on health and well-beng? What s the mpact of mgraton on those who move and those who are left behnd? Answerng these types of questons wth observatonal studes s challengng because the counter-factual s typcally not observed: what would have happened f the mgrant had not moved s not known. It s dffcult to separate the mpact of the stmulus (the decson to mgrate) from other, unobserved factors that dstngush those who experence the stmulus (those who choose to mgrate) from those that do not (those who do not choose to mgrate). The promse of socal experments s that they provde a method for puttng asde these unobserved dfferences and, thereby, dentfy causal effects. In the deal randomzed controlled tral (RT), study subjects are randomly assgned to receve a stmulus (the treatment group) or a placebo (the control group). After the stmulus, the dfference between the average outcome of the treatment group and the average outcome of the control group provdes a measure of the causal effect of the stmulus on the outcome. Because the study subjects are randomly assgned to receve the stmulus, concerns about what drves selecton nto each group (or unobserved dfferences between the groups) are, by desgn, put asde. Implementaton of the deal RT s far from straghtforward n the best of crcumstances. Implementng socal experments to address mportant questons n the populaton scences s very challengng. To examne the mpact of mgraton, for example, one cannot smply move a randomly selected group of people to a new locaton and force them to reman there whle forcng those n the control group to stay put. It may be possble, however, to subsdze the costs of mgratng, broadly defned, or, n the case of nternatonal mgraton, adopt a polcy of randomly assgnng vsas to a subset of those who apply. Aspects of each of these desgns rase mportant ssues regardng the queston that s addressed and the nterpretaton of the estmates. The central pont s that socal experments have the potental to be very valuable n populaton research but, as s the case wth observatonal studes, they have strengths and weaknesses. Indeed, beyond dentfyng causal effects, t s mportant to understand the mechansms that underle behavoral choces. To ths end, t seems the greatest promse of RTs and socal experments les n ther fuller ntegraton wth theory and observatonal desgns to advance understandng of populaton processes, a poston advocated by Heckman (2010). 1

3 2. Hstorcal perspectve RTs have a long hstory that, accordng to Stgler (1992) dates back at least to Perce s work n psychology (Perce and Jastow, 1885). The statstcal foundatons were formalzed n the 1920s by Neyman (1923) and Fsher (1935) and the methods were used n the agrculture scences to evaluate the mpact of nputs and practces on output. RTs were later adopted n medcne to evaluate the mpact of treatments on patents and, t s often argued, that double-blnd RTs are the gold standard for evdence n that feld. Whereas randomzaton of agrcultural plots s relatvely easy, randomzaton wth human subjects s more complcated, partcularly when the stakes lfe or death, for example -- are hgh. Wthout queston, randomzed controlled socal experments are even more complcated to desgn and mplement. In the 1960s and 1970s, several large-scale, nfluental socal experments were felded to examne, for example, the mpact of provdng an ncome guarantee through a negatve ncome tax (Orcutt and Orcutt, 1968), of varyng the costs of health nsurance (n the Rand Health Insurance Experment, Newhouse et al, 1996) and of provdng job tranng to youth and unsklled adults (Bloom et al, 1997). In the populaton feld, studes n demographc survellance stes n Navrongo, Ghana, and Matlab, Bangladesh poneered the use of socal experments to nvestgate the mpact of mproved access to famly plannng and maternal and chld health programs on health and well-beng (Phllps, et al, 1988; Phllps, et al, 2006). In recent years, the adopton of RTs has exploded, partcularly n development economcs, led n large part by Banerjee and Duflo along wth ther colleagues, many of whom are assocated wth the Abdul Latf Jameel Poverty Acton Lab at MIT. (Duflo 2007; Duflo et al, 2008; Banerjee and Duflo, 2009, 2010, 2011.) Deaton (2009, 2010) has been a leadng crtc of the rse of RTs n the feld. Over the last 20 years, Heckman (1992, 2010) has played a poneerng role n explotng the advantages and addressng the dsadvantages of socal experments. 3. The deal RT We begn wth the deal or gold standard RT n whch study subjects are randomly assgned, at the ndvdual level, to a treatment or a control group, wth nether the study subjects nor nvestgators knowng who has been assgned to whch group (that s, the study s double-blnd). To lay the foundaton, consder the lnear model, T [1] 0 1 2

4 n whch T =1 f ndvdual s treated and 0 f not. The goal s to estmate 1 to measure the causal mpact of a stmulus or treatment, T, on the outcome of nterest,. Put another way, we are nterested n estmatng T 1 E[( ) T 1] [2] T where s the outcome when an ndvdual s treated and s the outcome when the ndvdual s not treated. For example, we may be nterested n measurng the causal mpact of a treatment that T mproved physcal health on hourly earnngs. In ths example s hourly earnngs of a person who receved the treatment (took ron supplements); s what the same person would have earned per hour n the absence of recevng the treatment. Obvously, the counterfactual, the outcome f the ndvdual who was treated had not been treated (denoted n [2]) s not observed. However, the dfference between the average earnngs of those who receved the health-mprovng nterventon and those who dd not can be observed and so we can estmate Rewrtng [3] we have ˆ T E[ T 1] E[ T 0] [3] 1 1 ˆ T E[( ) T 1] E[ T 1] E[ T 0] [4] ˆ E[ T 1] E[ T 0] [5] where the second and thrd terms n [5] capture the effect of selecton, that s the mpact on ˆ 1 of unobserved heterogenety,, correlated wth the treatment, T, n [1]. Returnng to our example, the second term s the average earnngs that the treatment group would have earned f they had not been treated, the counterfactual that cannot be observed. The thrd term s the average earnngs of those who were not treated. If the dfference between the second and thrd terms s zero then ET [ ] 0 and ˆ 1 s a consstent estmate of the causal mpact of T, on : plm ˆ n 1 1 [6] On one hand, f study partcpants who receved the treatment would have earned more n the absence of the treatment than partcpants n the control group, then E[ T 1] E[ T 0] > 0 and the treatment group would be postvely selected, relatve to the controls. In that case, ET [ ] 0and ˆ 1 would be an upward based estmate of the causal mpact of T on. Ths mght arse, for example, f subjects have to apply to partcpate n the treatment and those who apply are

5 those who would have earned more even n the absence of the treatment. On the other hand, f, relatve to controls, the study partcpants would have earned less n the absence of the treatment, then E[ T 1] E[ T 0] < 0 and the treatment group would be negatvely selected, ET [ ] 0and ˆ 1 wll be downward based. Ths mght arse f those wth a lower value of tme (and lower potental hourly earnngs) are also more lkely to apply for and partcpate n the treatment. Expermental studes are desgned so that the selecton term E[ T 1] E[ T 0] s zero so that ET [ ] 0and [6] s true. In a double-blnd randomzed controlled tral (RT), ths can be acheved by randomly assgnng subjects to ether the treatment or control group and then exposng only the treatment group to the stmulus and, after the treatment has had an mpact, comparng the outcomes (or changes n outcomes) of the treated and control subjects. In an RT that s desgned and mplemented well, the second and thrd terms n [5] wll be zero by constructon. The average treatment effect (ATE) s an unbased estmate of the causal effect of the treatment on the outcome, 4. Key features of the deal RT There are at least three features of the deal or gold standard RT model that are crtcal for estmated treatment effects to be consstent. Frst, study subjects assgned to the treatment should be exchangeable wth those assgned to the control group;.e., n terms of any characterstc that s related to the outcome of nterest, the treatments and controls are effectvely nterchangeable. In some studes, randomzaton s conducted after stratfyng on observable characterstcs (such as age, gender, race, educaton or locaton) and study subjects are randomzed wthn each subgroup. Analyses conducted on each sub-group satsfy the condtons necessary to nterpret the estmated effects as causal. The reverse s not true: f study subjects are randomly assgned to treatment and control wthout stratfyng by group, t s napproprate to assume that the selecton term n [5] s zero wthn groups. For example, say you are nterested n examnng the effect of a treatment on males and females separately but randomzaton was not conducted separately by gender. We cannot assume that the selecton term n [5] s zero for all the study subjects and also for males as well as for females, taken separately. That may be true, and f so, then we are lucky. However, snce t s not bult nto the study desgn, one cannot assume that t s true. Ths alsohghlghts that RTs lean heavly on the assumpton of lnearty n [1] snce the mean of the dfference between treatment and control outcomes s the same as the dfference between the mean treatment and mean control outcome. Estmaton of heterogenety of effects calls for addtonal assumptons that may not be correct and wll typcally need to be justfed by drawng on a 4

6 theory or at least a mechansm that les outsde the RT study desgn. (Deaton, 2014; Heckman and Smth, 1995). Second, attrton, whch s a key concern n every longtudnal study, has the potental to be partcularly perncous n an RT. If post-treatment follow-up s selected on benefts (or costs) to those n the treatment group relatve to the benefts (and costs) to those n the control group, the attrton wll potentally contamnate estmates of the treatment effect. Snce outcomes of subjects who are lost to follow-up are not measured, the effects of attrton are necessarly absorbed n unobserved heterogenety,, n [1] and t s not obvous that t s reasonable to assume ET [ ] 0. To address ths concern, some RT studes establsh that attrton s unrelated to treatment status and, n some cases, to observed baselne characterstcs. Whle such results are comfortng, they do not establsh that attrton s gnorable. For example, f attrton s related to the mpact of the treatment say those who do not beneft much from the treatment ext the study then t may be dffcult to detect ths sort of selecton n the absence of a post-treatment measure of the outcome of nterest. In the gold standard RT there s no place for attrton; achevng that n a populaton-based study s a tall order. The thrd crtcal feature of the double-blnd ndvdual RT s that outcomes of the control group must not be affected by the treatment to assure that the second and thrd terms n [5] are equal. Ths rules out, for example, any response by the controls to the fact that they were not selected for the treatment. In a double-blnd RT, nether the study subjects nor the study nvestgators know who s assgned to receve the treatment or who s assgned to the control group. In that case, nether the treatments nor the controls have the nformaton necessary to change ther behavor because they know they are (or are not) n the treatment group. The gold standard RT study s double-blnded because a smlar concern apples to study nvestgators who may treat those n each group dfferently, albet mplctly. Ths sort of behavor has been descrbed n some drug trals, even n double-blnded studes, as nvestgators have been able to nfer who s n the treatment group (by examnng bomarkers, for example) and have adjusted ther own behavor or treatment n order to help the subject (see, for example, Laurtsen, 1990). Studes have also documented nstances of non-random assgnment of subjects to the treatment or control group. For example, n studes that have been mplemented by people who have a stake n the outcome, or want the best for the subjects, and allocate those subjects whom they beleve wll beneft most from the treatment to that group. The double-blnd desgn s ntended to assure these types of contamnaton do not occur. 5

7 5. Implementaton of double-blnd RTs n populaton studes In an deal study desgn, the RT nvolves a comparson of the mpact of a treatment wth an dentcal-lookng placebo (whch may be an alternatve treatment) to assure the study s double-blnd. Ths approach has been used n populaton research. For example, a RT mplemented n a dstrct n entral Java, Indonesa, tested the hypothess that health has a causal effect on economc productvty by comparng changes n hourly earnngs of ron-defcent ndvduals who had been randomly assgned to receve ron supplements wth earnngs of controls who receved an dentcal-lookng sugar pll (Thomas et al 2014). However, many studes n the populaton scences seek to measure the mpact of treatments such as mproved access to famly plannng or other health-related servces, partcpaton n an educaton or tranng program, access to credt, fnancng or an ncome transfer program. In such cases, t s not possble to desgn a double-blnd RT because there s no placebo that can be gven to the controls that appears to be dentcal to the treatment. Ths rases concerns wth contamnaton of the estmated treatment effect because of behavoral responses by study partcpants to random assgnment or to the treatment tself. In a study of a tranng program, for example, f some of the subjects assgned to the treatment group choose to not partcpate n the program or drop out of the program before t s completed then t wll not be possble to measure the program effect on those subjects. On the other sde of the con, f some of the subjects who are assgned to the control group partcpate n another, related tranng program, then the estmated treatment effect wll be based. One of the motvatons for randomly assgnng some people to partcpate n a program s because the program s over-subscrbed. In ths stuaton, treatments are more lkely to partcpate n the program, but control subjects are lkely to be more motvated to seek out alternatves to the program treatment. Move to Opportunty (MTO) s a major randomzed housng moblty experment that was mplemented n the 1990s n whch 4,600 low ncome famles wth chldren lvng n publc housng n dsadvantaged urban neghborhoods were offered the chance to move to prvate-market housng n better neghborhoods. Famles were randomzed to receve a housng voucher that subsdzed prvate market rents (wth half of them carryng some constrants on where they could relocate) whle the controls receved no vouchers. Only about 50% of those who receved a voucher managed to relocate to a better neghborhood and a substantal fracton of the control subjects also relocated. (Ludwg et al, 2012.) 6

8 6. Intent to treat and treatment on the treated estmates One approach to addressng these concerns s to estmate the ntent to treat (ITT) effect whch measures the mpact of the program on all the subjects who were assgned to the treatment relatve to all the subjects who were assgned to control, rrespectve of whether they actually partcpated n the program. Ths contrasts wth the estmate of the effect of the treatment on the treated (TOT) whch ncludes only those subjects who complete the treatment and compares them wth the controls. learly, the TOT estmate cannot be assgned a causal nterpretaton. Studes have used random assgnment status as an nstrument for partcpaton n the treatment n an nstrumental varables framework to dentfy the causal mpact of the treatment on those who chose to partcpate. Ths does not provde evdence on the lkely mpact of the treatment on those who dd not choose to partcpate. (Heckman, 1992). 7. Level of randomzaton Another approach to addressng the concern that there may be behavoral responses by subjects to beng assgned to the treatment or control group s to eschew randomzaton at the ndvdual level and, nstead, randomze at a hgher level such as the communty, school, health center or mcro-credt agency. In ths case, f the study s able to exclude people who are not n the communty (or school etc.) at the tme of randomzaton from recevng the treatment, then t s possble to mtgate the mpact of behavoral responses to the treatment and control assgnments. In practce, ths can be dffcult and may rase ethcal ssues. In addton, such a desgn may have unntended mpacts on behavor. PROGRESA s a very large and ambtous condtonal cash transfer (T) that was mplemented n rural Mexco after a rgorous randomzed control tral. Parker, Rubalcava and Teruel (2008) provde a descrpton and revew results from the evaluaton. The evaluaton study selected 506 low ncome communtes and randomly assgned about two-thrds of those communtes to receve the T program n 1998 wth the other thrd of communtes beng told they would receve the treatment three years later (although the treatment was provded to households n these communtes 18 months after the evaluaton began). The T was means tested at the household level based on a baselne survey. Over and above a basc transfer that depended on household sze and composton, addtonal payments were made condtonal on chldren s school attendance and partcpaton by household members n actvtes at the health center. In some households, the payments were very large relatve to ther pre-transfer ncome. 7

9 Studes have establshed that households that were elgble to receve the transfers had hgher levels of spendng, mproved health and schoolng outcomes and nvested more n productve assets. However, there s also evdence that non co-resdent famly members lvng n the same communty as a benefcary household also benefted from the program, ndcatng sharng of resources among extended famles. Ths suggests that mpacts of the T on outcomes n benefcary households may be under-estmated. On the other hand, the program was desgned so that ndvduals and households could not move to a treatment communty and receve the treatment; elgblty for the treatment was determned at the baselne survey. Moreover, f a member of a treatment household moved away from the communty, the ndvdual dd not take the T wth hm or her and the baselne household payment was reduced accordngly. Mgraton out of poor rural communtes s wdespread n Mexco and t s a mechansm used to mprove the well-beng of both the mgrant and those left behnd. By desgn, the PROGRESA evaluaton provdes a dsncentve for young adults n benefcary households to move away n search of a better lfe. The same dsncentve does not apply to young adults n households n control communtes. Ths further complcates nterpretaton of the estmated mpact of PROGRESA. Snce the evaluaton study dd not follow movers, t s dffcult to assess the lkely magntude or even drecton of ths effect. Moreover, attrton s a serous problem n the evaluaton study: loss to follow-up s substantal (wth about 20% of the study partcpants beng lost by the second follow-up) and t s potentally selectve. Nonetheless, PROGRESA has been judged a great success, was mplemented across all rural and later to urban Mexco, and renamed OPORTUNIDADES. 8. Estmaton of varablty In model [1], T [1] 0 1 the varance of errors may not be the same among subjects n the treatment and control groups. onsstent estmaton of the standard error of the treatment effect, ˆ 1, s straghtforward wth an estmator that takes heteroskedastcty nto account, such as the Huber estmator or the jackknfe (Huber, 1967; Efron, 1982). In studes, lke PROGRESA, n whch randomzaton s at the communty (or group) level, rather than at the ndvdual level, estmaton of varablty s more complcated. Rewrte model [1] T [7] g 0 1 g g 8

10 where treatment, T, s defned at the group, g, level. Unobserved heterogenety, g, vares across ndvduals and may be correlated wthn groups. Falure to take nto account wthn-group correlatons (or clusterng) of resduals yelds estmates of varablty that are based, typcally (but not always) downwards. Adopton of an estmator lke the Huber or jackknfe estmator that allows for both heteroskedastcty and the block structure of the resduals provdes consstent estmates of the standard error of the treatment effect. Huber-type estmators do not perform well when the number of groups s small (say less than 15), the bas n estmators that gnore the problem can be qute large. A wld clustered bootstrap procedure has been shown to perform well n these stuatons. (ameron, Gelbach and Mller, 2008). Whle the concern arses n both expermental and non-expermental studes, t s partcularly salent n RTs that randomze at the group level wth few groups. In some studes, the model s extended to nclude covarates, X, whch may be measured at the ndvdual or group level: T X [8] g 0 1 g 2 g g The dea s that, f randomzaton has not been volated, the covarates should have no mpact on the estmated treatment effect, ˆ 1, because they are, by desgn, orthogonal to the treatment. The role of the covarates s to absorb unobserved heterogenety and thus ncrease effcency of the estmates. The strategy s, however, not wthout controversy (Freedman, 2008). 9. Generalzablty of evdence Populaton scentsts are often concerned wth the generalzablty of results n both observatonal and expermental studes. It s, however, t s dscussed most often n the context of experments that are conducted on selected sub-samples. For example, some expermental studes are based on convenence samples of respondents who sgn up for a study or are recruted for a study wthout payng attenton to whether the study sample s representatve of a well-defned populaton of nterest. The external valdty or generalzablty of results from such studes to a broader populaton s dffcult to assess snce there are potentally unobserved characterstcs n the broader populaton that are not present n the study sample. In that case, one of the key advantages of the RT desgn, that ET [ ] 0, may not be true n the broader populaton and so lttle can be nferred about the generalzablty of the evdence. Expermental studes that use the over-subscrpton approach to recrut subjects are potentally prone to ths crtcsm f the goal of the study s to provde evdence on the mpact of the program 9

11 beyond those who have sgned up for t. For example, t s not clear that the results of the evaluaton of a tranng program based on an over-subscrpton recrutment strategy wll generalze to all the unemployed. Ths concern wll arse f, relatve to all the people who are elgble for a tranng program, those who sgn up to partcpate are more motvated to do well, possbly because they have recently experenced a dp n earnngs or a recent, short unemployment spell (Heckman, 1992). As wth non-expermental studes, t s possble to make addtonal behavoral assumptons n order to estmate bounds on the treatment effect and ts dstrbuton, potentally explotng both the expermental varaton tself and also the desgn of the experment such as the potental outcomes for those who receve the treatment(s) and those who do not (Heckman, Smth and lements, 1997). Drawng out generalzable fndngs s more complcated when there are behavoral responses by the subjects to the study tself or to the fact that the subjects have been assgned to ether the treatment or control group. These are often called Hawthorne effects whch arse when study subjects respond to the stmulus of beng studed. Agan, ths s a general concern n socal research, partcularly studes that nvolve ntensve nteracton between the subjects and nvestgators. The ssue can be more complcated n expermental studes f subjects react to the fact that they were assgned to ether the treatment or control group and draw nferences from that assgnment. One of the dffcultes wth desgnng expermental studes s that t can be dffcult to predct behavoral responses to the study or ts desgn. For example, n an ambtous large-scale study of the mpact of ncentves on polce performance n Inda, subjects n one arm of the study who were not selected for the treatment felt dscouraged. Ths had a deleterous mpact on ther work effort and so the treatment (or lack of t) affected the behavors and outcomes of the controls, substantally complcatng nterpretaton of the results (Banerjee et al, 2014). In many cases, observatonal and expermental research studes are small n scale whch rases questons about both generalzablty and whether the study results are lkely to hold at a larger scale. For logstcal and cost reasons, experments are often conducted n a small number of study stes and, n some cases, the stes are selected because an mplementng partner or nsttuton s operatng n that area. Generalzablty wll be compromsed f the study results are drven by characterstcs that are unque to the ste or mplementng partner. For example, f the mplementng partner n a study of the mpact of mcro-credt s a partcularly effectve mcro-credt provder n an area where the program has been establshed, t s not clear that study results wll generalze to settngs n whch provders are not as effectve or programs have not already been establshed. Smlar ssues arse wth studes that nvolve schools or health provders that are commtted to mprove outcomes but are not representatve of the full set of nsttutons that wll be called upon to mplement changes f the 10

12 treatment s scaled up. These ssues are famlar to populaton scentsts and studes n the populaton feld are often desgned wth subjects and study stes (and possbly partners and mplementers) selected so that they are representatve of a well-defned underlyng populaton. 10. Equlbrum and longer-term effects Scalng up an experment rases a related but dfferent ssue: there may be equlbrum effects that are not captured by measurng only the treatment effect. onsder an experment n whch the treatment that mproves the health of subjects that results n ther beng more productve (by provdng ron supplements n an area where ron defcency s wdespread) and, as a result, say the treated subjects have more rce to sell on the market. There may well be equlbrum effects f the ncreased supply of rce results n a reduced prce (f the local market s closed) or ncreased ncome n the communty (f more rce s sold outsde the local market) whch may, n turn led to ncreased opportuntes for work. To explore these effects, the ron supplementaton study n entral Java, Indonesa, (Thomas et al, 2014) was desgned so that the fracton of the populaton recruted for the study n each of the 146 desa (vllages) vared and the fracton of the study subjects who were at rsk of beneftng from the supplements also vared. In addton, some desa were randomly assgned to not receve ether the treatment or the control so that they serve as addtonal controls, uncontamnated by the study tself. In an RT that provded ranfall ndex nsurance to cultvators n Inda, demand for hred labor and market wages was more senstve to ranfall wth the nsurance. Ths resulted n reduced uncertanty among cultvators but ncreased uncertanty among agrcultural wage laborers. Ths effect would not have been observed f the study had focused only the mpact of the RT on cultvator choces (Mobarak and Rosenzweg, 2014). The pont s not that equlbrum effects are nherently bad (or good) but, as s the case n observatonal studes, drawng on theory to dentfy potental equlbrum effects and desgnng expermental studes to explore the emprcal mportance of such effects s lkely to have a substantal pay-off. Few experments are desgned to provde evdence on longer-term effects. From a theoretcal perspectve, the mpact of a short-run ncrease n ncome s lkely to be dfferent from the mpact of a permanent ncrease n ncome or an ncrease n ncome that s expected to last for the long term. It s dffcult to nfer the mpact of a permanent ncrease n ncome wth the PROGRESA desgn. Ths s an mportant lmtaton, nherent n many RT studes that are desgned to provde a treatment for a lmted perod of tme. Ths s partcularly mportant when effects of the nterventon are lkely to cumulate over tme as subjects save enough to start a busness, for example. A smlar ssue arses 11

13 n the RAND Health Insurance Experment whch randomly assgned health nsurance wth dfferent copayments and deductbles to subjects. The program was provded for three years. If the benefts of health nsurance cumulate, the experment s unlkely to uncover the full benefts of the nsurance. In fact, subjects who receved the most generous benefts were more lkely to have a brth durng the three-year wndow, ndcaton shfts n the tmng of fertlty n response to the tme lmted nature of the experment (Newhouse et al, 1996). A related, but dfferent, feature of many RTs s that they are desgned as stand-alone studes that do not follow subjects for the long-term. In those cases, t s not possble to measure the longerterm mpact of the specfc (short-term) nterventon. Ths s not a feature of RTs, per se, but rather reflects the way many have been mplemented, the costs of the studes and the costs of longer-term follow-ups. In some cases, t s unlkely to be feasble to desgn and mplement an RT that wll provde answers to queston many years hence. For example, measurement of the mpact of pre-natal exposures or early chldhood experences on health and mortalty at older ages s unlkely to garner much support. However, trackng subjects over the lfe course, and measurng the evoluton of mpacts over ths perod has the potental to be very proftable. Indeed the MTO evaluaton as well as evaluatons of the mpact of early chld nterventons n adulthood (Hoddnott et al, 2008; Heckman et al, 2013; ampbell et al, 2014, ) are superb examples of the value-added to scence of followng treatments and controls n RTs over the long haul. A complementary approach s to buld randomzed varaton nto broad-purpose longtudnal populaton-based surveys that are desgned to follow ndvduals over the longer term, ndependent of the RT. The Matlab Health and Socoeconomc Survey, MHSS (Rahman et al, 1999) has been a poneer n ths regard. MHSS s a large scale populaton-based longtudnal study that has been conducted n conjuncton wth a long-term demographc survellance study conducted by IDDR,B n Matlab dstrct, Bangladesh, and has been used to evaluate, for example, maternal and chld health and famly plannng nterventons (Schultz and Josh, 2013). 11. Integratng theory and estmaton wth expermental desgns Recent studes have better ntegrated RTs wth theory and other research desgns. An ncreasng number of experments n the socal scences are desgned to reach beyond measurement of an average treatment effect of a program or polcy and test theores about behavor or dentfy mechansms that plausbly explan behavors and outcomes. Many of these desgns have roots n behavoral economcs. 12

14 In an early example, Ashraf et al (2006) desgned a commtment savngs product that was offered to a randomly selected sample of current and former clents of a bank n the Phlppnes n order to provde nsghts nto nter-temporal preferences and savngs behavors. Several studes have used expermental varaton n combnaton wth models to examne how prce varaton affects consumer demand; these studes have descrbed, for example, the role that prces play n sgnalng nformaton and how short-run subsdes affect consumpton over the longer-run (Dupas, 2014). Another lne of research explots randomly assgned varaton n expermental desgns to valdate estmaton of complex structural models that have a strong lnk to the underlyng theory and, by necessty, make potentally strong assumptons. Todd and Wolpn (2006) use data from the PROGRESA evaluaton study to estmate a dynamc behavoral model of parental decsons about fertlty and chld schoolng. Estmates are based on data from subjects collected pror to recept of program ncome and model estmates are valdated by comparng predctons of program effects from ths model wth expermental results. The model performs very well. Ths s mportant because, wthout ths external valdaton, t s very dffcult to confdently assess the qualty of model estmates and, therefore, the assumptons underlyng model. Furthermore, the structural model s desgned to conduct smulatons of alternatve polcy desgns n ths case condtonal transfer schedules and the authors conclude that the same program mpacts could be acheved wth fewer resources a result that s consstent wth conclusons that Schultz (2004) drew based on estmaton of PROGRESA program mpacts. Results from expermental studes have been proftably used to evaluate assumptons n observatonal studes (Lalonde, 1986). urre and Thomas (1995) and Garces, Thomas and urre (2002) use longtudnal non-expermental survey data to examne the mpact of Head Start on chldren, adolescents and young adults by comparng sblngs, one of whom attended Head Start and one dd not. It s not obvous that ths dentfcaton strategy s vald and so they assess the strategy by frst replcatng the man results from the Perry Preschool Project. That project s an RT that measured the mpact of early chld educaton on poor chldren, and served as a model for Head Start. The use of non-expermental data and methods provded evdence of the mpact of Head Start on a broader and more dverse populaton than the Perry Preschool study populaton as well as the mpact on outcomes and at ages that were not assessed n the Perry project. The credblty of the nonexpermental results depends crtcally on the valdaton from replcaton of the RT results. 13

15 12. onclusons In prncple, the deal or gold standard RT s an extremely powerful tool for dentfcaton of causal effects n the populaton scences. However, mplementaton of an RT that meets ths deal s far from straghtforward n medcne and s even more complcated n the context of socal experments. In practce, the desgn and mplementaton of an RT n the populaton scences s fraught wth problems. But the same s true, arguably a fortor, of observatonal studes. The key advantage of an RT s that the nvestgator has the potental to randomly assgn who s exposed to a stmulus or treatment that s of central nterest to the study. One of the dsadvantages s that the bgger the mpact of the treatment, the more lkely that study partcpants wll seek to undo the advantages of randomzaton. Garber et al (2014) argue that In the absence of theoretcal or methodologcal breakthroughs, the only possblty for further learnng comes from experments, partcularly experments wth strong external valdty. It s unclear ths s true: the argument that one approach to emprcal nqury domnates all others under all crcumstances s extreme. Whereas some questons n the populaton and socal scences are unlkely to be credbly answered wthout an expermental desgn others are not amenable to expermental manpulaton and cannot possbly be addressed wth an RT. RTs are but one arrow n the populaton scentst s quver and a potentally very powerful one at that. Rather than toss out all the other arrows, t seems lkely that scentfc knowledge about populaton behavor and processes wll be advanced through the ntegraton of creatve use of theory wth the clever desgn of both expermental and non-expermental studes. Recent studes suggest that, at least n ths regard, future prospects for the populaton scences are very brght. 14

16 References Ashraf N, Karlan D & Yn W (2006) Tyng Odyssesus to the mast: Evdence from a commtment savngs product n the Phlppnes. Quarterly Journal of Economcs 121.2: Banerjee A & Duflo E (2009) The expermental approach to development economcs. Annual Revews of Economcs, 1: Banerjee A & Duflo E (2010) Gvng credt where t s due. Journal of Economc Perspectves, 24(3): Banerjee A & Duflo E (2011) Poor Economcs: A radcal rethnkng of the way to fght global poverty. Perseus, Phladelpha. Banerjee A, hattopadhyay R, Duflo E, Kenston D & Sngh N (2014) Improvng polce performance n Rajasthan, Inda: Expermental evdence on ncentves, manageral autonomy and tranng. Mmeo. Bloom H, Orr L, Bell S, ave G, Doolttle F, Ln W & J Bos (1997) The benefts and costs of JTPA Ttle II-A Programs: Key fndngs form the Natonal Job Tranng Partnershp Act Study. Journal of Human Resources, 32.3: ameron A, Gelbach J & Mller D (2008) Bootstrap-based mprovements for nference wth clustered errors. Revew of Economcs and Statstcs 90.3: ampbell F, ont G, Heckman J, Moon SH, Pnto R & Pungello E (2014) Early chldhood nvestments substantally boost adult health. Scence 343(6178): urre J & Thomas D (1995) Does Head Start make a dfference? Amercan Economc Revew 85.3: Deaton, A (2009) Instruments of development: Randomzaton n the Tropcs and the Search for the Elusve Keys to Economc Development, Keynes Lecture n Economcs. Proceedngs of the Brtsh Academy, 162. Deaton, A (2010) Instruments, randomzaton and learnng about development. Journal of Economc Lterature, 48(2): Duflo E (2007) Feld experments n development economcs. In Blundell R, Newey W & Persson T (eds.) Advances n Economc Theory and Econometrcs, Econometrc Socety Monograph 42, hapter 13. ambrdge Unversty Press, ambrdge. Duflo E, Kremer M & Glennerster R (2008) Usng randomzaton n development economcs research: A toolkt. In Schultz TP & Strauss J (eds.) Handbook of Development Economcs, vol 4, chapter 15. Elsever. Amsterdam. Dupas P (2014) Short run subsdes and long-run adopton of new health products: Evdence from a feld experment. Econometrca, 82.1: Efron B (1982) The jackknfe, the bootstrap and other resamplng plans, SIAM BMS-NSF Regonal onference Seres n Appled Mathematcs, Phladelpha. Fsher RA (1935) The desgn of experments. Macmllan, London. Freedman DA (2008) On regresson adjustments to expermental data. Advances n Appled Mathematcs, 40:

17 Garber AS, Green DP & Kaplan EH (2014) The lluson of learnng from observatonal research n Teele DL (ed.) Feld experments and ther crtcs: Essays on the uses and abuses of expermentaton n the socal scences. Yale Unversty Press, New Haven Garces E, Thomas D & urre J (2002) Longer-term effects of Head Start. Amercan Economc Revew 92.4: Heckman J (1992) Randomzaton and socal polcy evaluaton. In Mansk, Garfnkel I (eds.) Evaluatng Welfare and Tranng Programs , Harvard Unversty Press, ambrdge. Heckman (2010) Buldng brdges between structural and program evaluaton approaches to evaluatng polcy. Journal of Economc Lterature, 48(2): Heckman J, Pnto R & Savelyev P (2013) Understandng the mechansms through whch an nfluental early chldhood program boosted adult outcomes. Amercan Economc Revew, 103.6: Hoddnott J, Malucco J, Behrman J, Flores R & Martorell R (2008) Effect of a nutrton nterventon durng early chldhood on economc productvty n Guatemalan adults. Lancet, 371: Huber PJ (1967) The behavor of maxmum lkelhood estmates under nonstandard condtons. Proceedngs of the Ffth Berkeley Symposum on Mathematcal Statstcs and Probablty, 1:221-33, Berkeley. Heckman J,Smth J & lements N (1997) Makng the most out of programme evaluatons and socal experments: Accountng for heterogenety n programme mpacts. Revew of Economc Studes 64: Lalonde R (1986) Evaluatng the econometrc evaluatons of tranng programs wth expermental data. Amercan Economc Revew, 76.4: Laurtsen J (1990) Poson by prescrpton: The AZT story. Asklepos Press, New York. Ludwg J, Duncan G, Gennetan L, Katz L, Kessler R, Klng J & Sanbonmatsu L (2012) Neghborhood effects on the long term well-beng of low-ncome adults. Scence 337(6101): Newhouse J & the Insurance Experment Group (1996) Free for all? Lessons from the RAND Health Insurance Experment. Harvard Unversty Press, ambrdge. Neyman J (1923) On the applcaton of probablty theory to agrcultural experments. Essays on Prncples. Secton 9. Statstcal Scence 5(4): Orcutt G & Orcutt A (1968) Incentve and dsncentve expermentaton for ncome mantenance polcy purposes. Amercan Economc Revew 58: Parker S, Rubalcava L & Teruel G (2008) Evaluatng condtonal schoolng and health programs. In Schultz T P & Strauss J Handbook of Development Economcs Volume 4, hapter Perce S & Jastrow J (1885) On small dfferences n sensaton. Memors of the Natonal Academy of Scences, 3: Phllps J, Smmons R, Koeng M & hakraborty J (1988) Determnants of reproductve change n a tradtonal socety: Evdence from Matlab, Bangladesh. Studes n Famly Plannng, 19.6:

18 Phllps J, Bawah A & Bnka F (2006) Acceleratng reproductve and chld health programme mpact wth communty-based servces: The Navrongo experment n Ghana. Bulletn of the World Health Organzaton 84: Rahman O, Menken J, Foster A, Peterson E, Khan M N, Kuhn R & Gertler P (1999) The 1996 Matlab Health and Socoeconomc Survey: Overvew and User s Gude. RAND, Santa Monca. Schultz TP (2004) School subsdes for the poor: Evaluatng the Mexcan Progresa poverty program. Journal of Development Economcs, 74.1: Schultz T P & Josh S (2013) Famly plannng and women s and chldren s health: onsequences of an outreach program n Matlab, Bangladesh. Demography 50.1: Stgler S (1992) A hstorcal vew of statstcal concepts n psychology and educaton research. Amercan Journal of Educaton, 101(1): Thomas D, Frankenberg E, Fredman J, Habcht JP, Ingwersen N, McKelvey, Hakm M, Jaswad, Pelto G, Skok B, Seeman T, Smth JP, Sumantr, Surastn W & Wlopo S (2014). ausal effect of health on labor market outcomes: Expermental evdence. Mmeo. Todd P & Wolpn K (2006) Assessng the mpact of a school subsdy program n Mexco: Usng a socal experment to valdate a dynamc behavoral model of chld schoolng and fertlty. Amercan Economc Revew, 96.5: Relevant web stes The ochrane ollaboraton AEA RT Regstry Abdul Latf Jameel Poverty Acton Lab 17