Evaluating School Choice Programs

Transcription

1 DRAFT: PLEASE DO NOT CITE OR REDISTRIBUTE Evaluating Schl Chice Prgrams Jennifer Hill Clumbia University March 9, Intrductin Educatin refrm has becme an imprtant public plicy issue. The public fcus n educatin refrm is driven, at least in part, by statistics that suggest that ur public schl system may be failing. Fr instance, the U.S. Department f Educatin reprts that currently nly 1/3 f the natin s furth graders read at a prficient level and nly 1/4 f furth and eighth graders perfrm math at a prficient level. The numbers are even mre dire in urban areas. Whether these indicatrs reflect a genuine crisis r nt, they have helped t bring the debate ver educatin refrm int the limelight. Tw events that ccurred in 2002 have paved the way twards the pssible adptin f ne f the mst cntrversial educatin refrms, publicly-funded private schl vuchers. In January f 2002, President Bush s N Child Left Behind Act became law. One f the mst imprtant implicatins f this new law is that any child attending a failing schl (schls fr which test scres fr certain ppulatins fail t imprve ver a tw year perid) has t be prvided with the pprtunity t transfer t a schl that is nt failing. Critically, schl districts cannt deny this right t a child simply due t capacity cnstraints in their schls. Given that vercrwding is already a prblem in many f the affected Crrespndence shuld be addressed t the authr at Clumbia University Schl f Internatinal and Public Affairs, 740 IAB, 420 W. 120th St., New Yrk, NY jh1030@clumbia.edu. Phne: (212)

2 schl districts, this law may put pressure n these districts t turn t alternatives such as publicly-funded private schl vuchers. Als, in June f 2002, the Supreme Curt, in a 5-4 ruling, upheld the cnstitutinality f a publiclyfunded schl vucher prgram in Cleveland that went int effect in The cntrversy arse because sme (in fact the vast majrity) f the private-schl schlarships were being used by participants t attend parchial schls. Oppnents argued that this vilated Cnstitutinal mandates regarding the separatin f Church and state. The Supreme Curt decisin remves this bstacle frm states wanting t implemement vucher prgrams. Private-schl vuchers are ne manifestatin within the mre general rubric f schl chice. Other variatins include magnet schls, charter schls, and varius frms f intra- and inter-schl district chice systems (that is, chice within the public schl system). Private schl vuchers have been amng the mst cntentius f these schl chice initiatives. Critics maintain that publicly-funded vuchers drain mney away frm already under-funded public schls, fster greater racial segregatin, d nt prvide better educatinal pprtunities, steal the best students frm the public schls, and create mre instability fr affected students. Oppnents als argue that vuchers are nt a realistic apprach given the large number f children in public schls. Prpnents f vuchers cunter that private schls create greater racial integratin, bst achievement, prvide a safer and mre rderly learning envirnment, allw parents t chse the mst apprpriate schls fr their children, and can frce public schls t cmpete and becme mre efficient. Arguments abut whether r nt vuchers shuld be adpted in sme frm and fr sme ppulatins (e.g. children in failing schls) traditinally have nt been infrmed by strng empirical evidence. This article reviews varius appraches t measuring the effect f vuchers n children s educatinal achievement. Our first step is t cnsider why previus attempts at cmparing public and private schl achievement have nt prvided cnvincing evidence. A secnd step is t cnsider the use f randmized eperiments in evaluating educatinal prgrams. Randmized eperiments, where a study ppulatin is randmly divided int tw r mre treatment grups, have been very successful fr determining the effects f treatments in fields like medicine. This article then eamines a randmized eperiment that was perfrmed in the cntet f private-schl vuchers. A private fundatin s annuncement that it planned t ffer 1300 schlarships ($1400 each) t lw-incme children in New Yrk City t use in private schls prvided researchers with a naturally 2

3 ccurring randmized eperiment. What this natural eperiment can and cannt tell us is the subject f ur stry. 2 Early studies and the prblems with bservatinal data It is instructive first t cnsider what was learned prir t the New Yrk eperiment (and ther related eperiments). In the absence f any prgrams implementing private-schl vuchers it is quite difficult t eamine the ptential efficacy f vuchers. This is the cnceptual prblem that faced researchers in the 1970s and 1980s. A number f researchers attempted t make such an eaminatin thrugh cmparisns f students attending public and private schls. One f the earliest and best knwn studies was published in a series f articles and the 1982 bk High Schl Achievement: Public, Cathlic and Private Schls Cmpared by James Cleman, Thmas Hffer, and Sally Kilgre. After cnducting analyses n data frm the High Schl and Beynd survey carried ut by the Natinal Center fr Educatin Statistics they cncluded that private schls led t higher achievement than public schls. These findings were cntrversial and sparked a series f further studies and analyses. In general these studies did nt fully supprt the strng findings f the riginal wrk; they shwed nn-eistent, much weaker r mied advantages t the private schls. Since then many mre studies have been prefrmed with a range f findings. Hw is it pssible that researchers were able t find s many different answers t the same questin? Fr ne thing, there were a number f differences amng the many studies. Nt every study used the same survey data, r even the same type f student ppulatin (age and race varied). The utcme used t measure student success als varied acrss studies; amng the utcmes used are test scres, high schl graduatin, and cllege entry. Sample size als varied a great deal amng studies and it is imprtant t remember that large datasets are mre likely t yield statistically significant results than small datasets. These many differences are nt the key issue hwever. Even sme studies that used the same data surce, the same student ppulatin, the same utcme measure and the same statistical methds reached substantially different cnclusins. Why? The majr limitatin f the early studies is that they rely n bservatinal, rather than eperimental, data. The students attending private schls were ding s because they (r their parents) chse t d s. The students attending public schls were ding s because they (r their parents) chse t d s (perhaps because f financial, lgistical, r ther cnstraints that ruled ut private schls). This creates a situatin 3

4 where the students attending private schls are likely t be different than thse attending public schls. They might cme frm families with mre mney (particularly in the case f thse attending secular private schl), be mre mtivated, r have higher educatinal epectatins placed n them. Thus if there are differences in average test scres between these grups, it can be impssible t disentangle t what etent these differences were caused by variatin in the types f students attending each (knwn as the effects f selectin bias) and t what etent the differences were caused by the schls themselves (knwn as the causal effects f attending private schl). The early studies tried t adjust r cntrl fr the differences between the ppulatins using statistical mdeling. T illustrate hw such an adjustment might wrk cnsider the hypthetical data relating test scres, type f schl and family incme in Figure 1. In the first panel it appears that private schl students, represented by the circles, utperfrmed public schl students, represented by the s, n the test. Suppse, hwever, that we als have infrmatin regarding family incme demstrating that the children wh g t private schl cme frm families that have higher incme n average. Perhaps incme culd eplain this differential in test scres between private-schl and public-schl students. The middle panel f the Figure shws the same data with the best fitting line relating test scres t incme. Fr these (hypthetical) data a single line des a gd jb fr bth types f students. This means that the difference in test scres that we bserved in the left panel is cmpletely eplained by difference in family incme. If we make an adjustment we will find that students with cmparable incmes perfrm similarly n the test regardless f which schl they attend. The third panel shws a different hypthetical data set where even after adjusting fr incme it appears that private schl students have a test scre advantage relative t public schl students (the best-fitting line fr private-schl students is higher than the best-fitting line fr public-schl students). Clearly the many discrepant results described earlier suggest that this apprach t addressing bias due t self selectin desn t always wrk. One difficulty is that we can t be sure that we have prperly adjusted fr all relevant differences between grups. Fr instance what if we have adjusted fr age, race, and religin, but the mst imprtant variable t cntrl fr is incme? Our statistical methds will generate an answer, and it will be wrng, but we wn t have any way f knwing this frm the result. Anther prblem is that even if we smehw knew that we had included all relevant variables, the simple regressin apprach t adjustment relies n a number f difficult t verify assumptins. One key assumptin is that the relatinship between the utcme (e.g. test scres) and ur backgrund variables (e.g. incme) 4

5 public private test scres incme test scres incme test scres Figure 1: Illustratin f statistical adjustment f bservatinal data using hypthetical test scre data. Panel A plts test scre against schl type indicating that private schl students have higher test scres n average. Panel B plts the same test scres against incme with circles fr private-schl students and s fr public-schl children. The best-fitting line is als shwn; evidently the bserved differences in test scres are all due t incme. Panel C is similar t Panel B but fr these hypthetical data there appears t be a private schl advantage even after cntrlling fr incme. has t be linear. If the relatinship is nt linear, fr eample perhaps test scres increase with incme but nly t a pint after which they plateau, then the regressin adjustment will lead t a pr estimate f the difference between private and public schl students. There are, t be sure, mre sphisticated appraches fr trying t cntrl fr the differences between the private schl and public schl ppulatins. Yet even amng the subset f papers using mre advanced techniques t address selectin bias, the results still are mied with sme supprting private schl gains in general, sme supprting gains fr Cathlic schls but nt ther private schls, and sme finding n differences between educatinal sectrs. Observatinal studies d nt appear t be ideal fr calculating the causal effect f being given a private schl educatin rather than a public schl educatin because peple self-select int their schl envirnment yielding different types f students in each type f envirnment. 3 The rle f randmized eperiments The difficulties with bservatinal studies are well knwn in many fields. Fr eample, the same prblem arises in medicine if we cmpare patients that select ne particular frm f treatment t a secnd grup 5

6 that prefers an alternative. In that cntet, and in many ther fields, it has becme cmmn t rely n randmized eperiments where the subjects are randmly assigned t the different treatments in rder t assess the relative effectiveness f thse treatments. The randmizatin prcess ensures that the grups being cmpared are similar n average. T study schl chice then we might like t randmly assign students t either private r public schls. But, taken at face value, this seems an unethical and impractical plan. We can t easily manipulate the student ppulatin in this way. Wuld siblings be randmized int different schls? What abut transprtatin issues? What if parents culdn t affrd the schl t which their child was randmly assigned? Smetimes there are situatins, hwever, when the sught after randmizatin ccurs naturally. Frtunately fr schl chice research, several such pprtunities arse in the 1990 s. One such natural randmized eperiment ccurred in New Yrk City. In February 1997, a private fundatin in New Yrk called the Schl Chice Schlarships Fundatin (SCSF) annunced that it wuld ffer 1300 schlarships (in the amunt f $1400 each) t lw-incme children in New Yrk City t use twards private schl tuitin. Since the number f applicants was sure t eceed the number f available schlarships, a lttery was planned as the mst equitable way f allcating the schlarships. This lttery perfrms the randmizatin needed fr a fair cmparisn. During the time that the schlarship initiative was being develped, a Harvard prfessr, Paul Petersn, cnvinced the SCSF t take advantage f the randmizatin and allw the prgram t be evaluated by utside researchers. Mathematica Plicy Research jined the prject with researcher David Myers as the lead principal investigatr jint with Petersn and they helped t shape bth the implementatin and the evaluatin f the SCSF Prgram. The SCSF Prgram targeted primarily lw-incme grade schl children frm under-perfrming schls. Eligibility required the children t have family incmes lw enugh t qualify fr schl lunch and t be in public schl in kindergarten thrugh furth grade when applying. Only 15% f the children awarded schlarships were allwed t be frm schls whse average test scres were abve the city-wide median (high-perfrming schls) because the SCSF was mre interested in vuchers as a ptential plicy initiative fr students in trubled schls. The evaluatin plan invlved preliminary testing f students and then fllwup testing ver the three years f the prgram fr students receiving the schlarships and selected students amng the unsuccessful applicants. Hw des a lttery create fair cmparisns? A lttery randmly divides the applicants int tw grups: 1) a chsen grup f schlarship recipients and 2) a secnd grup f unsuccessful applicants. Because f the randmizatin, the tw grups shuld be similar in terms f their characteristics measured befre the lttery 6

7 tk place (r characteristics nt changed by the lttery, such as gender). Indeed cmparisns acrss grups fr a hst f characteristics ranging frm child pretest scres and gender t parental incme and educatin levels reveal that, as epected, the use f randmizatin creates grups that are similar but nt identical. Any differences are caused by chance they are nt systematic. This means that if we were t perfrm the lttery again we wuld be just as likely t see differences in the ppsite directin. Because ur tw grups are similar in terms f all their pre-treatment characteristics, cmparing the perfrmance f the tw randmized grups n achievement tests taken sme time after the children have been randmized shuld allw fr a fair evaluatin f the effect f receiving a vucher/schlarship because nw the nly systematic difference between the tw grups is that in ne grup peple had schlarships t private schls and in the ther grup they did nt. Tw large-scale randmized vucher evaluatins similar in spirit t the New Yrk City prgram fllwed in Washingtn, D.C.and Daytn, Ohi. There have als been randmized evaluatins f vucher prgrams in ther cities but each had sme prblems with eecuting the randmizatin r in analyzing the data and thus are nt discussed here. All f the randmized studies targeted lw-incme, primarily urban ppulatins. Majr characteristics f the studies can be fund in Table 1. A few features regarding the study characteristics are ntewrthy. First, ntice that the prprtin f students awarded vuchers that actually used them is between 50% and 70%. This seems lwer than ne might epect. Secnd, ntice that a significant fractin f the children dn t take the fllw-up tests. Bth f these ccurrences, the failure f subjects t fllw thrugh with the assigned treatment and the failure t cllect all needed data, are cmmn and pse challenges t evaluatrs. We return t these issues belw. Fr reasns that will becme clear belw, the actual results are nt the mst imprtant pint f this statistical stry. Table 2 presents results fr each eperiment frm its final year as presented in the evaluatin reprts. This was year 3 fr the New Yrk eperiment and year 2 fr Daytn and Washingtn. The numbers in the table represent the average gain n standardized tests measured in percentile pints fr students that attended private schl at least ne year n standardized tests measured. Thus in the New Yrk City eperiment math scres are abut 2 percentile pints higher fr private schl attenders. Specifically, in this case this treatment effect reflects the fact that the children wh had attended private schl scred 26.2 in terms f natinal percentile rankings (26.2% f the children natinwide wh tk the test scred belw this mean scre) and their cmparisn grup frm amng the public schl children scred 23.9 in terms f natinal percentile rankings n average. The results are intriguing in that when the results are reprted by ethnic- 7

8 Start Applic. Participatin Fllw-up Lcatin Dates Years Grades Rates Rates New Yrk City K-4 1st year: 64% 1st(1998): 78% (86% Chice, 75% Cntrl) 2nd year: 62% 2nd(1999): 65% (69% Chice, 60% Cntrl) all 3 years :53% 3rd(2000): 67% (69% Chice, 65% Cntrl) D.C st year: 53% 1st (1999): 63% bth years: 38% 2nd(2000): 50% Daytn, Ohi K-12 1st year: 54% 1st(1999): 56% Chice, 57% Cntrl 2nd year: 50% 2nd(2000): 48% Chice, 50% Cntrl Wlf et al. (2001, p. 15) nte that appimately half the families in the treatment grup actually used it [a schlarship] during the prgram s secnd year. Surces fr the results in this table are Mayer et al. (2002) and Hwell et al. (2000). Table 1: Majr features f the three majr randmized vucher eperiments including the lcatin, duratin, grade levels f participating children, participatin rates and rates f missing utcme data. ity there are statistically significant imprvements nly fr African Americans. Statistically significant means that the increases are larger than can be eplained by gd luck during the allcatin f individuals via the lttery and thus the increase can be attributed t attending private schl rather than chance. On the ther hand the vuchers seem t hurt the perfrmance f ther ethnic grups in reading (thugh these results are nt statistically significant) and d little fr the ther ethnic grups in math. It is nt clear why there shuld be such a large difference amng ethnic grups since the vuchers were nt thught t be especially advantageus fr a particular ethnic grup. The purpse f the present article is t demnstrate the rle f statistical arguments in analyzing scial eperiments s we wn t treat these results as the end f ur stry but rather as a jumping ff pint fr discussing hw they might have been affected by decisins made regarding hw t analyze the data. 4 Challenges in designing and analyzing scial eperiments Since randmized eperiments are suppsed t be the gld-standard in terms f identifying the true causal effects f an eperimental prgram why d we dwnplay the results? Desn t the randmized eperiment address the questin f private versus public schl efficacy nce and fr all? T be blunt the answer is n. Hints f the challenges faced by researchers were mentined abve in the discussin f the study characteristics presented in Table 1, namely a failure f vucher recipients t 8

9 Reading Math Lcatin Overall AA Other Overall AA Other NYC DC Daytn AA stands fr mther is African American ; Other represents all ther racial/ethnic grups. Table 2: Estimates f theffects f ever-attending private schl n test scres (in natinal percentile rankings) frm the final fllw-up year fr the three majr randmized vucher eperiments. Results are brken dwn by ethnicity and type f test (reading/math). significant at.05 level, 2-tailed test; significant at.01 level, 2-tailed test. Surces fr the results in this table are Mayer et al. (2002) and Hwell et al. (2000). attend private schls and missing fllw-up test scres. Even thugh these studies made use f randmized eperiments, the participants were human subjects with free will wh culd nt be frced t cmply with study prtcl. Of curse such cmplicatins can and d ccur in randmized medical studies but they are typically less dramatic in medicine especially if participants are blinded s that they d nt knw which drug they have been randmized t receive. Here thse nt selected t receive a vucher knw that they have nt been selected and there is n way t avid that knwledge. Nncmpliance: Peple dn t always d what they are tld Nt all f the students wh were fferred a vucher were able t actually use it t attend private schl fr a range f financial, lgistical r ther reasns. The implicatins f this cmplicatin, ften called nncmpliance, fr ur analyses are a bit subtle. In general this means that a simple cmparisn between the randmized grups will nt yield an estimate f the effect f ging t private schl, it will yield an estimate f the effect f being ffered a schlarship. This is knwn as the intent-t-treat (ITT) estimate since it measures the effect f ur intentin t apply a treatment (ffer a schlarship) withut regard fr whether that treatment was carried ut r nt. This is a cmmn prblem in randmized drug trials as well, particularly when the drugs cause unpleasant side effects. In sme situatins it is argued that nncmpliance is nt a big prblem because a simple cmparisn between the randmized grups, that is the ITT analysis, is a cnservative estimate f the true treatment effect (i.e. it shuld be clser t zer if anything). In ther wrds the ITT effect shuld be a watered-dwn 9

10 versin f the true treatment effect therefre the results wuld nly get strnger if everyne had fllwed thrugh. Unfrtunately this reasning des nt necessarily hld when we have missing data as well (as we will see mre clearly in the net sectin). In this case ur estimate culd ptentially verstate the benefits f the treatment leading yu t believe a treatment is effective when in fact it is nt. Hw can ne get a mre realistic estimate f the value f attending private schl? Sme bvius prpsals dn t wrk very well. Fr eample, we can t just cmpare the scres f thse wh used the vuchers t the randmized cmparisn grup because the types f peple wh tk advantage f the vucher pprtunity culd be quite different frm thse wh did nt. This means that we lse the benefit f ur randmized eperiment and end up with a frm f bservatinal study. It is hwever pssible t generate a gd estimate f the value f attending private schl. It requires a fairly sphisticated statistical apprach using what is knwn as an instrumental variable. A detailed discussin is beynd the scpe f this chapter but the intuitin in this scenari is that the apprach tries t make cmparisns between the grup f peple wh actually attended with the grup f public schl attendees wh we believe wuld have attended private schl if they had instead wn a schlarship. This subset f the cntrl grup shuld be mre similar t the private schl attendees than the full cntrl grup. The results in Table 2 rely n such instrumental variable adjustments. Fr insight int the differences in cnclusins that might be reached between analyses that estimate the ITT effect and analyses that fcus n the treatment effect fr thse wh actually attended a private schl, cnsider the tw sets f results in Table 3. The first panel shws ITT results frm the 1st year f the NY study fr children in grades 1-4 (when they applied t the prgram) frm families where nly ne child participated (see Barnard, Frangakis, Hill, and Rubin 2003). These represent the effect n test scres f being fferred a schlarship and they are brken dwn by the type f schl the child was attending at the time f applicatin (lw-test-scre/high-test-scre) and reading versus math tests. As discussed earlier, fewer students frm high-test-scre schls were allwed t win schlarships (nly 15%) because it was assumed that they were in less need f the ptential benefits f private schl. The secnd panel shws adjusted results that reflect the effect f actually attending a private schl (these als adjust fr missing data). Ntice that fr each quantity f interest the adjusted estimates are larger in magnitude than the ITT effects. Sme researchers argue that the ITT effect is the mst apprpriate fr evaluating the ptential average effect f a plicy because it reflects the fact that nt all eligible individuals will chse t participate in the 10

11 Applicant Reading Math Schl ITT Attended ITT Attended Lw High Table 3: Treatment effect estimates after 1 year in the prgram brken dwn by lw-scring and highscring schls and reading and math scres. The table cmpares the intentin-t-treat (ITT) effect and the effect f attending private schls (Attended). Test scres are natinal percentile rankings. significant at.05 level, 2-tailed test; significant at.01 level, 2-tailed test. Surce fr the results in this table is Barnard et al. (2003). prgram being investigated. Hwever, fr this lgic t be crrect, the effect when the actual plicy is implemented wuld have t be just as watered dwn (that is just as high a percentage f peple wuld have t chse nt t participate) as bserved in the preliminary investigatin. In additin, the same types f peple wuld have t chse t participate. Mrever, if we can assume that the same types f peple wuld chse t participate, then we can use the estimated treatment effect fr thse wh participated (here, the effect f actually attending private schl) t calculate ITT effects crrespnding t any ptential cmpliance rate specified. Thus, the mst cnservative strategy is prbably t estimate bth effects. Missing Data: Yu can t frce peple t stay in the study Missing utcme data ccurred in these studies because nt all children participating in each study actually shwed up t take fllw-up tests. Althugh participants were ffered a range f incentives (including mney) in each study t encurage them t shw up fr the pst-tests, the fllw-up rates achieved are well belw perfect (thugh still quite gd fr studies f this kind) as can be seen in Table 1. This reduces the number f peple in the study but wuldn t necessarily be a majr prblem if we culd rely n randmizatin t ensure that the students missing pst-test scres were just like the students wh tk the pst-tests. Unfrtunately randmizatin desn t help us here because the fllw-up tests happened after the lttery! We see in Table 1 that the fllw-up rate is generally lwer fr the cntrl grup than fr the schlarship grup in mst f the lcatins which is trubling. The n-shws may be different n ther traits as well; they may be less well mtivated r they may have parents wh wrk n weekends and hence they were nt able t attend the weekend testing. If the peple wh failed t shw up fr their pst-tests are systematically different frm thse wh did attend, and we have reasn t believe this is true, then a simple analysis f the bserved test scres can be biased. 11

12 Applicant Reading Math Schl CC All Data CC All Data Lw High Table 4: Treatment effect estimates after 1 year in the prgram brken dwn by lw-scring and high-scring schls and reading and math scres. The table cmpares estimates frm cmplete-case analyses (CC) with estimates frm mre frmal missing data mdels (All Data). Test scres are natinal percentile rankings. significant at.05 level, 2-tailed test; significant at.01 level, 2-tailed test. Surce fr the results in this table is Barnard et al. (2003). There are a number f statistical appraches t addressing these missing fllw-up test scres. The apprach that was used in Table 2 uses weighting t adjust fr the missing utcme data. The basic idea fr this is as fllws. Backgrund infrmatin n each persn (e.g. pre-test scres, incme, race/ethnicity) is used t develp a statistical mdel that can predict the prbability that each persn will have missing test scres. If this prbability is lw fr a grup then the individuals in that grup that d shw up fr the fllwup test are given increased weight in the analysis. S, fr eample, if students with lwer pre-test scres were less likely t shw up fr the pst-test, the weights wuld adjust analysis results t reflect that fact that mre students with lw pre-test scres shuld have had their pst-test scres included in the analysis. The weights will give the test scres fr students with lw pre-test scres wh did shw up mre weight in the final analysis. In rder t eplre the ptential differences in results caused by missing data, cnsider the analyses f first-year NY data presented in Table 4 (again fr children wh applied in grades 1-4 frm families where nly ne child participated). The results in the secnd panel are the same as thse in the first panel fr Table 3, ITT effect estimates brken dwn by schl type and math versus reading tests. The analysis in the first panel uses nly data fr New Yrk City students wh shwed up fr the pst-test. This is called a cmplete-case analysis. The analysis in the secnd panel uses all the students in the analysis sample. The analysis in Panel 2 uses an apprach fr the missing data that is mre frmal and cmplicated than the weighting strategy described abve but relies n the same type f intuitin (Barnard et al. 2003). The treatment effect estimates fr three f the fur categries are higher fr the analyses that mre frmally handle the missing data. In the case f the math scres fr children applying frm lw-perfrming schls the estimate als became statistically significant. The estimate that becmes smaller (reading tests 12

13 fr children applying frm high-perfrming schls) als lses significance. These cmparisns highlight the need t apprpriately handle missing data issues as they culd lead t different cnclusins abut the effectiveness f private schl educatin fr these children. What if: Results may be sensitive t statisticians chices Statistical analyses are ften criticized with remarks like, yu can get statistics t say anything r if yu trture the data lng enugh it will cnfess t anything. Thugh unethical uses f statistics are pssible, these criticisms als reflect anther legitimate cncern in a cmple study like the New Yrk City schl vucher prgram. Analyzing the data frm such a study requires that a number f chices be made by the prgram evaluatrs. We have already seen that ne must decide what eactly t estimate (the intent-t-treat effect r the actual effect f attending private schl) and hw t handle the missing data. In additin, a number f ther decisins are required in carrying ut the data analysis. In the New Yrk City study, the initial analysis mitted kindergarten students (because they had n pretest scres and such scres were part f the preferred analysis) and defined children as African American based n the race nly f their mther. Wuld different chices change the results? One subsequent study by ecnmists Alan Krueger and Pei Zhu suggests that they culd. They perfrm a variety f analyses making different chices regarding these issues (e.g. including kindergarteners, perfrming cmplete case analyses, defining race f child differently) and estimates frm their analyses, while still psitive fr African Americans, are nt all statistically significant. 5 Cnclusins Addressing scial plicy questins, like assessing the benefits f schl vuchers, presents unique statistical challenges. It can be difficult t carry ut randmized eperiments fr such questins. And, even when randmized eperiments can be implemented we must deal with missing data, nn-cmpliance, and the sensitivity f ur results t the chices made by the prgram evaluatr. In fact, the challenges f evaluating scial plicies g beynd these technical issues. Are the peple wh vlunteer t particpate in prgrams like the Schl Chice Schlarship Fundatin prgram in New Yrk similar t thse that gvernment plicies wuld ultimately target? Can ur results with a relatively small number f students be generalized t frecast what wuld happen with a large prgram? Fr eample, what wuld happen t a public schl if a large prprtin f their students left? What wuld happen t a private schl flded with applicatins? 13

14 Prviding sund quantitative evidence t aide in the public debate cncerning schl refrm is clearly a difficult prblem. In the schl vucher case, natural randmized eperiments thrugh the ltteries used t distribute the vuchers prvide a unique pprtunity. It is imprtant t remember that these randmized eperiments, thugh challenging t analyze, are much mre likely t yield reliable cnclusins than the bservatinal studies available up t this pint. The challenge t mdern statistical science is t develp methds that allw prgram evaluatrs t reliably address imprtant real-wrld cmplicatins like missing data and nncmpliance. References Barnard, J., Frangakis, C. E., Hill, J. L., and Rubin, D. B. (2003), Principal Stratificatin Apprach t Brken Randmized Eperiments: A Case Study f Schl Chice Vuchers in New Yrk City, Jurnal f the American Statistical Assciatin 98, Brandl, J. E. (1998), Mney and Gd Intentins are nt Enugh, r Why Liberal Demcrat Thinks States Need Bth Cmpetitin and Cmmunity, Washingtn, D.C.: Brkings Institutin Press. Cleman, J. S., Hffer, T., and Kilgre, S. (1981), Public and Private Schls, Tech. rep., Natinal Center fr Educatin Statistics. Cksn, P. W. (1994), Schl Chice: The Struggle fr the Sul f American Educatin, New Haven, CT: Yale University Press. Culsn, A. J. (1999), Market Educatin: The Unknwn Histry, Bwling Green, OH: Scial Philsphy & Plicy Center;Transactin Publishers. Fuller, B. and Elmre, R. F. (1996), Wh Chses? Wh Lses? Culture, Institutins, and the Unequal Effects f Schl Chice, New Yrk: Teachers Cllege Press. Hwell, W. G., Wlf, P. J., Petersn, P. E., and E., C. D. (2000), Test-Scre Effects f Schl Vuchers in Daytn, Ohi, New Yrk City, and Washingtn D.C.: Evidence frm Randmized Field Trials, Tech. rep., Prgram f Educatin and Plicy Gvernance. 14

15 Krueger, A. B. and Zhu, P. (2003), Anther Lk at the New Yrk City Schl Vucher Eperiment, Tech. rep., Natinal Center fr the Study f Privatizatin in Educatin, New Yrk. Mayer, D. P., Petersn, P. E., Myers, D., Tuttle, C. C., and Hwell, W. G. (2002), Schl Chice in New Yrk City After Three Years: An Evaluatin f the Schl Chice Schlarships Prgram, Tech. rep., Mathematica Plicy Research and Prgram f Educatin and Plicy Gvernance. Wlf, P. J., Petersn, P. E., and West, M. R. (2001), Results f a Schl Vucher Eperiment: The Case f Washingtn, D.C. After Tw Years, Tech. rep., Prgram n Educatin Plicy and Gvernance, Cambridge, MA. 15