A comparison of statistical methods in interrupted time series analysis to estimate an intervention effect

Transcription

1 Peer revew stream A comparson of statstcal methods n nterrupted tme seres analyss to estmate an nterventon effect a,b, J.J.J., Walter c, S., Grzebeta a, R. & Olver b, J. a Transport and Road Safety, Unversty of New South Wales, Australa, b School of Mathematcs and Statstcs, Unversty of New South Wales, Australa, c Centre for Health Systems and Safety Research, Unversty of New South Wales, Australa Abstract Snce the ntroducton of mandatory helmet legslaton (MHL) n Australa, debate on the effect of MHL on cyclst head njures has been ongong. The debate sometmes revolves around the statstcal methodology used to assess nterventon effectveness. Supporters of rescndng the MHL thereby encouragng cyclsts to rde wthout helmets, regularly dsmss statstcal evaluatons as beng flawed for varous reasons. In a more general context, researchers want to estmate whether and how a polcy nterventon changed an outcome of nterest. Quas-expermental nterrupted tme seres (ITS) s the most approprate desgn to evaluate the longtudnal effects of polcy nterventons and segmented regresson analyss s often used as a powerful statstcal method for ITS. Recent research has employed a log-lnear regresson model for the hosptal admsson counts of head and lmb njures from New South Wales, Australa, from a 36 month perod centred at the tme of legslaton. Estmaton of the model was done usng a frequentst approach. In ths paper, we re-analyse ths data usng emprcal Bayes and full Bayesan methods, snce the use of these methods has become popular n road safety studes. In partcular, we show how a full Bayesan method can be readly mplemented n WnBUGS software. We dscuss the advantages and dsadvantages of each method and descrbe and compare the dfferent estmaton methods n terms of parameter estmates. The results show that all three estmaton methods gve consstent conclusons regardng the postve effect of compulsory helmet wearng on cyclst head njures n New South Wales. Introducton For the before-after evaluaton of the effects of mplemented polcy nterventons, nterrupted tme seres (ITS) s the strongest quas-expermental approach (Wagner et al., 2002). An approprate statstcal method for analysng effects of nterventons n ITS data s segmented regresson analyss. In a segmented regresson analyss or a change-pont model, each segment of the tme seres s allowed to exhbt dfferent levels and trends. In other words, the segmented regresson model allows the outcome of nterest to evolve dfferently over the tme perods before and after the nterventon. A change n level of the outcome after the nterventon may consttute an abrupt nterventon effect due to polcy mplementaton. Segmented regresson analyss also allows analysts to control for varables other than the nterventon that can potentally cause the change n level and/or trend of the outcome of nterest. A range of statstcal approaches can be appled to estmate pre-nterventon level and trend and estmate post-nterventon changes n level and trend. Despte the flexblty of a change-pont model for analysng before-after studes, t has not yet receved much attenton n road safety lterature. The use of general Bayesan technques n road safety studes has become very popular over the past two decades, whch began wth the ntroducton of emprcal Bayes (EB) by Persaud and Hauer (1984). The EB approach has been used wdely and successfully for before-after evaluaton of the effects of mplemented treatments. Typcally, a Posson or negatve bnomal crash model s used for the untreated reference ste and t s combned wth pre-nterventon data at the treated ste to estmate the expected number of crashes that would have occurred wthout nterventon. Ths estmate s then compared wth the observed crash count after nterventon to assess the effect of Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

2 Peer revew stream polcy mplementaton. Ths approach effectvely copes wth several threats to valdty such as the regresson-to-the-mean bas. Extensve lterature on EB methods are avalable (Hauer, 1986; Persaud, 1988; Pendelton et al., 1991; Hauer, 1997) and these authors argue that EB methods have advantages over the classcal approach. Begnnng wth computer and methodologcal advances n the early 1990s, the feasblty of mplementng full Bayesan (FB) approaches has been explored by road safety researchers. The FB approach has been suggested as a useful alternatve to the EB approach due to several advantages such as the ablty to fully account for uncertanty n data. However, there have not been many applcatons of the FB methods n before-after studes manly due to complexty n modellng and estmaton. Out of the few authors who have used FB methods, Carrqury and Pawlovch (2004) demonstrated the feasblty of mplementng a FB framework and compared t wth the EB approach usng a hypothetcal example. Pawlovch et al. (2006) ntroduced a Bayesan herarchcal Posson regresson model wth a change pont to assess whether road dets appear to result n crash reducton. Persaud et al. (2010) compared the EB and FB usng two emprcal applcatons and concluded that the two approaches provde smlar results. However, the FB approach n Persaud et al. (2010) (and Lan et al., 2009) was close to the EB framework. Park et al. (2010) developed a FB multvarate approach to model crash counts for a before-after evaluaton. In ths paper, we perform segmented regresson analyss for a partcular ITS data set and estmate the model usng FB and EB approaches. We compare the estmaton procedures as well as emprcal results wth a prevous study that used only classcal frequentst method of estmaton. The partcular ITS data we consder s the hosptal admsson counts of head and arm njures of cyclsts from New South Wales (NSW), Australa. Australa was the frst country to ntroduce mandatory cycle helmet legslaton and debate on the effcacy of the legslaton on reducng cyclst head njures has been ongong. In NSW, the study by Walter et al. (2011) was the frst to assess the effect of mandatory helmet legslaton on cyclst head njures employng an ITS approach. More specfcally, a negatve bnomal regresson model was estmated to dentfy dfferental changes n head and lmb njury rates at the tme of legslaton and model nference was performed usng classcal maxmum lkelhood method. The authors dentfed evdence of a postve effect of mandatory helmet legslaton on cyclst head njures at a populaton level and concluded that repealng the law cannot be justfed. Rssel (2012) however, made some crtcsms about the data and methodology used n the study, thus questonng the valdty of the study and suggested reanalyss of the data. Walter et al. (2013) responded to those crtcsms and to whch demonstrated that the orgnal analyss s robust. In ths paper, we am to compare the FB and EB methods for assessment of road safety nterventons usng the NSW cyclst hosptalsaton data. In addton to practcal benefts and dsadvantages of each approach, we wll also consder the mplcatons for the conclusons drawn from the orgnal frequentst analyss. Methodology Data The data we used n ths study s taken from Walter et al. (2013). Hosptal admssons were recorded by the NSW Admtted Patents Data Collecton where external causes and dagnoses were classfed exclusvely usng the Internatonal Classfcaton of Dseases, 9 th Revson, Clncal Modfcaton (ICD 9-CM). Head njury admssons were defned as all njures to the skull, face and scalp whle arm njures were defned as all njures to the shoulder grdle, arm, wrst and hand. In our study, eghteen months of pre- and post-law head and arm hosptal admssons were ncluded. Snce the mandatory helmet law n NSW was ntroduced at separate tmes for adults and chldren, the 36 month analyss perod was from July 1989 to June 1992 for adults and January 1990 to December 1992 for chldren. To adjust for cyclcal effects, monthly counts of hosptal admssons were adjusted usng the X11 method separately for adults and chldren and then combned nto a sngle tme seres. The X11 method (Shskn et al., 1967) uses a seres of movng averages and Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

3 Peer revew stream smoothng calculatons to decompose the orgnal seres nto trend, seasonal and rregular components. Model specfcaton Injury counts are usually modelled as Posson or negatve bnomal random varables. We adopt the same model as the one descrbed n Walter et al. (2011) where the log-mean s expressed as a functon of tme perod, njury type and law ndcator. However to facltate Bayesan estmaton, we formulate a herarchcal Posson-Gamma mxture model where the Posson mean s modelled as a Gamma random varable. Ths addtonal parameter accommodates extra varablty n the data and effectvely overcomes the problem wth overdsperson. The negatve bnomal log-lnear regresson of admsson counts s as follows: Y η ~ Posson ( η µ ) η ~ Gamma( α, α) log( µ ) = β + β TIME + β INJURY + β LAW + β TIME INJURY + β TIME LAW 0 + β INJURY LAW + β TIME INJURY LAW + log(exposure), (1) where Y are the admsson counts of head and arm njures. In ths approach, the mean of the Posson dstrbuton s wrtten as a product of two parameters n whch the frst parameter η s a random varable drawn from a Gamma dstrbuton wth mean 1. Ths formulaton s convenent snce we can explctly model the mean of the negatve bnomal dstrbuton µ. Arm njury counts are used as a comparatve tme seres whch acts as a proxy for the unmeasured tme-dependent confoundng snce the cyclng arm njures are subjected to the same threats to nternal valdty but are unaffected by the nterventon. Thus any changes n level and/or slope after the nterventon s an adjusted effect of the nterventon after accountng for these threats. More nformaton on choosng approprate comparatve tme seres can be found n Olver et al. (2013). Three covarates, TIME, INJURY and LAW are ncluded n the above model. TIME s treated as a contnuous varable whch represents the 18 monthly ntervals pre-law and 18 monthly ntervals post-law. The ndcator varable INJURY takes a value of one for a head njury and zero for an arm njury. The varable LAW has a value of zero pror to legslaton and one post-legslaton. Ths model s able to capture temporal trend n njury counts and the nteracton term between TIME and INJURY allows for dfferent tme trends for arm and head njures. Any dfference n the post-law trend and the pre-law trend s captured n the nteracton between TIME and LAW. The nteracton between INJURY and LAW ndcates any dfferental changes n head njures as compared to arm njures. Fnally, the three-way nteracton accounts for the possble dfference n the rate of change of head and arm njures between pre- and post-law tme perods. Full Bayesan Estmaton The basc concept of a Bayesan estmaton procedure nvolves combnng both the observed data nformaton and some pror belef to generate the posteror dstrbuton of the unknown parameters. Pror nformaton s ncorporated nto the model through the use of a pror dstrbuton, whch s a dstrbuton of lkely values for a parameter. In the absence of specfc pror nformaton on a parameter, a non-nformatve pror dstrbuton wth large varance s usually adopted. As compared to the classcal maxmum lkelhood approach, the FB approach offers a number of advantages. Frstly, whle the maxmum lkelhood methods depend on ther asymptotc propertes, the small sample propertes of Bayesan estmaton may allow estmaton of models wth smaller sample szes. For a large sample sze, Bayes estmators are asymptotcally equvalent to maxmum lkelhood estmators under approprate regularty condtons. Secondly, the ablty to nclude pror knowledge on the values of parameters n the model before observng any data appears attractve when the nvestgator wants to utlse fndngs from prevous or smlar studes. Thrdly, a Bayesan Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

4 Peer revew stream approach allows one to specfy very complex model forms where the lkelhood functon s ntractable, partcularly n the context of herarchcal models. Lastly, n the classcal maxmum lkelhood approach, model parameters are consdered as fxed quanttes and the emphass s on obtanng pont estmators for the parameters and ther standard errors. The Bayesan approach on the other hand, s able to provde the entre posteror dstrbutons of outcomes based on whch nference can be made. For example, a common Bayes estmator for a parameter s the mean of ts posteror dstrbuton. Hence n ths respect, a Bayesan approach enables representaton and takes fuller account of the uncertantes related to models and parameter values. A major lmtaton for mplementng of Bayesan approaches n the past was that the posteror dstrbutons were analytcally tractable for only a small number of smple models. In most cases, the posteror dstrbuton does not have a closed form expresson and requres the evaluaton of hgh-dmensonal ntegrals. Smulaton-based MCMC methods became wdely known among statstcans snce the early 1990 s and have now become the routne tool for Bayesan computaton for a wde range of complcated statstcal models. Due to the advancement of computatonal power, the emergence of the statstcal software WnBUGS (Lunn et al. 2000) enables a more straghtforward and feasble mplementaton of the Bayesan methodology. WnBUGS adopts the Gbbs sampler to smulate realsatons of a Markov chan whose lmtng dstrbuton s the posteror dstrbuton of the parameters. Ths conssts of samplng each parameter n turn from ther full condtonal posteror dstrbutons. WnBUGS has a system for choosng the best samplng method to draw from these unvarate dstrbutons. The system frst checks for conjugacy, n whch case the full condtonal dstrbuton reduces analytcally to a well-known dstrbuton and drect samplng usng standard algorthms can be appled. If conjugacy s not detected, but nstead the densty s log-concave, then the adaptve rejecton samplng algorthm s employed. If the target densty s not log-concave, then a Metropols-Hastngs algorthm wll be used. We mplement the FB approach n WnBUGS. To complete the Bayesan estmaton procedure, we need to specfy a pror dstrbuton for each of the model parameters. For the regresson coeffcent parameters, we assgn a normal pror wth mean 0 and a large varance, that s β ~ Normal (0, 1000), = 0, 1, K, 7, to reflect a pror none of varables n the model may be assocated wth the njury count. We assgn a unform pror to the dsperson parameter α of the Posson dstrbuton, that s α ~ Unform (0.5, The hyper-parameters are chosen to reflect gnorance about possble values of parameters. We ran three parallel Markov chans ntated from dfferent startng values for 75,000 teratons. The ntal 20,000 teratons are dscarded as the burn-n perod. We then take smulated values every 20 th teraton to reduce autocorrelaton. A fnal sample of sze 9,000 (pooled over three chans) almost ndependent draws were obtaned for each parameter n the model and are used for posteror nference. Trace plots, autocorrelaton plots and posteror densty plots are examned carefully to ensure convergence s acheved for each parameter. Convergence of the parameters s also extensvely examned usng the Brooks, Gelman and Rubn convergence plots (Gelman and Rubn, 1992; Brooks and Gelman, 1998). Emprcal Bayesan Estmaton EB s a related approach that has been used extensvely n the analyss of traffc safety data, partcularly for the before-after evaluaton of road safety treatments. An EB approach s related to the full Bayesan approach n the sense that pror nformaton s combned wth current data to obtan an estmate. However, dfferent to the FB approach, the parameters of the pror dstrbuton 200) Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

5 Peer revew stream are estmated from exstng data and are used n subsequent steps assumng there s no uncertanty related to the parameter. For example, n evaluatng the effect of some treatments n the number of road crashes, the pror nformaton s obtaned from comparson stes smlar to those treatment stes. An estmate of the sample mean or a functon of the mean (commonly known as the Safety Performance Functons (SPF)) can be calculated and s then combned wth the ste-specfc crash count to acqure an mproved estmate of a ste s expected crash count. However, ths mples that n the EB approach, populaton-level estmates do not contrbute to the uncertanty n the ste specfc estmate and as a result, ntersecton-level estmates may have lower standard errors. The FB approach on the other hand, better accounts for ths uncertanty by recognsng that the populaton-level estmates follow certan dstrbutons whch n turn depend on hgher-level parameters. Also, mplementng a FB approach does not requre the estmaton of SPF, whch s one of the largest lmtatons of an EB approach. French and Heagerty (2008) appled a partcular EB procedure whch nvolves fttng a regresson model pror to polcy nterventon and use the resultng model to form a trajectory of outcomes n tme perods subsequent to the polcy nterventon. Ths s followed by contrastng post-nterventon observatons wth ther expected outcomes (predcted by the regresson model) under the absence of a polcy nterventon. Regresson-based models have the flexblty to account for temporal trends as well as adjust for mportant covarates and ther nteractons. The dfferences between these expected outcomes and the data observed after the polcy change are then averaged to estmate the polcy effects. Formal statstcal tests such as the two-sample t-test, can be used to examne whether there exsts a sgnfcant dfference n polcy effect between the prmary target group of nterest and ts comparson group. For EB methods, we follow the approach descrbed n French and Heagerty (2008) snce t s an EB procedure specfcally desgned for drawng nference from longtudnal partal crossover data. The dea s to frst ft a regresson model to capture any temporal trend pror to polcy nterventon and then use ths model to forecast the expected outcome subsequent to the polcy mplementaton. More specfcally we ft the followng model to the data pror to the nterventon Y ~ Posson ( η µ ) η ~ Gamma ( α, α,) log( µ ) = δ + δ TIME + δ INJURY + δ TIME INJURY 0 1 PRE 2 3 PRE + log(exposure) (2) where Y are the admsson counts of head and arm njures pror to the helmet legslaton, TIME PRE s 18 monthly ntervals pre-law and INJURY s agan an ndcator varable takng a value of one for a head njury and zero for an arm njury. These expected outcomes predcted from the model are then subtracted from the observed data after the polcy change to create contrasts,. That s, log( ) log( ˆ = Y µ ) Smple statstcal analyss such as a t-test can be used to assess the sgnfcance of a polcy effect. Moreover, we ft a smple lnear regresson to these contrasts to obtan estmates of baselne level and slope change after the nterventon as well as dfferental changes n head njures as compared to arm njures after the law. That s, = δ 4 + δ 5TIMEPOST + δ 6INJURY + δ 7TIMEPOST INJURY (3) where TIME POST s the 18 monthly ntervals post-law. δ 4 and δ 5 represent the baselne level for arm njures and slope change after the polcy nterventon respectvely, whle δ 6 and δ 7 effectvely capture dfferental changes n head and arm njures post-law. A negatve value for δ 6 suggests that the head njures decreased more than arm njures after the law. Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

6 Peer revew stream Results Monte Carlo estmates of the posteror mean, standard devaton, 2.5 th and 97.5 th percentles of the margnal posteror dstrbuton of each parameter were obtaned from the draws and they are reported n Table 1. Maxmum lkelhood estmates obtaned from Walter et al. (2011) are ncluded n Table 1 as a comparson. There s lttle dfference between the estmates obtaned under the two methods. The full Bayesan 95% credble ntervals are reasonably close to the classcal 95% confdence ntervals, although they are generally wder than the correspondng confdence nterval for all parameters. Ths s consstent wth the Bayesan concept as the pror dstrbutons take nto account the varablty of the unknown parameters and the Bayesan credble nterval reflects both parametrc and samplng uncertanty, whereas the classcal confdence nterval reflects only samplng uncertanty. Smlar to the maxmum lkelhood estmate, we obtaned a negatve estmate for the coeffcent β 3, whch suggested that overall njury count decreased followng the helmet law. Table 1. Negatve bnomal model estmates usng frequentst MLE, FB and EB methods Varable Intercept (β 0 or δ 0 ) TIME (β 1 or δ 1 ) INJURY (β 2 or δ 2 ) LAW (β 3 or δ 4 ) TIME INJURY (β 4 or δ 3 ) TIME LAW (β 5 or δ 5 ) INJURY LAW (β 6 or δ 6 ) TIME INJURY LAW (β 7 or δ 7 ) Frequentst MLE Full Bayesan Emprcal Bayes Estmate Estmate Estmate (95% CI) (95% CI) (95% CI) ( , ) ( , ) ( , ) (-0.019,0.009) (-0.020,0.009) (-0.018,0.007) (-0.128,0.272) (-0.136,0.287) (-0.111,0.255) (-0.318,0.093) (-0.330,0.106) (-0.314,0.024) (-0.022,0.016) (-0.023,0.017) (-0.021,0.014) (-0.005,0.034) (-0.006,0.036) (0.000,0.033) (-0.618,-0.027) (-0.635,-0.014) (-0.514,-0.064) (-0.018,0.038) (-0.021,0.040) (-0.016,0.030) We also obtaned a sgnfcant negatve estmate for β 6, whch ndcates dfferental changes n head njures as compared to arm njures after the helmet legslaton. In ths case, a sgnfcantly negatve estmate suggested that the head njures dropped by more than arm njures at the tme of legslaton, provdng evdence for a legslaton attrbutable beneft. Head and arm njury counts and predcted counts estmated usng the FB method are shown n Fgure 1. The graph looks almost dentcal to the one presented n Walter et al. (2011), snce very smlar parameter estmates were obtaned. The most notceable characterstc s the downward step n head njury counts havng a larger magntude than the correspondng arm njury counts. Results from the EB approach are qute smlar to those obtaned usng classcal MLE and FB methods, especally n terms of parameter estmates. Apart from the ntercept, only δ 5 and δ 6 are sgnfcant and δ 5 s only margnally sgnfcant. The sgnfcant negatve estmate of δ 6 has smlar mplcaton as β 6, that s, the decrease n head njury counts s more than that for arm njury wth the change n helmet wearng legslaton. The drop n arm njury after the law s non-sgnfcant snce Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

7 Peer revew stream the 95% confdence nterval or credble nterval cover the value 0 for the estmate of β 3 or δ 4, respectvely. Snce the mandatory helmet law s a safety nterventon amed specfcally at head njures but not lmb njures and we assume that other changes to the cyclng envronment would affect both head and arm njures, the sgnfcant dfferental decrease n njury counts s most lkely due to the compulsory helmet legslaton. Usng FB and EB methods, the estmated legslaton attrbutable decrease n head njury are 27.6% (1-exp(-0.323)) and 26.1% respectvely, whch are comparable to the estmated 27.5% found n the orgnal study (Walter et al., 2011). Therefore, usng two dfferent estmaton methods, namely FB and EB, we confrmed the orgnal fndng, that there s evdence for a postve effect of the compulsory cycle helmet legslaton on head njures. Note that the confdence ntervals of all parameters are the tghtest n the EB method among three estmaton methods. However, the parameter standard errors estmated n the EB method are not drectly comparable to those n classcal MLE and FB methods snce smpler regresson models (2) and (3) (wth less parameters) are ftted separately to pre- and post- nterventon data as opposed to the full model (1). Fgure 2 shows the plot of the ftted log-lnear regresson model usng pre-law data (sold lne) and predcted outcomes (dashed lnes) after the polcy nterventon separately for arm and head njures. At frst sght, the post-law arm njures seem to fluctuate around the trend formed by the expected outcomes. Result from the t-test suggests that the mean contrast for arm njures s not sgnfcantly dfferent from zero ( ˆ δ 4 =-0.145, s.e.=0.086, p=0.102). On the other hand, the observed post-law head njury counts are, n general, less than the predcted outcomes and the one sample t-test for the head contrast rejects the null hypothess that the mean s zero ( ˆ δ ˆ 4 + δ 6 = , s.e.=0.086, p= ). The dfference between head and arm contrasts, ˆ δ 6, s also statstcally sgnfcant ( ˆ δ 6 =-0.302, s.e.=0.122, p=0.019). Hence, relatve to arm njures, there s an estmated 26.1% drop n head njures. Fgure 1. Cyclst head vs. arm njury counts and ftted model for 18 months pror and post helmet legslaton Fgure 2. Cyclst head and arm njury counts wth pre-polcy estmaton (sold lne) and postpolcy predcton(dashed lne) Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

8 Peer revew stream Dscusson In ths study, we ft a herarchcal negatve bnomal regresson model to the hosptal admsson of head and arm njury counts, from a 36 month perod centred at the mandatory helmet legslaton n NSW. The explanatory varables n the segmented regresson ncluded njury type, tme, ndcator for the polcy nterventon and all two-way and three-way nteractons between the varables. The data and the model was prevously analysed n Walter et al. (2011) n whch they estmated sgnfcant decreased head njury rates relatve to arm njury rates at the tme of legslaton among cyclsts and concluded ths addtonal beneft was attrbuted to compulsory helmet legslaton. The model estmaton was done usng a classcal maxmum lkelhood method. In ths study, we reanalysed the data usng two dfferent estmaton methods, namely full Bayesan and emprcal Bayes approaches. For the FB methods, nonnformatve prors were used for all parameters n the model, and samplng from the posteror dstrbuton was accomplshed through MCMC methods mplemented n WnBUGS. Parameters estmated usng a FB method are very smlar to those obtaned under MLE, although the 95% credble ntervals are (as expected) wder than the correspondng 95% confdence ntervals as to account for the uncertanty n the parameters. For the EB method, we adopted the procedure descrbed n French and Heagerty (2008) specfcally desgned for evaluatng longtudnal effects of nterventons. A negatve bnomal regresson model was ft to head and arm njury counts pror to the nterventon and the resultng model was used to predct expected outcomes under the absence of the polcy change. These expected outcomes w then compared wth the njury counts observed after the polcy change to compute contrasts. These contrasts were then analysed by a lnear regresson model to estmate the (dfferental) changes n the level and slope of head and arm njures after the law. Parameter estmates usng the EB method are comparable to classcal MLE and FB estmates, Persaud et al. (2010) also found that EB and FB for treatment evaluaton studes gve comparable results. As ponted out n French and Heagerty (2008), the standard error calculaton for the contrasts s not straghtforward. Our t-tests are performed based on the assumpton that varaton due to estmaton of pre-polcy model parameters do not contrbute to post-polcy parameter estmaton. Hence, standard error computaton s smplfed and an estmate of the varance of contrast s smply var( ). A more complcated estmator for the varance of contrast can be found n the Supplementary Materal of French and Heagerty (2008). Regardless of the estmaton lmtaton n EB, we estmated a sgnfcant decrease n head njures more than arm njures at the tme of legslaton and the estmated effect had a smlar magntude to that estmated usng MLE and FB approaches. Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

9 Peer revew stream Generally speakng, when we have a large amount of nformaton avalable, the choce of whether model parameters are estmated usng a classcal frequentst, FB or EB framework makes lttle dfference. However, when the number of observatons s small, both classcal and Bayesan paradgms can result n nvald conclusons, and t s exacerbated n EB snce only the prenterventon data are used to construct the model whch s then used to predct expected outcomes. Furthermore, the FB method s more flexble than the classcal frequentst and EB methods as pror dstrbutons are used to better account for uncertanty n the sample. In our study, the classcal MLE and EB approaches rely on the assumpton of a negatve bnomal dstrbuton of njury counts n the estmaton process. The FB approach on the other hand, s able to model ths as a herarchcal Posson-Gamma mxture dstrbuton. Ths herarchcal structure allows other dstrbutons, such as the Posson-Lognormal dstrbuton, to be mplemented and nvestgated n a smlar fashon. Major crtcsms for the FB approach nclude the choce of pror dstrbutons and the computatonal cost nvolved n samplng from the posteror dstrbuton. Typcally, parallel Markov chans are run for a large number of teratons to ensure all parameters have converged and successve terates are not correlated. It s worth mentonng that the EB method employed n our study s not the same approach commonly used for analysng traffc safety data such as n Carrqury and Pawlovch (2004). Generally, n a parametrc EB procedure, the model s formulated n the same way as a Bayesan herarchcal model and pror dstrbutons are assgned to model parameters. The pror dstrbutons are assumed to belong to a famly of dstrbutons ndexed by a low-dmensonal parameter (hyperparameter). An estmate of ths hyperparameter can be computed usng the margnal dstrbuton of all data, usually as a MLE. For our full model specfed n (1), the parameter space s θ = β, β, β, β, β, β, β, ) and the pror dstrbutons are ( β7 β ~ Normal ( µ, σ α ~ Unform ( a α β, b α 2 β ) ), = 0, 1, K, 7, Hence to obtan MLE estmates for the hyperparameters µ β, σβ 2, aα, bα for each model parameter, we need to maxmse the margnal dstrbuton of the data whch s 9-dmensonal ntegral snce we need to ntegrate out model parametersθ. Snce the FB method s readly mplementable n ths case, we feel that the use of a parametrc EB method s not justfed for our segmented regresson model. Snce the data and model were the same as n Walter et al. (2013), some lmtatons assocated wth the data and the analyss method also apply n ths case. For nstance, due to lack of populaton level exposure and helmet wearng data, we used arm njures as a comparson group to cope wth ths problem. However, there may be more than one such comparson group (for example, cyclst leg njures) and t remans unclear how to choose the most approprate comparson group. Furthermore, our study s lmted by the amount of data avalable pror to helmet legslaton and thus conductng a smlar study usng data from a jursdcton wth more pre-law data s hghly recommended. Lastly, t s worthwhle to emprcally assess the relatve performance of these methods when they are appled to more complcated tme seres models and state space models. Ths comparson of three estmaton methods for road safety nterventon assessment showed that MLE, FB and EB gve smlar results for large samples. The agreement between the methods for the example data also confrmed the orgnal fndngs of Walter et al. (2011, 2013). Generally speakng, the FB method ncorporates parameter uncertanty va pror dstrbutons and does not depend on Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

10 Peer revew stream asymptotc propertes, whch are the advantages over the classcal MLE. However, the FB method may be computatonally costly to mplement, especally when WnBUGS s not used. The EB method s a relatvely smpler method but ssues may arse for small samples. Hence n performng detaled polcy change evaluatons, analysts must be mndful of the mportant dfferences between these estmaton approaches both n terms of the mplementaton procedure and nference. Acknowledgements Data for ths study were provded under agreement by the NSW Mnstry of Health, Centre for Epdemology and Evdence. References Brooks, S., Gelman, A. (1998). General methods for montorng convergence of teratve smulatons. Journal of Computatonal and Graphcal Statstcs 7, Carrqury, A, Pawlovch, M.D. (2004). From emprcal Bayes to full Bayes: methods for analysng traffc safety data. French, B., Heagerty, P.J. (2008) Analyss of longtudnal data to evaluate a polcy change. Statstcs n Medcne 27, Gelman, A., Rubn, D. (1992). Inference from teratve smulaton usng multple sequences. Statstcal Scence 7, Hauer, E. (1986). On the estmaton of the expected number of accdents. Accdent Analyss and Preventon 18, Hauer, E. (1997). Observatonal Before-After Studes n Road Safety: Estmatng the Effect of Hghway and Traffc Engneerng Measures on Road Safety. Pergamon Press, Elsever Scence, Ltd., Oxford, Unted Kngdom. Lan, B., Persaud, B., Lyon, C., Bhm, R. (2009). Valdaton of a Full Bayes methodology for observatonal before-after road safety studes and applcaton to evaluaton of rural sgnal conversons. Accdent Analyss and Preventon 41, Lunn, D.J., Thomas, A., Best, N. and Spegelhalter, D. (2000). WnBUGS -- a Bayesan modellng framework: concepts, structure, and extensblty. Statstcs and Computng 10, Olver, J.,, J.J.J., Walter, S., Grzebeta, R. (2013). On the use of emprcal Bayes for comparatve nterrupted tme seres wth an applcaton to mandatory helmet legslaton. Submtted to Road Safety, Research, Polcng and Educaton Conference, Park, E.S., Park, J., Lomax, T.J. (2010). A fully Bayesan multvarate approach to before-after safety evaluaton. Accdent Analyss and Preventon 42, Pawlovch, M.D., Wen, L., Carrqury, A., Welch, T. (2006). Iowa s experence wth road det measures: use of Bayesan approach to assess mpacts on crash frequences and crash rates. Transportaton Research Record 1953, Pendelton, O.J., Ofela, G., Horaco, D. (1991). Applcaton of new accdent analyss methodologes Volume II A Users Manual for BEATS. FHWA-RD Federal Hghway Admnstraton (FHWA), McLean, VA. Persaud, B.N. (1988). Do traffc sgnals affect safety? Some methodologcal ssues. Transportaton Research Record 1185, Persaud, B.N., Hauer, E. (1984). Comparson of two methods for debasng before-and-after accdent studes. Transportaton Research Record 975, Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland

11 Peer revew stream Persaud, B., Lan, B., Lyon, C., Bhm, R. (2010). Comparson of emprcal Bayes and full Bayes approaches for before-after road safety evaluatons. Accdent Analyss and Preventon 42, Persaud, B., Lan, B., Lyon, C., Bhm, R. (2010). Comparson of emprcal Bayes and full Bayes approaches for before-after road safety evaluatons. Accdent Analyss and Preventon 42, Rssel, C. (2012). The mpact of compulsory cycle helmet legslaton on cyclst head njures n New South Wales, Australa: A rejonder. Accdent Analyss and Preventon 45, Shskn, J., Young, A.H., Musgrave, J.C. (1967). The X-11 Varant of the Census Method II Seasonal Adjustment Program. In: Techncal Report 15. US Department of Commerce, Bureau of the Census. Walter, S., Olver, J., Churches, T., Grzebeta, R. (2011). The mpact of compulsory cycle helmet legslaton on cyclst head njures n New South Wales, Australa. Accdent Analyss and Preventon 43, Walter, S., Olver, J., Churches, T., Grzebeta, R. (2013). The mpact of compulsory cycle helmet legslaton on cyclst head njures n New South Wales, Australa: A response. Accdent Analyss and Preventon 52, Wagner, A.K., Soumera, S.B., Zhang, F., Ross-Degnan, D. (2002). Segmented regresson analyss of nterrupted tme seres studes n medcaton use research. Journal of Clncal Pharmacy and Therapeutcs 27, Proceedngs of the 2013 Australasan Road Safety Research, Polcng & Educaton Conference 28 th 30 th August, Brsbane, Queensland