Studies With Staggered Starts: Multiple Baseline Designs and Group-Randomized Trials

Transcription

1 Stuies With Staggere Starts: Multiple Baseline Designs an Group-Ranomize Trials Dale A. Rhoa, MAS, MS, MPP, Davi M. Murray, PhD, Rebecca R. Anrige, PhD, Michael L. Pennell, PhD, an Erinn M. Hae, MS The ranomize controlle trial (RCT) is the gol stanar for causal inference for iniviual-level interventions. 1 When interventions are applie at the group level an outcomes are measure at the iniviual level, the clusteror group-ranomize trial (GRT) is the gol stanar for causal inference. 2,3 However, GRTs are often costly an time-consuming, prompting researchers to look for alternatives. One alternative that has been suggeste is the multiple baseline esign (MBD), which has a venerable history in eucation an applie behavior research with iniviual-level interventions but is relatively new in public health research with group-level interventions. 4-6 The MBD makes repeate measurements over a perio of time an introuces a sustaine intervention on a staggere scheule; intervention effects synchronize with the staggere start times provie evience for causal inference. Hawkins et al. escribe the MBD as a viable alternative to the RCT an suggeste that it will be lower cost, use smaller sample sizes, an still be statistically rigorous. 5 Biglan et al. suggeste complementary roles, with MBDs use to evelop an sort through potentially effective intervention methos, followe by evaluation in RCTs both to test efficacy an to etermine the extent of generalizability. 4 We review the structural features that have mae MBDs useful in other fiels an consier whether similar success is likely in public health. We also compare the statistical power of MBDs an GRTs. METHODS We reviewe the MBD literature to ientify key structural features. We reviewe recent suggestions that the MBD be aopte in public health research. Finally, we reviewe the literature on GRTs with staggere starts an compare the power of that esign with the more traitional parallel esign. Objectives. Multiple baseline esigns (MBDs) have been suggeste as alternatives to group-ranomize trials (GRT). We reviewe structural features of MBDs an consiere their potential effectiveness in public health research. We also reviewe the effect of staggere starts on statistical power. Methos. We reviewe the MBD literature to ientify key structural features, recent suggestions that MBDs be aopte in public health research, an the literature on power in GRTs with staggere starts. We also compute power for MBDs an GRTs. Results. The features that have contribute to the success of small MBDs in some fiels are not likely to translate well to public health research. MBDs can be more powerful than GRTs uner some conitions, but those conitions involve assumptions that require careful evaluation in practice. Conclusions. MBDs will often serve better as a complement of rather than as an alternative to GRTs. GRTs may employ staggere starts for logistical or ethical reasons, but this will always increase their uration an will often increase their cost. (Am J Public Health. 2011;101: oi: / AJPH ) RESULTS AND DISCUSSION We use the Murray convention, in which conition refers to the stuy arm, group refers to a collection of participants assigne together to a conition, an member refers to an iniviual participant. 3 Experiments involve a single intervention, so at any particular time a group will be in either the control conition or the intervention conition. The MBD has several variations with an without ranomization; we focuse on MBDs that introuce the intervention to groups on a staggere scheule an in a ranom orer. Before any intervention occurs, the outcome is measure in each group. Then the intervention is initiate in 1 or more groups while the others continue in the control conition. After sufficient time has passe for the intervention to affect the outcome in the first group(s), outcome measurements are conucte in all groups an the intervention is introuce in 1 or more aitional groups. This procees until all groups receive the intervention. Once a group starts the intervention, it remains in that conition until the en of the stuy. Figure 1 illustrates an MBD involving 14 measurement perios an 11 groups with 1 group crossing to intervention at each start time. If every group shows a similar change after crossing to the intervention conition an oes not change at other times, the experiment provies compelling evience that the changes resulte from the intervention. Even with a limite number of groups, consistently replicate effects can be persuasive, an that is why some researchers have been rawn to the MBD. The term group-ranomize trial covers a broa array of esigns in which groups are ranomize to conitions an measurements are taken from the members of those groups. In a parallel GRT, baseline measurements are conucte in every group an then the intervention commences simultaneously in half of the groups; the remainer serves as controls an oes not receive the intervention uring the stuy. One or more aitional measurements occur in all groups. MBDs for public health interventions always have multiple members per group, an 2164 Research an Practice Peer Reviewe Rhoa et al. American Journal of Public Health November 2011, Vol 101, No. 11

2 Note. The intervention commences on a staggere scheule across groups with the iea that a measureable effect in the outcome variable(s) will follow a similar pattern. FIGURE 1 Conceptual iagram of the multiple baseline or steppe wege esign. ranomization is use to select the orer in which groups start the intervention. Because these 2 conitions efine a GRT, the MBDs we consiere are GRTs as well. These MBDs are also calle steppe wege esigns (SWDs). The name steppe wege originate with the Gambia Hepatitis Stuy an refers to the wege shape of the intervention timeline across groups, as epicte in Figure 1. 7 Brown an Lilforrevieweaozensteppewegestuies; the state reasons for using staggere starts inclue easier implementation of the intervention, ethical requirements to give the intervention to all participants, a esire to estimate trens over time, an a esire to use participants as their own controls. 8 Structural Features of Multiple Baseline Designs The following features of the MBD have contribute to its success in fiels such as applie behavior research an eucation It is important to consier whether they are likely to be as effective in public health research. Designs with staggere start times can be persuasive if the timing of the effects is synchronize with the timing of the introuction of the intervention. Furthermore, they guar the internal valiity of the stuy by ruling out the possibility that a single external event (e.g., a celebrity cancer iagnosis or a change in legislation) coul explain the results. In some cases it woul be prohibitively expensive or impossible to start the intervention in half of the groups simultaneously. This is especially true when a single team trains all the intervention personnel or when groups are separate by large geographical istances. With staggere start times, each group can start the intervention shortly after being traine. Many measurements may be require to reliably estimate baseline trens an intervention effects within each group. In some situations, ata collection an reporting will be stanarize processes that occur with fortunate frequency. Otherwise, the cost of many measurements may make MBDs prohibitively expensive. Ranomization of the orer in which the groups start the intervention protects against bias associate with reainess or eagerness to participate. Each group experiences a single transition from the baseline conition to the intervention conition. MBDs are typically use in settings in which it woul not be ethical, healthy, or practical to withraw the intervention or in which it is unrealistic to expect participants to revert to their pretreatment conition quickly after the withrawal. The time between intervention onsets in ifferent groups is long enough for the intervention to show its full effect in the most recently treate group. Treatments with a long latency are not goo caniates for the MBD. Analysis Methos for Multiple Baseline Designs Matyas an Greenwoo reporte that 75% of the experiments they examine in the Journal of Applie Behavior Analysis looke for effects that shifte the mean outcome by more than 5 stanar eviations; 50% looke for effects larger than 10 stanar eviations. 20 Thus it is no surprise that MBDs in applie behavior research have traitionally been analyze by simple visual inspection for a substantial change in within-unit outcomes shortly after the November 2011, Vol 101, No. 11 American Journal of Public Health Rhoa et al. Peer Reviewe Research an Practice 2165

3 intervention starts. 10,12 This metho works best if the intervention effects are large. More recent work has explore formal hypothesis testing both within an between groups to etect more moest effects. The methos escribe inclue Box-Jenkins time series analysis, interrupte time series analysis, ranomization tests, an multilevel moeling. 11,21-24 Approaches to the analysis of steppe wege GRTs are complex an are escribeinbrownanlilfor, 8 Brown et al., 25 Hayes an Moulton, 26 an Hussey an Hughes. 27 Optimistic Claims for Multiple Baseline Designs in Public Health Research Hawkins et al. suggeste, Conceptually a multiple baseline esign may use as few as two groups to test an intervention, reucing costs an alleviating some of the ifficulties in obtaining a sufficient sample size require in RCTs. 5(p163) Success in 2 staggere groups may ispel the counterfactual suggestion that success resulte from a single external event, an it may provie justification for testing in a broaer set of groups. But numerous examples in the MBD literature emonstrate intervention success in some, but not all, groups. 4,11 Given the small effect sizes that are typical in public health interventions, mixe results will be ifficult to interpret. Biglan et al. observe that in aition to knowing whether the intervention was successful or not, ata on the ifferential effects of alternative forms of the intervention implemente in iniviual communities an implemente at ifferent times will help avance knowlege on which components influence [the stuy outcome]. 4(p39) With iniviual-level interventions, MBD researchers may be able to guess at the reasons for their intervention failures if they interact with the stuy participants frequently. 28 In public health stuies, it will be impractical for any single investigator to have contact with all stuy participants, an those participants will not be observe often. In aition, community-base investigators will have a multitue of confouning factors to consier. Data on ifferential effects will be confoune with the community effects, the times of implementation, ifferent baseline trens, an possibly ifferent implementation teams. Investigators will have ifficulty attributing ifferential effects to specific features of the intervention. Finally, Hawkins observe that [the MBD] is a viable alternative to the RCT. 5(p167) We believe that an MBD woul nee to have many of the features of an RCT to provie compelling evience in public health stuies. Moest effects require many groups, an simple visual inspection of the time series of results will usually not suffice. There are conitions uner which SWDs (MBDs for groups) can be more powerful than parallel GRTs with the same number of groups. However, the SWD esign will often require more measurements an more time than the parallel GRT, sometimes substantially more, so stuy esigners must carefully consier the traeoffs involve. Unless the intervention s effect sizes are large, we o not share Hawkins s optimism that the SWD or MBD can provie substantiallyfasterorlessexpensiveevience than the parallel GRT for causal inference for group-level interventions. Steppe Wege Versus Parallel Design There may be situations in which the stuy esigner is able to choose between an SWD an parallel GRT. We compare the statistical power of those esigns an highlight some consierations that might influence the choice. Several questions interest us: Which esign is more statistically powerful given the same number of groups an measurement perios? What are the esign parameters that influence the answer? In cases in which the SWD is more powerful than the parallel esign, how many fewer groups or measurements can be use without sacrificing statistical power? Brown et al. provie an illustrative Poisson event example in which the SWD intervention effect was estimate using a weighte average of between-groups ifferences at each measurement time point. 25 Those authors acknowlege the point raise by Hayes an Moulton: the number of groups in intervention an control conitions are markely unbalance near the beginning an en of the steppe wege stuy, an that leas to loss of efficiency for this analytic moel. 26 Hussey an Hughes escribe the orthogonal complement to that approach: analyzing an SWD with an estimator that is a weighte average of within-groups ifferences when it is reasonable to assume that there is no time effect. 27 The same problem applies in that case: some groups spen most of the stuy in the control conition an some spen most of the stuy in the intervention conition, an that leastolossofefficiencyforthisanalyticmoel. Hussey an Hughes propose an alternative weighte least squares estimator for SWDs that combines information from within an between groups. 27 This estimator is more complicate but usually more efficient than either the purely between-groups or purely within-groups estimators, an we use it to make observations about conitions uner which the SWD can be more powerful than a parallel GRT. They assume a mixe effects moel that escribes the iniviual-level response Y ijk for member k in group i an measurement perio j (i in1,..., I; j in 1,..., T ), ð1þ Y ijk ¼ l ij 1 a i 1 b j 1 X ij h 1 " ijk ; where l ij is the average response in group i uring measurement perio j, a i ; N(0, s 2 ) is a ranom effect for group i, b j is a fixe effect for time perio j, h is the intervention effect, X ij takes the value 1 if group i is in the intervention conition uring perio j an 0otherwise,an" ijk ;N0; r 2 e.unerthis moel, q ¼ s 2 =ðs 2 1r 2 e Þ is the intraclass correlation coefficient (ICC), escribing the correlation between 2 iniviuals in the same group, either at the same point in time or across 2 ifferent measurement occasions. This moel can be use to escribe parallel GRTs, SWDs, an traitional crossover esigns by varying the pattern of X ij values. Figure 2 shows the pattern of Xs for a parallel GRT esign with a single baseline an 3 follow-up measurements an for a SWD with 4 measurements; both esigns involve 6 groups. Hussey an Hughes observe that if s 2 an r 2 r 2 e =m are known, where m is the number of members per group, then estimates of the fixe effects can be obtaine with a weighte least squares analysis of the cluster means. 27 If Z is the IT (T + 1) esign matrix corresponing to the parameter g =(l, b 1, b 2,..., b T 1, h)for the SWD, then ^g =(Z9V 1 Z ) 1 (Z9V 1 Y )an 1, thecovariancematrixof^g ¼ Z 0 V 1 Z where V is an IT IT block iagonal matrix. The estimate of the intervention effect, ^h,isthet +firstelementof^g. EachT T block within V escribes the correlation structure between the repeate (in time) cluster means an has r 2 + s 2 in every element along the iagonal an s 2 in the off-iagonal elements Research an Practice Peer Reviewe Rhoa et al. American Journal of Public Health November 2011, Vol 101, No. 11

4 Perio 1 an half the groups (I/2) transition to intervention in Perio 2. For this esign, Equation 3 may be rewritten in a form that is vali for parallel GRTs with T >2: (5) Var ^hgrt ¼ 41 q ð Þ r2 e 1 s2 ½1 1 ðmt 1ÞqŠ : mi ðt 1Þ½1 1 ðm 1ÞqŠ Note. Intervention an control conitions are represente with both shaing an the 0/1 X ij nomenclature of Hussey an Hughes. 27 A 0 means the group is in the control conition, an a 1 means the group is in the intervention conition. FIGURE 2 Conceptual comparison of (a) steppe wege an (b) parallel group-ranomize trial esigns. Hussey an Hughes suggeste that power can be compute for the weighte least squares analysis using 0 1 ð2þ power ¼ U h alternative rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Z 1 a=2 C A ; Var ^h where U is the cumulative stanar normal istribution function, Z 1 a/2 is the (1 a/2)th quantile of the stanar normal istribution function, an h alternative is the hypothesize treatment effect. 27 They gave a close form expression for Var ^h : (3) Var ^h ¼ ðiu W mr 2 e I mr2 e 1 T s2 Þmr 2 e 1 ð U 2 1 ITU TW IV Þs 2 where all groups are in the control conition in Perio 1 an then the same number of groups, k=i/(t 1), crosses over from control to intervention in each remaining perio. In that case, Equation 3 may be rewritten as (4) Varð^h SWD Þ¼ 6ðT 1 Þð1 q Þ r2 e 1 s2 ½1 1 ðmt 1ÞqŠ n h i o : mit ðt 2Þ 11 mt1 ð 1Þ 2 1 q Similarly, we constrain a parallel GRT to be a stuy with T measurement perios in which all I groups are in the control conition in Using Equations 4 an 5, we can compare the power of the parallel GRT with the SWD uner the Hussey an Hughes mixe effects moel. Figure 3 plots Varð^h GRT Þ=Varð^h SWD Þ as a function of the ICC (q) an number of time perios (T ) with m =50 iniviuals per group. This result hols for any number of groups entering the intervention in each time perio in the SWD. For T=3 the parallel GRT has a smaller variance for all values of the ICC. For T >3 the SWD yiels a smaller variance than the parallel GRT for all but the very lowest values of the ICC. We can also use Equations 4 an 5 to investigate the extent to which the SWD can employ fewer or smaller groups, measure on fewer occasions, or look for a smaller effect size than that of the parallel GRT without compromising power. Hussey an Hughes where V ¼ P P 2, P P 2; X ij W¼ X ij an i j j i U ¼ P X ij. Equation 3 is general an may ij be solve for any pattern of 0 an 1 X ij values. They also use values of X that increase from 0 to1graually to inicate that the intervention effect sometimes takes more than 1 measurement perio to be fully realize. Although we have not one so here, it is possible to use the same approach to moel problems with intervention aherence in long stuies. Note that although the weighte least squares estimator works with values of X between 0 an 1, Equation 3 is only vali when X takes the value 0or1. For the remainer of this section we efine a balance SWD to be a stuy of T perios with I groups, each of which has m members, Note. GRT=parallel group-ranomize trial; ICC=intraclass correlation coefficient; SWD=steppe wege esign; T = number of measurement occasions. FIGURE 3 The variance of the estimator using a parallel group-ranomize trial esign from Equation 5 ivie by the variance of the estimator using a steppe wege esign from Equation 4, assuming equal numbers of groups an measurement occasions, an 50 persons per group for various numbers of measurement occasions an ICCs from 0.0 to 0.5. November 2011, Vol 101, No. 11 American Journal of Public Health Rhoa et al. Peer Reviewe Research an Practice 2167

5 escribe a cross-sectional SWD to evaluate an intervention to reuce the prevalence of chlamyia infections in 24 counties of Washington State. 27 We expane on that example to explore the relative sizes of parallel GRTs an SWDs that have comparable power. We start with a parallel GRT with 3 measurement perios an 24 counties. In Perio 1, all counties are in the control conition. In Perio 2, 12 ranomly selecte counties receive the intervention an continue to o so through Perio 3. The remaining 12 counties stay in the control conition. Equations 5 an 2 inicate that a parallel GRT with 3 measurement occasions an 24 groups woul have 81% power to etect a 50% reuction in prevalence, assuming prevalence=l=0.05, m=100, T=3, I=24,q=0.01, r 2 e ¼ ðlþð1 lþð1 qþ; s2 =(l)(1 l)q, h alternative = 0.025, an a=0.05. Again, the parallel GRT has a smaller variance than a SWD with the same number of groups when T=3; Equations 4 an 2 inicate that a balance SWD where T=3 an I=24 woul have only 69% power to etect the same 50% reuction in prevalence. Table 1 shows the total number of groups necessary for the SWD to reach or excee the power of the parallel GRT uner various values of ICC. The ICC=0.010 row shows that a SWD using as few as 8 counties coul be use in the Washington chlamyia project an achieve the same power as the parallel GRT with 24 counties. Note, however, that the ICC = SWD where I SW =8 woul require 9 measurement perios, which is 3 times as many as the parallel GRT. Both woul require 72 group-level measurement efforts: 8 counties on 9 occasions for the SWD an 24 counties on 3 occasions for the parallel GRT. Table 1 uses a superscript to ientify cells in which SWDs woul require fewer total measurements than woul the parallel GRT. Time is often as important a consieration as cost. The parallel GRT in the T GRT column of Table 1 woul require 3 measurement perios. To achieve the same level of power, the SWDs woul always require aitional measurement perios. ICC values <0.050 are common in public health; with such ICCs, appreciably more measurement perios may be require for the SWD to achieve the same power as the parallel GRT. Thus when choosing between the parallel GRT an SWD, investigators woul nee to consier the time require to complete each esign in aition to the other factors. Table 1 also lists the power of those stuy esigns to etect a 50% reuction in prevalence (from to 0.025). By esign, the power of the SWDs is at least as high as that of the parallel GRT. In some cases the SWD is substantially higher in power than the parallel GRT because the number of SWD groups was roune up to be an integer multiple of T SW 1 so the same number of groups woul start the intervention in each of T SW 1 measurement perios. Limitations We benefite greatly from Hussey an Hughes s weighte least squares formulation to write close form expressions for the variance of the intervention effect estimator. Even so, our approach has several limitations. The intervention effect, h, is assume to be constant across groups an to persist throughout the stuy. The correlation between earlier an later measurements within a group is assume to be constant no matter how much time elapses. An we roune the number of groups in the SWD up to ensure a balance number of groups starting the intervention in each time perio. Note that the Hussey an Hughes moel assumes repeate cross-sectional measurements, so we o not have to worry about loss to follow-up of iniviual group participants; this woul be an important consieration in a steppe wege cohort esign. Further work is neee to evelop results that are free of these limitations. Conclusions TheMBDisagooesignwheninterventions are applie to iniviuals an result in rapi, large changes in the outcome variable. The esign is especially compelling when the change is large enough to be obvious just from TABLE 1 Total Groups as a Function of Measurement Perios Require in Steppe Wege Designs to Provie Power at Least as Great as That in the Parallel Group-Ranomize Trial With 24 Counties an 3 Measurement Perios T GRT =3 a T SW =3 T SW =4 T SW =5 T SW =6 T SW =7 T SW =8 T SW =9 T SW =10 T SW =11 T SW =12 T SW =13 ICC I GRT P, % I SW P,% I SW P,% I SW P,% I SW P,% I SW P,% I SW P,% I SW P,% I SW P,% I SW P,% I SW P,% I SW P,% > > >99 11 >99 12 > > > b 71 b 10 b 73 b b 71 b b 73 b 10 b 75 b b 73 b b 75 b 12 b 78 b 10 b 79 b b 78 b >99 Note. GRT= group-ranomize trial; I =number of groups; ICC=intraclass correlation coefficient (q); P=power to etect a rop in prevalence from to with cross-sectional measurements mae on m= 100 persons per group per measurement perio, assuming a =0.05; SW =step wege; T = number of measurement perios. Steppe wege table entries are roune up to be integer multiples of T SW 1, so the same number of groups start the intervention in each measurement perio. a The T GRT =3 column shows the power for the parallel GRT with 24 groups an 3 measurement perios at various values of ICC. Each of the remaining T SW columns to the right lists the number of groups necessary to achieve at least as much power as the 3-group GRT using an SWD with T SW measurement perios. b Designs with fewer total measurements (I SW T SW ) than the 24 3 =72 measurements require by the group-ranomize trial Research an Practice Peer Reviewe Rhoa et al. American Journal of Public Health November 2011, Vol 101, No. 11

6 looking at the time series of measure outcome ata. These conitions are not likely to hol in most public health interventions in which effect sizes are often on the orer of stanar eviations There may be a place for small, quick MBDs in the early stages of protocol evelopment, when interventions that will eventually be applie at the group level are teste out at the iniviual level. But to raw causal inferences, we o not share the enthusiasm expresse by Hawkins et al. that small MBDs might be a viable alternative to RCTs. To support strong conclusions an to estimate a generalizable treatment effect in group-level public health interventions, investigators who have reason to use a staggere start esign shoul use a steppe wege GRT. Before oing so, they shoul recognize the limitations of the SWD an the available evience on its power. The SWD will take longer than the parallel GRT an may require as many measurements even if there are fewer groups involve. With fewer groups, ajustment for group-level confouning factors will be important because ranomization will be less likely to balance ifferences, an it will be challenging because there will be fewer egrees of freeom. Investigators will want to consier all these factors as they choose between the parallel GRT an the SWD. j About the Authors Dale A. Rhoa is with the Centers for Public Health Research an Evaluation, Battelle Memorial Institute, Columbus, OH. Dale A. Rhoa, Rebecca R. Anrige, Michael L. Pennell, an Erinn M. Hae are with the Division of Biostatistics, College of Public Health, The Ohio State University, Columbus. Davi M. Murray is with the Division of Epiemiology, College of Public Health, The Ohio State University. Erinn M. Hae is also with the Center for Biostatistics, The Ohio State University. Corresponence shoul be sent to Dale A. Rhoa, Centers for Public Health Research an Evaluation, Battelle Memorial Institute, 505 King Ave., Columbus, OH ( RhoaD@battelle.org ). Reprints can be orere at by clicking the Reprints/Eprints link. This article was accepte April 25, Contributors D. A. Rhoa le the multiple baseline esign an steppe wege esign literature reviews, rafte an eite the article, an coorinate revisions an corresponence. D. M. Murray inspire the article by raising questions about the relative merits of multiple baseline esigns an group-ranomize trials (GRTs), facilitate the group s meetings, provie leaership on GRT issues, an eite the article. R. R. Anrige provie key insight into the relative variances of parallel GRTs an steppe wege esigns an eite the article. M. L. Pennell an E. M. Hae participate in the literature review, helpe shape the article s arguments, an eite the article. Acknowlegments The National Cancer Institute supporte this work (grant supplement P50CA105632). The authors are very grateful to Will Shaish for helpful comments on an early raft of this article an to the anonymous reviewers for their helpful comments. Human Participant Protection Human participant protection was not neee for this article because no human participants were involve. References 1. Shaish WR, Cook TD, Campbell DT. Experimental an Quasi-experimental Designs for Generalize Causal Inference. Boston: Houghton Mifflin; Donner A, Klar N. Design an Analysis of Cluster Ranomization Trials in Health Research. Lonon: Arnol; Murray D. Design an Analysis of Group-Ranomize Trials. New York: Oxfor University Press; Biglan A, Ary D, Wagenaar AC. The value of interrupte time-series experiments for community intervention research. Prev Sci. 2000;1(1): Hawkins NG, Sanson-Fisher RW, Shakeshaft A, D Este C, Green LW. The multiple baseline esign for evaluating population-base research. Am J Prev Me. 2007;33(2): Sanson-Fisher RW, Bonevski B, Green LW, D Este C. Limitations of the ranomize controlle trial in evaluating population-base health interventions. Am J Prev Me. 2007;33(2): The Gambia Hepatitis Stuy Group. The Gambia Hepatitis Intervention Stuy. Cancer Res. 1987;47(21): Brown CA, Lilfor RJ. The steppe wege trial esign: a systematic review. BMC Me Res Methool. 2006;6: Baer DM, Wolf MM, Risley TR. Some current imensions of applie behavior analysis. J Appl Behav Anal. 1968;1(1): Barlow DH, Hersen M. Single Case Experimental Designs: Strategies for Stuying Behavior Change. 2n e. New York: Pergamon Press; Barlow DH, Nock M, Hersen M. Single Case Experimental Designs: Strategies for Stuying Behavior for Change. 3r e. Boston: Pearson/Allyn an Bacon; Hersen M, Barlow DH. Single Case Experimental Designs: Strategies for Stuying Behavior Change. New York: Pergamon Press; Kazin AE. Single-Case Research Designs: Methos for Clinical an Applie Settings. New York: Oxfor University Press; Kazin AE. Research Design in Clinical Psychology. 3r e. Boston: Allyn an Bacon; Siman M. Tactics of Scientific Research. New York: Basic Books; Mertens D. Research an Evaluation in Eucation an Psychology: Integrating Diversity With Quantitative, Qualitative, an Mixe Methos. Thousan Oaks, CA: Sage; Horner R, Carr E, Halle J, Mcgee G, Oom S, Wolery M. The use of single-subject research to ientify eviencebase practice in special eucation. Except Chil. 2005; 71(2): Kenney C. Single-Case Designs for Eucational Research. Boston: Allyn & Bacon; Tawney JW, Gast DL. Single Subject Research in Special Eucation. Columbus, OH: Merrill; Matyas TA, Greenwoo KM. Visual analysis of single-case time series: effects of variability, serial epenence, an magnitue of intervention effects. J Appl Behav Anal. 1990;23: Onghena P, Egington E. Customization of pain treatments single-case esign an analysis. Clin J Pain. 2005;21(1): Bulté I, Onghena P. Ranomization tests for multiple baseline esigns: an extension of the SCRT-R package. Behav Res Methos. 2009;41(2): Ferron JM, Bell BA, Hess MR, Renina-Gobioff G, Hibbar ST. Making treatment effect inferences from multiple-baseline ata: the utility of multilevel moeling approaches. Behav Res Methos. 2009;41(2): Marascuilo LA, Busk PL. Combining statistics for multiple-baseline AB an replicate ABAB esigns across subjects. Behav Assess. 1988;10(1): Brown CH, Wyman PA, Guo J, Pena J. Dynamic wait-liste esigns for ranomize trials: new esigns for prevention of youth suicie. Clin Trials. 2006;3(3): Hayes RJ, Moulton LH. Cluster Ranomise Trials. Boca Raton, FL: CRC Press; Hussey MA, Hughes JP. Design an analysis of steppe wege cluster ranomize trials. Contemp Clin Trials. 2007;28(2): Kazin AE. Methoological an interpretive problems of single-case experimental esigns. J Consult Clin Psychol. 1978;46(4): Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2n e. Hillsale, NJ: Erlbaum; Fishbein M. Great expectations, or o we ask too much from community-level interventions? Am J Public Health. 1996;86(8 pt 1): Snyer LB, Hamilton MA. Meta-analysis of U.S. health campaign effects on behavior: emphasize enforcement, exposure, an new information, an beware the secular tren. In: Hornik R, e. Public Health Communication: Evience for Behavior Change. Hillsale, NJ: Erlbaum; 2002: November 2011, Vol 101, No. 11 American Journal of Public Health Rhoa et al. Peer Reviewe Research an Practice 2169