Statistical Power and Optimal Design in Experiments in Which Samples of Participants Respond to Samples of Stimuli

Transcription

1 Journal of Exerimental Psychology: General 014 American Psychological Association 014, Vol. 143, No. 5, /14/$1.00 htt://x.oi.org/ /xge Statistical Power an Otimal Design in Exeriments in Which Samles of Particiants Reson to Samles of Stimuli Jacob Westfall University of Colorao Bouler Davi A. Kenny University of Connecticut Charles M. Ju University of Colorao Bouler Researchers esigning exeriments in which a samle of articiants resons to a samle of stimuli are face with ifficult questions about otimal stuy esign. The conventional roceures of statistical ower analysis fail to rovie aroriate answers to these questions because they are base on statistical moels in which stimuli are not assume to be a source of ranom variation in the ata, moels that are inaroriate for exeriments involving crosse ranom factors of articiants an stimuli. In this article, we resent new methos of ower analysis for esigns with crosse ranom factors, an we give etaile, ractical guiance to sychology researchers lanning exeriments in which a samle of articiants resons to a samle of stimuli. We extensively examine 5 commonly use exerimental esigns, escribe how to estimate statistical ower in each, an rovie ower analysis results base on a reasonable set of efault arameter values. We then evelo general conclusions an formulate rules of thumb concerning the otimal esign of exeriments in which a samle of articiants resons to a samle of stimuli. We show that in crosse esigns, statistical ower tyically oes not aroach unity as the number of articiants goes to infinity but instea aroaches a maximum attainable ower value that is ossibly small, eening on the stimulus samle. We also consier the statistical merits of esigns involving multile stimulus blocks. Finally, we rovie a simle an flexible Web-base ower alication to ai researchers in lanning stuies with samles of stimuli. Keywors: statistical ower, exerimental esign, otimal esign, stimulus samling, mixe moels Stuies in which samles of articiants reson to samles of stimuli in various exerimental conitions are ubiquitous in exerimental sychology. In stuies of memory, articiants often memorize lists of wors that are rawn from a larger corus of wors. In stuies of social cognition, articiants often make jugments about sets of faces or rea vignettes about hyothetical ersons. In stuies of emotion, articiants are often exose to hotograhs or film clis of emotion-rovoking scenes. In all of these examles, the articular iniviual stimuli use are not of intrinsic theoretical interest, excet insofar as they instantiate the general categories to which they belong (e.g., monosyllabic common wors; African American male facial hotos). What are tyically of interest are any ifferences in the resonses given in Jacob Westfall, Deartment of Psychology an Neuroscience, University of Colorao Bouler; Davi A. Kenny, Deartment of Psychology, University of Connecticut; Charles M. Ju, Deartment of Psychology, University of Colorao Bouler. We thank Markus Brauer, Deborah Kashy, John Lynch, Dominique Muller, Gary McClellan, Thom Baguley, an members of the Stereotying an Prejuice Lab at the University of Colorao for helful comments on an earlier version of this article. Corresonence concerning this article shoul be aresse to Jacob Westfall, Deartment of Psychology an Neuroscience, University of Colorao, Bouler, CO jake.westfall@colorao.eu the ifferent conitions to those stimuli (e.g., better wor memory in one conition than another; more negative jugments of African American faces than of White faces). In esigning such stuies, researchers are face with a variety of esign ecisions concerning articiants, stimuli, an the secifics of the esign use. There is a series of ecisions to be mae about articiants: Shoul one samle articiants broaly to increase generalization, or shoul one restrict the samle of articiants use to maximize ower? Shoul one use a esign where articiants are only in one conition, or shoul a within-articiant esign be use? An finally, how many articiants shoul be run? These are ecisions that researchers realize have consequences for statistical ower an the kins of generalizations that are ermitte from the subsequently collecte ata. Similar ecisions are mae, although too often in a cursory manner, about stimuli: How variable shoul the stimuli be? How many stimuli shoul be use? An shoul stimuli be ifferent in the ifferent conitions or shoul the same stimuli be use across conitions? Unfortunately, researchers often o not realize that these ecisions also have major consequences for statistical ower an the kins of generalizations that they ermit. All too often, these ecisions about stimuli are base more on the traitional ractices within an exerimental araigm rather than wellthought-out rinciles about the otimal esign of exeriments. This article focuses on issues of statistical ower an otimal esign in exeriments in which samles of articiants reson to 1

2 WESTFALL, KENNY, AND JUDD samles of stimuli in ifferent conitions. Assuming that one seeks to generalize conclusions about conition ifferences to future stuies that might be conucte, then one ought to be concerne about generalization not only to other samles of articiants but also to other samles of stimuli that might have been use. Our goal is to rovie the tools to enable researchers to think in an informe manner about esign ecisions involving not only samles of articiants but also samles of stimuli. We consier a range of ossible esigns that might be use. We consier the ower imlications of each, an we consier the ower imlications of the number of stimuli as well as the number of articiants. Treating Particiants an Stimuli as Ranom In an earlier article (Ju, Westfall, & Kenny, 01), we consiere one articular esign in which a samle of articiants resons to a samle of stimuli. The hyothetical examle use in that article involve articiants giving resonses to a series of facial hotograhs of White an African American male targets. Target ethnicity was the exerimental factor of theoretical interest (i.e., were ifferent resonses given to the White targets on average comare to the African American targets?). Accoringly, in this esign articiant was crosse with ethnicity (conition), but stimuli were either of one ethnicity or the other. Most tyically when face with ata from such a esign researchers conuct what we calle a by-articiant analysis, analyzing two means for each articiant one for the White targets an one for the African American targets an testing whether the mean within-articiant ifference in these two means is significantly ifferent from zero. Such an analysis imlicitly treats articiants as a ranom factor but oes not treat stimuli as ranom, thus ermitting generalization to other samles of articiants but not to other samles of stimuli. Ju et al. (01) an others have shown that this analysis gives rise to seriously inflate Tye I error rates if in fact one seeks generalization not only to other samles of articiants but also to other samles of stimuli (e.g., articular White an African American target hotos) that might be use (Baayen, Davison, & Bates, 008; Clark, 1973; Coleman, 1964; Santa, Miller, & Shaw, 1979). Traitionally, if both articiants an stimuli were treate as ranom factors in the analysis, an analytic aroach involving quasi-f ratios was the only ractical solution, as outline by Clark (1973) an many others. What Ju et al. (01) mae clear (as others ha reviously; see Baayen et al., 008) was that an analysis base on linear mixe moels coul now be easily imlemente, roucing aroriate Tye I error rates assuming generalization was sought across both ranom factors articiants an stimuli. Many of the shortcomings of the traitional quasi-f methos (e.g., restrictive assumtions about the stuy esign, comlicate esign-secific erivations, lack of availability in major statistical software ackages) are avoie by the more moern aroach base on linear mixe moels. Although treating stimuli as ranom through the aroriate use of linear mixe moels oes bring the benefit of increase generalizability, it often oes so at the cost of lower statistical ower: More robust an general inferences must necessarily be suorte by stronger statistical evience. Thus, it is reasonable to exect that if we wish to urchase increase generalizability from our exeriments, we must be reare to ay at least some statistical ower cost. However, in numerous stuies aearing in the literature toay, the ower costs associate with treating stimuli as ranom are neelessly exacerbate by oor esign choices. More secifically, in esigns in which samles of articiants reson to samles of stimuli, researchers tyically ignore the ower imlications of the stimuli they use both the variability of the stimuli an their number. This neglect of stimulus samling (Wells & Winschitl, 1999) has le to many inefficiently esigne stuies. This unfortunate fact is articularly salient in light of the growing awareness in sychology an other exerimental sciences of the roblems create an maintaine by a roliferation of chronically unerowere stuies (Asenorf et al., 013; Bakker, van Dijk, & Wicherts, 01; Button et al., 013; Ioanniis, 008; Schimmack, 01). Our goal in the rest of this article is to make researchers aware of the ower imlications of stimulus samling when they aroriately treat stimuli as a ranom factor. We offer guiance about how one otimally esigns stuies with ranom factors of articiants an stimuli to increase the robability that any conition ifferences that are foun can in fact be relicate in future stuies that emloy ifferent articiant an ifferent stimulus samles. We consier a broa range of esigns in which samles of articiants reson to samles of stimuli. For each of these, we evelo aroriate linear mixe moels that treat both factors as ranom. We further evelo effect size an ower estimation roceures for all of these esigns, roviing illustrative ower results as a function of the numbers of articiants an stimuli, an their variances. Aitionally, we rovie a Web-base alication that ermits generalization of these ower results to any secific set of assumtions about esigns, numbers of articiants an stimuli, an the relevant variances. Finally, we consier more general issues of otimal esign in research eneavors in which samles of articiants reson to a samle of stimuli in ifferent conitions. Designs With Crosse Particiant an Stimulus Ranom Factors In consiering the range of ossible esigns, we focus on esigns in which articiants are crosse with stimuli, as oose to esigns in which stimuli are neste in articiants or vice versa. That is, we consier esigns in which multile articiants reson to the same stimuli. We mae the ecision to limit the range of esigns that we iscuss in etail in this article because crosse esigns are use more frequently than neste esigns in exerimental sychology. Aitionally, there is an extensive literature in eucation an alie statistics on neste esigns, commonly referre to as hierarchically neste esigns, an this literature has consiere issues of ower an otimal esign (Rauenbush, 1997; Rauenbush & Liu, 000; Snijers, 001; Snijers & Bosker, 1993). We know of no work that has aresse issues of statistical ower in esigns with crosse ranom factors. In the following aragrahs, we efine the five exerimental esigns consiere in this article. For these esigns, we assume a single exerimental maniulation of interest having two levels. This maniulation, which we refer to as Conition, constitutes the only fixe factor in the esign, ifferentiating it from the ranom

3 STATISTICAL POWER WITH SAMPLES OF STIMULI 3 factors of stimuli an articiants. A secon assumtion is that there is only a single relication in the esign; that is, each articiant resons to any iniviual stimulus within a conition only one time. Table 1 rovies schematics that efine the five esigns. Each esign is escribe by a matrix in this table, with articiants efining the rows an stimuli efining the columns. The cells of these matrices inicate the levels of conition (A an/or B) uner which articular observations occur. If a cell has a ash, it means that that observation (a articular articiant aire with a articular stimulus) is not collecte. Table 1 Schematics for Five Exerimental Designs Particiants Stimuli Fully crosse esign 1 AB AB AB AB AB AB AB AB AB AB AB AB 3 AB AB AB AB AB AB 4 AB AB AB AB AB AB 5 AB AB AB AB AB AB 6 AB AB AB AB AB AB Counterbalance esign 1 A A A B B B A A A B B B 3 A A A B B B 4 B B B A A A 5 B B B A A A 6 B B B A A A Stimuli-within-conition esign 1 A A A B B B A A A B B B 3 A A A B B B 4 A A A B B B 5 A A A B B B 6 A A A B B B Particiants-within-conition esign 1 A A A A A A A A A A A A 3 A A A A A A 4 B B B B B B 5 B B B B B B 6 B B B B B B Both-within-conition esign 1 A A A A A A 3 A A A 4 B B B 5 B B B 6 B B B Note. Each schematic illustrates which articiant (labele 1 to 6) views which stimulus (labele 1 to 6) uner which conition (labele A or B). AB means that this articiant resons to this stimulus uner both conitions; A means that this articiant resons to this stimulus only uner Conition A; B means that this articiant resons to this stimulus only uner Conition B. A ash means that this articiant never resons to this stimulus. The first esign is the fully crosse esign in which every articiant resons to every stimulus twice, once in each conition. In the matrix for this esign, every articiant by stimulus cell contains both A an B, inicating that the articiant stimulus air occurs in both conitions. The secon esign in Table 1 is the counterbalance esign in which, for half of the articiants, each stimulus is in either conition A or B, an for the other half, the stimulus is resone to in the other conition. The thir esign is the stimuli-within-conition esign in which each stimulus is resone to in only one of the two conitions, although articiants are crosse with conition. This is the esign that was the focus of Ju et al. (01). The fourth esign, the articiants-within-conition esign, reverses the roles of stimuli an articiants in that each articiant resons in only one of the two conitions, but stimuli are resone to in both conitions, albeit by ifferent articiants. Finally, the fifth esign is the both-within-conition esign. Here both articiants an stimuli are neste uner conition, an within each conition each articiant resons to every stimulus. Estimating Power in Designs With Crosse Particiant an Stimulus Ranom Factors The starting oint for conucting a ower analysis is to state exlicitly the moel unerlying the observations in these esigns, making exlicit all the sources that contribute to variation in those observations. To o this, we assume that we have two ranom factors, one involving a samle of articiants an the other involving a samle of q stimuli. Aitionally there is a single fixe conition factor, c, with two levels. In this article, we consier exeriments where articiants reson to each stimulus at most only once er conition; in other wors, we assume only a single relication at the lowest level of observation. Uner these assumtions the fully secifie mixe moel for the resonse of the ith articiant to the jth stimulus in the kth conition is y ijk 0 1 c k P i PC i c k S j SC j c k PS ij ε ijk, var ip P, PC var i PC, var js S, SC PS var j SC, var ij PS, var(ε ijk ) E. In this moel, 0 an 1 c k reresent the fixe effects an cature, resectively, the overall mean resonse an the conition ifference in resonses. We assume the values of c k are contrast or eviation coe so that c 1 c 0 an c 1 c c. For examle, the values of c k might be 1, 1, or 1/, 1/. Imortantly, the values of c k are assume not to be ummy or treatment coe (e.g., c 1 0, c 1), as this totally alters the meanings of the variance comonents (see also Barr, Levy, Scheeers, & Tily, 013, footnote ), an because we rely on the contrast-coing assumtion to eal with otential covariances between the ranom effects (see the Aenix). The other comonents in this moel are the ranom effect comonents an are efine in Table. These reresent all the otential sources of variation that might be execte in resonses.

4 4 WESTFALL, KENNY, AND JUDD Table Definitions of Ranom Variance/Covariance Comonents in Mixe Moel Comonent Definition Interretation P Variance ue to articiant (Particiant intercet variance) To what extent o articiants have ifferent mean resonses? PC Variance ue to articiant by conition interaction (Particiant sloe variance) To what extent oes the mean ifference between conitions vary across articiants? S Variance ue to stimulus (Stimulus intercet variance) To what extent o stimuli elicit ifferent mean resonses? SC Variance ue to stimulus by conition interaction (Stimulus sloe variance) To what extent oes the mean ifference between conitions vary across stimuli? PS Variance ue to articiant by stimulus interaction (Particiant by stimulus intercet variance) To what extent o articiant ifferences in resonses vary with ifferent stimuli? E Resiual error variation To what extent is there resiual variation in resonses not ue to the above sources? The etails of the ower calculations are given in the Aenix. For these, we assume balance esigns with no missing ata. Aitionally we assume that all variance comonents, incluing resiual error, are normally istribute. We follow there the general roceure secifie by Cohen (1988) for simler esigns. This roceure involves first calculating an estimate effect size, analogous to Cohen s : the execte mean ifference ivie by the execte variation of an iniviual observation, which in our case accrues from all the variance comonents secifie above: 1 P c PC 1 S c SC PS. E In this equation, 1 an are the two execte conition means. The enominator of the effect size is the square root of a oole variance, albeit a comlicate one. It reresents the variation within each conition across both articiants an stimuli. The Aenix shows how that variance can be etermine for ifferent esigns. One then calculates an oerative effect size (Cohen, 1988,. 13) estimate that makes ajustments to eening on the secific esign use. Next one weights this esign-secific effect size by the relevant samle sizes to obtain the noncentrality arameter for a noncentral t istribution an also then comutes the aroriate egrees of freeom. Cohen (1988,. 544) at this oint relie on an aroximation from Dixon an Massey (1957) to estimate ower. We have aote a more exact aroach that irectly comutes areas uner the aroriate noncentral t istribution. When we consier ower results for secific esigns an when we comare results across esigns, it is necessary for us to refer to some of the secific results that are in the Aenix. Haily, however, for most uroses the researcher can ignore the technical etails that we resent in the Aenix because we have imlemente all of the stes in a user-frienly Web-base alication (htt://jakewestfall.org/ower/), which erforms the rather extensive comutations. Figure 1 islays a screenshot of this ower alication. In aition to comuting ower estimates, users can also secify a esire ower level an solve for the minimum number of articiants, minimum number of stimuli, or minimum effect size that woul lea to the esire ower level. The ower alication also rovies syntax for the R, SAS, an SPSS statistical software ackages that one can use to secify the aroriate mixe moel for each esign an can be use to comute effect sizes from a set of unstanarize esign estimates. Accoringly, all one nees to estimate ower from the a are the following: 1. The esign to be use.. The anticiate effect size or the mean conition ifference. 3. Estimates of the relevant variance comonents. 4. The anticiate numbers of articiants an stimuli. One otential ifficulty in oing this is the etermination of the estimates of the relevant variance comonents (Item 3 in the list above). In the Aenix, we evelo formulas for the noncentrality arameters an egrees of freeom in terms of these unstanarize variances an also in terms of variance artitioning coefficients (VPCs; Golstein, Browne, & Rasbash, 00), which we enote using V in lace of (the subscrits remain the same). These in essence are stanarize variances, exressing the roortion of the total variation in the observations that is ue to a articular variance comonent. Thus, it is ossible to estimate ower (an use the alication) base on estimates of the relative rather than absolute sizes of the variance comonents. There are six ossible variance comonents that must be consiere (as lai out in Table ), an the sum of the six VPCs for these comonents must equal one. We now move to a more extene iscussion of the interretations of these variance comonents. Reasoning About Variance Comonents an VPCs Reasoning about statistical ower always requires that one think carefully about the factors that influence the variability of resonses that is, about the variance comonents or VPCs. For mixe moels, this is more comlicate because there are multile sources of ranom variation. Table rovies substantive interretations of all ossible variance comonents in a general, abstract setting. In this section we try to make the meanings of the ifferent VPCs more intuitive an concrete by walking through a hyothetical exerimental scenario an escribing all ossible VPCs in terms of this scenario. Consier an exeriment in which heterosexual male articiants consume an alcoholic rink or a nonalcoholic lacebo rink an then are aske to juge the attractiveness of a stimulus set of

5 STATISTICAL POWER WITH SAMPLES OF STIMULI 5 Figure 1. Screenshot from Web-base ower alication (htt://jakewestfall.org/ower/). The alication allows users to comute statistical ower for the five main esigns iscusse in this article; or to secify a esire ower level an solve for the minimum effect size, minimum number of articiants, or minimum number of stimuli that woul lea to that ower level. The alication also rints useable syntax in the R, SAS, an SPSS software ackages for fitting a linear mixe moel in any of the esigns an can be use to comute effect sizes an VPC values from a set of unstanarize esign estimates. Stim stimulus.

6 6 WESTFALL, KENNY, AND JUDD hotograhs of female faces. 1 Thus, the resonse variable is some continuous rating of erceive attractiveness, the Conition reictor is alcoholic versus lacebo rinks, the articiants are the erceivers, an the stimuli are the hotograhs or targets of ercetion. The reicte beer goggle effect is the tenency for attractiveness ratings to be higher on average following consumtion of the alcoholic rink comare to the nonalcoholic lacebo rink. In this hyothetical exeriment, V refers to the variance in the erceivers average tenencies to view all targets as attractive or unattractive. That is, the erceiver Allen may be a high resoner (Allen views most targets as being relatively attractive, comare to other erceivers) while the erceiver Bob may be a low resoner (Bob is more icky an views most targets as being relatively unattractive), an V refers to the variation across erceivers in their average resonse tenencies. V S refers to the variance in the targets average tenencies to be erceive as attractive or unattractive. That is, the target Carol may ten to elicit high resonses (most erceivers agree that Carol is attractive) while the target Diane may ten to elicit low resonses (most erceivers agree that Diane is unattractive), an V S refers to the variation across targets in their average erceive attractiveness. In mixe moel terminology, V an V S refer to the variance in the articiant intercets an stimulus intercets, resectively. Neither of the variance comonents escribe above were concerne with the beer goggle effect; they referre to attractiveness ratings on average across the alcohol an lacebo conitions. The V PC comonent refers to the variance of the erceivers beer goggle effects. That is, the erceiver Allen may exhibit a large beer goggle effect (Allen tens to give higher attractiveness ratings following consumtion of the alcoholic rink comare to the lacebo rink), whereas the erceiver Bob may exhibit a small or even a negative beer goggle effect (Bob s attractiveness ratings o not ten to be affecte by alcohol consumtion, or erhas he even gives lower attractiveness ratings following alcohol consumtion), an V PC refers to the variance across erceivers in the magnitues of their beer goggle effects. V SC refers to variation in how much ifferent targets ten to benefit from beer goggle effects. That is, the target Carol may benefit greatly from beer goggle effects (Carol tens to be rate as more attractive by erceivers who consume the alcoholic comare to the lacebo rink), whereas the target Diane may not benefit much from beer goggle effects, or may even ten to elicit negative beer goggle effects (Diane s attractiveness is juge to be about equal by erceivers who consume alcoholic or lacebo rinks, or erhas is even viewe as less attractive following alcohol consumtion). In mixe moel terminology, V PC an V SC refer to variance in the articiant sloes an stimulus sloes, resectively. Next we have V PS, which refers to the variance in the relationshi effects for each airing of erceiver an target (Kenny, 1994). Previously we suose that the erceiver Allen tene to juge targets as attractive an that the target Carol tene to elicit high attractiveness ratings. On the basis of this, we certainly exect Allen to give Carol high attractiveness ratings. The extent to which Allen s ratings of Carol s attractiveness ten to systematically iffer from this execte aitive attractiveness rating is the erceiver target interaction (or relationshi effect) for Allen an Carol. If Allen systematically rates Carol as being even more attractive than execte base on their iniviual erceiver an target effects, then they have a high relationshi effect. If Allen systematically rates Carol lower than execte, then they have a low relationshi effect. V PS refers to the variance across all erceiver target airings in these interaction effects. Finally, V E refers to the resiual or error variance. When the attractiveness ratings of a erceiver towar a target iffers from what is execte base on both the intercet an sloe for both the articiant an stimulus, as well as their interaction effect, then this is consiere unexlaine variation an attribute to the error variance. Power Analyses for the Stanar Case In this section, we rovie ower estimates for each of the reviously efine five esigns. 3 For these estimates we mae some relatively arbitrary but not unreasonable ecisions about values of the VPCs an effect sizes that one might encounter. Secifically, we make the following assumtions about the relative magnitue of the various variance comonents: V E.3; V P V S.; V PC V SC V PS.1. We refer to these values as the stanar case. These VPCs reflect first our informal observation, from fitting mixe moels to many ifferent ata sets in exerimental sychology, that variance ue to resiual error tens often to account for the largest roortion of ranom variation; variance ue to articiant an stimulus mean resonse tenencies (i.e., ranom intercets) tens to account for a noticeable but smaller fraction of ranom variation; variance ue to ranom articiant-by-conition an stimulus-by-conition interactions (i.e., ranom sloes) tens to account for still less of the total ranom variation; an finally variance ue to articiant-bystimuli interactions in many contexts is tyically small as well. These observations about the tyical relative magnitues of the variance comonents are also consistent with regularities that have been frequently remarke uon in the statistical literature on the esign of exeriments, where this henomenon has been referre to as the hierarchical orering rincile (Li, Suarsanam, & Frey, 006; Wu & Hamaa, 000,. 143) or the sarsity-ofeffects rincile (Montgomery, 013,. 90). Of course, in articular cases, researchers who feel that these efault VPC assumtions are unlikely to be a reasonable escrition of the ata from their exeriment can use the ower alication escribe in the revious section to get ower estimates 1 One coul easily imagine an extene version of this esign in which both geners serve as both erceivers an targets, but we wishe to kee our examle more consistent with the simler single-fixe-factor esigns that are the focus of this article, an hence we focus arbitrarily on males erceiving females. Note that in the esigns that we consier in this article, V E also imlicitly contains variance ue to the P S C interaction, as well as covariance between the P S an P S C terms. However, it is only ossible to uniquely estimate these two aitional arameters if every articiant receives every stimulus multile times uner both conitions in other wors, in the fully crosse esign with multile relicates. As mentione in the main text, we o not consier the case of multile relicates, but we mention these otential aitions to the full mixe moel here for the sake of comleteness. 3 All ower results were erive as exlaine in the Aenix. Aitionally, in the case of every esign, selecte results were emirically confirme through simulations. The simulation coe an results are oste in links at the bottom of the ower alication Web age (htt://jakewestfall.org/ower/).

7 STATISTICAL POWER WITH SAMPLES OF STIMULI 7 tailore to their research omain. We certainly acknowlege that in many ifferent omains our assumtions about these variance comonents, secifically that the variances ue to articiants an stimuli are aroximately equal, o not hol. In introucing the ower results for each esign in this stanar case, we rovie the algebraic exressions for the noncentrality arameter for each, ulle from the Aenix. This is one to make clear how the esigns iffer in their ower estimates. Factors that increase the noncentrality arameter (i.e., move it further from 0) lea to greater ower, while factors that ecrease the noncentrality arameter (i.e., move it closer to 0) lea to lower ower. Fully Crosse Design In this esign, articiants an stimuli are crosse with each other an both are crosse with conition. As an examle, imagine that articiants are aske to juge the attractiveness of a set of faces; they o this uner two ifferent context conitions, with the same faces in each conition. Because articiants make two resonses to each stimulus in this esign, there is a ossibility of observing orer or carryover effects in articiants two resonses to each stimulus; for the sake of simlicity, we assume here that no such effects are oerative. In this esign, the fully secifie mixe moel can be estimate, with all the variance comonents efine as in that equation. In other wors, there are no comonents of variance that are confoune given the mixe moel secification. From the Aenix the noncentrality arameter for this esign is nc V PC V SC q V E q Thus, the relevant variance comonents for calculating ower in this esign are the articiant sloe variance, the stimulus sloe variance, an the resiual error variance. Variations ue to articiant an stimulus intercets, while estimable, o not contribute to the stanar error of the treatment effect. On the basis of the values of the VPCs given earlier, in Figure we lot ower results for this esign, as a function of (the number of articiants), q (the number of stimuli), an small, meium, an large values of the stanarize effect size ( 0., 0.5, an 0.8, resectively). This figure contains two ways of looking at the ower results. The grahs in the to row lot ower (as a function of an q) for the three effect sizes. Those in the bottom row lot minimum effect sizes neee to have ower equal at least.5,.8, an.95, resectively. These results show that ower is ramatically affecte by both the number of articiants an the number of stimuli in this esign, an, given the arallel magnitue of the relevant variance comonents, the ower curves are erfectly symmetric as a function of the two samle sizes. Although most researchers are reasonably attune to thinking about the nee to gather ata from a sufficiently large samle of articiants to achieve accetable ower levels, it is rare for researchers to think in a arallel manner about the aroriate size of the stimulus samles they use. What is remarkable about our results here is that, given the assumtions we are making about the variance comonents, maximum achievable ower with a meium effect size when using eight stimuli a. fairly tyical value of q in many exerimental stuies is only aroximately.50, even with an infinite number of articiants. Another way of saying this is that if one anticiates a meium effect size an one woul like ower to roughly equal.80, then the minimum number of stimuli that can be use, even with a very large number of articiants, is about 16. We iscuss the iea of maximum attainable ower in more etail in the next major section on rinciles of otimal esign. Counterbalance Design In this esign, each articiant resons to every stimulus, one half of which are resone to in one conition an the other half in the other, an those halves are counterbalance across articiants. Again imagine that the attractiveness of a set of faces is juge, half in one conition an half in the secon, but which set is juge in which conition varies between articiants. Thus, in this esign, each articiant woul see a given face only once. Technically, when analyzing ata from the counterbalance esign, a secon fixe reictor reresenting the counterbalancing factor (e.g., Set A vs. Set B for faces aearing in one of two sets) coul be ae to the moel, an this woul be sensible to o if, for examle, the stimuli were assigne at all nonranomly to the levels of the counterbalancing factor. Here, for the sake of simlicity, we assume there is no main effect of the counterbalancing factor an omit this secon reictor from the moel. In this esign, only five of the variance comonents are estimable. The variance ue to the articiant-by-stimulus interaction (V PS ), is not estimable an is confoune with V E. From the Aenix the noncentrality arameter for this esign is nc V PC V SC q [V E V PS ] q Here the confouning of the articiant-by-stimulus interaction variance an the error variance is inicate by the brackets enclosing these two VPCs in the enominator. This noncentrality arameter iffers from that of the fully crosse esign, given above, as a function of this confouning. Aitionally, in this esign, comare to the fully crosse esign, there is half the number of total observations here, given constant an q. As in the fully crosse esign, variation ue to articiant an stimulus intercets oes not affect ower. Because of the ifferences just note, the ower results for this esign, given in Figure 3, are a bit below those given for the fully crosse esign. Actually, for the values of the variance comonents that we are consiering, the ower ifference is remarkably small. Now with eight stimuli an a meium effect size, the maximum achievable ower, even with an extremely large number of articiants, is still aroximately.5. Put another way, to achieve ower of.8 with a meium effect size an an extremely large number of articiants, one woul nee at least 16 stimuli. At a later oint, we iscuss in greater etail the relative efficiency of this esign comare to the fully crosse esign. Stimuli-Within-Conition Design This is the esign that Ju et al. (01) consiere in some etail, in which stimuli are in one conition or the other, but each articiant resons to each stimulus an thus articiant is crosse with coni-.

8 8 WESTFALL, KENNY, AND JUDD tion. 4 In terms of the beer goggle examle given earlier, each articiant is in both the alcohol an lacebo conitions but ifferent target ersons are juge for attractiveness in the two conitions. In this esign, only four of the variance comonents are estimable. The variance ue to the stimulus-by-conition interaction or V SC,is not estimable because stimuli are not crosse with conition. Instea this variance is confoune with the stimulus mean variance V S. An as in the counterbalance esign, the articiant-by-stimulus interaction or V PS is confoune with the resiual error variance. From the Aenix, the noncentrality arameter for this esign is nc Figure. Contour lots for the fully crosse esign. To anel: Statistical ower as a function of the effect size, the number of articiants, an the number of stimuli. Bottom anel: Minimum effect sizes for ifferent esire ower levels as a function of the number of articiants an number of stimuli. The VPCs are hel constant at V V S., V PC V SC V PS.1, an V E.3. The samle sizes are on log scales. VPCs variance artitioning coefficients. V PC V S V SC V E V PS q q Again, the confouning of variance comonents is inicate by the brackets aroun the VPCs in the enominator. The ower of this esign is less than the ower of the first two esigns consiere. largely as a function of the variation ue to stimulus intercets or means which now figures in the enominator of the noncentrality arameter or equivalently in the stanar error for testing the conitions ifference. The ower results for this esign are given in Figure 4. Unlike the earlier two esigns, the ower results here are no longer symmetric with resect to the numbers of articiants an stimuli. Uner the assumtions that we have mae, ower in this esign is more influence by q, the number of stimuli, than by, the number of articiants. Here, with a moerate effect size an a very large number of articiants, one achieves ower of.80 only by using 4 We caution the reaer that in our treatment of this esign in Ju et al. (01), q was efine as the number of stimuli in each conition, not as the total number of stimuli, as it is here.

9 STATISTICAL POWER WITH SAMPLES OF STIMULI 9 Figure 3. Contour lots for the counterbalance esign. To anel: Statistical ower as a function of the effect size, the number of articiants, an the number of stimuli. Bottom anel: Minimum effect sizes for ifferent esire ower levels as a function of the number of articiants an number of stimuli. The VPCs are hel constant at V V S., V PC V SC V PS.1, an V E.3. The samle sizes are on log scales. VPCs variance artitioning coefficients. more than 3 stimuli er conition. Another way of saying this is that ower increases as a function of q begin to asymtote only at values of q greater than 64. On the other han, ower increases as a function of begin to asymtote at values of greater than 3. Statistically this is the same esign as the revious one, excet now the nesting or crossing of the two ranom factors with conition is reverse: Particiants are now in only one conition or the other, but all stimuli are resone to in both. In terms of the beer goggle exeriment, some articiants receive only the alcoholic rink while other articiants receive only the nonalcoholic lacebo rink, but all articiants reson to the same set of stimulus hotograhs. In this esign the variance attributable to the articiant-byconition interaction, V PC, is not estimable. Instea, it is confoune with the articiant mean variance, V P. An as in the revious two esigns, the articiant-by-stimulus interaction or V PS is confoune with the resiual error variance. The noncentrality arameter for this esign, taken from the Aenix, is Particiants-Within-Conition Design nc V P V PC V SC q V E V PS q with confouning again inicate by the brackets in the enominator. This time, to the extent there is large variation from articiant to articiant in their means, ower woul be reuce. In other wors, in the revious esign, variation in stimulus means increases the stanar error in testing the conition ifference; now what increases that stanar error is variation in articiant means.,

10 10 WESTFALL, KENNY, AND JUDD Figure 4. Contour lots for the stimuli-within-conition esign. To anel: Statistical ower as a function of the effect size, the number of articiants, an the number of stimuli. Bottom anel: Minimum effect sizes for ifferent esire ower levels as a function of the number of articiants an number of stimuli. The VPCs are hel constant at V V S., V PC V SC V PS.1, an V E.3. The samle sizes are on log scales. VPCs variance artitioning coefficients. Uner the assumtions that we have mae, the ower results for this esign, given in Figure 5, are the same as those given for the revious esign excet they have been transose so that now ower is more ramatically affecte by the number of articiants than the number of stimuli. Even in the case of this esign, however, it remains true that ower of.80 is only achievable, given a moerate effect size, when the number of stimuli is greater than 16, a number which is larger than that tyically use. In this esign three ifferent variance comonents are not estimable an are confoune with other comonents: the stimulusby-conition comonent, V SC, the articiant-by-conition comonent, V PC, an the articiant-by-stimulus interaction comonent, V PS. The first of these is confoune with the stimulus intercet variance, the secon with the articiant intercet variance, an the thir with the resiual error variance. From the Aenix, the noncentrality arameter for this esign is Both-Within-Conition Design The final esign has each articiant an also each stimulus in only one of the two conitions, but within each conition all / articiants juge all q/ stimuli. From the beer goggle examle, each articiant consumes either alcohol or the lacebo, an each target is juge for attractiveness in only one of the two conitions. nc V P V PC V S V SC V E V PS q q As a result, the stanar error for testing the conition ifference is inflate an ower reuce to the extent that there exist both large articiant an large stimulus ifferences in their means..

11 STATISTICAL POWER WITH SAMPLES OF STIMULI 11 The ower results given in Figure 6 for this esign are again symmetric with resect to an q uner the assumtions we are making. An these ower results are articularly oor, relative to the other esigns we have consiere. With a moerate effect size an as many as 5 articiants resoning to 5 stimuli in each conition, ower only equals.5. Even with unlimite numbers of articiants, ower of.80 with a moerate effect size is achievable only if the number of stimuli er conition is greater than roughly 48. Illustrative Examle Figure 5. Contour lots for the articiants-within-conition esign. To anel: Statistical ower as a function of the effect size, the number of articiants, an the number of stimuli. Bottom anel: Minimum effect sizes for ifferent esire ower levels as a function of the number of articiants an number of stimuli. The VPCs are hel constant at V V S., V PC V SC V PS.1, an V E.3. The samle sizes are on log scales. VPCs variance artitioning coefficients. To make our iscussion of ower analysis for these comlex esigns more concrete, here we work through an examle for a hyothetical exeriment. This examle illustrates the rocess of coming u with some reasonable variance comonent an effect size estimates given what we know about our stuy, using the online ower alication to comute ower an minimum numbers of articiants an/or stimuli, an investigating how these answers woul change uner ifferent assumtions about the variance comonents an effect size. Consier a stuy where we are investigating the imact of a cognitive loa maniulation (e.g., memorizing short lists of integers) on articiants erformance on a series of items from a task that eens on working memory caacity. The eenent variable is some continuous measure of each articiant s erformance on each item. We woul like to emloy a within-subject esign where each articiant resons to items both uner cognitive loa an not uner cognitive loa, but we wish to avoi any otential carryover effects that might result from each articiant resoning to each item twice (once uner each loa conition). Therefore,

12 1 WESTFALL, KENNY, AND JUDD Figure 6. Contour lots for the both-within-conition esign. To anel: Statistical ower as a function of the effect size, the number of articiants, an the number of stimuli. Bottom anel: Minimum effect sizes for ifferent esire ower levels as a function of the number of articiants an number of stimuli. The VPCs are hel constant at V V S., V PC V SC V PS.1, an V E.3. The samle sizes are on log scales. VPCs variance artitioning coefficients. the items are ivie into two lists, List A an List B, an articiants are ranomly assigne to either reson to the List A items uner cognitive loa an the List B items not uner loa, or to reson to the List B items uner cognitive loa an the List A items not uner loa. Thus, we emloy the counterbalance esign. We have two ranom factors, Particiant an Item, an one fixe factor reresenting the overall ifference in erformance between the loa an no-loa conitions. Suose that we have eveloe a samle of 16 items (eight on each list) for this working memory task. How many articiants must we recruit to achieve statistical ower of 0.8? We first examine some answers to this question using the efault set of VPCs that we roose above (V E.3;V P V S.; V PC V SC V PS.1), an a meium effect size of 0.5. The online ower alication (locate at htt://jakewestfall.org/ower/) has these VPCs as the efault. We secify the esign as the counterbalance esign. Accoring to some recent recommenations (Simmons, Nelson, & Simonsohn, 011), a stuy shoul emloy at least 0 articiants er between-subjects conition, so we start with that value as the number of articiants an enter 16 for the number of stimuli (items). Pressing the Solve for X button, we fin that uner the VPC assumtions above, ower woul be.571. If we want to know how many articiants woul be require to achieve statistical ower of.80, we can enter the x symbol for number of articiants, fill in.80 for ower, an ress Solve for X; the alication informs us that we require 154 articiants. An examination of the ower lots for this esign (in Figure 3) makes clear that with only 16 items, the maximum ower that can be achieve, uner the assumtions mae, even with a very large number of articiants, is only slightly above.8. These ower results also make clear that if more ower is to be achieve, more items are necessary. Accoringly, as a result of reaing what has been resente so far in this article, it is ecie to increase the number of items to 30. Solving now for the number of articiants

13 STATISTICAL POWER WITH SAMPLES OF STIMULI 13 to achieve ower of.80, the alication tells us that 7 articiants are neee. Clearly, increasing the number of stimuli makes a large ower ifference. In the absence of any information about the stuy we are lanning other than the ossible samle sizes, the efault set of VPCs that we use above an a meium effect size together reresent a reasonable set of assumtions that we might exect to be aroximately true for our stuy. But if we o know a little about the samles of articiants an stimuli we are using an the effect which we are stuying, we can o better by tailoring what we exect the VPCs to be for the research being conucte. For examle, suose that we know that the items we eveloe for the working memory task vary consierably in their average ifficulty (some items ten to yiel low erformance scores, while others ten to yiel high erformances scores); as a consequence, the more ifficult ones might be affecte more by the cognitive loa maniulation than the easier items. We also might know that our articiants are likely to be rather homogeneous in their average working memory caacities an, we susect, also in the egree to which cognitive loa woul interfere in task erformance. To reflect this knowlege, it seems reasonable to increase the values of both V S an V SC an to ecrease the values of both V P an V PC. Accoringly, we ecie to increase V S an V SC to.5 an.15, resectively, an to ecrease both V P an V PC by the same amounts (to.15 an.05, resectively). We leave the other VPCs at their efault values. These changes, tailore to our knowlege of the research omain, reveal that ower of.80 is achievable with 5 articiants, again assuming that we have increase the number of items to 30. In the rocess illustrate above, we starte with some efault values of the variance comonents an effect size, an then we mae ajustments to some of these efault values base on substantive knowlege that we ha about the etails of the articular stuy we woul be running. However, we note that all of these values, even the values we secifically ajuste, are ultimately eucate guesses about the true arameter values, an so they shoul be seen as imlicitly containing a egree of uncertainty. In acknowlegment of this, our recommenation is that researchers use these eucate guesses as a starting oint in comuting ower an samle size estimates for a range of lausible values of the arameters for the stuy at han. Otimal Design With Crosse Ranom Factors On the basis of the above ower analyses for our five esigns, we can raw some general conclusions an formulate a variety of rules of thumb concerning the otimal esign of exeriments where a samle of articiants reson to a samle of stimuli. In this section, we iscuss the maximum attainable ower in an exeriment once the number of stimuli is fixe; when statistical ower is best serve by increasing the number of articiants or by increasing the number of stimuli; the relative efficiency of the five esigns; an finally the statistical merits of esigns involving multile blocks of stimuli as a way of ealing with the roblem of time-consuming stimulus resentation. Maximum Attainable Power One otentially surrising fact that we learn from our ower analyses about the use of esigns with crosse ranom factors is that statistical ower oes not aroach unity as the number of articiants aroaches infinity, whenever there is variation ue to stimuli. Instea, statistical ower asymtotes at a maximum theoretically attainable ower value that eens on the effect size, the number of stimuli, an the variability of the stimuli. To see this in the fully crosse esign, consier what haens to the noncentrality arameter (nc FC ) as the number of articiants () goes to infinity: lim nc FC lim V PC V SC q V E q q. V SC Notice here that when goes to infinity, the terms in the enominator involving V E an V PC isaear, but the numerator an the term involving V SC are both unchange, so that in the limit the noncentrality arameter converges to a finite (an ossibly small) value. In Figure 7 we lot the maximum attainable ower in the fully crosse esign as a function of the effect size (), the number of stimuli (q) an the variance in the stimulus sloes (V SC ). These are contour lots just like the lots islaye earlier; the ifference is that instea of lotting the observe ower for ifferent combinations of arameters, they lot the uer boun for ower at various combinations of arameters. The lots of maximum ower for the other four esigns all look nearly ientical to those for the fully crosse esign. The rimary lesson for exerimenters following from the statistical fact that maximum attainable ower oes not aroach unity with increasing numbers of articiants is that it is very imortant to think carefully about the samle of stimuli in the exeriment before the ata collection begins. Once ata collection begins, the effect size an the number an roerties of the stimuli can no longer be change, an thus the maximum ower that woul be attainable in the exeriment has in essence alreay been ecie. All that then remains is to recruit a certain number of articiants, which etermines how close to this maximum ower level the actual ower ens u being. Exerimenters may believe that they can comensate for a subotimal samle of stimuli by simly recruiting a larger number of articiants, but in fact the egree to which this sort of comensation can take lace is quite limite, a oint which we iscuss in more etail in the next section. The fact is that in many entirely realistic exerimental situations, the maximum attainable ower with an infinite number of articiants can be quite low even for etecting true, large effects. For examle, in an exeriment emloying the stimuli-within-conition esign, uner the stanar case VPCs escribe above, where the true effect size is large at 0.8, an where there are a total of eight stimuli (four stimuli er conition) a samle size which we susect many exerimenters woul consier erfectly aequate for a stimulus samle the maximum attainable ower is only about.41. However, if we just ouble the samle size of stimuli to a still relatively moest 16 (eight er conition), then the maximum ower to etect a large effect goes u to about.78. Another imlication of the maximum attainable ower being less than one is that in stuies involving crosse ranom factors, a irect relication with high statistical ower is often theoretically imossible when the original stuy emloye a relatively small number of stimuli. Recently, researchers have stresse the imor-

14 14 WESTFALL, KENNY, AND JUDD Figure 7. Contour lots of the maximum attainable ower with an infinite number of articiants in the fully crosse esign as a function of the effect size, number of stimuli (on a log scale), an the roortion of stimulus sloe variance. The contour lots of maximum ower for the other four esigns all look nearly ientical to the lots shown here. tance of conucting irect relications (Francis, 01; Ioanniis, 01; Koole & Lakens, 01; Nosek, Sies, & Motyl, 01; Oen Science Collaboration, 01) an also emhasize that relication attemts shoul ieally have high statistical ower, even when (erhas esecially when) the original stuy was unerowere (Brant et al., 014). Most tyically, those who woul relicate a stuy with higher ower than the original might emloy the same stimuli but increase the number of articiants to a level they consier aequate. However, to the extent that there is substantial stimulus variability, relications with high ower may in fact not be feasible using only the stimuli inclue in the original stuy. Sufficiently high ower may be obtainable only by increasing the number of stimuli, as well as the numbers of articiants. We oint out that a irect relication woul virtually never use the same articiants as the original stuy. Instea, it woul robably just be assume that the new articiants were rawn from the same general oulation as the original articiants. Why then shoul we require the stimuli to be exactly the same as in the original exeriment, rather than just being rawn from the same oulation? We strongly suggest that, when conucting a relication of a stuy involving crosse ranom factors, it woul be beneficial for the researchers conucting the relication to augment the stimulus set, rawing from the same oulation of stimuli as in the original stuy, in orer to ensure that statistical ower is accetably high. We iscuss these issues in more etail in a comanion article (Westfall, Ju, & Kenny, 014). Increasing the Samle Sizes of Particiants Versus Stimuli Not surrisingly, exeriments with larger samle sizes of both articiants an stimuli always have greater ower than exeriments with smaller samle sizes. However, the question of whether one can exect a greater benefit to statistical ower by increasing the samle size of articiants or the samle size of stimuli turns out to een on several ifferent asects of the exeriment. Here we formulate two rules of thumb that researchers can use to ientify situations where it woul be better to increase the samle size of articiants or of stimuli. Formally, these rules of thumb are base on an analysis of the conitions uner which the rate of change in the noncentrality arameter with resect to the number of articiants is greater than the rate of change in the noncentrality arameter with resect to the number of stimuli (or vice versa). For examle, to examine when it is better to increase the samle size of stimuli rather than the samle size of articiants in the fully crosse esign, we first set nc FC q nc FC. With some work, we can simlify this inequality to obtain V E V S C qv E q V P C. (1) We refer to this inequality in iscussing the rules of thumb below. The first rule of thumb is that it is generally better to increase the samle size of whichever ranom factor is contributing more ranom variation to the ata, consiering the nature of the resonse variable an the relevant roerties of the articiants an stimuli. Statistically seaking, this is because the rimary statistical consequence of aing articiants or aing stimuli is that the corresoning articiant or stimulus variance comonents, resectively, woul become further iminishe in their contribution to the enominator of the noncentrality arameter, an there is a greater avantage to iminishing the contribution of larger variance comonents comare to smaller variance comonents. An equivalent way of stating this is the following: