A Reevaluation of Assessment Center Construct-Related Validity

Transcription

1 A Reevlution of Assessment Center Construct-Relted Vlidity Milton V. Choon 1, Mrk C. Bowler 2 & Jennifer L. Bowler 2 1 RTI Interntionl, Reserch Tringle Prk, USA 2 Deprtment of Psychology, Est Crolin University, Greenville, USA Correspondence: Mrk C. Bowler, Deprtment of Psychology, Est Crolin University, 104 Rwl, Greenville, NC 27858, USA. Tel: E-mil: owlerm@ecu.edu Received: Jnury 10, 2012 Accepted: Mrch 7, 2012 Pulished: My 1, 2012 doi: /ijm.v7n9p3 URL: Astrct Recent Monte Crlo reserch (Lnce, Woehr, & Mede, 2007) hs questioned the primry nlyticl tool used to ssess the construct-relted vlidity of ssessment center post-exercise dimension rtings (PEDRs) confirmtory fctor nlysis of multitrit-multimethod (MTMM) mtrix. By utilizing hyrid of Monte Crlo dt genertion nd univrite generlizility theory, we exmined three primry sources of vrince (i.e., persons, dimensions, nd exercises) nd their interctions in 23 previously pulished ssessment center MTMM mtrices. Overll, the person, dimension, nd person y dimension effects ccounted for comined 34.06% of vrince in ssessment center PEDRs (16.83%, 4.02%, nd 13.21%, respectively). However, the lrgest single effect cme from the person y exercise interction (21.83%). Implictions nd suggestions for future ssessment center reserch nd design re discussed. Keywords: ssessment center, construct-relted vlidity, generlizility theory, multitrit-multimethod 1. Introduction Over the pst three decdes, ssessment centers hve een stedily gining populrity with orgniztions worldwide, for the purposes of employee selection nd development (Eurich, Kruse, Cigulrov, & Thornton, 2009; Joiner, 2002; Spychlski, Quinones, Gugler, &Pohley, 1999). Constructed from the currencies of exercises nd dimensions (Hoffmn, Melcher, Blir, Kleinmnn, & Ldd, 2011), ssessment centers ttempt to evlute individul performnce levels on set of jo-relted skills. However, reserch on the design nd function of ssessment centers hs rought to light some prolemtic results. Of prticulr concern is the construct-relted vlidity prdox (Arthur, Woehr, &Mldegen, 2000). Specificlly, ssessment centerspper to exhiit oth content- nd criterion-relted vlidity while simultneously lcking construct-relted vlidity. Tht is, despite utilizing high-fidelity work simultions tht provide excellent predictors of jo performnce (Thornton & Mueller-Hnson, 2004), the primry source of vrince in post-exercise dimension rtings (PEDRs) ppers to e differences in the ssessment center exercises rther thn differences in the dimensionson which the ssessment center is sed (Lnce, Lmert, Gewin, Lievens, & Conwy, 2004; Lnce et l., 2000; Petrides, Weinstein, Chou, Furnhm, & Swmi, 2010). However, the results of recent Monte Crlo study (Lnce, Woehr, & Mede, 2007) cll into question the ccurcy of the primry nlyticl technique tht ws used in the mjority of the studies tht hve concluded tht ssessment centers lck construct-relted vlidity, nmely, confirmtory fctor nlysis (CFA) of multitrit-multimethod mtrix (MTMM). In light of the strong possiility of erroneous results from CFA of n MTMM mtrix, we re left with lck of clrity regrding the current stte of ssessment centers. However, there is new evidence tht univritegenerlizility theory (Hrtley, Ro, & LMotte, 1978; Hemmerle& Hrtley, 1973) constitutes n pproprite nlyticl methodology for evluting the construct-relted vlidity of ssessment center PEDRs (Bowler &Woehr, 2008). As with CFA of MTMM mtrix, when pplied to ssessment center PEDRs, generlizility theory llows dimension nd exercise effects to e exmined. Additionlly, this method fcilittes the evlution of severl other relevnt fctors such s the person eing rted (i.e., the rtee), the ssessor ssigning the rting (i.e., the rter), nd ll of the relevnt interctions. Moreover, generlizility Pulished y Cndin Center of Science nd Eduction 3

2 theory does not require n itertive estimtion procedure tht focuses on fitting theoreticl models it simply identifies the proportion of vrince ssocited with ech component. 1.1 Study Ojectives The present study sought to clrify the ssessment center construct-relted vlidity issue y utilizing hyrid of Monte Crlo dt genertion nd generlizility theory. Previously pulished MTMM mtrices were used s the popultion prmeters for generting Monte Crlo dt sets, which in turn were nlyzed vi generlizility theory to estimte the reltive contriutions of the person, dimension, nd exercise effects, s well s theirrespective interctions. 2. Construct-Relted Vlidity of Assessment Centers The current unitrin frmework of vlidity holds tht content-, construct-, nd criterion-relted vlidity re ll points long the roder spectrum of construct vlidtion (Binning & Brrett, 1989). Content-relted vlidity is n indictor of the similrity etween the suject mtter of mesure nd the domin tht it purportedly represents, construct-relted vlidity is n indictor of the reltionship etween mesure nd the theoreticl concept it is intended to mesure, nd criterion-relted vlidity is n indictor of the reltionship etween mesure nd relevnt ehviorl indictor of performnce. The process of construct vlidtion involves ll three components nd seeks to determine whether test indeed mesures wht it is designed to mesure, the strength of the reltionship etween the mesure nd this construct, nd how redily one cn drw inferences from the scores the mesure produces. These three vlidity estimtes re linked to one nother nd one cn logiclly ssume tht the estlishment of two of them necessrily sserts the existence of the third (Arthur et l., 2000). Therefore, if mesure is shown to possessoth content- nd criterion-relted vlidity, then one cn resonly ssume tht the mesure in question lso possesses construct-relted vlidity. This logic, however, hs not een shown to pply to ssessment center PEDRs. In fct, reserch hs frequently demonstrted tht ssessment centers disply oth criterion- nd content-relted vlidity while filing to demonstrte construct-relted vlidity. More specificlly, ssessment centers demonstrte sustntil content-relted vlidity in tht they re high-fidelity simultions of mngeril work (Thornton & Mueller-Hnson, 2004) tht re typiclly sed on jo nlysis tht generted criticl work-relted performnce dimensions (Thornton, 1992). Moreover, whilessessment centers hve evidenced strong criterion-relted vlidities (Arthur, Dy, McNelly, &Edens, 2003; Chn, 1996), the construct-relted vlidity of ssessment centersppers to e prolemtic. Theoreticlly, the vrince in ssessment center PEDRs should e lrgely due to the individul s performnce on the prticulr dimensions tht re eing mesured. Tht is, rtee s PEDR on prticulr dimension should e function of the his or her ehvior relting to tht dimension (i.e., person y dimension effect) rther thn the exercise from which the dimension is eing mesured. Previous primry studies, however, hve produced inconsistent results (cf.arthur et l., 2000; Bycio, Alvres, & Hhn, 1987; Lnce, Foster et l., 2004; Lnce et l., 2000). In n ttempt to ddress these inconsistencies, three extensive met-nlytic reevlutions of ssessment center construct-relted vlidity hve een conducted (Bowler &Woehr, 2006; Lnce, Lmert et l., 2004; Lievens& Conwy, 2001; ). In one wy or nother, ll three of these reviews utilized previously pulished ssessment center MTMM mtrices. Moreover, ech one utilized CFA techniques to nlyze these mtrices, with ech study generting somewht different results nd conclusions. 2.1 Previous Reviews of Assessment Center Construct-Relted Vlidity Lievens & Conwy (2001) In their review of ssessment center construct-relted vlidity, Lievens nd Conwy (2001) renlyzed 34 MTMM mtrices from 24 ssessment center studies. Their nlyses focused on fitting six different models to ech of the MTMM mtrices vi CFA. These models included (1) correlted dimension model, (2) correlted exercise model, (3) correlted dimension-correlted exercise model, (4) single dimension-correlted exercise model, (5) direct product model, nd (6) correlted uniqueness model. The correlted dimension model represents n ssessment center with PEDRs tht re only influenced y the dimensions eing rted, wheres the correlted exercise model represents n ssessment center with PEDRs tht re only influenced y the exercises utilized to evlute the dimensions. The correlted dimension-correlted exercise model is mixedmodel in whichssessment center PEDRs re function of oth the dimension eing rted nd the exercise in which they re eing rted. The single dimension-correlted exercise model represents n ssessment center with PEDRs tht re influenced y single dimension fctor (e.g., g) s well s the exercises. Both the direct product nd the correlted uniqueness models re sttisticl vritions of the correlted dimension-correlted exercise model. In the direct product model the correltions etween mnifest vriles (e.g., PEDRs) re modeled s multiplictive function etween dimensions nd exercises (rther thn s n 4 ISSN E-ISSN

3 dditive function), nd in the correlted uniqueness model the exercise effects re not explicitly modeled in fvor of estimting them from the correltions mong the uniquenesses. Results from the Lievens nd Conwy (2001) nlyses indicted tht oth the correlted dimension nd the correlted exercise models fit the dt poorly, providing n dequte fit for only 3% nd 29% of the mtrices, respectively. The single dimension-correlted exercise model performed somewht more fvorly, producing n cceptle fit for 53% of the mtrices. In contrst, oth the correlted dimension-correlted exercise model nd direct product model demonstrted cceptle fit with 85% nd 81% of the dt, respectively. However, the correlted uniqueness model emerged s the est performnce model, fitting 88% of the MTMM mtrices. Bsed on the conclusion tht the correlted uniqueness model provided the most pproprite fit for the dt, Lievens nd Conwy estimted tht the men proportion of vrince tht ws ttriutle to dimensions ws.34 nd tht the men proportion of vrince tht ws ttriutle to exercises ws lso.34. Furthermore, they noted tht these vlues vried gretly nd tht severl models fetured highly intercorrelteddimensions, with the verge dimension intercorreltion eing.71. Nonetheless, they concluded tht dimensions ply greter role in ssessment center rtings thnws previously suspected, ut the study fell short of demonstrting tht dimensions hve greter impct thn the exercises from which they re drwn Lnce, Lmert et l. (2004) Due to severl prolemtic sttisticl issues with the LievensndConwy (2001) review, prticulrly concerns with the utiliztion of the correlted uniqueness model, Lnce, Lmert et l. (2004) reexmined thelievens nd Conwy (2001) dt s well s five dditionl MTMM mtrices (for totl of 39 MTMM mtrices). As with the Lievens nd Conwy review, Lnce, Lmert et l. individully nlyzed ech of the MTMM mtrices vi CFA. However, in their evlution, only three models were tested: (1) the correlted dimension-correlted exercise model, (2) the correlted exercise model, nd (3) the single dimension-correlted exercise model. Overll, their results indicted tht the correlted dimension-correlted exercise model provided good fit for only two of the 39 MTMM mtrices (5%) nd the correlted-exercise model provided good fit for only 2% of the mtrices. In contrst, the single dimension-correlted exercise model produced dmissile solutions for sustntil numer of the MTMM mtrices (49%) with men dimension fctor loding of.14 nd men exercise fctor loding of.52. Thus, they concluded tht exercise effects preside over dimension effects in ssessment center PEDRs Bowler nd Woehr (2006) Bowler nd Woehr (2006) conducted similr study tht lso reexmined similr set of previously reported ssessment center MTMM mtrices. However, rther thn individully renlyze ech MTMM mtrix, they chose to recode the dt from 35 MTMM mtrices into single MTMM mtrix comprised of six dimensions nd six exercises. They then conducted CFA on this single MTMM mtrix to ssess the fit of severl different nlyticl models: (1) thecorrelted dimension model, (2) thecorrelted exercise model, (3) thesingle dimension-correlted exercise model, (4) single dimension-uncorrelted exercisesmodel, (5) thecorrelted dimension-correlted exercise model, nd (6) thecorrelted dimension-uncorrelted exercise model. Their results noted tht, with the exception of the correlted dimension model, ll of the models demonstrted resonle fit; however, the correlted dimension-correlted exercise model ws mrginlly superior, generting men dimension fctor loding of.47 nd men exercise fctor loding of.58. Thus, contrry to the findings of Lnce, Lmert et l. (2004), Bowler nd Woehr concluded tht exercise effects do not necessrily tke precedence over dimension effects. Overll, these three forementioned reviews produced notly different results. Wheres results from the Lnce, Lmert et l. (2004) nlysis sided with other individul studies tht concluded tht ssessment center PEDRs re function of exercises nd not dimensions (cf.bycio et l., 1987; Lnce et l., 2000; Lnce, Foster, et l., 2004; Lnce et l., 2007; Lievens& Conwy, 2001), Bowler nd Woehr (2006) nd Lievens nd Conwy (2001) produced results to contrry. Interestingly, the findings of ll of these studies were sed on the sme sttisticl technique CFA of MTMM mtrix. Unfortuntely, the suitility of this method hs recently een clled into question. 2.2 Confirmtory Fctor Anlysis nd MTMM Mtrices To ssess the ppropriteness of conducting CFA of n MTMM mtrix, Lnce et l. (2007) conducted Monte Crlo evlution of this procedure. For this study, they generted three popultion models,ech representing different model of ssessment center functioning found in previous ssessment center reserch. Ech of these models ws similr in tht echmodeled n ssessment center comprised of five dimensions, ech of which ws mesured in three exercises. However, the models differed with regrd to the nture of their ltent structure. The Pulished y Cndin Center of Science nd Eduction 5

4 first two models represented the correlted dimension-correlted exercise(cdce) nd single dimension-correlted exercise (1DCE) models. The third model ws sed on n uncorrelted dimension-correlted exercise model tht lso took into ccount n overll person effect (UDCE+g). Using these three models s popultion prmeters, Lnce et l. (2007) generted 500 smple MTMM mtrices for ech model. A CFA ws then conducted to ssess the fit of ech of the three popultion models (i.e., CDCE, 1DCE, UDCE+g) to ech of the smple MTMM mtrices. For ech model, they noted whether the model converged within 1,000 itertions nd whether it produced n dmissile solution. When fitting the three popultion models (i.e., CDCE, 1DCE, UDCE+g) to the smple dt, Lnce et l. (2007) noted severl prolemtic results. First nd foremost, when the CDCE model converged to n dmissile solution, it fit only 61% of the CDCE dt. Thus, for 39% of the CDCE dt, the CFA ws unle to generte n pproprite solution. However, when the 1DCE model ws pplied to the CDCE dt it converged to n dmissile solution for 100% of the dt. Similrly, when fitting the popultion models to the UDCE+g dt, the UDCE+g model produced n dmissile solution for only 52% of the dt. However, when the 1DCE model ws pplied to the UDCE+g dt, it converged to n dmissile solution for 99% of the dt. In contrst, when the 1DCE smple dt ws exmined, the CDCE model converged to n dmissile solution for only 1% of the dt nd the UDCE+g model converged to n dmissile solution for only 10% of the dt. However, the 1DCE model converged to n dmissile solution for 100% of the dt. Overll, the findings of Lnce et l. (2007) suggest tht the results produced y CFA of n MTMM mtrix my e inccurtely ised towrds the 1DCE model. Tht is, regrdless of the true nture of the dt, the 1DCE model is most likely to emerge s the pproprite model. Thus, the conclusions of the pst 20 yers of reserch regrding the construct-relted vlidity of ssessment centers, nd more specificlly, the results of numerous studies tht hve concluded tht ssessment center PEDRs re est represented y the 1DCE model (e.g., Bycio et l., 1987; Fleenor, 1996; Lnce, Foster, et l., 2004; Lnce, Lmert, et l., 2004; Lnce et l., 2000; Schneider & Schmidt, 1992), my include some erroneous results. 2.3 Applying Generlizility Theory to Assessment Center Rtings A solution for voiding the prolems ssocited with CFA of n MTMM mtrix is the ppliction of generlizility theory to the evlution of ssessment center PEDRs. Generlizility theory (i.e., vrince prtitioning) exmines the different sources of vrince ssocited with PEDRs nd estimtes the reltive impct tht ech source hs on the rtings (Cronch, Gleser, Nnd, &Rjrtnm, 1972). Bowler nd Woehr (2008) hve sserted tht this method is etter suited for exmining the construct-relted vlidity of ssessment centers in tht (1) it tkes into ccount sources of vrince tht cnnot e ssessed with the trditionl CFA of n MTMM mtrix, nd (2) it genertes results tht re representtive of the popultion dt Novel Sources of Vrince Three primry sources of vrince my e redily ssessedvi the ppliction of generlizility theory tossessment center PEDRs: the person eing rted, the dimension eing rted, nd the exercise from which the rting ws mde. The dimension effect represents the vrince in PEDRs tht cn e ttriuted to certin dimensions receiving overll higher or lower rtings in comprison with other dimensions, wheres the exercise effect represents the degree to which certin exercises engender higher/lower PEDRs cross ll dimensions. The person effect represents generl performnce fctor tht is not due to interctions with the dimension effect or exercise effect. In ddition to the three min effects, the generlizility theory pproch evlutesthree relevnt interction effects. The person y dimension interction effect represents the mount of vrince ttriuted to individuls scoring higher/lower on certin dimensions regrdless of the exercise eing employed. This is indictive of cross-situtionl consistency, which represents the degree to which person s dimension rtings re consistent cross exercises nd serves s n indictor of construct-relted vlidity. In contrst, the person y exercise interction effect represents the mount of vrince ttriuted to person receiving generlly high/low scores on certin exercises, regrdless of the dimension eing mesured. This provides evidence of situtionl specificity, mening tht it represents the degree to which person does not score consistently on dimension rtings cross exercises. Situtionlspecificity is counterintuitive to ssessment center functioning ecuse it suggests tht person s dimension rtings re lrgely function of their performnce on prticulr exercise rther thn the dimension tht ws intended to e mesured (Lnce et l., 2000). Lstly, the dimension y exercise interction effect represents the mount of vrince ttriuted to specific dimension eing mesured in specific exercise. This effect hs previously een exmined y Lievens et l. (2006) nd concerns the oservility of prticulr dimension in prticulr exercise. 6 ISSN E-ISSN

5 Overll, exmining these sources of vrince provides sustntilly more informtion regrding the functioning of ssessment centerpedrs thn does trditionl CFA. Trditionl CFA methods only provide rudimentry view sed on dimension nd exercise effects. These two effects long with their noted levels of intercorreltion do not provide much detil regrding wht is occurring with the PEDRs. Dimension effects re regrded s good nd exercise effects re d, ut little dditionl informtion is provided. In contrst, the generlizility theory pproch provides more detiled informtion regrding the dimension nd exercise effects. For exmple, wht would simply e descried s dimension effect vi CFA cn e decomposed into (1) pure dimension effect tht comes from different dimensions eing systemticlly rted differently cross ech other, (2) person y dimension interction tht stems from differences in individuls performnce on the dimensions, nd (3) dimension y exercise interction tht comes from differences in rtings sed on the exercise from which it is drwn Monte Crlo Support In their Monte Crlo exmintion of ssessment center PEDRs, Bowler nd Woehr (2008) generted four popultion models. The first ws n optiml model tht fetured low exercise lodings nd high dimension lodings. The second ws mixed model with oth high dimension nd high exercise lodings. The third ws worst-cse model tht hd high exercise lodings nd low dimension lodings. The fourth nd finl popultion represented null model tht fetured low dimension lodings nd low exercise lodings. For ech of these models, Bowler nd Woehr generted 500 smple dt sets. This dt ws then sujected to generlizility theory nlyses. Overll, the nlyses from Bowler nd Woehryielded pproprite results for ech popultion model tht ws nlyzed. For exmple, the primry source of vrince in the optiml model ws the person y dimension effect with 29.79%. In comprison, for the worst-cse model, the primry source of vrince ws the person y exercise effect with 27.35%. Thus, they concluded tht the ppliction of univrite generlizility theory is n ccurte nd pproprite method for evluting ssessment center functioning Previous Empiricl Applictions Generlizility theory hs rrely een pplied to exmintions of ssessment center construct-relted vlidity. Arthur et l. (2000) first pplied this technique to ssessment centers when they exmined four sources of vrince in ssessment center PEDRs s well s the relevnt two-wy interctions. Specificlly, they evluted the vrince ttriuted to the person, dimension, exercise, rter, nd person y dimension, person y exercise, nd dimension y exercise interction effects. Their results indicted tht the person, dimension, nd person y dimension effects ccounted for sustntil mount of the totl systemtic vrince in ssessment center PEDRs (59%). Specificlly, the dimension min effect ccounted for roughly 21% of the totl vrince, the person effect ccounted for 18%, nd the person y dimension effect ccounted for 20% of the totl vrince. In contrst, the exercise effect ccounted for less thn 1% of the vrince, nd the person y exercise effect ccounted for roughly 5% percent of the totl vrince.similrly, Jckson et l. (2005) exmined the reltive vrince ccounted for y person, dimension, nd exercise fctors (including the ssocited two-wy interctions). Their results indicted tht tht the person effect, dimension effect, nd person y dimension interction ccounted for pproximtely 36% of the vrince (31.9%, 2.2%, nd 1.8%, respectively). However, Jckson et l. lso noted tht pproximtely 37% of the vrince in ssessment center PEDRs ws ttriutle to comintion of the exercise effect (3.2%) nd the person y exercise interction (34%). Most recently, Bowler &Woehr (2009) exmined the vrince ccounted for y person, dimension, exercise, nd rter fctors. They noted sustntil person y exercise effect (28.4%) s well s strong person y dimension effect (16.0%), nd moderte dimension nd person effects (6.7% nd 4.6%, respectively). All of these studies re reltively unique in tht they provide considerle support for the expected person, dimension, nd person y dimensions effects (i.e., those effects ssocited with the oject of mesurement nd considered supportive of construct-relted vlidity). However, results of these three studies diverge with respect to vrince estimtes ssocited with exercise nd person y exercise effects (i.e., those effects trditionlly considered unsupportive of ssessment center construct-relted vlidity). Unfortuntely, these re the only three studies to pply generlizility theory to ssessment center PEDRs from functioning ssessment centers (i.e., not experimentl dt). Thus, there is reltively smll smple of studies tht do not rely solely on CFA of n MTMM mtrix from which to se ny conclusions regrding the overll nture of the construct-relted vlidity of ssessment centers. 3. Reevluting Assessment Center Construct-Relted Vlidity Despite the plethor of studies conducted on the ssessment center construct vlidity prdox, few hve chosen to utilize generlizility theory to exmine wht role exercises nd dimensions ply in ssessment center Pulished y Cndin Center of Science nd Eduction 7

6 PEDRs (Arthur et l., 2000; Jckson, Stillmn, & Atkins, 2005). The present study seeks to rectify this shortcoming y pplying hyrid of Monte Crlo dt genertion nd generlizility theoryin n effort to summrize the current stte of ssessment center without utilizing prolemtic methodology. Specificlly, we sought to pply generlizility theory to the sustntil numer of ssessment centers tht hve een previously studied (e.g., Arthur et l., 2001; Bycio et l., 1987; Schneider & Schmitt, 1992). However, in most cses it is impossile to retrieve the necessry rw dt from these studies only the pulished MTMM mtrices re ville. Thus, for ech MTMM mtrix tht met our inclusion criteri, we generted 500 smple dtsets sed on the MTMM mtrix. Ech of these dtsets ws then sujected to generlizility theory nlysis, nd the results were ggregted to provide generl informtion out the likely nture of tht prticulr ssessment center. 3.1 Assessment Center Design Fetures In ddition to the primry nlyses, nlyses were lso conducted to evlute the reltionship etween prticulr ssessment center design fetures nd differences in sources of vrince. In prticulr, we exmined differences in vrince components in reltion to (1) the totl numer of dimensions ssessed, (2) crossed versus non-crossed design, (3) the use of ehviorl checklists, (4) the purpose of the ssessment center (selection versus development), nd (5) the occuption of the ssessor Numer of Dimensions The numer of dimensions rted y n ssessment center hs previously een shown to e vitl to the construct-relted vlidity of ssessment center PEDRs (Woehr& Arthur, 2003). Trditionlly, it is held tht the greter the numer of dimensions tht re utilized y n ssessment center, the more difficult it ecomes for rter to distinguish etween sid dimensions (Gugler& Thornton, 1989). Thus, ssessment centers with five or fewer dimensions should demonstrte superior construct-relted vlidity. Hypothesis 1: Assessment centers with five or fewer dimensions will hve significntly higher person, dimension, nd person y dimension effects,nd significntly lower exercise nd person y exercise effects, thn ssessment centers with more thn five dimensions Crossed vs. Non-crossed Design Assessment centers tht utilize fully crossed design re those in which every dimension included in the ssessment center is mesured in ech exercise(thornton & Mueller-Hnson, 2004). In contrst, ssessment centers with non-crossed design typiclly include only prticulr suset of dimensions in ech of the exercises. For exmple, n ssessment center might feture n in-sket exercise tht exmines three dimensions (nlysis, judgment nd delegtion), role-ply exercise tht only mesures two dimensions (nlysis nd confronttion), nd lederless group discussion exercise tht mesures three dimensions (nlysis, judgment, nd confronttion). The fundmentl nture of ssessment center design the mesurement of multiple trits vi multiple methods is ligned with the trditionl construct-relted vlidity model of Cmpell nd Fiske (1959). Thus, the fully crossed design should demonstrte superior construct-relted vlidity. Hypothesis 2: Assessment centers utilizing fully crossed design will hve significntly higher person, dimension, nd person y dimension effects, nd significntly lower exercise nd person y exercise effects, thn ssessment centers with more thn five dimensions Behviorl Checklists A ehviorl checklist is tool utilized in some ssessment centers to ssist rters in ssigning their rtings. Behviorl checklists llow rters to simply indicte if ny of set of prticulr ehviors ws exhiited y the ssessee during the exercise. Typiclly, ehviorl checklists re employed in n ttempt to reduce the cognitive lod on ssessors nd to improve the ccurcy of their rtings (Donhue, Truxillo, Cornwell, &Gerrity, 1997). Thus, the use of ehviorl checklists should improve the construct-relted vlidity of ssessment center rtings. Hypothesis 3: Assessment centers tht utilize ehviorl checklists will hve significntly higher person, dimension, nd person y dimension effects, nd significntly lower exercise nd person y exercise effects, thn ssessment centers with more thn five dimensions Assessment Center Purpose Selection-oriented ssessment centers re used to distinguish individuls who re est suited for prticulr position sed on their overll ssessment center performnce, wheres developmentl ssessment centers re used to exmine the strengths nd weknesses of current employees in n ttempt to highlight their weknesses nd fcilitte their improvement. Trditionl thinking holds tht developmentl ssessment centers should hve 8 ISSN E-ISSN

7 clerer dimensions, in tht their purpose is to provide dimension-level feedck (Kudish, Ldd, & Doins, 1997; Woehr& Feldmn, 1993). Thus, developmentl ssessment centers should demonstrte superior construct-relted vlidity thn those tht re designed for selection purposes. Hypothesis 4: Assessment centers tht re intended for developmentl purposes will hve significntly higher person, dimension, nd person y dimension effects, nd significntlylower exercise nd person y exercise effects thn ssessment centers tht re designed for selection purposes Assessor Occuption Trditionlly, ssessment center employees re clssified s either professionl ssessors or mngers employed y the orgniztion utilizing the ssessment center. Mngers re rters tht typiclly come from the within the orgniztion in which the ssessment center is eing used. They my e supervisors or individuls who re knowledgele in terms of the jo eing ssessed (e.g., suject mtter experts). In contrst, professionl ssessors generlly come from outside the orgniztion nd hve extensive knowledge of ssessment center functioning nd the ehviors tht re representtive of the dimensions eing ssessed. Previous reserch hs suggested tht rtings mde y professionl ssessors re typiclly superior to those of mngers (Woehr&Arthus, 2003). Hypothesis 5: Assessment centers tht employ professionl ssessors will hve significntly higher person, dimension, nd person y dimension effects, nd significntly lower exercise nd person y exercise effects, thn ssessment centers tht employ mngers employed y the orgniztions ssessors. 4. Method 4.1 Literture Serch nd Inclusion Criteri To retrieve the dt necessry for the nlyses, serch of the relevnt online dtses ws conducted (e.g., PsycINFO, PsycARTICLES, Disserttion Astrcts, etc.). Serch terms included ssessment center in conjunction with multitrit-multimethod nd construct vlidity. In ddition to serching the relevnt dtses, studies tht were included in the previous reviews (i.e., Bowler &Woehr, 2006; Lnce, Lmert, et l., 2004; Lievens& Conwy, 2001) were exmined for potentil sources of dt. In order to e included in the nlyses, studies needed to provide n ssessment center sed MTMM mtrix s well s the mens nd stndrd devitions of ech vrile. Unlike previous ssessment center met-nlyses (e.g., Lnce et l., 2004) tht utilized MTMM mtrices, mens nd stndrd devitions were necessry to estimte the dimension nd exercise effects so tht we could provide the most ccurte nlysis of ech ssessment center. Susequently, some more recently pulished MTMM mtrices were not included in our nlyses (e.g., Lnce, Foster, Nemeth, Gentry, &Dollinger, 2007). Additionlly, the studies MTMM mtrices hd to (1) come from n ssessment centers in which rtings of individul dimensions were collected for ech exercise (i.e., PEDRs) nd (2) report the smple size on which the correltions were sed. When ny of this informtion ws missing (e.g., if study provided only men correltions cross dimensions nd/or exercises), the uthors were contcted in n ttempt to otin the full mtrix. Overll, 14studies dting from 1987 to 2004 nd reporting 23 unique MTMM mtrices met the inclusion criteri. Of the 23 MTMM mtrices, 52% were included in the Lievens nd Conwy (2001) review, 61% were included in the Lnce, Lmert, et l. (2004) review, nd 65% were included in the Bowler nd Woehr (2006) review. Tle 1 presents the complete list of studies providing mtrices for this study s well s severl of the ssessment centers design fetures. Tle 1. Study demogrphics Smple size Fully crossed design? Study No. of dimensions No. of exercises Behviorl checklists? AC purpose Assessor occuption Arthur et l. (2000) Yes No Developmentl Mngers Becker (1990) No No Selection Mngers No No Selection Mngers No No Selection Mngers No No Selection Mngers Pulished y Cndin Center of Science nd Eduction 9

8 Bycio et l. (1987) Yes No Both Mngers Chorvt (1994) No Yes Developmentl Assessors Fredericks (1989) No No Selection Mngers Jnsen & Stoop (2001) Yes No Selection Mngers Kolk et l. (2002) Yes No Selection? Yes No Selection? Kolk et l. (2003) Yes No Developmentl Assessors Yes No Developmentl Assessors Yes No Developmentl Assessors Yes No Developmentl Assessors Kudisch et l. (1997) No No Developmentl Assessors Lnce et l. (2004) No Yes Promotionl Mngers No Yes Promotionl Mngers Lievens et l. (2003) No Yes Selection Assessors Prker (1991) No No Developmentl Mngers Roie et l. (2000) Yes Yes Developmentl Assessors Yes Yes Developmentl Assessors Schneider & Schmitt (1992) Yes Yes Developmentl Assessors Notes. Severl other fctors re often included in AC reviews (e.g., prticipnt-to-ssessor rtio, trnsprtent dimensions, exercise similrity, ssessor trining, length of trining). However, do to lck of informtion regrding these fcets they could not e included in the modertor nlyses nd re thus not included in this tle. ftc = MTMM mtrices tht filed to converge to n dmissle solution in the Lnce et l. (2004) nlyses. *MTMM mtrices not included in the Lnce et l. (2004) nlyses. 4.2 Dt Genertion Similr to the Lnce et l. (2007) Monte Crlo study, we treted MTMM mtrices s popultion mtrices tht served s the sis from which we generted dt. However, unlike Lnce et l., we utilized MTMM mtrices tht were drwn from ctul ssessment centers. Furthermore, rther thn generte dditionl MTMM mtrices for CFA-sed nlyses, we used ech of the MTMM mtrices s the sis for generting 500 smple dt sets. To generte the dt, Cholesky decomposition ws first clculted for ech MTMM mtrix. A set of rndom, normlly distriuted numers were then multiplied y the Cholesky mtrix. Ech set contined the sme numer of vriles s the originl MTMM mtrix with the numer of sets eing equl to the originl smple size of the prticulr ssessment center. Thus, the smple dt sets were ech creted with the smple size reported in the originl study nd were comprised of the pproprite dimensions nd exercises. For exmple, ech of the 500 dt sets tht were generted sed on the Bycio et l. (1987) MTMM mtrix hd smple size of 1170 nd 40 vriles representing the comintion of eight dimensions nd five exercises in their fully-crossed design. 4.3 Dt Anlyses Ech smple dt set ws nlyzed vi the SAS VARCOMP procedure with the MIVQUE0 method (Hrtleyet l., 1978).The MIVQUE0 method ws chosen over trditionl Mximum Likelihood or Type 1 methods due to the sustntil difference in computtionl time. The MIVQUE0 method is one of the most efficient computtionl methods ville (Bell, 1985; Brennn, 2001). For one smple dtset, the verge MIVQUE0 nlysis runtime ws less thn one second. In comprison, the verge mximum likelihood nlysis runtime for one smple dtset ws pproximtely one minute. Thus, given the sustntil numer of nlyses tht were conducted for this study (~12,000), the mximum likelihood method ws not fesile. Furthermore, the MIVQUE0 method mkes no ssumptions regrding the normlity of the dt nd cn e utilized for nlyzing unlnced designs (i.e., ssessment centers tht do not fully cross dimensions nd exercises; Hrtleyet l., 1978). Thus, for this study we felt tht this ws the superior nlyticl technique. 10 ISSN E-ISSN

9 All nlyses were sed on three-fcet (person, dimension, nd exercise) rndom effects design with one repeted mesure (ssessment center rtings). In ddition to the min effects, ll of the two-wy interctions were included (i.e., person y dimension, person y exercise, nd dimension y exercise). However, not ll studies ssessed ll dimensions in ll exercises. Therefore, not ll MTMM mtrices tht were ssessed were fully crossed. However, ll dimensions were ssessed in t lest two exercises, so dimensions were not nested within exercises. This represents type of frctionl fctoril design tht hs een shown to e firly roust with respect to lower-order effects in the model (Cochrn & Cox, 1957; Connor & Young, 1961; Federer, 1955). 5. Results The results for the initil simultions sed on the 23 MTMM mtrices re displyed in Tle 2. Vrince estimtes were otined which corresponded to the seven modeled effects: person, dimension, exercise, person y dimension, person y exercise, dimension y exercise, nd error (i.e., vrince ttriutle to ll components not estimted in the model s well s mesurement error). Overll, n initil exmintion of the vrince estimtes indictes firly complex pttern of results. Specificlly, those components ssocited with the person nd dimensionmin effects ccounted for sustntil proportion of the totl vrince (34.06%). Overll, this vrince ws distriuted cross the person min effect (16.83%), the dimension min effect (4.02%), nd the person y dimension interction (13.21%). Moreover, the vrince ssocited with the exercise min effect ws miniml (2.60%), s ws the vrince ttriuted to the dimension y exercise interction (4.17%). However, other thn error, the lrgest effect ws ccounted for y the person y exercise interction (21.83%). Overll, this suggests tht, cross ssessment centers, significnt proportion of the vrince in ssessment center rtings is in fct ssocited with the oject of mesurement (person, dimension, nd person y dimension), with sustntil proportion of this vrince ttriuted to the person y dimension interction. Tht is, considerle mount of vrince is ssocited with fcets tht re supportive of construct-relted vlidity nd proper ssessment center functioning. However, sustntil proportion of the vrince in rtings is lso ttriutle to person y exercise interction. Thus, long with the good vrince, there is lso considerle mount of d vrince in ssessment center PEDRs or, t the very lest, vrince tht supports the situtionl specificity issue discussed y Lnce et l. (2000). Although some ssessment centers pper to hve PEDRs tht re ppropritely sed on the person y dimension interction (e.g., Fredericks, 1989; Roie, Osurn, Morris, Etchegry, & Adms, 2000, mtrix 2) others do not (e.g., Bycio et l., 1987, Prker, 1991). Similrly, lthough some ssessment centers pper to engender reltively smll mount of situtionl specificity (e.g., Becker, 1990, mtrix 1; Chorvt, 1994; Fredericks, 1989; Roie et l., 2000, mtrix 1), others pper to engender mssive mounts of situtionl specificity (e.g., Bycio et l., 1987; Jnsen & Stoop, 2001). This finding is disturing ecuse it implies tht true ssessment center functioning vries gretly cross differing ssessment centers. Further exmintion of these results highlights n interesting issue regrding the rnge of oserved vrince ssocited with ech of the effects in the model. The stndrd devitions of the verge oserved vrince for oth the person y dimension nd person y exercise interction effects were greter thn 10% (SD = 11.70% nd 16.89%, respectively). In contrst, the stndrd devitions of the person, dimension, exercise, nd dimension y exercise effects were lower thn 10% (SD = 8.24%, 5.51%, 5.06%, nd 3.95%, respectively). Thus, there ppers to e greter similrity cross ssessment centers regrding the sources of vrince ssocited with the person, dimension, exercise, nd dimension y exercise effects. In contrst, the person y dimension nd person y exercise effects seem to vry sustntilly cross ssessment centers. An dditionl issue of note is tht the Bycio et l. (1987) study produced results tht differed sustntilly from those of the other studies exmined. In prticulr, the Bycio et l. MTMM mtrix produced person y exercise interction tht ccounted for roughly 66% of the totl vrince. This is considerly greter thn the next highest person y exercise effect of 47% tht ws generted y the Jnsen et l. (2001) MTMM mtrix. To evlute the impct of this potentil outlier, seprte mens nd stndrd devitions were clculted with the Bycio et l. study excluded. Overll, this nlysis yielded only sutle differences. The components ssocited with the person nd dimension effects ccounted for sustntil proportion of the totl vrince (34.06%), with this vrince distriuted cross the person min effect (17.31%), the dimension min effect (4.15%) nd the person y dimension interction (13.80%). The vrince ssocited with the exercise min effect ws miniml (2.66%), s ws the vrince ttriuted to the dimension y exercise interction (4.35%). However, other thn error, the lrgest effect ws still ttriutle to the person y exercise interction (19.84%). Pulished y Cndin Center of Science nd Eduction 11

10 Tle 2. Monte Crlo simultion results Proportions of Vrince Study s p s d s x s pd s px s dx s pdx,e Arthur et l. (2000) 23.70% 23.00% 0.00% 19.20% 5.10% 3.70% 25.30% Becker (1990) % 1.50% 0.80% 24.60% 0.10% 2.00% 58.80% % 0.70% 0.60% 20.80% 2.40% 4.80% 58.60% % 0.50% 1.00% 17.60% 13.90% 1.20% 46.80% % 0.40% 0.90% 9.70% 15.90% 1.20% 48.30% Bycio et l. (1987) 6.30% 1.20% 1.20% 0.30% 65.70% 0.20% 25.10% Chorvt (1994) 4.10% 9.0% 0.70% 16.40% 3.00% 8.00% 58.80% Fredericks (1989) 4.00% 0.40% 2.70% 43.90% 2.30% 6.00% 40.40% Jnsen et l. (2001) 20.40% 0.10% 0.50% 5.40% 47.20% 1.90% 24.50% Kolk et l. (2002) % 7.00% 0.20% 7.80% 16.20% 0.40% 50.90% % 2.00% 0.40% 10.30% 30.20% 1.20% 35.50% Kolk et l. (2003) % 2.80% 0.30% 9.60% 24.70% 1.20% 44.10% % 1.90% 0.40% 11.00% 31.20% 0.80% 42.80% % 0.00% 0.50% 14.50% 41.30% 3.90% 11.90% % 0.20% 0.00% 13.40% 39.50% 6.20% 8.40% Kudisch et l. (1997) 8.30% 1.70% 1.10% 7.30% 22.70% 11.30% 47.60% Lnce et l. (2004) % 6.60% 18.60% 2.30% 32.40% 5.90% 14.60% % 8.90% 15.40% 2.00% 34.40% 12.00% 8.95% Lievens et l. (2003) 11.00% 13.40% 10.70% 18.50% 9.90% 6.80% 29.20% Prker (1991) 4.50% 0.00% 1.20% 0.01% 15.00% 13.40% 65.20% Roie et l. (2000) % 6.40% 2.10% 5.70% 22.10% 2.40% 31.00% % 3.80% 0.20% 42.20% 2.10% 0.40% 26.00% Schneider & Schmitt (1992) 19.30% 1.00% 0.20% 1.30% 24.90% 1.10% 52.30% Including Bycio et l. (1987) M 16.83% 4.02% 2.60% 13.21% 21.83% 4.17% 37.18% SD 8.24% 5.51% 5.06% 11.70% 16.89% 3.95% 17.14% Excluding Bycio et l. (1987) M 17.31% 4.15% 2.66% 13.80% 19.84% 4.35% 37.73% SD 8.10% 5.61% 5.17% 11.63% 14.26% 3.94% 17.33% Notes. p = person; d = dimension; x = exercise; pd = person y dimension; px = person y exercise; dx = dimension y exercise; pdx,e = error. Vrince components greter thn 10% re in oldfce. The lrgest vrince component for ech MTMM mtrix is underlined. 5.1 Assessment Center Design Fetures As previously noted, we exmined differences in vrince components in reltion to (1) the totl numer of dimensions ssessed, (2) crossed versus non-crossed design, (3) the use of ehviorl checklists, (4) the 12 ISSN E-ISSN

11 purpose of the ssessment center(selection versus development), nd (5) the occuption of the ssessor. Results from ech of the 23 dt sets were seprted ccording the chrcteristic eing exmined, nd t-tests were conducted to evlute the significnce of the differences (see Tle 3).However, due to the mssive mount of dt, ll differences were significnt. Thus, we clculted Cohen s d effect sizes (Cohen, 1988) for ech comprison to etter judge the differences etween design fetures. To this end, we considered effects sizes tht were greter thn.40 to e sustntil. Tle 3. Modertors of vrince prtitioning results Modertor s p s d s x s pd s px s dx s pdx,e Proportions of Vrince Numer of dimensions 5 (k = 14, n = 7000) 20.89% 3.59% 0.57% 15.16% 20.89% 2.24% 36.64% > 5 (k = 9, n = 4500) 11.43% 4.69% 5.76% 9.33% 25.61% 7.20% 38.02% Fully crossed design? No (k = 11, n= 6000) 13.07% 3.92% 4.89% 14.08% 15.71% 6.61% 43.39% Yes (k = 12, n= 5500) 20.96% 4.12% 0.51% 11.78% 29.17% 1.96% 31.48% Behviorl checklists? Yes (k = 8, n= 3500) 19.28% 7.01% 6.86% 11.39% 21.23% 5.25% 30.77% No (k = 15, n= 8000) 16.27% 2.72% 0.07% 13.54% 23.39% 3.72% 40.59% AC purpose c Selection (k = 9. n= 5500) 16.38% 2.88% 1.98% 16.65% 17.63% 2.84% 43.67% Developmentl (k = 11, n= 4500) 18.54% 4.52% 0.61% 12.89% 21.07% 4.77% 37.58% Assessor occuption d Non-Mngers (k = 10, n= 5500) 19.53% 2.97% 0.61% 13.55% 23.48% 3.93% 34.82% Mngers (k = 10, n= 4500) 14.84% 3.95% 3.91% 13.29% 21.38% 4.76% 39.13% Cohen's D Effect Sizes Numer of dimensions 5 (k = 14, n = 7000) > 5 (k = 9, n = 4500) (.29,.37) (.16,.24) (.96, 1.04) (.43,.51) (.24,.31) (1.33, 1.41) Fully crossed design? No (k = 11, n= 6000) Yes (k = 12, n= 5500) (.91,.98) (.00, 07) (.77,.85) (.16,.23) (.80,.88) (1.23, 1.30) Behviorl checklists? Yes (k = 8, n= 3500) No (k = 15, n= 8000) (.91,.98) (.78,.86) (1.15, 1.25) (.02,.14) (.08,.16) (.32,.40) AC purpose c Selection (k = 9. n= 5500) Developmentl (k = 11, n= 4500) (.19,.27) (.25,.32) (.32,.40) (.29,.36) (.19,.27) (.43,.51) Assessor occuption d Non-Mngers (k = 10, n= 5500) Mngers (k = 10, n= 4500) (.46,.54) (.14,.22) (.54,.62) (.01,.06) (.07,.15) (.15,.23) Notes: Effect/design feture pirs with different superscripts re sttisticlly different from one nother t p<.05. c The purpose of the Bycio et l. (1987) AC did not fll exclusively into either ctegory, thus it ws not included in this nlysis. d TheKolk et l. (2003) study did not identify the ssessor occuption, thus it ws not included in this nlysis. k = numers of studies with the chrcteristic in question. n = totl smple size of Monte Crlo dt sets with the chrcteristic in question. p = person; d = dimension; x = exercise; pd = person y dimension; px = person y exercise; dx = dimension y exercise; pdx,e = error. Effect sizes greter thn.80 re in oldfce type Numer of Dimensions A lrge effect ws found for oth the exercise nd dimension y exercise effects (see Tle 3). Assessment centers with five or fewer dimensions demonstrted sustntilly lower exercise effect (d =.99) nd sustntilly higher person y dimension interction (d =.47). However, no sustntil differences emerged for the person, dimension, or person y exercise interction effects. Thus, Hypothesis 1 ws only prtilly supported. Furthermore, it should lso e noted tht the dimension y exercise interction effect ws lrger when greter thn five dimensions were utilized (d = 1.37). This suggests tht, for these ssessment centers, dimension oservility is of greter concern when more thn five dimensions re fetured. Pulished y Cndin Center of Science nd Eduction 13

12 5.1.2 Crossed vs. Non-crossed Design When compring crossed nd non-crossed designs, lrge effects were found for the person, exercise, nd person y exerciseeffects. Both the person nd person y exercise effects were sustntilly lrger for fully crossed ssessment centers (d =.94 nd d =.84, respectively). In contrst, the exercise effects were smller for those ssessment centers tht were fully crossed (d =.81).Thus Hypothesis 2 ws only prtilly supported. A fully crossed design improved the construct-relted vlidity vi two sources of vrince (i.e., the person nd exercise effect) ut reduced the construct-relted vlidity for nother source of vrince (i.e., the person y exercise interction). Additionlly, similr to the finding concerning the numer of dimensions exmined y n ssessment center, dimension oservility is greter concern for ssessment centers tht re not fully crossed (d = 1.27) Behviorl Checklists For the ssessment centers nlyzed, lrge effects were displyed for the person (d =.95), dimension (d =.82), nd exercise effects (d = 1.20). Specificlly, ll three of these effects were lrger when ehviorl checklists were used. However, there were no sustntil differences for the remining effects. Thus, Hypothesis 3 ws only prtilly supported. Although ehviorl checklists improved the construct-relted vlidity of ssessment center rtings vi two sources of vrince (i.e., the person nd dimension effects), the use of these checklists reduced the construct-relted vlidity rtings vi nother source of vrince (i.e., the exercise effect) Assessment Center Purpose Overll, none of the sources of vrince demonstrted lrge effect sizes sed on the purpose of the ssessment center (selection vs. development). Thus, Hypothesis 4 ws not supported. The purpose of the ssessment center ppers to e irrelevnt to construct-relted vlidity. Interestingly, developmentl ssessment centers demonstrted sustntilly greter dimension y exercise interction (d =.47) Assessor Occuption Moderte effects were displyed for the person nd exercise effects. The person effect ws sustntilly greter in the ssessment centersin which non-mngers were used s ssessors in comprison with those ssessment centers tht utilized mngers (d =.50). In contrst, the exercise effect ws sustntilly smller when non-mngers were used s ssessors (d =.58). Thus, Hypothesis 5 ws only prtilly supported. Assessment centers tht utilized professionl ssessors exhiited significntly higher person effect nd significntly lower exercise effect. Otherwise, the occuption of the ssessor hd little impct on the construct-relted vlidity of ssessment center rtings. 6. Discussion There hve now een lmost three decdes of reserch on the construct-relted vlidity of ssessment center PEDRs. Much of this reserch hs een less thn promising nd tends to diminish the effects of dimensions on the vrince of PEDRs (e.g., Lnce, Lmert et l., 2004). Primrily utilizing CFA techniques, reserch hs shown tht models with lrge exercise effects tend to fit PEDR dt etter thn those with lrge dimension effects (e.g., Bycio et l., 1987; Fleenor, 1996;Lnce, Foster, et l., 2004; Lnce, Lmert, et l., 2004; Lnce et l., 2000; Schneider & Schmidt, 1992). The present study sought to chllenge these findings y utilizing reltively overlooked technique, univritegenerlizility theory, to exmine the sources of vrince in ssessment center PEDRs. Our results led us to elieve tht the current view of ssessment center construct-relted vlidity is not s dire s some suggest (e.g., Lnce, 2008) nd my e etter clrified with the utiliztion of generlizility theory. Specificlly, lthough the person y exercise interction ws the lrgest single source of vrince (21.83%), the person, dimension, nd person y dimension effects together ccounted for even more vrince (34.06%). Thus, the very effects tht re commonly considered indictors of construct-relted vlidity (cf. Arthuret l., 2000; Bowler &Woehr, 2009) contriuted more vrince thn the effects tht re considered to e indictive of situtionl specificity (i.e., the exercise nd person y exercise effects ccounted for 24.43% of the vrince). Overll, the previling view tht ssessment centersdisply either dimension effects or exercise effects is too simplistic. This is rther rod generliztion tht does not provide much useful informtion for the improvement of ssessment center design. With the utiliztion of generlizility theory, we now hve etter lnguge with which to discuss the issue. Our results suggest tht lthough PEDRs re comprised of sustntil mount of vrince ssocited with construct-relted vlidity (i.e., person, dimension, nd person y dimension effects), sustntil mount of situtionl specificity lso exists in mny ssessment centers (i.e., person y exercise effects). However, oth of these results were highly vrile cross ssessment centers. Thus, the most 14 ISSN E-ISSN