SBORNÍK PRACÍ FILOZOFICKÉ FAKULTY BRNĚNSKÉ UNIVERZITY STUDIA MINORA FACULTATIS PHILOSOPHICAE UNIVERSITATIS BRUNENSIS P 4 / 200 Martn Jelínek, Petr Květon, Dalbor Vobořl Assessment of Response Pattern Aberrancy n Eysenck Personalty Inventory Abstract The presented study deals wth two well standardzed and relatvely powerful ndces based on Item Response Theory z 3 and F 2 whch we used for detectng aberrant respondng n the feld of personalty assessment. The ndces were used on the Neurotcsm and Extraverson scales from Eysenck Personalty Inventory. The research sample conssted of 427 subjects. Both ndces were computed for the whole sample and t was found that they yelded smlar results. We selected subjects wth the lowest z 3 ndex (hghest aberrancy) values and further analyzed ther response patterns. Our results suggest that n some cases the nconsstency was caused by faulty or careless respondng, but n other cases by dscrepancy between the subject s test behavor (tem respondng) and the theoretcal construct beng measured. Keywords measurement, personalty, psychometrcs Assessment of Response Pattern Aberrancy n Eysenck Personalty Inventory Most psychologcal assessments stand on nterpretatons of questonnares. But there s no secret that valdty of questonnares nferences s often dsputed. In one case a tested person can respond carelessly or randomly at all, n another case the tested person gves answers truthfully but hs/her answers are not consstent wth a normatve pattern accordng to the theoretcal background of the method. Accordng to Rese and Waller (993), at least three factors nfluence respondng consstency: a) mperfecton shared by all probablstc measurement systems; b) faulty tem respondng (careless, msreadng, etc.); c) ndvdual dfferences to the normatve theoretcal expectatons. Factor c) can be consdered as lack of tratedness whch refers to the agreement between subject s behavor and the construct measured. Inconsstences n respondent s answers can be psychometrcally detected usng varous ndces. Bascally these ndces are based ether on Classcal test the-
38 Martn Jelínek, Petr Květon, Dalbor Vobořl ory (CTT) or on Item response theory (IRT). The general approach to the problem n CTT can be demonstrated on Guttman s deas. Consder a test consstng of several tems. When answered by a relatvely large sample we can sort the tems by ther dffcultes (based on proporton correct score p) n ascendng order. A perfectly consstent ndvdual pattern wll then look lke a lne of s (correct answers) followed by 0s (wrong answers) and s called Guttman vector. Guttman error s such a case when any s rght to any 0 n the sorted lne (.e. subject s able to correctly answer a certan tem but fals on a less dffcult one). Pattern consstng of maxmum number of Guttman errors (all 0s rght to the s) represents Guttman reversed vector. Ths dea gave a ground for many other ndces lke Sato s Cauton Index C, or Tatsuoka and Tatsuoka s Norm Conformty Index NCI (Mejer, Sjtsma, 994). In order to examne the valdty of an ndvdual response pattern n the IRT framework, varous person-ft (PF) ndces are used. Although a number of ndces usng dfferent computaton methods have been developed (see e.g. Tatsuoka, 996; Drasgow, Levne, McLaughln, 987, etc.), ther prncple wll always rely on the degree of consstence of a gven response pattern and a vald pattern based on the relevant IRT model (Embretson, Rese, 2000). Advantageously, the PF ndces based on IRT do not requre the tested persons to go through the whole test wth exact number of tems and thus may be useful e.g. n the feld of computerzed adaptve testng (CAT). Although many ndces have been proposed n the scentfc texts, there are two of them that are well standardsed (ther values do not systematcally vary across dfferent levels of θ) and show suffcent power for detectng aberrant response patterns (Drasgow, Levne, McLaughln, 987). Better known s the ndex orgnally proposed by Drasgow, Levne, Wllams (985) whch was orgnally known as the z 3 ndex but others refer to t also as the Z L ndex (Embretson, Rese, 2000) or l z (Mejer, Sjtsma, 994). It has been dscussed that ths ndex s also computably extendable for use wth polytomous tems (Drasgow, Levne, Wllams, 985). The latter one s F 2 ndex (Rudner, 983). These two ndces wll be further dscussed. The logc of calculaton The core of IRT s the model of tem respondng. It allows determnng a way n whch the tested person assumng we know hs/her level of ablty s lkely to respond to an tem. Intutvely t apples that e.g. a very able person should resolve a very easy tem wth almost 00% certanty. The person-ft ndces are obtaned n the followng way: when the tested person has completed the whole test and we have thereby ganed the nformaton about the person s level of ablty, we shall revew the ndvdual tems and determne at each response whether t corresponds wth hs/her ablty. The level of correspondence of the ablty and the response s expressed by means of lkelhood. It s then possble to determne the general credblty for the whole set of responses (.e. test).
Assessment of Response Pattern Aberrancy n Eysenck Personalty 39 Calculaton of z 3 The calculaton (Drasgow, Levne, Wllams, 985) s based on the lkelhood of a specfc observed response pattern of a concrete subject expressed n the logarthmc form,.e. as follows: l o n = u log P ( ˆ) θ + ( u)logq ( ˆ) θ = where denotes the concrete tem from the sample of n tems, log s the natural logarthm, θˆ = estmated ablty, u = concrete response by a proband ( correct response, 0 wrong response), P = probablty of the correct response and Q = -P (probablty of the wrong answer). Probablty of the correct response s computed as follows: P ( ˆ) c ( c ) Da ( ˆ ) b e where a denotes tem dscrmnaton parameter, b dffculty, and c guessng. When c =0 ths three-parameter model s equal to two-parameter model wth guessng parameter omtted. D s a constant equal to.7, whch makes the logstc functon close to the normal ogve functon. The expected value of l o can be computed as: E n 3 ( ˆ) θ = P ( ˆ)log θ P ( ˆ) θ + Q ( ˆ)log θ Q ˆ ( θ ) = In comparson to l o, concrete response u s replaced by the probablty of a correct response. The standardzed z 3 ndex s then defned as: lo E3( ˆ) θ z3 = σ ( ˆ) θ 3 where condtonal standard devaton: σ [ log( P( ˆ) θ / Q ( ˆ )] 2 n 3 ( ˆ) θ = P ( ˆ) θ Q ( ˆ) θ θ ) = If the total log-lkelhood of a specfc subject s response pattern (l o ) equals ts expected value for a gven level of the latent trat (E 3 ), then the z 3 ndex value s zero. When z 3 s postve, the credblty of responses s hgher than that predcted by the model. The unlkely response patterns are ndcated by hgh negatve ndex values. Consderng that the ndex dstrbuton should be roughly consstent wth the standard normal dstrbuton wth an average of 0 and a standard devaton of
40 Martn Jelínek, Petr Květon, Dalbor Vobořl, we can set e.g. a -2.33 value as an approxmate value separatng % of the most unsutable patterns n the populaton. The mathematc sequence of calculaton of F 2 Ths ndex assesses person-ft by determnng the degree of devaton from the expected response based on the latent trat level, summarzed across tems. The formula (Rudner, 983) s as follows: F n = 2 = n = 2 [ u P( ˆ) θ ] P( ˆ) θ Q ( ˆ) θ F 2 can be consdered as dscrepancy scores and so larger values ndcate napproprate response patterns. Gven that, we can assume negatve and very hgh correlaton wth z 3 ndex. In the presented study we apply the procedures to data from Eysenck Personalty Inventory wth the am to ) descrbe and compare z 3 and F 2 ndces, 2) dentfy persons wth low credblty of response pattern and suggest possble reasons for that. Method Instrument The Eysenck Personalty Inventory, Czech verson (Mglern, Vonkomer, 979): the questonnare dentfes two basc personalty trats, extraverson and neurotcsm. Each scale contans 24 dchotomous tems; the respondent expresses hs/ her agreement or dsagreement wth a gven statement by Yes/No response. Both scales provde suffcent levels of relablty (KR20 E =0.78; KR20 N =0.8). The nventory ncludes also a le scale consstng of 8 tems. Items from all scales are mxed together and some of them are reversed (wth the excepton of neurotcsm scale). Sample The data comes from the research by Blatný, Osecká and Hrdlčka (998). The sample comprsed 427 grammar school students from varous towns of the South- Moravan Regon (57% of women; average age of 6). Data analyss We calbrated the tems of the neurotcsm and extraverson scales wth two parameter logstc model (2PL) n the Blog 3. (Mslevy, Bock, 997) software usng margnal maxmum lkelhood estmaton method. Trat level scores (θ) for all subjects were estmated usng maxmum lkelhood method. Together eght
Assessment of Response Pattern Aberrancy n Eysenck Personalty 4 subjects (four n Neurotcsm scale and four n Extraverson scale) wth perfect negatve or postve profles were omtted from analyss because fnte estmate of θ could not be reached. The mean and varance of θ were fxed at 0.0 and.0, respectvely. Usng tems characterstcs (b and a), ndvdual trat levels and tem responses, we were able to compute z 3 and F 2 ndces for all subjects n our sample. For those nterested, we gve our nteractve spreadsheet at http://www. psu.cas.cz/cato/z3.html free for download. Results The tems of extraverson and neurotcsm scales were calbrated usng 2PL model because we found out that PL model would be too restrctve (tem dscrmnaton parameters, as can be seen n table, are dversfed) and 3PL model s sutable only f one could expect guessng aspect n answerng behavor. We present obtaned tem parameters n table. TABLE. Item Parameters for EPI Scales neurotcsm extraverson # n EPI b a # n EPI b a 2 n -0.99 0.65 e -0.93 0.78 5 n2 0.0 0.48 3 e2.68 0.39 8 n3-0.59 0.59 6 e3.75 0.45 0 n4 -.0 0.52 9 e4-0.32 0.49 3 n5 -.65 0.43 e5.07 0.30 6 n6 -.08 0.80 5 e6-0.79 0.42 9 n7.60 0.62 8 e7-0.80.6 22 n8 -.29 0.7 20 e8-0.72.3 25 n9 -.76 0.55 24 e9 2.30 0.80 27 n0 0.20 0.86 26 e0.27 0.22 30 n 0.40 0.93 29 e -.74 0.92 32 n2 -.74 0.43 3 e2-0.40 0.92 37 n3-0.8 0.50 35 e3-0.50.24 39 n4-0.49 0.70 38 e4-0.93 0.5 4 n5.20 0.72 40 e5 0.3 0.5 45 n6 0.73 0.42 44 e6 0.42 0.34 47 n7-0.0 0.8 46 e7 0.73 0.5 50 n8 2.0 0.63 48 e8-2.43 0.2 53 n9.06 0.63 5 e9.55 0.60 55 n20 0.27 0.79 54 e20 0.03 0.70 58 n2 0.39 0.40 57 e2-0.03 0.45 60 n22 0.74 0.92 59 e22 -.2 0.79 64 n23 0.6 0.27 62 e23 0.7 0.90 66 n24 2.08 0.6 65 e24-0.54 0.3
42 Martn Jelínek, Petr Květon, Dalbor Vobořl Descrpton and comparson of z 3 and F 2 ndces Table 2 presents descrptve statstcs for both scales (extraverson and neurotcsm) and both ndces. z 3 shows mean value around zero and standard devaton around (z-dstrbuton), whle F 2 shows mean value around and standard devaton 0.2. TABLE 2. Descrptve Statstcs for F 2 and z 3 Indces z 3n z 3e F 2n F 2e Mean 0.08 0.09 0.98 0.99 Medan 0.2 0.9 0.96 0.96 Std. devaton 0.94 0.97 0.20 0.20 Note. z 3n and z 3e z 3 ndces for neurotcsm and extraverson; F 2n and F 2e F 2 ndces for neurotcsm and extraverson Further analyss revealed that both ndces are almost equvalent (r e =-.96; r n =-.98). Therefore, further results wll present only one of the ndces (z 3 ). FIGURE. Dstrbuton of z 3 Index for Neurotcsm and Extraverson,00 0,90 0,80 cumulatve proporton. 0,70 0,60 0,50 0,40 0,30 0,20 z3n 0,0 z3e normsnv 0,00-2,50-2,00 -,50 -,00-0,50 0,00 0,50,00,50 2,00 2,50 z score Fgure dsplays a slghtly negatvely skewed dstrbuton of z 3 ndces for both EPI scales. Lower number of values s located below the zero value than above zero, whch s consstent wth comparson of mean and medan n table 2. It s also mportant to note that z 3 ndex does not correlate (r n =-.02; r e =.05) wth θ and
Assessment of Response Pattern Aberrancy n Eysenck Personalty 43 thus measure of nconsstency level s not confounded by the overall θ value. Moreover, we found no relaton between z 3 ndces for both scales (r=.09). Identfyng persons wth low credblty of response pattern Table 3 and 4 brng a lot of useful nformaton. The z 3n and z 3e lnes show the essental nformaton about the credblty of ndvdual patterns (accompaned by values of the other scale ndex). Grayed felds n the tables clearly dfferentate those fve cases wth hgh negatve values of the ndces from the ones wth hgh postve values. TABLE 3. Fve Lowest and one Hghest z 3n Scores wth Assocated Response Patterns Subject # 6 # 24 # 49 # 43 # 338 # 8 Item Resp. P Resp. P Resp. P Resp. P Resp. P Resp. P n9 0.9 0.92.79.88 0.7.8 n2 0.85.87 0.73 0.82.67.76 n5.85.87 0.72.8 0.65.75 n8 0.9 0.93 0.76 0.88.64.79 n4.83 0.86 0.66.78 0.57.69 n6.92.94.73 0.87.60.78 n.86.89 0.67.8 0.55.7 n3 0.78.8.60.73 0.50.63 n3.78.82 0.56 0.72.45.60 n4 0.80 0.84 0.54.72 0.4.59 n7.72 0.78.39 0.6 0.26.45 n2.62.66.4 0.55 0.32 0.44 n23.56 0.58 0.44.52 0.39 0.46 n0 0.67.74.3 0.55 0.9 0.37 n20.63.70.30.52 0.9 0.35 n2.55.59 0.38.49.3 0.4 n.6.69 0.23.47 0.3 0.29 n6 0.49.53 0.32.43 0.25 0.34 n22.48.57 0.5.35 0.08 0.9 n9.40 0.46 0.8 0.3.2 0.2 n5 0.35.4.3 0.26 0.08 0.6 n7.28.33 0..2.07 0.3 n8 0.9 0.24 0.07 0.4 0.05 0.09 n24.9.23.07.4.05 0.09 θ n 0.68 0.9-0.35 0.33-0.80-0.7 z 3n -3.39 =-0.58) -3. =-0.57) -3.05 =0.27) -3.05 =-0.64) -2.65 =0.08) 2.34 =-0.3) Note. Grayed felds desgnate unexpected (by means of tem response functons) answers. Items are ordered by ascendng dffculty. Resp. subject s responses ( keyed; 0 nonkeyed); P expected response probablty.
44 Martn Jelínek, Petr Květon, Dalbor Vobořl TABLE 4. Fve Lowest and one Hghest z 3e Scores wth Assocated Response Patterns Subject # 95 # 25 # 2 # 353 # 96 # 47 Item Resp. P Resp. P Resp. P Resp. P Resp. P Resp. P e8 0.68.74 0.69.63 0.6.66 e 0.90.97.92.80 0.7.86 e22 0.77 0.9.8 0.62 0.5.70 e4.63.78 0.66 0.52 0.45.57 e.69.87.73 0.53.42.6 e7 0.72.93 0.77 0.48 0.32.60 e6 0.58.72.6.49 0.43.54 e8.7 0.94.77 0.43 0.27.58 e24 0.53.64.55 0.46 0.42.49 e3 0.60 0.90.67 0.32 0.9 0.45 e2 0.53.8 0.59.33.23 0.43 e4.50 0.67.53.39 0.33 0.44 e2.45.6 0.47 0.35.29 0.39 e20.40.64 0.44 0.26 0.9 0.32 e5.47 0.53 0.48 0.44 0.42.45 e23.32.63 0.37.7. 0.24 e6.40 0.52 0.42 0.33 0.29 0.36 e7.29.46 0.32.20.6 0.24 e5.33.43.34.27.24 0.30 e0.36 0.43 0.37 0.3 0.28 0.33 e9.3.26 0.5 0.08 0.06 0.0 e2.2.32.23.6.3 0.8 e3 0.7.28.9.2 0.0 0.4 e9 0.03.08.03 0.0 0.0 0.02 θ e -0.32 0.53-0.7-0.84 -.8-0.59-4.87-4.5-2.69-2.67-2.62 2.25 z 3e (z3n=-2.8) (z3n=0.30) (z3n=-0.3) (z3n=n/a) (z3n=-.93) (z3n=-0.98) Note. Grayed felds desgnate unexpected (by means of tem response functons) answers. Items are ordered by ascendng dffculty. z 3n score for subject ID 353 s assgned N/A (= not avalable) because of a postve nfnte estmaton of theta. Resp. subject s responses ( keyed; 0 nonkeyed); P expected response probablty. Graphcal depcton of unexpected responses helps us to dentfy most problematc tems where all our nconsstent subjects gve responses totally aganst the model probablty. We dentfed one such tem n case of neurotcsm scale (n24: Do you suffer from sleeplessness? ) and two n case of extraverson scale (e2: Are you usually carefree? and e5: Would you do almost anythng for a dare? ).
Assessment of Response Pattern Aberrancy n Eysenck Personalty 45 Dscusson It s evdent that psychologcal testng, and partcularly the testng of mass character, s affected, apart from other sources of errors, also by the knd of errors stemmng from the very behavor of persons wthn the testng procedure. PF ndces can be used n the area of performance testng to dentfy persons cheatng n the test, but also to dentfy persons wth specfc abltes or dagnose cogntve errors of the concrete subjects (Tatsuoka, Tatsuoka, 983). Interestng s also the use of PF ndces n personalty testng as presented n our study. The personalty research based on the questonnare methods s faced wth the problem of dentfcaton of faulty answers, partcularly n certan groups of subjects such as adolescents (Handel et al. 2006). Another source of bas can be lack of, so called, tratedness. In general, measurement methods (lke questonnare EPI) stand on nomothetc trat construct and are usually constructed usng advanced statstcal procedures (factor analyss). It s obvous that ths approach does not allow us to expect that every sngle ndvdual wll perfectly ft to the construct. The way of nterpretng PF ndces depends on the aspratons. Researcher workng wth large amounts of data can use them to clear hs/her dataset from unnterpretable (nconsstent wth the model) records wthout lookng after the causes. Another stuaton arses n clncal settng. A sklled psychologcal professonal knows that the same questonnare score does not always mean the same qualty for two dfferent clents. Usng a PF ndex he/she can assure that wth suffcent value of the ndex, the score s nterpretable by means of the theory behnd the method. But when non-nterpretable score s found, he/she should ask for the reasons. From ths pont of vew, there may be two man types of nconsstences: a) clent dd not understand tem contents, msread tems, or e.g. flled out the answer sheet carelessly (faulty answerng); b) clent honestly flled out the questonnare but the theoretcal construct for some reason does not apply to hm/her and there s a need for addtonal nformaton to explore. In our study we dentfed several subjects wth hghly nconsstent answerng n scales of extraverson and neurotcsm. Our goal was not only to dentfy them but also try to suggest possble reasons for t. Rese and Waller (993) explored data obtaned usng Multdmensonal Personalty Questonnare (MPQ). For dstncton between faulty respondng and lack of tratedness they used ) several valdty scales ncorporated n MPQ (e.g. Varable Response Inconsstency Scale, whch s based on comparson of answers on tems wth smlar content) and 2) comparson of PF ndex values between dfferent scales. The EPI questonnare used n our study contans Le Scale whch s focused rather on socal desrablty than on consstency of respondng and thus s not sutable for our purposes. Therefore, we had only one clue for dstncton between faultness and lack of tratedness comparson of z 3 ndex scores from the two scales. When the ndex ndcates nconsstency n both scales, then we assume that t s faultness that plays a major role here. Such cases are ID # 95 and # 96 as shown n table 4.
46 Martn Jelínek, Petr Květon, Dalbor Vobořl When the nconsstent tendency was found only n one of the scales, we nclned to nterpret t rather as a lack of tratedness. It was also nterestng to look on tems wth a hgh rate of hghly unexpected responses. Such an tem can be found n the Neurotcsm scale (n24: Do you suffer from sleeplessness? ). Although sleeplessness ndeed s an mportant ndcator of neurotcsm, there surely exst people that are sleepless from other reasons than that (e.g. neurologcal dsorders). Even though the PF ndces were found useful n many areas (dscussed above), also less promsng attempts for applcaton can be found n the relevant lterature. For example, Brown and Harvey (2003) tred to dentfy fakng n Fve Factor Model personalty test (Conscentousness and Agreeableness scales) but wth no substantal success. As t s well known from practce, motvated ndvduals are often capable of pretendng desred (n hs/her opnon) characterstcs and dong t consstently. References Blatný, M., Osecká, L., & Hrdlčka, M. (998). Zdroje sebehodnocení u temperamentových typů [Sources of self-esteem n temperament types]. Československá psychologe 42, 297-305. Brown, R. D., & Harvey, R. J. (2003). Detectng personalty test fakng wth approprateness measurement: fact or fantasy? Paper presented at the 2003 Annual Conference of the Socety for Industral and Organzatonal Psychology, Orlando. Drasgow, F., Levne, M. V., & McLaughln, M. (987). Detectng napproprate test scores wth optmal and practcal approprateness ndces. Appled Psychologcal Measurement, 59-79. Drasgow, F., Levne, M. V., & Wlams, E. A. (985). Approprateness measurement wth polychotomous tem response models and standardzed ndces. Brtsh Journal of Mathematcal and Statstcal Psychology 38, 67-86. Embretson, S. E., Rese, S. P. (2000). Item response theory for psychologsts. London: Lawrence Erlbaum Assocates. Handel, R. W., Arnau, R. C., Archer, R. P., & Dandy, K. L. (2006). Evaluaton of the MMPI-2 and MMPI-A True Response Inconsstency (TRIN) scales. Assessment 3, 98-06. Mejer, R. R., & Sjtsma, K. (994). Detecton of Aberrant Item Score Patterns: A Revew of Recent Developments. Research Report, Faculty of Educatonal Scence and Technology, Unversty of Twente 94, 3-26. Mglern, B., & Vonkomer, J. (979). Eysenckov osobnostný dotazník EOD [Eysenck Personalty Inventory EPI]. Bratslava: Psychodagnostcké a ddaktcké testy. Mslevy, R. J., & Bock, R. D. (997). Blog 3.. Mooresvlle, IN: Scentfc Software Internatonal. Rese, S. P. & Waller, N. G. (993). Tratedness and the assessment of response pattern scalablty. Journal of Personalty and Socal Psychology 65, 43-5. Rudner, L. M. (983). Indvdual Assessment Accuracy. Journal of Educatonal Measurement 20, 207-29. Tatsuoka, K. K. (996). Use of generalzed person-ft ndces, zetas for statstcal pattern classfcaton. Appled Measurement n Educaton 9, 65-75. The study was performed wth the support of The Czech Scence Foundaton (project nr. 406/09/P284) and s part of Research project of the Insttute of Psychology, Academy of Scence of the Czech Republc, dentfcaton code: AV0Z70250504.