Psychometric limitations of the Personality Assessment Inventory: A reply to Morey's (1995) rejoinder

Size: px

Start display at page:

Download "Psychometric limitations of the Personality Assessment Inventory: A reply to Morey's (1995) rejoinder"

Calvin Douglas
6 years ago
Views:

Bond University epublications@bond Humanities & Social Sciences papers Faculty of Humanities and Social Sciences 6-1-1996 Psychometric limitations of the Personality Assessment Inventory: A reply to

1 Bond University Humanities & Social Sciences papers Faculty of Humanities and Social Sciences Psychometric limitations of the Personality Assessment Inventory: A reply to Morey's (1995) rejoinder Gregory J. Boyle Bond University, Gregory_Boyle@bond.edu.au Follow this and additional works at: Part of the Psychology Commons Recommended Citation Gregory J. Boyle. (1996) "Psychometric limitations of the Personality Assessment Inventory: A reply to Morey's (1995) rejoinder" Journal of Psychopathology and Behavioral Assessment, 18 (2), : ISSN This Journal Article is brought to you by the Faculty of Humanities and Social Sciences at epublications@bond. It has been accepted for inclusion in Humanities & Social Sciences papers by an authorized administrator of epublications@bond. For more information, please contact Bond University's Repository Coordinator.

2 1 Psychometric Limitations of the Personality Assessment Inventory: A Reply to Morey's (1995) Rejoinder Gregory J. Boyle Department of Psychology Bond University Note. Morey referred to the PAI instrument as a "test." However, it is not really appropriate to refer to self-report personality scales as "tests," particularly when there are no right or wrong answers, as such. Only objective, performance measures (T-data) can rightly claim the title "test" (cf. Cattell & Kline, 1977).

3 2 Some psychometric problems with the Personality Assessment Inventory (PAl) were observed by Boyle and Lennon (1994). However, Morey (1995) asserted there were "several methodological and conceptual limits" and alternative explanations of the Boyle and Lennon data. Although Morey asserted that age and clinical status were confounded, Boyle and Lennon statistically partialled out variance due to age, using ANCOVA procedures. Morey's description of Boyle and Lennon's sample as "unusual" was strange-schizophrenic and alcoholic patients in a psychiatric hospital comprised the clinical groups-the PAl was designed specifically to assess psychopathology in such patients. Although Morey claimed that alpha coefficients were misinterpreted, Boyle and Lennon based their conclusions solely on the obtained coefficients. Morey's attempt to downplay the finding of suboptimal stability for several PAI scales also runs counter to the empirical results actually observed. Finally, Morey attempted to minimize the role of factor analysis in investigating construct validity, apparently to deflect attention from deficiencies in the factor analysis of a clinical sample reported in the PAl manual. Morey (1995) criticized Boyle and Lennon's (1994) finding that the median test-retest reliability coefficient for the Personality Assessment Inventory (PAI) 2 scales measured over a 28-day interval was only.73. Morey claimed that there was a restriction in the range of PAl scores, so that the less than optimal reliability was largely a statistical artifact. Yet perusal of actual scores obtained by a large subsample of 70 normal individuals from the general adult population at large (in whom personality characteristics would be expected to be relatively stable across an interval of only 4 weeks) revealed a wide range of PAl scores, despite Morey's suggestion to the con-trary (mean scale scores alone ranged from

4 to 22.16, and standard deviations ranged from 2.75 to 9.67). In any event, restriction of variance was not a particularly problematic issue in Boyle and Lennon's study, because the primary concern was whether or not PAl scale scores were ranked similarly or dissimilarly by the normal group on each measurement occasion. Had the PAl scales been more stable, a greater similarity of rankings would have been observed, than was the case. Although Morey regarded his argument as "self-explanatory," the alleged "restriction of range" of PAl scores within Boyle and Lennon's normal sample was not supported by the actual empirical evidence. Morey argued about minimum reliability "cutoff" points, despite the absence of any such claim by Boyle and Lennon. Ideally, if perfectly reliable, PAl scales would have exhibited stability coefficients of A median coefficient of.73 indicates that half the observed stability coefficients were actually lower than this value-a clearly undesirable finding for a personality trait inventory. Indeed, only 53% of the variance was common across both measurement occasions (i.e., at least 47% of the measurement variance was associated with dissimilar rankings of PAl scores). In calculating test-retest coefficients, Morey stated, admittedly, in the psychopathology field this is rather difficult, since ethical considerations preclude withholding treatment for purposes of generating reliability estimates." It was precisely because of this dilemma that Boyle and Lennon chose to assess stability in a normal sample-which was expected to produce upper-bound estimates of reliability. There is no a priori reason not to investigate the psychometric properties of a clinical personality instrument in more readily accessible, nonpsychiatric samples. For Morey to claim that, "this artifact must be recognized as a

5 4 complication" seems strange. This assertion was not based on empirical evidence, but was merely presumed. Morey set up "a straw man" and then proceeded to "knock it down." Scientific discourse should be based on concrete evidence, rather than on unsupported allegations. Morey stated that, "the PAl includes measures of anxiety, depressed affect, and suicidal ideation, features that might be expected to fluctuate widely over the course of one month." That might be true for certain clinical samples but would not be expected among psychologically stable normal individuals. Indeed, the proportion of variance common across both measurement occasions for the anxiety scale was only 38%, so that no less than 62% of variance was due to dissimilar rankings of scores, even among normal, psychologically stable individuals. Likewise, for the Paranoia (PAR), and Antisocial Features (ANT) scales, 58% and 60% of the variance was discrepant-i.e., more of the variance was unreliable than reliable! Using normal samples himself (despite his criticism of Boyle & Lennon), Morey (1991) reported a stability coefficient of.90 for the Antisocial Features (ANT) scale, thereby contradicting his assertion that normal samples are inappropriate for evaluating the psychometric properties of the PAl instrument and, by implication, contradicting his (1995) claim that there is undue restriction of range in scores for normal samples. Evidently, Morey's argument about "restriction of range" was refuted by his own empirical findings! In stating that the WAIS-R scales do not all exhibit retest coefficients of.80, Morey referred to intelligence as "a more stable construct" but failed to acknowledge that intellectual functioning fluctuates markedly in response to biological, neuropsychological, and situational factors (see Stankov, Boyle, & Cattell, 1995). In any event, sub-optimal

6 5 stability of WAIS-R scales would suggest that the instrument could stand considerable refinement, especially in light of the work showing that such models of intelligence are overly narrow (e.g., Boyle, 1995a; Cattell, 1987; Gardner, 1993; and Sternberg, 1994). The finding of less than optimal test-retest coefficients led Boyle and Lennon to point to the possibility that the PAl scales exhibit only relative stability, somewhat akin to dynamic traits (cf. Cattell & Child, 1975), rather than more enduring personality traits. Although Morey claimed that Boyle and Lennon had confused dynamic traits with states, Morey himself failed to make the necessary conceptual discriminations. As Boyle had pointed out in many previous papers (e.g., Boyle, 1979, 1983, 1985, 1988; Boyle & Cattell, 1984), stable trait measures should exhibit high immediate test-re- test (dependability) coefficients, as well as high longer-term retest (stability) coefficients, whereas transitory state measures should exhibit high depend- ability but considerably lower stability-if the measures are truly sensitive to situational fluctuations on different measurement occasions (cf. Fernan- dez, 1990; Fernandez, Nygren, & Thorn, 1991). Dynamic traits [including motivational dynamic traits such as those measured in the objective IPAT Motivation Analysis Test (see Cattell, 1992)] theoretically should exhibit test-retest coefficients intermediate in magnitude between those observed for stable traits, on the one hand, and transitory states, on the other (see Cattell, 1973, p. 354) for a detailed account of consistency coefficients and their implications for state, dynamic trait, and enduring trait scales]. Without considering the complex issues regarding test-retest reliability, as a function of the state-dynamic-trait continuum, Morey's denial of the observed instability of several of the PAl scales cannot be sustained logically.

7 6 With regard to item homogeneity estimates, Morey claimed that Boyle and Lennon interpreted high alpha coefficients as "bad." Despite being a rather dogmatic assertion, Morey contradicted himself when admitting that "internal consistency can be too high." In fact, as Boyle (1991) has discussed, both "internal consistency" and "item redundancy" are value judgments. Cronbach alpha coefficients merely indicate the level of item homogeneity of a scale. Morey further claimed that Boyle and Lennon's findings only raised the possibility of excessive item redundancy in the PAl scales. Yet the obtained median coefficient was no less than.83, indicating that for half of the PAl scales, alpha coefficients were exceptionally high, exceeding this figure (cf. Boyle, 1991). Morey asserted that, "There are simply too many influences upon the alpha coefficient to warrant the conclusion that high alphas invariably indicate problems." However, Boyle and Lennon only stated (p. 182) that the observed high alpha coefficients "suggested the possibility of rather narrow scales, with excessive item redundancy." (cf. Boyle, 1991). Morey subsequently asserted that "high internal consistency is obviously not a problem if a scale can be validated against external criteria." However, Morey has not considered the implications of the breadth of measurement of a construct. As Boyle (1991) pointed out, high alpha coefficients can be obtained if all items in a scale are merely paraphrases of each other but the construct is therefore being measured in a very limited, narrow fashion, and many of the items are redundant and could be dispensed with. Just because an instrument is popular (Morey referred to the Beck Depression Inventory and the Hamilton Rating Scale as good measures against which to assess the external or concurrent validity of the PAl), it does not follow that such instruments necessarily make good external

8 7 criterion measures, especially since both the BDI and the HRS also have some psychometric deficiencies (see Boyle, 1985). As for prevalence rates of alcoholic problems, it is unlikely that there are major differences between the United States and Australia. Thus, 18% would be the comparable figure for Queensland. Nevertheless, Morey was correct in highlighting the particular composition of the non-psychiatric sample, since the sample composition may have contributed to the apparent number of false positives on the Alcohol Problems (ALC) scale. In these circumstances, it seems likely that the ALC scale did not seriously overestimate the number of alcoholic cases. Morey questioned Boyle and Lennon's finding that several PAl scales did not discriminate between schizophrenic and alcoholic clinical samples, and asserted that there is no need for all PAl scales to exhibit discriminative validity. However, it would seem reasonable to expect that schizophrenic and alcoholic patients should differ significantly on the PAl scales labelled Schizophrenia (SCZ), Dominance (DOM), and Warmth (WRM). That these scales failed to differentiate between the two clinical groups raises questions concerning their validity. Alluding to the inadequate discriminative validity of the MMPI clinical scales (see the comprehensive critique by Helmes & Reddon, 1993) does not detract from the poor discriminative validity of several of the PAl scales (and of abnormal personality instruments, in general). Furthermore, Morey's suggestion about removing protocols with high Negative Impression (NIM) may not necessarily improve PAl scale discriminative validities. Morey's criticism of the possible confounding effect of age differences in the clinical and non-psychiatric groups ignored the very careful consideration

9 8 given the statistical handling of this variable. According to Boyle and Lennon (p. 178), "MANCOVAs were carried out on the data with gender and age as covariates, in order to correct statistically for distorting effects due to those variables. The main effect across groups was still significant... after the effects of age and gender were partialled out." Partialling out the variance of demographic variables by treating them statistically as covariates clearly refutes Morey's claim that demography and clinical status were confounded. Morey also asserted that differing numbers of subjects in the cells of the design was problematic for the discriminative validity of PAl scales. However, Boyle and Lennon (p. 178) specifically pointed out that "in order to counteract the effects of heterogeneity of variance due to unequal group sizes, a more stringent criterion of statistical significance was used... In addition... Bonferroni corrections were applied to minimize finding significant between-group differences due to chance alone." In any event, heterogeneity of variance due to unequal group sizes should have increased the likelihood of finding significant differences on the PAl scales. Since several of the scales still failed to discriminate between the clinical groups, this further suggests inadequate discriminative validities. Morey attempted to downplay the finding that his factor analytic solution for the clinical subjects was poorly replicated when identical "Little Jiffy" procedures were employed (cf. Comrey & Lee, 1992). He suggested that a minor typographical error in a single correlation coefficient was responsible for the discrepant results. Thus, Morey suggested that, "I would encourage them [Boyle & Lennon] to rerun their analyses correcting this value and I suspect they will replicate the solution provided in the manual." Accordingly, the suggested minor

10 9 correction was made and the factor analysis rerun. This minor change made no significant difference to the outcome (factor loadings, in general, differed only at the third or fourth decimal place from the previous analysis by Boyle and Lennon). Again, SPSS gave a warning message that the correlation matrix was "illconditioned," that it was "not positive definite," and again, the Bartlett test of sphericity could not be calculated, indicating that the matrix for the clinical sample reported in Table 10.1 of the PAl manual fails to satisfy multivariate normality assumptions required for a valid factor analysis. Morey criticized Boyle and Lennon's factor analytic methodology, but his own procedures and results were to some extent deficient. It is irrelevant how many other studies Morey cites to support his higher-order factor solutions (e.g., Deisinger, 1995). Philosophical debate about theory and factor analysis and seeking support from Costa and McCrae (1985) does not overcome the problem. Indeed, the factor analytic work of Costa and McCrae itself leaves much to be desired from a methodological and psychometric standpoint (see detailed critiques by Block, 1995; and Boyle, Stankov, & Cattell, 1995, pp ). Morey stated that "using a principal components-varimax factor technique does not imply that I believe psychopathological constructs are orthogonal " Nevertheless, construct validation of an instrument such as the PAl is a detailed, extensive process, part of which includes consideration of its factor analytic validity [see Grossarth-Maticek, Eysenck, & Boyle (1995) for a technical discussion of construct validity in relation to personality instruments]. Morey's higher-order factoring procedure unnecessarily precluded the possibility of checking on the PAl factor validity.

11 10 Factor analytic procedures have been well discussed in several authorative publications (e.g., Cattell, 1978; Comrey & Lee, 1992; Gorsuch, 1983; McDonald, 1985). Consideration of what constitutes appropriate factor analytic methodology is critically important if valid factor solutions are to be obtained (cf. Boyle, 1988, 1993; Boyle & Stanley, 1986). For example, Morey recommended factor analyses at the item-subscale level, yet it is well documented that item correlations are notoriously unreliable. That is why Comrey (e.g., Hahn & Comrey, 1994) has advocated the use of factored homogeneous item dimensions (FHIDs), Cattell (1978) has recommended the use of item parcels, and Marsh (see Boyle, 1994) has employed item-dyads as his smallest correlational units. As for Morey's call for non-linear factor analytic procedures, the comments of Gorsuch (pp ) and McDonald (1981) seem germane. Finally, Morey concluded that "an unreflective application of criteria such as coefficient alpha is bad or simple structure is good can be quite misleading depending on the issue in question." Yet Boyle and Lennon never made such dogmatic claims. It appears that Morey has misread the evidence in attempting to deflect attention away from the empirically observed shortcomings highlighted in Boyle and Lennon's study (cf. Boyle, 1995; and Boyle, Ward, & Steindl, 1994). Discussion grounded more firmly in the empirical evidence, rather than on supposition or semantics about the definition of terms such as moderator variables, would do greater justice to the scientific issues. References Block, J. (1995). A contrarian view of the five-factor approach to personality description. Psychological Bulletin, 117,

12 11 Boyle, G. J. (1979). Delimitation of state-trait curiosity in relation to state anxiety and learning task performance. Australian Journal of Education, 23, Boyle, G. J. (1983). Effects on academic learning of manipulating emotional states and motivational dynamics. British Journal of Educational Psychology, 53, Boyle, G. J. (1985). Self-report measures of depression: Some psychometric considerations. British Journal of Clinical Psychology, 24, Boyle, G. J. (1988). Exploratory factor analytic principles in motivation research. In J. R. Nesselroade & R. B. Cattell (Eds.), Handbook of multivariate experimental psychology (pp ). New York: Plenum. Boyle, G. J. (1991). Does item homogeneity indicate internal consistency or item redundancy in psychometric scales? Personality and Individual Differences, 12, Boyle, G. J. (1993). Special review: Evaluation of the exploratory factor analysis programs provided in SPSSX and SPSS/PC+. Multivariate Experimental Clinical Research, 10, Boyle, G. J. (1994). Self-Description Questionnaire II. In D. J. Keyser & R. C. Sweetland (Eds.), Test Critiques (Vol. 10, pp ). Kansas City, MO: Test Corporation of America. Boyle, G. J. (1995a). Measurement of intelligence and personality within the Cattellian psychometric model. Multivariate Experimental Clinical Research, 11, Boyle, G. J. (1995b). Review of the Personality Assessment Inventory. In J. C. Conoley & J. lmpara (Eds.), Twelfth mental measurements yearbook. Lincoln, NE: Buros Institute of Mental Measurements.

13 12 Boyle, G. J., & Cattell, R. B. (1984). Proof of situational sensitivity of mood states and dynamic traits-ergs and sentiments-to disturbing stimuli. Personality and Individual Differences, 5, Boyle, G. J., & Lennon, T. J. (1994). Examination of the reliability and validity of the Personality Assessment Inventory. Journal of Psychopathology and Behavioral Assessment, 16, Boyle, G. J., & Stanley, G. V. (1986). Application of factor analysis in psychological research: Improvement of simple structure by computer assisted graphic oblique transformation: A brief note. Multivariate Experimental Clinical Research, 8, Boyle, G. J., Ward, J., & Lennon, T. J. (1994). Personality assessment inventory: A confirmatory factor analysis. Perceptual and Motor Skills, 79, Boyle, G. J., Stankov, L., & Cattell, R. B. (1995). Measurement and statistical models in the study of personality and intelligence. In D. H. Saklofske & M. Zeidner (Eds.), International Handbook of Personality and Intelligence. New York: Plenum. Cattell, R. B. (1973). Personality and mood by questionnaire. San Francisco: Jossey-Bass. Cattell, R. B. (1978). The scientific use of factor analysis in behavioral and life sciences. New York: Plenum. Cattell, R. B. (1987). Intelligence: Its stmcture, growth and action. Amsterdam: North Holland. Cattell, R. B. (1992). Human motivation objectively, experimentally analyzed. British Journal of Medical Psychology, 65,

14 13 Cattell, R. B., & Child, D. (1975). Motivation and dynamic stmcture. London: Academic. Cattell, R. B., & Kline, P. (1977). The scientific analysis of personality and motivation. New York: Academic. Comrey, A L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum. Costa, P. T., & McCrae, R. R. (1985). The NEO Personality Inventory manual. Odessa, FL: Psychological Assessment Resources. Deisinger, J. A (1995). Exploring the factor structure of the Personality Assessment Inventory. Assessment, 2, Fernandez, E. (1990). Artifact in pain ratings, its implications for test-retest reliability, and correction by a new scaling procedure. Journal of Psychopathology and Behavioral Assessment, 12, Fernandez, E., Nygren, T. E., & Thorn, B. E. (1991). An open-transformed scale for correcting ceiling effects and enhancing retest reliability: The example of pain. Perception and Psychophysics, 49, Gardner, H. 0. (1993). Intelligence and intelligences: Universal principles and individual differences. Archives de Psychologie, 61, Gorsuch, R. L. (1983). Factor analysis (rev. 2nd ed.). Hillsdale, NJ: Erlbaum. Grossarth-Maticek, R., Eysenck, H. J., & Boyle, G. J. (1995). Method of test administration as a factor in test validity: The use of a personality questionnaire in the prediction of cancer and coronary heart disease. Behaviour Research and Therapy, 33, Hahn, R., & Comrey, A L. (1994). Factor analysis of the NEO-PI and the Comrey Personality Scales. Psychological Reports, 75, Helmes, E., & Reddon, J. R. (1993). A perspective on developments in assessing

15 14 psychopathology: A critical review of the MMPI and MMPI- Psychological Bulletin, 113, McDonald, R. P. (1981). The dimensionality of tests and items. British Journal of Mathematical and Statistical Psychology, 34, McDonald, R. P. (1985). Factor analysis and related methods. Hillsdale, NJ: Erlbaum. Morey, L. C. (1991). The Personality Assessment Inventory Professional Manual. Odessa, FL: Psychological Assessment Resources. Morey, L. C. (1995). Critical issues in construct validation: Comment on Boyle and Lennon (1994). Journal of Psychopathology and Behavioral Assessment, 17, Stankov, L., Boyle, G. J., & Cattell, R. B. (1995}. Models and paradigms in personality and intelligence research. In D. H. Saklofske & M. Zeidner (Eds.), International handbook of personality and intelligence. New York: Plenum. Sternberg, R. J. (1994). Experimental approaches to human intelligence. European Journal of Psychological Assessment, 10,

Content Similarities and Differences in Cattell s Sixteen Personality Factor Questionnaire, Eight State Questionnaire, and Motivation Analysis Test

Bond University From the SelectedWorks of Gregory J. Boyle 1987 Content Similarities and Differences in Cattell s Sixteen Personality Factor Questionnaire, Eight State Questionnaire, and Motivation Analysis