EXAMINING THE RELIABILITY AND VALIDITY OF THE FIVE-FACTOR MODEL SCORE SHEET. Lauren R. Pryor. (Under the direction of Joshua D.

Transcription

1 EXAMINING THE RELIABILITY AND VALIDITY OF THE FIVE-FACTOR MODEL SCORE SHEET by Lauren R. Pryor (Under the direction of Joshua D. Miller) ABSTRACT The current study extended research in the assessment of the Five-Factor Model (FFM) by examining the reliability and validity of the Five-Factor Model Score Sheet (FFMSS; Widiger & Spitzer, 2002), a single-item clinician rating form of the 30 facets of the FFM. Participants included 130 outpatients who were rated by clinicians using the FFMSS. Results revealed that the factor structure of the FFMSS is consistent with previous research examining alternative measures of the FFM. Furthermore, patients were rated with reasonable reliability following minimal training and FFMSS ratings provided personality data convergent with other self-report measures of personality and clinician ratings of personality disorders from the Fourth Edition of the Diagnostic and Statistical Manual for Mental Disorders (American Psychological Association, 1994). The ratings also demonstrated clinical utility in predicting concurrent and prospective impairment across a number of domains. Results provide initial support for the use of the FFMSS in clinical settings. INDEX WORDS: Five-Factor Model, Personality assessment, Clinician personality ratings

2 EXAMINING THE RELIABILITY AND VALIDITY OF THE FIVE-FACTOR MODEL SCORE SHEET by LAUREN R. PRYOR B.S., Furman University, 2006 A Thesis Submitted to the Graduate Faculty of The University of Georgia in Partial Fulfillment of the Requirements for the Degree MASTER OF SCIENCE ATHENS, GEORGIA 2009

3 2009 Lauren R. Pryor All Rights Reserved

4 EXAMINING THE RELIABILITY AND VALIDITY OF THE FIVE-FACTOR MODEL SCORE SHEET by LAUREN R. PRYOR Major Professor: Joshua D. Miller Committee: Amos Zeichner Karen Calhoun Electronic Version Approved: Maureen Grasso Dean of the Graduate School The University of Georgia May, 2009

5 iv CHAPTER TABLE OF CONTENTS Page 1 INTRODUCTION METHOD RESULTS DISCUSSION...37 REFERENCES...55 FOOTNOTES...67 TABLES...68 APPENDIX...81

6 1 CHAPTER 1 INTRODUCTION One of the most widely researched and accepted models of general personality is the fivefactor model (FFM; Digman, 1990). Although the FFM is an established dimensional model of general personality and has proven quite helpful in integrating findings from various personality models, conceptualizing personality disorder (PD), and bridging the divide between normal and abnormal personality, there is less research addressing the clinical utility of this model. However, the limited research that is available for the FFM, as it pertains to clinical utility, suggests that it has promise as it has been demonstrated that the FFM predicts treatment satisfaction and compliance (Miller, Pilkonis, & Mulvey, 2005), functional impairment (e.g., Miller, Pilkonis, & Clifton, 2005), and clinicians view it as a more useful model than the current conceptualization of personality pathology (Samuel & Widiger, 2006) found in the Diagnostic and Statistical Manual for Mental Disorders (DSM; e.g., American Psychological Association, 1994). Despite its demonstrated utility, clinicians have been slow to use measures of the FFM. There may be several reasons for this. First, the most popular measure of the FFM, the Revised NEO Personality Inventory (NEO PI-R; Costa & McCrae, 1992a), is a 240-item measure that takes approximately minutes to complete. The NEO PI-R can be completed using self or informant ratings but the self-report methodology has been the most commonly used form. Selfreport personality data typically demonstrate correlations with informant ratings of personality at approximately.30 to.55 (Ready & Clark, 2002) in general, and approximately.40 to.50 for the FFM specifically (McCrae et al., 2004). While both self and informant ratings of personality

7 2 demonstrate predictive validity (Klein, 2003), informant reports often provide stronger predictive or incremental validity over self-report data. For instance, Miller, Pilkonis, and Clifton (2005) compared self and informant reports on the NEO PI (Costa & McCrae, 1985) and expert ratings of three domains of functional impairment (i.e., work, social, romantic); informant reports on the FFM were stronger statistical predictors than self-reports for all three. It is clear that personality data related to the FFM show promise with regard to patient conceptualization, prediction of treatment outcomes (treatment utilization, satisfaction), and level of impairment. However, the current assessment methodology may be regarded as too unwieldy to warrant frequent use. As an attempt to rectify this problem, Widiger and Spitzer (2002) developed the FFM Score Sheet (FFMSS), which uses a single rating for each of the thirty personality facets captured in the NEO PI-R. This methodology has great potential as a clinical tool because of the speed and ease with which it can be completed. However, because of its recent development, there are no existing data on the reliability and validity of this measure. The aims of the current study are to evaluate the factor structure and reliability of clinician ratings on the FFMSS. In addition, the validity of the FFMSS is investigated by examining how the FFMSS relates to pathological personality traits, personality disorder (PD), psychological distress (i.e., anxiety and depression), and functional impairment. Predictive and incremental validity of the FFMSS will also be examined in relation to impairment. Before considering the details of the current study, however, it is necessary to provide relevant background information related to the FFM and its link to Axis I and Axis II psychopathology. Furthermore, assessment of the FFM and its current clinical applications will be described.

8 3 FFM Background The FFM/Big Five was originally derived from Allport s (1937) lexical hypothesis, which posits that the most critical personality traits are encoded in language as single words (i.e., adjectives). In order to identify these traits, several researchers systematically studied the English language and developed lists of descriptive personality terms (see John & Srivastava, 1999 for a review). Using factor analytic techniques to reduce these comprehensive adjective lists, researchers have consistently extracted five factors, referred to as the Big Five (Goldberg, 1981). These factors include 1) extraversion or surgency, 2) agreeableness, 3) conscientiousness, 4) emotional stability versus neuroticism, and 5) intellect or openness. Although the Big Five provides a descriptive taxonomy of personality traits, it was not intended to be a theory of general personality (John & Srivastava, 1999). In response to this, McCrae and Costa s (1996, as cited in John & Srivastava) five factor theory was developed in an attempt to explain general personality through the lens of the Big Five. The theory works on the premise that the five personality dimensions are genetically based and are basic tendencies that, when interacting with the environment, influence behavior, attitudes, and goals. Support for the biological bases of FFM personality dimensions is derived from research elucidating the cross-cultural universality of these traits (McCrae & Costa, 1997), as well as the heritability of the FFM (Jang, Livesley, and Vernon, 1996; Riemann, Angleitner, & Strelau, 1997). Jang et al., for example, estimated that the heritability of the FFM dimensions ranges from 41% (Neuroticism) to 61% (Openness). In order to help define the poles of each domain, Costa and Widiger (2002) describe characteristics of individuals who are high and low in each domain. Specifically, individuals who are high in Extraversion tend to be sociable, fun-loving and assertive, whereas those low in

9 4 Extraversion are often more reserved and quiet. Agreeable individuals are characterized as trustworthy, helpful, and empathic, whereas those low in Agreeableness tend to be cynical, manipulative, and uncooperative. Individuals who are high in Conscientiousness are punctual, able to persevere in the face of boredom and frustration, and demonstrate good impulse control, whereas those low in Conscientiousness tend to be unreliable, careless, and are predisposed to acting without adequate forethought. Individuals high in Neuroticism are typically vulnerable to a variety of negative emotional states such as depression, anxiety, and anger, whereas those low in Neuroticism are more emotionally stable and tend to be less likely to experience depression and anxiety. Lastly, individuals high in Openness are characterized as curious and imaginative, whereas those low in Openness tend to be more conventional in their beliefs and attitudes (p.6). In addition to these five broad domains, Costa and McCrae (1995) delineated six underlying facets subsumed by each of the FFM domains. For example, the facets of Agreeableness include trust, straightforwardness, altruism, compliance, modesty, and tendermindedness. These facets correspond with trait adjectives identified in lexical research (John & Srivastava, 1999). FFM and Psychopathology The FFM has evidenced utility beyond simply describing individual differences in general personality traits. More specifically, FFM inventories have been used to gain understanding about the basic personality traits underlying various forms of psychopathology. McCrae and colleagues (Costa & McCrae, 1992b; McCrae, 1991) examined FFM traits, as measured by the NEO PI, with clinical scales from several dimensional measures of psychopathology, including the Millon Clinical Mutiaxial Inventory (MCMI; Millon, 1983), Minnesota Multiphasic Personality Inventory (MMPI; Hathaway & McKinley, 1983), Basic

10 5 Personality Inventory (Jackson, 1989), and Personality Assessment Inventory (PAI; Morey, 1991). Across instruments, results consistently yielded a negative association between Extraversion and depression related scales, and positive associations between Neuroticism and both anxiety and depression. These findings are consistent with Clark and Watson s (1991) tripartite model, in which anxiety and depression are both characterized by negative affect (i.e. Neuroticism), whereas low positive affect (i.e. Extraversion) is uniquely related to depression. FFM traits have also been examined in relation to DSM Axis I disorders. For example, Trull and Sher (1994) examined the relations between FFM domains and substance abuse, anxiety disorders, and major depression, and found that the FFM demonstrated incremental validity beyond a measure of Axis I psychopathology, the Brief Symptom Inventory (BSI; (Derogatis & Spencer, 1982). In general, Trull and Sher found that FFM Neuroticism, Extraversion, and Conscientiousness were most relevant to Axis I pathology. Specifically, their results affirmed previous literature suggesting an association between high Neuroticism, low Extraversion and Axis I depressive disorders (Widiger & Trull, 1992), as well as low Conscientiousness and anxiety and depression (Costa & McCrae, 1992b; McCrae, 1991). A more recent study by Durrett and Trull (2005) examined the amount of variance in lifetime Axis I disorders accounted for by the FFM domains compared to an alternative seven factor model of personality, the Big Seven (Almagor, Tellegen & Waller, 1995). This model includes the five dimensions encompassed in the FFM, as well as two additional dimensions (Positive Valence, Negative Valence) that assess more evaluative personality descriptors (e.g., superior, ordinary, awful, decent ). Durrett and Trull found that the FFM accounted for a significant amount of the variance in mood, anxiety, substance use, and eating disorders (29%,

11 6 20%, 17%, and 24% respectively). These findings lend support for the use of the FFM in predicting Axis I disorders. FFM and Personality Disorder Greater attention, from both theoretical and empirical perspectives, has been devoted to examining the link between the FFM and Axis II psychopathology (i.e., personality disorder). Many researchers support the notion that PD symptoms are simply extreme, maladaptive variants of general personality traits (Widiger, 2000; Widiger & Trull, 1992; Wiggins & Pincus, 1989). In line with this hypothesis, Widiger, Trull, Clarkin, Sanderson, and Costa (1994; 2002) translated each DSM-IV PD into the language of the FFM, using the specific facets. These translations are based on the DSM-IV diagnostic criteria for each PD. For example, they hypothesized that Antisocial PD is characterized by high angry hostility, a facet of Neuroticism, in addition to high excitement seeking, a facet of Extraversion. In addition, Antisocial PD includes low scores on the Agreeableness facets of straightforwardness, altruism, compliance, and tender mindedness, as well as the Conscientiousness facets of dutifulness, self-discipline, and deliberation. More recently, Lynam and Widiger (2001) developed expert-ratings of each of the DSM-IV PDs by asking experts to rate the prototypical individual with a given DSM-IV PD on all 30 facets of the NEO PI-R. This approach appears to result in a fuller, more comprehensive description of individuals with each PD as it is not limited to the description provided by the DSM-IV PD criteria. Although these are hypothesized translations of DSM-IV PDs, there has been substantial empirical support that PDs can, in fact, be conceptualized using the FFM. A meta-analysis aimed at exploring this theoretical perspective revealed expected patterns of relations between each of the DSM-IV PDs and FFM domains (Saulsman & Page, 2004). For example, disorders

12 7 characterized in the DSM-IV by emotional distress consistently exhibited positive associations with Neuroticism (e.g., Borderline PD), whereas disorders characterized by interpersonal problems were negatively associated with Agreeableness (e.g., Antisocial PD). This metaanalysis did, however, reveal a drawback of using the FFM to understand PDs, in that the FFM is better for conceptualizing certain disorders compared to others. Specifically, the magnitude of effect sizes was significantly lower for Obsessive-Compulsive PD (OCPD) and Schizoid PD (SPD). Thus, the FFM may not adequately represent all PDs. Several studies have observed weak relations between FFM domains and certain Axis II symptomatology. Specifically, studies have failed to reveal an expected relation between Conscientiousness and OCPD (e.g. Costa & McCrae, 1990; Trull, 1992), Agreeableness and Dependent PD (DPD), and Openness and Schizotypal PD (STPD). Haigler and Widiger (2001) hypothesized that this failure of certain FFM domains to predict the symptomatology of various PDs may be a result of the relative emphasis on adaptive rather than maladaptive variants of these domains of personality functioning within the NEO PI-R (p.339; Costa & McCrae, 1992a). In other words, the NEO PI-R (Costa & McCrae, 1992a), a self-report assessment of the FFM, may lack items tapping maladaptive extremes of high Conscientiousness, Agreeableness, and Openness. In order to test this hypothesis, Haigler and Widiger reversed wording of NEO PI- R items in an effort to make them appear less adaptive. For example, 90% of items were considered to be more adaptive when assessing high Conscientiousness rather than low Conscientiousness. Therefore, they inserted words in these items to make them suggestive of maladaptivity (e.g. original item: I keep my belongings neat and clean. New item: I keep my belongings excessively neat and clean. ). Results using the new FFM assessment measure (the EXP-NEOPIR) supported the researchers hypothesis, in that predicted relations between

13 8 Conscientiousness, Agreeableness, Openness and Obsessive-Compulsive PD, Dependent PD, and Schizotypal PD, respectively, were observed when items reflective of maladaptivity at the high end of these domains were included. However, reversing the items changed the content of some items such that they did not exhibit expected correlations with their respective FFM domain. This led to changes in the overall correlations between domains of the FFM. For example, EXP-NEOPIR Agreeableness correlated as highly with NEO PI-R Neuroticism as it did with NEO PI-R Agreeableness. Also, correlations of.43,.53, and.67 between the NEO PI-R and EXP-NEOPIR scales of Conscientiousness, Agreeableness, and Openness respectively were lower than desired. So, it is not clear if the inclusion of greater maladaptivity at the positive poles of the FFM domains would result in a model with the same structure. In addition to investigating the link between FFM domains and psychopathology, various studies have examined the lower order FFM facets in relation to Axis I and Axis II disorders. For example, Costa and McCrae (1992b) suggest that high self-consciousness, a facet of Neuroticism, should direct clinicians toward consideration of an Axis I social phobia diagnosis. With respect to PDs, Dyce and O Connor (1998) originally noted that FFM facets did not predict substantial variance beyond the FFM domains in a normal sample. However, this finding was challenged by results from a clinical sample indicating that FFM facets significantly predicted 12 of 13 DSM-IV PDs (Reynolds & Clark, 2001). Moreover, Reynolds and Clark compared the predictive validity of FFM domains, FFM facets, and the Schedule for Nonadaptive and Adaptive Functioning (SNAP; Clark, 1993). The SNAP is a dimensional model of pathological personality, which includes 12 lower order, primary trait scales and 3 higher order temperament dimensions and was derived from analyses of DSM PD criteria, non-dsm conceptualizations of PDs (e.g. psychopathy), and criteria from Axis I disorders that resemble PDs (e.g. dysthymia).

14 9 Results revealed that the FFM domains showed the least amount of predictive validity when compared to the FFM facets and the SNAP. In further support of the usefulness of FFM facets, Axelrod, Widiger, Trull, and Corbitt (1997) found that Borderline PD, Antisocial PD, Narcissistic PD, and Paranoid PD all correlated positively with the Agreeableness domain; however, the Agreeableness lower-order facets significantly differentiated between these disorders. For example, Paranoid PD was characterized primarily by low scores on trust, whereas Narcissistic PD was characterized by low scores on modesty, tender-mindedness, and altruism. Thus, the variance in PD symptomatology is better accounted for when considering the lowerorder facets of the FFM (Bagby, Costa, Widiger, Ryder, & Marshall, 2005). Furthermore, assessment of DSM-IV PDs using FFM facets, as measured by the NEO PI-R, has been shown to be as effective as more explicit assessments of DSM-IV PD symptomatology (Miller, Reynolds, Pilkonis, 2004), such as the Structured Interview for DSM-IV Personality Disorders (SIDP-IV; Pfohl, Blum, & Zimmerman, 1997). Clinical Utility of the FFM The aforementioned discussion suggests that the utility of the FFM is not limited to explaining basic traits underlying psychopathology but is, in fact, useful in clinical practice. MacKenzie (2002) outlines several benefits of approaching therapy from a FFM perspective, including assisting the therapist in their selection of therapeutic strategies, alerting the therapist to potential barriers in the therapeutic alliance, as well as facilitating empathy by helping the therapist to understand the patient s internal experience. From an empirical standpoint, Samuel and Widiger (2006) sought to determine whether individuals actively providing therapeutic services view the FFM as beneficial. They had clinicians conceptualize case vignettes using DSM-IV diagnostic categories, as well as FFM descriptions, and then asked them to rate the

15 10 clinical utility of each model. Results indicated that clinicians could reliably conceptualize the case vignettes regardless of the model used. More importantly, clinicians rated the FFM as more clinically useful in communicating information to the client, capturing the comprehensiveness of the client s difficulties, and treatment planning. The use of the FFM in treatment planning may be particularly beneficial, in that changes in underlying FFM traits have been shown to result in later changes in PD symptoms while the reverse was not the case (i.e., changes in PD symptoms did not result in changes in FFM traits; Warner et al., 2004). The FFM may provide additional benefits in clinical settings beyond those identified by clinicians in the therapeutic process. For instance, the FFM has evidenced incremental validity beyond measures of functioning and PD symptoms in predicting treatment utilization and satisfaction (Miller, Pilkonis, & Mulvey, 2006). Specific to utilization, individuals high on Extraversion and low on Agreeableness tend to use fewer clinical services and are less compliant, whereas individuals high on Openness and Conscientiousness tend to adhere more to therapy. In regard to treatment satisfaction, results indicated that patients high on Openness are generally more satisfied with medication and therapy. These findings support the effectiveness of the FFM in predicting important treatment variables and the importance of considering these personality constructs when conducting treatment planning. From a diagnostic standpoint, Widiger, Costa, and McCrae (2002) have delineated a fourstep process that uses FFM data to make PD diagnoses. This process involves (1) describing individuals using FFM domains and facets, (2) identifying overt difficulties associated with each trait, (3) determining the clinical significance of these difficulties, and (4) deciding whether the FFM profile matches the profile of specific DSM-IV PDs. Widiger et al. describe this process as particularly useful in that the FFM description also includes beneficial, adaptive traits that will

16 11 likely facilitate decisions concerning treatment (p. 436). Thus, this process provides a useful framework for utilizing the FFM in clinical diagnosis. To assist in diagnosis, researchers (Lynam & Widiger, 2001; Miller, Lynam, Widiger, and Leukefeld, 2001) have developed a technique enabling the assessment of PDs from selfreport measures of the FFM. As previously mentioned, FFM prototypes can be generated for each PD using expert ratings of prototypical individuals using the 30 FFM facets (Lynam & Widiger). These resulting prototypes can then be compared to individuals self-reported profiles on measures of the FFM in order to assess for PDs. In examining the validity of the FFM PD prototypes, Miller, Reynolds, and Pilkonis (2004) found empirical support for the use of the prototype matching technique with self-reported NEO PI and NEO PI-R data in two clinical samples. Research (Miller, Bagby and Pilkonis, 2005) has also suggested that the FFM PD prototype scores can be generated successfully using the Structured Interview for the Five-Factor Model of Personality (SIFFM; Trull & Widiger, 1997). The implications of the latter finding are substantial, in that structured interviews conducted by trained clinicians can assist in eliminating some of the problems involved in using self-report measures. Length, however, is a drawback of using the SIFFM (120 items) in assessing FFM traits. This suggests the need for a shorter, otherrated assessment of the FFM that could be used to generate prototype scores. Although the prototype matching technique has been validated, it is rather cumbersome and requires transformation of NEO data and computation of a double-entry Q correlation between an individual s NEO data and a prototypic PD profile. Recently, an alternative and less unwieldy approach to assessing DSM PDs using the FFM has been developed. The count technique requires summation of facets considered prototypically low or high for each PD (based on expert FFM ratings of prototypical DSM-IV PDs; Lynam & Widiger s, 2001). Facets

17 12 considered prototypically low are reverse scored so that that higher scores represent greater maladaptivity for that specific DSM-IV PD. For example, the FFM PD count for Borderline PD would be generated by adding the ratings from the following facets: anxiousness (N), angry hostility (N), depressiveness (N), impulsiveness (N), vulnerability (N), feelings (O), actions (O), compliance (A; reverse scored) and deliberation (C; reverse scored). Normative data in samples from three countries (United States, France, and Belgium-Netherlands) has recently been collected enabling interpretation of FFM PD count scores generated from NEO PI-R data (Miller, Lynam, Rolland, Fruyt, Reynolds, Pham-Scottez, A et al., 2008), thus providing further support for the clinical utility of this approach. FFM Assessment Given the mounting evidence supporting the usefulness of assessing FFM traits in clinical settings, it is necessary to consider the various measures that have been developed to assess these traits. These include the Goldberg Big Five Markers (Goldberg, 1992), the Interpersonal Adjectives Scale Big Five (Trapnell & Wiggins, 1990), the Hogan Personality Inventory (Hogan, 1986), as well as the Structured Interview for the Five-Factor Model of Personality (SIFFM; Trull & Widiger, 1997). As evidenced from the literature reviewed thus far, the majority of empirical work investigating the FFM has used the NEO PI-R (Costa & McCrae, 1992b). Several advantages of the NEO PI-R compared to alternative measures of the FFM have been suggested. For example, in contrast to the NEO PI-R, some measures do not include items to assess facets for each FFM domain (e.g. Goldberg s Big Five). Widiger and Trull (1997) further suggest that the use of statements in NEO PI-R items, rather than adjectives (such as in the IAS-B5), is beneficial because it increases the precision of the measure in evaluating FFM domains and facets. Additionally, Costa and McCrae (1992b) delineate clinical uses of the NEO

18 13 PI-R such as using these scores to better understand the client, assist in diagnosis, select therapeutic strategies, and anticipate the course of therapy. Despite these benefits, there are several disadvantages to using the NEO PI-R in clinical settings. Consistent with self-report measures in general, there are concerns regarding socially desirable responding and malingering. This may be particularly problematic, in that there are no validity scales included in the NEO PI- R. Miller, Bagby, and Pilkonis (2005) also point out that self-report measures do not assess for impairment and describe this as a drawback given that the DSM requires impairment in order for a PD diagnosis. It has become increasingly clear that an explicit assessment of impairment is necessary when considering personality disorder (Clark, 2007). As noted earlier, clinicians must also consider administration time as the NEO PI-R consists of 240 items. Therefore, a more abbreviated measure of the FFM could be useful when there are time constraints for assessment. In response to these concerns, several abbreviated measures of the FFM have been developed. These include the Big Five Inventory (BFI; John & Srivastava, 1999), the NEO-Five Factor Inventory (NEO-FFI; Costa & McCrae, 1992a), Saucier s (1994) Mini Markers, and the 10-Item Personality Inventory (TIPI; Gosling, Renfrow, & Swann, 2003). One potential limitation of these measures, however, is that there are no items or scales assessing the facets of the FFM (Mullins-Sweatt, Jamerson, Samuel, Olson, & Widiger, 2006). This deficiency is problematic in that, as previously mentioned, FFM facets have successfully differentiated between both Axis I and Axis II disorders and are better able to capture the different PDs (see Lynam & Widiger, 2001). A recently developed measure, the FFM rating form (FFMRF; Widiger, 2004), has addressed concerns regarding the need to include lower-order facets in other-rated assessment of the FFM, as well as the need for brevity. The FFMRF is a single-item assessment of the 30 facets

19 14 of the FFM, which includes an identifying term for each facet (e.g. self-consciousness), along with two to four adjectives that describe both poles of each facet (e.g. timid, embarrassed vs. self-assured, glib, shameless). The FFMRF has been reliably used by clinicians, as well as researchers, in describing prototypical cases of PD in terms of the FFM (Lynam & Widiger, 2001; Samuel & Widiger, 2004). A recent study by Mullins-Sweatt, Jamerson, Samuel, Olson, and Widiger (2006) examined the validity of the FFMRF in self-report college samples. They found good convergence with alternative measures of the FFM, including the NEO PI-R, IASR- B5, and other abbreviated measures of the FFM. In addition, self-descriptions of FFMRF domains and facets correlated as expected with maladaptive personality traits, as well as PDs. For example, those rating themselves high on FFM facets theoretically linked to Borderline PD (e.g. impulsivity, angry hostility) also endorsed more Borderline PD traits, as measured by the SNAP (Clark, 1993), OMNI Personality Inventory (Loranger, 2001), and Personality Disorder Questionnaire-4 (Hyler, 1994). Current Study The current study evaluates the reliability and validity of the Five-Factor Model Score Sheet (FFMSS; Widiger & Spitzer, 2002), an instrument that uses single-items to assess each of the 30 facets of the FFM. This score sheet is similar to the FFMRF, although the FFMSS utilizes a scale ranging from 1 (problematic, very low on the trait) to 7 (problematic, very high on the trait), whereas the FFMRF utilizes a 5-point scale ranging from extremely low to extremely high. This difference is important to consider because the FFMSS scale includes indications of maladaptivity at both poles of each FFM facet which allows individuals to be assessed as being problematic high or low on any given trait. Based on Haigler and Widiger s (2001) finding that reversing NEO PI-R items to reflect more maladaptivity at the high end of

20 15 certain FFM domains altered the nature of the domains, we examine the correlations between the FFMSS domains. Based on previous research examining discriminant correlations between NEO PI-R domains (Costa & McCrae, 1992a), it is expected that Neuroticism will evince small but significant negative correlations with Extraversion and Openness and will be moderately negatively related to Conscientiousness. Extraversion should also be moderately positively related to Openness and should exhibit a small positive correlation with Conscientiousness. Lastly, a small but significant positive correlation between Agreeableness and Conscientiousness is expected. Exploratory factory analysis will also be conducted because this specific measure of the FFM has not been previously used or evaluated in research. Unique to the current study is the use of clinician ratings of actual patients on FFM facets, rather than ratings of prototypical PDs (Samuel & Widiger, 2004) or PD vignettes (Sprock, 2002). This study examines the reliability and validity of these clinician ratings, as opposed to self-report descriptions (as in Mullins-Sweatt et al., 2006). Specifically, we evaluate whether clinicians can reliably rate FFM traits using a single-item assessment of the 30 FFM facets, the FFMSS, in a clinical setting. To quantify this, a double-entry intraclass correlation (ICC DE ) coefficient will be computed for each participant (McCrae, 2008). Given the reliability of clinician ratings of prototypical PDs and PD vignettes using the 30 FFM facets (Samuel & Widiger; Sprock), adequate inter-rater reliability is expected using the FFMSS. In addition, the current study investigates which FFM traits, as well as which types of individuals (i.e. personality profiles determined by consensus PD ratings), are more or less difficult to rate using the FFMSS. Research examining self and other ratings of personality may be valuable in evaluating the potential difficulty for clinicians in rating FFM dimensions using the FFMSS. For example, in a clinical sample of predominately depressed individuals, Bagby,

21 16 Rector, Bindseil, Dickens, Levitan, and Kennedy (1998) found that Openness and Neuroticism were rated somewhat less reliably relative to the other FFM domains, and Conscientiousness and Agreeableness were rated most reliably, followed by Extraversion. Based on these findings from the literature examining self-other convergence of personality traits, clinicians in the current study may have more difficulty providing reliable ratings for individuals with personality profiles characterized by Openness and Neuroticism. For example, Lynam and Widiger (2001) had experts rate the prototypic case for each DSM-IV PD and found that Schizotypal PD was characterized, in part, as being comprised of high scores on Openness. The prototypic Borderline PD case was rated as being high in Neuroticism. The DSM- IV states that the essential feature of Borderline PD is a pervasive pattern of instability of interpersonal relationships, self-image, and affects (p.710). Therefore, in the current study, it is expected that individuals with greater Schizotypal and Borderline pathology will be rated less reliably. Results of a recent study explicitly support the latter hypothesis. Specifically, Flury, Ickes, and Schweinle (2008) found that individuals high in Borderline PD symptomatology were more difficult to rate accurately on trait dimensions. The researchers further concluded that this was a result of Borderline PD individuals being more difficult to read and having more unusual and unpredictable personalities. Alternatively, it is hypothesized that individuals with personality profiles characterized primarily by Conscientiousness, Agreeableness, or Extraversion will be rated more reliably by clinicians. Based on prototypical ratings of PDs (Lynam & Widiger), it is expected that individuals with Obsessive-Compulsive (i.e., high Conscientiousness), Antisocial (low Agreeableness and Conscientiousness), Dependent (high Agreeableness), and Schizoid (low Extraversion) pathology will be rated more reliably. The current study also generates convergent and discriminant validity data for the

22 17 FFMSS. Associations between FFMSS traits and SNAP trait and temperament scales are also investigated. Results consistent with those found by Clark, Vorhies, and McEwen (2002) are expected. For example, FFMSS Extraversion should correlate significantly with SNAP Positive Temperament, Exhibitionism, Detachment, Entitlement, and Impulsivity, whereas FFM Neuroticism should be significantly related to SNAP Negative Temperament, Self-Harm, Mistrust, Dependency, and Aggression. FFMSS Conscientiousness should correlate significantly with SNAP Disinhibition, Workaholism, Impulsivity, Manipulativeness, Dependency, and Propriety. The relations between FFMSS ratings and expert consensus ratings of DSM-IV PDs are also examined. Findings should reflect theoretical conceptualizations of PDs from a FFM perspective (Widiger, Trull, Clarkin, Sanderson, & Costa Jr., 2002). For instance, individuals with higher expert consensus ratings for Borderline PD should be rated higher on the FFMSS Neuroticism facets of anxiety, angry-hostility, depression, impulsiveness, and vulnerability, and they should be rated lower on FFMSS Agreeableness facets of trust and compliance, in addition to the competence facet of Conscientiousness. The success of the FFMSS in calculating FFM PD counts, which have proven to be a successful method of scoring FFM data to evaluate DSM-IV PDs (Miller, Bagby, Pilkonis, Reynolds, & Lynam 2005), is also examined in the current study. To determine success, correlations between FFMSS generated FFM PD counts and expert consensus PD ratings will be computed. Results should be consistent with Miller and colleagues findings when using the NEO PI-R to generate FFM PD counts, in that the median correlation should range from.40 to.45. The association between FFMSS ratings and ratings of psychological distress (depression and anxiety) and impairment (romantic relationships, other social relationships, occupational impairment, distress caused to significant others, and overall impairment) is also examined in the

23 18 current study. Positive associations between Neuroticism and both anxiety and depression ratings are expected. Conscientiousness is also expected to be negatively related to anxiety and depression, whereas Extraversion should be negatively associated with depression ratings only. In terms of impairment, positive associations with Neuroticism, and negative associations with Extraversion, Agreeableness and Conscientiousness are hypothesized. The current study also examines the predictive validity of the FFMSS with regard to functional impairment. More specifically, domain and facet ratings, as determined by the FFMSS, will be used to predict impairment at a two-year follow-up. It is expected that individuals rated high in Neuroticism will have higher impairment ratings at the two-year follow up. Also, individuals rated low in Agreeableness, Conscientiousness, and Extraversion should also have higher impairment ratings at follow-up. Finally, the current study evaluates the incremental validity of the FFMSS by examining whether the FFMSS domains account for additional variance beyond SNAP traits in the prediction of consensus PD ratings. Based on Reynolds and Clark s (2001) findings, it is expected that FFM domains will significantly predict each of the 10 DSM-IV PDs, with the exception of Schizotypal PD. Although the FFM domains did not significantly predict Obsessive-Compulsive PD in the Reynolds and Clark study, this may not be the case in the current study given the FFMSS rating scale which allows for maladaptively high levels of Conscientiousness. Based on Reynolds and Clark s findings, the FFMSS should demonstrate incremental validity only for Avoidant PD. In contrast, the SNAP should exhibit substantial incremental validity across the 10 PDs (i.e., between 16% [Avoidant PD] and 35% [Paranoid PD]).

24 19 One important consideration in examining the potential usefulness of the FFMSS is whether familiarity with the FFM has an impact on reliability and validity. Given that the primary investigator (i.e., Josh Miller) in this study has had extensive experience working with the FFM and was involved in construction of the FFM PDs (Miller, Bagby, Pilkonis, Reynolds, & Lynam, 2005), this is an important issue to address. Therefore, FFMSS data with and without inclusion of his ratings is examined. Therefore, FFMSS data with and without inclusion of his ratings will be examined. I will first compare the reliability of clinician ratings when the primary investigator s ratings are included versus excluded. In order to examine validity, it will then be determined whether data from the other raters, who received limited instruction about the FFM domains and facets, can be used to generate FFM PD counts. In other words, will this FFM PD data correlate with consensus DSM-IV PD ratings to the same extent as data with the primary investigator s ratings? If the FFMSS is found to be a reliable and valid assessment of FFM personality traits, this would provide important support for the use of the FFMSS in clinical samples as it would be an easy to use but incredibly flexible instrument. The implications of this are substantial, in that a considerably shorter, self or other-rated assessment of facets of the FFM could be utilized for diagnosis, case conceptualization, and treatment planning.

25 20 CHAPTER 2 METHOD Participants Participants included 130 outpatients (97 females; 33 males) from the Western Psychiatric Institute and Clinic (WPIC) in Pittsburgh, Pennsylvania between the ages of 21 and 60 (M = 37.91, SD = 10.63). The majority were white (74.6%); of the remaining participants, the largest percentage were African American (23.8%), with the final 1.6% comprised of individuals self-identified as either Asian or American Indian. Fifty percent were single, 26.9% were either separated or divorced, 16.1% were married, 5.4% lived with a long-term domestic partner, and 1.5% were widowed. With regard to education,.8% of participants did not attend high school, 5.4% of participants attended but did not complete high school, 12.3% completed only high school, 39.2% had some college or vocational training, 24.6% graduated from college, and 17.7% had graduate or professional training. In addition, 68.8% of 128 participants (2 participants did not provide this information) reported an annual income of less than $25,000, 17.9% between $25-50,000, and 13.3% reported more than $50,000 per year. Measures Five Factor Model Score Sheet (FFMSS, see Appendix). The FFMSS (Widiger & Spitzer, 2002) is a one-page rating sheet consisting of 30 items representing each of the 30 facets of the FFM, as conceptualized in the NEO PI-R. These facets are organized with respect to the five FFM domains, such that there are six items beneath a listed domain. Each item includes a list of two to four adjectives describing the trait. For example, high straightforwardness, a facet of Agreeableness, is described as confiding and honest, whereas low straightforwardness is

26 21 described as cunning, manipulative, and deceptive. Each item is rated on a 1 (problematic, very low on the trait) to 7 (problematic, very high on the trait) scale. Of the 130 participants, 18 were rated by only one rater, 109 were rated by two raters, and 3 participants were rated by three, four, or five raters respectively. The primary investigator rated 50 of the 130 participants and was never the rater for a participant with only set of ratings. Consensus ratings of DSM-IV PD criteria. These ratings were determined in each participant s case conference. A consensus rating of each DSM-IV PD criteria was determined using a 0-2 scale, with 0 indicating absent, 1 indicating present, and 2 indicating strongly present. Criterion scores for each participant can be added to generate a symptom count for each PD. The LEAD method (Spitzer, 1983) was utilized in determining consensus ratings. This method emphasizes the contribution of expert clinical judgment, but includes the use of multiple information sources in arriving at that judgment. These sources included assessment interviews with the patient, self-report measures completed by the patient, as well as judgments of other professionals. Schedule for Nonadaptive and Adaptive Functioning (SNAP). The SNAP (Clark, 1993) is a 375-item, true-false inventory that assesses 15 traits relevant to PD: 12 lower order primary traits, and three broad temperament dimensions Negative temperament, Positive temperament, and Disinhibition (vs. constraint). The scales were developed to assess 22 PD symptom clusters, and each of the 15 scales corresponds to one or two of the symptom clusters and is composed of separate, non-overlapping items. The SNAP scales have demonstrated good reliability in both college and clinical samples with median alpha coefficients ranging from.76 to.83 (Clark, Vorhies, and McEwen, 2002). In terms of validity, SNAP scales converge well with other measures of personality, both general and pathological, including the NEO PI-R and the

27 22 Dimensional Assessment of Personality Pathology Basic Questionnaire (Livesley, 1990). In addition, the SNAP has been shown to exhibit theoretically expected relationships with DSM Axis II diagnoses (Clark, McEwen, Collard, & Hickok, 1991; Clark, Vorhies, and McEwen, 2002). Ratings of psychological distress. These ratings were conducted with the HAM-D and HAM-A by the primary interviewer for each participant. Studies have supported the reliability of HAM-D ratings by clinicians (Carroll, Fielding, & Blashki, 1973). The HAM-A has also been shown to be both reliable and valid when utilized with individuals with depressive disorders and anxiety disorders (Maier, Buller, Philipp, & Heuser, 1988). It has also demonstrated acceptable reliability and good construct validity when used with adolescent psychiatric populations (Clark & Donovan, 1994). Consensus ratings of impairment. Consensus ratings were determined separately for romantic relationships, other social relationships (e.g. friends, family members), occupational impairment, distress caused to significant others (e.g. friends, children), and overall impairment. The three judges in the case conference rated each participant using a 1 to 5 scale with higher scores indicative of greater impairment. As with the consensus PD ratings, the LEAD method (Spitzer, 1983) was used in generating impairment ratings. The two-year follow-up impairment data were generated using this same method. Procedure Announcements describing the characteristics of Borderline PD, Avoidant PD, and levels of social functioning indicative of no PD were posted in the outpatient clinic at WPIC. Although the current study does not focus specifically on these subgroups, it is expected that there is enough variability in general personality traits and PD pathology in the sample to be useful for

28 23 exploring a broader array of PD symptoms. In addition, research has shown considerable overlap in PD diagnoses, and thus it was expected that these groups would evidence heterogeneity with respect to Axis II diagnoses. Interested participants contacted the research staff and were screened (for exclusion purposes) via telephone to determine whether they met any of the following criteria: psychotic disorders, organic mental disorders, mental retardation, and major medical illnesses that influence the central nervous system and might be associated with organic personality disturbance (e.g. Parkinson s disease, cerebrovascular disease, and seizure disorders). Eligible participants provided written consent after all study procedures had been explained. Subsequently, they were assessed by a primary interviewer in a minimum of three sessions lasting approximately 2-3 hours each. Session 1 included the administration of the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I; First, Gibbon, Spitzer, & Williams, 1997) and other measures of current symptomatology. In session 2, a detailed social and developmental history was taken, using a semi-structured interview, the Interpersonal Relations Assessment (IRA; Heape, Pilkonis, Lambert, & Proietti, 1989). During session 3, the Structured Clinical Interview for DSM-IV Axis II PDs (SCID-II; First, Gibbon, et al. 1997) was administered. The participants also completed self-report measures in between assessment sessions, such as the SNAP. Following the three assessment sessions, the primary interviewer rated the participant on the Hamilton Rating Scale for Depression (HAM-D; Hamilton, 1960), the Hamilton Rating Scale for Anxiety (HAM-A; Hamilton, 1959), and the FFMSS. Subsequent to the three assessment sessions, the primary interviewer presented the case at a three-hour diagnostic conference with colleagues from the research team. A minimum of three judges participated. All available data (historical and concurrent) were reviewed and discussed at the conference. Judges were given access to all data that had been collected: current

29 24 and lifetime Axis I information, symptomatic status, social and developmental history, and personality features acknowledged on the Axis II interviews. The relevant data resulting from the case conference include (1) consensus ratings of DSM-IV PD criteria, (2) consensus ratings on impairment variables, and (3) FFMSS ratings completed by at least one of the judges. All FFMSS raters, including the primary interviewer and judges from the diagnostic case conference, received training in a one-hour session, in which they reviewed the FFM facets using descriptions from Costa and McCrae s (1992a) NEO PI-R manual. Raters also received copies of these descriptions to use as a reference when completing the FFMSS.

30 25 CHAPTER 3 RESULTS Prior to completing reliability and validity analyses, descriptive statistics for the FFMSS were examined. Means and standard deviations for the facets and domains are presented in Table 1. Several analyses were conducted to examine reliability of the FFMSS. First, internal consistency of the FFMSS domains was examined. Second, individual profile agreement, or reliability of ratings for each participant on the FFMSS facets, was determined. Reliability of ratings across the FFMSS domains and facets was also examined to evaluate whether certain traits are more difficult to rate reliably. Third, reliability coefficients (i.e., ICC DE generated from individual profile agreement analyses) were used as an independent variable to determine whether individuals who are more difficult to reliably rate have higher levels of PD pathology (e.g., Borderline pathology). Finally, given that the FFMSS has not been used or evaluated in previous research, exploratory factor analysis of facet scores was conducted. Intercorrelations between FFMSS domains were also computed. Following the reliability analyses, several analyses were conducted in order to examine convergent and divergent validity. First, composite ratings were generated by averaging each of the participant s ratings along the 30 FFMSS facets. For the 18 participants with only one rater, generation of composite ratings was not possible and the single ratings were used. The resulting composite facet ratings were then used to compute FFMSS domain scores for each participant (simply an average of the six composite facet ratings subsumed by each domain). Second, these composite ratings were examined in relation to (1) SNAP scales, (2) consensus PD ratings,

31 26 (3) measures of psychological distress (as measured by the HAM-A and HAM-D), and (4) consensus impairment ratings across a variety of domains (e.g., romance, work). Third, the composite facet ratings were used to generate FFM PD counts and compared to consensus PD ratings to test whether the FFMSS data can be used to derive DSM-IV PD scores in a manner similar to the NEO PI-R. We also examined the amount of variance in consensus PD ratings accounted for the by FFMSS domains, as well as the incremental validity of the five domains above and beyond SNAP traits. Finally, both predictive and incremental (relative to Axis I and Axis II pathology) validity of the FFMSS were evaluated by predicting two-year follow-up impairment data from the FFMSS ratings. 1 Analyses were also included to examine whether familiarity with the FFM impacts the reliability and validity of the FFMSS. To determine this, individual profile agreement was examined for participants that were not rated by the primary investigator. Additionally, data excluding the primary investigator s ratings were used to generate FFM PD counts and subsequently compared to consensus PD ratings. As a final note, to control for Type I error, we lowered our significance level to p <.01 for all analyses. Internal Consistency and Inter-rater Reliability Coefficient alphas (presented in Table 2) for the FFMSS domains ranged from.58 (Neuroticism) to.96 (Conscientiousness). More detailed analysis of the Neuroticism domain revealed that the angry hostility and impulsiveness facets were either weakly or negatively related to the other four Neuroticism facets, thus substantially decreasing the alpha coefficient for this domain. To examine individual profile agreement, a double-entry intraclass correlation (ICC DE ) was computed for 112 of the 130 participants. 2 One benefit of this approach is that it takes into