Psychometric Details of the 20-Item UFFM-I Conscientiousness Scale

Transcription

1 Psychometric Details of the 20-Item UFFM-I Conscientiousness Scale Documentation Prepared By: Nathan T. Carter & Rachel L. Williamson Applied Psychometric Laboratory at The University of Georgia Last Updated: July 2014 Development of the UFFM-I was supported by funding from the UGA Provost Research Seed Grant awarded to the first author. Note: This scale may be used freely by any not-for-profit researcher or practitioner with appropriate citation of the scale and its scoring program in resulting publications and/or reports. For-profit entities must obtain permission from the first author.

2 Psychometric Details of the 20-Item UFFM-I Conscientiousness Scale 1. Content Development and Item Pool Selection This document describes the technical details of the Unfolding Five Factor Model (UFFM-I) 20-item scale and reports its psychometric properties. The UFFM-I Conscientiousness scale was developed by first creating six 9-item facet scales according to the Costa and McCrae (1995) structure of FFM trait facets: 1. Competence 2. Orderliness 3. Dutifulness 4. Achievement Motivation 5. Self-Discipline 6. Cautiousness From these six scales, two 14-item measures to represent the higher-order factors, Orderliness and Industriousness identified by DeYoung et al. (2007) Items were written by graduate and undergraduate research assistants in the Applied Psychometric Laboratory trained by the author. For each facet, each of three item writers wrote 60 items (20 positively-worded, 20 negatively-worded, and 20 moderatelyworded), resulting in a pool of 180 items per construct. This pool of items was checked for quality of wording and redundancies. The resulting item pool (approximately 120 items per facet) was then evaluated independently by four undergraduate research assistants by: (1) rating the extremity of the item on a 7-point scale; and (2) conducting a Q-sorting task by assigning each item to the facet they believed the item belonged to (raters were provided with all facet definitions). For an item to be retained, it had to meet two criteria: (1) at least 75% of raters had to assign the item to the correct facet dimension; and (2) the standard deviation of the item extremity ratings had to be 1 or less. For each facet, 30 items with mean extremity ratings that spanned the entire 7-point scale range uniformly were selected. 2. Selection of Items for the 20-Item General Conscientiousness Scale After selection of 180 items (30 items per facet), item response theory (IRT) analyses were conducted on a sample of 732 participants. Data were collected via the Amazon Mechanical Turk crowdsourcing website. The generalized graded unfolding model (GGUM; Roberts, Donoghue, & Laughlin, 2000) was chosen as the unfolding IRT model

3 due to its applicability to polytomous items. Each 30-item set for the six facets were analyzed separately using the GGUM2004 software program (Roberts, Fang, Cui, & Wang, 2006). For each facet, a 9-item scale was developed by selecting items that maximized score model-data fit and scale reliability across the trait continuum (details on facets are also available on the Unfolding Project website). Next, the 9-item scales for Orderliness, Dutifulness, and Cautiousness were combined to form a scale represented the higher-order Orderliness factor, and the Competence, Achievement Motivation, and Self-Discipline were combined to form a scale of the higher-order Industriousness factor. These two 27-item combinations were analyzed separately, and 14 items were chosen that maximized model-data fit and score reliability across the trait continuum while maintaining nearly equal representation of each facet (details on the Orderliness and Industriousness scales are also available on the Unfolding Project website). Finally, the two scales representing the DeYoung et al. (2007) factors were combined and analyzed together. Items again were selected to maximize model-data fit and score reliability while maintaining a balanced representation of the lower-order facets. 3. Item Response Theory Analysis of the UFFM-I Conscientiousness Scale In this section, we present analyses of the UFFM-I Conscientiousness scale. The scale was analyzed using the full GGUM (Model 8) using the GGUM2004 program (Roberts, Fang, Cui, & Wang, 2006) which using marginal maximum likelihood in estimation of item parameters and expected a posteriori (EAP) in estimation of person scores. Modeldata fit was assessed using the MODFIT v2.0 program (Stark, 2007), which calculates the 2 /df ratio (adjusted to N=3,000 to promote more accurate fit assessment; see Chernyshenko et al., 2001) item fit statistic. Ratios less than three are considered indicate of good fit. Table 1 shows that all item 2 /df ratio singles were below 3, and indicates excellent model-data fit. Doubles and triples were slightly high, indicating the potential for a moderate degree of local independence (i.e., items are interrelated for reasons other than conscientiousness), which is not surprising given the hierarchical nature of this general trait variable (i.e., due to the presence of lower-order facets). Table 1. Frequency table for Adjusted (N=3,000) 2 /df ratios for the UFFM-I Conscientiousness scale. <1 1<2 2<3 3<4 4<5 5<7 >7 M SD Singles Doubles Triples

4 Figure 1. Model-data fit plot Graphical analyses comparing observed scores versus the scores that would be expected (or predicted) by the estimated model confirmed the excellent model-data fit of the UFFM-I Conscientiousness scale (see Figure 1). The GGUM has two primary parameters for consideration: and. The parameter is an estimate of how extreme an item is, and corresponds to the trail level,, of persons whom would be likely to fully endorse the item. For example, a person whom fully endorses Item 12, I always go above and beyond what is expected, would be expected to be 3.18 SD above the mean. On the other hand, a person whom fully endorses Item 16, I wouldn t describe myself as messy or clean; my organization is average, would be expected to be around.13 SD below the mean. Figure 2. Item characteristic curve A truly unfolding scale Figure 3. Test characteristic curve would be expected to have as approximately as many moderate items as there are extreme items, which would imply curvilinear relationship between traditional sumscoring and unfolding scores. Indeed, the Test Characteristic Curve (TCC), which relates unfolding scores ( ) to traditional Likert-type sum scores. As shown in Figure 3, the TCC clearly shows the expected curvilinear relationship between the two types of scores, with the inflection point of the curve occurring approximately at the mean (0) of unfolding scores. Finally, a crucial consideration in all IRT analyses is the Test Information Curve (TIC), which describes the reliability of measurement (called information in IRT) for different levels of estimates (i.e., unfolding scores. One of the major advantages of a truly unfolding measure is that including a variety of items allows for more reliable measurement across the trait continuum than scales developed using traditional approaches. Show in Figure 4, the TIC shows that reliability of measurement is relatively constant across the Figure 4. Test information curve

5 distribution of unfolding scores. This means that at most levels of conscientiousness, the UFFM-I scale is similarly reliable. To summarize, the UFFM-I scale for measuring general conscientiousness showed excellent model-data fit, truly unfolding properties, and high reliability across the trait continuum. Use of this scale is likely to be highly advantageous in applications wherein it is important to distinguish between persons whose conscientiousness is extremely high and those whose conscientiousness is moderately high. 4. Scoring the UFFM-I Conscientiousness Scale in Small Samples A major goal of the Unfolding Project is to make the use of unfolding measures more accessible. This is accomplished by providing a scoring program for each scale developed. This program is a SPSS macro titled UFFM-I CONSCORE, and is a specialized modification of the GUMSCORE SPSS (Carter & LoPilato, 2014), both available at the Applied Psychometric Laboratory website. The difference between these program is that whereas GUMSCORE SPSS requires the user to enter their own item parameters, the UFFM-I CONSCORE program has item parameters built into it specifically for the use of the 20-item UFFM-I Conscientiousness measure. To run this program, the user need only: (1) collect data using the full UFFM-I conscientiousness scale with a 6-point scale (strongly agree to strongly disagree); (2) download the UFFM-I CONSCORE folder directly to the C:\ drive; (3) enter the data into the Resp_File.sav file, in the order shown in the file titled UFFM-I Conscientiousness (20 Items).pdf or in the order indicated by the Item # column shown in Appendix A (note, in Appendix A items have been sorted by their item location and data should not be entered in the order shown); (4) Run the portion of the SPSS syntax below the first line (from the command DEFINE to the command!enddefine; this defines the macro titled!conscscore); (5) Change the parameter p on the first line to reflect the number of persons entered into the dataset (the default of this parameter is p=1); and (6) Select and run the first line of the syntax (beginning with!conscscore). The result will be a SPSS.sav file titled RESULTS_Theta_SE.sav which will include unfolding scores, or Thetas, and their standard errors (SEs). Additionally, this file will include the response pattern for each person so that the user can ensure the data were read into the program correctly. This program uses EAP estimation (using 30 quadrature points) to score respondents (the same approach used in GGUM2004) with the item parameters reported on the last page of this document. Therefore, respondents scored from different samples will be placed on the same scale as in the calibration sample. The UFFM-I measure and associated scoring program may be used by any not-forprofit research or practice. Any for-profit use requires the permission of the first author of this report.

6 REFERENCES Carter, N.T., & LoPilato, A. (2014). GUMSCORE SPSS [software program]. University of Georgia: Athens, GA. Available at: Chernyshenko, O.S., Stark, S., Chan, K.Y., Drasgow, F., & Williams, B. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, Costa, P.T., & McCrae, R.R. (1995). Domains and facets: Hierarchical personality assessment using the revised NEO personality inventory. Journal of Personality Assessment, 64, DeYoung, C.G., Quilty, L.C., & Peterson, J.B. (2007). Between facets and domains: 10 aspects of the Big Five. Journal of Personality and Social Psychology, 93, Roberts, J.S., Donoghue, J.R., & Laughlin, J.E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24, Roberts, J. S., Fang, H., Cui, W. And Wang, Y. (2006). GGUM2004: A Windows-based program to estimate parameters in the generalized graded unfolding model. Applied Psychological Measurement, 30, Stark, S. (2007). MODFIT version 2.0 [software program].

7 APPENDIX: Item Parameters and Test Content for the 20 Item UFFM-I Conscientiousness Scale (Note: In applications using the UFFM-I CONSCORE program, data should be entered in the order indicated by Item #) ITEM # ITEM CONTENT 1 I tend to do just enough work to get by I procrastinate a lot I do not keep my room clean My performance at work is always adequate, no more and no less I am good about getting things done on time but sometimes I do not manage my time well I let my room get kind of messy but I don t let it get out of control I wouldn t describe myself as messy or clean, my organization is average I would say my self-discipline is about the same as most people I prefer to be above average at things but don t have to be the very best I would say I understand things at a normal pace I usually excel in what I m doing but occasionally I ll do mediocre at something I follow the rules about as much as most people I would say I am more disciplined than most, but there a lot of people with better self-discipline than 7 me I love to win, but I am not a sore loser I always respect authority, even if I disagree with 20 them If there is a problem, I can usually solve it I would never jump into doing something without thinking about it I am very well organized I always follow through with my plans I always go above and beyond what is expected