ACCURACY OF INTERVIEWER JUDGMENTS OF JOB APPLICANT PERSONALITY TRAITS

Transcription

1 PERSONNEL PSYCHOLOGY 2000,53 ACCURACY OF INTERVIEWER JUDGMENTS OF JOB APPLICANT PERSONALITY TRAITS MURRAY R, BARRICK Department of Management Michigan State University GREGORY K. PATTON, SHANNA N, HAUGLAND Department of Management and Organizations University of Iowa This study investigated whether interviewers can assess Big Five personality traits during a job interview. Four raters (self, interviewer, friend, and stranger) assessed the apphcant's personahty. Results from ratings for 73 applicants demonstrated that interviewer ratings of applicant personality correlate higher with self-ratings (f =.28) than do stranger ratings (f =.09) but less than ratings from close friends (f =.39). However, correlations between interviewer ratings and selfratings were smaller for the two job-relevant personality traits. Conscientiousness (r =.16, n.s.) and Emotional Stability (r =.17, n.s.) than for the other three personality traits. Variance in ratings suggested the applicants managed their self-presentation on these two traits during the interview. Thus, although interviewers can and do assess personality during the interview, they are not able to assess those traits that would best predict later job success. Finally, the moderating effect of interview design (i.e., structure and content) was assessed. The results revealed that job-relevant interviews, situational interviews, and behavioral interviews did not affect an interviewer's ability to assess personality. Although there was a small (positive) effect for more structured interviews, this affect was modest. The results of this study suggest that future research should examine whether the interview can be designed to assess personality directly, and what the gains to predictive validity are by doing so. Somewhat surprisingly, little research exists that examines the empirical relationships between applicant personality characteristics and interviewer judgments of those characteristics. Nevertheless, it has been argued that one important purpose of an interview is to assess the applicant's personality (Burbage, 1991; Caldwell & Burger, 1998; Rothstein & Jackson, 1984; White, 1993). A recent meta-analysis examining the incremental validity of the interview (Cortina, Goldstein, Payne, Davison, & Gilliland, 2000) found nine correlation coefficients (N = 966) Correspondence and requests for reprints should be addressed to Murray R, Barrick, Department of Management, Eli Broad Graduate School of Management, Michigan State University, East Lansing, MI ; barrick@msu,edu, COPYRIGHT 2000 PERSONNEL PSYCHOLOGY, INC. 925

2 926 PERSONNEL PSYCHOLOGY between measures of conscientiousness and structured interviews. The results revealed a modest relationship (p =.26). Further, other selection tools have also been linked to applicant personality and employment decisions. Recent evidence suggests that recruiters also infer personalitylike traits from biodata (Bretz, Rynes, & Gerhart, 1993; Brown & Campion, 1994). Dunn, Mount, Barrick, and Ones (1995) found that interviewers can accurately use personality information in hiring decisions if it is available to them. Furthermore, Caldwell and Burger (1998) demonstrated that inferences of applicant personality traits influence the number of follow-up interviews and job offers. In addition, much of the person-organization fit literature (Cable & Judge, 1997; Judge & Bretz, 1992; O'Reilly, Chatman, & Caldwell, 1991) focuses on matching the traits valued by the organization with the traits displayed by the applicants. Yet, no studies have directly assessed whether interviewers actually draw inferences about the applicant's personality. Consequently, the purposes of this article are to examine the extent to which specific personality traits can be assessed during a typical job interview and the manner in which those ratings relate to evaluations of the applicant's suitability. From a practitioner's perspective, the degree to which interviewers typically access personality will help clarify the potential benefits or limitations of the employment interview. Once these boundaries are identified, practitioners will be better able to decide the importance of supplementing the interview with other and often less costly predictors (e.g., personality tests) to increase the utilify of their selection decisions. In addition, a better understanding of one's abilify to assess personality constructs in interviews could lead to improvements in interview design and training. Such improvements could ultimately enhance the predictive validity of employment interviews. This study is a step toward a better understanding of the likely relations among various selection predictors. In this study, we rely on the five-factor model of personality (Digman, 1990) to classify personalify traits in a meaningful way. Labels and prototypical characteristics associated with constructs in the five-factor model include Extraversion (sociable, active, energetic), Agreeableness (cooperative, considerate, trusting). Conscientiousness (dependable, organized, persistent). Emotional Stabilify (calm, secure, unemotional), and Openness to Experience (imaginative, intellectual, artistically sensitive). Recent meta-analyses have found that the predictive validify of Conscientiousness and Emotional Stabilify to job performance generalizes across jobs (Barrick & Mount, 1991; Barricl^ Mount, & Judge, in press; Hough, 1992; Salgado, 1997). Consequently, these two personalify traits are expected to be relevant predictors of job performance in virtually all jobs. Other personalify dimensions have been found relevant

3 MURRAY R. BARRICK ET AL. 927 for specific criteria or in specific jobs (Barrick & Mount, 1991; Barrick et al., in press; Hough, 1992; Salgado, 1997). Taken together, these findings underscore the importance of assessing whether interviewers judge applicant personalify traits during a job interview that is not structured specifically to assess personalify. Although there is little research on interviewer inferences of applicant personalify in selection settings, there are a number of studies in the personalify literature that have examined interjudge agreement on personalify traits. Some studies have examined the extent to which peer judgments of personalify agree with each other, and others have examined the extent to which they agree with a subject's self-judgments (Colvin & Funder, 1991; Connolly & Viswesvaran, 1998; Funder & Colvin, 1988; Watson, 1989). A review of this literature makes clear that: (a) people draw inferences about the personalify of others, even after relatively short exposure, (b) there can be high levels of agreement on personalify traits between observer and self-ratings, and (c) the accuracy of self-observer agreement depends on the information available to the observer about the subject's thoughts, feelings, and actions and the extent to which it matches the information available to oneself (Colvin & Funder, 1991; Connolly & Viswesvaran, 1998; Funder & Colvin, 1988; Hayes & Dunning, 1997; Watson, 1989). These observations lead to an obvious set of empirical questions: (a) To what extent do interviewers infer personalify traits of applicants during a typical interview that is not structured specifically to assess personalify? (b) Do those inferences agree with the applicant's self-ratings? (c) Is the level of agreement between interviewer-self ratings less than that obtained for close friends yet greater than that for strangers? Despite their importance, these questions have received little attention in the research literature. Interestingly, Motowidlo et al. (1996) assessed the correlation between observers' personalify ratings of 60 managers derived from recorded interviews with supervisor ratings of the managers' performance. Results indicated that Extraversion (r =.27) and Conscientiousness (r =.20) significantly predicted supervisory ratings of job performance. Mount, Barrick, and Strauss (1994) also demonstrated that observer ratings (supervisor, coworker, and customer, N = 105,105, 92, respectively) of personalify, particularly Conscientiousness and Extraversion, accounted for significant variance in ratings of sales performance, beyond self-ratings of personalify data. Taken together, these results demonstrate the potential value to be derived from an interviewer's personalify inferences about an applicant during a job interview. Yet the questions remain, can interviewers judge applicant personalify traits, and do those ratings converge with self-ratings? Although

4 928 PERSONNEL PSYCHOLOGY the research literature shows there is agreement between observer and self-ratings, studies consistently demonstrate that convergence increases as the rater becomes well acquainted with the target-self. The meanweighted correlation between self and observer-ratings from a recent meta-analysis by Connolly and Viswesvaran (1998) was considerably lower for strangers (K = 6to8; N ranges from 609 to 724 across the FFM constructs) than well-acquainted peers (K = 22 to 33; N ranges from 3,181 to 4,001): Extraversion (f =.29 vs..41, strangers and peers, respectively), Agreeableness (f = -.01 vs..30), Conscientiousness (f =.23 vs..39), Emotional Stabilify (f =.05 vs..27), and Openness to Experience (f =.14 vs..35). Nevertheless, the stranger-self mean-weighted correlations for Extraversion (f =.29), Conscientiousness (f =.23), and to a lesser extent. Openness to Experience (f =.14) exhibited surprisingly high convergent validify. In many of these studies, stranger ratings were based on at most 15 minutes of nonverbal contact with the self-targets, and most studies allowed less than a minute of nonverbal or very limited (and structured) verbal contact (e.g., subject states his or her name). Thus, although these convergent validify correlations are not high, it is puzzling that correlations of this magnitude can be obtained under such circumstances. Such results are important for this study, as they suggest that interviewer assessments of applicant personalify are likely to converge with self-ratings for at least some dimensions of personalify. Tkken together, these results show that observer judgments of personalify are often valid, particularly with greater acquaintanceship. Recent research revolves around one additional determinant of the magnitude of self-observer agreement in personalify judgements. Several studies suggest that "easiness" of trait judgment or trait visibilify relates to agreement (Colvin & Funder, 1991; Funder & Colvin, 1988; Hayes & Dunning, 1997; John & Robins, 1993). Trait ambiguify, trait observabilify, trait social desirabilify or favorabilify, and the frequency of trait occurrence have been found to influence how easy it is to accurately judge a trait. This research (Colvin & Funder, 1991; Funder «fe Colvin, 1988; Hayes & Dunning, 1997; John & Robins, 1993) consistently identifies Extraversion as the easiest trait to assess, and Emotional Stabilify is the most difficult to judge. In the case of Extraversion, being active, talkative, and energetic are characteristics that enhance the visibilify of the trait, and hence agreement. Conversely, Emotional Stabilify, which is primarily an internally oriented psychological trait, is less visible and consequently, harder for observers to judge. Based on these findings, one would expect observer-self agreement to be higher for those traits that are more visible and easier to judge. This is supported by the results reported by Connolly and Viswesvaran's (1998) meta-analysis where the sample-weighted mean correlations

5 MURRAY R. BARRICK ET AL. 929 between stranger-self-ratings for Extraversion and Conscientiousness (f =.29 and.23, respectively) were larger, and smaller for Emotional Stabilify (f =.05). Taken together, this research shows that agreement is higher between stranger ratings and self-ratings on traits that are easier to judge because they are more observable (e.g., Extraversion, Conscientiousness) as opposed to less visible traits (e.g.. Emotional Stabilify). In summary, very little is known about the interviewer's abilify to assess personalify traits in the selection context. However, given the literature cited above, it is likely that interviewer ratings correlate with selfratings more than stranger ratings, yet less than ratings from close acquaintances, due to the amount of contact between the observer and the applicant. Furthermore, the magnitude of agreement will differ across personalify traits, as previous research demonstrates that "visibilify" or "easiness" of trait judgment influences observer-self-rating agreement. Finally, interviewer ratings of applicant personalify are useful because they are expected to be related to important outcomes at work, such as evaluations of applicant suitabilify and ultimately job performance. Three hypotheses were tested in this study. First, we hypothesized that interviewer ratings of fraits would correlate higher with applicant ratings than stranger ratings of traits but lower than close acquaintance ratings, such as friends (Hypothesis 1). Second, we hypothesized that those correlation's between interviewer ratings of Extraversion and Conscientiousness with applicant ratings would be highest, and those for Emotional Stabilify would be lowest (Hypothesis 2). Third, we hypothesized that interviewer judgments of applicant personalify would be related to interviewer evaluations of applicant's suitabilify to the organization (Hypothesis 3). The first two hypotheses examine whether relations found in other (nonwork) settings extend to the employment setting (the interview). However, to help us better understand whether personalify constructs are assessed infypicalinterviews, this study also examined two (unique) potential moderators of the interview-personalify relationship. Potential Moderators of the Interview-Personality Relationship It is now widely accepted that the predictive validify of an interview is increased when the interview questions and scoring procedures are structured and the content is job-related (e.g., Huffcutt & Arthur, 1994; McDaniel, Whetzel, Schmidt, & Maurer, 1994). The incremental gain in validify of interviews designed this way could be due to gains in one's abilify to capture personalify constructs when using increased level of structure or more job-related interview questions. Thus, this study will examine the effect of the content and structure of the interview on the relationship between interviews and personalify. Interestingly, a recent

6 930 PERSONNEL PSYCHOLOGY meta-analysis (Huffcutt, Roth, & McDaniel, 1996) of the relationship between the interview and cognitive abilify underscored the importance of interview content and structure. Huffcutt et al. found that both the level of structure and the content of the interview affected interviewer assessments of the applicant's cognitive abilify. In a similar manner, it may be that the degree of interview structure (assuming the interview is not structured around personalify assessment) affects interviewer assessments of personalify. One might argue that applicant personalify would vary more in the unsfructured interview, because it is not "constrained" by the need to answer technical or even perhaps specific questions. Based on this argument, the interviewer would be expected to better assess personalify in an unsfructured interview. Conway, Jako, and Goodman (1995) demonstrated that increasing the sfructure of an interview improved interviewer reliabilify and suggested that through more consistent assessment across applicants, might also improve construct validify. Due to these "psychometric" improvements, increasing the degree of interview sfructure should enhance the magnitude of the personalify-interview relationships. However, the use of structured interviews does not increase the focus on personalify per se. Consequently, large gains in the magnitude of these correlations are not expected. In contrast, the use of job-related interview questions is likely to result in more substantial increases in the personalify-interview correlation. Schmit, Ryan, Stierwalter, and Powell (1995) found the validify of a personalify inventory was increased when work-context specific items were used. For example, Schmit et al. (1995) reported that the predictive validify was greater than.40 when using context-specific items of Conscientiousness, but only ranged from -.02 to.25 when using general context items. Furthermore, the use of context-specific items was found to provide a common frame of reference to all respondents, which reduced error variance and increased construct validify. Thus, to the extent interviewer assessments are saturated with personalify, the use of job-related interview questions should enhance the personalify-interview relationship. The content of an interview has also been operationalized by the type of questions asked (i.e., content). The two most common types of questions used in interviews are situational (future-oriented questions, which use hypothetical situations and ask the applicant what they would do) and behavioral (past-oriented questions, which ask the applicant to describe a situation from the past and describe what they did) approaches (Huffcutt et al., 1996). Huffcutt et al. found interviewer ratings better reflected cognitive abilify with situational interviews (p =.32) than behavioral description interviews (p =.18) when the interview was

7 MURRAY R, BARRICK ET AL, 931 highly structured. This was primarily atfributed to the belief that situational interview questions may load more heavily on verbal and inductive reasoning abilities than behavioral interviews. However, this would not affect interviewer judgments when evaluating personalify. Although most personalify inventories are past-oriented, there is no evidence that past-oriented questions have greater validify than future-oriented items (Burisch, 1984). Consequently, there is no theoretical basis for expecting a sfrong moderator effect based on the type of question. Thus, two final hypotheses involving the unique effects of interview design are examined in this study. First, a more sfructured interview is expected to result in stronger correlations with personalify assessments (Hypothesis 4). Second, the use of job-related questions should increase the interview-personalify correlation (Hypothesis 5). For both hypotheses, the hypothesized effects should be consistent across all five personalify constructs. Finally, although no hypothesis is proposed, we also examine whether the fype of interview content affects an initerviewer's assessment of personalify. Specifically, we examine whether having futureoriented content (situational interview) or past-oriented content (behavioral interview), differentially influences personalify assessments. Subjects Method Participants were 73 undergraduate business majors at a large midwestern universify. These students were on average, 21 years old (20.97), seniors (3.24, where 4 = senior, 3 = junior, etc.), and women (68%), with moderately high GPAs (3.28). Half of these students had prior internships (.47), had previously had two job interviews (2.05), and had received a moderate amount of training in preparation for participating in an interview (mean = 3.05, where moderate amount = 3 on a 4-point scale). The close acquaintances (friends) were approximately 21 years of age (21.33), seniors (3.37), primarily female (76%), and were well acquainted with the applicant (M = 3.93, where quite a lot = 4 on a 4- point scale). The interviewers were practitioners in the human resources field (N = 26), 42 years old (41.73), had over 12 years of interviewing experience (12.32), extensive interview training over the years (3.78; where quite a lot = 4 on a 4-point scale), and were primarily male (54%). Finally, other interviewers participating in the study provided stranger assessments (based on the videotaped statement from the applicants). In all cases, the strangers said they were unfamiliar with the applicant (M = 1.0; where did not know at all = 1 on a 4-point scale).

8 932 PERSONNEL PSYCHOLOGY Interviews The students attended a mock interview session and participated in this study to acquire greater interviewing experience. It should be noted that students seemed to take these mock interviews seriously as all wore formal attire and some appeared outwardly nervous. The interviewers participated as part of the local SHRM (Sociefy for Human Resource Management) chapter's oufreach initiative. As such, each interviewer had received significantfrainingon issues of validify, reliabilify, and utilify in the selection process. Each interviewer interviewed up to three students. Both students and interviewers knew the interview would not result in a job offer. However, at least two job offers did emerge due to the interviews. During each session, a student (applicant) was interviewed for approximately 30 minutes by an experienced interviewer. The interviewers were not prompted to use a specific sfyle of interviewing, nor were they given an interview script. Rather, interviewers were encouraged to use the interview method actually used in their organization. In this manner, the research design relies on the interview approach judged to be most effective within the consfraints of validify, reliabilify, adverse impact, and practical utilify. This also enabled us to more fully capitalize on the interviewers' previous experiences. After all interviews were completed each interviewer was asked to provide information about the sfructure and content of his or her interviews. Similar to current interview research (Conway et al., 1995; Huffcutt & Arthur, 1994; Huffcutt et al., 1996; & McDaniel et al., 1994) sfructure was dichotomized as being either high or low, and content was operationalized as job-related or not specifically job-related. In addition, interview content was also categorized as situational, behavioral, or a combination of the two. Post-Interview Debriefing At the conclusion of the interview, the applicant and interviewer went to separate rooms where they completed a questionnaire assessing applicant personalify and suitabilify to the job. Both the applicant and interviewer were encouraged to respond candidly to the questionnaire. The applicant then returned to the interview room, and received constructive feedback on her or his interviewing skills and presentation sfyle from the interviewer. Prior to leaving, the applicant was videotaped for 15 seconds. During this time, the applicant stated her or his name and study design number. The applicant was then given a questionnaire and a stamped envelope, and was asked to give the questionnaire to a close friend who would then return it directly to the researchers. Each

9 MURRAY R, BARRICK ET AL. 933 interviewer was given a videotape containing 15-second "statements" from three applicants and asked to complete a personalify questionnaire for each applicant. This rating served as the stranger rating. Consequently, a questionnaire containing a personalify measure and other relevant information (e.g., demographic data, evaluation rating, etc.) was completed by the applicant (self-rating), by the interviewer, a close acquaintance, and a stranger (another interviewer). Complete data was obtained for all subjects (N = 73). Measures Personality. Personalify was measured using a shortened version of the adjective checklist developed by Goldberg (1992), previously used by Mount et al. (1994). The adjective checklist contains 10-item scales for each of the five factor markers of Exfraversion, Agreeableness, Conscientiousness, Emotional Stabilify, and Openness to Experience. The coefficient alpha reliabilify estimates of these scales in this sample are reported in Table 1 and range from.71 to.90, with a mean of.78 for self-ratings,.79 for friend ratings,.82 for interviewer ratings, and.83 for the sfranger ratings. These reliabilify estimates are generally consistent with those reported elsewhere (Mount et al., 1994). Interviewer ratings of applicants' effectiveness. Stevens and Kristof s (1995) 6-item measure of interviewers' assessments of applicants' suitabilify and probable interview outcomes was used in this study to analyze interviewer ratings of applicant effectiveness. Sample items include: "This applicant appears to be very qualified for this position," "This applicant knows what is important for this position," and "How well did this applicant do in the interview." The reliabilify of this 6-item scale was.82. Interview structure and content. After all information was received from the interviewer a final set of questions was asked of each interviewer,^ addressing the design of the interview used during the study. These questions were developed using the framework followed in prior studies (Conway et al., 1995; Huffcutt & Arthur, 1994; Huffcutt et al., 1996). The first sets of questions were used to categorize the structure of the interview. More structured interviews were characterized as those with standardized questions ("Did you have specific questions to ask prior to the interview?" and "Did each candidate get asked the same ^The authors acknowledge and thank the anonymous reviewer for raising the concern about interview structure and content. The follow-up survey was a direct response to that concern, and provided more in-depth information on responses initially collected on interview design.

10 934 PERSONNEL PSYCHOLOGY TABLE 1 Correlations Between Interviewer Ratings With Ratings From, s, and s for Early and Later Interviews (N = 73) Extraversion Agreeableness Conscientiousness Emotional Stability Openness to Experience Number of interviews Early interviews Correlation between interviewer ratings and:,15,63,27,30,37,25,02,43,05,15,23,01,43,44,18 26 Later interviews Correlation between interviewer ratings and:,53,44,16,35,10,15,26,06,06,21,17 -,14,30,15,07 47 'core' questions as all other candidates?") and formalized response evaluations ("Did you form a single overall evaluation?" or "Did you evaluate responses to each question?"). Thirty-five interviews were categorized as structured and 38 were categorized as relatively unstructured. Interview content was categorized according to whether the interview was based on specific job requirements ("Were questions based on requirements of the job or a job analysis?"). A majority of interviews (61) were based on the requirements of the job and 12 were not based on job requirements. Although the primary issue was whether the interview questions were based on the job, we also examined the type of questions asked. That is, were questions related to past behavior (e.g., what did you do when?) or future-oriented (e.g., what would you do if?). "Eventy-five of the interviews were situational interviews (future-oriented), 31 were behavioral interviews (past-oriented), and 17 included both situational and behavioral questions.

11 Manipulation Checks MURRAY R. BARRICK ET AL. 935 Because interviewers provided data for more than one case, interpretation of the results could be confounded by systematic differences across interviewers due to "practice" effects (the interviewer might change his or her interviewing style for later interviews after completing the postinterview personality questionnaire after the first interview), and due to individual tendencies to "accurately" evaluate applicants. Tb test for "practice" effects, we compared interviewer ratings with ratings from self, friends, and strangers for first interviews and later (second and third) interviews. As reported in Table 1, correlations obtained from the first interviews were not significantly different from correlations obtained during later interviews. In fact, over all three sources (self, friend, and stranger ratings), the average correlation with interviewer ratings firom earlier interviews was slightly larger than from later interviews (f =.26 vs..19, respectively). However, the only pair of correlations that were significantly different were those between interviewer ratings of Extraversion and self-ratings (r =.15 and.53 for early and later interviews, respectively). In this case, the correlation for later interviews was significantly larger than for early interviews. To test whether there are interviewer differences in ratings, we conducted an analysis of variance on interviewer ratings using interviewer as the grouping variable.^ For four of the FFM constructs (Extraversion, Agreeableness, Conscientiousness, and Openness to Experience), the F ratios were not significant (F [25, 47] =.36,1.88,1.09, and 1.21). However, one of the FFM traits (Emotional Stability) was found to have significantly different mean ratings across interviewers F (25,47) = This analysis indicates there may be idiosyncratic effects for at least ^A concern noted by an anonymous reviewer pointed out that our research design, having each interviewer rate two to three applicants, created a confound when interpreting our results. As the reviewer pointed out, a possible solution would be to treat the data as if it were a meta-analysis. That is, treat each interviewer as an individual study and develop a single correlation, combine all interviewers (fc = 26, JV = 73), and correct for attenuation. The objective of this set of analyses would be to assess whether sampling error accounted for most of the observed variance. However, this approach creates two issues that we believe make it inappropriate. First, this approach is not effective when each interviewer only interviews two or three people, as each "analysis" (interviewer) produces a correlation that is not very "informative" (e,g,, in the case of two people, the correlation would be 1,0), Second, one of the assumptions, which underlie meta-analysis, is that differences between study results are largely attributable to measurement error. Therefore, the goal would be to identify differences across studies as measurement error rather then unknown error by correcting for unreliability and when applicable range restriction. Due to these concerns, we chose not to use this approach. Instead, we used a one-way ANOVA with interviewer as the grouping variable to test for heterogeneity. In so doing we were able to address the reviewer's concern about interrater differences without creating the problems related to a meta-analytic approach.

12 936 PERSONNEL PSYCHOLOGY one personality construct. However, heterogeneity in interviewer "accuracy" is not the only explanation for variation in interviewer ratings across FFM traits in this study. Furthermore, as we will see, idiosyncratic interviewer effects were not found for at least one of the personality traits that had low correlations between interviewer-self ratings. Thus, systematic differences across interviewers in their decision making "accuracy" do not appear to confound the findings reported below. Taken together, this evidence suggests that there are not influential differences in individual interviewers due to "practice" effects in this study or in their ability to provide "accurate" personality predictions. To the extent they do exist, however, they represent a downward bias on the magnitude of the correlations reported in the following tables. Results Observer- Agreement in the Overall Sample Tkble 2 presents means, standard deviations, and correlations between the applicants' self-ratings and observer ratings fi-om a close acquaintance (friend), interviewer, and stranger (based on the videotape). We first examined whether there were significant differences among rating sources for each of the personality dimensions by conducting a one-way analysis of variance, followed by Tukey's Honestly Significant Difference test. The mean ratings across sources differed for all five personality dimensions. Those means that were significantly different are denoted with subscripts in Tkble 2. For Extraversion, F(3,288) = , p <.01, with applicant, friend, and interviewer ratings greater than stranger ratings. For Agreeableness, F(3,288) = , p <.01, with applicant and friend ratings greater than interviewer ratings, which in turn, were greater than stranger ratings. For Conscientiousness, F(3,288) = , p <.01, with applicant ratings greater than interviewer ratings, which in turn were greater than stranger ratings. For Emotional Stability, F(3,288) = , p <.01, with applicant ratings greater than interviewer ratings, which were greater than friend and stranger ratings. Finally, for Openness to Experience, F(3,288) = , p <.01, with applicant ratings greater than interviewer ratings and stranger ratings, and friend ratings greater than stranger ratings. The most consistent finding from these analyses was that applicant ratings had the highest mean score, excluding Extraversion, where applicant ratings were only slightly lower than the highest mean (friend ratings). In contrast, stranger ratings were always (significantly) the lowest mean scores across all four rating sources.

13 MURRAY R. BARRICK ET AL. 937 TABLE2 Corrected and Uncorrected Correlations Between -Ratings With Ratings From s, Interviewers, and s (N = 73) M SD (applicant) Interviewer Extraversion Interviewer Agreeableness Interviewer Conscientiousness Interviewer Emotional Stability Interviewer Openness to Experience Interviewer 38,486 38,686 37,676 32,19^ 45,99c 45,01c 40,846 34,79a 44,03,: 42,366,c 40,896 34,11a 37,95c 32,60a 35,456 30,89a 36,266,c 35,486,c 33,34a,6 31,40a 5,81 5,93 7,09 5,47 3,56 4,64 5,39 4,51 3,82 5,98 4,39 5,17 4,05 5,35 3,33 4,47 5,41 5,45 5,59 5,98 (,84),41*,42*,17 (,78),38*,30*,18 (,79),41*,16,12 (,72),29*,17,01 (,75),46',34* -,02,49 (,82),50',11,47 (,85),20*,09,51 (,82),20*,05,41 (,71),21*,31',61 (,77),25*,17,49,59 (,89),19,37,24 (,85),18,19,24 (,86),07,24,30 (,71) -,09,44,32 (,81),11,20,13,21 (,88),22,10,21 (,88),14,06,08 (,90),01,42 -,12 (,75) -,03,23,14 (,74) Notes: Means with different subscripts within a personality trait are statistically different. Corrected correlations are provided in the upper diagonal. Corrections are based on ALPHA for each rating source. Because it does not take into account transient error, ALPHA is somewhat of an overestimate, which leads to an undercorrection for measurement error. In interpreting the correlations reported in Table 2, recall that the Connolly and Viswesvaran (1998) meta-analysis found the largest sampleweighted mean correlations between observer-self ratings for Extraversion, Conscientiousness, and Openness to Experience. Table 2 indicates that these results are largely replicated in the current data for close acquaintances (friends), as the correlation was.41 for Extraversion and Conscientiousness (p =.49 and.51, respectively), and,46 for Openness to Experience (p =.61). However, the evidence suggests that the strangers' trait ratings are not convergent with the applicant's own selfratings. In fact, only Extraversion and Agreeableness even approach significance (r =.17 and.18, respectively; p =.20 and.22). This is an

14 938 PERSONNEL PSYCHOLOGY important point to note, as the stranger ratings are provided by the interviewers, but are based only on impressions derived from a 15-second videotape. Consequently, stranger ratings provided by experienced interviewers did not converge with applicant ratings after a very short period of interaction in this study. Recognizing this, examination of the self-interviewer correlations based on the interview will demonstrate the "contribution" toward agreement derived from the interview itself.^ As shown in Table 2, only three dimensions of personality indicate significant self-interviewer convergence after the interview. In the current data, Extraversion again shows the highest convergent validity (r =.42, p =.49), and is very similar to the correlation reported for the close acquaintance (friend, r =.41, p =.49). Openness to Experience and Agreeableness also have significant self-interviewer correlations (r =.34 and.30, respectively; p =.44 and.37), but the correlations are consistently lower than those reported with the close acquaintances (friend r =.46 and.38, respectively; p =.61 and.47). In other words, although an interviewer's judgment of the applicant's Openness to Experience and Agreeableness corresponded to self- (applicant) ratings, they were not able to judge these dimensions as well as the applicant's friend. The results for interviewer ratings of Conscientiousness (r =.16, n.s., p =.19) and Emotional Stability (r =.17, n.s., p =.24) differ from findings in earlier studies of self-acquaintance correlations. In the current data, there was no significant interviewer-self agreement nor strangerself agreement on either factor. In contrast, there was significant friendself agreement in this study and in Connolly and Viswesvaran's (1998) meta-analysis for Conscientiousness (r =.41 and f =.39, respectively) and for Emotional Stability (r =.29 and f =.27). Taken together, these results demonstrate that interviewers were not able to assess the two most important job-relevant personality traits as well as close acquaintances. One explanation for this finding could be that the applicant was trying to "manage" his or her presentation in order to create a favorable impression with the interviewer. Prior research of impression management behaviors during actual interviews (Stevens & Kristof, 1995) demonstrated that applicants increase the use of self-promotion tactics to manage their impression during interviews. Inspection of the standard deviations reported in Table 2 illustrates these two personality dimensions did This assumes any differences between interviewer ratings and stranger ratings are primarily due to the additional information acquired during the employment interview. An alternative explanation is these differences are due to the large difference in time spent in contact with the subject, as stranger ratings were based on a 15-second videotape, whereeis the interview lasted for 30 minutes. However, to enhance generalizability with prior studies of stranger ratings, we kept the contact period short. Consequently, we cannot rule out the possible affects of this alternative explanation.

15 MURRAY R. BARRICK ET AL, 939 in fact have the lowest variance estimates for interviewer ratings, Conscientiousness (SD = 4.39) and Emotional Stability (SD = 3.33), followed by Agreeableness (SD = 5.39), Openness to Experience (SD = 5.59), and Extraversion (SD = 7.09). Consequently, applicants may have consciously presented themselves more favorably during the interview on the two job-relevant personality traits. This in turn may have reduced the agreement correlations between interviewer ratings and self-applicant ratings. To further investigate this explanation, the interitem variability on each personality dimension was computed by taking the standard deviation across a participant's responses after standardization (Baumeister, 1991; Baumeister & Tice, 1988; Biesanz, West, & Graziano, 1998; Britt, 1993; Chaplin, 1991). To do this, participant responses on each item were converted to z scores using the mean and standard deviation from the full sample for each rater (self, friend, interviewer, and stranger). A standard deviation less than one demonstrates this personality trait has less interitem variability than the average variance found across all items. In the current data we found support for this explanation, as interitem variability scores for interviewer ratings were more consistent for Conscientiousness (SD =.84) and Emotional Stability (SD =.87), and less consistent for Agreeableness (SD =.94), Openness to Experience (SD = 1.07), and Extraversion (SD = 1.15). The small standard deviation across items from these personality constructs provides additional evidence that the applicants may have "managed" their level of Conscientiousness and Emotional Stability during the interview. Effects of Interview Structure and Content Ratings of Personality It is informative to examine the effects interview design (degree of structure and job-related content) have on the interviewer's ability to assess personality traits. Table 3 reports the correlations between interviewer ratings across all five FFM traits with ratings from the applicant (self), the friend, and the stranger for structured and relatively unstructured interviews. As shown, the correlations between ratings from the more structured interviews tended to be larger (f =.27 vs..18 over all correlations for structured interviews vs. unstructured interviews), but none were significantly different from the more unstructured interviews. Nevertheless, the direction of the relations are consistent with our hypotheses. More importantly, for correlations between interviewer ratings and applicant ratings (self), we found the largest differences in correlations (f =.37 vs.,22 over all FFM traits for structured interviews vs. unstructured interview).

16 940 PERSONNEL PSYCHOLOGY TABLE 3 Correlations Between Interviewer Ratings Wtth Ratings From, s, and s for High and Low Structure Interviews (N = 73) Extraversion Agreeableness Conscientiousness Emotional Stability Openness to Experience Number of interviews Low structure interviews Correlation between interviewer ratings and:,30,48,16,33,16,19,06,38 -,08,16,14 -,22,26,30,02 38 High structure interviews Correlation between interviewer ratings and:,50,51,24,35,20,19,26,05,21,20,29,21,42,24,25 35 The second interview design hypothesis (Hypothesis 5) predicted that the interview-personality correlation would be greatest when the interview was based on requirements of the job. As reported in Table 4, contrary to our hypothesis, there was no consistent difference between correlations when using job-related interviews versus nonjobrelated interviews. Across all correlations, the average correlation was quite comparable (f =.20 vs..18). Furthermore, there were no substantial differences in correlations for any specific source (e.g., self, friend, or stranger). Finally, to examine whether the nature of the interview content (situational vs. behavioral interviews) affected the interviewer's ability to assess personality. Table 5 reports the correlations for situational interviews, behavioral interviews, and those interviews that were a combination of both situational and behavioral interviews. As shown, the nature of the interview was not consistently related to differences in correlations. Although the behavioral interview had slightly higher correlations overall (f =.26 for behavioral and f =.19 for both situational and the combined situational/behavioral), most of this difference emerged due to higher correlations with stranger ratings (f =.18 for behavioral, f =.10 for the combined situational/behavioral, and f = -.03

17 MURRAY R, BARRICK ETAL. 941 TABLE4 Correlations Between Interviewer Ratings Wtth Ratings From, s, and s for Job-Related Questions and Nonjob-Specific Interviews (N = 73) Extraversion Agreeableness Conscientiousness Emotional Stability Openness to Experience Number of interviews Job-related interviews Correlation between interviewer ratings and:,44*,50*,22,28*,18,16,12,14,08,19,01 -,04,36*,26*,12 61 Noniob-SDecific interviews Correlation between interviewer ratings and:,32,48,02,54*,15,16,41,52* -,03,11 -,03 -,32,18,23,01 12 for situational interviews). Consequently, interviewers were not able to more effectively assess applicant personality based on any of the interview approaches. Taken together, these results show that interview content, whether operationalized as job-related, situational, or behavioral in nature, did not affect the interviewer's ability to assess the applicant's personality. Effects of Applicants' Personality Traits on Interview Outcomes Another purpose of this study was to examine whether interviewer ratings of applicant personality traits predicted interviewer evaluations of suitability for the organization. Specifically, we were interested in the incremental validity of interviewer ratings of applicant personality over self-ratings. We used hierarchical regression analyses, and regressed the evaluation measure (perceived applicant suitability ratings by interviewer) on several control variables (applicants' gender, GPA, and amount of applicant experience interviewing) and then on the set of personality traits. -ratings of personality were entered immediately after the control variables, followed by interviewer ratings to assess the in-

18 942 PERSOlSfNEL PSYCHOLOGY TABLE 5 Correlations Between Interviewer Ratings Wtth Ratings From, s, and s for Different Content of the Interview (N = 73) Situational interviews Correlation between interviewer ratings and: Behavioral interviews Correlation between interviewer ratings and: Both situational and behavioral Correlation between interviewer ratings and: Extraversion Agreeableness Conscientiousness Emotional Stability Openness to Experience Number of interviews,47,50,16,27,16,12,28,36,07,17,03 -,29,42,37 -,22 25,37,66,21,35,18,17,20,13,01,17,46,08,31,19,41 31,54,07,14,29,22,21 -,02,20,10,19,05 -,01,39,37,07 17 cremental contribution of the interviewer source over self-ratings alone. Finally, all rating sources (self, interviewer, friend, and stranger) were entered together as a block to determine the percentage of variance accounted for by all sources. Results of these regression analyses are shown in Tkble 6. Analyses indicated the control variables (gender, GPA, and applicant interviewing experience) bore little relationship to interviewer ratings of perceived suitability (Adj. R"^ =.005). After controlling for these variables, the set of applicant personality traits significantly increased (A Adj. R'^ =.055) for interviewer ratings of applicant suitability. Extraversion (/? =.26) and Emotional Stability (/3 =.23) were strongly (positively) related to these evaluations. Interviewer ratings of the applicant's personality provided an additional significant incremental contribution (A Adj. i?2 = 249) to the explanation of interviewer ratings of applicant suitability. Interviewer ratings of the applicant's Conscientiousness (0 =.31) and Openness to Experience (/3 =.30) provided the greatest

19 MURRAY R, BARRICK ET AL. 943 TABLE 6 Results of Hierarchical Regression Analyses for the Five-Factor Model ThaUs (N = 73) Step 1: Applicant controls Sex GPA Experience Step 2: Applicant ratings of: Extraversion Agreeableness Conscientiousness Emotional Stability Openness to Experience Step 3: Interviewer ratings of: Extraversion Agreeableness Conscientiousness Emotional Stability Openness to Experience Step 4: All variables Extraversion Agreeableness Conscientiousness Emotional Stability Openness to Experience Extraversion Agreeableness Conscientiousness Emotional Stability Openness to Experience Interviewer ratines of applicant suitabilitv 0 Adj, fi= A Adj, R? -0,10 0,05 0,17 0,26* -0,10 0,00 0,23* -0,09 0,12-0,09 0,31* 0,06 0,30* -,02 0,04-0,11 0,27* -0,09 0,18 0,15-0,13 0,18-0,05,005,060*,304*,343*,055*,249*,039 incremental gains in validity in this analysis. Finally, inclusion of ratings of applicant personality from all four rating sources increased explained additional variance (A Adj. R^ =.039), with friend ratings of the applicant's Emotional Stability providing the largest incremental gain (0 =.27). Discussion Interviews are the most commonly used selection practice. Yet research has just begun to unravel what constructs are assessed in a typical interview (e.g., Huffcutt et al., 1996). The major finding of this study with regard to personality assessment is that although the interview can effectively measure some personality traits (e.g., Extraversion, Agreeableness, and Openness to Experience), the two traits most closely as-

20 944 PERSONNEL PSYCHOLOGY sociated with job performance across jobs (Conscientiousness and Emotional Stability; Barrick & Mount, 1991; Barrick et al., in press; Hough, 1992; Salgado, 1997) are the two most difficult for an interviewer to identify. These findings challenge the notion that interview validity is largely due to its ability to assess the candidate's personality. Our results also illustrate that the design of the interview does not substantially affect the interviewer's ability to assess personality traits. For example, using job-related questions did not increase the correlations between interviewer assessments and applicant ratings (f =.28 for job-related interviews and f =,31 for those interviews not tied to job requirements). Similarly, the use of behavioral interviews (pastoriented questions) or situational interviews (future-oriented) did not substantially increase these correlations (f =.26 or.19, respectively). The largest effect of interview design on personality assessment was attributed to the use of more structured interviews. Even here, however, the correlational gains between interviewer ratings and other ratings (self, friend, and stranger) were modest (f =.27 with high structure, f =.18 with low structure). Taken together, these findings further question the utility of using interviews to assess personality traits. Nevertheless, this study does illustrate "how well" the typical interview assesses personality. Our first hypothesis was that an interviewer's rating of applicant personalify would correlate more with the applicant's self-rating than a stranger's would, but less than a close friend's ratings. In every case except one, our results support this hypothesis. The only exception was the correlation between friend and applicant ratings of Extraversion (r =.41), which is nearly equal to the correlation between interviewer and applicant ratings (r =.42). These findings suggest that the interviewer is assessing personalify during the employment interview. Looking at the correlations of applicant personalify for interviewer-friend and comparing them to interviewer-stranger strengthens this finding. In every situation, the correlation between the interviewer and friend is higher then the correlation between the interviewer and stranger. This suggests that during a 30-minute interview, an interviewer is actively and effectively moving away from "stranger" status toward "friend" status as it relates to an evaluation of the applicant's personalify. This finding supports the assumption that the interview provides an opportunify to "get to know" the applicant. The second hypothesis suggested that due to observabilify, interviewers will be able to identify Extraversion and Conscientiousness more accurately (i.e., higher correlation with the applicant's self-rating) then Emotional Stability. The data suggests support for two aspects of this hypothesis. Interviewers were able to assess Extraversion quite well (r =.42), and were less effective assessing Emotional Stabilify (r =.17)

21 MURRAY R. BARRICK ET AL, 945 as hypothesized. However, contrary to our hypothesis, interviewers were also generally unable to accurately identify the applicant's level of Conscientiousness (r =.16). This result is similar to the correlation between self-reports of conscientiousness and structured interviews (p =.26), reported by Cortina et al. (2000). The implication for practitioners is the two traits that consistently increase predictive validify (Conscientiousness and Emotional Stabilify) are the two most difficult traits to assess through an interview at least in settings where the interview was not explicitly designed to assess personalify. Thus, the argument that the inclusion of both an interview and a personalify assessment in a selection system is unnecessary and redundant is not supported by these findings. One explanation for these findings is that an interview is simply too short a period of time to observe and correctly evaluate Conscientiousness and Emotional Stabilify. This explanation is supported by the evidence for interrater agreement in assessing a subject's level of Emotional Stabilify, where stranger ratings have notoriously low correlations with self-ratings (r =.05; Connolly & Viswesvaran, 1998). However, this is contradicted by our correlation for Conscientiousness, which was lower than the moderate correlation typically found for stranger ratings (f =.23; Connolly & Viswesvaran, 1998). In fact, the evidence in this study supports the conclusion that applicants actively try to manage their impressions during the interview. This is supported by the relatively small standard deviations and interitem variabilify indices found for interviewer radngs of these traits. From a practical perspective, it seems likely that applicants are motivated to present themselves as dependable, hardworking, and persistent (high in Conscientiousness) and calm, stable, and confident (high in Emotional Stabilify). Stevens and Kristof (1995) reported that applicants used more self-promotion than ingratiation tactics. This is consistent with applicants promoting themselves as more Conscientious and Emotionally Stable. Thus, to the extent applicants can manage their impression during the interview, the interview will not be an effective method for assessing performance-related personalify. There are those who would suggest that through training an organization could increase an interviewer's abilify to assess these traits (Conscientiousness and Emotional Stabilify) more effectively. However, to the extent applicants monitor their self-presentation, accuracy may not be corrected with training. That is, given the strength of the interview situation and the training available to applicants, an interviewee may be able to script responses to create a conscientious, calm, and competent self-presentation. Consequently, it may be that for interviewers to assess the traits of Conscientiousness and Emotional Stability more accurately.

22 946 PERSONNEL PSYCHOLOGY interviews may need to be longer and include events intentionally placed to break up scripted behavior. More important, our results suggest that personalify assessment is not likely to improve dramatically by simply redesigning the interview. As stated earlier, our findings show that using job-related questions, behavioral questions, or situational questions does not improve personalify assessment. Although we found more structured interviews did enable interviewers to assess personalify more accurately, these gains were not large. These results do, however, dispel the notion that interviewer-self personalify correlations would be higher in unstructured interviews. It should be noted, our study does not test the effect of asking job-related questions that are specifically designed to assess personalify. Future research must examine whether the interview can be designed to assess personalify directly, and what the gains to predictive validify are by doing so. Another hypothesis examined whether a relationship exists between interviewer assessment of applicant personalify and interviewer assessment of applicant suitabilify with the organization. As indicated previously, through training and the relatively structured nature of the interview, applicants may be able to tailor their self-presentation. The assumption here is that those traits valued by an organization will be those things the applicant will make a special effort to display. For example, if the organization values innovation rather than status quo performance, an applicant may choose to present the aspects of intellect, creativify, and inquiry rather then his or her desire for comfort and stabilify (assuming the applicant desires employment with that company). Results from this study suggest that after controlling for gender, GPA, interview experience, and self-assessed personalify, assessment of applicant personalify by the interviewer added an additional.25 to the variance explained in ratings for applicant suitabilify, and thus supports our hypothesis. Although the hypothesis is supported, interpretadon of these results is a bit problematic. Our current design does not allow us to idendfy if applicant suitabilify caused the positive assessment of personalify by the interviewer, or if an accurate assessment of an applicant's personality predicts the applicant's suitabilify. However, it does show that the interviewer's assessment of applicant personalify is related to the interviewer's assessment of applicant suitabilify. Given that both assessments are taking place simultaneously during the interview, it is difficult to identify a research design that would aid in identifying the causalify of this relationship. One other observation regarding the finding that ratings of applicant personalify are related to interviewer ratings of applicant suitabilify. In Step 2 of the hierarchical regression reported in Table 6, we show that