The Reliability of Profiling Within the Workplace - A Comparison of Two Personality Measures

The Reliability of Profiling Within the Workplace - A Comparison of Two Personality Measures Geoffrey Chapman*, PhD candidate Centre for Industry and Innovation Studies University of Western Sydney Locked bag 1797 Penrith South 1797 ge.chapman@uws.edu.au

THE RELIABILITY OF PROFILING WITHIN THE WORKPLACE A COMPARISON OF TWO PERSONALITY MEASURES ABSTRACT This study aims to examine the test-retest reliability of the Myers-Briggs Type Indicator and the Instinctive Drives System in order to determine their effectiveness in the workplace. It was hypothesised that the Instinctive Drives System would demonstrate significantly higher test-retest reliability. The study consisted of 50 participants, who completed both personality measures twice. Unlike previous studies which only compare test-retest reliability by examining the averages of the entire sample, this study directly compared changes in individual scores. The Instinctive Drives System was found to be significantly more likely to return reliable results than the Myers-Briggs Type Indicator. The results of this study provide valuable information for business managers interested in improving performance through personality profiling in the workplace. Keywords: Personnel Psychology, Human Resources Management and Organisational Performance, Recruitment, Comparative Human Resource Management BACKGROUND Numerous researchers have commented on the effectiveness of personality measurement within organisational environments (Boyle, 1995; Clack, Allen, Cooper, & Head, 2004; Fitzgerald, Dadich, & Fitzgerald, 2006). The majority of research in this field agrees that, used correctly, an effective personality measurement tool will lead to increases in employee satisfaction, performance, productivity, and general workplace cohesion. The Myers-Briggs Type Indicator (MBTI) is arguably the most widely known and utilised personality measurement tool in organisations today. Developed from Jungian type theory by Katherine Myers and her daughter Isabel Briggs Myers, the MBTI succinctly categorises people into one of sixteen types (Myers, McCaulley, Quenk & Hammer, 1999). However, despite the widespread popularity of the MBTI, many researchers claim that the test lacks reliability and validity, and should be used sparingly, if at all, in organisational environments (McCrae & Costa, 1989; Boyle, 1995; Pittenger, 2005). As evidenced by Fitzgerald, Ferres, Hamilton and Fitzgerald (2005), a number of employees in organisational settings commented that they have completed the MBTI several times and have often received different scores when taking the test again. This diminishes the desired effect of the personality 1

measure, as participants don t believe the feedback applies to them specifically enough, and consequently do not incorporate the feedback from the profiling tool into their everyday work environment. Recognising the need for more effective communication in the workplace, Paul Burgess developed the Instinctive Drives (ID) System (Fitzgerald & Fitzgerald, 2004). This personality measurement tool also utilises four scales, however, unlike the MBTI each scale is measured on a continuum from one to nine. Whilst studies examining the ID System are still in their initial phases, early reports indicate that it is more reliable and valid than the MBTI, and also provides more useful feedback for employees and employers when conducted in the workplace (Fitzgerald & Fitzgerald, 2004; Fitzgerald, Ferres, Hamilton & Fitzgerald, 2005). However, these studies only utilised pre-existing data on the MBTI for the purpose of comparison, not data arising from the participants who completed the ID System questionnaire. The purpose of this exploratory study is to directly compare the two personality tests using the same sample, and therefore obtain valuable knowledge regarding the reliability of the two personality measurement tools. Overview of the Personality Measures Myers-Briggs Type Indicator The MBTI was developed more than 50 years ago using Jung s theory of psychological types, and is arguably the most widespread personality measurement assessment tool in circulation within organisational environments (Myers et al., 1999). Through application of Carl Jung s work from the 1920 s onwards, 4 distinct scales, each with 2 distinct directions, were created to measure a person s type: Extraversion-Introversion (EI), Sensing-Intuition (SN), Thinking-Feeling (TF) and Judging-Perceiving (JP). The MBTI combines these four scales and measures these dimensions using a number of dichotomous multiple choice questions. Whilst this simple and practical feedback has undoubtedly assisted in the MBTI in achieving the popularity it has at present, not all agree that the results participants receive are meaningful. Virtually every aspect of the MBTI has come into question, from its theoretical 2

background and its construction, to its reliability, validity and overall usefulness in organisational environments. Pittenger (1993), is one of the most outspoken critics of the MBTI, and has spent many years analysing its various aspects and applications. His main concern with the MBTI is that it does not measure what it purports to measure, and that Jung s type theory is not being applied correctly, which reflects the comments made in an earlier study by McCrae & Costa (1989). Pittenger (1993) details a number of potential problems, beginning by highlighting the fact that the MBTI does not display the bimodal distribution on each scale, as the theory underlying it suggests it should do. Furthermore, Pittenger (1993) reports that approximately one third of participants changed their type when taking the test for a second time, which brings the reliability of the MBTI into question. The problem of unreliability can be seen as the most concerning for business managers, as managing employees whose personality profile changes every year can be extremely frustrating. The Instinctive Drives System A profiling tool that supposedly lacks this problem of unreliability is the I.D. System, which was developed in 1991 and has been expanding into organisational environments ever since (Burgess, 2007). The present study lacks the scope necessary for examining all aspects of the ID System, and accordingly focuses mainly on the reliability of the two measures. For a more complete description of the ID System, see Burgess (2007). Put briefly, the ID System utilises 4 distinct scales, similar to the MBTI; the Instinct to Verify (Verify), the Instinct to Authenticate (Authenticate), the Instinct to Complete (Complete), and the Instinct to Improvise (Improvise). For a detailed discussion of the possible theoretical underpinnings for the four scales that constitute the ID System, see Fitzgerald et al. (2005). Another similarity to the MBTI is that each of the scales used in the measure has two directions (i.e. a person with a high score on Verify would be said to use the Instinct to Verify, and a person with a low score would be said to avoid the Instinct to Verify ). 3

The ID System does not claim to be a measure of personality, but rather it measures what drives us to be a certain way (Burgess, 2007 p.5). However, both the MBTI and the ID System use preference-based questions (i.e. routine vs. variety) to measure participants on four separate scales. For this reason, and for ease of reading, both the profiling tools will be referred to as measures of personality. The scope of the present study does not allow for a comprehensive discussion regarding the definition of personality, for details on how the concept of personality has changed over the past 40 years, see Dahlstrom (1970), Wiggins and Pincus (1992), and Livesley and Jang (2005). Fitzgerald et al. (2005) report that the test-retest reliability of the ID System is similar to the many of the findings reported for the MBTI. For both instruments, approximately 75% of people score the same preferences when taking the test a second time. However, Fitzgerald et al. (2005) make note of the fact that many of the studies examining this aspect of the MBTI do not address whether the scales in the MBTI questionnaire are being measured as a continuous variable (as the resulting data would suggest) or a dichotomous variable (as the underlying theory would suggest). For this reason, Fitzgerald et al. (2005) consider the test-retest reliability results of the ID System to be statistically more sound than those of the MBTI. Comparing the MBTI and the ID System This study aims to examine the value of both the MBTI and ID System in applied environments. The research presented by Clack et al. (2004) and Fitzgerald et al. (2005) discuss a wide range of benefits associated with successful implementation of personality profiling in the workplace including better communication between employees and employers, better understanding of customer needs and better methods of managing conflict. These benefits combine and result in more productive workers, and eventually, a more successful business (Fitzgerald et al., 2006). It is therefore important to examine the aspects of both the MBTI and the ID System that affect their practical application. The previous research combined with the aims of this study led to the following question: 4

RQ1: What is the difference between the test-retest reliability of the MBTI and the ID System? Examination of existing literature shows that previous studies have almost exclusively measured test-retest reliability by averaging the Pearson correlations demonstrated by the separate scales of the personality measure in question. However, this procedure does not allow for a test to determine whether the difference between the reliability of the two measures is significant. Fitzgerald et al. (2005) also demonstrate that test-retest reliability can be measured via the use of the Intra-Class Correlation (ICC) coefficient. However, this procedure examines only the means of the entire sample, and does not account for individual variations in score. The present study aims to evaluate the test-retest reliability of the two personality measures using a more holistic method, examining the changes between participants results more directly in order to achieve a more meaningful comparison. As the research regarding the test-retest reliability of the MBTI is equivocal at best, and the research on the ID System, whilst limited, suggests that it has significantly high test-retest reliability, the following hypothesis was established: Hypothesis 1) the test-retest reliability of the ID System is significantly different to the test-retest reliability of the MBTI. Participants METHOD A total of 50 participants were used in the study (15 male, 35 female). All participants were university students, but no other demographic information was collected as it was not relevant to the present study. Participants were selected via a convenience sampling technique, and as there were no specific requirements for participation, the first 50 students that agreed to participate were included. 5

Questionnaires The MBTI questionnaire and the ID System questionnaire were used in this study. The MBTI questionnaire has 93 dichotomous questions, and was filled out on paper. The ID System Questionnaire consists of 32 multiple choice questions in which participants rank the four options in order of preference, and was completed on a computer. Procedure In the first round of the study, each participant was required to complete the MBTI and the ID System in the same sitting. The order in which the questionnaires were completed was counter-balanced across all participants, with half receiving the MBTI questionnaire first, and half receiving the ID System questionnaire first. These tests were then scored, but results were not returned to participants at this stage. This was to ensure that knowledge of first round results did not affect the way participants responded in the second phase of the study. The second round of the study was conducted for each participant at least one month (usually 5 to 6 weeks) after they first completed the surveys, and both questionnaires were administered in a similar manner to the first round. Once participants completed the second phase of the study, they received their results from both the first phase and the second phase. All participants were also given a short debriefing by the researcher, which included an explanation of their individual results. Design This study was designed in order to examine the test-retest reliability of the MBTI and the ID System. Accordingly, the study was split into two rounds to facilitate the use of paired-sample t-tests. To investigate the guiding research question, all data analysis was carried out by comparing the results of the first phase to the results of the second phase. Reliability scores of each scale was compared first, then the raw data from the MBTI questionnaire was scaled into a form identical to the form of the raw data 6

from the ID System. The practice of collapsing raw data and measuring the MBTI as a continuous variable has been debated in the literature, but several authors (McCrae and Costa, 1989; Furnham et al., 2003) suggest that this practice is not only acceptable, but also allows for greater meaning to be gained from the results. A t-test was used to analyse whether or not there was a significant difference between the difference scores for each personality measure. RESULTS Assumptions for test-retest reliability analysis were met, including normality, linearity, and singularity. Eight participants had to be removed from the analysis as they received an additional intervention between the first and second rounds which was designed to alter their scores on the I.D. System Questionnaire. This procedure is part of the I.D. System and consists of a phone interview with participants whose I.D. Profile does not display any strong drives. As these participants are expected to display a significant change in their I.D. scores, they could not be included in the analysis. No outliers were identified in the data and the sample size was judged to be adequate for test-retest analysis according to the criteria suggested by Tabachnick and Fidell (2005). Test-Retest Analysis A total of 42 participants were used for the reliability analysis. The test-retest reliability for each of the four scales in each of the two tests was measured first. All of the scales demonstrated satisfactory levels of reliability, ranging from r =.679 to r =.918. The test-retest reliability of all scales was significant at p<.001. This simply means that if someone is high (or low) in one of the scales present in the two measures, they are more likely to get the same score when taking the tests again. However, this information does not answer the question of whether or not a person is more or less likely to get the same score across all four of the scales present in each personality measure. 7

When combining the four scales and examining the differences between the I.D. System and the MBT, the MBTI was found to have a higher total difference (M = 3.62, SD = 2.26) than the ID System (M = 2.88, SD = 1.90). A paired samples t-test demonstrated this difference to be significant, (N=42) t(41)=2.321 p =.025. The first hypothesis was therefore supported. Complete details of the means, standard deviations and t-test results are presented in Table 1. These results indicate that a person s MBTI score is more likely to change the second time they take the test by an amount which will affect the personality type which they are given. Table 1: Means, Standard Deviations, and t-test Results for Total Change Between Test and Retest for the MBTI and the ID System Mean N Std. Std. 95% confidence interval t df Sig. Dev Error of the difference Mean Lower Upper MBTI 3.62 42 2.26 0.35 ID System 2.88 42 1.90 0.29 t-test 0.74 42 2.06 0.32 0.10 1.38 2.32 41.025 DISCUSSION The present study was designed to determine the relative test-retest reliability of both the MBTI and the I.D System. In regards to the research question posed, a significant difference was found between the test-retest reliability of the MBTI, and the test-retest reliability of the ID System. The null hypothesis that the test-retest reliability of the ID System would not be significantly different to the testretest reliability of the MBTI was rejected. Furthermore, there was a clear direction indicated by the results which demonstrated that the test-retest reliability of the ID System was significantly higher than the test-retest reliability of the MBTI. Whilst all 4 scales in both tests had significant reliability in 8

isolation, direct comparison of first and second round scores demonstrated that scores on the ID System were significantly more stable. These findings have many implications for both application and prospective future research, particularly for managers within organisational environments looking to implement a profiling tool within their business. The Test-Retest Reliability of the MBTI and ID System The analysis used for this aspect of the study was useful in exploring the research question posed. The results of the test-retest reliability analysis suggest that participants are significantly more likely to achieve a similar score when retaking the ID System questionnaire than when retaking the MBTI questionnaire. This finding was achieved through direct examination of individual differences between first and second round in both of the personality tests. This approach was taken as the methodology used in existing literature would not have been as useful in terms of exploring the second research question posed by the present study. The overall goal of profiling tools within the workplace is to increase communication, morale, cohesion, and ultimately increase productivity. If a profiling tool does not give reliable results to employees, this goal is seldom achieved. Naturally then, managers should look for a profiling tool which offers a high level of reliability, however, there are many profiling tools which claim to have this. This study provides a clear comparison of two existing profiling tools in regards to test-retest reliability. Whilst traditional methods of testing reliability would suggest that both measures have sufficient reliability, the direct comparison of the two measures shows the ID System to be significantly more reliable than the MBTI. Limitations and Delimitations As previously mentioned, much of the research in this area is still being conducted, and further studies will undoubtedly shed more light on the two measures examined in this study. Specifically, studies 9

examining performance as a direct result of implementation of the personality measures need to be carried out before a conclusive statement can be made regarding the relative value of the tool in organisational settings. A potential confounding variable for this study was the lack of any control over the actions of participants in the time between test and retest. Whilst this is arguably the point of test-retest studies, no qualitative information was gathered in regards to how participants may have incorporated changes into their lives that could affect their personality scores. As both the MBTI and the ID System purport to give stable results across the life span, this confound was not considered substantial enough to control for, however, future research could be directed at gathering information from participants regarding life changes to determine whether these changes are significantly affecting their scores on these measures. Additionally, due to the time constraints of this study, several delimitations were incorporated in order to focus the study specifically on the research question posed. No analysis was made regarding any demographic information, such as gender, age, culture, religion, education. Numerous other studies do examine these factors (Furnham et al., 2003; Clack et al., 2004), however further research into many of these factors would still make for highly interesting and informative results. Implications and Considerations for Future Research This finding has many implications, particularly for the implementation of personality profiling tools in organisational settings. Combined with the previously presented research (Fitzgerald et al., 2005; Fitzgerald et al., 2006), the findings of this study suggest that the ID System is a more applicable, and more reliable tool for implementation into organisational environments. As stability is one of the most important factors for employers considering the use of a personality measurement tool, the implications of this finding are readily apparent. Future studies could utilise the findings of the test-retest reliability analysis as a starting point for research into how applicable the personality tests are in the applied environment. Whilst research in this 10

area is predominantly qualitative, there is potential for a quantitative study that examines team performance to be conducted, using different personality tests as levels of an independent variable, and measures of performance and efficiency as dependant variables. The results of such a study, combined with results of the present study and existing qualitative research could play a large role in the widespread implementation of a personality tool that is significantly more reliable than other personality tools, and may also result in a higher level of organisational performance improvement. Conclusion The present study examined the relative test-retest reliability of the MBTI and the ID System. From this examination, the ID System was demonstrated to have significantly higher test-retest reliability than the MBTI as hypothesised. Whilst there were several limitations of the present study, including the relatively limited existing literature regarding the ID System, the results still provide valuable information for the field of personality psychology. Whilst many more studies are required before conclusive statements can be made about any of the concepts discussed in the present study, the results provide an interesting and promising starting point for future research. 11

REFERENCES Burgess, P. (2007). Natural born success: discover the instinctive drives that make you tick, Milton: Australia: Wrightbooks. Clack, G. B., Allen, J., Cooper, D. and Head, J. O. (2004). Personality differences between doctors and their patients: implications for the teaching of communication skills, Medical Education, 38, 177-186. Dahlstrom, G. W. (1970). Personality, Annual Review of Psychology, 21, 1-48. Fitzgerald, J. and Fitzgerald, J. A. (2004). What makes students tick: The Instinctive Drives System as a means to improve study group performance, Paper presented at the 4th IERA (Australia and New Zealand) Teaching, Learning and Research Conference. Fitzgerald, J. A., Ferres, N., Hamilton, K. and Fitzgerald, J. (2005). What is Instinctive Drive : A report on the validation process of the ID system used to improve team performance, Paper presented at the Australian Business and Behavioural Sciences Association Annual Conference 2005. Fitzgerald, J. A., Dadich, A. and Fitzgerald, J. (2006). What can the Instinctive Drives System offer the workplace: A qualitative exploration, Journal of Business Systems Governance & Ethics, 1, 29-46. Furnham, A., Moutafi, J. and Crump, J. (2003). The relationship between the revised Neo-Personality Inventory and the Myers-Briggs Type Indicator, Social Behaviour and Personality, 31, 577-584. Humphries-Wadsworth, T. M. (1998). Features of published analyses of canonical results, Paper presented at the Annual Meeting of the American Educational Research Association 2005. Livesley, W. J. and Jang, K. J. (2005). Differentiating normal, abnormal and disordered personality, European Journal of Personality, 19, 257-268. McCrae, R. R. and Costa, P. T. Jr. (1989). Reinterpreting the Myers-Briggs Type Indicator from the perspective of the five-factor model of personality, Journal of Personality, 57, 17-40. Myers, I. B., McCaulley, M. H., Quenk, N. L. and Hammer, A. L. (1999). MBTI Manual: a guide to the development and use of the Myers-Briggs Type Indicator (3rd ed.), Melbourne: Australia: Australian Council for Educational Research. Pittenger, D. J. (1993). Measuring the MBTI and coming up short. Journal of Career Planning and Employment, 54, 48-52. Tabachnick, B. G. and Fidell, L. S. (2005). Using multivariate statistics (5th ed.), Boston: Allyn and Bacon. Wiggins, J. S. and Pincus, A. L. (1992). Personality: Structure and assessment, Annual Review of Psychology, 43, 473-504. 12