Square pegs in round holes: has psychometric testing a place in choosing a surgical career? A preliminary report of work in progress

Transcription

1 Ann R Coll Surg Engl 1999; 81: Square pegs in round holes: has psychometric testing a place in choosing a surgical career? A preliminary report of work in progress J Herman Gilligan BA Management Consultant Fenella K S Welsh FRCS Colin Watts MSc Director of Human Resources Tom Treasure MD MS FRCS SHO in General Surgery Professor of Cardiothoracic Surgery St George's Hospital, London Key words: Surgical training; Career choice; Psychometrics Methods of selection of candidates for training in surgery has long been regarded as lacking explicit criteria and objectivity. Our purpose was to discover the aptitudes and personality types of applicants for surgical posts at the outset, in order to discover which were most likely to result in a satisfactory progression through training and which were associated with career difficulties. This longitudinal predictive validation study has been undertaken in a London Teaching Hospital since After short-listing, but immediately before interview, all candidates for senior house officer posts in basic surgical training and in geriatric medicine were asked to undertake psychometric tests of numerical (GMA) and spatial (SIT7) reasoning, personality type (MBTI), and selfrating of competency. There were no differences in ability scores between surgeons or geriatricians. Personality differences were revealed between the surgeons and the geriatricians, and between male and female surgeons. This study suggests that while there are no differences in ability between surgeons and geriatricians at the start of training, there are differences in personality. Long-term follow-up of the career development of this cohort of surgical SHOs is required to determine whether the psychometric measures described correlate with achievements of milestones in their surgical careers. Correspondence to: Professor Tom Treasure, Professor of Cardiothoracic Surgery, Regional Cardiothoracic Unit, St George's Hospital, Blackshaw Road, London SW17 OQT With the introduction of 'Calman', surgical training has undergone considerable change in the past 4 years. Modern surgical training programmes have been designed to provide the trainee with a more focused career path, shorter and more structured training, and earlier specialisation. It is important for both the trainer and the trainee to make the right choice in the initial selection process. In 1987, a symposium at The Royal College of Surgeons of England, on the potential for use of personality assessment and aptitude testing in the selection of surgical trainees, Gough suggested that 'present methods of selection for surgical training depend on the candidate's academic record, the impression given at interview, the references submitted and a combination of opportunity and luck' (1). A recent paper by Lambert et al. (2) suggests that trainees chose careers based on their experience as students, the influence of role models, or 'just a hunch'. Thus, current selection methods may be neither valid nor reliable and, perhaps, there should be more objective assessments which provide better predictors of success in the selection and training of surgeons. Psychometrics are a range of standardised psychological measures of two distinct kinds: on the one hand there are measures which assess general cognitive ability and more specific aptitudes, and on the other are questionnaires relating to personality and work style. Psychometric testing is widely used in selection of personnel in managerial and professional groups (3) and is now an established component in the selection and development of managers within the NHS (4). The

2 74 H Gilligan et al. Association of Surgeons of the Netherlands has incorporated psychometric testing in the selection of surgical trainees since 1983 (1), although it does not appear that its effectiveness has been evaluated. The purpose of this study is not to reject applicants from surgical training on the basis of some template of the ideal surgeon. This would be unacceptably prescriptive. However, if we could identify those for whom training would present some difficulties, we could help their career development in a constructive way. We chose to measure numerical and spatial reasoning, and identify personality type preferences in prospective basic surgical trainees. To provide a reference point, we invited contemporaneous cohorts of applicants for SHO posts in geriatric medicine to act as a comparison group. It is further aimed to follow the career development of these candidates and to establish whether these measures have any predictive value in achievement of the accepted milestones of progress in a surgical career. The study was set up in 1994, and it will be some years before the questions outlined above can be answered. However, in this initial report, we have aimed to establish baseline comparisons of numerical and spatial reasoning ability and personality preferences between the surgeon and the physician at the commencement of training. Methods Study design Ethical approval was obtained from the Ethics Committee of St George's Hospital, according to the Helsinki agreement. The study was designed as a longitudinal predictive validation study. Candidates short-listed for basic surgical training posts and for SHO posts in geriatric medicine were invited to undertake two ability tests (numerical and spatial reasoning) and complete a personality measure and a self-rating questionnaire on the day of interview. They were assured that this was a research project independent of the selection procedure, and that the selection committee was blinded to the results of the tests. All candidates received written feedback on their results and personality profiles. Further verbal feedback was available on request. Psychometric testing The following psychometric measures were used. GMA numerical reasoning test (5) This is a 30-min timed test of the ability to reason using numbers. Saville and Holdsworth SIT7 test (6) This is a 20-min timed test of spatial reasoning, in which candidates are asked to select which three-dimensional object would be formed by folding and manipulating a two-dimensional pattern. Numerical and spatial reasoning scores generated in the above two tests were expressed both as a raw score, and as a percentile relative to scores from current UK university graduates. That is, a score that was on the 50% percentile was equivalent to the average score obtained by UK graduates. Myers-Briggs Type Indicators' (7) The Myers-Briggs Type Indicator (MBTI) is a 126 item questionnaire based on the psychological type theory of Carl Jung. It provides an indicator of an individual's broad personality preferences. These preferences are combined into four dimensions, each dimension consisting of two polar preferences (Table I). It is the combination of candidates' preferences across these four dimensions that produces their Myers-Briggs type profile. Candidates self-rating questionnaire Candidates were asked to rate themselves on 11 dimensions of competency, using a 1-10 scale (Table II). These factors have been previously identified as being key characteristics associated with effective performance in surgery (8). Statistical analysis Data are expressed as medians, with interquartile range in parentheses. Statistical analysis was performed using Statview software (Apple Computers Inc, Abacus Table I. Myers-Briggs Type Indicator Dimension Polar preferences Energising Extroversion Introversion Energised by the external world of people/ Energised by the internal world of ideas/ activities/things emotions/impressions Attending Sensing Intuition Focused on what is actual Focused on what might be Deciding Thinking Feeling Logical/objective/detached Value-oriented/subjective/personal Living J7udging Perceiving Planned/organised Spontaneous/flexible

3 Table II. Dimensions of self-rating Interpersonal skills Communication skills Responsibility and leadership skills Evaluation and analytical skills Broad and balanced perspective Decision-making skills Personal organisation skills Stress tolerance Self-motivation Political awareness Self-insight and integrity Concepts Inc, Berkeley, CA). Non-parametric analysis was performed using the Mann-Whitney U test. Analysis of categorical variables was performed using Fisher's exact test. A value of P < 0.05 was considered significant. Results Demographics In all, 159 senior house officers, 133 surgical and 26 geriatric, were recruited to the study between 1994 and The median age of the surgeons was 26 years (range years), significantly greater than that of the geriatricians (25 years (range years), P<0.001). In addition, there was a significantly greater percentage of men in the surgical group (80%) compared with the geriatricians (24%, P< 0.01 Fisher's exact test). Numerical reasoning scores The median score in the surgical trainees' group was 13 (range 9-16). This was not significantly different from that in the geriatrics group (12 (range 10-13)). When results were expressed as a percentile, in comparison to UK graduates (median score 50%), the median scores were 45% (range 15-70%) for the surgeons and 35% (range 20-48%) for the geriatricians, that is just below average compared with current UK graduates. Furthermore, there was no significant difference in median score between those surgical applicants appointed (13 (range 10-17)), and those who were not (13 (range 8-16)). Spatial reasoning scores The median spatial reasoning score in the surgical trainees group was 19 (range 14-23) and this was not significantly different from that in the geriatrics group (17 (range 14-20)). Expressed as a percentile, the median scores were 58% (range 27-79%) for the surgeons and 46% (range 26-62%) for the geriatricians, the surgeons thus being above average compared with current UK graduates. In addition, there was no significant difference in median score between those surgical applicants appointed (18 (range 14-23)), and those who were not (20 (range 15-23)). Psychometric testing in choosing a surgical career 75 Personality preferences The distribution of personality preferences is shown in Fig. 1. Both surgeons and geriatricians showed similar preferences for 'Extroversion', and also for preferring a more planned and organised ('Judging') lifestyle. In addition, the surgeons and geriatricians were both evenly distributed across the dimension as to whether they paid attention to what was actual ('Sensing') versus which might be ('Intuition'). However, a significantly higher percentage of surgical candidates (85%) expressed a preference for making decisions in a detached, logical and objective manner ('Thinking'), compared with the geriatrics candidates (43%) (P < 0.01, Fisher's exact test). The distribution of personality types measured using the MBTI is illustrated in Fig. 2. It shows that the distribution is skewed in the surgical trainees, with 64% of surgeons preferring one of four personality types (ESTJ, ENTJ, ISTJ, ENTP), compared with the geriatricians who were more evenly distributed among all personality types. We asked whether these differences in personality preference could be explained by the significant gender differences between the groups. The distribution of personality preferences in male and female surgeons is shown in Fig. 3, and there were no significant differences between gender (Fisher's exact test). Again, both male and female surgeons showed a marked preference for making decisions in a detached, logical and objective manner ('Thinking'). The distribution of personality types in male and female surgeons measured using the MBTI, is shown in Fig. 4. This distribution is again skewed, reflecting the distribution in all surgical trainees and shows that 62% of male and 70% of female surgeons preferred one of four personality types (ESTJ, ENTJ, ISTJ, ENTP). 80. %of Trainees o0 Dimensions F P..1 JE E-Extroversion vs I-Introversion S-Sensing vs N-Intuition *T.Thinking vs F-Feeling, P<0.01 vs geriatricians J-Judging vs P-Perceiving Figure 1. Distribution of Myer-Briggs personality preferences across the four dimensions in surgical (hatched) and geriatrics (stippled) trainees.

4 - : _', ^ Gb: i - J 76 J H Gilligan et al. ESTJ zzi ENT..._ :::: :;:. ;itj o ISTJ ENTP INTJ INTP Distribution of MBTI ESTP ENFJ 2;3mm ISTP INFJ INFP ESFP ENFP -7,77. ISFJ ESFJ 'SFP TwAr---- I I 1 I r 0 5 to % Trainees Figure 2. Percentage distribution of Myers-Briggs Type Indicator in surgical (hatched) and geriatric (stippled) candidates. Self-rating The results of the self-rating analysis are shown in Table III. The surgeons rated themselves significantly higher than the geriatricians in five areas: responsibility and leadership skills, evaluation and analytical skills, decision 'N.80 %of Surgical 60v Trainees ), P.R. c-fl LiC IZ n---: i U.1 Al-C::@ E-Extroversion vs I-Introversion S-Sensing vs N-Intuition T-Thinking vs F-Feeling J-Judging vs P-Perceiving Figure 3. Distribution of Myers-Briggs personality preferences across the four dimensions in male (hatched) and female (stippled) surgical trainees. ENTJ ESTJ i@s : w,.-: e ; -.' z I ISTJ ENTP JNTJ INTP Distibion of MBTI ESTP ENFJ ISTP INFJ INFP ESFP ENFP ISFJ ESFJ ISFP I F I~ ~ I I I I' % Trainees Figure 4. Percentage distribution of Myers-Briggs Type Indicator in male (hatched) and female (stippled) surgical candidates. making, stress tolerance and self-motivation. In no area did the surgeons rate themselves lower than the geriatricians. There were no significant differences between those surgeons appointed to post and those not. When the self-rating scores of the surgical candidates were analysed for possible sex differences, the female applicants scored themselves significantly higher than the males with regard to their organisation skills (9 (range 8-10) vs 8 (range 7-9), P = 0.005), and in no area did female surgical applicants score themselves lower than the male surgical applicants. We also asked whether the gender differences in self-rating were reflected in different success rates in terms of appointment: 63% of female surgical trainees were appointed to post, compared with 39% of male surgical trainees, but this failed to reach statistical significance. Discussion I i I: ; :...'.: ;..' 1.1-: :.!... The current methods used in the selection of surgical trainees are questionable in predicting career success. This problem is illustrated in a recent paper which asked whether admission variables on 39 general surgical residents could be used to predict success during residency (9). These variables included the resident's National Resident Matching Programme Rank, academic record, research activity and publications, hobbies and

5 Psychometric testing in choosing a surgical career 77 Table III. Results of candidates' self-assessment rating* Surgeons All Dimension candidates Appointed Not appointed Geriatricians Interpersonal skills 8 (7-9) 8 (7-8) 8 (8-9) 8 (8-9) Communication skills 8 (7-9) 8 (8-9) 8 (7-9) 8 (7-8) Responsibility/leadership skills 8 (7-9)t 8 (8-9) 8 (7-9) 7 (5-8) Evaluation/analytical skills 8 (6-9)t 8 (7-9) 8 (6-8) 7 (6-8) Broad/balanced perspective 8 (7-8) 8 (7-8) 8 (6-8) 8 (7-8) Decision-making skills 8 (7-9)t 8 (7-9) 8 (7-9) 6 (6-8) Personal organisation skills 8 (8-9) 8 (8-9) 8 (7-9) 8 (7-9) Stress tolerance 8 (6-8)t 8 (6-8) 8 (7-9) 6 (5-7) Self-motivation 9 (8-9)t 9 (8-10) 9 (8-9) 8 (6-8) Political awareness 8 (7-9) 8 (7-9) 8 (7-9) 8 (6-8) Self-insight/integrity 8 (8-9) 8 (8-9) 9 (8-9) 8 (7-9) *Results are expressed as median score with interquartile range in parentheses tp <0.001, tp <0.02 vs Geriatricians, Mann-Whitney U test sports. The authors found no relationship between admission data and the resident's final evaluation. They suggested this may be because the candidates selected on to the residency programme were a relatively homogeneous group and, perhaps more significantly, qualities such as interpersonal skills, character and technical ability, which were important in the final evaluation, were difficult to assess at admission. Thus, there is increasing interest in the use of ability and aptitude testing as a tool to aid the selection and career development of surgical trainees (10). Such psychometric tests have been used for a number of years in dentistry, a profession which, like surgery, requires a combination of practical and intellectual ability. Fifty years ago, the American Dental Association began attempts to forecast undergraduate success. It devised the Dental Aptitude Test (DAT) (later renamed the Dental Admission Test) which became a compulsory part of the selection process of North American dentists as early as 1951 (11). As well as assessing intellectual ability, it tested candidates' ability to visualise patterns and relationships and to use their hands and fingers dextrously. It has been validated in reducing student drop-out rates; however, its value in assessing dexterity is questionable (12). Spatial reasoning has been identified as being predictive of undergraduate success in dentistry (13). Furthermore, while the results of spatial reasoning tests performed by applicants to dental school bear little relation to GCE 'A' level grades, poor test scores strongly correlate with dropping out or delayed graduation (14). This is supported by a study commissioned by The Royal College of Surgeons of England of over 100 surgeons, which concluded that spatial reasoning was the single most useful predictor of performance in surgery (15). Thus, it is somewhat surprising that in this study we have shown no significant differences in spatial reasoning scores between surgical trainees and geriatrics trainees. However, surgeons' scores were slightly above average compared with a reference population of UK graduates. Spatial reasoning was not predictive of success in the surgical trainees in terms of appointment to post. Similarly, our study revealed no differences in numerical reasoning scores between the surgeons and geriatricians, and the surgeons who were or were not appointed to post. Moreover, both groups scored slightly below average compared with the reference graduate population. These data are supported by work from Harris et al. (16) who found no demonstrable psychomotor differences between trainee surgeons and psychiatrists, anaesthetists or physicians. Our current selection techniques do not appear to be selecting a group of SHOs who are measurably better at the onset of training, but there is a wide range of scores within the surgical group, and it remains to be seen whether those with certain aptitudes have a more successful career. In a study of 22 surgical trainees, Deary et al. (17) examined the relationship between trainees' scores in aptitude tests and ability ratings by their consultant (trainer). They found no significant correlation between ability ratings and intelligence test scores. However, those trainees rated as superior tended to have better stereoscopic depth perception. The authors concluded that there was little role for aptitude testing in the selection of surgeons until there were more objective criteria of surgical ability in different surgical specialties. A recent study addresses this by using the rating of performance on a surgical training exercise as a more objective measure of ability (18). The performance of surgical trainees attending a microsurgery course was scored and correlated with a test of dexterity, and a test of spatial ability. A significant correlation was found between surgical performance and both manual dexterity and spatial ability. Moreover, it was suggested that the

6 78 J H Gilligan et al. test of dexterity used (the wire-loop test) might be particularly discriminating in identifying those individuals unsuited for a career in microsurgery, because although proficiency at this test could be improved upon with practice, some individuals who lacked stereoscopic vision are unable to perform the test even after practice. This is of interest because it has been argued elsewhere that dexterity is not an important dimension of surgical skill (17), and can be improved with practice (19). Our investigation has identified significant differences in personality type between the surgical and geriatrics trainees, according to the Myers-Briggs Type Indicator. The surgical trainees showed a marked preference for 'Thinking', ie making decisions in a logical and objective fashion, rather than for 'Feeling', a preference for subjectivity and for more people-orientated concerns. This is in contrast to the geriatricians, who did not show any preference bias across this personality dimension. Moreover, there was a marked skew in general personality types measured in the surgeons, compared with the geriatricians. Is this evidence that surgeons conform to a 'type' and, if so, what are the consequences of not conforming to type? The MBTI may be used to identify the different work and management styles which people adopt, on the assumption that people function best when they are able to express their own preferences. It is suggested that when an individual is forced to use a style for a long time that does not utilise their own preferences, loss of comfort and satisfaction, inefficiency, and in extremis 'burn-out' may well result. If future work identifies particular personality preferences which correlate with failure to achieve surgical milestones such as passing examinations, obtaining an HST post, or dropping out of a surgical career, this may enable selectors to counsel such individuals as to the suitability of a surgical career for them. The potential savings in terms of personal and financial costs, not only to the individual, but also to the trainers, the National Health Service and ultimately the State, are considerable. The surgeons also differed from geriatricians in their self-rating, rating themselves higher overall than the geriatricians, and significantly better in terms of leadership, analytical and decision-making skills, stress tolerance and self-motivation. Such personality traits are important to the surgeon, and in a study of surgical residents, Schueneman et al. (20) showed that stress tolerance was the personality factor most predictive of operative skill. Deary et al. (17) demonstrated that those trainees rated as superior by their trainers tended to be more introverted and conscientious. The present study also revealed demographic differences between the surgeons and the geriatricians. The number of geriatricians in the study was small, but the male to female ratio was 1:3, in contrast with that of the surgeons, which was 4:1. These data reflect the career choices of medical graduates as a whole, the most recent data being from doctors who qualified in the UK in 1993; of those doctors who chose a career in surgery (16.9%), three times as many were male. Similarly, three times as many of those who chose a career in geriatrics (0.9%/), were female. The geriatricians in our study were also younger than the surgeons, perhaps reflecting the traditional tendency for surgical trainees to spend the first SHO year in casualty/anatomy demonstrator posts. However, we do not intend to read too much into the differences between the surgeons and geriatricians, but they do provide a comparison group. Differences in personality preferences demonstrated may simply have been owing to the preponderance of females in the geriatrics group; however, examination of the surgical trainees revealed no differences in numerical or spatial reasoning or personality preference between male or female surgeons. In terms of self-rating, in no parameter did female surgeons rate themselves lower than males and, moreover, they rated themselves higher than males in terms of organisational skills. In conclusion, this study suggests that while there may be no differences in ability between surgeons and geriatricians at the start of training, there are differences in personality preferences and work style. Long-term follow-up of the career development of this cohort of surgical SHOs is required to determine whether the psychometric measures described correlate with success in their surgical careers. It is intended to follow their career course, using objective outcome measures such as examination success, obtaining satisfactory appraisals, moving into a HST post, gaining a CCST, and eventually a consultant post in surgery. Any pattern of results that would predict failure to achieve these milestones might be of use in guiding the career choice of prospective surgical trainees. We are grateful to Professor Millard and his colleagues in geriatric medicine for allowing us to study their candidates. We appreciate the help of Catherine Berwick, Human Resources Department, St George's Hospital, who supervised some of the tests. We are, of course, very grateful to the large numbers of juniors who agreed to undergo these tests, and at a time of great stress and anxiety. South West Thames Regional Research and Development funded our initial pilot study into the use of psychometric tests to assist the career development of junior doctors (RDP 14/1993/ G51). See Gilligan et al. (21) for a discussion of our early findings and an overview of the related literature on surgeons and other clinical groups. References 1 Gough MH, Holdsworth R, Bell JA et al. Personality assessment techniques and ability testing as aids to the selection of surgical trainees. Ann R Coll Surg Engl 1988; 70: Lambert TW, Goldacre MJ, Edwards C, Parkhouse J. Career preference of doctors who qualified in the United Kingdom in 1993 compared with those of doctors qualifying in 1974, 1980 and BMJ 1996; 313: Toplis J, Dulewicz V, Fletcher C. Psychological Testing: A Manager's Guide. London: 1PM, Spurgeon P. Psychometric Testing in Selection and Develop-

7 Psychometric testing in choosing a surgical career 79 ment: A Guide for Managers. Bristol: NHS Training Authority, Blinkhorn SF. Graduate and Managerial Assessment: Manual and User's Guide. Published by ASE Windsor, Information Technology Test Series: Manual and User's Guide. Thames Ditton: Saville and Holdsworth, Krebs Hirsh S, Kummerow JM. Introduction to Type in Organisations. 2nd Edition. Oxford: Psychologists Press, Stevenson H, Henley S. 7ob Analysis Report on the Role of the Surgeon. Thames Ditton: Saville and Holdsworth, Papp KK, Polk HC, Richardson D. The relationship between criteria used to select residents and performance during residency. Am J Surg 1997; 173: Graham KS, Deary IJ. A role for aptitude testing in surgery? J R Coll Surg Edinb 1991; 36: The dental aptitude testing programme. J Am Dent Assoc 1950; 41: Spratley MH. Aptitude testing and the selection of dental students. Aust Dent J 1990; 35: Smith BGN. The value of tests of spatial and psychomotor ability in selecting dental students. Br DentJ 1976; 141: Smith BGN. A longitudinal study of the value of a spatial relations test in selecting dental students. Br Dent J 1989; 167: Fitzgerald C. Royal College of Surgeons Validation Study. Thames Ditton: Saville and Holdsworth, Harris CJ, Herbert M, Steele RJC. Psychomotor skills of surgical trainees compared with those of different medical specialists. Br Jf Surg 1994; 81: Deary IJ, Graham KS, Maran AG. Relationships between surgical ability ratings and spatial abilities and personality. Jf R Coll Surg Edinb 1992; 37: Murdoch JR, Bainbridge LC, Fisher SG, Webster MHC. Can a simple test of visual-motor skill predict the performance of microsurgeons? J R Coll Surg Edinb 1994; 39: Barnes RW. Surgical handicraft: teaching and learning surgical skills. Am J Surg 1987; 153: Schueneman AL, Pickelman J, Hesslein R, Freearck RJ. Neuropsychological predictors of operative skill among general surgical residents. Surgery 1984; 96: Gilligan JH, Treasure T, Watts C. Incorporating psychometric measures in selecting and developing surgeons. J Management in Medicine 1996; 10: Received 3 August 1998