Validating a Self-Confidence Scale for Surgical Trainees

Transcription

1 Validating a Self-Confidence Scale for Surgical Trainees Roxana Geoffrion, MD, 1 Terry Lee, PhD 2, Joel Singer, PhD 2 1 Faculty of Medicine, University of British Columbia, Vancouver BC 2 Centre for Health Evaluation and Outcome Sciences, University of British Columbia, Vancouver BC Abstract Objective: is a feeling of trust in one s abilities, qualities, and judgement. It is one of the cognitive mechanisms underlying behavioural change. Variations in self-confidence influence motivation and predict performance success. We sought to validate a new tool for measuring self-confidence in surgical residents. Methods: A self-confidence rating scale was developed and consisted of six questions about the attitudes of gynaecology residents while performing a vaginal hysterectomy. The residents were participants in a randomized controlled trial (2008 to 2011) and performed a vaginal hysterectomy before and after an educational intervention. They rated their own surgical performance on a validated global rating scale (GRS) of surgical skill and their self-confidence on the new rating scale. Supervising surgeons concurrently rated the residents performance on the GRS. Correlations were sought between self-confidence scale scores and measures of competence. Results: There was no difference in self-confidence scores between intervention and control residents at baseline. The number of vaginal hysterectomies performed before the educational intervention was associated with a significantly higher confidence level (P = 0.024). Other demographic variables such as age and gender did not influence confidence levels. Internal consistency between the individual scale items was good (Cronbach alpha 0.85). scores were significantly higher after the educational intervention (P = 0.04). was positively correlated with both self-assessed and supervising-surgeon assessed GRS scores in both intervention and control residents. Conclusion: The self-confidence scale is a psychometrically valid tool to measure residents self-confidence during surgical learning. More research is needed to establish the role of this scale for feedback, to channel self-confidence, and to optimize surgical skill acquisition. J Obstet Gynaecol Can 2013;35(4): Key Words: scale, validation, surgical resident, gynaecology Competing interests: None declared Received on September 4, 2012 Accepted on December 12, 2012 Résumé Objectif : La confiance en soi est un sentiment de confiance dans ses capacités, ses qualités et son discernement. Il s agit de l un des mécanismes cognitifs qui sous-tendent les changements comportementaux. Les variations en matière de confiance en soi influencent la motivation et permettent de prédire la qualité du rendement. Nous avons cherché à valider un nouvel outil de mesure de la confiance en soi chez les résidents en chirurgie. Méthodes : Une échelle d évaluation de la confiance en soi a été mise au point; elle comptait six questions au sujet des attitudes des résidents en gynécologie au moment où ils menaient une hystérectomie vaginale. Les résidents participaient à un essai comparatif randomisé (de 2008 à 2011) et ont mené une hystérectomie vaginale avant et après une intervention pédagogique. Ils ont évalué leur propre rendement chirurgical au moyen d une échelle d évaluation globale (EEG) validée de leurs compétences chirurgicales, en plus d évaluer leur confiance en soi au moyen de la nouvelle échelle d évaluation. Des chirurgiens superviseurs ont, de façon concomitante, évalué le rendement des résidents au moyen de l EEG. Nous cherchions à établir des corrélations entre les scores de confiance en soi et les mesures de la compétence. Résultats : Aucune différence n a été constatée en matière de scores de confiance en soi entre les résidents du groupe «intervention» et les résidents du groupe «témoin» au départ. Le nombre d hystérectomies vaginales menées avant l intervention pédagogique a été associé à une confiance en soi considérablement accrue (P = 0,024). D autres variables démographiques (comme l âge et le sexe) n ont pas exercé d influence sur les niveaux de confiance. La cohérence interne d un article d échelle à l autre était bonne (Cronbach alpha, 0,85). Les scores de confiance en soi étaient considérablement accrus à la suite de l intervention pédagogique (P = 0,04). La confiance en soi était en corrélation positive avec les scores EEG issus tant de l autoévaluation que de l évaluation menée par les chirurgiens superviseurs, et ce, tant chez les résidents du groupe «intervention» que chez les résidents du groupe «témoin». Conclusion : L échelle de confiance en soi est un outil psychométrique valable pour ce qui est de la mesure de la confiance en soi des résidents au cours de la formation chirurgicale. La tenue d autres recherches s avère requise pour établir le rôle de cette échelle aux fins de la rétroaction, pour canaliser la confiance en soi et pour optimiser l acquisition de compétences chirurgicales. APRIL JOGC AVRIL

2 INTRODUCTION is a feeling of trust in one s abilities, qualities, and judgement. 1 The social cognitive theory of human behaviour refers to this belief in one s ability to perform as perceived self-efficacy. 2 In other words, selfconfidence is one of the cognitive mechanisms underlying behavioural change; variations in self-confidence influence motivation and predict performance success. 2 A large national survey of general surgery residents in the United States revealed that 27.5% worry they will not feel confident performing procedures independently. 3 can influence career choice, 4 satisfaction with training, comfort with asking for help during training, and choice of a fellowship after residency. 5 It is often used to assess the success of an educational intervention within a curriculum. 6,7 Certain demographic variables such as gender, residency level, marital status, and residency program type are strongly correlated with confidence. 5 Female residents are significantly more concerned than their male counterparts about their ability to perform procedures independently before completion of a general surgery residency (37% vs. 21.9%, P < 0.01). 5 This can be of particular concern in obstetrics and gynaecology residencies, where a majority of trainees are female. Confidence also increases with increasing exposure to clinically relevant situations, 8 and obstetrics and gynaecology residents spend fewer months in purely surgical rotations than other surgical residents (4.9 vs. 8.5 for all years, P = 0.001). 9 To our knowledge, there is no psychometrically valid tool to assess selfconfidence in specific situations such as the performance of a surgical procedure by a surgical trainee. The objective measurement of self-confidence could assist surgical educators in implementing appropriate interventions to channel self-confidence and optimize success during surgical skill acquisition. With this in mind, our objective was to validate a new tool for measuring self-confidence in surgical residents. Our primary hypothesis was that our selfconfidence scale would successfully distinguish residents who had been enrolled in an educational intervention by their improved self-confidence scores compared with control subjects. Our secondary hypothesis was that self-confidence, global assessments of competence and self-confidence, and certain demographic variables, such as resident level and number of procedures previously performed, would be correlated. METHODS This was an ancillary study related to a larger randomized controlled trial investigating whether mental imagery is effective as an educational intervention to teach the performance of a vaginal hysterectomy to obstetrics and gynaecology residents. Details of the methodology have been previously described. 10 In brief, 50 junior gynaecology residents performed a pre-test vaginal hysterectomy and were then randomized to receive mental imagery or read a surgical textbook. All residents then performed a test vaginal hysterectomy. They were evaluated on pre-test and test performance by their attending surgeons, who were blinded to the residents group and who used a validated global rating scale of surgical skill. 11,12 All residents also evaluated their own performance on pre-test and test vaginal hysterectomy via the global rating scale (GRS) of sugical skill and a self-confidence scale. Data were collected prospectively, between 2008 and 2011, in eight academic surgical centres across Canada and the United States. The self-confidence scale originally consisted of seven questions about trainee attitudes during the surgical procedure (online eappendix). Selfrating was performed by selection on a Likert scale ranging from 1 to 5. Maximum score attainable was 35. The individual questions for the self-confidence scale were selected on the basis of a study by Sanders et al., 13 who examined self-confidence in relation to a mental imagery exercise for medical students. Their questionnaire items were briefly alluded to but not fully provided in their publication. To the best of our knowledge, this questionnaire was not subsequently validated elsewhere. For ease of understanding and administration, the Likert scale used for the self-confidence in our study was designed to be similar to the GRS, a surgical skill scale that is fully validated for use with obstetrics and gynaecology residents. 11,12 Before our surgical trial, the self-confidence scale was evaluated by expert medical educators at the American Urogynecologic Society fellows research forum (2007). The individual items were found to adequately measure the intended construct of self-confidence and to cover its full breadth. Subsequently, eight centres in Canada and the United States were recruited to participate in the surgical trial. The centres were selected on the basis of surgical volume and interest in surgical education methods. The principal investigator at each centre also reviewed the self-confidence scale for ease of understanding and administration, as well as face and content validity. Correlations were sought between self-confidence scores and demographic variables such as age, gender, level of residency training, and number of hysterectomies assisted or performed. Correlations with age and gender were assessed using the Kruskal-Wallis test, while the Jonckheere-Terpstra test was used to determine whether 356 APRIL JOGC AVRIL 2013

3 Validating a Self-Confidence Scale for Surgical Trainees Table 1. Demographics of participants Variable All subjects Control subjects Mental imagery Gender, n (%) Male 13 (26.0) 7 (26.9) 6 (25.0) Female 37 (74.0) 19 (73.1) 18 (75.0) Age, years Median (IQR) 28.0 (27.0 to 29.0) 28.0 (27.0 to 29.0) 28.0 (27.0 to 30.0) Mean (SD) 29.1 (4.0) 29.3 (4.7) 29.0 (3.2) Range (24.0 to 45.0) (24.0 to 45.0) (25.0 to 37.0) Year of residency training, n (%) 1 11 (22.0) 6 (23.1) 5 (20.8) 2 18 (36.0) 10 (38.5) 8 (33.3) 3 19 (38.0) 9 (34.6) 10 (41.7) 4 2 (4.0) 1 (3.8) 1 (4.2) Vaginal hysterectomies observed or assisted, n (%) < (58.0) 14 (53.8) 15 (62.5) 10 to (40.0) 11 (42.3) 9 (37.5) 50 to (2.0) 1 (3.8) 0 (0.0) Vaginal hysterectomies performed, n (%) 0 25 (50.0) 12 (46.2) 13 (54.2) 1 8 (16.0) 6 (23.1) 2 (8.3) 2 5 (10.0) 2 (7.7) 3 (12.5) 3 6 (12.0) 4 (15.4) 2 (8.3) 4 6 (12.0) 2 (7.7) 4 (16.7) SD: standard deviation; IQR: interquartile range Previously published data 10 reproduced with permission The Authors BJOG An International Journal of Obstetrics and Gynaecology 2012 RCOG self-confidence increased with increasing levels of other predictor variables. Internal consistency between the items on the self-confidence scale evaluated the extent of the interrelationship of individual scale items via the Cronbach alpha coefficient. Responsiveness of the selfconfidence scale, or the ability to detect change after an educational intervention, was evaluated by comparing self-confidence scores after mental imagery in the intervention versus the control group. Criterion validity of the self-confidence scale was also evaluated. It was expected self-confidence would increase with increasing feelings of competence. The most direct test of this relationship would be the relationship between self-confidence and self-assessed competence, buttressed by the relationship between self-confidence and competence as judged by an expert. We assessed the correlations between self-confidence scores and competence scores before and after intervention in both control and intervention groups using the Spearman correlation coefficient. Data analysis was performed using SAS version 9.2 (SAS Institute Inc., Cary NC). Ethics approval for the study was obtained from the Conjoint Health Research Ethics Board in Calgary and from each participating site individually. RESULTS The demographic characteristics of the residents in the trial, with uniform distribution between control and intervention groups, are shown in Table 1. Correlations between self-confidence and several predictor variables at baseline are shown in Table 2. Age and gender did not show significant relationships with self-confidence. The relationships between resident level and self-confidence and number of hysterectomies assisted and selfconfidence did not reach statistical significance; however, the number of hysterectomies performed was positively associated with self-confidence in all residents at baseline (P = 0.024). APRIL JOGC AVRIL

4 Table 2. Pre-intervention self-confidence score at various levels of predictor variables Variable n Median (IQR) P Gender Male (14.0 to 18.0) Female (14.0 to 19.0) Age, years* (14.0 to 18.0) 28 to (16.0 to 19.0) > (12.5 to 18.5) Year of residency training (14.0 to 18.0) (14.0 to 18.0) (16.0 to 21.0) (-) Vaginal hysterectomies observed or assisted < (13.0 to 19.0) 10 to (16.0 to 20.5) > (-) Vaginal hysterectomies performed (13.0 to 18.0) (12.5 to 17.5) (19.0 to 21.0) (16.0 to 18.0) (16.0 to 23.0) IQR: interquartile range *Cut-off based on interquartile range Table 3. Difference in score between pre- and post intervention Scale Control subjects Mental imagery Difference in mean (95% CI) P Mean (SD) 3.2 (3.1) 5.8 (4.8) 2.5 (0.2 to 4.8) Median (IQR) 2.0 (1.0 to 5.0) 5.0 (3.5 to 8.5) Range ( 2.0 to 11.0) ( 3.0 to 14.0) GRS by resident Mean (SD) 3.2 (3.7) 6.6 (4.3) 3.4 (1.1 to 5.6) Median (IQR) 3.5 (1.0 to 5.0) 6.0 (4.5 to 9.5) Range ( 3.0 to 14.0) ( 2.0 to 14.0) GRS by attending surgeon Mean (SD) 2.4 (4.8) 4.4 (5.8) 2.0 ( 1.1 to 5.1) Median (IQR) 3.0 (1.0 to 5.0) 5.0 ( 2.0 to 9.0) Range ( 10.0 to 11.0) ( 5.0 to 14.0) GRS: global rating scale score; SD: standard deviation; IQR: interquartile range. Previously published data 10 reproduced with permission The Authors BJOG An International Journal of Obstetrics and Gynaecology 2012 RCOG 358 APRIL JOGC AVRIL 2013

5 Validating a Self-Confidence Scale for Surgical Trainees Relationship between self-confidence scale and self- or attending physician assessment GRS r = 0.73, P < 0.01 r = 0.75, P < 0.01 r = 0.73, P < 0.01 r = 0.29, P = 0.04 r = 0.79, P < 0.01 r = 0.60, P < 0.01 When we assessed the relationships between items on the self-confidence scale, all items correlated well with one another, except for item 7, which asked whether the resident intended to pursue a subspecialty of gynaecology. Cronbach alpha, reflecting the internal consistency of the other six items, was Question 7 was therefore dropped from the questionnaire and further analyses were conducted using only the first six questions on the scale. At baseline, self-confidence scores were similar between the two groups of residents. levels increased more in the mental imagery educational intervention group (mean change = 5.8) than in the control group (mean change = 3.2) (Wilcoxon rank P = 0.039), as did the GRS self-ratings of competence (mean changes of 6.6 and 3.2, respectively) (Wilcoxon rank P = 0.003). The mean change in competence as judged by GRS scores of attending physicians was not significantly different between the mental imagery group and control subjects (4.4 vs. 2.4; 95% CI 1.1 to 5.1, P = 0.26) (Table 3). Most residents (29/48 or 60.4%) scored themselves lower on the GRS than the attending physicians on the test vaginal hysterectomy. On both pre-test and test vaginal hysterectomies, there were good correlations between self-confidence and selfassessed GRS score (pre-test overall r = 0.73, P < 0.001; test control r = 0.75, P < 0.001; test intervention r = 0.73, P < 0.001) (Figure). The correlation between the selfconfidence score and the GRS score by attending physician was positive but lower on the pre-test (r = 0.29, P = 0.04). This correlation was good on the test hysterectomy in both control and intervention groups (control r = 0.79, P < 0.001; intervention r = 0.60, P < 0.001) (Figure). DISCUSSION In this study, we evaluated the psychometric validity of a self-confidence scale for surgical residents. Our primary hypothesis, that residents enrolled in a randomized trial of a surgical educational intervention would have improved self-confidence compared with control subjects, was shown to be true. This is consistent with previous studies showing that practical learning experiences increase confidence. 14,15 Our study did not show any relationship between selfconfidence and gender. This is contrary to a large survey by Bucholz et al., 5 who showed that women are twice as likely as men to worry about their competence after training. Our study included only 13 male residents, compared with 37 female residents; therefore, men may have been APRIL JOGC AVRIL

6 insufficiently represented to demonstrate an association of gender with self-confidence. Level of training also showed no relationship with self-confidence. We had a majority of residents in the relatively junior second, third, and fourth years of training. Although the survey by Bucholz et al. 5 did show increased confidence with increased level of training, it also showed that a greater percentage of first and fifth year residents were confident than were residents in the second, third, and fourth years. Residents in middle years may display similarly low confidence levels in performing vaginal hysterectomy than those in the first and fifth years, and our study would have to be repeated with more junior and more senior residents to demonstrate an association. We did demonstrate an association between the number of procedures performed prior to entry into the trial and self-confidence. This is consistent with previous research showing a positive correlation between practice and confidence. 5,14,16 18 Evaluating the internal consistency of the scale helped us to eliminate the question about the pursuit of a surgical subspecialty in gynaecology: Will you pursue a surgical subspecialty of gynaecology if you continue operating at this level? It is reasonable that a resident might decide not to pursue such a residency for reasons other than confidence or competence. In addition, previous research shows that when residents are confident about their skills, they are actually less likely to pursue post-residency specialty training. 5 Self-assessed global rating scale scores correlated well with self-confidence, both on pre-test and test hysterectomies. Our data showed that as residents become more confident, the positive correlation between their confidence and the assessment of their surgical skill by supervising attending surgeons becomes stronger. Self-assessment is key for continuing professional development. 19 However, our ability to self-assess accurately is generally poor. 19 The selfconfidence scale goes a step further in enabling reflection on performance. It allows educators to focus on specific determinants of confidence when discussing performance with residents. A low self-confidence score coupled with a high GRS score from an attending supervising surgeon would prompt the attending surgeon to provide some positive feedback to the resident, thus increasing confidence. The self-efficacy literature indicates that those who expect to do well are more likely to do well than those who expect to do poorly. 20 Therefore, much could be gained from improving under-confident residents self-confidence when they are actually skilled at surgical tasks. On the other hand, mildly negative expectations have been shown to increase effort and attention to strategy. 21 Overconfidence in surgical trainees may need to be harnessed with the introduction of appropriate feedback to improve performance. Such feedback may include a preoperative discussion of the difficulty of the case or the need for particular attention to details of certain surgical steps. Our study introduces a psychometrically valid selfconfidence scale to the surgical community. Validation was achieved through a large cohort of resident participants in a randomized controlled trial who were undergoing training at several sites in Canada and the United States. A limitation of the study is that we were unable to assess testretest reliability. We found that confidence increases with an increasing number of procedures performed; therefore, we could not test this aspect of validity with our control residents, who, although not enrolled in an educational intervention, did perform an additional hysterectomy before assessing self-confidence a second time. Another limitation of our study is its limited generalizability, as the study participants were obstetrics and gynaecology residents and mostly female. It would be worthwhile to extend use of the self-confidence scale to other surgical residency programs and to more male residents. CONCLUSION The self-confidence scale is a psychometrically valid tool to measure residents self-confidence during surgical learning. More research is needed to establish the role of the selfconfidence scale during feedback to residents, as selfconfidence may play a significant role in the acquisition of surgical skill. ACKNOWLEDGEMENTS The authors acknowledge indirect support for the related Mental Imagery trial from an American Urogynecologic Society fellowship grant awarded in REFERENCES 1. Oxford Dictionaries [website]. Oxford University Press. Available at: Accessed July 1, Cervone D. Thinking about self-efficacy. Behav Modif 2000;24: Yeo H, Viola K, Berg D, Lin Z, Nunez-Smith M, Cammann C, et al. Attitudes, training experiences and professional expectations of US general surgery residents: a national survey. JAMA 2009;302: Sobral DT. Influences on choice of surgery as a career: a study of consecutive cohorts in a medical school. Med Ed 2006;40: Bucholz EM, Sue GR, Yeo H, Roman SA, Bell RG Jr, Sosa JA. Our trainees confidence. Results from a national survey of 4136 US general surgery residents. Arch Surg 2011;146: APRIL JOGC AVRIL 2013

7 Validating a Self-Confidence Scale for Surgical Trainees 6. Hoover SJ, Berry MP, Rossick L, Rege RV, Jones DB. Ultrasound-guided breast biopsy curriculum for surgical residents. Surg Innov 2008;15: Esterl RM Jr, Henzi DL, Cohn SM. Senior medical student Boot Camp can result in increased self-confidence before starting surgery internships. Curr Surg 2006;63: Binenbaum G, Musick DW, Ross HM. The development of physician confidence during surgical and medical internship. Am J Surg 2007;193: Geoffrion R, Choi JW, Lentz GM. Training surgical residents: the current Canadian perspective. J Surg Educ 2011;68: Geoffrion R, Gebhart J, Dooley Y, Bent A, Dandolu V, Meeks R, et al. The mind s scalpel in surgical education: a randomised controlled trial of mental imagery. BJOG 2012;119: Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84: Goff BA, Lentz GM, Lee D, Houmard B, Mandel LS. Development of an objective structured assessment of technical skills for obstetrics and gynecology residents. Obstet Gynecol 2000;96: Sanders CW, Sadoski M, Bramson R, Wiprud R, Van Walsum K. Comparing the effects of physical practice and mental imagery rehearsal on learning basic surgical skills by medical students. Am J Obstet Gynecol 2004;191(5): Mitchell EL, Sevdalis N, Arora S, Azarbal AF, Liem TK, Landry GJ. A fresh cadaver laboratory to conceptualize troublesome anatomic relationships in vascular surgery. J Vasc Surg 2012;55: Klingensmith ME, Brunt LM. Focused surgical skills training for senior medical students and interns. Surg Clin North Am 2010;90: Malekzadeh S, Malloy KM, Chu EE, Tompkins J, Battista A, Deutsch ES. ORL emergencies boot camp: using simulation to onboard residents. Laryngoscope 2011;121: O Toole RV, Playter RR, Krummel TM, Blank WC, Cornelius NH, Roberts WR, et al. Measuring and developing suturing techniques with a virtual reality surgical simulator. J Am Coll Surg 1999;189: Cooke JM, Larsen J, Hamstra SJ, Andreatta PB. Simulation enhances resident confidence in critical care and procedural skills. Fam Med 2008;40: Eva KW, Regehr G. I ll never play professional football and other fallacies of self-assessment. J Cont Ed Health Prof 2008;28: Bandura A. Human agency in social cognitive theory. Am Psych 1989;44: Stone DN. Overconfidence in initial self-efficacy judgments: effects on decision processes and performance. Organ Behav Hum Decis Process 1994;59: APRIL JOGC AVRIL