JBI Database of Systematic Reviews & Implementation Reports 2014;12(11)

Transcription

1 Measuring competence in healthcare learners and healthcare professionals by comparing self-assessment with objective structured clinical examinations: a systematic review Kim Sears, RN, PhD 1 Christina M. Godfrey, RN, PhD 1 Marian Luctkar-Flude, RN, MScN 2 Liane Ginsburg, RN, PhD 3 Deborah Tregunno, RN, PhD 2 Amanda Ross-White, MLIS, AHIP 4 1 The Queen's Joanna Briggs Collaboration for Patient Safety: a Collaborating Center of the Joanna Briggs Institute; Queen's University, Kingston, Ontario, Canada 2 School of Nursing, Queen's University, Kingston, Ontario, Canada 3 York University, Toronto, Ontario, Canada 4 Bracken Library, Queen's University, Kingston, Ontario, Canada Corresponding author: Kim Sears kim.sears@queensu.ca Executive summary Background The measure of clinical competence is an important aspect in the education of healthcare professionals. Two methods of assessment are typically described; an objective structured clinical examination and self-assessment. Objectives To compare the accuracy of self-assessed competence of healthcare learners and healthcare professionals with the assessment of competence using an objective structured clinical examination. doi: /jbisrir Page 221

2 Inclusion criteria Types of participants All healthcare learners and healthcare professionals including physicians, nurses, dentists, occupational therapists, physiotherapists, social workers and respiratory therapists. Types of intervention Studies in which participants were first administered a self-assessment (related to competence), followed by an objective structured clinical examination; the results of which were then compared. Types of outcomes Competence, confidence, performance, self-efficacy, knowledge and empathy. Types of studies Randomized controlled trials, non-randomized controlled trials, controlled before and after studies, cohort, case control studies and descriptive studies. Search strategy A three-step search strategy was utilized to locate both published and unpublished studies. Databases searched were: Medline, CINAHL, Embase, ERIC, Education Research Complete, Education Full Text, CBCA Education, GlobalHealth, Sociological Abstracts, Cochrane, PsycInfo, Mosby s Nursing Consult and Google Scholar. No date limit was used. Methodological quality Full papers were assessed for methodological quality by two reviewers working independently using the Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument (JBI-MAStARI). Data collection Details of each study included in the review were extracted independently by two reviewers using an adaptation of the standardized data extraction tool from JBI-MAStARI. Data synthesis Meta-analysis was not possible due to methodological and statistical heterogeneity of the included studies. Hence study findings are presented in narrative form. The data was also analyzed using The Four Stages of Learning model by Noel Burch. Results The search strategy located a total of 2831 citations and 18 studies were included in the final review. No articles were removed based on the critical appraisal process. For both competence and confidence, the majority of studies did not support a positive relationship between self-assessed performance and performance on an OSCE. doi: /jbisrir Page 222

3 Conclusions Study participants self-assessed competence or confidence was not confirmed by performance on an objective structured clinical examination. An accurate self-assessment may be threatened by over confidence and high performers tend to underestimate their ability. It is theorized that this disparity may in part be due to the stage that the learner or professional is in, with regard to knowledge and skill acquisition. Educators need to examine their evaluation methods to ensure that they are offering a varied and valid approach to assessment and evaluation. Notably, if self-assessment is to be used within programs, then learners need to be taught how to perform consistent and accurate self-assessments. Implications for practice It is important that educators understand the limitations within the evaluation of competence. Key aspects are the recognition of the stage that the learner is in with regard to skill acquisition and equipping both learners and professionals with the ability to perform consistent and accurate self-assessments. Implications for research There is a need for standardization on how outcomes are identified and measured in the area of competence. Further, identifying the leveling of an OSCE and the appropriate number of stations is required. Keywords Self-assessment; objective structured clinical examinations; education; healthcare learners; healthcare professionals Background Establishing the effectiveness of the health professional education process is complex and requires a multifaceted approach to assess the outcomes. 1 Typically, outcomes are assessed in terms of the competence of the professional, level of confidence, performance and/or skills. Throughout the literature on this topic, these terms are used interchangeably, but there is overlap and some terms may encompass others. Descriptions/definitions of these terms are provided as follows. Competence In their paper that discusses the definition and assessment of professional competence, Epstein and Hundert define professional competence as: the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection in daily practice for the benefit of the individual and community being served. 2(p.226) Confidence Holland s concept analysis of professional confidence describes four components; namely affect (feelings associated with action), reflection (thoughtfully examine one s actions and intentions), higher cognitive functioning (which includes aspects such as learning and integration of concepts, decision making, attention, motivation and memory) and action. 3 doi: /jbisrir Page 223

4 Performance The on-line Merriam-Webster dictionary defines performance as the execution of an action or something accomplished a deed or feat. 4 Skill Skill is defined as proficiency, facility, or dexterity that is acquired or developed through training or experience. 5 Self-efficacy Self-efficacy is defined as people's beliefs about their capabilities to produce designated levels of performance that exercise influence over events that affect their lives. 6 Knowledge Knowledge is defined as: (1) the fact or condition of knowing something with familiarity gained through experience or association; (2) acquaintance with or understanding of a science, art, or technique. 7 Empathy Empathy is defined as the ability to understand and share the feelings of another. 8 Empathy is an essential quality for health professionals and is often measured along with other skills and competence. Looking at the above definitions of competence, confidence and performance it is clear that there is considerable overlap. The term competence was found to be the most inclusive. Given the nuances involved in each term, this review refers to two main concepts: that of competence (including knowledge and performance) and that of confidence (including self-efficacy). Further, the findings were examined for healthcare learners and healthcare professionals. There are a variety of ways to measure health professionals competence. Although Objective Structured Clinical Examinations (OSCEs) are considered to be valid and reliable for assessing clinical skills, they are labor-intensive and costly to design and implement. 9,10 By comparison selfassessment requires less resources to implement; however, it is unclear whether self-assessment is an effective measure of competence and how well it correlates with actual performance. Self-assessment has been defined as: the evaluation or judgment of the worth of one s performance and the identification of one s strengths and weaknesses with a view to improving one s learning outcomes. 11(p.146) For example, self-reported patient safety competence may provide data about learners insights into and likely safety of their own practice 12, and about their perceived strengths or limitations. 13,14 The value of using more objective methods to assess competence is unclear. Recent studies examining self- versus expert assessment of technical and non-technical skills have produced mixed results. Surgeons seem to be able to accurately assess their own technical skills but not their non-technical skills 15 ; however, an earlier study of junior medical officers found no correlation between their self-assessments of confidence and their measured competencies on routine procedural skills. 16 A 2006 systematic review examined how accurately physicians subjectively evaluated their own competence compared with external observations of their competence. 17 Davis and colleagues doi: /jbisrir Page 224

5 concluded that physicians have a limited ability to accurately self-assess. 17 This may be particularly true among those rated as the least skilled and those who were the most confident. These results were found to be consistent with other professions. 18 The OSCE is another method that has shown to be a useful means to assess the competence of a learner. Typically, an OSCE consists of a specific scenario established by the examiners that requires the learner to demonstrate their proficiency in that area. The evaluator can control the environment and standardize the patient and in this manner use the OSCE to objectively assess competencies (i.e. knowledge, attitudes and behaviors). There is growing recognition that OSCEs are appropriate for evaluating the interpersonal skills associated with breaking bad news or cross-cultural interviewing. 19 The use of the OSCE to assess physician communication skills is also becoming more common In the realm of patient safety, there is a small but emerging body of literature encouraging the use of OSCEs to assess aspects of patient safety competence among medical trainees In this area, most OSCEs assess the technical aspects of patient safety or quality improvement competence 27,29-31, or clinical aspects of patient safety such as hand hygiene compliance and medication labeling. 27 Few studies describe the use of OSCEs to assess socio-cultural aspects of patient safety 26,28, and those that do tend to focus on communicating/disclosing an error and are discipline-specific in nature. 32,33 In nursing, a recent integrative review by Walsh and colleagues located 41 papers and identified major gaps regarding the psychometrics of nursing OSCEs. 34 In concluding their review, the researchers highlighted the need for additional research on using the OSCE as an evaluative tool in nursing. The OSCE is thought to be a more objective measure than self-assessment. However, while limited, examinations of the extent to which OSCE performance predicts outcomes on other performance metrics are somewhat equivocal. Some studies have failed to detect a significant positive relationship between OSCE performance and other forms of summative evaluations of health profession learners. 25 A study by Tamblyn found that scores achieved in a patient-physician communication and clinical decision-making OSCE that was part of a national licensing examination predicted complaints to medical regulatory authorities up to 10 years later. 35 In an environment where providing optimal student learning and quality patient care is a goal, there is a need to explore whether a link exists between self-assessment scores and OSCEs in light of providing the best learning for the most affordable means. It has been noted that some studies comparing self- and external assessments of competence (such as the OSCE) have had several methodological problems. Davis and colleagues report that fewer than half of the studies they included in their systematic review: (1) used pretested or validated OSCEs or standardized patients or assessment instruments, or (2) described objective criteria for performance assessment. 17 Others have noted there is insufficient methodological detail in most published research involving standardized patients (SP), in particular details pertaining to SP characteristics and their training. 36 An examination of the Cochrane Library of Systematic Reviews, the JBI Database of Systematic Reviews and Implementation Reports and the PROSPERO database indicates that no systematic reviews have been completed (or proposed) on this topic since the Davis review in Building on the Davis review which focused solely on physicians, this systematic review explored research that included all healthcare learners and healthcare professionals and examined the relationship between doi: /jbisrir Page 225

6 self-assessed competence and objective assessments of competence using the OSCE. As this review overlaps the same time period as Davis, it is worth noting that three studies (Fox 37, Barnsley 16, and Leopold 38 ) that were in the Davis systematic review met the inclusion criteria for this study. The proposed synthesis is part of a broader program of research which builds on recommendations from numerous international bodies regarding the need to restructure health professional education to ensure it equips learners with the knowledge, skills and attitudes they need to function safely. 1,39-42 Notably, there is also recognition that what is evaluated drives what is taught and learnt. 43,44 Accordingly, development of an OSCE for adoption by various health professional education programs may be crucial for truly integrating patient safety into health professional education. Just as written examinations and OSCEs assess different things 45,46, so do subjective and objective assessments; however, both are understood to yield important data. 12 The objectives, inclusion criteria and methods of analysis for this review were specified in advance and documented in a protocol. 47 Review question/objective The objective of this systematic review was to compare the use of self-assessment instruments to an OSCE to measure the competence of healthcare learners and healthcare professionals. The question used to guide the review was: when measuring the competence of healthcare learners and healthcare professionals, is the evaluation of competence using self-assessment instruments comparable to using an OSCE? Inclusion criteria Types of participants This review considered all healthcare learners and healthcare professionals including but not limited to physicians, nurses, dentists, occupational therapists, physiotherapists, social workers and respiratory therapists. Types of intervention This review considered studies in which participants were first administered a self-assessment (related to competence), followed by an OSCE; the results of which were then compared. Types of outcomes This review considered studies that included the following outcome measures: competence, confidence, performance, self-efficacy, knowledge and empathy as defined above. Types of studies This review considered both experimental and epidemiological study designs including randomized controlled trials, non-randomized controlled trials, quasi-experimental, before and after studies, prospective and retrospective cohort studies, case control studies and analytical cross-sectional studies for inclusion. Descriptive epidemiological study designs including case series, individual case reports and descriptive cross-sectional studies were also considered for inclusion. doi: /jbisrir Page 226

7 Search strategy The search strategy aimed to find both published and unpublished studies (Appendix I). A three-step search strategy was utilized in this review. An initial limited search of MEDLINE and CINAHL was undertaken, followed by an analysis of the text words contained in the title and abstract and of the index terms used to describe the article. A second search using all identified keywords and index terms was then undertaken across all included databases. Thirdly, the reference lists of all identified reports and articles were searched for additional studies. This review only included studies published in English. In order to provide the broader picture of all available literature on this topic, the non- English literature was tallied (but not translated). Although this review was building on a review done in 2006, in order to be thorough the search included articles from the inception of each database. The databases searched included: Medline, CINAHL, Embase, ERIC, Education Research Complete, Education Full Text, CBCA Education, GlobalHealth, Sociological Abstracts, Cochrane, Mosby s Nursing Consult and PsycInfo. The search for unpublished studies included Dissertation Abstracts and Google Scholar. Initial keywords used were: OSCE; objectiv$ structur$ clinic$ exam$; self-assessment; self-report; competence; confidence; self-efficacy. Method of the review Assessment of methodological quality Quantitative papers selected for retrieval were assessed by two independent reviewers for methodological validity prior to inclusion in the review using standardized critical appraisal instruments from the Joanna Briggs Institute Meta-Analysis of Statistics Assessment and Review Instrument (JBI- MAStARI) (Appendix II). Any disagreements that arose between the reviewers were resolved through discussion, or with a third reviewer. Studies reporting experimental/ quasi-experimental designs were assessed using the JBI Critical Appraisal Checklist for Randomized Control/Pseudo-randomized Trials. Studies reporting descriptive designs were assessed using the JBI Critical Appraisal Checklist for Descriptive/Case Series. Data collection Data was extracted from papers included in the review using an adapted form of the standardized data extraction tool from JBI-MAStARI (Appendix III). The adapted tool (Appendix IV) included specific details about the interventions, populations, study methods and outcomes of significance to the review question and specific objectives. Data synthesis Statistical pooling was not possible; therefore the findings were presented in narrative form including tables and figures to aid in data presentation where appropriate. The strength of the reported correlations (r values) were interpreted in accordance with the method described by Munro and apply to positive or negative correlations: little if any correlation (r =.00 to.25); low (r =.26 to.49); moderate (r =.50 to.69); high (r =.70 to.89); and very high (r =.90 to 1.00). 48 The data was also analyzed using The Four Stages of Learning model by Noel Burch. 49 doi: /jbisrir Page 227

8 Results Characteristics of included studies The search strategy located a total of 2831 citations (Figure 1). Of this set, 698 duplicates were removed and 127 articles were retrieved for full review. One hundred and nine articles were excluded on the basis of not meeting the inclusion criteria. No articles were removed based on the critical appraisal process, leaving a final set of 18 included studies. Eighteen studies compared learners self-assessment of competence with performance on an OSCE (Appendix IV and Appendix V). Within the review, there was a wide range of study designs: randomized control trials (two studies) 50,51, quasi-experimental (five studies) 52-56, and descriptive (11 studies) 16,37,38, Of the descriptive studies, six were cross-sectional 16,57-59,61,63, and two were correlational. 37,38 Eight studies were conducted in the U.S.A. 38,50,51,57-59,61,62, four studies in the U.K. 37,53,60,64, three studies from Canada 52,54,55, and one each from Australia 16, Malaysia 56, and the Netherlands. 63 There were no non-english studies located on this topic. Publication dates spanned The studies focused on a variety of healthcare learners and healthcare professionals. Thirteen studies examined medical learners 16,37,51,53,56,57,59,61,62,64 ; of which five studies evaluated residents 16,37,54,58,62, two studies examined nursing learners 52,55, and one explored nurses and midwives pre-registration. 60 Of the studies that explored healthcare professionals, one examined nurses and medical assistants 50, and one examined multidisciplinary practitioners. 38 One study examined both healthcare learners and practitioners. 63 Self-reported competence was measured by a range of instruments including: an 11- point scale 64, a 10-point scale 38, a 7-point scale 52,53,57,59, a 6-point scale 61, a 5-point scale 37,51,54,55, and a 4-point scale 16,58,60,62, a four-category 17-item behavioral checklist 58, two sub-scales with 18 items 50, a 40-item inventory 56, and a five-item scale. 63 doi: /jbisrir Page 228

9 Medline CINAHL Embase PsycInfo ERIC Education CBCA FT 179 Education 1 GlobalHealth 47 Sociological Abstracts 150 Cochrane 2 Hand search / grey literature 4 Total number of articles retrieved from searching 2831 Duplicates removed 698 Number of articles reviewed at metadata level (title/abstract) 2133 Number of articles excluded at metadata level: not on topic 2006 Number of articles reviewed in full text 127 Number of articles excluded after reviewing full text: not meeting inclusion criteria 109 Final number of articles reviewed for critical appraisal 18 Number of articles excluded: not meeting critical appraisal criteria 0 Final number of articles included in review 18 Figure 1: Search decision flow diagram doi: /jbisrir Page 229

10 Assessment of methodological quality Given the paucity of research in this area, a cut-off score of four was established for each critical appraisal checklist (Tables 1a and 1b). Critical appraisal scores ranged from four to six out of nine for descriptive studies, or four to six out of 10 for experimental studies; hence all eligible studies were included. Only 22% of the included studies on this topic were experimental/quasi-experimental in design; therefore, implications for inference and inherent bias must be considered. Four studies reported experimental/quasi-experimental methods involving comparisons between an experimental and control group. 50,51,53,55 Studies were evaluated according to the research design specified by the author. However, regardless of the design, the results that pertained to the comparison of selfassessment and OSCEs tended to be descriptive in nature. Table 1a: Assessment of methodological quality for experimental studies Citation Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Ault al. et Y U U N Y U Y Y Y Y (2002) Dornan et al. (2003) Doyle et al. (2010) Luctkar- Flude et al. (2012) N N N U N N Y Y Y Y U N U U Y Y Y Y Y Y N N N N U U Y Y Y Y % doi: /jbisrir Page 230

11 Table 1b: Assessment of methodological quality for descriptive studies Citation Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Barnsley al. (2004) Baxter Norman (2011) Berg et al. (2011) et & N Y U Y N/A N/A N Y Y N Y Y Y N/A N/A U Y Y N Y U Y Y N/A N Y Y Biernat et al. (2003) Chen et al. N Y N Y N/A N/A N/A Y Y N Y Y Y Y N/A N/A Y Y (2009) Fox et al. (2000) N Y U Y N/A N/A N Y Y Langhan al. (2009) et N Y N Y N/A Y N Y Y Lauder et al. N Y N Y Y N/A N/A Y Y (2008) Leopold et al. (2005) Lukman et al. N/A Y U Y N/A N/A N Y Y N Y N Y N/A Y N Y Y (2009) Mavis et al. N Y Y Y N/A N/A Y Y Y (2001) Parish et al. N Y N Y N/A N/A N Y Y (2006) Turner et al. N Y N Y N/A N/A N/A Y Y (2009) Vivekenanda- Schmidt et al. (2007) Y Y Y Y N/A N/A N Y Y % doi: /jbisrir Page 231

12 In the unlikely event that people evaluating the OSCE were aware of the students self-assessed grading prior to the OSCE this could compromise the results. However, no studies provided this level of detail. Further, the quality of the evidence may have been limited as some studies did not provide in-depth descriptions of methods, outcomes and use of statistical tests; an omission also noted by Davis et al. 17 Additionally some studies did not report the exact correlation data for their findings, but rather used terms such as limited or moderate. Many of the assessment tools were descriptive in nature and only six of the 18 studies reported on the validity of their tools. 52,53,55,60,63,64 Five studies reported on the reliability of the OSCE assessment 16,37,60,62,64, and 10 studies reported on the reliability of at least one tool ,57,60,61,64 Additionally, not all studies reported the precise correlation results, rather they reported the correlation as limited or mild but did not provide the data. 37,53,56,62 Correlational analysis is the appropriate statistical analysis to ascertain the relationship between self-assessment and OSCE performance. When the studies were not precise, the preset limits proposed by Munro were used to quantify the leveling of limited (negligible, r<0.26), mild (low, r = ), or moderate (r = ) correlations. 48 One study conducted a subgroup analysis to compare learners to professionals 63, and two studies compared gender 38,64, which are potential confounds to the overall study results. Criteria 5 and 6 of the appraisal tool for descriptive studies were not applicable to many studies in the review as they did not involve between-group comparisons and only reported cross-sectional correlations with no follow-up conducted. Findings of the review As described in the background, there is an overlap between the concepts of confidence and competence and these terms are frequently used interchangeably in the literature. As a consequence, this review referred to two main concepts: that of competence (including knowledge and performance) and confidence (including self-efficacy) (see Appendix VI for characteristics of included studies). For both competence and confidence the majority of studies did not support a positive relationship between self-assessed performance and performance on an OSCE. Competence The concept of competence included both knowledge and performance. Of the 18 studies that were involved in this review, seven studies focused on competence including performance and skill 37,53,54,57-59,62, and two studies focused on both competence and confidence. 52,60 Of the studies that examined self-report compared to the results from an OSCE in the area of competence, two identified a negative correlation 52,58, four identified no correlation 37,53,60,62, two identified a negligible (small) correlation 57,59, and only one reported low (mild) to moderate correlations (Table 2). 54 Confidence The concept of confidence included self-efficacy. Of the 18 studies that were involved in this review, nine studies focused on confidence 16,38,50,51,55,56,61,63,64, and two studies examined both competence and confidence. 52,60 Five studies compared OSCE performance to learner self-assessment of confidence and six studies compared OSCE performance to learner self-assessment related to selfefficacy. Of the studies that examined self-report compared to the results from an OSCE in the area of doi: /jbisrir Page 232

13 confidence three identified a negative correlation 38,52,63, four no correlation 16,50,60, three negligible (small) correlation 38,51,60, and three identified a low (mild) correlation. 55,63,64 There were no studies reporting moderate or high correlations between confidence and OSCE performance. Competence and Confidence Of the three studies that examined both self-reported competence and confidence as compared to the results from an OSCE, two identified a negative correlation 38,52, and one identified no correlation. 60 Self-Assessment A variety of concerns about the process of self-assessment were raised. For example, Lauder et al. identified concerns related to the measurement of self-reported competence. 60 In particular, they identified issues related to the variation in reliability of instruments and suggested an expansion of the range of components included in a self-reported assessment. The inability of participants to identify their own weaknesses 60, or to have an accurate perception of their abilities 62, has also raised concerns. 16,37,38,50,52,58,60 Ultimately, Lauder et al. cast doubt on the value of self-assessment and the benefit of correlating self-assessment with the assessment of performance on an OSCE. 60 Baxter and Norman 52 caution that the use of a self-assessment tool in nursing education to evaluate clinical competence requires serious examination and a clear rationale for its use. They note that overconfidence related to inaccurate self-assessment may lead to negative outcomes for new graduates and that overconfident new graduates may threaten patient care. Examination of healthcare professionals and healthcare learners As the review performed by Davis et al. focused solely on physician competence, a post hoc analysis was conducted to explore whether results would differ for other healthcare professionals or healthcare learners. 17 Two studies focused on healthcare professionals. One study found there was no correlation 50, and one study demonstrated a negative correlation. 38 Turner et al. examined both healthcare learners and healthcare professionals, but did not differentiate between the findings of the learners and professionals. 63 However, they did mention that there was no correlation between selfefficacy and the OSCE. In total, there were 15 studies that examined healthcare learners. The results of the examination of healthcare learners indicated that there was either no correlation 37,50,53,56,61,62, a negative correlation 16,52,58,59, a weak 64, or limited correlation 51, a mild to moderate correlation 54, or a moderate correlation 55,57, and no significant association between self-assessment and OSCE performance. 60 An inference may be made from the healthcare professionals and healthcare learners data suggesting that there is no increase in the level of insight into one s competence or confidence with increased experience in the profession. However this interpretation needs to be accepted with caution given the small sample of studies examining healthcare professionals and the quality of the evidence. doi: /jbisrir Page 233

14 Table 2: Comparison of self-assessment and OSCE (n=18) Author Domain Healthcare Review results /year/country professionals/learners Luctkar-Flude et Confidence Learners: Low correlation between self-confidence & performance: Significant correlations with al. 55 (2012) undergraduate secondary medication scores (Pearson correlation: r =.309, p =.044) & total Canada nursing performance scores (r =.368, p =.015). Baxter & Confidence & Learners: Negative correlation: All but one of 16 correlations were negative (inversely related to Norman 52 (2011) competence undergraduate actual performance). Only two of the self-assessment items were significantly Canada nursing correlated with OSCE total scores: post-test level of confidence dealing with acute care situations (Pearson correlation: r = , p<0.01); ability to manage a crisis situation (r = , p<0.01). Berg et al. 57 Competence Learners: Negligible correlation between self-reported empathy & assessments by (2011) USA (empathy) undergraduate standardized patients (Pearson correlation: r =0.19, p<0.05). Medical Chen et al. 59 Competence Learners: medical Negligible overall correlation between self-reported empathy (JSPE-S) scores and (2010) USA (empathy) OSCE empathy (r =0.22; p<0.001). Learners in their 2 nd year had a higher perceived self-assessment compared to the OSCEs. By their 3 rd year their self-assessment was lower than their OSCEs. Doyle et al. 50 Confidence (self- Healthcare High magnitude of self-efficacy improvement did not correlate with improved (2010) USA efficacy) professionals performance: On self-assessment there was significant improvement for self-efficacy (nursing, medical) (ANCOVA: F=24.43, p<0.001); on performance there was no significant improvement between intervention and control groups (F=3.46, p=0.073). Langhan et al. 54 Competence Learners: medical Low to moderate correlations between expert assessment total rating scores and (2009) Canada (knowledge & residents residents self-assessed knowledge (Pearson correlation: r =0.40, p<0.05) & clinical performance) skills scores (r =0.51, p<0.01). Sears et al. Measuring competence in healthcare learners and healthcare professionals by comparing self-assessment with objective structured clinical examinations: a systematic review the authors 2014 doi: /jbisrir Page 234

15 Author Domain Healthcare Review results /year/country professionals/learners Lukman et al. 56 Confidence (self- Learners: No correlation between self-efficacy & performance (actual correlational data not (2009) efficacy) undergraduate provided) Malaysia medical Turner et al. 63 Confidence (self- Practitioners: Little correlation between self-efficacy & OSCE performance. No differentiation made (2009) efficacy) pediatric & between healthcare learners & professionals. Significant correlation between self- Netherlands anaesthesia trainees & specialists reported self-efficacy and only two of seven OSCE performance scores: global resuscitation score (Spearman correlation: r=0.467, p =0.002) and time to intention to intubate (r= ; p<0.001). Lauder et al. 60 Competence & Learners: pre- Negligible significant association between confidence (self-efficacy) and the drug (2008) UK confidence (self- registration nursing & calculation OSCE (Spearman correlation: r =0.239, p=0.028); no significant efficacy) midwifery associations between self-reported competence & any of the OSCE scores. Vivekenanda- Schmidt et al. 64 Confidence Learners: undergraduate Correlation between self-assessment & OSCE scores were non-significant: total scores (Pearson correlation: r =0.13, p=0.11), shoulder assessment scores (r =0.16, (2007) medical p=0.150, knee scores (r =0.13, p=0.21); however, significant but weak correlations UK were noted for two measures in female learners only: total scores total (r =0.22, p=0.04) & shoulder scores (r =0.29, p=0.03). Parish et al. 62 (2006) USA Competence Learners: residents medical learners No correlations between summary OSCE scores & either interest or competence. (Actual correlational data not provided). High performers underestimated their competence. Residents self-assessed competence was not associated with OSCE performance. Leopold et al. 38 Confidence Practitioners: Negative correlation. Before instruction, participants confidence was significantly but (2005) USA multidisciplinary inversely related to competent performance (r =-0.253, p=0.02); that is greater confidence correlated with poorer performance. Both men & physicians displayed Sears et al. Measuring competence in healthcare learners and healthcare professionals by comparing self-assessment with objective structured clinical examinations: a systematic review the authors 2014 doi: /jbisrir Page 235

16 Author Domain Healthcare Review results /year/country professionals/learners higher pre-instruction confidence (p<0.01) that was not correlated with better performance. After instruction, confidence correlated with objective competence in all groups (r =0.24, p=0.04); men & physicians disproportionately overestimated their skills both before & after training, a finding that worsened as confidence increased. Barnsley et al. 16 (2004) Australia Biernat et al. 58 (2003) USA Confidence Learners: residents medical learners Competence Learners: residents medical learners No correlation. Application of the Wilcoxon s signed rank test confirmed a true difference between self-reported confidence scores and the assessed competence scores for seven procedural skills: venipuncture (Z=14.556, p=0.001), IV cannulation (Z=-3.598, p=0.001), CPR (Z=-4.371, p=0.001), ECG (Z=-3.306, p=0.001), catheterization (Z=-4.769, p=0.001), NG tube insertion (Z=2.737, p=0.01), blood cultures (Z=-3.974, p=0.001). Thus there was no association between any of the selfassessments and actual performance. For all skills, the OSCE performances were lower than self-reported confidence skills. Mixed results: self-assessment and rater assessments of competence were congruent for six out of seven categories: communication, history of present illness, past medical history, social history, mini mental status and geriatric depression scale and were significantly different for the 7 th : functional assessment (p<0.01) in which residents rated themselves higher than the assessors. However, when self-ratings were compared to assessors individual item ratings within the categories large discrepancies were revealed. There was a negative association between the scores on the self-assessment & the results from the OSCE. Dornan et al. 53 Competence Learners: Assessment results did not correlate with real patient learning. (Actual correlational (2003) UK (real patient undergraduate data not provided). learning) medical Ault et al. 51 Confidence (self- Learners: Limited association between self-report examination & OSCE. Students who Sears et al. Measuring competence in healthcare learners and healthcare professionals by comparing self-assessment with objective structured clinical examinations: a systematic review the authors 2014 doi: /jbisrir Page 236

17 Author /year/country Domain Healthcare professionals/learners (2002) USA efficacy) undergraduate medical Review results participated in the breast exam workshop reported higher self-efficacy related to their breast exam skills (t =10.72, p<0.05) and performed significantly higher in clinical exam skills (t =-2.99, p<0.05) than students who did not attend the workshop. (Actual correlational data not provided). Mavis et al. 61 Confidence (self- Learners: Learners with high self-efficacy were more likely to score above the mean OSCE (2001) efficacy) undergraduate performance compared to low self-rated learners (71% versus 51%); however self- USA medical efficacy was not significantly correlated to OSCE performance (r =0.12, p>0.05). Fox et al. 37 (2000). UK Competence Learners: residents medical learners There were no significant correlations between skills performed on OSCE stations & participants self-ratings. (Actual correlational data not provided). Sears et al. Measuring competence in healthcare learners and healthcare professionals by comparing self-assessment with objective structured clinical examinations: a systematic review the authors 2014 doi: /jbisrir Page 237

18 Discussion The findings of this review suggest that the evaluation obtained by self-assessment instruments is not comparable to performance on an OSCE. Notably, there are concerns emerging from the literature as to the validity and reliability of self-report, as well as the standardization of measurement using an OSCE. Currently in the practice of health education, there is no gold standard method for the evaluation of competence. It is thought that the OSCE provides a closer depiction of the students competence but there is no standardization across programs, years or segments of learning to identify when it is appropriate to use OSCEs, how many stations should be involved and what content lends itself best to this form of evaluation. There is no consistency in terms of measurement for either an OSCE or self-assessment. Self-assessment was measured by different instruments including scales with a variety of categories and items (range 4-point scale to 40 item inventory). Similarly, OSCEs lacked standardization in the number of stations used and method of evaluation (range 1-17 stations). Using the criteria described by Munro as previously outlined in the data synthesis section, the overall findings from this review demonstrate that out of the 18 studies examined, four studies found negative correlations, seven found no correlation and seven found positive correlations that ranged from negligible to moderate in strength. 48 This finding is similar to the review that Davis conducted. Davis found that out of the 20 studies on physicians that were included in their review, 13 demonstrated little, no or an inverse relationship between self-assessment and other indicators. They found that seven studies demonstrated a positive relationship between self-assessment and external observations. Interpretation according to the four stages of learning model The findings were also interpreted using the Four Stages of Learning theory developed by Noel Burch (Gordon Training International) in the early 1970s. 49 The model is comprised of four stages including: stage 1 unconscious incompetence (do not know what they do not know); stage 2 conscious incompetence (know what they do not know); stage 3 conscious competence (know what they know) and stage 4 unconscious competence (knowledge becomes a part of one s being, almost unconscious the expert). According to Burch, everyone progresses through the same four stages regardless of the skill that needs to be acquired. From the beginning stage of acquiring a new skill the learner progresses from stage 1 towards stage 4 as they gain experience and their level of competence increases. This systematic review focused on the assessment of competence, thus this model provided a useful framework to interpret the levels of competence demonstrated by the participants in the included studies. Three reviewers read through each article independently and interpreted the study findings with respect to the Burch framework in order to align the findings with the stages of the framework. Alignment was based on the stage of competence of the participant learners. Consensus was reached for interpretation of each article (Table 3). The stage of competence assigned by the reviewers was based on the level of agreement between the participants self-assessed level of competence and their actual performance on an OSCE. For example, if participants self-assessment scores were high but their performance scores were low, these findings were interpreted as being representative of the unconscious incompetence stage of the framework. doi: /jbisrir Page 238

19 According to Burch, the skill that one is attempting to acquire is irrelevant as everyone has to progress through the same four stages. Burch identified that by being aware of these stages one can better anticipate and accept that learning can be a slow and frequently uncomfortable process. In stage 1 (unconscious incompetence) - learners are unaware of their own shortcomings. 49 In this stage, learners are seen to have little knowledge of the extent of skill and abilities they will need to master and do not realize their own knowledge base that currently exists. According to the interpretation of the findings of this review, more novice learners thought they were doing better and lacked a deeper level of self-awareness. In six studies, the participants were at this stage of competence. 16,37,38,50,52,58,60 Stage 2 (conscious incompetence) - learners are aware of their former state of ignorance/naïveté and begin to process how much material/skill/knowledge they will need to absorb. 49 Learners may experience some emotional distress during this stage as they realize the consequences of their actions or become overwhelmed at the process ahead. When interpreting the review findings in the context of the framework, it was noted that as learners knowledge increased, their self-assessment became more accurate. Further, stronger learners underrated themselves, which may be related to the fact that although they had mastered the theoretical content of a skill, they appeared to have not mastered the practical application or recognized the level of complexity that was involved to become competent. Examples of this stage are provided by Parish et al. 62, and Baxter and Norman 52, who found that high performers underestimated their competence. Stage 3 (conscious competence) - learners begin absorbing; processing and practicing the skills and abilities they will need to advance to the unconscious competence stage. 49 In the conscious competence stage learners become proficient, effective and reliable at completing required tasks and assignments. It is at this conscious competence stage that learners change from unskilled amateurs to skilled professionals. Learners at this stage still need to consciously think about performing the correct skill/ability. In three studies, the participants were at this stage of competence as demonstrated by congruence between their self-assessments and their OSCE performance. 55,57,60 Stage 4 (unconscious competence) - following best practice becomes routine and learners no longer need to consciously think about performing the skill/ability properly. 49 Learners are comfortable with their skill level and feel confident in their abilities and this is verified by others responses to their work. As healthcare learners or healthcare professionals were not followed to the point where they would have been performing at a stage of unconscious competence, it is not possible to link the review findings to this stage of the model. Several studies illustrated progressions between the stages. In the study by Chen et al., second year learners had a higher perceived self-assessment compared to the OSCEs (in other words they were in stage 1 unconscious incompetence); however, in their third year, their self-assessment decreased while their ability on the OSCE increased (they progressed to stage 2 conscious incompetence). 59 Participants in the study by Leopold et al., 38 demonstrated a progression from stage 1 unconscious incompetence prior to an educational session, to stage 3 conscious competence following instruction. The participants in the study by Mavis 61 trended towards confidence, but the finding was not significant; whereas, Langhan et al. s 54 and Ault et al. s 51 participants progressed from stage 1 to stage 3. Further the findings of Turner et al. 63, Lukman et al. 56 and Dornan et al. 53 were unclear in relation to the model. doi: /jbisrir Page 239

20 Table 3: Interpretation and alignment of review results according to the four stages of learning model (n=18) Author /year/ Domain Healthcare Alignment with the four stages of learning country professionals/ learners model: i) unconscious incompetence ii) conscious incompetence iii) conscious competence iv) unconscious competence Luctkar-Flude Confidence Learners: Conscious competence et al. 55 (2012) undergraduate nursing Canada Baxter & Confidence Learners: Unconscious incompetence Norman 52 (2011) & competence undergraduate nursing conscious incompetence Canada Berg et al. 57 Competence Learners: Conscious competence (2011) USA (empathy) undergraduate medical Chen et al. 59 Competence Learners: medical Progression (2010) USA (empathy) unconscious incompetence (learners in their 2 nd year) conscious incompetence (learners in their 3 rd year) Doyle et al. 50 Confidence Healthcare unconscious incompetence (2010) USA (self-efficacy) professionals (nursing, medical) Langhan et Competence Learners: medical Progression from unconscious al. 54 (2009) (knowledge residents incompetence to conscious competence Canada & performance) Lukman et Confidence Learners: Stage of model unclear al. 56 (2009) (self-efficacy) undergraduate medical Malaysia Turner et al. 63 Confidence Practitioners: pediatric Stage of model unclear (2009) (self-efficacy) & anaesthesia trainees Netherlands & specialists doi: /jbisrir Page 240

21 Author /year/ Domain Healthcare Alignment with the four stages of learning country professionals/ learners model: i) unconscious incompetence ii) conscious incompetence iii) conscious competence iv) unconscious competence Lauder et Competence Learners: pre- Unconscious incompetence (competence) al. 60 (2008) UK & confidence (self-efficacy) registration nursing & midwifery conscious competence (confidence) Vivekenanda- Confidence Learners: Conscious competence Schmidt et undergraduate medical al. 64 (2007) UK Parish et al. 62 (2006) USA Competence Learners: residents medical learners Conscious incompetence Leopold et Confidence Practitioners: Unconscious incompetence (pre- al. 38 (2005) multidisciplinary instruction) USA conscious competence (post-instruction) unconscious incompetence (pre & post instruction) Barnsley et al. 16 (2004) Australia Biernat et al. 58 (2003) USA Confidence Learners: residents medical learners Competence Learners: residents medical learners Unconscious incompetence Unconscious incompetence Dornan et Competence Learners: Stage of model unclear al. 53 (2003) (real patient undergraduate medical UK learning) Ault et al. 51 Confidence Learners: Progression from unconscious (2002) USA (self-efficacy) undergraduate medical incompetence to conscious competence Mavis et al. 61 Confidence Learners: Trend towards conscious competence but (2001) USA (self-efficacy) undergraduate medical not significant Fox et al. 37 (2000) UK Competence Learners: residents medical learners Unconscious incompetence doi: /jbisrir Page 241

22 Limitations of the review There are several limitations of this review. First of all, there is a lack of conceptual clarity of the terms competence, performance, confidence and self-efficacy. These four terms were often used interchangeably in the literature, with some studies focusing on more than one concept. Secondly, a variety of assessment tools were used to establish the level of competence; hence it was difficult to establish a consistent measure of competence across the studies. Thirdly, not all studies identified the number of OSCE stations, nor is there a consensus of how many stations and the leveling of skills at these stations to ensure consistency of measurement. To be an effective measure of competence, the OSCE needs to be aligned with the skills, knowledge or performance being tested. Notably, there is no consensus in the literature regarding how many stations and the leveling of skills at these stations that is needed to ensure a comprehensive assessment and consistency of measurement. The implications of these limitations to the review highlight the need for caution when exploring and measuring the concept of competence. This review has used study authors definitions of the term and the multiple descriptors associated with it. These results may not be generalizable to other research or situations in which competence is examined in a broad or narrow manner. Conclusion In this review, it was found that the evaluation of competence obtained by self-assessment instruments was not comparable to an OSCE. It was shown that an accurate self-assessment is threatened by overconfidence and that high performers underestimate their ability. It is theorized that this is in part due to the stage of the learner with regard to skill acquisition according to the four stages of learning. 49 It is also important to note that both healthcare professionals and healthcare learners were not accurate when assessing their own abilities and competencies. The goal of healthcare education is to provide optimal student learning thereby facilitating quality patient care. However, the findings of this review are important as the quality of care delivered to patients and families is heavily affected by the competence of healthcare professionals. Professional programs are being held more accountable for the competence of the individuals that they graduate. Therefore, educators need to examine evaluation methods to ensure that we are offering a varied and valid approach to assessment and appraisal. A standardized method of evaluation is required for both selfassessments and OSCEs and evaluations need to be standardized across years of a program to confirm the degree of complexity is appropriate for learners. Notably, learners need to be taught how to perform accurate self-assessments if these evaluations are to be used within programs. The findings of this review are drawn from the level of evidence rated as a three (3) (observational analytic designs). Implications for practice The following implications are based on a level of evidence rated as a three (3). Preparing novice healthcare professionals to enter their chosen profession with an entry level of competence is complicated. However, ensuring that they are functioning at this level is even more complex. The findings of this review indicate that self-assessment is threatened by overconfidence that is often seen in very novice learners. Furthermore, high performers were seen to underestimate their ability. Educators need to understand this limitation within the evaluation approach and apply varied methods of evaluation in an attempt to gather a holistic picture of a learner s competence. Key messages were doi: /jbisrir Page 242