Assessing resident performance is an essential part of the

Size: px

Start display at page:

Download "Assessing resident performance is an essential part of the"

Lydia Thomas
5 years ago
Views:

Ophthalmic Education Update Assessment s and Tools Karl C. Golnik ABSTRACT The goal of ophthalmology residency training is to produce competent ophthalmologists.

1 Ophthalmic Education Update Assessment s and Tools Karl C. Golnik ABSTRACT The goal of ophthalmology residency training is to produce competent ophthalmologists. Competence can only be determined by appropriately assessing resident performance. There are accepted guiding principles that should be applied to competence assessment methods. These principles are enumerated herein and ophthalmology specific assessment tools that are available are described. Access this article online Website: DOI: / Quick Response Code: Key words: Assessment, Education, Feedback INTRODUCTION Assessing resident performance is an essential part of the training process and integral to the development of a competent ophthalmologist. Without good assessments in place, one cannot be sure if the resident has learned what is expected. Appropriate tools are needed for assessment, which should be based on accepted principles. This article will review relevant principles of assessment and ophthalmology specific assessment tools that have already been published and are available for use. Many variations of the principles of assessment have been described. Williams previously described ten specific guidelines for good assessment and I describe those I believe to be most relevant to medical training assessment. 1 ASSESSMENT PRINCIPLES Time is valuable and limited. Ophthalmologist teachers are typically very busy and are paid little or nothing for their teaching activities. Thus, any assessment should be as time efficient as possible but still provide the needed information. The need for brevity must be balanced and at the same time gather adequate valid and reliable information. Too brief an assessment may not provide what is needed and if it is too long, the assessor may not complete the assessment. The most important performance aspects only should be assessed and one should be ready to act on the assessment results to maximize their effect. Test application of knowledge/skills rather than just facts. Although there is clearly a need to assess basic memorized ophthalmic knowledge, assessment of surgical and clinical skill is also essential. A multiple choice test may be adequate for measuring medical knowledge but it is not a good measure of procedural skill. This means that multiple assessment tools must be used to assess important aspects of the resident s performance. Ideally, multiple assessors will also be involved. Performance is situation specific. Good performance in one situation (e.g. medical knowledge multiple choice test) does not mean there will be good performances in other situations (e.g. surgical skill or examination techniques). Nevertheless, sometimes it is assumed that the smart resident is also professional or a good communicator. Clearly this is not always the case and thus assessment tools must be used to evaluate multiple aspects of the resident s performance. Define assessments clearly in advance. Assessments should not be kept a secret. Let the residents know exactly how they University of Cincinnati and the Cincinnati Eye Institute, Cincinnati, Ohio, USA Corresponding Author: Dr. Karl C. Golnik, University of Cincinnati and the Cincinnati Eye Institute, Cincinnati, Ohio 45242, USA. E mail: golnikkarl@gmail.com Middle East African Journal of Ophthalmology, Volume 21, Number 2, April - June

2 Golnik: Assessment s and Tools will be assessed and be sure the assessments are aligned with your objectives and performance expectations. Doing this will minimize unnecessary assessments and assure relevant assessments are performed. Make every assessment a learning experience. This may not always be possible if the assessment is an examination that leads to passing or failing a course, rotation or residency. This type of assessment is known as summative feedback. On the contrary, formative feedback is assessment that is designed to improve performance and thus must be shared with the resident. The vast majority of assessments should be formative. When the assessment is shared with the resident on a frequent basis and is designed to improve performance it is called teaching Good formative feedback is timely and specific and ideally given every day. It should be used to reinforce good behavior (make a good resident even better) and used to change undesirable behavior. Good assessment methods and tools have become increasingly important as the spectrum of desired competencies and abilities are more clearly defined and emphasized. Both the Accreditation Council for Graduate Medical Education (ACGME) 2 in the United States and the Royal College of Physicians and Surgeons in Canada 3 have developed and emphasized competencies thought essential to become a good physician. These competencies are similar to certain extent and are listed in Tables 1 and 2. Assessment tools must be available to assure the resident that he/she has achieved these competencies. Fortunately, variety of ophthalmology specific assessment tools has already been developed to assess these competencies. ASSESSMENT TOOLS Many ophthalmology programs have unpublished and very subjective methods of resident assessment. Typically, these will be evaluations carried out by the teaching faculty where residents are rated on a Likert scale (e.g. 1-5 or 1-10). This form of assessment is probably better than not having a rating scale but it does not meet many of the assessment principles described above. Additionally, medical knowledge is usually assessed by multiple choice tests at a program or national level. The International Council of Ophthalmology (ICO) offers the only internationally developed test of ophthalmic medical knowledge. The ICO examinations include both basic and advanced levels. 4 However, other forms of assessment tools are necessary to assess procedures and skill. These tools usually consist of rubrics ideally with behavioral descriptors. A rubric is a tool that can help one give timely, specific, structured feedback and is defined as an explicit set of criteria used to assess a particular skill. Rubrics can be used to assess virtually any skill including playing a musical instrument, lecturing, or performing a surgical procedure. Table 1: The ACGME core competencies Patient care Residents must be able to provide patient care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health Medical knowledge Residents must demonstrate knowledge of established and evolving biomedical, clinical, epidemiological and social behavioral sciences, as well as the application of this knowledge to patient care Practice based learning and improvement Residents must demonstrate the ability to investigate and evaluate their care of patients, to appraise and assimilate scientific evidence, and to improve continuously on patient care based on constant self evaluation and life long learning Interpersonal and communication skills Residents must demonstrate interpersonal and communication skills that result in the effective exchange of information and collaboration with patients, their families, and health professionals Professionalism Residents must demonstrate a commitment to carry out professional responsibilities and an adherence to ethical principles Systems based practice Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care, as well as the ability to call effectively on other resources in the system to provide optimal health care ACGME: Accreditation council for graduate medical education Table 2: The CanMEDS competencies Medical expert Possess the knowledge and skills required to provide up to date, ethical, and resource efficient clinical care. This is the central role of physicians and requires all of the roles listed below Communicator Ability to efficiently manage the doctor patient relationship Collaborator Ability to work effectively in the healthcare team and to provide optimal patient care Manager Ability to organize practices, allocate resources appropriately, and contribute to the effectiveness of the healthcare system Health advocate Ability to advance the health and well being of the patients, communities and population Scholar Ability to demonstrate life long learning principles to enhance professional activities, create and apply new medical information and educate students, patients and peer groups Professional Practice ethically and have high standards of personal behavior Good rubrics consist of three parts: (1) Dimensions (e.g. steps of a surgical procedure); (2) Levels (e.g. score of 1-5 or novice, beginner, advanced beginner, competent, expert); and (3) Behavioral Descriptors (what it means to perform at a certain level for any of the dimensions). For example, an assessment rubric for cataract surgery might include dimensions of prepping and draping the patient; and levels from 1-5; and descriptions of exactly what behavior is necessary to score 1-5. Specific examples of procedural rubrics will be described below and are available at ( Rubrics should be given to the 110 Middle East African Journal of Ophthalmology, Volume 21, Number 2, April - June 2014

3 Golnik: Assessment s and Tools resident in advance so they will serve both, as a tool for teaching and assessing. The resident can read the rubric to learn what is required to be competent in various stages of the procedure. The rubrics should be completed by the teacher immediately after the observed performance and then reviewed with the resident to provide structured, timely and specific formative feedback as described above. Additionally, the rubric allows more objective, structured assessment of resident performance thus allowing areas of deficiency to be identified and remediated. The first of these ophthalmology specific assessment rubrics was developed to assess patient care and called the Ophthalmic Clinical Evaluation Exercise (OCEX). 5 The specified tool is completed by a teaching physician as they observe the resident performing a patient history, examination and then listens to the case presentation. The teaching physician completes scoring in 33 categories that rate the residents ability to communicate effectively, perform a history and examination, and synthesize the information into a differential diagnosis and plan. Importantly, a rubric that describes the behavior necessary to achieve each grade on the OCEX was developed. The OCEX has been shown to have content validity and inter rater reliability. 5,6 It is available on the ICO s website in multiple languages ( It was not developed by an international panel and thus may need to be modified to reflect cultural differences. The OCEX is a valid and reliable tool for assessing the competencies of patient care, medical knowledge and communication skills. Several methods of surgical skill assessment have been devised. Cremers and associates developed the Objective Assessment of Skills in Intraocular Surgery (OASIS), a one page objective evaluation form to assess residents skills in cataract surgery. 7 The form is completed by an evaluator who directly observes the surgical procedure and includes objective data such as wound placement and size, phacoemulsification time, and total surgical time, etc., They showed that the OASIS had both face and content validity. To complement this objective assessment the same group developed a subjective rating of surgical skills named Global Rating Assessment of Skills in Intraocular Surgery (GRASIS). 8 This one page form allows the evaluator to assign scores from 1-5 based on a behaviorally anchored rubric to domains such as preoperative knowledge, microscope use, instrument handling, and tissue treatment in addition to seven other areas. Thus the use of the combination of the OASIS and GRACIS provides objective and subjective evaluation of surgical skills. Feldman and Geist described the Subjective Phacoemulsification Skills Assessment as an evaluative instrument designed specifically for intraoperative assessment of resident phacoemulsification cataract extraction (PCE) surgery. 9 This form delineates PCE into overall performance and specific steps of the procedure (e.g. capsulorrhexis, hydrodelineation, IOL implantation, etc). The performance was graded with a rubric defining a good outcome at each step and asking the evaluator to rate on a 1-5 spectrumfrom strongly agree to strongly disagree. They were able to show a degree of inter rater reliability. Saleh and colleagues described an assessment tool called the Objective Structured Assessment of Cataract Surgical Skill (OSACSS). 10 This tool breaks down the phacoemulsification procedure into 20 steps that are scored on a 5 point Likert scale. The scale anchors are: 1= poorly or inadequately performed, 3= performed with some errors or hesitation and 5= performed well with no prompting or hesitation. There are no scale anchors for scores of 2 or 4. Further, there is no behavioral or skill based rubric for the evaluators to use when assessing the resident s competence. An international panel of author s modified the OSACSS by producing a globally applicable rubric with levels based on the Dreyfus model of skill acquisition (novice, beginner, advanced beginner, competent, and expert) and with behavioral anchors for each level in each step of the surgical procedure was created. 11 Once drafted, the content and face validity were achieved by having an international panel of 15 experts review the draft instrument and provide feedback. The international panel was composed of representatives from North America, Latin America, Asia, the Middle East, Europe and Africa. After incorporating suggestions from the international panel, a final document, the ICO Ophthalmology Surgical Competency Assessment Rubric (OSCAR) phacoemulsification was produced. 11 Furthermore, it was subsequently shown to have inter rater reliability. 12 In a similar fashion internationally applicable assessment tools for extracapsular cataract surgery (ICO OSCAR: ECCE), 11 small incision cataract surgery (ICO OSCAR: SICS), 13 lateral tarsal strip surgery (ICO OSCAR: LTS), 14 and strabismus surgery (ICO OSCAR: Strabismus) 15 were developed. Additional ICO OSCARS are being developed for panretinal photocoagulation, corneal transplant, vitrectomy, trabeculectomy and epiretinal membrane peeling. Recently, another cataract surgery assessment tool was developed in Canada and was shown to have a degree of validity and reliability. 16 It was not created by an international panel nor does it has a specific behaviorally grounded rubric. The ICO OSCAR assessment tools serve a variety of purposes: (1) They are internationally applicable, as comments from an international panel of experts were used to adapt it and make it flexible to any setting; (2) they will decrease subjectivity of the assessment by clearly defining for the assessor what behavior must be observed for each level of proficiency; (3) the rubric will clearly communicate to the learner what is expected to attain competence, and thus this tool can be used for both assessment and teaching. Other authors have investigated surgical skills outside of actual human surgery. Fisher and associates developed the Eye Surgical Middle East African Journal of Ophthalmology, Volume 21, Number 2, April - June

4 Golnik: Assessment s and Tools Skills Assessment Test (ESSAT); a 3 station (skin suturing, muscle recession, phacoemulsification/wound construction and suturing technique) wet laboratory surgical skills obstacle course for ophthalmology residents. 17 In contrast to other surgical assessments, the ESSAT is designed to evaluate residents basic skills before entering the operating room. Lee and associates developed an ophthalmology wet laboratory curriculum for teaching and assessing cataract surgical competency. 18 The specified curriculum includes pre and post tests of cognitive skills in addition to a structured wet lab curriculum with observed ratings of surgical skill. The same group from Iowa has shown that changes in their surgical curriculum have decreased resident complications during cataract surgery. 19 Finally, surgical simulation may afford non patient methods of surgical skill assessment. Le and associates have shown that the anti tremor and anterior forceps modules of the Eyesi simulator s have construct validity and may be valuable assessment (and teaching) tool in the future. 20 Medical educators have also been devising methods to teach and assess the competencies other than patient care and medical knowledge. Lee and associates described an assessment tool involving a structured assessment of journal club that leads to assessment of practice based learning and improvement. 21,22 Golnik and associates developed and showed validity of a tool to assess resident on call performance, the On Call Assessment Tool (OCAT). 23 The OCAT is a one page checklist to be used retrospectively during random chart review of on call consultations. Finally, a 360 degree evaluation tool can be very helpful in evaluating professionalism and communication skills. The intention is to get feedback of the residents from all the groups they interact. Thus, faculty evaluations can be considered 90 degrees and the patients surveys, ancillary staff and peer evaluations constitute the other 270 degrees. Self evaluations may be utilized in addition. Of course, patients and staff are not asked about the resident s medical knowledge but rather about professionalism and communication skills. Jagadeesan and associates have shown that their patients satisfaction surveys can discriminate levels of resident communication skill and thus may be useful to assess this competency. 24 Internationally, cultural differences may produce difficulties with this type of tool. The information must be anonymous and the evaluators must believe this as the case. Emphasizing to the assessors that this is an attempt to improve the young doctors performance can facilitate acceptance of this method. To my knowledge, no internationally valid 360 degree assessment tool exists and thus the ICO is currently developing one in a manner similar to the ICO OSCARs. CONCLUSION There are accepted principles of assessment that should guide our assessment of resident performance. A number of ophthalmology specific resident assessment tools has been developed. Ideally, every resident training program would utilize a minimum test of medical knowledge, rubrics to assess procedural and clinical skill and a 360 degree evaluation. Assessment of the resident s performance is essential in both teaching and demonstrating the graduating residents, capable to function as competent ophthalmologists. REFERENCES 1. Williams Ten Commandments of Assessment. Available from: html. [Last accessed on 2013 Oct 26]. 2. ACGME Common Program Requirements. Pgs Available from: ProgramRequirements/CPRs2013.pdf. [Last accessed on 2013 Oct 26]. 3. The CanMEDS 2005 Physician Competency Framework. Available from: documentos/canmeds.pdf. [Last accessed on 2013 Oct 26]. 4. The ICO examinations. Available from: [Last accessed on 2013 Oct 26]. 5. Golnik KC, Goldenhar LM, Gittinger JW Jr, Lustbader JM. The Ophthalmic Clinical Evaluation Exercise (OCEX). Ophthalmology 2004;111: Golnik KC, Goldenhar L. The Ophthalmic Clinical Evaluation Exercise (OCEX): Interrater reliability determination. Ophthalmology 2005;112: Cremers SL, Ciolino JB, Ferrufino Ponce ZK, Henderson BA. Objective assessment of skills in intraocular surgery (OASIS). Ophthalmology 2005;112: Cremers SL, Lora AN, Ferrufino Ponce ZK. Global rating assessment of skills in intraocular surgery (GRASIS). Ophthalmology 2005;112: Feldman BH, Geist CE. Assessing residents in phacoemulsification. Ophthalmology 2007;114: Saleh GM, Gauba V, Mitra A, Litwin AS, Chung AK, Benjamin L. Objective structured assessment of cataract surgical skill. Arch Ophthalmol 2007;125: Golnik KC, Beaver H, Gauba V, Lee AG, Mayorga E, Palis G, et al. Cataract surgical skill assessment. Ophthalmology 2011;118:427.e Golnik KC, Beaver H, Gauba V, Lee AG, Mayorga E, Palis G, et al. Development of a new valid, reliable, and internationally applicable assessment tool of residents competence in ophthalmic surgery (An American Ophthalmological Society Thesis). Trans Am Ophthalmol Soc 2013;111: Golnik KC, Haripriya A, Beaver H, Gauba V, Lee AG, Mayorga E, et al. Cataract surgical skill assessment. Ophthalmology 2011;118: e Golnik KC, Gauba V, Saleh GM, Collin R, Naik MN, Devoto M, et al. The ophthalmology surgical competency assessment rubric for lateral tarsal strip surgery. Ophthal Plast Reconstr Surg 2012;28: Golnik KC, Motley WW, Atilla H, Pilling R, Reddy A, Sharma P, et al. The ophthalmology surgical competency rubric for strabismus surgery. J AAPOS 2012;16: Rootman DB, Lam K, Sit M, Liu E, Dubrowski A, Lam WC. Pyschometric properties of a new tool to assess task specific and global competency in cataract surgery. Ophthalmic Surg Lasers Imaging 2012;43: Fisher JB, Binenbaum G, Tapino P, Volpe NJ. Development 112 Middle East African Journal of Ophthalmology, Volume 21, Number 2, April - June 2014

5 Golnik: Assessment s and Tools and face and content validity of an eye surgical skills assessment test for ophthalmology residents. Ophthalmology 2006;113: Lee AG, Greenlee E, Oetting TA, Beaver HA, Johnson T, Boldt C, et al. The Iowa ophthalmology wet laboratory curriculum for teaching and assessing cataract surgical competency. Ophthalmology 2007;114:e Rogers GM, Oetting TA, Lee AG, Grignon C, Greenlee E, Johnson AT, et al. Impact of a structured surgical curriculum on ophthalmic resident cataract surgery complication rates. J Cataract Refract Surg 2009;35: Le TDB, Adatia FA, Lam WC. Virtual reality surgical simulation as a feasible training and assessment tool: Result of a multicenter study. Can J Ophthalmol 2011;46: Lee AG. Using the American Journal of Ophthalmology s website for assessing residency subcompetencies in practice based learning. Am J Ophthalmol 2004;137: Lee AG, Boldt CH, Golnik KC, Arnold AC, Oetting TA, Beaver HA, et al. Using the journal club to teach and assess competence in practice based learning and improvement: A literature review and recommendation for implementation. Surv Ophthalmol 2005;50: Golnik KC, Lee AG, Carter K. Assessment of ophthalmology resident on call performance. Ophthalmology 2005;112: Jagadeesan R, Kaylan DN, Lee P, Stinnet S, Challa P. Use of a standardized patient satisfaction questionnaire to assess the quality of care provided by ophthalmology residents. Ophthalmology 2008;115: e3. Cite this article as: Golnik KC. Assessment principles and tools. Middle East Afr J Ophthalmol 2014;21: Source of Support: Research to Prevent Blindness, Conflict of Interest: None declared. Middle East African Journal of Ophthalmology, Volume 21, Number 2, April - June

Cataract Surgical Skill Assessment

Cataract Surgical Skill Assessment Dear Editor: In the United States, the Accreditation Council for Graduate Medical Education (ACGME) has mandated that all residency training programs (including ophthalmology)