Development and Validation of a Bronchoscopy Competence Assessment Tool in a Clinical Setting

Development and Validation of a Bronchoscopy Competence Assessment Tool in a Clinical Setting Nha Voduc 1, Nancy Dudek 2, Christopher M. Parker 3, Krishna B. Sharma 2, and Timothy J. Wood 4 1 Division of Respirology, Department of Medicine, Ottawa Hospital, Ottawa, Ontario, Canada; 2 Department of Medicine, and 4 Department of Innovation in Medical Education, University of Ottawa, Ottawa, Ontario, Canada; and 3 Department of Medicine, Queen s University, Kingston, Ontario, Canada Abstract Rationale: Flexible bronchoscopy is performed by clinicians representing multiple medical specialties in a variety of settings. Given the increasing importance of competency-based assessment in postgraduate training, it is important that this skill be assessed within a competency-based framework using a valid measurement tool. Objectives: The purpose of this study was to design and validate a practical, competency-based bronchoscopy assessment tool that could be applied to trainees in a clinical setting. Methods: Focus groups of expert physicians were formed in Ottawa, Canada representing adult medical specialties routinely engaged in preparing trainees to perform flexible bronchoscopy (respiratory medicine, critical care, thoracic surgery and anesthesia). The focus groups were charged with identifying themes and items relevant to the assessment of competency in bronchoscopy. By an iterative process, a bronchoscopy assessment tool was developed, the Ontario Bronchoscopy Assessment Tool (OBAT). The tool was evaluated by first using it to assess learners in a pilot study, refining it based on the results, and then testing the OBAT again in a validation study. Measurements and Main Results: The initial tool consisted of 19 items, organized into the following groups: preprocedure planning, sedation and monitoring, technical skill, diagnostic skill, and post-procedure planning. The tool demonstrated high reliability (0.91) and discriminated junior from senior trainees. Based on the results of the pilot, the tool was simplified to a 12-item scale with three subscales: preprocedure planning, technical skills, and postprocedure planning. In the validation study, the assessment tool remained highly reliable (0.92) and discriminated junior from senior trainees with an estimated eight assessments per trainee. Conclusions: The OBAT demonstrates promise as a reliable tool to assess trainee competence for bronchoscopy in clinical settings. Keywords: assessment of clinical competence; bronchoscopy; workplace-based assessment (Received in original form August 21, 2015; accepted in final form February 5, 2016 ) Supported by The Ottawa Hospital Academic Medical Organization Innovation Fund and University of Ottawa Educational Initiatives in Residency Education Fund. Author Contributions: N.V.: study conception, developed and revised the tool, and provided intellectual content in the writing of the manuscript; N.D.: led the initial focus group, developed and revised the tool, and provided intellectual content for manuscript preparation; C.M.P.: developed and revised the bronchoscopy assessment tool, coordinated the enlistment of teachers and trainees at Queen s University for the validation study, and provided intellectual content in the writing the manuscript; K.B.S.: developed and revised the bronchoscopy assessment tool, coordinated the enlistment of teachers and trainees at the University of Ottawa for the validation study, and provided intellectual content in writing the manuscript; and T.J.W.: developed and revised the bronchoscopy assessment tool, conducted the statistical analysis for both validation studies, and provided intellectual content in the writing of the manuscript. Correspondence and requests for reprints should be addressed to Nha Voduc, M.D., Associate Professor, Department of Medicine, Division of Respirology, University of Ottawa at The Ottawa Hospital, General Campus, 501 Smyth Road, Box 211, Ottawa, ON, K1H 8L6 Canada. E-mail: nvoduc@ottawahospital.on.ca This article has an online supplement, which is accessible from this issue s table of contents at www.atsjournals.org Ann Am Thorac Soc Vol 13, No 4, pp 495 501, Apr 2016 Copyright 2016 by the American Thoracic Society DOI: 10.1513/AnnalsATS.201508-548OC Internet address: www.atsjournals.org Training in flexible fiberoptic bronchoscopy is an essential component of several subspecialty training programs, including thoracic surgery, respiratory medicine, and critical care medicine. However, although guidelines on bronchoscopy training have been published (1), training and competency standards remain heterogeneous (2). There is therefore a need for valid metrics for bronchoscopy competency. An ideal tool would be simple, be usable in diverse clinical environments to allow use by multiple specialties, allow repeated assessments to document growing competence, use competency-based Voduc, Dudek, Parker, et al.: Bronchoscopy Competency Assessment Tool Validation 495

behavioral anchors to improve interrater reliability, and assess all of the components of the procedure to facilitate formative feedback and summative competency evaluation. Existing tools for the assessment of competency with bronchoscopy include the Bronchoscopy Skills and Tasks Assessment Tool and the Bronchoscopy Step by Step Evaluation Tool (3). These tools can distinguish between novices and pulmonary fellows. However, they require specialized simulation equipment, have not been validated in a real-time clinical setting, and are not designed for repeated application to track individual trainee skill acquisition. More recently, Konge and colleagues developed a simple assessment tool for use in a clinical setting (4). Their tool, however, focused on the technical aspects of bronchoscope manipulation and description of findings and lacked assessment of other important components of bronchoscopy: patient monitoring, sampling, and post-procedure management and documentation. Therefore, our objectives were to develop and collect validity evidence for an assessment tool that could assess not only technical proficiency in bronchoscopy but also all clinically relevant areas of the procedure. Preliminary results of this study have been previously presented in abstract form at the Canadian Conference for Medical Education (2014) (5) and at the Academy of Innovation in Medical Education conference, hosted by the University of Ottawa (2015) (6). Methods The study was approved by the Ottawa Health Science Network Research Ethics Board (protocol 20130069-01H) and Queen s University Ethics Board (protocol 6007786 DMED-1568-13). It was completed in three phases: tool development, piloting to assess the tool validity (including reliability), followed by an informed revision and repeat validity evaluation of the revised tool. Tool Development A multidisciplinary focus group was assembled, including experienced clinician teachers from the University of Ottawa representing the adult medical specialties of respirology, thoracic surgery, critical care medicine, and anesthesiology. Additional members included a respirology trainee and experts in medical education. The group identified the items to be included in the tool and the scoring system. The draft tool was modified based on the feedback from focus group members and other clinician teachers from both the University of Ottawa and Queen s University. This cycle was repeated until all participants were satisfied with the prototype (Figure 1). A more detailed description of the focus group and the development process is available in the online supplement. Pilot Study An initial pilot study of the tool was conducted between March and September 2013 to assess its psychometric properties and its ability to distinguish between junior and senior subspecialty trainees. Trainees from the respirology, thoracic surgery, and critical care medicine training programs at the University of Ottawa and the respirology program from Queen s University were invited to enroll. Involvement was voluntary and confidential. Junior trainees in these 2-year programs had some prior experience with bronchoscopy but were not expected to be competent for independent practice, whereas senior subspecialty trainees were expected to have attained or be nearing competence for independent practice. All participating trainees and supervising attendings were shown the tool at the time of consent. Most of the attendings had previously seen one or more versions of the tool during development. To familiarize the remainder, a 10-minute presentation was provided at their divisional education committee meeting. No other training or mentoring was provided. Attendings were asked to complete the assessment tool immediately after supervising the bronchoscopy. For analysis, trainees were divided into three groups: (1) junior subspecialty trainees who were starting their subspecialty training, (2) midlevel subspecialty trainees who were completing their first year of subspecialty training and starting their second during the study period, and (3) senior subspecialty trainees who were completing their training during the study period. The psychometric characteristics of the initial tool were assessed using descriptive statistics for each item on the tool, including means, frequencies, missing data, and correlations between items. A generalizability analysis was conducted to determine the reliability of the scores, adjusting for the trainee, the assessment items used, the raters, and differences between cases. The findings of the first validation study were circulated to the original focus group members to inform the additional revisions. Validation Study The revised tool, the Ontario Bronchoscopy Assessment Tool (OBAT), contained 12 items (Appendix). For the primary objective, we once again chose to study the OBAT s ability to differentiate between different levels of trainees, as we believed that training level would provide the simplest surrogate measure of competency in the absence of an established gold standard. The study data were collected between July 2014 and December 2014, sufficient time to permit multiple measurements of each trainee by different raters but short enough to limit the learning effect within each trainee produced by experience. For the validation study, trainees were divided into two groups for analysis, to ensure sufficient power to detect significant differences between groups: 1. First-year (post-graduate year [PGY]-4) subspecialty trainees, enrolled during the first 6 months of subspecialty training (July 2014 to Dec 2014). All trainees of this level had completed at least 8 weeks of respirology training during their internal medicine residency and thus had some previous exposure to bronchoscopy. Many trainees also received simulator training during the first week of their PGY4 year, before study enrolment. 2. Second-year (PGY-5) subspecialty trainees. These trainees had 12 months of previous subspecialty training and were expected to be approaching or have attained competency in bronchoscopy. Trainees were enrolled for a 6-month period during their second year of training. Supervising attendings were asked to complete the OBAT immediately after completing a diagnostic bronchoscopy with a trainee. Interventional procedures and endobronchial ultrasound procedures were not evaluated. 496 AnnalsATS Volume 13 Number 4 April 2016

Bronchoscopy Assessment Tool Staff: Trainee: Date: Level: Relative complexity of procedure: Low Medium High Samples Obtained / Interventions Performed Scale 1- I had to take over 2- I had to talk the trainee through 3- I had to prompt the trainee from time to time 4 - I needed to be in the room just in case 5- I did not need to be there N/A- not applicable The purpose of this scale is to evaluate the trainee s ability to perform all activities related to bronchoscopy safely and independently. The scores provided should reflect the trainee s performance in this case and should not be based on level of training. N/A (not applicable) should only be selected if the item is not relevant to the case (e.g. localization of vocal cords in an intubated patient). If an item was not assessed because the supervisor had assumed control, 1 should be selected. 1. Pre-procedure planning Performs appropriate pre-procedure evaluation of patient, including review of pertinent investigations Communicates procedure plan and precautions with bronchoscopy team, ensuring availability of appropriate equipment N/A N/A 2. Sedation and Monitoring Selects and administers (or directs administration of) appropriate sedation and local anesthesia N/A Attentive to patient s vital signs and comfort during procedure; modifies procedure or sedation appropriately N/A 3. Technical Localizes / visualizes vocal cords N/A Intubates trachea N/A Maintains good orientation /localization during procedure; correctly identifies bronchial N/A anatomy Performs inspection in an orderly manner N/A Maintains good visualization of bronchi during procedure while avoiding excessive trauma to N/A mucosa Demonstrates good body and bronchoscopy mechanics N/A Able to obtain appropriate samples and/or perform appropriate intervention N/A Avoids complications if possible or recognizes and manages complications during procedure if they arise Performs procedure efficiently (completes procedure in a timely manner without degradation in quality) N/A N/A 4. Diagnosis Recognizes abnormalities and appropriately describes bronchoscopy findings N/A 5. Post-Procedure Ensures appropriate processing of samples (labels, samples, completes requisitions) N/A Develops post-operative plan (provides appropriate verbal and written orders as appropriate) N/A Assesses for and manages post-procedure complications N/A Communicates bronchoscopic findings to patient and/or caregiver as appropriate N/A Documents procedure appropriately N/A Suggestions for improvement: Figure 1. A preliminary draft of the nineteen-item Bronchoscopy Assessment Tool. Voduc, Dudek, Parker, et al.: Bronchoscopy Competency Assessment Tool Validation 497

Statistical Analysis and Sample Size A total OBAT score was derived by averaging the scores across all items on the scale. To determine reliability, we required a balanced design with ratings on all items. To avoid having to exclude all forms with one or more unscored items, the 12 items on the scale were grouped into three subscales based on related content (Appendix). The subscales were preprocedure (which included preprocedure assessment and sedation, items 1 and 2), technical (which included diagnosis, items 3 10), and post-procedure related items (items 11 and 12). Subscale totals were calculated by deriving the average score for the scored items within that subscale. Therefore, as long as the form had at least one scored item in each subscale it could be included. This procedure allowed us to include OBAT forms that were missing data but also allowed a limited analysis of the items. The psychometric characteristics of the OBAT were determined by calculating the descriptive statistics for the items and the subscales as well as the correlation between the ratings. Reliability of the OBAT was determined using a generalizability analysis of the total OBAT ratings with OBAT forms nested with participant (i.e., trainee) and participant nested within PGY level and crossed with subscale. In addition to reliability, the generalizability analysis also allows one to identify the degree that each factor contributes to the variability in the scores, as well as the average number of procedures that must be rated to produce a reliable score of bronchoscopy competency. To determine if the tool could discriminate between performances by different training levels, an analysis of variance was conducted using the scale mean (mean score of all completed tools for a given participant) as the dependent measure and training level as a grouping factor. Sample size for the validation phase was based on a convenience sample of 15 residents who were enrolled in respirology (at University of Ottawa and Queen s University), critical care, and thoracic surgery (at University of Ottawa) within a year. If each trainee was evaluated for 10 bronchoscopies, it should lead to a sample of 150 completed forms. Given the number of items on the OBAT, this sample of forms was deemed sufficient to achieve a reliability of 0.80 with a 95% confidence interval that reliability will fall between 0.75 and 0.84. Results Tool Development The focus group agreed on a total of 19 separate items. The group decided that a rating system previously developed by Gofton and colleagues for the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) was best suited for the needs of the tool, as it was focused on competency as opposed to rating a resident according to the expectations for performance based on level of training (7). This scale also incorporates the concept of entrustability, which captures the progression to independent competence and is easily understood by educators (8). Scores range from 1 ( I had to do i.e., requires complete hands-on guidance, did not do, or was not given the opportunity to do) to 5 ( I did not need to be there i.e., complete independence, understands risks and performs safely, practice ready). Pilot Study Twenty-four trainees were enrolled and 145 tools were completed by 19 faculty members engaged in supervision of bronchoscopies performed in a variety of settings (bronchoscopy suite, operating room, and intensive care unit). Mean 6 SD scores from all 145 tools were statistically significantly different for group 1 (early trainees; 3.79 6 1.00) versus group 3 (late trainees; 4.73 6 0.38; P = 0.02) Group 2 (midlevel trainees) had a mean rating of 4.23 6 0.89 but did not differ from either group 1 or 3. Reliability was 0.91, with seven forms per person. A reliability of 0.80 could be achieved with just three forms per person. Correlation between ratings on items ranged from 0.59 to 0.95, suggesting that some items may be redundant (see online supplement for more detailed pilot results and analysis). Validation Study Based on the results and iterative feedback from participants and the focus group, the tool was modified to provide specific instructions for bronchoscopies in intubated patients, omitting two nonrelevant items when used in an intubated patient. When thematically appropriate, highly correlated (.0.90) items were merged. The number of items was reduced from 19 to 12. Based only on correlations between item scores, it would have been possible to reduce the number of items further, but focus group members believed that would diminish the ability of the tool to provide specific feedback to the trainee. Please see the online supplement for the details of the changes between the original and final versions of the tool. The validation study conducted with the revised tool included a total of 148 OBAT forms completed on 19 trainees by 15 attending staff, for an average of 7.79 forms per trainee. For PGY-4 trainees, there were 12 trainees and 86 forms (mean of 7.17 forms per person) and a mean 6 SD rating of 4.23 6 0.67. For PGY-5 trainees, there were seven trainees with 62 forms (mean of 8.86 forms per person) and a mean rating of 4.52 6 0.58 (P = 0.008). The frequencies with which each rating was chosen for all of the 12 items are displayed in Table 1. The majority of items had ratings that ranged from 3 to 5, although the full range of scores was used for many items on the tool. Fifty-three of 148 tools had missing data for at least one of the 12 items. The missing data were largely due to the use of the OBAT for intubated patients, for whom two items do not apply. Thirty-one (21%) of the observed bronchoscopies were performed on intubated patients. Some missing data were also attributed to the performance of bronchoscopy in operating room or ICU settings, when the trainee was not involved in pre- or post-procedure management. The distribution of missing items is also shown in Table 1. For the generalizability analysis, any form missing a subscale (i.e., no rating for any of the items in that subscale) was removed from the analysis to ensure equal number of ratings per trainee across the three subscales. This process eliminated 13 forms, leaving 135 forms from 18 participants (12 PGY4s and 6 PGY5s), completed by 14 attending staff. There were 6.17 and 10.17 forms per person for PGY4s and PGY5s, respectively. Table 2 displays the results of the generalizability analysis as well as an interpretation of each variable. Training level (facet l) accounted for 24% of the variance in scores. People within training level (facet p:l) accounted for 17% of the variance in scores, which indicates 498 AnnalsATS Volume 13 Number 4 April 2016

Table 1. The frequency of scale rating selection for each item from 148 completed tools Subscale/Question that there was some variation across people within a particular training level. As shown in Table 2, subscales by themselves did not account for a large portion of variance; however, the people by subscale within level (facet Ps:l) accounted for 22% of the variability. This finding demonstrates that for any given trainee, there were some differences in their ratings on the three subscales and suggests that the subscales are assessing different aspects of bronchoscopy competency. The variance components in Table 2 were used to determine the reliability of the OBAT. With 7.5 (rounded to 8) forms per person, the reliability is 0.92. With three forms per person, the reliability would equal 0.82. Table 3 displays the mean ratings for each subscale and the scale total for the 135 forms containing data in all three subscale Ratings Missing Pre-procedure planning 1. Pre-procedure evaluation 1 0 9 47 89 3 2. Selects/administers sedation 5 5 36 43 25 34 Technical skills 3. Localizes vocal cords 2 3 15 30 56 42 4. Maintains good orientation 0 3 23 40 82 0 5. Maintains good visualization 0 2 21 39 86 0 6. Good body mechanics 0 2 20 45 81 0 7. Obtains appropriate samples 8 4 22 40 74 2 8. Recognizes and manages complications 2 0 21 42 77 6 9. Efficient 0 3 30 58 57 0 10. Recognizes abnormalities 0 0 18 50 80 0 Post-procedure 11. Develops post-procedure plan 0 0 8 48 80 13 12. Communicates findings 0 0 13 44 68 23 domains. There was a significant difference in mean total ratings between senior (PGY- 5) and junior (PGY-4) trainees (4.50 vs. 4.13, P, 0.001). Additional results and analysis areshownintheonlinesupplement. Discussion We have developed a 12-item tool, the OBAT, to assess competency in bronchoscopy. The OBAT was designed to assess all clinically relevant aspects of bronchoscopy from initial setup to post-procedure documentation and uses a competency-based rating system based on entrustability. Furthermore, the OBAT was designed to be sufficiently brief to be used in a clinical setting and sufficiently simple to be used by experienced clinical teachers with Table 2. Generalizability analysis for Ontario Bronchoscopy Assessment Tool (n = 135 tools) Facet Variance % Variance Interpretation l 0.13 24 Variance due to differences between training levels p:l 0.09 17 Variance due to differences between trainees within a training level f:p:l 0.03 6 Variance due to differences in scores the raters provided on any given trainee s form s 0 0 Variance due to differences between subscales ls 0.05 9 Variance due to difference between subscales for any training level ps:l 0.11 22 Variance due to trainees having different ratings on the subscales fs:p:l 0.12 23 Overall error Definition of abbreviations: f = forms; l = training level; p = participants (trainees); s = subscales. minimal additional training. Our findings indicate that the OBAT can provide a reliable rating of bronchoscopy competency with as few as three observed procedures for trainees from a number of different specialties and in a variety of clinical settings and can distinguish between junior and senior trainees. The statistically significant difference in mean scores between PGY-4 and PGY-5 trainees may appear numerically small from a purely numerical perspective (0.37). However, the difference of a full point on this scale is large from an educational perspective. A score of 4 still requires the trainee to be supervised, but a score of 5 indicates the ability to perform the bronchoscopy independently with no supervision. Given that OBAT scores for each trainee were collected over a 6-month period, it would not be expected to see a full point difference between the two groups. There are some limitations to this study. First, the assessor could not be blinded to the training level of the trainee, potentially biasing their assessment. However, our study design reflected the real-world application of the tool, in which the supervisor would typically ascertain the experience of the trainee before any procedure. We acknowledge that future studies could address the issue of blinding using video recordings. A second limitation is that we did not include truly novice learners in our validation studies. All trainees enrolled in our study had some prior bronchoscopy experience before starting their subspecialty training. This may have reduced the range of responses, and it also remains unknown how well the OBAT would measure lower levels of competence. A third limitation is that we designed this as a cohort study, comparing ratings between more junior and more senior trainees, with each set of ratings obtained in a relatively short time interval for each trainee to minimize the effects of their own training. A more relevant design might have been to track the ratings of individual trainees over time to see how sensitive the OBAT was to their learning and to correlate the OBAT ratings to other measures of procedural competence. By eliminating intertrainee differences, it is likely that such a design would have been even more sensitive to changes in competence, but that remains untested. Voduc, Dudek, Parker, et al.: Bronchoscopy Competency Assessment Tool Validation 499

Table 3. Descriptive statistics for the Ontario Bronchoscopy Assessment Tool by training level (n = 135 tools) Level Pre Technical Post Total 4 Mean 4.0608 4.1081 4.3108 4.1275 No. 74 74 74 74 SD 0.76280 0.71388 0.57756 0.64304 Minimum 2.00 2.38 3.00 2.91 Maximum 5.00 5.00 5.00 5.00 5 Mean 4.4016 4.4856 4.6885 4.4967 No. 61 61 61 61 SD 0.63127 0.61900 0.56394 0.57219 Minimum 3.00 2.75 3.00 3.00 Maximum 5.00 5.00 5.00 5.00 Total Mean 4.2148 4.2787 4.4815 4.2943 No. 135 135 135 135 SD 0.72416 0.69624 0.59977 0.63708 Minimum 2.00 2.38 3.00 2.91 Maximum 5.00 5.00 5.00 5.00 Unfortunately, the influence of the specific patient encounter is confounded with the rater and cannot be separated. Given that this combined facet only accounts for 6% of the variance in the ratings, the small amount of variance suggests that there is high agreement between any set of raters. To definitively address this issue, future studies could include two assessors or use video recording to allow a second assessor to evaluate the learner. In conclusion, the OBAT demonstrates promise as a tool to assess competency in bronchoscopy in a diverse range of clinical settings. n Author disclosures are available with the text of this article at www.atsjournals.org. Finally, we did not report either the internal consistency of the subscales or the interrater reliability. We chose to use a generalizability analysis to derive the reliability of the OBAT. The strength of this analysis is that it considers the simultaneous influence of a number of factors on reliability including the subscales and is therefore a more accurate measure of reliability than we would have obtained by just reporting the internal consistency of the subscale. Regarding interrater reliability, only one rater completed each form; therefore, it was notpossibletoderiveameasureof interrater reliability in this study. The closest facet that could represent a proxy for interrater reliability is the f:p:l facet in the model of Table 2, because this facet captures the rating assigned by each rater. Acknowledgment: The authors thank Dr. Carla Lamb for her insight on the need for a bronchoscopy assessment tool that could be usable in a clinical setting. They also thank the following physicians for their participation in the original focus group and follow-up discussions: Kaissa De Boer, Sebastian Gilbert, Donna Maziak, Viren Naik, and Rakesh Patel. The authors also thank Ms. Rosalie Labelle for her assistance in the preparation of the manuscript; and the editor and anonymous reviewers for their helpful comments and suggestions for the manuscript. References 1 Ernst A, Silvestri GA, Johnstone D; American College of Chest Physicians. Interventional pulmonary procedures: guidelines from the American College of Chest Physicians. Chest 2003;123:1693 1717. 2 Haponik EF, Russell GB, Beamis JF Jr, Britt EJ, Kvale P, Mathur P, Mehta A. Bronchoscopy training: current fellows experiences and some concerns for the future. Chest 2000;118:625 630. 3 Davoudi M, Osann K, Colt HG. Validation of two instruments to assess technical bronchoscopic skill using virtual reality simulation. Respiration 2008;76:92 101. 4 Konge L, Larsen KR, Clementsen P, Arendrup H, von Buchwald C, Ringsted C. Reliable and valid assessment of clinical bronchoscopy performance. Respiration 2012;83:53 60. 5 Voduc N, Dudek N, Wood T. Validation of a new bronchoscopy assessment tool in a clinical setting [abstract]. Presented at the 12th Canadian Conference on Medical Education 2014 (16th Ottawa Conference). April 28, 2014, Ottowa, Ontario, Canada. Abstract OTT-PA5-11. 6 Voduc N, Dudek N, Wood T. Validation of a new bronchoscopy assessment tool in a clinical setting [abstract]. Presented at the AIME Medical Education Day 2015. April 10, 2015, Ottowa, Ontario, Canada. 7 Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med 2012;87:1401 1407. 8 Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. Entrustability scales: outlining their usefulness for competency-based clinical assessment. Acad Med 2016;91:186 190. 500 AnnalsATS Volume 13 Number 4 April 2016

Ontario Bronchoscopy Assessment Tool Staff: (study number) Date: Trainee: (study number) Relative complexity of procedure: Low Medium High Intubated Patient: Yes No Samples Obtained / Interventions Performed Scale 1- I had to take over 2- I had to talk the trainee through 3- I had to prompt the trainee from time to time 4 - I needed to be in the room just in case 5- I did not need to be there Pre-procedure planning 1. Performs appropriate pre-procedure evaluation of patient, including review of pertinent investigations; discusses procedure with patient, obtains consent; communicates plans with bronchoscopy team The purpose of this scale is to evaluate the trainee s ability to perform all activities related to bronchoscopy safely and independently. The scores provided should reflect the trainee s ability to competently perform tasks relevant to bronchoscopy and not be based on expectations related to level of training. For intubated patients, omit questions 2 and 3. If an item was not assessed because the supervisor had assumed control, 1 should be selected. Sedation and Monitoring 2. Selects and administers (or directs administration of) appropriate sedation and local anesthesia; attentive to patient s vitals and comfort during the procedure, modifies procedure and sedation appropriately Technical 3. Localizes / visualizes vocal cords / intubates trachea 4. Maintains good orientation /localization during procedure; correctly identifies bronchial anatomy and performs inspection in an orderly manner 5. Maintains good visualization of bronchi during procedure while avoiding excessive trauma to mucosa 6. Demonstrates good body and bronchoscopy mechanics 7. Able to obtain appropriate samples and/or perform appropriate intervention 8. Avoids complications if possible or recognizes and manages complications during procedure if they arise 9. Performs procedure efficiently (completes procedure in a timely manner without degradation in quality) Diagnosis 10. Recognizes abnormalities and appropriately describes bronchoscopy findings Post-Procedure 11. Develops post-procedure plan (provides appropriate verbal and written orders as appropriate); Ensures appropriate processing of samples (labels, samples, completes requisitions); Assesses for and manages post-procedure complications 12. Communicates bronchoscopic findings to patient and/or caregiver as appropriate; Documents procedure Suggestions for improvement: Appendix. The Ontario Bronchoscopy Assessment Tool. Voduc, Dudek, Parker, et al.: Bronchoscopy Competency Assessment Tool Validation 501