Free Online Otolaryngology Educational Modules A Pilot Study

Transcription

1 Research Original Investigation A Pilot Study Cristina Cabrera-Muffly, MD; Paul C. Bryson, MD; Kevin J. Sykes, PhD, MPH; Yelizaveta Shnayder, MD IMPORTANCE Otolaryngology residents need concise, easily accessible modules to expand educational opportunities between surgical cases. These modules should be inexpensive to create and improve learning outcomes. OBJECTIVE The purpose of this pilot study was to assess whether otolaryngology residents at multiple institutions used online video modules to supplement their studying for the Otolaryngology Training Exam, whether the modules had any effect on their Otolaryngology Training Examination Scores, and to obtain survey feedback about the modules. DESIGN, SETTING, AND PARTICIPANTS This randomized trial was conducted in 3 academic departments of otolaryngology in the United States among 37 residents enrolled in 3 otolaryngology residency programs. INTERVENTIONS Residents were randomized into 2 groups, one with access to the educational modules and the other with no access. MAIN OUTCOMES AND MEASURES Otolaryngology training examination scores were obtained from the year prior to the intervention (2012) and the year following module access (2013). Residents with access to the modules were also surveyed to assess use and obtain feedback about the modules. RESULTS Otolaryngology training examination scores improved significantly from 2012 to 2013 among both residents who had access to the modules and those who did not in the sections of head and neck, laryngology, and sleep medicine. However, scores in the sections of pediatric otolaryngology (8% increase, P =.03), otology (7% increase, P =.02), and facial plastic surgery (10% increase, P =.02) improved from 2012 to 2013 only among residents with access to the modules. All respondents rated the videos as very helpful, with a rating of 4 of 5 on a Likert scale. CONCLUSIONS AND RELEVANCE Online otolaryngology educational modules are an inexpensive way to expand resident learning opportunities. Despite the lack of quantifiable improvement in otolaryngology training examination scores in this study, use of online modules sends a message to otolaryngology residents that their education is a priority; self-study outside the hospital and clinics is necessary and expected; and that instructors are willing to try more nontraditional or progressive forms of education to increase resident knowledge. JAMA Otolaryngol Head Neck Surg. 2015;141(4): doi: /jamaoto Published online February 19, Author Affiliations: Department of Surgery, University of Colorado School of Medicine, Aurora (Cabrera-Muffly); Department of Surgery, Cleveland Clinic, Cleveland, Ohio (Bryson); Department of Surgery, University of Kansas School of Medicine, Kansas City (Sykes, Shnayder). Corresponding Author: Cristina Cabrera-Muffly, MD, Department of Surgery, University of Colorado School of Medicine, E 17th Ave, MS B205, Aurora, CO (Reprinted) jamaotolaryngology.com

2 Original Investigation Research Surgical education has not changed in many years. The traditional model of classroom, bedside, and operating room teaching continues to be the mainstay of resident education. As digital technologies emerge, however, there is greater potential for other teaching methods. Prior studies have shown that web-based instruction modules can effectively supplement traditional teaching methods to achieve learning outcomes. 1,2 Benefits to the use of web-based supplemental resident education include the appeal to learners familiar with digital technology, increased time flexibility, and ability to standardize the curriculum and assessment. Previous studies have shown that learners exhibiting technology readiness are more likely to be optimistic and innovative with new educational technologies. 3 Younger students (age <25 years in 2008) were found to have higher rates of technology readiness. 3 As learners are exposed to technology earlier in their childhood, this age boundary will likely continue to increase. Web-based instruction modules also provide for increased flexibility, both for faculty teaching residents and for residents looking to improve their knowledge and conceptualization of complex surgeries. The power of online, asynchronous learning allows content (eg, videos, self-quizzes, and other tools) to be accessed any time of day or night by surgical residents with busy schedules. When created through responsive design, videos can be viewed on any digital device, be it a laptop or smart phone, allowing for portable learning opportunities. This also allows for a flipped classroom model, where residents can better prepare in advance for upcoming surgical cases or other teaching moments. 4 Web-based modules can make teaching time more efficient by having residents prepare ahead of time. Furthermore, as prior studies have shown, the flipped classroom model can increase both test scores and enthusiasm for learning. 5,6 Unlike operative training, web-based modules can be used to standardize education. Instead of residents performing cases with various attending physicians, the case or topic being reviewed is the same each time for each resident. Currently in otolaryngology education, each residency program varies in terms of number of faculty and comprehensiveness of subspecialty representation among the faculty. For example, a program may have 3 otologists and no sleep specialists, which skews resident operative and clinic experience. Some programs have 2:1 faculty to resident ratios, while some have more residents than faculty. While the American Board of Otolaryngology put together a Comprehensive Core Curriculum 7 in 2007, there is no requirement that all programs have specific numbers or types of faculty members. Certainly, faculty to resident ratios and overall clinical productivity will affect availability of teaching time even from week to week. Residents on rotation at varying times during the year may have completely different experiences. Use of web-based modules standardizes both exposure to content and assessments for each resident. Web-based modules can standardize assessment of residents by evaluating the same knowledge and skills in the same manner for each learner. For example, for teaching anatomy, a model can be used with labels that the residents need to correctly attach to the pertinent structures. Anatomy knowledge can be easily stratified by the resident s scores, and areas where a specific resident needs further coaching to achieve topic mastery become evident. In our goal to continue to train excellent otolaryngologists, determining the efficacy of digital technologies as a supplement to traditional teaching methods for the improvement of resident knowledge and skill is a worthwhile endeavor. The purpose of this pilot study was to assess whether otolaryngology residents at multiple institutions used online video modules to supplement their studying for the otolaryngology training examination (OTE), whether the modules had any effect on their OTE scores, and to obtain survey feedback about the modules. Methods Institutional review board approval was obtained from the Colorado Multiple Institution Review Board, the Cleveland Clinic Institutional Review Board, and the University of Kansas Medical Center Human Research Protection Program. The need for written informed consent was waived by all review boards. Online Modules Online modules were created for supplemental otolaryngology resident education. Videos were produced using Microsoft Windows Paint ( /en-us/windows7/products/features/paint) as the background for the images, a Wacom Bamboo tablet (Intuos) to allow for drawing and writing on the background, HyperCam 2 software ( to record the video, and a microphone to record sound. The videos were edited with Microsoft Windows Movie Maker ( The Wacom Bamboo tablet was specifically chosen because it was successfully used to create videos on a free, online educational website ( Inspiration for these otolaryngology modules was drawn from this educational website. Videos were each made shorter than 15 minutes for 2 reasons: First, research has suggested that the average attention span is approximately 15 minutes and that a student s retention of material precipitously drops after this period. Second, YouTube will not upload videos longer than 15 minutes without a verified account. The general format of each video included both an audio description of the topic and pertinent learning points along with a visual representation of the pertinent anatomy or learning points. For example, in the audiogram interpretation lecture, the diagrams were visual representations of audiograms and tympanograms, whereas in the laryngeal anatomy lecture, public domain anatomy cartoons were used. Some videos contained deidentified patient photographs to illustrate clinical findings. To illustrate the format more specifically, we describe the video on external carotid anatomy. The video opens with a cartoon of the head and neck in profile. A brief introduction is narrated about why this topic is relevant. The viewer is oriented jamaotolaryngology.com (Reprinted) JAMA Otolaryngology Head & Neck Surgery April 2015 Volume 141, Number 4 325

3 Research Original Investigation Table 1. Resident Participant Characteristics Characteristic Residents, No. (%) Sex Male 21 (57) Female 16 (43) Institution 1 13 (35) 2 12 (32) 3 12 (32) Access Yes 18 (49) No 19 (51) PGY level during 2013 test 2 8 (22) 3 10 (27) 4 11 (30) 5 8 (22) Abbreviation: PGY, postgraduate year. to the general anatomy and hears specific descriptions of the neck cartilages and the common carotid. The branches of the external carotid artery are then sequentially drawn on the cartoon; each vessel is named in order from inferior to superior in accordance with the direction of blood flow. As each branch is named, the structures supplied by that vessel are described. Finally, the names of the branches are reviewed a final time, and the viewer is encouraged to draw the anatomy for themselves to reinforce their knowledge. Once finalized, each video was posted onto a private You- Tube channel. A free blog site, Wordpress.blogspot.com, was used to link to the completed videos. A total of 10 videos were piloted. Six videos covered laryngology topics, including laryngeal anatomy, benign laryngeal lesions, benign laryngeal neoplasms, vocal cord immobility, congenital laryngeal lesions, and airway stenosis. Two videos discussed audiology topics, while 2 discussed general head and neck anatomy. Cost The website used to post the web-based modules is free once registered. Posting videos to YouTube is also free. The Hyper- Cam 2 software used to record the video is available for free online. Microsoft Windows Paint and Microsoft Windows Movie Maker come with almost every Windows-based computer or can be downloaded for free. The only true costs to make the videos were therefore the microphone ($20) and the Wacom Bamboo tablet ($80). These costs were personally funded by the first author (C.C.-M.). Resident Exposure to Modules Residents from all 3 institutions were randomly assigned to either have access (experimental arm) or not have access (control arm) to the online modules prior to the 2013 OTE. Scores were collected from both the 2012 OTE and the 2013 OTE. Only residents who took both tests were included in the study. The OTE is divided into 9 subspecialty sections, including allergy, fundamentals, pediatric otolaryngology, rhinology, otology, head and neck, laryngology, facial plastic surgery, and sleep medicine. The percentage correct scores were collected from each section for each resident. A survey was sent to all residents who had access to the online modules after the 2013 OTE was complete to assess use and obtain feedback about the modules. Data Analysis Paired t tests were first performed to compare each resident s percentage correct scores from the 2012 and 2013 OTEs. JMP Pro 11 software was used to perform all analyses. Subsequently, score data were analyzed in 2 groups, one group from subjects with access to the modules and the other group with no access. The mean change in scores within each topic was calculated for access and no access groups. Results A total of 37 otolaryngology residents were included in the study. Residents from each institution were randomized so that approximately 50% had access to the modules. See Table 1 for resident characteristics. Only 3 residents had a significant difference in their overall scores from the 2012 to 2013 OTE (all were increased scores). One of these 3 residents was in the experimental arm, while the other 2 had no access to the modules. When comparing 2012 and 2013 OTE scores among all residents by topic, there were no differences in scores within the allergy, fundamentals, or rhinology test sections. All of the other test sections showed an overall increase in scores in 2013 compared with 2012, including head and neck, laryngology, otology, pediatric, facial plastic surgery, and sleep. The 2012 individual topic scores were compared between the experimental arm and the control arm to determine if there was a difference prior to any intervention. Mean scores were the same between the 2 groups in every topic except for allergy, in which the access group trended toward a higher mean score (P =.048). The 2013 mean individual topic scores did not show any difference between access and no access groups. The OTE scores improved significantly from 2012 to 2013 among both residents who had access to the modules and those who did not in the sections of head and neck, laryngology, and sleep medicine (Table 2 and Table 3). Scores in the sections of pediatric otolaryngology, otology, and facial plastic surgery improved from 2012 to 2013 only among residents with access to the modules. Scores decreased among residents with access to the modules in the allergy section. After the 2013 OTE, a survey was distributed by to all residents who had access to the modules. Of the 18 residents who had access, there was a 22% response rate. Those who responded reported they had watched a mean of 66% of the available lectures. All respondents rated the videos as very helpful, with a rating of 4 of 5 on a Likert scale. Fifty percent of the respondents believed that the videos had not helped them to answer any questions on the OTE correctly, while the other 50% believed the videos may have helped them answer a few questions correctly. Although the surveys were anonymous, it was possible to track website traffic after the resi- 326 JAMA Otolaryngology Head & Neck Surgery April 2015 Volume 141, Number 4 (Reprinted) jamaotolaryngology.com

4 Original Investigation Research Table 2. OTE Scores Among Residents With Access to Modules OTE Score, Mean (SD) OTE Section Change, % P Value Allergy 79 (12.0) 66 (15.2) 13 <.001 Facial plastics 59 (12.1) 69 (10.6) Fundamentals 74 (9.8) 71 (9.6) 3.30 Head and neck 57 (13.1) 67 (10.8) Laryngology 57 (15.9) 71 (12.7) Otology 60 (9.9) 67 (12.0) 7.02 Pediatric 70 (13.8) 78 (12.2) 8.03 Rhinology 63 (14.2) 66 (12.2) 3.45 Sleep 63 (20.3) 79 (17.1) Abbreviation: OTE, otolaryngology training examination. Table 3. OTE Scores Among Residents With No Access to Modules OTE Score, Mean (SD) OTE Section Change P Value Allergy 68 (18.6) 69 (15.2) 1.87 Facial plastics 63 (10.8) 67 (11.0) 4.06 Fundamentals 72 (9.8) 72 (8.1) 0.95 Head and neck 56 (13.0) 65 (12.3) 9.01 Laryngology 53 (15.0) 67 (12.8) Otology 62 (12.1) 66 (17.0) 4.38 Pediatric 69 (10.4) 71 (10.6) 2.52 Rhinology 62 (9.0) 61 (13.3) 1.74 Sleep 68 (17.5) 79 (13.8) Abbreviation: OTE, otolaryngology training examination. dents were given access to the modules. Since the website was private and not accessible by web search, the traffic was considered to be by residents only. Residents were given access beginning in January prior to the OTE in March. The page was viewed 31 times in January, 91 times in February, and 87 times in March. Discussion Online modules about a variety of otolaryngology topics can be a useful supplement to traditional didactic lectures and bedside clinical teaching. This study was meant as a pilot to describe the creation and implementation of otolaryngology topic modules online for very little cost as an additional teaching tool for resident education. With approximately $100 in startup cost, it is possible to create and post hundreds of videos to supplement the traditional didactic curriculum. The second part of this pilot study was to attempt to quantify the impact of such modules on OTE scores. Since the study compared the same group of residents from one year to the next, we expected that the residents would significantly increase their scores overall in each topic section. This was not the case: scores in the topics of allergy, fundamentals, and rhinology did not change from 2012 to Scores in each of the other topics did increase overall. Furthermore, when looking at individual residents, we found that only 3 significantly increased their overall scores from 2012 to Overall 2013 OTE scores did not differ between resident groups who either did or did not have access to the modules. There were, however, significant improvements in the experimental arm vs the control arm in 3 of the subspecialty sections (facial plastic surgery, otology, and pediatric otolaryngology). There are several likely reasons for this finding. First, there are many factors responsible for significant improvements in OTE scores, including exposure to didactic curriculum, clinical experiences, and self-study. The online modules were meant to be a supplement, and we did not control for the other methods of learning. Second, residents would be expected to empirically improve their scores from year to year since they are participating in year-round instruction and have repeated exposure to the OTE each year. It is difficult to differentiate between expected progressive improvements and improvements due to other factors. Third, since the online modules were meant as a pilot, the topics covered were limited and would not be expected to cause improvement in most of the subspecialty sections. The majority of videos represented laryngology topics, so we would not necessarily expect differences in the OTE scores on other sections directly from use of the video modules. Fourth, the web-based blog was used without tracking software, so it was difficult to quantify what percentage of the modules the residents actually watched. Finally, while we can quantify the change in scores that is statistically significant, it is difficult to quantify how much of a difference in OTE scores translates to clinical significance in board passage rates or improved patient care. Since performance on the OTE correlates with performance on the otolaryngology board examination, a few percentage points may make the difference between passing and failing the board examination. jamaotolaryngology.com (Reprinted) JAMA Otolaryngology Head & Neck Surgery April 2015 Volume 141, Number 4 327

5 Research Original Investigation Itisimportanttonotethateaseofaccess, alone, canmakevideos such as those piloted in the present study a valuable resource for resident learning. Residents can access these modules wherever they have a web connection, including during turnover between operating room cases. It is easier to access these modules on a smart phone than to carry around a text or journal article. Limitations of this pilot study include the small number of videos tested, the low response rate to the resident survey, and the inability to eliminate confounding factors affecting OTE scores. Future studies could involve a larger group of videos that cover a majority of otolaryngology educational topics or videos targeting areas of weakness within a specific residency program. Response rates could be improved if this educational tool becomes more formalized in the otolaryngology residency didactic curriculum. Many other specialties, including emergency medicine 1 and obstetrics and gynecology, 2 have implemented webbased educational modules into their curriculum and shown improvements in resident knowledge and testing scores. We predict that online learning will be a key component in the future of resident education within all specialties. We also anticipate a continued need for educational strategies to be costeffective. With this study, we contribute to future resident education by demonstrating the feasibility of easily accessible online resources created for minimal cost. Conclusions Online otolaryngology educational modules are an inexpensive way to expand resident learning opportunities. Despite the lack of quantifiable improvement in OTE scores found in this study, use of online modules sends a message to otolaryngology residents that their education is a priority; self-study outside the hospital and clinics is necessary and expected; and that instructors are willing to try more nontraditional or progressive forms of education to increase resident knowledge. Expansion of the online otolaryngology modules would allow for better coverage of all otolaryngology subspecialties. Use of testing created specifically for the modules would likely give better feedback on the effectiveness of the modules as a teaching strategy. ARTICLE INFORMATION Submitted for Publication: November 7, 2014; final revision received December 18, 2014; accepted December 31, Published Online: February 19, doi: /jamaoto Author Contributions: Dr Cabrera-Muffly had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Cabrera-Muffly, Bryson, Sykes. Acquisition, analysis, or interpretation of data: Cabrera-Muffly, Bryson, Sykes, Shnayder. Drafting of the manuscript: Cabrera-Muffly. Critical revision of the manuscript for important intellectual content: Cabrera-Muffly, Bryson, Sykes, Shnayder. Statistical analysis: Cabrera-Muffly, Sykes. Administrative, technical, or material support: Cabrera-Muffly, Bryson, Sykes, Shnayder. Study supervision: Cabrera-Muffly, Shnayder. Conflict of Interest Disclosures: None reported. REFERENCES 1. Burnette K, Ramundo M, Stevenson M, Beeson MS. Evaluation of a web-based asynchronous pediatric emergency medicine learning tool for residents and medical students. Acad Emerg Med. 2009;16(suppl 2):S46-S De Silva NK, Dietrich JE, Young AE. Pediatric and adolescent gynecology learned via a web-based computerized case series. J Pediatr Adolesc Gynecol. 2010;23(2): Caison AL, Bulman D, Pai S, Neville D. Exploring the technology readiness of nursing and medical students at a Canadian university. J Interprof Care. 2008;22(3): McDonald K, Smith CM. The flipped classroom for professional development, part I: benefits and strategies. J Contin Educ Nurs. 2013;44(10): Tune JD, Sturek M, Basile DP. Flipped classroom model improves graduate student performance in cardiovascular, respiratory, and renal physiology. Adv Physiol Educ. 2013;37(4): McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Acad Med. 2014;89(2): Gantz BJ, Brookhouser PE, Chole RA, et al; American Board of Otolaryngology. Surgery Comprehensive Core Curriculum [2007]. /43a8ba21a7ece670acf25ac4234c90c5 /curriculum final. Accessed January 8, JAMA Otolaryngology Head & Neck Surgery April 2015 Volume 141, Number 4 (Reprinted) jamaotolaryngology.com