Effect of training frequency on the learning curve on the da Vinci Skills Simulator

Transcription

1 ORIGINAL ARTICLE Effect of training frequency on the learning curve on the da Vinci Skills Simulator Ute Walliczek, MD, 1 Arne F ortsch, 1 Phili Dworschak, MD, 2 Afshin Teymoortash, MD, 1 Magis Mandaathil, MD, 1 Jochen Werner, MD, 1 Christian G uldner, MD 1 * 1 Deartment of Otorhinolaryngology Head and Neck Surgery, University Hosital of Marburg, University of Gießen and Marburg, Marburg, Germany, 2 Deartment of Orthoedics, University Hosital of Marburg, University of Gießen and Marburg, Marburg, Germany. Acceted 20 Setember 2015 Published online 17 December 2015 in Wiley Online Library (wileyonlinelibrary.com). DOI /hed ABSTRACT: Background. The urose of this study was to evaluate the effect of training on the erformance outcome with the da Vinci Skills Simulator. Methods. Forty novices were enrolled in a rosective training curriculum. Particiants were searated into 2 grous. Grou 1 erformed 4 training sessions and grou 2 had 2 training sessions over a 4-week eriod. Five exercises were erformed 3 times consecutively. On the last training day, a new exercise was added. Results. A significant skills gain from the first to the final ractice day in overall erformance, time to comlete, and economy of motion was seen for both grous. Grou 1 had a significantly better outcome in overall erformance, time to comlete, and economy of motion in all exercises. There was no significant difference found regarding the new exercise in grou 1 versus grou 2 in nearly all arameters. Conclusion. Longer time distances between training sessions are assumed to lay a secondary role, whereas total reetition frequency is crucial for imrovement of technical erformance. VC 2015 Wiley Periodicals, Inc. Head Neck 38: E1762 E1769, 2016 KEY WORDS: da Vinci Skills Simulator, robotic surgery, robotic surgery education, otorhinolaryngology, training frequency, transoral robotic surgery INTRODUCTION With the introduction of the da Vinci Surgical System (Intuitive Surgical, Sunnyvale, CA) in 1999, the necessity for training to reare surgeons for comlex robotic skills has been steadily increased. In Germany, in more than 40 clinics, the da Vinci Surgical System is used. 1 Currently, robotic-assisted surgery is mainly used in the domain of urology and gynecology. However, in the field of otorhinolaryngology, the robotic-assisted interventions are increasing. The first reorted transoral robotic surgery was the excision of a vallecular cyst in A coule of years later, in 2009, the U.S. Food and Drug Administration aroved the use of transoral robotic surgery for the management of selected benign and malignant tumors of the head and neck area. 3 It may be used for lesions of the oroharynx, hyoharynx, and larynx. 4 Advantages for surgeons using robotic-assisted systems are increased freedom of movement, better visualization, imroved ostoerative outcome, and reduction of hand tremor Another field of alication for robotic-assisted surgery in the head and neck area is the sinus and skull base surgery. Here, robotic-assisted rocedures are still in their investigatory hase. 11,12 Further ossible uses for robotic surgery are in thyroidectomy and obstructive slee anea hyonea syndrome surgery. 13,14 Single case reorts can *Corresonding author: C. G uldner, University Hosital of Marburg, Deartment of Ear, Nose, and Throat, Head and Neck Surgery, Baldingerstraße, Marburg, Germany. gueldner@staff.uni-marburg.de be found for robotic surgery of nasoharyngeal and recurrent nasoharyngeal carcinoma. 15,16 The comlex da Vinci Surgical System consists of several key elements: a high definition 3D-vision system, a atient side cart where the atient is located during the surgery, the Si console where the surgeon sits while oerating, and the EndoWrist instruments. Secial skills are needed to erform robotic-assisted surgery, such as the EndoWrist instrument maniulation and clutching, 3D visualization of the oerative field, and, last but not least, camera control. These secialized features are unique for this tye of technology and differ from techniques used in laaroscoic or oen surgery. 17 Secial training is needed to develo and deeen these secial skills. Currently, there are 3 robotic training latforms available: the Robotic Surgical Simulator (Simulated Surgical Systems, Williamsville, NY), the da Vinci Trainer (Mimic Technologies, Seattle, WA), and the da Vinci Surgical Skills Simulator (dvss; Intuitive Surgical). The only simulator that works with the original surgeon Si console is the dvss. It has furthermore been suggested to have a better face and content validity and is simler to use than the Mimic da Vinci Trainer. 18 The dvss runs with the Mimic virtual reality training software (Mimic Technologies), which contains more than 30 exercises develoed to give trainees the oortunity to acquire secific robotic skills and to imrove their handling of the da Vinci surgeon console. The face-construct and content validity has been roven by various studies. 1,19,20 Although the recondition to use the dvss is the resence of a da Vinci E1762 HEAD & NECK DOI /HED APRIL 2016

2 LEARNING CURVES ON THE DA VINCI SKILLS SIMULATOR TABLE 1. Training curriculum of grou 1 and grou 2. Grou Day 1 Day 8 Day 15 Day 22 1 Exercise 1 5 Exercise 1 5 Exercise 1 5 Exercise 1 5 Reetition 1 3 Reetition 4 6 Reetition 7 9 Reetition new exercise 2 Exercise Exercise 1 5 Reetition 1 3 Reetition new exercise Surgical System, it needs to be taken in mind that the acquisition of a surgical robot, for examle the da Vinci Surgical System, is an investment estimated between 1 and 2.5 million U.S. dollars, excluding simulation systems for training, ongoing costs of maintenance, and instrument relacement. 21 This might lead to a restricted availability of dvss to large academic centers. Because of the high costs, the use of a da Vinci Surgical System is focused to medical education rograms in order to amortize before the atient s surgery. This leads to a shift in time of availability of the dvss to be used for surgical education toward the evenings and the weekends. To otimize time and outcome effectiveness, there is a need to analyze the training frequency to gain roficiency with the da Vinci Surgical System. The urose of the resent study was to evaluate the effect of the training frequency on the erformance outcome of robotic novices on the dvss. MATERIALS AND METHODS The dvss latform was chosen to conduct this rosective study, which was erformed from January 2013 to December The training was absolved using the original da Vinci surgeon console with the simulator backack. Forty robotic novices, in which none of them had significant exerience in endoscoic or laaroscoic surgery, were enrolled in this 4-week robotic training study conducted within the Deartment of Otorhinolaryngology Head and Neck Surgery University Hosital of Marburg, Germany. They were randomly divided into 2 grous consisting of 20 trainees in each grou (grous 1 and 2). Each grou erformed a total of 5 exercises 3 times consecutively. Of the whole rogram, the chosen exercises were Match Board (MB)1, MB2, MB3, Ring and Rail (RR)1, and RR2, in the defined order. This was done at the recommendation of the manufacturer s original training rogram, the need of different difficulty levels, and the missing measurable and comarable outut of some of the other rogram arts. On the last day, a new exercise (Needle Targeting [NT]) was added and erformed in only 1 reetition. Grou 1 erformed the exercises in 4 days within the 4-week curriculum (days 1, 8, 15, and 22), whereas grou 2 conducted training only on 2 days (days 1 and 22; Table 1). The first training session was initiated by a brief introduction to the dvss by the research coordinator to familiarize trainees with the console and its handling. Each task was receded by a short verbal exlanation of the exercise. After this, the trainees fulfilled their robotic training autonomously in the resence of the study team. Each training session took aroximately 1.5 hours. After each exercise, a detailed erformance reort, including time to comlete the exercise, economy in motion, number of instrument collisions, excessive instrument force, instruments out of view, master work sace range, and number of dros was given. The overall erformance score was defined as the total result of the terminated exercise and is calculated by the software of the simulator from the above-measured metrics. Particiants were able to ask questions and receive feedback on how to better erform each of the exercises. The study was aroved by the local ethics committee. The following exercises were erformed in order. MB1 to MB3: the MB exercises consisted of icking u 3D objects (wooden letters and number blocks) and lacing them into corresonding attern cutouts. From MB1 toward MB3, the required dexterity increased. In MB2: in confront to MB1, an extra robotic arm was used. In MB3: the difficulty was raised again while the third arms came into lay and was used to free the attern cutouts where the wooden objects were laced (an examle is given in Figure 1, left side). RR1 and RR2: within the RR1 exercise, 1 ring was led along a twisted rod. In RR2, 3 colored rings were led along several twisted rods toward the color equal goal (an examle is given in Figure 1, middle). Needle targeting In this exercise, the exact ositioning of the colored needles into 2 colored identical targets of different sizes was erformed. One after another, the larger target and the smaller one were required to be ricked (an examle is given in Figure 1, right side). The order in which the exercises were erformed was MB1, MB2, MB3, RR1, and RR2, and, on the last day, NT was added. Statistics The statistical data analysis was erformed with the rogram SPSS version 22.0 (SAS Statistics, Cary, NC). The used statistical test was the 2-samle test for indeendent and deendent variables under the assumtion of equal variance of the oulation. The test was erformed with the arithmetic mean of the overall score with the individual arameters of 1 session day, in accordance with the exercises. Statistical significance was set at.05. RESULTS A total of 40 articiants were enrolled in this study and were divided into 2 grous: grou trainees (10 women and 10 men; mean age, years) and grou trainees (10 women and 10 men; mean age, of years). A significant imrovement was seen in the technical erformance (overall score, time to comlete, and economy of HEAD & NECK DOI /HED APRIL 2016 E1763

3 WALLICZEK ET AL. FIGURE 1. Examle of the different exercises is given. On the left side, Match Board 1 is demonstrated. In the middle, Ring and Rail is shown, and on the right side is the final exercise of Needle Targeting is resented. motion) between the first and the last reetition in all exerciseswithingrous1and2( <.001; Figures 2 and 3). No statistically significant difference could be found between the first reetition of each exercise of grou 1 versus grou 2 for the overall score, time to comlete, and economy of motion (Tables 2 4), showing the fact of similar grous starting the training. Interestingly, indeendent of the time between reetition 1 and 6 (grou 1: 7 days vs grou 2: 21 days), no significant difference of overall score, time to comlete, and economy of motion could be found between the grous (Tables 2 4). Between reetition 12 of each exercise in grou 1 and reetition 6 in grou 2, a statistically significant difference could be demonstrated for overall erformance, time to comlete, and economy of motion in all exercises (Tables (2 4)). Moreover, the roficiency level of grou 1, according to the recommendation of the develoer defined as overall score >80%, was reached in 3 of 5 exercises (MB1, MB2, and RR1) after all reetitions, whereas articiants of grou 2 reached the roficiency level in only 1 exercise (RR1), which is a relatively easy one. Looking at the new exercise (NT) after the comlete training, there was no statistically significant difference found for grou 1 versus grou 2 in nearly all arameters (Table 5). DISCUSSION Exanding the alication of minimally invasive techniques, such as robotic-assisted interventions, challenge surgical educators to reare residents on this secial kind of surgery in which secific skills are required, such as 3D visualization of the oerative field, maniulation and clutching of the EndoWrist instruments, and camera control. Furthermore, it needs to be taken in mind that mentoring for robotic surgery is also secial because, contrary to oen surgery where the mentor and trainee are in close roximity allowing hands-on teaching during robotic surgery, the trainee is left to his own, at least hysically, because only 1 erson can be seated at the console of the robotic surgery system This makes it even more necessary to reare surgeons adequately before erforming robotic surgery. Simulation training has been shown to be an effective rearation for laaroscoic surgery to shorten oeration time and reduce surgical errors. 25,26 Simulation has been demonstrated to be an effective educational tool for robotic surgeons as well. 1,27 It might even be used as a warmu before surgery on atients, or as a refresher for surgeons during hases with little active robotic-assisted interventions. One ossible simulator is the da Vinci Skills Simulator for which face, content, and construct validity have already been reorted. 19,20 However, little data are available as to how exactly the training curricula needed to be structured to esecially how training frequency should be to reare trainees in order to reach the roficiency defined as >80% overall score. This study was a rosective study to analyze the effect of different training frequencies on the erformance outcome on the da Vinci Skills Simulator. It was seen that an imrovement of the outcome of technical erformance esecially overall score, time to comlete, and economy of motion from the first toward the last reetition was achieved. Consistent with these findings is another study from 2014 in which the training curriculum was conducted with 10 consecutive reetitions in 1 training session. They could demonstrate a significant imrovement in technical erformance, esecially overall score, time to comlete, and economy of motion, within 4 exercises (MB1, RR1, Suture Songe 1, and Energy Switching 1). 28 In the resent study, it was furthermore seen that there was no statistically significant difference between reetition 6 of grou 1 versus grou 2 in overall score and time to comlete in nearly all exercises. This means that it makes no difference whether the time between the reetitions is 1 week or 4 weeks. In the economy of motion, a significant difference could be seen for the exercises MB2 and MB3. In MB3, grou 1 reached a significant better overall score comared to grou 2. In conclusion, longer training-free intervals might lead to a oorer outcome in overall score and economy of motion in more comlex exercises. Furthermore, it could be shown that the outcome of the erformance of a comlete new exercise, in this case NT, did statistically almost not differ for grou 1 versus grou 2. Critically remarked, it could not be ruled out that this is because of the fact that the exercise is too easy to differentiate between both grous. A longer time-interval of training absence (there was a 2- E1764 HEAD & NECK DOI /HED APRIL 2016

4 LEARNING CURVES ON THE DA VINCI SKILLS SIMULATOR FIGURE 2. Overview of the learning curves for the Match Board examinations is given. The left middle column shows the results for grou 1 and the right middle column shows the results for grou 2. The horizontal dotted line marks the result of reetition 6 of grou 2 to show the difference comared to grou 1. The outer columns visualize the results of the new exercise at the end of the training (Needle Targeting; left side grou 1; right side grou 2). week break in this study) seems not to influence the outcome of technical erformance at all. A statistically significant difference in overall erformance, time to comlete, and economy of motion in all exercises could be demonstrated between the last (12) reetition of grou 1 and the last (6) reetition of grou 2. We therefore assume that less of a time interval between training sessions lays a rimary role but total frequency of reetitions of each exercise is crucial for imrovement of technical erformance. Interestingly, the overall score >80% was neither reached in grou 1 ( ) nor in grou 2 ( ) for the new exercise of NT after finishing the regular training rogram. This might be because of the single reetition of this exercise. Therefore, it seems to be necessary to erform exercises more than one time, even in well-trained ersons, to reach an overall score of >80%. However, a slightly better overall score for grou 1 could be seen, as well as a better outcome in time to comlete and economy in motion, although not statistically significant. A transfer of gained robotic skills to new tasks could be assumed. It could be suggested that a higher training frequency also suorts the skills transfer to before untrained exercises. The non-reaching of roficiency for the new exercise could be exlained by the ossibility of differing secific skills required for NT in confront to the racticed MB and RR exercises. In this study, the trainee collective was homogenous with no robotic or endoscoic/laaroscoic surgical exerience. For the evaluation of a training curriculum for HEAD & NECK DOI /HED APRIL 2016 E1765

5 WALLICZEK ET AL. FIGURE 3. Overview of the learning curves for the Ring and Rail examinations is given. The left middle column shows the results for grou 1 and the right middle column shows the results for grou 2. The horizontal dotted line marks the result of reetition 6 of grou 2 to show the difference comared to grou 1. The outer columns visualize the results of the new exercise at the end of the training (Needle Targeting; left side grou 1; right side grou 2). rearation for robotic surgery, it might be advantageous to include a study oulation of robotic novices to be able to give evidence about how to reach the roficiency level. In this study, no statistically significant difference could be found between reetition 1 in grou 1 versus grou 2 in overall score, time to comlete, and economy of motion, which showed that the study oulation of grou 1 and 2 was comarable. Whereas in various other studies, the examined oulation was more heterogeneous and smaller, as Schreuder et al 27 showed in their review. Thirteen studies were included in that review, 11 studies were comrised of <30 articiants, 8 studies consisted of both: exerts and robotic novices. In 11 studies, less than the 5 different exercises had been comleted. They reorted about a significant learning effect through training on virtual reality simulators. 27 In this study, trainees were asked to erform 5 different exercises. We suggest that concentration might be higher when having variety in training sessions. The validity for 1 exercise grou (MB3 and RR3) has already been exemlarily demonstrated by TABLE 2. Comarison of the overall score for the several exercises. Exercise Grou Grou 1 reetition 1 vs grou 2 reetition 1 Grou 1 reetition 6 vs grou 2 reetition 6 Grou 1 reetition 12 vs grou 2 reetition 6 in % in % in % < < < Exercise 1 5 Match Board 1; Exercise 2 5 Match Board 2; Exercise 3 5 Match Board 3; Exercise 4 5 Ring and Rail 1; and Exercise 5 5 Ring and Rail 2 for different time oints. First reetition 1 of both grous, second reetition 6 of both grous (grou 1 on day 8 and grou 2 on day 22), and third reetition 12 of grou 1 and reetition 6 of grou 2 (both on day 22). E1766 HEAD & NECK DOI /HED APRIL 2016

6 LEARNING CURVES ON THE DA VINCI SKILLS SIMULATOR TABLE 3. Comarison of the time to comlete for the several exercises. Exercise Grou Grou 1 reetition 1 vs grou 2 reetition 1 Grou 1 reetition 6 vs grou 2 reetition 6 Grou 1 reetition 12 vs grou 2 reetition 6 in s in s in s < < < < < Exercise 1 5 Match Board 1; Exercise 2 5 Match Board 2; Exercise 3 5 Match Board 3; Exercise 4 5 Ring & Rail 1; and Exercise 5 5 Ring & Rail 2) for different time oints. First reetition 1 of both grous, second reetition 6 of both grous (grou 1 on day 8 and grou 2 on day 22), and third reetition 12 of grou 1 and reetition 6 of grou 2 (both on day 22). Lyons et al, 20 in They evaluated 46 surgeons (25 novices, 8 intermediate exerience, and 13 exerts) on their technical erformance. They could demonstrate that overall score, time to comlete, and economy of motion differed significantly between robotic novices and exerts. 20 This might suggest that evaluation of these metrics is advisable to assess gained roficiency of robotic novices. Furthermore, it was assumed that it might be imortant to be safe and efficient in economy of motion to become a better robotic surgeon. 29 In this study, the roficiency level of these metrics was reached in 3 of 5 exercises (MB1, MB2, and RR1) in grou 1 within 12 reetitions, whereas articiants of grou 2 reached the roficiency level only in 1 exercise (RR1) after 6 reetitions. A total frequency of 6 reetitions in a oulation of robotic novices seemed to not be sufficient to reach the roficiency level. In a similar study from 2014, Gomez et al 17 enrolled 24 articiants (4 faculty, 4 senior, and 14 junior residents) in a 4-week robotic training curriculum and a significant skills gain from the retesting to the osttesting hase was reorted, but only in 5 of the 7 exercises (Camera Targeting 2, Energy Dissection 2, Peg Board 2, Suture Songe 2, Suture Songe 3, Thread the rings, and Ring Walk 3) an overall score >80% was reached. Proficiency was not reached in the more comlex exercises Thread the rings and Ring Walk 3. Their study consisted of a retesting hase in which trainees were asked to comlete each trial of the 7 exercises, followed by a training hase in which 2 individual training sessions had different exercises, although they contained the same robotic core skills, were erformed, as in the retesting hase, the last ste of the curriculum was a osttesting session with the same set of exercises used during the retesting hase. The training exercise for Ring Walk 3 was MB3. 17 Matching with these observations, also in the resent study, the roficiency level for MB3 was not reached. We suose that, esecially for more comlex exercises, a higher frequency than 12 reetitions is needed within a robotic training curriculum for robotic novices to reach roficiency. In the current study, the roficiency level was achieved, in average, after 5 attemts. The overall score of >80% was reached in reetition 8 for MB1 (overall score ), reetition 7 for MB2 (overall score ), and reetition 3 for RR1 (overall score ; grou 1), resectively, after 1 attemt (grou 2; RR1). Consistent with these findings, Patel et al 29 reorted in 2013 that between 4 and 5 attemts to achieve TABLE 4. Comarison of the economy of motion for the several exercises. Exercise Grou Grou 1 reetition 1 vs grou 2 reetition 1 Grou 1 reetition 6 vs grou 2 reetition 6 Grou 1 reetition 12 vs grou 2 reetition 6 in cm in cm in cm < < Exercise 1 5 Match Board 1; Exercise 2 5 Match Board 2; Exercise 3 5 Match Board 3; Exercise 4 5 Ring & Rail 1; and Exercise 5 5 Ring & Rail 2 for different time oints. First reetition 1 of both grous, second reetition 6 of both grous (grou 1 on day 8 and grou 2 on day 22), and third reetition 12 of grou 1 and reetition 6 of grou 2 (both on day 22). HEAD & NECK DOI /HED APRIL 2016 E1767

7 WALLICZEK ET AL. TABLE 5. Comarison of the arameters for the new exercise (needle targeting) of grou 1 and grou 2. Missed targets Dros Range in cm Outview in s Force in s Collisions Economy in cm Time to comlete in s Overall in % Grou >80% roficiency were necessary. Exercises chosen for their study were Camera Targeting, Energy Switching, and Energy Dissection. No data on the exerience of users was given to the reader. 29 In contrast, in this study, MB and RR exercises were erformed and only robotic novices were enrolled. CONCLUSION This study demonstrates that longer time eriods of training absence might lay a minor role for outcome of technical erformance on the da Vinci Skills Simulator, whereas the total frequency of reetitions seems to be crucial. For more comlex exercises, a higher number than 12 reetitions might be necessary to reach the roficiency level in a oulation of robotic novices. A skills transfer to a before unracticed exercise is assumed. REFERENCES 1. Find a da Vinci surgeon. Available at: htt:// com/. Accessed March 1, McLeod IK, Melder PC. Da Vinci robot-assisted excision of a vallecular cyst: a case reort. Ear Nose Throat J 2005;84: Weinstein G, O Malley BW Jr. FDA clears transoral robotic surgery develoed at Penn for tumors of mouth, throat and voice box Available at: htt:// tors-robotic-head-neck-surgery/. Accessed March 1, Rinaldi V, Pagani D, Torretta S, Pignataro L. Transoral robotic surgery in the management of head and neck tumours. Ecancermedicalscience 2013; 7: O Malley BW Jr, Weinstein GS, Snyder W, Hockstein NG. Transoral robotic surgery (TORS) for base of the tongue neolasms. Laryngoscoe 2006;116: Genden EM, Desai S, Sung CK. Transoral robotic surgery for the management of head and neck cancer: a reliminary exerience. Head Neck 2009; 31: Genden EM, Kotz T, Tong CC, et al. Transoral robotic resection and reconstruction for head and neck cancer. Laryngoscoe 2011;121: Hurtuk AM, Marcinow A, Agrawal A, Old M, Teknos TN, Ozer E. Quality-of-life outcomes in transoral robotic surgery. Otolaryngol Head Neck Surg 2012;146: Iseli TA, Kulbersh BD, Iseli CE, Carroll WR, Rosenthal EL, Magnuson JS. Functional outcomes after transoral robotic surgery for head and neck cancer. Otolaryngol Head Neck Surg 2009;141: Clayburgh DR, Gross N. Surgical innovations. Otolaryngol Clin North Am 2013;46: Schneider JS, Burgner J, Webster RJ III, Russell PT III. Robotic surgery for the sinuses and skull base: what are the ossibilities and what are the obstacles? Curr Oin Otolaryngol Head Neck Surg 2013;21: Eichhorn KW, Bootz F. Clinical requirements and ossible alications of robot assisted endoscoy in skull base and sinus surgery. Acta Neurochir Sul 2011;109: Byeon HK, Holsinger FC, Tufano RP, et al. Robotic total thyroidectomy with modified radical neck dissection via unilateral retroauricular aroach. Ann Surg Oncol 2014;21: Vicini C, Montevecchi F, Pang K, et al. Combined transoral robotic tongue base surgery and alate surgery in obstructive slee anea-hyonea syndrome: exansion shincter haryngolasty versus uvuloalatoharyngolasty. Head Neck 2014;36: Wei WI, Ho WK. Transoral robotic resection of recurrent nasoharyngeal carcinoma. Laryngoscoe 2010;120: Yin Tsang RK, Ho WK, Wei WI. Combined transnasal endoscoic and transoral robotic resection of recurrent nasoharyngeal carcinoma. Head Neck 2012;34: Gomez PP, Willis RE, Van Sickle KR. Develoment of a virtual reality robotic surgical curriculum using the da Vinci Si surgical system. Surg Endosc 2015;29: Liss MA, Abdelshehid C, Quach S, et al. Validation, correlation, and comarison of the da Vinci trainer( TM ) and the davinci surgical skills simulator( TM ) using the Mimic( TM ) software for urologic robotic surgical education. J Endourol 2012;26: Hung AJ, Zehnder P, Patil MB, et al. Face, content and construct validity of a novel robotic surgery simulator. J Urol 2011;186: Lyons C, Goldfarb D, Jones SL, et al. Which skills really matter? Proving face, content, and construct validity for a commercial robotic simulator. Surg Endosc 2013;27: Mandaathil M, Teymoortash A, G uldner C, Wiegand S, Mutters R, Werner JA. Establishing a transoral robotic surgery rogram in an academic hosital in Germany. Acta Otolaryngol 2014;134: E1768 HEAD & NECK DOI /HED APRIL 2016

8 LEARNING CURVES ON THE DA VINCI SKILLS SIMULATOR 22. Tergas AI, Sheth SB, Green IC, Giuntoli RL, Winder AD, Fader AN. A ilot study of surgical training using a virtual robotic surgery simulator. JSLS 2013;17: Guzzo TJ, Gonzalgo ML. Robotic surgical training of the urologic oncologist. Urol Oncol 2009;27: Perrenot C, Perez M, Tran N, et al. The virtual reality simulator dv-trainer(vr ) is a valid assessment tool for robotic surgical skills. Surg Endosc 2012;26: Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch Jensen P. Randomized clinical trial of virtual reality simulation for laaroscoic skills training. Br J Surg 2004;91: Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training imroves oerating room erformance: results of a randomized, doubleblinded study. Ann Surg 2002;236: ; discussion Schreuder HW, Wolswjik R, Zweemer RP, Schijven MP, Verheijen RH. Training and learning robotic surgery, time for a more structured aroach: a systematic review. BJOG 2012;119: Sheth SS, Fader AN, Tergas AI, Kushnir CL, Green IC. Virtual reality robotic surgical simulation: an analysis of gynecology trainees. J Surg Educ 2014;71: Patel A, Patel M, Lytle N, et al. Can we become better robot surgeons through simulator ractice? Surg Endosc 2014;28: HEAD & NECK DOI /HED APRIL 2016 E1769