The Development and Validation of Hand Motion Analysis to Evaluate Competency in Central Line Catheterization

Transcription

1 EDUCATIONAL ADVANCE The Development and Validation of Hand Motion Analysis to Evaluate Competency in Central Line Catheterization David Clinkard, MSc, Matthew Holden, MSc, Tamas Ungi, MD, PhD, David Messenger, MD, FRCPC, Colleen Davison, PhD, Gabor Fichtinger, PhD, and Robert McGraw, MD, FRCPC Abstract Objectives: Traditionally, technical skills proficiency has been assessed by direct observation. While direct observation and feedback are essential components in technical skills learning, they demand considerable investment of faculty time, and as an assessment tool, direct observation is inherently subjective and has been criticized as unreliable. The purpose of this study was to determine if quantitative electromagnetic motion tracking is feasible and can discriminate between experts and nonexperts during simulated ultrasound (US)-guided insertion of a central venous catheter (CVC) guidewire. Methods: Ten nonexperts (junior emergency medicine residents) and 10 experts (critical care fellows or attending physicians) were recruited. Electromagnetic sensor probes were used to capture hand motion during an US-guided internal jugular cannulation on a standardized manikin. Hand, US, and needle motion were analyzed for the following metrics: total path length, total time, translational movements, and rotational movements. Subjects were also videotaped and evaluated using a modified, validated global rating scale (GRS) by a blinded expert. Results: There was a significant difference in almost all examined motion parameters between experts and nonexperts. Experts took 66% less time (50.2 seconds vs seconds, p < 0.001) and had significantly less right-hand and US motion (total path length and translational and rotational movements). Left-hand total path length was the only parameter that was not significantly different between groups. Concurrent validity of motion parameters was established by strong correlations (r 2 > 0.74) to a previously published, modified GRS. Conclusions: Electromagnetic hand and instrument motion analysis is technically feasible for assessing competence in US-guided insertion of a CVC guidewire in a simulation setting. In showing that it discriminates between the performances of nonexperts and experts, this study has provided evidence for construct validity. It also shows excellent correlation with a modified version of a previously validated GRS, providing evidence of concurrent validity. ACADEMIC EMERGENCY MEDICINE 2015;22: by the Society for Academic Emergency Medicine The past decade has witnessed a fundamental shift in medical education. Competency-based medical education (CBME), with a focus on outcomes, has been embraced at all levels of training and endorsed by the Royal College of Physicians and Surgeons of Canada and the Association of Faculties of Medicine of Canada. In general terms, CBME requires identifying abilities needed of graduates, followed by the development of a program to ensure trainees achieve competency. 1,2 Two of the biggest challenges in implementing CBME are the need for explicit definitions of required competencies and the development of feasible, valid, and reliable assessment tools for measuring achievement of competence. 3 From the Department of Emergency Medicine (DC, DM, CD, RM), the Department of Computing (MH, TU, GF), and the Department of Community Health and Epidemiology (CD), Queen s University, Kingston, Ontario, Canada. Received June 30, 2014; revision received August 19, 2014; accepted August 28, Presented at the Canadian Emergency Medicine Conference, Ottawa, Ontario, Canada, June The authors have no relevant financial information or potential conflicts to disclose. Supervising Editor: John Burton, MD. Address for correspondence and reprints: David Clinkard, MSc; dclinkard@qmed.ca. 212 ISSN by the Society for Academic Emergency Medicine 212 PII ISSN doi: /acem.12590

2 ACADEMIC EMERGENCY MEDICINE February 2015, Vol. 22, No Proficiency in procedural skills, including central venous catheter (CVC) insertion, is one of the key competencies of the Medical Expert Role in the CanMEDS framework. 4 Traditionally learned at the bedside, CVC insertion has a serious and life-threatening complication rate of 0 to 25%, with the rate being inversely related to operator experience. 5 There is a growing body of evidence that simulation-based training reduces the incidence of complications in central line insertion by providing learners the opportunity to gain this experience in a safe, simulated setting prior to real patient exposure. 6 To enable this, however, educational programs require measures of proficiency to determine when individual learners are ready to make the transition to real patient encounters. 7 Traditionally, technical skills proficiency was assessed by direct observation. While direct observation and feedback are essential components in technical skills learning, they demand considerable investment of faculty time, and as an assessment tool, direct observation is inherently subjective and has been criticized as unreliable. 8 This has led to the pursuit of more objective assessment tools, such as checklists, global rating scales (GRS), and more recently, hand motion analysis. 9,10 Specific performance criteria global rating systems and checklists are an improvement on subjective, unstructured observational assessments, but they still retain subjective elements and are resource-intensive, requiring multiple faculty raters. 8 As well, these systems do not provide objective measures of hand motion efficiency a critical difference between novice and expert performance. 10,11 Recently, hand motion analysis has been investigated as a method to measure proficiency in procedural skills, with the most widely used system being the Imperial College Surgical Assessment Device. 12 Utilizing trackers on both hands, this device has been validated for multiple surgical techniques, 13 but to our knowledge similar hand motion analysis has not been used in ultrasound (US)-guided CVC insertion. The purpose of this study was to determine if quantitative electromagnetic motion tracking is feasible during simulated US-guided insertion of a CVC guidewire. Additionally, we sought to investigate if it could discriminate between the performance of nonexperts (firstand second-year residents) and experts (critical care attending physicians and fellows). Our hypothesis was that we could discriminate between the performances of these two groups based on differences in hand and instrument movement during the procedure. A significant difference between the two groups would be evidence of construct validity. Further, we hypothesized there would be correlation between scores on hand and US motion and scores on a previously validated GRS (Data Supplement S1, available as supporting information in the online version of this paper), 14 which would provide evidence of concurrent validity. METHODS Study Design This exploratory study was conducted in the Clinical Simulation Centre at Queen s University (Kingston, Ontario, Canada) and all participants were residents or faculty at Queen s University. The project was approved by Queen s University Health Science Research Ethics Board. All participation was voluntary. All participants provided written informed consent. Study Setting and Population Ten resident physicians (novices) and 10 staff physicians/fellows (experts) were recruited. All residents had completed an online teaching module (the standard approach for teaching central venous access at Queen s University). The teaching module describes the indications, safety considerations, and procedure required to complete a central line insertion. Following the module, all novice study participants participated in an interactive simulation session, including practice of the procedure on a manikin until they felt comfortable performing the procedure (maximum time of 2 hours). This teaching module was completed 1 month prior to the experiment start. Sample sizes were informed by previous hand and instrument tracking studies at our center, although for this pilot, group size was also limited by resource constraints. At a total of 20 participants we were able to detect expected levels of effect with at least 80% power. The expert group consisted of teaching hospital based critical care physicians and critical care fellows. The nonexpert group consisted of residents in their first or second year of emergency medicine training. Demographics (Table 1) were collected as best estimates, with many of the experts unable to remember exactly how many procedures they had completed. Study Protocol All participants had electromagnetic sensor probes attached to the dorsal aspect of the third metacarpal on both hands. Participants were given 30 seconds to orient themselves to the US, and screen brightness was adjusted to personal preference. All participants were then instructed to perform the US-dependent portion of an internal jugular CVC insertion without feedback. Specifically, they were instructed to use left-handed US guidance to correctly landmark the internal jugular vein and, with the right hand insert a needle into the vein, pass the guide wire through the needle and then remove the needle. Equipment preparation, sterile draping, and catheter insertion were not required, as these steps are not specifically dependent on US guidance. All participants were instructed to start with hands and instruments in a standardized position. Measures The sensing device consisted of a DriveBay EM tracker (Ascension Technology Corporation, Milton, VT) consisting of an electromagnetic field generator with four sensors (Model 800 6DOF Sensor, Ascension Technology). Sensors were attached to the proximal aspect of a standard CVC needle and syringe, the dorsal aspects of both hands, and the head of an US probe to record their respective trajectories and movements. The three-dimensional coordinates of each sensor were recorded with root-mean-square accuracy of 1.4 mm at 50 Hz using the open-source software package

3 214 Clinkard et al. HAND MOTION ANALYSIS TO EVALUATE COMPETENCY Table 1 Participant Demographics Characteristic Experts (n = 10) Nonexperts (n = 10) Age (yr) 40 (31 54) 28 (25 34) Training year (years training & practice) 12 (6 21) 2 (1 3) Number of months training in the ICU specifically 13 (4 30) 0.5 (0 2) Number of observed CVC placements 250 (200 1,000) 5 (0 10) Number of CVC placements with US 50 (10 100) 10 (0 25) Number of CVC placements without US 300 (3 1,000) 1 (0 6) Number of CVC placement failures 30 (2 100) 1 (0 3) Number of simulated CVC placements with US 15 (0 100) 5 (0 10) Number of simulated CVC placements without US 20 (0 100) 5 (0 5) Data are reported as median (range). Participants were asked to give best estimates for the number of procedures they had completed. Given the nature of this estimation, all data were rounded to the nearest 5. CVC = central venous catheter; ICU = intensive care unit; US = ultrasound. PLUS ( PLUS sends the data to another open source program, 3D Slicer ( org), via the OpenIGTLink Network protocol. Using the PerkTutor extension ( for 3D Slicer, trajectories were recorded and metrics associated with the procedure were calculated. 15,16 All data were filtered for movement velocity > 5 cm/millisec. Several metrics were used to evaluate motion efficiency: total time of procedure, total path length traced by each sensor (distance), number of translational motions for each sensor (movements), and number of rotational motions for each sensor (repositions). The number of translational motions was calculated as the number of times the translational velocity of a sensor changed by greater than 7.5 mm/sec. The number of rotational motions was calculated as the number of times the rotational velocity of a sensor changed by greater than 5 degrees/sec. 17 Subjects hands were videotaped with a Sony HDR- XR160 camcorder mounted on a tripod. Video did not include any sound or identifying features, to ensure blinding. At the end of the data collection period, an independent critical care physician, who is considered a local expert in central line teaching and evaluation, assessed the videos in a blinded fashion (unaware of study hypothesis and video source) using a modified version of a previously validated GRS. 14 Data Analysis All data analysis was conducted with PRISM 6.0, GraphPad software. A previously published GRS, modified to account for the limited components of the overall procedure that subjects performed, was used to look for evidence of concurrent validity (Data Supplement S1). 14 The GRS evaluates domains of competence (flow, instrument handling, time and motion, overall ability) on a five-point Likert scale, with behavioral anchors. In modifying the GRS, specific domains of evaluation were omitted based on local expert consensus about nonrelevance for the purposes of this study and lack of correlation to previously published checklist scores (e.g., aseptic technique, analgesia). 14 Pearson s correlations were then calculated to evaluate the relationship between GRS and hand motions. Nonnormally distributed data are presented as medians with interquartile ranges; normally distributed data are presented as means with standard deviations (SDs). A D Agostino and Pearson omnibus normality test showed motion data to be nonnormally distributed, so Mann-Whitney nonparametric tests were used. Time and GRS scores were normally distributed, and thus Student s t-test was used. It should be noted that we did not adjust for the type 1 error rate to account for multiple comparisons, and findings close to a p-value of 0.05 should be interpreted conservatively. RESULTS Experts had significantly more experience with CVC than nonexperts (Table 1). Experts had, on average, 12 years of training and practicing in critical care, with more than 300 central line insertions each. There was a significant difference in almost all examined parameters between the expert and nonexpert groups (Table 2). The experts took 66% less time (Figure 1) to complete the task than nonexperts and performed significantly better on most of the US and needle motion parameters (Figure 2, distance; Figure 3, motions; and Figure 4, repositions). Left-hand distance was not significantly different between groups, although left-hand motions and repositions were both significantly less in the expert group. All aspects of right-hand motion were significantly different between experts and nonexperts. Concurrent validity of the motion parameters was assessed by comparing them to the modified GRS (Figure 5). As a group, experts scored significantly higher than the nonexpert group on the four assessed components. On time and motion, experts scored 4.28 vs. 2.17, p < 0.005; instrument handling 4.57 vs. 2.84, p < 0.005; flow 4.42 vs. 3.23, p < 0.005; and overall ability 4.57 vs. 2.71, p < The Pearson correlation coefficient between hand motions and the GRS demonstrated strong correlations (needle distance 0.74, repositions 0.80, and movements 0.77). DISCUSSION For procedural skills, movement economy is considered a key difference between expert and nonexpert operators. 18 In simple terms, nonexperts tend to move their

4 ACADEMIC EMERGENCY MEDICINE February 2015, Vol. 22, No Table 2 Median Movement Parameters and p-values for Mann-Whitney U-Test Parameter Experts (n = 10) Nonexperts (n = 10) p-value Total procedure time (sec) 50.2 ( ) ( ) <0.001 US Path length (mm) 1,530 (1,289 2,742) 3,167 (1,026 10,499) Discreet motion count 1,078 (610 1,920) 3,263 (1,520 10,725) <0.001 Tool reposition count 751 ( ) 2,273 (944 6,551) <0.001 Left hand Path length (mm) 2,455 (1,725 4,178) 3,235 (1,488 10,756) 0.12 Discreet motion count 1,181(555 2,190) 3,075 (1,817 6,149) <0.001 Tool reposition count 848 (531 1,456) 2,305 (1,053 4,275) <0.01 Needle Path length (mm) 2,377 (1,694 4,598) 5,591 (2,337 10,011) <0.01 Discreet motion count 1,523 (921 3,027) 4,040 (2,163 10,011) <0.001 Tool reposition count 1,067 (634 1,773) 2,704 (1,439 5,772) <0.001 Right hand Path length (mm) 4,622 (3,952 7,105) 8,892 (5,027 14,063) <0.001 Discreet motion count 1,914 (1,197 3,248) 4,424 (2,764 13,180) <0.001 Tool reposition count 1,348 (916 1,818) 2,771 (1,666 9,552) <0.001 Data are reported as median (range). US = ultrasound. Figure 1. Total time of procedure for experts and nonexperts. Mean 1 SD is shown. *Statistically significant at p < hands and instruments excessively during performance of a procedure, while experts tend to be more deliberate, with fewer pauses and greater motion efficiency. Multiple assessment modalities have attempted to capture this fluidity qualitatively, but only recent technical advances have allowed hand motions to be captured quantitatively. One such system, the Imperial College Surgical Assessment Device, measures hand motion and has been shown to be a feasible, valid, and reliable measure of proficiency in general, laparoscopic, and microscopic surgeries. 13 Building on this work, our study examines hand and instrument motion analysis as a measure of proficiency in central venous line insertion. The primary aim of this study was to determine if it is feasible to use electromagnetic hand and instrument motion during a simulated CVC guide wire insertion and if the quantitative measures generated could be used to differentiate between the performances of experts and nonexperts during the US-dependent portion of CVC insertion. Our results, which show a significant difference between expert and nonexperts on nearly all measures, are evidence for the construct validity of motion analysis as a measure of competence in this procedure. A secondary aim of the study was to collect evidence for the concurrent validity of these measures by correlating the results with observed assessment using a previously validated GRS. The high correlation between the results is evidence for the concurrent validity of the measures as well. The most common complications of central venous insertion are due to wire/catheter embolism and multiple insertion attempts, suggesting that wire and needle control has significant safety implications. 19 In confirming that experts exhibit fewer hand and tool movements and repositions than nonexperts, we can begin to determine the threshold motion-economy metrics that trainees need to achieve to proceed to the next phase of training and clinical care (insertion on real patients). These quantitative benchmarks will enable the study of important parameters such as the volume and type of practice required by nonexperts to achieve competency. Further, such quantitative benchmarks may provide a valuable asset for CBME evaluation. CBME is premised on the belief that learners need to achieve a specific level of competence independent of time spent in training. It requires that programs provide sufficient opportunities for individual trainees to practice to the point of competence and that they have ways of measuring achievement of competence. 2,3 For this approach to succeed, however, valid and reliable tools for assessing competency are essential. 2 The hand and instrument motion analysis reported here removes any observer bias from the assessment process, provides specific quantitative indices of performance, and

5 216 Clinkard et al. HAND MOTION ANALYSIS TO EVALUATE COMPETENCY Figure 2. Total path length of the left hand, right hand, ultrasound, and needle for experts and nonexperts. Median 1 interquartile range. *Statistically significant at p < Figure 3. Total discreet motions of the left hand, right hand, ultrasound, and needle for experts and nonexperts. Median 1 interquartile range. *Statistically significant at p < Figure 4. Total repositions of the left hand, right hand, ultrasound, and needle for experts and nonexperts. Median 1 interquartile range. *Statistically significant at p < allows tracking of improvement that can illuminate curriculum development and the determination of the proper amount of time needed to learn and perfect a skill. Having shown that hand and instrument motion analysis provides both a valid and a feasible measure of proficiency in US-guided CVC insertion, we intend to investigate the temporal learning curves of nonexperts

6 ACADEMIC EMERGENCY MEDICINE February 2015, Vol. 22, No technical domains have been shown to have positive correlation with both a 10-item checklist score and a 21-item checklist score, something that the safety factors did not. 13 CONCLUSIONS Figure 5. Pearson s correlations between global rating scores and procedure motion metrics. Straight lines indicate linear regressions using the sum of squares methodology. r 2 = Pearson s correlation coefficient. as they practice the skill. The goal is to establish evidence-based guidelines for the volume and type of practice required to achieve competence in this skill. LIMITATIONS This study must be regarded as preliminary due to several limitations. Our study examined US-guided CVC insertion among postgraduate medical trainees in a simulation setting. After consultation with local experts, we chose to limit our focus to venous access and guide wire insertion as the critical and US-dependent portion of the larger procedure. As motion sensor accuracy degrades with distance, other elements that involve large hand movements, such as tray preparation and sterile draping, are difficult to study using hand and instrument motion and thus were not included. Second, the study involved just 10 novice trainees in EM and 10 expert critical care physicians and fellows. We recognize the variation that could exist across types of trainees and experts and thus the potential limitation in generalizability of these findings to other groups. Third, success was defined a priori as placement of a guide wire in the internal jugular vein. In this study, this was ensured by both direct supervision and by capture in video recordings. We do recognize that the potential does exist to game the system should direct oversight not be provided. In future studies, we plan to incorporate internal verifications using the three-dimensional position of the needle tip in relation to the manikin s internal jugular vein to prevent this from happening. Finally, the external validation of this study was performed using a blinded local expert with a modified GRS. Using only four domains of the GRS for evaluation, as opposed to the eight described in the originally validated tool, may have resulted in unintentional bias. This potential bias was minimized due to the proportional reduction in both demonstrated (subjects were not required to perform anesthesia and establish and maintain sterility) and evaluated steps. Further, these We have demonstrated that electromagnetic hand and instrument motion analysis is technically feasible for assessing competence in ultrasound-guided insertion of a central venous catheter guide wire in a simulation setting. In showing that this assessment method can discriminate between the performances of nonexperts and experts, we have provided evidence for construct validity. Hand and instrument motion analysis also shows excellent correlation with a modified version of a previously validated GRS, thus providing evidence of concurrent validity. With the adoption of competency-based medical education in medical curricula throughout North America, the development of valid assessment strategies is critical. Electromagnetic hand and instrument motion tracking, as reported here, has potential to provide a useful tool for assessment in a competency-based medical education program targeting central venous catheter insertion. We acknowledge the anonymous reviewers of an earlier draft of this manuscript for the thorough and helpful comments. References 1. Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a definition of competencybased education in medicine: a systematic review of published definitions. Med Teach 2010;32: Frank JR, Snell LS, Cate OT, et al. Competencybased medical education: theory to practice. Med Teach 2010;32: Snell LS, Frank JR. Competencies, the tea bag model, and the end of time. Med Teach 2010;32: Frank JR (editor). The CanMEDS 2005 Physician Competency Framework: Better Standards, Better Physicians, Better Care. Royal College of Physicians and Surgeons of Canada. Available at: ollege.ca/portal/page/portal/rc/common/documents/can meds/resources/publications/framework_full_e.pdf. Acc essed Nov 12, Dong Y, Suri HS, Cook DA, et al. Simulation-based objective assessment discerns clinical proficiency in central line placement: a construct validation. Chest 2010;137: Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, Wayne DB. Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med 2009;4: Sanfey H, Dunnington G. Verification of proficiency: a prerequisite for clinical experience. Surg Clin North Am 2010;90: Moorthy K, Munz Y, Sarker SK, Darzi A. Objective assessment of technical skills in surgery. Br Med J 2003;327:1032.

7 218 Clinkard et al. HAND MOTION ANALYSIS TO EVALUATE COMPETENCY 9. Martin J, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84: Reznick RK, MacRae H. Teaching surgical skills changes in the wind. N Engl J Med 2006;355: Fitts PM, Posner MI. Human Performance. Oxford, England: Brooks/Cole, Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB. Observational tools for assessment of procedural skills: a systematic review. Am J Surg 2011;202: Van Hove P, Tuijthof G, Verdaasdonk E, Stassen L, Dankelman J. Objective assessment of technical surgical skills. Br J Surg 2010;97: Ma IW, Zalunardo N, Pachev G, et al. Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ 2012;17: Ungi TT. Perk Tutor: An open-source training platform for ultrasound-guided needle insertions. IEEE Trans Biomed Eng 2012;59: Moult E, Ungi T, Welch M, Lu J, McGraw R, Fichtinger G. Ultrasound-guided facet joint injection training using Perk Tutor. Int J Comput Assist Radiol Surg 2013;8: Hayter MA, Friedman Z, Hanlon JG, Katznelson R, Borges B, Naik VN. Validation of the Imperial College Surgical Assessment Device (ICSAD) for labour epidural placement. Can J Anesth 2009;56: Datta V, Chang A, Mackay S, Darzi A. The relationship between motion analysis and surgical technical assessments. Am J Surg 2002;184: Kumar A, Chuan A. Ultrasound guided vascular access: efficacy and safety. Best Pract Res Clin Anaesthesiol 2009;23: Supporting Information The following supporting information is available in the online version of this paper: Data Supplement S1. Modified global ratings scale.