Learning Curves in Emergency Ultrasound Education

Transcription

1 EDUCATIONAL ADVANCE Learning Curves in Emergency Ultrasound Education David J. Blehar, MD, Bruce Barton, PhD, and Romolo J. Gaspari, MD, PhD Abstract Objectives: Proficiency in the use of bedside ultrasound (US) has become standard in emergency medicine residency training. While milestones have been established for this training, supporting data for minimum standard experience are lacking. The objective of this study was to characterize US learning curves to identify performance plateaus for both image acquisition and interpretation, as well as compare performance characteristics of learners to those of expert sonographers. Methods: A retrospective review of an US database was conducted at a single academic institution. Each examination was scored for agreement between the learner and expert reviewer interpretation and given a score for image quality. A locally weighted scatterplot smoothing method was used to generate a model of predicted performance for each individual examination type. Performance characteristics for expert sonographers at the site were also tracked and used in addition to performance plateaus as benchmarks for learning curve analysis. Results: There were 52,408 US examinations performed between May 2007 and January 2013 and included for analysis. Performance plateaus occurred at different points for different US protocols, from 18 examinations for soft tissue image quality to 90 examinations for right upper quadrant image interpretation. For the majority of examination types, a range of 50 to 75 examinations resulted in both excellent interpretation (sensitivity > 84% and specificity > 90%) and good image quality (90% the image quality benchmark of expert sonographers). Conclusions: Educational performance benchmarks occur at variable points for image interpretation and image quality for different examination types. These data should be considered when developing training standards for US education as well as experience requirements for US credentialing. ACADEMIC EMERGENCY MEDICINE 2015;22: by the Society for Academic Emergency Medicine The use of clinician-performed ultrasound (US) examination has increased dramatically over the past several decades. Initially adopted by relatively few physicians, it has become part of the standard practice of emergency medicine (EM) in academic and community settings alike and is now considered a requisite skill for graduating EM residents. The American College of Emergency Physicians (ACEP) instated US training guidelines based on expert consensus in 2001 and again in In 2008 the Council of Emergency Medicine Residency Directors (CORD) introduced minimum training guidelines for clinician-performed US. In 2012 the Accreditation Council for General Medical Education (ACGME) designated US as one of the milestones competencies for graduating EM residents. 2 The initial ACEP guidelines focused on the number of US examinations that were required to be performed by a physician prior to being considered competent. In the most recent ACEP guidelines, it is recommended that 25 to 50 examinations be performed for each of the core applications. While the ACEP guidelines note that other metrics may be used to determine competency, the absolute number of US examinations performed remains the most common (and easiest to obtain) metric. Likewise, while the ACGME milestone competencies focus on a variety of metrics, they do include a cutoff of 150 total examinations as minimum experience to complete residency training. Choosing a specific number as a benchmark for competency remains a theme for national guidelines in EM and other specialties. From the Department of Emergency Medicine (DJB, RJG) and the Department of Quantitative Health Sciences (BB), University of Massachusetts Medical School, Worcester, MA. Received July 29, 2014; revision received November 7, 2014; accepted November 10, Presented at the Society for Academic Emergency Medicine, Dallas, TX, May The authors have no relevant financial information or potential conflicts to disclose. Supervising Editor: John H. Burton, MD. Address for correspondence and reprints: David J. Blehar, MD; david.blehar@umassmemorial.org by the Society for Academic Emergency Medicine ISSN doi: /acem PII ISSN

2 2 Blehar et al. LEARNING CURVES IN EMERGENCY US EDUCATION The recommended 25 to 50 examination cutoff was chosen based on expert consensus, as there has been very little published literature regarding the learning curves for US. Previous studies on learning curves for clinician-performed US have focused primarily on interpretation metrics without consideration for image acquisition skill. 3 A consensus conference on how to evaluate for US competency identified a number of elements, including image optimization and image interpretation, but little data exist on how these metric change as individuals gain experience. 4 The objective of this study was to characterize learning curves for novice physician sonographers to identify experience levels where educational performance plateaus occur. We additionally sought to identify experience levels requisite for training sonographers to approach the performance standards of expert physician sonographers for image acquisition and interpretation. This study uses a large educational database from a single site to calculate learning curves for each of the core EM US applications. We compare learner metrics to expert metrics to introduce a discussion on learning curves and competency for each individual US application. METHODS Study Design This study was a retrospective review of an educational database from a single EM residency training program over a 5-year period, from July 2007 to June The study was reviewed by the local institutional review board and was determined to be exempt from informed consent requirements. Study Setting and Population Ultrasound examinations in the database are a combination of those performed in the course of clinical care as well as those performed by learning sonographers solely for educational purposes across four separate emergency departments (EDs) staffed by a single academic EM physician group. The four sites ranged in character from a small community ED with annual volume of 20,000 patients, to a large urban Level I trauma center with annual volume over 100,000 patients. The US machines used during the study period varied by location and over time, with acquisition of newer equipment as older machines were removed from service. Equipment used included a range of Sonosite machines (Titan and MicroMaxx, Sonosite, Bothel, WA) and Zonare machines (Zone Ultra, ZS3, Zonare Medical Systems, Inc., Mountain View, CA). Still and video images were captured in DICOM video file format stored to DVD storage or USB drive (Zonare) or recorded as continuous video clips to DVD recorder (Sonosite). Changes in equipment did not have a significant effect on image quality or accuracy metrics. There was no significant difference in image quality (p = 0.62, Student s t-test) between the first and last year of our study data. Similarly, sensitivity (0.86% vs. 0.88%) and specificity (0.96% and 0.97%) did not differ between the first and last year in the study. Learning sonographers in the database included not only training residents, but also attending physician staff without prior US training or experience. All sonographers, regardless of whether they were residents or faculty, underwent similar education. Prior to inclusion in the database, they underwent a 1-day educational session focusing on basic principles of the core US applications. Residents participated in 1 dedicated month in each of their 3 years of training that included a focus on US, during which time they attended weekly educational sessions. These periodic educational sessions focused on core US applications including limited US of the aorta, chest wall, endovaginal uterus, focused assessment with sonography in trauma (FAST), right upper quadrant (RUQ), lower-extremity duplex, renal, soft tissue, and limited cardiac echo. Each educational session consisted of a 1-hour lecture, 60 to 90 minutes of hands-on education, and 60 to 90 minutes of image review. Faculty learners attended identical educational sessions over a similar 3-year time period. All learners, including both residents and faculty, attended between eight and 12 of these complete sessions. Additional hands-on sessions were available for learners. An additional US elective was available to third-year residents as either a 2- or a 4-week rotation. All learners were able to perform and record US during clinical shifts and received automated timely feedback on their images using the system described in the next section. Study Protocol Digital video of every US examination performed in the ED was recorded at the time of performance and uploaded into an electronic database for review. All pertinent patient information, the indication for imaging, and the initial image interpretation of the diagnostic study by the sonographer performing the examination were recorded. Examinations performed solely for the purposes of the educational experience of the learning sonographer were logged as such by the sonographer and designated as educational US in the database. All other examination indications were designated as clinical US. All images were reviewed in a standardized fashion by one of five physicians expert in bedside US. The US experience of the expert reviewers ranged from 5 years and over 4,000 US examinations to 15 years and over 20,000 US performed and/or reviewed. Image reviews were performed unblinded to the initial interpretation, the identity of the patient, or the performing sonographer. Required images and image elements differed for the different US protocols performed in this study. These requirements did not change regardless of the experience level or type of learner (resident or faculty). Some protocols required a specific number of images, such as four images for the FAST exam and two images for soft tissue examination, while other protocols could require a variable number of images. For example, chest wall US required at least one image of each hemithorax, but multiple views of each hemithorax could be submitted if needed. Lower-extremity duplex could include between six and 12 (or more) paired images of compressed and uncompressed veins in the lower leg. The image review focused on image interpretation, image acquisition skill, and resultant image quality. Image interpretation consisted of a predefined

3 ACADEMIC EMERGENCY MEDICINE 3 structured interpretation focusing on a primary finding specific to each individual US application. Interpretation was dichotomous, either positive or negative for the primary finding for that application. A listing of the core US applications and the primary findings are included in Table 1. Each examination was scored for agreement between the initial interpretation (learner) and final review (expert). During the initial study period (first 3 years of the database), image review was performed twice weekly by one of three expert physician reviewers. During the final 2 years of the study period, image review was performed on a daily basis Monday through Friday by one of five reviewers. This image review performed by the expert reviewer also focused on metrics related to image acquisition. These metrics included image quality ratings for the images as a whole on a predetermined ordinal scale from 1 to 8, with 8 representing perfect image quality and 1 representing poor image quality. Any score of 4 or less was predefined to indicate that the images were of sufficiently poor quality to adversely affect the interpretation. All data were recorded in the electronic database and sent back to the initial sonographer via for feedback at the time of the review. Data were downloaded from the electronic database for all users in the system. US examinations performed by any individual with experience that preceded inclusion in the electronic tracking system were excluded from data analysis. In addition, all US of noncore applications were excluded from analysis (e.g., musculoskeletal, US-guided nerve blocks, ocular US). For analysis of interpretative skill, those US that did not include an interpretation by the initial sonographer were excluded. Table 1 Definition of Ultrasound Terms and Their Findings Term Finding Short Definition of Positive Finding Aorta Abdominal aortic Diameter of aorta measuring 3 cm. aneurysm Cardiac Pericardial effusion Anechoic fluid visualized in pericardial space. Chest wall Pneumothorax Lack of visualized sliding of interface between chest wall and lung tissue. Endovaginal uterine Intrauterine pregnancy Visualized fetus and or yolk sac in endometrial cavity. FAST Free fluid Anechoic fluid visualized in peritoneal or pericardial space. Lowerextremity duplex Deep vein thrombosis Inability to compress deep veins of lower extremity. Renal Hydronephrosis Visualized branched anechoic center to kidney. Right upper quadrant Gallstones Mobile hyperechoic structures in lumen of gallbladder. Soft tissue Abscess Disruption of superficial soft tissue with focal collection of anechoic fluid. FAST = focused assessment with sonography in trauma. Image quality metrics were converted to an ordinal scale representing a dichotomous quality metric of poor quality limiting interpretation (score of 4 or below) or good quality (score of 5 or greater). Data Analysis The data were not normally distributed (Kolmogorov- Smirnov goodness-of-fit test for uniform distributions) and are presented as medians with interquartile ranges (IQR). Learner interpretation was analyzed using sensitivity and specificity analysis with the findings from expert review as the criterion standard. All examinations were included for analysis, regardless of image quality. Learner image quality was assessed by comparing their image quality to the image quality of the expert reviewers independent imaging performed during the study. To describe expert reviewer benchmarks related to image quality, the US images obtained and interpreted by the expert reviewers were analyzed separately. Each image was interpreted and reviewed in a fashion identical to those included in the study database. Data on the average image quality for each US application were calculated. Internal validity of the rating scale was analyzed by comparing the agreement of the five expert reviewers. All reviewers independently and blindly reviewed and rated 100 randomly selected US images from the database for image quality. Agreement between reviewers was analyzed using a kappa analysis on those image interpretations. A locally weighted scatterplot smoothing method (loess) was used to generate a model of predicted performance (+95% CI) for each individual examination type. Performance curves were analyzed to determine plateau points where experiential benefit diminished. In addition, performance curves were analyzed with a reference to the performance of the expert reviewers. SAS (version 9.3) software was used for data analyses. Not all patients will have perfect imaging, so data for image quality was normalized to the image quality for images obtained by the experts. In the displayed data, 100% references the average image quality of the experts for that US protocol and 0 represents the lowest possible score for the scale. While a plateau point within a curve can be estimated visually, for the purposes of this study the plateau was mathematically defined as the point where there was a change in slope for the interval immediately preceding a specific point of US experience (x axis) compared to the slope immediately following that point. Slopes were calculated as the change in percent experience (y axis) over the number of US performed (x axis). For the purpose of this study, only slope changes greater than 25% were considered eligible as a plateau point. For curves with multiple changes in slope, the point of the greatest slope change was used as the plateau point for this study. RESULTS A total of 191 EPs performed a total of 89,052 US examinations. After excluding experienced physicians (those

4 4 Blehar et al. LEARNING CURVES IN EMERGENCY US EDUCATION who began US imaging prior to starting this study) and noncore applications, 101 EPs and 52,408 examinations were included in the data set for analysis. US examinations in the database included those performed for educational purposes (33%), as well as clinical purposes (66%). There were nine image applications included in the final data set, representing the core US applications as defined by ACEP. The number of US examinations in each application ranged from 12,963 (FAST) to 1,253 (endovaginal uterus). Overall, 12.5% of imaging was positive for pathology, but the percentage of positive findings varied by US application, with a range of 4.5% (chest wall) to 56% (soft tissue; see Table 2). The median image rating (rated from 1 to 8) for images obtained by expert sonographers provided a benchmark for the image ratings of learners. The median image quality score for each of the US applications for those images obtained by expert physicians was as follows: aorta, 7 (IQR = 5 to 8); cardiac, 6 (IQR = 5 to 8); chest wall, 8 (IQR = 7 to 8); endovaginal uterus, 8 (IQR = 7 to 8); FAST, 7 (IQR 6 to 8); lower-extremity duplex, 7 (IQR = 6 to 8); renal, 7 (IQR = 6 to 8); RUQ, 7 (IQR = 6to 8); and soft tissue, 8 (IQR = 7 to 8). The overall image quality and agreement with final interpretation varied by US application for US acquired by learners. Agreement between reviewers related to rating the images on the image rating scale was good (j = 0.81). The median image rating for all US in the database was 6 on the 8-point ordinal scale. The US applications with the worst image quality were the cardiac and aorta examinations, with median scores of 6 (IQRs 5 to 8), while the chest wall and renal examinations had the best mean image rating, with medians of 7 (IQR = 7 to 8) for chest wall and 7 (IQR = 6 to 8) for renal. The overall agreement between the initial sonographer interpretation and the expert review was 95.9%, but there was variability by US application. The learning curves for image interpretation differed by US application with regard to initial agreement level, slope of learning curve, and overall shape of the learning curve. Most of the learning curves demonstrate a slow steady improvement in agreement as experience increases until a point of plateau, where further experience is associated with little or no improvement. All image interpretation learning curves are displayed in Figure 1. Interpretation performance plateaus differed for the different imaging protocols (Table 3). Some US protocols had plateau points that occurred at relatively lower experience levels. Interpretation plateaus for soft tissue (for abscess) and cardiac (for pericardial effusion) occurred at 27 and 30 US, respectively. Other examinations such as FAST, chest wall, and aorta required more experience (57, 60, and 66 US, respectively). Renal (78 US) and RUQ (90 US) required the most experience to reach interpretation plateaus. Plateau points for endovaginal uterus and lower-extremity duplex were not definable. Another way to analyze the changes in interpretation performance for learners over time is to compare sensitivity and specificity of learners as they accumulate experience. For most protocols, the sensitivity and specificity improved over time (Figures 2 and 3). Soft tissue and endovaginal uterus were the easiest to learn to interpret, with sensitivities in the mid-90s. The FAST exam was the hardest protocol to learn to interpret, as it demonstrated the lowest sensitivity, with a peak around 80%. All protocols had excellent specificity, with soft tissue and renal the only protocols below a specificity of 96%. Performing 50 US is a common goal for many learners based on current guidelines, and this would produce a sensitivity and specificity greater than 84 and 90%, respectively, for all US protocols with the exception of the FAST exam, where performing 50 US produces sensitivity of 80% and specificity of 96%. The learning curves for image acquisition also differed by application (see Figure 4). Performance plateaus were identified for five of nine examination types and occurred earliest for soft tissue (18), and latest for aorta (84; see Table 3). The four learning curves that did not display plateau points did so because their curves demonstrated no change in slope for three, and there was insufficient data for one. Image quality was easiest to obtain for chest wall US, where performing three US resulted in 95% of the image quality obtained by experts. Image quality was hardest to learn for cardiac, endovaginal uterus, and lower-extremity duplex, where learners never surpassed 90% of the image quality of Table 2 Characteristics of Ultrasounds Included in the Database Examination Type Number of US Examinations Number of Sonographers Positive Findings, n (%) Image Rating, Mean (95% CI) Interpretation Agreement (%) Aorta 6, (5.3) 5.8 ( ) 98.7 Cardiac 5, (8.9) 5.4 ( ) 95.6 Chest wall 6, (4.5) 7.0 ( ) 98.6 Endovaginal uterine 1, (24.6) 5.8 ( ) 90.7 FAST 12, ,290 (10.0) 6.2 ( ) 95.7 Lower-extremity duplex 2, (6.1) 5.8 ( ) 96.9 Renal 6, (13.0) 6.3 ( ) 93.2 Right upper quadrant 8, ,432 (17.6) 6.0 ( ) 95.5 Soft tissue 2, ,377 (56.3) 6.8 ( ) 92.2 Total 52, ,546 (12.5) 6.1 ( ) 95.9 FAST = focused assessment with sonography for trauma.

5 ACADEMIC EMERGENCY MEDICINE 5 (A) (B) (C) (D) (E) (F) (G) (H) (I) Figure 1. Image interpretation learning curves. Predicted percentage agreement with expert review is plotted as a function of examination experience with surrounding 95% CI. the experts. Both FAST and RUQ demonstrated long learning curves, where it took 183 and 96 US, respectively, to reach 90% of the image quality of an expert. As mentioned previously, performing 50 US is a common goal for many learners based on current guidelines, and this would produce different skill levels for the different US protocols. Performing 50 US resulted in an image quality relative to expert as follows: aorta (73%), cardiac (68%), chest wall (93%), endovaginal uterus (>76%), FAST (84%), lower-extremity duplex (51%), renal (87%), RUQ (79%), and soft tissue (94%). DISCUSSION It is logical to expect that individuals learning US will demonstrate an increase in skill level over time. The learning curves for interpretation in this article demonstrate a gradual improvement over time, but the degree of improvement was relatively shallow. This may be related to the fact that the agreement was relatively high for most US applications, even for the first few US in the learning experience. For some of the examination types, this initial high performance level may in part explain the absence of a discernable performance plateau (e.g., FAST exam image quality). A few of the learning curves (FAST, cardiac) demonstrate a small decrease over the later stages of experience. It is unclear what is responsible for this decrease in interpretation performance. One possibility is that more experienced learners interpreted trace fluid as negative, mistaking the clinical significance with the actual finding. Another possibility is that patient selection for

6 6 Blehar et al. LEARNING CURVES IN EMERGENCY US EDUCATION Table 3 Plateau Points for Interpretation and Image Acquisition Metrics Examination Interpretation Image Acquisition Aorta Cardiac Chest wall Endovaginal uterine None None FAST 57 None Lower-extremity duplex None None Renal Right upper quadrant 90 None Soft tissue Data represent the number of ultrasounds performed prior to reaching a plateau in the learning curve of the indicated metric. Plateau is defined as a decrease in slope of learning curve >25%. The curve for endovaginal uterus did not have enough data points to calculate a plateau point. All other curves without a plateau point did not demonstrate sufficient changes in slope to calculate a plateau point. Figure 2. Sensitivity of ultrasound by examination type compared to interpretation by expert reviewer. Figure 3. Specificity of ultrasound by examination type compared to interpretation by expert reviewer. learning sonographers with more relative experience includes difficult patients who are avoided by more novice sonographers. It is also possible that not all individuals will demonstrate increases in skill over time. A study examining cardiologists found no association between experience and proficiency. 5 One final possibility relates to the fact that FAST and cardiac examinations demonstrated proportionally more educational US during the earlier learning phases. These would be more likely to be obtained during nonclinical times when the sonographer has more time to obtain and interpret images, thus artificially enhancing performance. Even including the initial US experience, our overall interpretation agreement rate of 95.9% compares favorably to published rates for other specialties. Discrepancy rates of US interpretation between radiology residents and faculty range from 0.2% to 4.0%. 6 9 A renal US study in 2012 found good agreement between two experienced radiologists (j = 0.82), similar to the agreement in our study. 10 Studies comparing cardiology fellows using portable US to cardiology faculty found comparable agreement to our study (j = 0.66 to 0.89). 11,12 Echocardiography studies involving internists or EPs demonstrate similar agreement (j = 0.79). 13,14 A study by novice surgeons learning RUQ US demonstrated less agreement (j = 0.40), significantly lower than the agreement in this article (j = 0.87). 15 The most likely explanation for this difference is that their training was significantly less than the training of the learners in this article. Similar to image interpretation, it is logical to expect that imaging technique will improve as experience increases. However, not all of the learning curves for image quality improved over time. Three of the US applications (aorta, lower extremity duplex, and endovaginal uterus) demonstrated decreases in image quality over the initial experience before increasing back to baseline. We speculate that this decrease relates to patient selection, as new learners choose patients who are easier to image (thinner, fasting patients) and more experienced sonographers recorded examinations from more challenging patients based on clinical necessity. It is also possible that the decrease in image quality is independent of the sonographer, and some of the patients at different times had higher percentages of characteristics that decrease US imaging (for example, obesity, recent food intake, increased intestinal gas or pain tolerance, and ability to cooperate with emergent imaging). There are little published data related to image acquisition for US. One study focusing on acquisition of images for the FAST exam found that US technique improved even after 75 US examinations. 16 The learning curve in this article demonstrates similar findings, in that the image quality for the FAST exam increased even after 200 US. A study of internists found that even after 35 cardiac US, their image quality was not equal to experienced sonographers. 17 Our learning curve demonstrates that learners do not reach the quality equivalent of expert sonographers even after 120 US. A study exploring learning curves for US-guided interventions found that students needed between 37 and 109 US-guided procedures to gain competency. 18 Many national groups recommend or require a certain number of US for training, in effect using the number of US as a surrogate for length of experience and by extension competency. In most of these cases, the specific numbers are chosen by expert consensus and are not based on any data. In 2008 ACEP recommended that physician perform 25 to 50 US for each application except US-guided procedures, which required 10 US

7 ACADEMIC EMERGENCY MEDICINE 7 (A) (B) (C) (D) (E) (F) (G) (H) (I) Figure 4. Image quality learning curve. Image quality is expressed as percentage of examinations at a given experience level that are predicted to be of high quality (rating scale of 5 or greater) with surrounding 95% CI. each. 1 The American Registry for Diagnostic Medical Sonography requires that physicians perform 800 US prior to taking a test to certify competency ( The American College of Cardiology and the American Heart Association published recommendations in 2003 that learners perform 150 cardiac echoes prior to independently performing echocardiography. 19 The American College of Radiology requires that physicians document at least 500 US to certify competency ( Translating the performance curves into a metric for competency is more complicated, as the definition of competency with regard to US imaging is unclear. Performance plateaus found in our study offer a guide for US education to understand at what point additional experience offers minimal improvement in image acquisition or interpretation. We have additionally assessed the performance of the expert reviewers as a yardstick to serve as a comparison for the individuals who are learning US. Although the ideal rating for image quality on our scale would be 8.0, there are many contributors to decreased image quality, including patient factors (oral intake, body habitus, painful condition, urgency of imaging) and sonographer factors (skill level, attention to detail), to name a few. Similarly, the ideal agreement would be 100%, but even expert reviewers sometimes disagree with each other. Some statisticians define a kappa of 0.80 or greater as nearly perfect. 20 Some of the newer guidelines on proficiency in US have moved away from requiring a specific number of examinations. The guidelines for proficiency in critical care US describe educational goals without a reference to number of US performed. 21 CORD has instituted milestones, where US competency is assessed through

8 8 Blehar et al. LEARNING CURVES IN EMERGENCY US EDUCATION simulation, direct observation, and examinations. 2 This trend makes logical sense, as not every individual learns at the same speed, and periodic assessments will allow a greater certainty that an individual is competent. From a group perspective, the data presented in this article provide some understanding of how a given experience level translates into a predicted level of performance. LIMITATIONS This study was conducted at a single center with a common educational process for all learning sonographers supported by rapid interpretative and technical feedback. This potentially limits applicability in other centers with different educational programs. Specific educational styles can affect both individual skill acquisition rates and practice patterns of US utilization. However, it is unclear how unevenly distributed imaging, or periods of inactivity during a period of learning US, affect learning. It is possible that skill decay is seen following periods of inactivity, but our study does not address this issue. In this study our learners sometimes benefit from more experienced individuals (residents and staff) present during image acquisition and interpretation. Our database does not specifically track such interactions, and therefore the effect of this interaction cannot be quantified in the analysis of our data. Although it is possible that interactions with more experienced staff during image acquisition could influence learning curves, this situation exists in many EDs, and it should not limit the generalizability of our findings. The image rating scale used in this study has not been previously validated and is potentially unique to our center. Simplifying the scale to a dichotomous variable of good versus poor image quality not only assists in statistical analysis, but also converts the scale to one we believe is easily applied to other educational programs. Our primary outcome measure relates to performance of expert sonographers at our center. While all five experts in this study have extensive training and experience (as detailed under Methods), there is no standard to define expertise in emergency US, and this creates another limitation to the generalizability of the results of this study to other centers. CONCLUSIONS Educational performance curves vary by ultrasound application, not only for image acquisition skill but also for image interpretation. While not providing an absolute cutoff for requisite examination experience, our results would suggest that for the majority of ultrasound examination types, a minimum of 50 examinations, as suggested by the current American College of Emergency Physicians guidelines, will result in a reasonable performance level compared to expert sonographers. References 1. Emergency ultrasound guidelines. Ann Emerg Med 2009;53: Lewiss RE, Pearl M, Nomura JT, et al. CORD-AEUS: consensus document for the emergency ultrasound milestone project. Acad Emerg Med 2013;20: Gaspari RJ, Dickman E, Blehar D. Learning curve of bedside ultrasound of the gallbladder. J Emerg Med 2009;37: Tolsgaard MG, Todsen T, Sorensen JL, et al. International multispecialty consensus on how to evaluate ultrasound competence: a Delphi consensus survey. PloS One 2013;8:e Nair P, Siu SC, Sloggett CE, Biclar L, Sidhu RS, Yu EH. The assessment of technical and interpretative proficiency in echocardiography. J Am Soc Echocardiogr 2006;19: Ruma J, Klein KA, Chong S, et al. Cross-sectional examination interpretation discrepancies between on-call diagnostic radiology residents and subspecialty faculty radiologists: analysis by imaging modality and subspecialty. J Am Coll Radiol 2011;8: Ruutiainen AT, Durand DJ, Scanlon MH, Itri JN. Increased error rates in preliminary reports issued by radiology residents working more than 10 consecutive hours overnight. Acad Radiol 2013;20: Ruchman RB, Jaeger J, Wiggins EF 3rd, et al. Preliminary radiology resident interpretations versus final attending radiologist interpretations and the impact on patient care in a community hospital. Am J Roentgenol 2007;189: Ruutiainen AT, Scanlon MH, Itri JN. Identifying benchmarks for discrepancy rates in preliminary interpretations provided by radiology trainees at an academic institution. J Am Coll Radiol 2011;8: Rud O, Moersler J, Peter J, et al. Prospective evaluation of interobserver variability of the hydronephrosis index and the renal resistive index as sonographic examination methods for the evaluation of acute hydronephrosis. BJU Int 2012;110: E Giusca S, Jurcut R, Ticulescu R, et al. Accuracy of handheld echocardiography for bedside diagnostic evaluation in a tertiary cardiology center: comparison with standard echocardiography. Echocardiography 2011;28: Borges AC, Knebel F, Walde T, Sanad W, Baumann G. Diagnostic accuracy of new handheld echocardiography with Doppler and harmonic imaging properties. J Am Soc Echocardiogr 2004;17: Bustam A, Noor Azhar M, Singh Veriah R, Arumugam K, Loch A. Performance of emergency physicians in point-of-care echocardiography following limited training. Emerg Med J 2014;31: Vignon P, Mucke F, Bellec F, et al. Basic critical care echocardiography: validation of a curriculum dedicated to noncardiologist residents. Crit Care Med 2011;39: Eiberg JP, Grantcharov TP, Eriksen JR, et al. Ultrasound of the acute abdomen performed by surgeons in training. Minerva Chir 2008;63: Jang T, Kryder G, Sineff S, Naunheim R, Aubin C, Kaji AH. The technical errors of physicians learning

9 ACADEMIC EMERGENCY MEDICINE 9 to perform focused assessment with sonography in trauma. Acad Emerg Med 2012;19: Martin LD, Howell EE, Ziegelstein RC, Martire C, Shapiro EP, Hellmann DB. Hospitalist performance of cardiac hand-carried ultrasound after focused training. Am J Med 2007;120: de Oliveira Filho GR, Helayel PE, da Conceicao DB, Garzel IS, Pavei P, Ceccon MS. Learning curves and mathematical models for interventional ultrasound basic skills. Anesthes Analg 2008; 106: Quinones MA, Douglas PS, Foster E, et al. ACC/ AHA clinical competence statement on echocardiography: a report of the American College of Cardiology/American Heart Association/American College of Physicians-American Society of Internal Medicine Task Force on clinical competence. J Am Soc Echocardiogr 2003;16: Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med 2005; 37: Mayo PH, Beaulieu Y, Doelken P, et al. American College of Chest Physicians/La Societe de Reanimation de Langue Francaise statement on competence in critical care ultrasonography. Chest 2009;135: