ORIGINAL RESEARCH. Abbreviations LV, left ventricular; PE, pulmonary embolism; RV, right ventricular; 2D, 2-dimensional. doi: /ultra.15.

Transcription

1 ORIGINAL RESEARCH Interobserver and Intraobserver Agreement on Qualitative Assessments of Right Ventricular Dysfunction With Echocardiography in Patients With Pulmonary Embolism Anthony J. Weekes, MD, Laura Oh, MD, Gregory Thacker MD, Angela K. Johnson, MD, Michael Runyon, MD, Geoffrey Rose, MD, Thomas Johnson, MD, Megan Templin, MS, H. James Norton, PhD Received November 3, 2015, from the Department of Emergency Medicine (A.J.W., G.T., A.K.J., M.R.), Sanger Cardiology, (G.R., T.J.), and Biostatistics Facility (M.T., H.J.N.), Carolinas Medical Center, Charlotte, North Carolina USA; and Emory University, Atlanta, Georgia USA (L.O.). Revision requested December 3, Revised manuscript accepted for publication December 29, Address correspondence to Anthony J Weekes, MD, Department of Emergency Medicine, Carolinas Medical Center, 1000 Blythe Blvd, Charlotte, NC USA. anthony.weekes1@gmail.com Abbreviations LV, left ventricular; PE, pulmonary embolism; RV, right ventricular; 2D, 2-dimensional doi: /ultra Objectives To evaluate observer agreement using qualitative goal-directed echocardiographic criteria for right ventricular (RV) dysfunction prognostication in submassive pulmonary embolism (PE). Methods Two emergency physicians and 2 cardiologists independently reviewed 31 packets of goal-directed echocardiographic video clips consisting of at least 3 windows obtained by emergency physicians from normotensive patients with PE. Nine packets were repeated to assess for intraobserver agreement. Right ventricular dysfunction criteria on goal-directed echocardiography were as follows: RV enlargement was present, with a right-to-left ventricular basal diameter ratio of 1.0 or higher and blunting of the apex of the RV in 2 or more different windows; RV systolic dysfunction was present if the tricuspid annulus moved toward the apex 10 mm or less and there was RV free wall hypokinesis; and septal deviation was present with any flattening or deviation of the ventricular septum toward the left ventricle. Results Among the 4 participants, there was 83.9% agreement on the presence or absence of RV enlargement (κ = 0.84), 74.2% agreement on the presence or absence of RV systolic dysfunction (κ = 0.69), and 71.0% agreement on the presence or absence of septal deviation (κ = 0.59). Intraobserver agreement was 100% for each RV dysfunction variable for each observer (κ = 1.0). Conclusions Agreement was substantial for both severe RV enlargement and RV systolic dysfunction and moderate for septal deviation. Right ventricular dysfunction assessment with qualitative goal-directed echocardiographic criteria is reproducible for PE risk stratification. Key Words echocardiography; observer agreement; pulmonary embolism; right ventricular dysfunction Pulmonary embolism (PE) presentations are varied, and clinical management can be challenging. Although up to 5% of PE cases are clinically unstable at the onset, most patients present with normal blood pressure. It is important to identify the patients within this group of normotensive PE presentations that may benefit from closer monitoring for subsequent adverse clinical outcomes and consideration for escalated treatment (eg, thrombolysis) over standard anticoagulation therapy. Goal-directed 2016 by the American Institute of Ultrasound in Medicine J Ultrasound Med 2016; 35:

2 echocardiography is a valuable first-line risk stratification tool for physicians treating patients with PE. It allows the categorization of patients with normotensive PE into low and moderate risk groups based on the presence or absence of right ventricular (RV) dysfunction. Right ventricular dysfunction is associated with increased mortality in patients with PE and poorer functional outcomes in survivors. 1 4 Although the American College of Emergency Physicians and the American Society of Echocardiography support the use of goal-directed echocardiography for determining the presence or absence of RV dysfunction, the interobserver and intraobserver agreement of RV dysfunction assessment among emergency physicians and cardiologists interpreting goal-directed echocardiography is largely unknown. 5 Despite numerous studies on left ventricular (LV) function and pericardial effusion detection by emergency physicians, there are very few reports focused on qualitative RV dysfunction assessment. 6 9 With its complex crescentlike shape, size, geometry, and function, RV assessment can be challenging. Cardiologists usually interpret comprehensive echocardiography after a trained cardiac sonographer acquires images and measurements. Interpretation of comprehensive echocardiography involves review of multiple measurements and images with the use of many ultrasound modalities beyond 2-dimensional (2D) such as M-mode and various forms of Doppler technology (color, pulsed wave, continuous wave, tissue, and spectral Doppler). Comprehensive echocardiography also involves assessments of valve function, distinguishes between acute and chronic RV dysfunction, and assesses for the presence of a patent foramen ovale. Several studies have used different echocardiographic RV dysfunction definitions for PE risk stratification; although most included RV enlargement, there were varying thresholds: some included only RV hypokinesis, whereas others used various Doppler measurement thresholds Comprehensive echocardiography is not always immediately available for early risk stratification of patients with PE. In contrast, goal-directed echocardiography is usually immediately available. Emergency physicians perform and interpret goal-directed echocardiography. The most important and primary goal of goal-directed echocardiography for PE-related RV dysfunction is the identification of RV enlargement. Secondary goals include the detection of septal deviation and RV systolic dysfunction. We propose an approach to visually estimate RV size and systolic function using only 2D echocardiographic technology. We tested the performance of our RV assessment guidelines by determining the interobserver and intraobserver agreement among cardiologists and emergency medicine residents in the identification of RV dysfunction in patients with confirmed acute PE. Previous studies have either described the interobserver reliability of RV dysfunction criteria among cardiologists using measurements or retrospectively looked at interobserver agreement of RV dysfunction between emergency physicians and comprehensive echocardiography using different RV dysfunction criteria. 3,9,14 The primary goal of this study was to determine observer agreement using predefined qualitative goaldirected echocardiographic criteria for RV dysfunction prognostication in normotensive PE. The secondary goal was to determine interobserver and intraobserver agreement between emergency physicians and cardiologists. Materials and Methods Setting This study was conducted at an urban, academic emergency department with more than 85,000 annual adult visits. Goal-directed echocardiography was performed on normotensive ED patients with acute PE. The hospital s Investigational Review Board approved the research study. Waiver of informed consent was granted. Selected saved goal-directed echocardiographic video clips (with all patient identifiers removed) were presented to 4 participants for specific interpretations. Entry Criteria Any second- or third-year emergency medicine resident was eligible to participate in the research study. The emergency ultrasound curriculum at our emergency medicine program requires all residents to complete a 1-month emergency ultrasound rotation during the first year and goal-directed cardiac ultrasound integrated in the practice of emergency medicine at our institution. Detection of RV dilatation is one of the main objectives of goal-directed echocardiography. 5 Two emergency medicine physicians volunteered to participate in the research study. Each emergency medicine resident had completed the minimum criteria of 25 cardiac ultrasound examinations during the 1- month emergency ultrasound rotation, and the emergency medicine participants were required to take a 1-hour didactic in-service session on the detection of RV dysfunction. The emergency medicine physicians who volunteered were at the very beginning of the third year of residency training when the study was conducted. Two board-certified cardiologists, each with experience reading more than 500 comprehensive echocardiograms, volunteered to be part of the research study J Ultrasound Med 2016; 35:

3 Methods Physicians viewed a series of echocardiographic video clips from normotensive patients with acute PE. Goal-directed echocardiographic digital video images from confirmed patients with acute PE were stored during a previous clinical research study. We selected packets of video images that had interpretations confirmed by comprehensive echocardiography (documented presence or absence of RV dysfunction signs) within 9 hours of goal-directed echocardiography completion. We used video images acquired with a 3 5-MHz phased array transducer and the Sparq ultrasound platform (Philips Healthcare, Andover, MA). All goal-directed echocardiographic images were acquired by emergency physicians treating the patients with PE. None of the video clips included any direct or displayed measurements or calculations. See Figure 1 for examples of images viewed. Right ventricular dysfunction criteria were defined as follows: (1) presence or absence of severe RV dilatation/ enlargement (severe dilatation is defined as an RV/LV basal diameter ratio of 1.0 or higher and blunting of the apex of the RV from 2 or more different windows); (2) presence or absence of severe RV systolic dysfunction (using tricuspid annular plane systolic excursion visual estimates of <10 mm and RV free-wall hypokinesis); and (3) presence or absence of flattening or deviation of the interventricular septum toward the LV. We also included a 60/60 guideline that we commonly teach in our didactics for detection of RV dysfunction. Normally, the RV basal diameter is 60% smaller than the basal LV diameter and the RV longitudinal shortening is 60% greater than the LV longitudinal shortening. 15 After completing a didactic session on RV dysfunction detection by echocardiography, the physicians reviewed 40 goal-directed echocardiographic packets of video clips and gave their qualitative impressions on the presence or absence of each of these 3 key features of RV dysfunction. A few months before the testing session, the emergency physicians completed a 1-hour didactic education session presented by the emergency ultrasound fellowship director. Both cardiologists reviewed the qualitative goal-directed echocardiographic criteria for RV dysfunction a few weeks before the actual testing session. A 10-minute review session on the RV dysfunction criteria was presented to all participants immediately before the testing session. The series of imbedded videos was presented in a Power- Point (Microsoft Corporation, Redmond, WA) presentation slideshow. The first 31 goal-directed echocardiographic video packets were from different patients. The final 9 packets were used for intraobserver testing and consisted of 3 RV dysfunction positive and 6 RV dysfunction negative cases randomly sequenced and selected from the previous 31 packets. Observer agreement for RV dysfunction was calculated. Each video packet contained brief video clips of 3 or 4 transthoracic 2D ultrasound windows only (no Doppler). Participants were blinded to patient identifiers and the composition and sequence of the video packets. They were also blinded to the associated symptoms and signs of the patients when reviewing their videos, minimizing undue bias during their interpretations of ultrasound findings. Each video review session was proctored. Statistical Methods The level of interobserver and intraobserver agreement in the qualitative assessment of RV dysfunction signs on echocardiography was measured. For the primary outcome, we calculated that a sample size of 30 would allow us to detect a statistically significant κ value of at least 0.6 by a 2-tailed test, with a null value of 0 and 90% power. 19 Two emergency department residents and 2 board-certified cardiologists each reviewed the video clips and gave their qualitative impressions on the presence or absence of severe RV Figure 1. Different echocardiographic features. A, Apical 4-chamber window showing the normal RV appearance (arrow). B, Parasternal long window showing the RV (arrow) and bowing of the interventricular septum toward the LV. C, Parasternal short-axis view showing RV dilatation (arrow) and flattening of the septum. D, Apical 4-chamber view showing RV dilatation (arrow), RV apex blunting, and deviation of the septum toward the LV. J Ultrasound Med 2016; 35:

4 enlargement, severe RV systolic dysfunction, and deviation of the ventricular septum contour. Interobserver agreement was measured by comparing the assessments across the emergency department residents, the boardcertified cardiologists, and all 4 observers together. Intraobserver agreement was measured by comparing a single individual s repeated assessments of the same video clip. The percentage of agreement and the κ statistic were reported for the level of agreement between raters and intraobserver agreement. The κ statistic tests whether the percentage of agreement is higher than what would be expected by chance alone. For the comparison of all 4 observers, the multiple-rater SAS macro 20 was used. The suggested guidelines of Landis and Koch 21 were used to describe the strength of agreement for the κ statistic. They suggested, and we used, the following interpretations: less than 0, poor; 0 to 0.20, slight; to 0.40, fair; 0.41 to 0.60, moderate; 0.61 to 0.80, substantial; and 0.81 to 1.00, almost perfect. SAS Enterprise Guide 6.1 (SAS Institute Inc, Cary, NC) was used for all analyses. Results Selected Characteristics of Goal-Directed Echocardiography In this study, we assessed goal-directed echocardiographic video packets from 31 independent cases of normotensive patients with acute PE whose images had previously been acquired by emergency physicians in all stages of training. Of these 31 cases, there were 18 (58%) without RV dysfunction and 13 (42%) with RV dysfunction. Ten had all 3 RV dysfunction abnormalities. The other cases with features of RV dysfunction were 1 case with RV enlargement only, 1 case with RV enlargement and septal deviation only, and 1 case with isolated RV systolic dysfunction caused by a preexisting condition. Interobserver Agreement Stratified by Title We compared responses on 31 cardiac ultrasound video packets (did not include the duplicate videos). Comparing All 4 Observers Agreement among all 4 raters was statistically significant for all RV dysfunction variables. Agreement was substantial for both severe RV enlargement and systolic dysfunction and moderate for septal deviation (Table 1). Comparing Emergency Medicine Residents 1 and 2 For severe RV enlargement, resident 1 identified the disease in 13 cases, and resident 2 identified the disease in 12 cases; there was agreement on 11 of these cases. For severe RV systolic dysfunction, resident 1 identified the disease in 8 cases, and resident 2 identified the disease in 9 cases; there was agreement on 7 of these cases. For deviation of the ventricular septum, resident 1 identified the disease in 9 cases, and resident 2 identified the disease in 10 cases; there was agreement on 8 of these cases. The agreement between the third-year emergency medicine residents was considered substantial for RV enlargement, RV systolic dysfunction, and septal deviation (Table 2). Comparing Cardiologists A and B For severe RV enlargement, both cardiologists identified this disease in 14 cases. The cardiologists agreed on the Table 1. Comparison of Emergency Medicine Residents 1 and 2 and Cardiologists A and B Agreement Disease (All 4 Agreed), % κ SE P Severe RV enlargement <.001 Severe RV systolic dysfunction <.001 Deviation of ventricular septum <.001 Table 2. Interobserver Agreement for RV Dysfunction by 2 Emergency Medicine Residents Disease Agreement, % κ SE P Severe RV enlargement <.001 Severe RV systolic dysfunction <.001 Deviation of ventricular septum <.001 CI indicates confidence interval J Ultrasound Med 2016; 35:

5 presence of the disease in 13 of the cases and did not agree on 1 case. For severe RV systolic dysfunction: cardiologist A identified the disease in 13 cases, and cardiologist B identified the disease in 14 cases; there was agreement on 12 cases. For deviation of the ventricular septum, cardiologist A identified the disease in 10 cases, and cardiologist B identified the disease in 5 cases; of those 5 cases, there was agreement. The agreement between cardiologists was considered near perfect for RV enlargement and systolic dysfunction and moderate for septal deviation (Table 3). Intraobserver Agreement Nine goal-directed echocardiographic videos were displayed and assessed twice by each participant. There was 100% agreement for each participant for each disease; their responses were consistent both times they assessed the videos. Each participant s κ value was 1.0 (95% confidence interval, ; P =.013). Of the 9 duplicates, there were 6 without RV dysfunction and 3 with RV dysfunction present. Two of the RV dysfunction positive patients had all 3 RV dysfunction abnormalities. The third patient had RV enlargement and septal deviation but no RV systolic dysfunction. Accuracy of RV Dysfunction Detection The details of observer assessments on the cases with RV dysfunction are detailed in Table 4. Errors There was 1 case in which all 4 participants interpreted goal-directed echocardiography as showing RV enlargement, but RV size and function were normal on goaldirected echocardiography. In this case, the RV apex was not blunted and did not extend to the apex in the subcostal and apical views. There were 2 cases each in which there was interpretation of septal deviation by a cardiologist when there was none and RV systolic function by a cardiologist when there was none. Overall, cardiologist A had 4 errors in RV dysfunction features in 4 goal-directed echocardiographic examinations. Cardiologist B had 4 separate errors within 3 examinations. Resident 1 had 2 errors, and resident 2 had 1 error. Discussion The fundamentals of severe RV dysfunction detection using goal-directed echocardiography alone are reviewed in this report. Performance on the observer agreement testing in RV dysfunction prognostication with echocardiography was similar between emergency medicine residents and cardiologists using qualitative RV dysfunction criteria on goal-directed echocardiography. Our study evaluated RV dysfunction by making binary visual assessments in 3 categories: RV enlargement, RV systolic dysfunction using tricuspid annular plane systolic excursion and freewall hypokinesis, and septal deviation. Table 3. Interobserver Agreement for RV Dysfunction by 2 Cardiologists Disease Agreement, % κ SE P Severe RV enlargement <.001 Severe RV systolic dysfunction <.001 Deviation of ventricular septum CI indicates confidence interval. Table 4. Comparison of the Accuracy of the Participants for Detection of RV Dysfunction Variables Confirmed RV Result Dysfunction Cases Cardiologist A Cardiologis B Resident 1 Resident 2 Identified at least 1 of the abnormalities All 3 RV dysfunction abnormalities RV enlargement and septal deviation but no RV dysfunction RV enlargement only Only RV systolic dysfunction (no RV enlargement or septal deviation) J Ultrasound Med 2016; 35:

6 We found that emergency physicians had substantial interobserver agreement on RV dysfunction using our criteria. The cardiologists also had near-perfect agreement on both severe RV enlargement and severe RV systolic dysfunction, with moderate agreement on septal deviation. All 4 participants had substantial agreement on RV enlargement and RV systolic dysfunction and moderate agreement on septal deviation. Intraobserver agreement was perfect for each participant. Our findings were consistent with previous literature that compared interobserver agreement of signs of RV dysfunction between cardiologists only. In a study by Kline et al, 3 2 cardiologists interpreted the echocardiograms of 200 patients with normotensive PE. At the time of diagnosis, the interobserver agreement was excellent for RV dilatation (κ = 0.96), moderate for the presence of hypokinesis (κ = 0.76), and less good for interventricular septum flattening or paradoxical movement (κ = 0.51). Kopecna et al 14 also evaluated the interobserver reliability for measurements of RV dysfunction in 73 patients with normotensive PE. Two cardiologists performed and interpreted echocardiographic examinations within 24 hours of diagnosis and compared their measurements of the RV/LV diameter ratio (cutoff of 1:1), hypokinesis of the RV free wall, and M-mode tricuspid annular plane systolic excursion measurements (cutoff of <16 mm) against a central panel of 3 cardiologists. The interobserver agreements were as follows: for the RV/LV diameter ratio, there was a weighted κ of 0.65 (good); for hypokinesis of the RV free wall, there was a weighted κ of 0.70 (good); and for tricuspid annular plane systolic excursion, there was a weighted κ of 0.86 (very good). Several studies in the cardiology literature have demonstrated low intraobserver and interobserver variability in tricuspid annular plane systolic excursion measurement Emergency physicians, however, rarely make this measurement. Although tricuspid annular plane systolic excursion is typically measured by M-mode imaging, a study of 329 pediatric patients by Qureshi et al 25 found a correlation coefficient between M-mode and 2D tricuspid annular plane systolic excursion. In a smaller study of 30 patients, Kaul et al 26 compared the RV ejection fraction obtained by radionuclide angiography with 2D tricuspid annular plane systolic excursion and found a close correlation (r = 0.92) and close interobserver reproducibility of the tricuspid annular plane systolic excursion measurement (r = 0.93). A unique part of our study was our emphasis on an eyeball assessment of 2D tricuspid annular plane systolic excursion for evaluation of RV dysfunction, as movement of the tricuspid valve toward the apex during systole is affected by movement of both the interventricular septum and the RV free wall. In previous emergency medicine literature, more attention has been given to RV dilatation than to other markers of RV dysfunction in the identification of RV strain. In a study by Dresden et al, 7 in which 146 patients received bedside echocardiography performed by emergency physicians, limited echocardiography had high specificity (98%) for PE when RV dilatation was identified (RV/LV ratio 1:1) in patients with a Wells score of 2 or higher. This study also noted 96% observed agreement on all images reviewed between the principal investigator and 3 other emergency physicians for RV dilatation. 7 A retrospective cohort study by Taylor and Moore 9 identified 411 patients who had undergone goal-directed echocardiography performed by 69 emergency providers (residents and attending physicians) followed by comprehensive echocardiography for indications of chest pain, dyspnea, and hypotension. Right ventricular dilatation was defined as an RV/LV ratio of 1 or higher. The study found moderate agreement overall (κ = 0.44) for RV dilatation; in a subgroup analysis, junior residents showed a κ of 0.41 versus a κ of 0.54 for senior residents versus a κ of 0.55 for fellowship-trained providers. 9 Our study may have demonstrated more agreement between raters overall because we did not have a time delay between initial goal-directed echocardiography and comprehensive echocardiography during which RV function might have changed, since interpretation was of goal-directed echocardiographic images only. In addition, we used slightly different RV dilatation criteria, requiring blunting of the apex of the RV to be called severe RV dysfunction. In our study, septal deviation was the least helpful of the RV dysfunction criteria with the least agreement among our participants, although agreement was considered moderate. In contrast, the finding of RV enlargement is a requirement for PE-specific RV dysfunction. Overall, there was substantial to near-perfect agreement for the absence and presence of this fundamental PE-related dysfunction sign. There was also substantial agreement for the presence or absence of RV systolic dysfunction when combined with RV enlargement. The combination of RV enlargement and RV systolic dysfunction is indicative of a more severe stage of PE-related RV dysfunction. Limitations Both the acquisition and interpretation of images at the bedside by the treating physician are important components of goal-directed echocardiography. This study design did not address image acquisition but, rather, the inter J Ultrasound Med 2016; 35:

7 pretation of previously acquired goal-directed echocardiographic images. All of the images were and acquired by emergency physicians and derived from normotensive patients with acute PE. Images obtained are operator dependent. Interpretation of RV images and appearance is dependent on the transducer rotation and angulation, which may lead to suboptimal RV representation and potentially lead to image interpretation errors. For example, RV overestimation may occur if the apical view is foreshortened. Underestimation may occur if the intended apical 4-chamber window is overly rotated toward an apical 2-chamber view. Finally, an oblique or deviated appearance of the LV can occur if there is extra rotation (beyond 90 ) from the parasternal longaxis to the parasternal short-axis views. This study reports on the agreement between physicians in different medical specialties. Agreement does not always equate with accuracy, as observers can agree, yet both can be incorrect in their assessments. However, in our study, the goal-directed echocardiographic clips used were part of a previous research project in which goal-directed echo cardiography was highly sensitive and specific for RV dysfunction when compared with comprehensive echocardiography performed, on average, within 9 hours of each other. Our method of testing intraobserver agreement involved assessments of 9 randomly arranged video packets that were repeated after the first 31 packets. It is plausible that this process, and the lack of a substantial time lag between repeated assessments, may have introduced a recall bias; however, overall, there were 120 video clips reviewed, and none of the repeated studies included any unique distinguishing features, such as circulating thrombi, pacemaker wires, or pericardial effusions. This study was done at an institution with an established emergency ultrasound program. Results may not be generalizable to all institutions. Conclusions In summary, we studied observer agreement using qualitative RV dysfunction prognostication criteria on goaldirected echocardiography. There was substantial agreement among the 4 participants for RV enlargement and RV systolic dysfunction according to the goal-directed echocardiographic RV assessment guidelines. The agreement of emergency medicine residents on qualitative RV dysfunction assessments was substantial for RV enlargement, RV systolic dysfunction, and septal deviation. The agreement between board-certified cardiologists was almost perfect for RV enlargement and systolic dysfunction and moderate for septal deviation. The use of these specific RV dysfunction guidelines for goal-directed echocardiography may be of benefit as an early and time-sensitive risk stratification tool to accurately and qualitatively detect the presence and absence of substantial RV dysfunction and assist in dichotomizing normotensive patients with PE into intermediate and low-risk categories. References 1. Grifoni S, Olivotto I, Cecchini P, et al. Short-term clinical outcome of patients with acute pulmonary embolism, normal blood pressure, and echocardiographic right ventricular dysfunction. Circulation 2000; 101: Jaff MR, McMurtry MS, Archer SL, et al. Management of massive and submassive pulmonary embolism, iliofemoral deep vein thrombosis, and chronic thromboembolic pulmonary hypertension: a scientific statement from the American Heart Association. Circulation2011; 123: Kline JA, Steuerwald MT, Marchick MR, Hernandez-Nino J, Rose GA. Prospective evaluation of right ventricular function and functional status 6 months after acute submassive pulmonary embolism: frequency of persistent or subsequent elevation in estimated pulmonary artery pressure. Chest 2009; 136: Konstantinides SV ESC guidelines on the diagnosis and management of acute pulmonary embolism. Eur Heart J 2014; 35: Labovitz AJ, Noble VE, Bierig M, et al. Focused cardiac ultrasound in the emergent setting: a consensus statement of the American Society of Echocardiography and American College of Emergency Physicians. J Am Soc Echocardiogr 2010; 23: American College of Emergency Physicians. Emergency ultrasound guidelines American College of Emergency Physicians website. Accessed August 15, Dresden S, Mitchell P, Rahimi L, et al. Right ventricular dilatation on bedside echocardiography performed by emergency physicians aids in the diagnosis of pulmonary embolism. Ann Emerg Med 2014; 63: Jackson RE, Rudoni RR, Hauser AM, Pascual RG, Hussey ME. Prospective evaluation of two-dimensional transthoracic echocardiography in emergency department patients with suspected pulmonary embolism. Acad Emerg Med 2000; 7: Taylor RA, Moore CL. Accuracy of emergency physician performed limited echocardiography for right ventricular strain. Am J Emerg Med 2014; 32: Cho JH, Kutti Sridharan G, Kim SH, et al. Right ventricular dysfunction as an echocardiographic prognostic factor in hemodynamically stable patients with acute pulmonary embolism: a meta-analysis. BMC Cardiovasc Disord 2014; 14: Coutance G, Cauderlier E, Ehtisham J, Hamon M, Hamon M. The prognostic value of markers of right ventricular dysfunction in pulmonary embolism: a meta-analysis. Crit Care 2011; 15:R103. J Ultrasound Med 2016; 35:

8 12. Pollack CV, Schreiber D, Goldhaber SZ, et al. Clinical characteristics, management, and outcomes of patients diagnosed with acute pulmonary embolism in the emergency department: initial report of EMPEROR (Multicenter Emergency Medicine Pulmonary Embolism in the Real World Registry). J Am Coll Cardiol 2011; 57: Sanchez O, Trinquart L, Colombet I, et al. Prognostic value of right ventricular dysfunction in patients with haemodynamically stable pulmonary embolism: a systematic review. Eur Heart J 2008; 29: Kopecna D, Briongos S, Castillo H, et al. Interobserver reliability of echocardiography for prognostication of normotensive patients with pulmonary embolism. Cardiovasc Ultrasound 2014; 12: Bruhl SR, Chahal M, Khouri SJ. A novel approach to standard techniques in the assessment and quantification of the interventricular systolic relationship. Cardiovasc Ultrasound 2011; 9: Horton KD, Meece RW, Hill JC. Assessment of the right ventricle by echocardiography: a primer for cardiac sonographers. J Am Soc Echocardiogr 2009; 22: Rudski LG, Lai WW, Afilalo J, et al. Guidelines for the echocardiographic assessment of the right heart in adults: a report from the American Society of Echocardiography endorsed by the European Association of Echocardiography, a registered branch of the European Society of Cardiology, and the Canadian Society of Echocardiography. J Am Soc Echocardiogr 2010; 23: Lang RM, Badano LP, Mor-Avi V, et al. Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging. Eur Heart J Cardiovasc Imaging 2015; 16: Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005; 85: Chen B, Zaebst D, Seel L. Paper : a macro to calculate kappa statistics for categorizations by multiple raters. In: SUGI 30 Proceedings. Cary, NC: SAS Inc; 2005: Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: López-Candales A, Lopez FR, Trivedi S, Elwing J. Right ventricular ejection efficiency: a new echocardiographic measure of mechanical performance in chronic pulmonary hypertension. Echocardiography 2014; 31: Pinedo M, Villacorta E, Tapia C, et al. Inter- and intra-observer variability in the echocardiographic evaluation of right ventricular function. Rev Esp Cardiol 2010; 63: Tamborini G, Pepi M, Galli CA, et al. Feasibility and accuracy of a routine echocardiographic assessment of right ventricular function. Int J Cardiol 2007; 115: Qureshi MY, Eidem BW, Reece CL, O Leary PW. Two-dimensional measurement of tricuspid annular plane systolic excursion in children: can it substitute for an M-mode assessment? Echocardiography 2015; 32: Kaul S, Tei C, Hopkins JM, Shah PM. Assessment of right ventricular function using two-dimensional echocardiography. Am Heart J 1984; 107: J Ultrasound Med 2016; 35: