Over the recent decades, breast ultrasonography (US) has

Transcription

1 ORIGINAL RESEARCH Application of Computer-Aided Diagnosis on Breast Ultrasonography Evaluation of Diagnostic Performances and Agreement of Radiologists According to Different Levels of Experience Eun Cho, MD, Eun-Kyung Kim, MD, PhD, Mi Kyung Song, MS, Jung Hyun Yoon, MD, PhD Received November 22, 2016, from the Department of Radiology, Severance Hospital, Research Institute of Radiological Science (E.C., E.-K.K., J.H.Y.), and Department of Research Affairs, Biostatistics Collaboration Unit (M.K.S.), Yonsei University College of Medicine, Seoul, Korea. Manuscript accepted for publication April 10, This study was supported by the research fund of Samsung Medison Co, Ltd. Address correspondence to Jung Hyun Yoon, MD, PhD, Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University College of Medicine, 50-1 Yonse-ro, Seodaemun-gu, Seoul , Korea. Abbreviations AUC, area under the receiver operating characteristic curve; BI-RADS, Breast Imaging Reporting and Data System; CAD, computer-aided diagnosis; NPV, negative predictive value; PPV, positive predictive value; US, ultrasonography doi: /jum Objectives To investigate the feasibility of a computer-aided diagnosis (CAD) system (S-Detect; Samsung Medison, Co, Ltd, Seoul, Korea) for breast ultrasonography (US), according to radiologists with various degrees of experience in breast imaging. Methods From December 2015 to March 2016, 119 breast masses in 116 women were included. Ultrasonographic images of the breast masses were retrospectively reviewed and analyzed by 2 radiologists specializing in breast imaging (7 and 1 years of experience, respectively) and S-Detect, according to the individual ultrasonographic descriptors from the fifth edition of the American College of Radiology Breast Imaging Reporting and Data System and final assessment categories. Diagnostic performance and the interobserver agreement among the radiologists and S- Detect was calculated and compared. Results Among the 119 breast masses, 54 (45.4%) were malignant, and 65 (54.6%) were benign. Compared to the radiologists, S-Detect had higher specificity (90.8% compared to 49.2% and 55.4%) and positive predictive value (PPV; 86.7% compared to 60.7% and 63.8%) (all P <.001). Both radiologists had significantly improved specificity, PPV, and accuracy when using S-Detect compared to US alone (all P <.001). The area under the receiving operating characteristic curves of the both radiologists did not show a significant improvement when applying S-Detect compared to US alone (all P >.05). Moderate agreement was seen in final assessments made by each radiologist and S-Detect (j and 0.45, respectively). Conclusions S-Detect is a clinically feasible diagnostic tool that can be used to improve the specificity, PPV, and accuracy of breast US, with a moderate degree of agreement in final assessments, regardless of the experience of the radiologist. Key Words breast; Breast Imaging Reporting and Data System; computeraided diagnosis; neoplasm; ultrasonography; ultrasound equipment and products Over the recent decades, breast ultrasonography (US) has been widely available in everyday practice. Based on that trend, the American College of Radiology Breast Imaging Reporting and Data System (BI-RADS) was released and recently updated. 1 The ultrasonographic BI-RADS provides a sonographic lexicon and final assessment categories, which have excellent diagnostic performances when applied to breast US in the differential diagnosis of breast masses and suggest standardized management for patients. 2,3 In spite of the excellent performances reported by using the ultrasonographic BI-RADS, the final assessments made for breast VC 2017 by the American Institute of Ultrasound in Medicine J Ultrasound Med 2018; 37:

2 masses by different performers are known to vary significantly, 4,5 mostly because of the multiple BI-RADS ultrasonographic descriptors used for describing breast lesions and the subjectiveness of US. To increase the diagnostic accuracy of breast US, several additional ultrasonographic techniques have been developed and applied in clinical practice, such as elastography, 6 automated breast US, 7,8 and computer-aided diagnosis (CAD) systems Among these additional imaging modalities, CAD systems have been reported to enable efficient interpretation, in which consistent improved accuracy can be expected. 14,15 S-Detect (Samsung Medison, Co, Ltd, Seoul, Korea), a recently developed commercially available imaging analysis program for breast US, analyzes the morphologic features of a target breast mass according to the BI-RADS ultrasonographic descriptors and provides a final assessment, which is based on analysis of the ultrasonographic features. 16 This program may be useful in clinical practice, since it can be used to provide a second opinion on tumor characterization and guidance in deciding on patient treatment. In a recent study analyzing the diagnostic performances of S-Detect and a dedicated breast radiologist, 16 S-Detect had equivalent diagnostic performance and moderate agreement with the radiologist. The usefulness of CAD for breast US can differ according to the level of experience of the radiologist, as reported in a recent study, 17 but the previous study used in-house software that is not commercially available, which had its limitations in that the analytic results could differ according to different algorithms or models used for analysis. Therefore, the purpose of this study was to investigate the clinical feasibility of S-Detect when applied to breast US by comparing the diagnostic performances and agreement between S-Detect and radiologists with various degrees of experience in breast imaging. Materials and Methods This study was a prospective study, and it was approved by the Institutional Review Board of Severance Hospital. Informed consent was obtained from all patients for study inclusion. Patients From December 2015 to March 2016, 126 breast masses in 123 consecutive women who were scheduled for breast US-guided core needle biopsy, surgical excision, or a combination thereof were included in this study. Among them, 7 women were excluded for nonmasslesionsseenonbreastus(n5 4), since BI-RADS descriptors are hard to apply on these lesions, and lack of grayscale images of the targeted breast mass stored in the US machine for S-Detect analysis (n 5 3). Only elastogramsanddopplerusimageswerestoredforthe3 cases with images not fit for the S-Detect analysis. Finally, 119 breast masses in 116 women were included in this study. Ultrasonographic Examinations Ultrasonographic examinations were performed with a 3 12-MHz linear transducer (RS80A with Prestige; Samsung Medison, Co, Ltd). Two staff radiologists (J.H.Y. and E.-K.K.) with 7 and 19 years of experience in breast imaging, respectively, were involved in image acquisition. Bilateral whole-breast ultrasonographic examinations were routinely performed, according to scanning protocols including representative transverse and longitudinal images of the breast masses with and without calipers used for size measurements. The same radiologist who had performed breast US proceeded with US-guided core needle biopsy. Both radiologists had access to the mammographic images obtained before US, images of ultrasonographic examinations performed in the past, and medical records containing clinical information of the patients. Image Review and Application of S-Detect Ultrasonographic images of the 119 breast masses were retrospectively reviewed and documented for data analysis by 2 breast radiologists (J.H.Y., radiologist 1; and E.C., radiologist 2) with 7 and 1 years of breast imaging experience, respectively. All observers were blinded to the clinical information and pathologic results of each mass during image review. Breast masses were analyzed according to the descriptors and final assessment categories used in the fifth edition of the BI-RADS and final assessments (Table 1). 1 Calcifications were not analyzed because of the limited data analytic ability of S-Detect for calcifications. 16 The radiologists individually chose the most appropriate term to describe each lesion for each descriptor and made a final assessment accordingly. The ultrasonographic BI-RADS final assessment categories were made as follows: category 2 (benign), 3 (probably benign), 4a (low suspicion of malignancy), 4b (moderate suspicion of malignancy), 4c (high suspicion of malignancy), and 5 (highly suggestive of malignancy). 210 J Ultrasound Med 2018; 37:

3 After image review by the radiologists, S-Detect was applied to the same image the radiologists used for grayscale ultrasonographic feature analysis. A region of interest was either automatically or manually drawn along the border of the mass by S-Detect. Analytic results of S- Detect, including the BI-RADS lesion descriptors and final assessments, were immediately displayed and recorded for data analysis (Figure 1). After being informed of the final assessment made by S-Detect, each radiologist gave a final assessment for each breast mass, integrating the analytic results of S-Detect. Table 1. Ultrasonographic Descriptors Used for Image Analysis Characteristic Descriptors Shape Orientation Margin Echo pattern Posterior acoustic features Oval, round, irregular Parallel, nonparallel Circumscribed, not circumscribed: indistinct, angular, microlobulated, speculated Anechoic, hyperechoic, complex cystic and solid, hypoechoic, isoechoic, heterogeneous No posterior features, enhancement, shadowing, combined Data and Statistical Analyses Histopathologic results from US-guided core needle biopsy, vacuum-assisted excision, or surgery were regarded as the reference standards. Patients with a diagnosis of high-risk lesions, such as atypical ductal hyperplasia, intraductal papilloma, radial scar, and lobular carcinoma in situ, were recommended to have further surgical treatment, for which the final pathologic diagnosis was considered. Final histopathological diagnoses of high-risk lesions were considered benign for the statistical analysis. Because the final assessment data from S-Detect were in a dichotomized form, possibly benign and possibly malignant, final assessments from the radiologists based on the BI-RADS were also divided into 2 groups for statistical analysis: positive assessments consisted of categories 4a to 5, and negative assessments consisted of categories 2 and 3. Diagnostic performances of the individual radiologists, S-Detect, and the integration of S-Detect with each radiologist were analyzed and compared, including sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy. The generalized estimating equation method was used for comparing the diagnostic performances between radiologists and S-Detect. The area under the receiver operating characteristic curve (AUC) was Figure 1. Representative image of setting the region of interest for S-Detect analysis in a 52-year-old woman with a diagnosis of cancer in her left breast. The region of interest was set automatically along the margin of the breast mass for analysis (green line). After the region of interest was set, the ultrasonographic features were automatically analyzed by S-Detect, and a final assessment was automatically visualized. J Ultrasound Med 2018; 37:

4 obtained and compared by the Delong method. j statistics were calculated to assess the agreement for ultrasonographic descriptors and final assessments among the radiologists, S-Detect, and the integration of each radiologist and S-Detect. Estimation of the overall j was based on a study by Landis and Koch 18 :lessthan0indicated poor agreement; 0.00 to 0.20 indicated slight agreement; 0.21 to 0.40 indicated fair agreement; 0.41 to 0.60 indicated moderate agreement; 0.61 to 0.80 indicated substantial agreement; and 0.81 to 1.00 indicated almost perfect agreement. The weighted least squares approach of Barnhart and Williamson 19 wasusedtocomparej coefficients. All statistical analyses in this study were performed with SAS version 9.2 software (SAS Inc, Cary, NC), P <.05 was considered to have statistical significance. Results Of the 119 breast masses included in this study, 65 (54.6%) lesions were benign, and 54 (45.4%) lesions were malignant. The mean age 6 SD of the 116 women included in this study was years (range, years). The mean size of the breast masses was mm (range, 4 60 mm). Among the 119 breast masses, 39 (33.3%) lesions were diagnosed by surgery, 8 (6.7%) by US-guided vacuum-assisted excision, and 103 (86.6%) by US-guided core needle biopsy. Fourteen (11.8%) lesions were diagnosed as benign according to ultrasonographic features without additional biopsy, among which 8 were considered benign, since they showed benign ultrasonographic features along with stability during imaging follow-up for greater than 2 years, 1 being a cyst, and 5 proven as galactoceles on aspiration. The mean size of the malignant masses was significantly larger than that of the benign masses: 17.0 mm (range, mm) and 11.0 mm (range, mm),respectively(P <.001). Diagnostic Performances of the Radiologists Versus S-Detect The distribution of breast masses with histopathologic diagnoses analyzed by the radiologists and S-Detect is summarized in Table 2. There were no significant differences in S-Detect results between benign and malignant masses for both radiologists (P >.05). Table 3 summarizes the diagnostic performances of the radiologists and S-Detect. The sensitivity and NPV were significantly higher for both radiologists compared to S-Detect, whereas the specificity, PPV, and accuracy were higher for S-Detect (all P <.05). The AUCs of the radiologists were significantly higher than that of S- Detect (0.887 and compared to 0.815, respectively; P and.004). When the results of S-Detect Table 2. Distribution of Breast Masses Analyzed by the Radiologists and S-Detect Radiologist 1 Radiologist 2 Final Assessment S-Detect Benign Malignant Total P a Benign Malignant Total P a 2 Total 4 (100.0) 0 (0.0) 4 NA 0 (0.0) 0 (0.0) 0 NA Probably benign 4 (100.0) 0 (0.0) 4 0 (0.0) 0 (0.0) 0 Possibly malignant 0 (0.0) 0 (0.0) 0 0 (0. 0) 0 (0.0) 0 3 Total 28 (100.0) 3 (100.0) 31 NA 36 (100.0) 3 (100.0) 39 NA Probably benign 28 (100.0) 3 (100.0) (100.0) 3 (100.0) 39 Possibly malignant 0 (0.0) 0 (0.0) 0 0 (0.0) 0 (0.0) 0 4a Total 26 (100.0) 7 (100.0) (100.0) 9 (100.0) Probably benign 24 (92.3) 4 (57.1) (91.3) 6 (66.7) 27 Possibly malignant 2 (7.7) 3 (42.9) 5 2 (8.7) 3 (33.3) 5 4b Total 1 (100.0) 0 (0.0) 1 NA 4 (100.0) 8 (100.0) Probably benign 1 (100.0) 0 (0.0) 1 2 (50.0) 2 (25.0) 4 Possibly malignant 0 (0.0) 0 (0.0) 0 2 (50.0) 6 (75.0) 8 4c Total 4 (100.0) 24 (100.0) (100.0) 16 (100.0) 18 >.999 Probably benign 2 (50.0) 7 (29.2) 9 0 (0.0) 3 (18.7) 3 Possibly malignant 2 (50.0) 17 (70.8) 19 2 (100.0) 13 (81.3) 15 5 Total 2 (100.0) 20 (100.0) 22 > (0.0) 18 (100.0) 18 NA Probably benign 0 (0.0) 1 (5.0) 1 0 (0.0) 1 (5.6) 1 Possibly malignant 2 (100.0) 19 (95.0) 21 0 (0.0) 17 (94.4) 17 Data are presented as number (percent) where applicable. NA indicates not applicable. a Comparison between each radiologist and S-Detect in each BI-RADS category analyzed by the radiologist. 212 J Ultrasound Med 2018; 37:

5 were integrated, the specificity, PPV, and accuracy showed significant improvement compared to the performances of the individual radiologists alone (all P <.05). The sensitivity of radiologist 2 integrated with the result of S-Detect was lower than that of radiologist 2 alone(p <.05). There was no significant difference between the sensitivity of radiologist 1 integrated with the result of S-Detect and that of radiologist 1 alone (P 5.170). The AUCs of the radiologists integrated with S-Detect did not show significant differences when compared to the AUCs of the radiologists alone (0.895 compared to and compared to 0.901; all P >.05; Figure 2). Figure 2. Receiver operating characteristic curves for the radiologists, S-Detect, and integration of the radiologists and S-Detect. Agreement for Ultrasonographic Descriptors and Final Assessment Between the Radiologists and S- Detect Agreement between the radiologists and S-Detect for ultrasonographic descriptors and final assessments are summarized in Table 4. Between the radiologists, agreements were substantial for shape (j ), orientation (j ), and posterior acoustic features (j ) and fair for margin (j ) and echo pattern (j ). Moderate agreement (j ) was seen for final assessments between the radiologists. Agreement between radiologist 1 and S-Detect for the ultrasonographic descriptors ranged from fair to moderate (j ). Agreement between radiologist 2 and Table 3. Diagnostic Performances of the Radiologists With and Without Integration of S-Detect Diagnostic Radiologist Radiologist Performance Radiologist 1 Radiologist 2 S-Detect P a P b 1 1 S-Detect P c 2 1 S-Detect P d Sensitivity, % 94.4 ( ) Specificity, % 49.2 ( ) PPV, % 60.7 ( ) NPV, % 91.4 ( ) Accuracy, % 69.8 ( ) AUC ( ) 94.4 ( ) 55.4 ( ) 63.8 ( ) 92.3 ( ) 73.1 ( ) ( ) 72.2 ( ) 90.8 ( ) 86.7 ( ) 79.7 ( ) 82.4 ( ) ( ) <.001 < ( ) <.001 < ( ) <.001 < ( ) ( ) ( ) ( ) ( ) < ( ) < ( ) > ( ) < ( ) > ( ).040 <.001 < >.999 Values in parentheses are 95% confidence intervals. a Comparison between radiologist 1 and S-Detect. b Comparison between radiologist 2 and S-Detect. c Comparison between radiologist 1 and radiologist 1 1 S-Detect. d Comparison between radiologist 2 and radiologist 2 1 S-Detect. J Ultrasound Med 2018; 37:

6 S-Detect ranged from fair to substantial (j ). When the BI-RADS final assessments of the radiologists were dichotomously divided into benign (categories 2 and 3) and malignant (categories 4a 5), fair and moderate agreements were seen between the radiologists and S-Detect (j and 0.45, respectively). Interobserver agreements for the shape, orientation, and posterior features between the radiologists were significantly higher than those between the radiologists and S-Detect (P <.05). For the final assessment categories, there were no significant differences in the agreement between the radiologists versus that between the individual radiologists and S-Detect (P >.05). Discussion The ultrasonographic BI-RADS lexicon is widely used for breast lesion descriptions, but because of the subjective tendency of US, observer variability is inevitable, and it can lead to inconsistent diagnoses among performers. 2,5 Computer-aided diagnosis systems were recently applied to overcome the observer variability of breast US, 20 as well as to improve the diagnostic performances. 14,15 S-Detect applies a novel feature extraction technique and support vector machine classifier that categorizes breast masses into benign or malignant according to the suggested feature combinations integrated according to the contents of BI-RADS ultrasonographic descriptors. 21 In a recent study, Kim et al 16 reported that S-Detect had significantly higher specificity, PPV, accuracy, and AUC compared to a dedicated breast radiologist, with fair to substantial agreement in ultrasonographic feature analysis for breast masses. However, in clinical practice, radiologists with different levels of experience perform breast US, and the usefulness of S-Detect may be different according to the level of experience: for example, radiologists with less experience may benefit more by using S-Detect. In our study, S-Detect showed significantly higher specificity, PPV, and accuracy compared to the radiologists (all P <.001). When integrating the results of S- Detect, it led to significant improvements in specificity, PPV, and accuracy in both radiologists, similar to the results of previous studies for CAD systems. In addition, 24 of 26 (92.3%) and 21 of 23 (91.3%) benign breast masses initially assessed as category 4a by the radiologists were categorized as probably benign by S-Detect (Table 2). Based on our results, S-Detect could be used as an additional tool with breast US regardless of the level of experience the radiologist has and may be used to reduce the number of unnecessary biopsies of benign breast masses. Although the specificity, PPV, and accuracy were improved, the AUCs of the radiologists integrated with S- Detect had no significant differences compared to those of the radiologists alone (all P >.05). Both of the radiologists already had very high AUC values; therefore, little was left to improve, which may have been the cause for no differences in AUCs after the integration of S-Detect. Several reports have been published on applying different types of CAD to breast US. 16,17,22 These studies commonly reported that the CAD systems enable improvement in diagnostic performances of breast US, especially specificity and accuracy. Shen et al 22 suggested that computer-aided classification systems could be helpful in assessing indeterminate category 4 cases. Wang et al 17 concluded that the inclusion of CAD was more helpful for junior radiologists than the seniors, with greater improvement in the diagnostic performances in Table 4. Interobserver Agreement for BI-RADS Sonographic Descriptors and Final Assessment Categories Among the Radiologists and S-Detect BI-RADS Lexicon j, Radiologist 1 vs Radiologist 2 j, Radiologist 1 vs S-Detect j, Radiologist 2 Vs S-Detect P a P b P c Shape 0.72 ( ) 0.44 ( ) 0.44 ( ) <.001 < Orientation 0.72 ( ) 0.55 ( ) 0.68 ( ) Margin 0.35 ( ) 0.28 ( ) 0.31 ( ) Echo pattern 0.33 ( ) 0.23 ( ) 0.35 ( ) Posterior acoustic features 0.76 ( ) 0.35 ( ) 0.46 ( ) < Final assessment 0.57 ( ) 0.40 ( ) 0.45 ( ) Values in parentheses are 95% confidence intervals. a Comparison between radiologist 1 versus radiologist 2 and radiologist 1 versus S-Detect. b Comparison between radiologist 1 versus radiologist 2 and radiologist 2 versus S-Detect. c Comparison between radiologist 1 versus S-Detect and radiologist 2 versus S-Detect. 214 J Ultrasound Med 2018; 37:

7 the junior group. In our study, the specificity, PPV, and accuracy of both radiologists were improved, supporting the idea that S-Detect can be relied on to provide a second opinion that can be used to decide the next step for patient treatment. In addition, S-Detect is a commercially available, easy-to-use program that enables immediate visualization of analytic results during real-time ultrasonographic examinations, which can easily be incorporated in daily practice. In our study, agreement for the BI-RADS ultrasonographic descriptors between the radiologists and S- Detect was fair to substantial. For final assessments, the agreement between the radiologists and the agreement between each radiologist and S-Detect was moderate, without significant differences. The level of agreement for final assessments between each radiologist and S- Detect were in a similar range as the agreement between the radiologists in this study (j ), and also the agreement among radiologists reported in the literature (j ). 2,5,23 In addition, as S-Detect provides the final assessment in a dichotomized form of possibly benign and possibly malignant, we consider this factor to have affected the low levels of agreement among the radiologists. Further investigation with technical improvements is anticipated for a more sophisticated algorithm using the multiple final-assessment BI- RADS ultrasonographic categories. There were a few limitations to our study. First, present study included a relatively small number of cases (n 5 119). Second, the presence of calcifications, which is a strong predictor for breast malignancy, 24,25 was not included in the analyses because of the limited analytic ability of S-Detect for microcalcifications. Similarly, nonmass lesions were not included in this study, which may have affected the results. Third, representative still images stored for analysis were used during image analysis by the radiologists and S-Detect. Variability also exists in selecting representative images of a breast mass. Fourth, 60 cases (50.4%) of breast masses included this study were diagnosed by core needle biopsy only. The false-negative rate of core needle biopsy was less than 2% in the literature. 26 Therefore, this factor may have had little effect on our results. In conclusion, S-Detect is a clinically feasible diagnostic tool that can be used to improve the specificity, PPV, and accuracy of breast US, with a moderate degree of agreement in final assessments, regardless of the experience of the radiologist. References 1. D Orsi CJ, Sickles EA, Mendelson EB, et al. ACR BI-RADS VR Atlas: Breast Imaging Reporting and Data System. 5th ed. Reston, VA: American College of Radiology; Lazarus E, Mainiero MB, Schepps B, Koelliker SL, Livingston LS. BI- RADS lexicon for US and mammography: interobserver variability and positive predictive value. Radiology 2006; 239: Yoon JH, Kim MJ, Moon HJ, Kwak JY, Kim EK. Subcategorization of ultrasonographic BI-RADS category 4: positive predictive value and clinical factors affecting it. Ultrasound Med Biol 2011; 37: Hong AS, Rosen EL, Soo MS, Baker JA. BI-RADS for sonography: positive and negative predictive values of sonographic features. AJR Am J Roentgenol 2005; 184: Lee HJ, Kim EK, Kim MJ, et al. Observer variability of Breast Imaging Reporting and Data System (BI-RADS) for breast ultrasound. Eur J Radiol 2008; 65: Au FW, Ghai S, Moshonov H, et al. Diagnostic performance of quantitative shear wave elastography in the evaluation of solid breast masses: determination of the most discriminatory parameter. AJR Am J Roentgenol 2014; 203:W328 W Kim EJ, Kim SH, Kang BJ, Kim YJ. Interobserver agreement on the interpretation of automated whole breast ultrasonography. Ultrasonography 2014; 33: Shin HJ, Kim HH, Cha JH, Park JH, Lee KE, Kim JH. Automated ultrasound of the breast for diagnosis: interobserver agreement on lesion detection and characterization. AJR Am J Roentgenol 2011; 197: Chang RF, Wu WJ, Moon WK, Chen DR. Automatic ultrasound segmentation and morphology based diagnosis of solid breast tumors. Breast Cancer Res Treat 2005; 89: Chen CM, Chou YH, Han KC, et al. Breast lesions on sonograms: computer-aided diagnosis with nearly setting-independent features and artificial neural networks. Radiology 2003; 226: Drukker K, Giger ML, Vyborny CJ, Mendelson EB. Computerized detection and classification of cancer on breast ultrasound. Acad Radiol 2004; 11: Kim KG, Cho SW, Min SJ, Kim JH, Min BG, Bae KT. Computerized scheme for assessing ultrasonographic features of breast masses. Acad Radiol 2005; 12: Sehgal CM, Cary TW, Kangas SA, et al. Computer-based margin analysis of breast sonography for differentiating malignant and benign masses. J Ultrasound Med 2004; 23: Horsch K, Giger ML, Vyborny CJ, Lan L, Mendelson EB, Hendrick RE. Classification of breast lesions with multimodality computeraided diagnosis: observer study results on an independent clinical data set. Radiology 2006; 240: J Ultrasound Med 2018; 37:

8 15. Horsch K, Giger ML, Vyborny CJ, Venta LA. Performance of computer-aided diagnosis in the interpretation of lesions on breast sonography. Acad Radiol 2004; 11: Kim K, Song MK, Kim EK, Yoon JH. Clinical application of S-Detect to breast masses on ultrasonography: a study evaluating the diagnostic performance and agreement with a dedicated breast radiologist. Ultrasonography 2017; 36: Wang Y, Jiang S, Wang H, et al. CAD algorithms for solid breast masses discrimination: evaluation of the accuracy and interobserver variability. Ultrasound Med Biol 2010; 36: Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: Barnhart HX, Williamson JM. Weighted least-squares approach for comparing correlated kappa. Biometrics 2002; 58: Singh S, Maxwell J, Baker JA, Nicholas JL, Lo JY. Computer-aided classification of breast masses: performance and interobserver variability of expert radiologists versus residents. Radiology 2011; 258: Lee JH, Seong YK, Chang CH, et al. Computer-aided lesion diagnosis in B-mode ultrasound by border irregularity and multiple sonographic features. In: SPIE Medical Imaging. Bellingham, WA: International Society for Optics and Photonics; 2013: Shen WC, Chang RF, Moon WK. Computer aided classification system for breast ultrasound based on Breast Imaging Reporting and Data System (BI-RADS). Ultrasound Med Biol 2007; 33: Park CS, Lee JH, Yim HW, et al. Observer agreement using the ACR Breast Imaging Reporting and Data System (BI-RADS) ultrasound, first edition (2003). Korean J Radiol 2007; 8: Sickles EA. Mammographic features of early breast cancer. AJR Am J Roentgenol 1984; 143: Sickles EA. Breast calcifications: mammographic evaluation. Radiology 1986; 160: Schueller G, Jaromi S, Ponhold L, et al. US-guided 14-gauge core-needle breast biopsy: results of a validation study in 1352 cases. Radiology 2008; 248: J Ultrasound Med 2018; 37: