Retrospective study comparison of Arcadia Lab s Galileo and Parascript AccuDetect Galileo CAD systems in screening environment July 2011 Parascript Page 1 7/8/2011
Investigators Dr. Rossano Girometti (Researcher), Prof. Chiara Zuiani (Associate Professor), Dr. Anna Linda (Staff Radiologist), Dr. Viviana Londero (Staff Radiologist), Dr. Anna Dal Col (Staff Radiologist), Dr. Silvia De Stefani (Resident in Radiology), Dr. Luisa Battigelli (Resident in Radiology), Eleonora Di Gaetano (MD, Resident in Radiology), Prof. Massimo Bazzocchi (Chief of the Department). All the Investigators work for the Istituto di Radiologia Diagnostica Azienda Ospedaliero- Universitaria S. Maria della Misericordia di Udine, Via Colugna n.50-33100 Udine. University of Udine and their department of radiology conducted the first retrospective clinical study of Parascript AccuDetect Galileo in November and December 2010. This was a no profit, no sponsorship study conducted at the Radiology Department of University of Udine, Italy. The Udine Hospital has a long experience with digital images processed by Full Field Digital Mammography. The first mammography equipment was purchased from IMS Giotto in 2003 (IMS Giotto 3DL) and another FFDM (IMS Giotto SD) was acquired by the hospital in 2008. IMS DL and IMS SD systems are each set up with Acquisition Work Station (AWS) and there are also 2 Review Work Stations so each image from both FFDMs can be viewed on both workstations. For processing (RAW) and for presentation images are stored in hospital s picture archiving system. Udine Hospital is doing both screening and diagnostic breast imaging, therefore occurrence of malignant cancers is about 10-12 cancer cases per 1,000 women. 20-22 mammogram cases are completed each day. Background Computer-Aided Detection (CAD) tools are software programs that analyze digitized or digital mammography images to find features that are associated with breast cancer. The purpose of CAD is to "mark" suspected findings on mammographic images, in order to help Radiologist in the detection of lesions [1]. CAD is more suited to detect than to characterize/interpret breast lesions [1]. In other words, CAD marks potential malignant lesions that Radiologist could have missed. CAD is an intensive matter for research in the scenario of mammographic screening programs. Therefore, screening is a cost-effective strategy for earlier detection, and thus treatment of breast cancer [1-2]. However, prevalence of disease is lower in the screening scenario (no symptomatic patients) as compared to the clinical setting (different patient categories, see below), leading to subtler and less obvious mammographic features, more difficult to identify [1,3]. The rate of falsenegative cases at screening (missed cancers) is up to 25%, 27-70% of lesions being visible in retrospect according to different experiences [2]. In order to overcome the problem of false-negatives, and to improve sensitivity in the detection of breast cancer, two methods have been adopted: (a) conventional double reading strategies, involving two radiologists; (b) a single reading by radiologist followed by the use of CAD, at lesser human cost [1,4-5]. Despite the large use of CAD, especially in Parascript Page 2 7/8/2011
the USA screening programs, analysis of these methods is still incomplete because no randomized controlled trials have been performed to assess changes in survival [6]. Since these studies are difficult to achieve, most experiences use surrogate end-points such as: (a) CAD sensitivity and the proportion of retrieved cancers (retrospective studies); (b) cancer detection rate and recall rate due to further investigations in patients with suspect findings (prospective studies) [7]. Ideally, a CADinduced increase in cancer detection should be balanced by a small, clinically acceptable increase in recall rate due to further investigations on false-positive cases [8]. Overall, available results make CAD a controversial matter. Some Authors state that CAD provides no significant increase in cancer detection rate in clinical practice, at the expense of excessive recall rates [7,9]. Most evidence, on the contrary, supports its use as a second reader in the screening scenario, because of an adequate intrinsic sensitivity for cancer, especially for microcalcifications, and increased readers' sensitivity up to 23.7% [10]. In any case, the reduction of specificity invariably associated with the CAD use may result in no overall clinical benefit [1]. CAD is still less sensitive to masses, especially in dense breasts. Moreover, readers who benefit from CAD are mainly inexperienced ones [1]. Better CAD performances have been obtained by using full-field digital mammography, which provides digital images, compared to screen-film mammography [11]. One emerging issue is [9] the occurrence of many false positive marks created by CAD systems during the screening process. The purpose of this study was to compare two different CAD systems and determine if false positive rates differ significantly from system to system. Study Design Two different CAD systems were used during the study. Both Computer Aided Detection (CAD) systems were designed for a computerized second read of mammography images in order to help radiologists confirm that a suspicious area requires further investigation. AccuDetect Galileo is an adaptation of Parascript CAD solution called AccuDetect for IMS Giotto platform. Galileo is manufactured by Arcadia Lab S.r.L and was specifically developed for IMS Giotto, a Bologna, Italy based manufacturer of FFDM systems. The study was performed in collaboration with the Department of Radiology, University of Udine and focused on standalone results for malignant calcification and mass detection. The cases for the study were collected from two IMS Giotto Full Field Digital Mammography (FFDM) units and consequently retrieved from Udine Hospital s picture archiving system. There were 118 cancer cases (86 masses, 6 calcifications, 26 mixed) and 209 negative (normal) cases. The negative cases had a 12-month follow up radiology report to mark them as normal. The truth data was obtained by resident, experienced radiologist who read each case and accompanying radiology and pathology reports. The radiologist draw contours of mass lesions and outlined an area of malignant micro calcifications on the screen of a color monitor. The senior radiologist then reviewed the contours and confirmed their validity. Parascript Page 3 7/8/2011
All retrospective normal and cancer cases were sent to Galileo and AccuDetect Galileo software and the following results were calculated: a) Percent of mass cancer cases correctly marked by Galileo, b) Percent of calcification cancer cases correctly marked by Galileo, c) Percent of mass cancer cases correctly marked by AccuDetect Galileo, d) Percent of calcification cancer cases correctly marked by AccuDetect Galileo, e) Percent of mass cancer images correctly marked by Galileo, f) Percent of calcification cancer images correctly marked by Galileo, g) Percent of mass cancer images correctly marked by AccuDetect Galileo, h) Percent of calcification cancer images correctly marked by AccuDetect Galileo. Figure 1. Parascript AccuDetect Galileo CAD - system Parascript Page 4 7/8/2011
Results During the comparison as depicted in Table 1, AccuDetect Galileo achieved higher sensitivity per case for all malignant lesions (p<0.012). Table 1: True Positive (Sensitivity) versus False Positive rate per case and per image for AccuDetect Galileo and Galileo CAD. Parascript Page 5 7/8/2011
Figure 1: FROC curve for AccuDetect Galileo (ADG) and performance data for Galileo at 0.8 False Positive per case. AccuDetect Galileo achieved higher sensitivity per image for all malignant lesions (p<0.0001). Table 2: Sensitivity and False Positive rates per image % 1 0.9 0.8 Masses T P p e r c a s e 0.7 0.6 0.5 0.4 0.3 0.2 0.1 ADG Galileo 0 0 0.5 1 1.5 FP Masses/Image Figure 2: AccuDetect Galileo (ADG) FROC curve for Masses when compared to Galileo Parascript Page 6 7/8/2011
AccuDetect Galileo achieved more than 60% lower false positive rate per image for malignant lesion hypothesis (p<0.001) when the operating point for AccuDetect Galileo is set to match the sensitivity of Galileo. Table 3: False positive rate for all cancers (Total FP) and separately for masses and calcifications. Figure 3: AccuDetect Galileo FROC curve for calcifications when compared to Galileo. The paired Student s T-test and Wilcoxon matched-pairs signed-rank test were used to assess the statistical significance of AccuDetect Galileo results improvement for malignant lesions detection. Parascript Page 7 7/8/2011
The paired Student s T-test was used to assess the statistical significance of false-positives reduction by 60% achieved by AccuDetect Galileo. The results of the tests are summarized in the table below: Table 4: The paired Student T- test and Wilcoxon test results. Conclusions During the study AccuDetect Galileo was compared against state of the art product Galileo CAD, which had shown better performance than widely used icad s Second Look Digital according to a 2007 study [15]. The comparison proved that: AccuDetect Galileo has significantly better overall performance than Galileo in terms of both sensitivity and false positive rates. AccuDetect Galileo achieved more than 60% lower false positive rate per image for malignant lesion hypothesis (p<0.001) when operating point for AccuDetect Galileo is set to match the sensitivity of Galileo. The study proved that not all CAD products are alike and in this comparison study, Parascript s AccuDetect Galileo achieved a significant, 60% reduction in false positive rate per image and better overall performance in sensitivity. Therefore, hospitals should pay a close attention to performance details when selecting CAD systems, especially if they care about a significant false mark reduction on reviewed mammograms. Parascript Page 8 7/8/2011
About University of Udine The University of Udine was founded in 1978 as part of the reconstruction plan of Friuli after the 1976 earthquake. Its aim was to provide the Friulian community with an independent center for advanced training in cultural and scientific studies. The University currently has 10 faculties in: Agriculture, Economics, Engineering, Law, Foreign Languages, Education, Humanities, Medicine and Surgery, Veterinary Science and Mathematical, Physical and Natural Sciences. The University is actively involved in student and staff exchange projects with universities within the EU and is currently engaged in close collaboration with several universities from Eastern Europe and other non-eu countries. Moreover the University participates in many research projects at national and international level. The present number of students enrolled at the University is approx. 17000. About Parascript The Parascript image analysis suite extracts meaningful information from images. Employing patented digital image analysis and pattern recognition technologies, the Parascript image analysis suite improves decision quality in medical imaging, postal and payment automation, fraud detection, and forms processing operations. Parascript software processes over 100 billion imaged documents per year. Fortune 500 companies, postal operators, major government, and financial institutions rely on Parascript products. Organizations include the U.S. Postal Service, Böwe Bell + Howell, Fiserv, Elsag, Lockheed Martin, NCR, Siemens, and Burroughs. Parascript is online at http://www.parascript.com. Parascript Page 9 7/8/2011
References 1.Bazzocchi M et al. (2007) CAD systems for mammography: a real opportunity? A review of the literature. Radiol Med 112:329-353 2.Vyborny CJ et al. (2000) Computer-aided detection and diagnosis of breast cancer. Radiol Clin North Am 38(4): 725-740 3.Malich A. et al (2006) CAD for mammography: the technique, results, current role and further developments. Eur Radiol 16:1449-1460 4.Chan HP et al. (1999) Improvement of radiologists' characterization of mammographic masses by using computer-aided diagnosis: an ROC study. Radiology 212:817-827 5.Thurfjell EL et al. (1994) Benefit of independent double reading in a population-based mammography screening program. Radiology 191: 241-244 6.Helvie M (2007) Improving mammographic interpretation: double reading and computer aided diagnosis. Radiol Clin North Am 45:801-811 7.Taylor P, Potts HWW (2008) Computer aids and human second reading as interventions in screening mammography: Two systematic reviews to compare effects on cancer detection and recall rate. EJC 44;798-807 8.Gur D et al. (2004) Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst 96:185-190 9.Fenton JJ et al. (2007) Influence of computer-aided detection on performance of screening mammography. N Engl J Med; 356:1399-409 10.Ciatto S et al. (2004) Comparison of two commercial systems for computer-assisted detection (CAD) as an aid to interpreting screening mammograms. Radiol Med 107:480-488 Parascript Page 10 7/8/2011
11.Skaane P (2009) Studies comparing screen-film mammography and full-field digital mammography in breast cancer screening: updated review. Acta Radiol 1:3-14 12.Brancato B et al. (2008) Does computer-aided detection (CAD) contribute to the performance of digital mammography in a self-referred population? Breast Cancer Res Treat 111:373-376 13.The JS et al. (2009) Detection of breast cancer with full-field digital mammography and computeraided detection 192:337-340 14. Balleyguier C et al. (2007) BIRADS classification in mammography. Eur J Radiol 61:192-194 15. University of Udine, Department of Radiology (2007) CAD and mammography: state of the art and clinical impact. Materials of XXXVI Annual Meeting of the Radiologists of the Alpe - Adria Region. Verona, 11.17.2007 Parascript Page 11 7/8/2011