Women s Imaging Original Research

Transcription

1 Women s Imaging Original Research Women s Imaging Original Research David Gur 1 Andriy I. Bandos 2 Howard E. Rockette 2 Margarita L. Zuley 3 Jules H. Sumkin 3 Denise M. Chough 3 Christiane M. Hakim 3 Gur D, Bandos AI, Rockette HE, et al. 1 Department of Radiology, University of Pittsburgh, Radiology Imaging Research, 3362 Fifth Ave., Pittsburgh, PA Address correspondence to D. Gur (gurd@upmc.edu). 2 Department of Biostatistics, University of Pittsburgh, Graduate School of Public Health, Pittsburgh, PA. 3 Department of Radiology, Breast Imaging, Magee- Womens Hospital, Pittsburgh, PA. AJR 2011; 196: WOMEN S IMAGING Keywords: digital breast tomosynthesis, digital mammography, free-response receiver operating characteristic (FROC), full-field digital mammography (FFDM), observer performance study, recall rate DOI: /AJR Received April 9, 2010; accepted after revision July 20, Supported by grant EB to the University of Pittsburgh from the National Institute for Biomedical Imaging and Bioengineering, National Institutes of Health and by grant BCTR to the University of Pittsburgh from the Susan G. Komen Foundation. D. M. Chough, and C. M. Hakim have participated as readers in independent reader studies for Hologic. J. H. Sumkin and M. L. Zuley are principal investigators on other research projects funded by Hologic X/11/ American Roentgen Ray Society Localized Detection and Classification of Abnormalities on FFDM and Tomosynthesis Examinations Rated Under an FROC Paradigm OBJECTIVE. The purpose of our study was to assess diagnostic performance when retrospectively interpreting full-field digital mammography (FFDM) and breast tomosynthesis examinations under a free-response receiver operating characteristic (FROC) paradigm. MATERIALS AND METHODS. We performed FROC analysis of a previously reported study in which eight experienced radiologists interpreted 125 examinations, including 35 with verified cancers. The FROC paradigm involves detecting, locating, and rating each suspected abnormality. Radiologists reviewed and rated both FFDM alone and a combined display mode of FFDM and digital breast tomosynthesis (DBT) (combined). Observer performance levels were assessed and compared with respect to the fraction of correctly identified abnormalities, the number of reported location-specific findings (both true and false), and their associated ratings. The analysis accounts for the number and locations of findings and the location-based ratings using a summary performance index (Λ), which is the FROC analog of the area between the receiver operating characteristic curve and the diagonal (chance) line. RESULTS. Under the FROC paradigm, each reader detected more true abnormalities associated with cancer, or a higher true-positive fraction, under the combined mode. In an analysis focused on both the number of findings and associated location-based ratings, each of the radiologists performed better under the combined mode compared with FFDM alone, with increases in Λ ranging from 5% to 34%. On average, under the combined mode radiologists achieved a 16% improvement in Λ compared with the FFDM alone mode (95% CI, 7 26%; p < 0.01). CONCLUSION. We showed that DBT-based breast imaging in combination with FFDM could result in better performance under the FROC paradigm. With the availability of digital breast tomosynthesis (DBT) imaging systems for visualizing 3D breast examinations [1 5], there are a number of questions related to the acquisition, operation, training, and display of DBT examinations that need to be addressed if DBT is to be optimally incorporated in routine clinical practice. DBT is of great interest in screening as well as in diagnostic procedures because it enables the 3D reconstruction of images, thus allowing cross-sectional visualization of breast tissue. This representation of the breast reduces the difficulty associated with interpretations of projection mammograms that are inherent to these procedures because of superposition or overlapping tissue. Several studies have addressed technical, ergonomic, and performance issues associated with DBT, but the results from retrospective or subjective rating studies, although en- couraging, remain inconclusive. After a previous pilot study to begin to address some of the issues involved [6], we performed one of several multimodal observer performance studies planned as an integral part of a comprehensive assessment of DBT. Previously [7] we showed, based on screening BI-RADS ratings ( recall or not recall ), that the combined use of FFDM and DBT was superior to FFDM alone in terms of significantly reducing recall rates for further diagnostic workups. These results are relevant to the evaluation of the possible utility of DBT in some environments (e.g., screening), but the analyses performed do not characterize all relevant aspects of diagnostic performance, such as location of the abnormality in question or the identification of multiple abnormalities on the same examination. In this article we analyze the relative performance of these techniques in terms of detection and correct localization AJR:196, March

2 of abnormalities associated with verified cancers on the same data set. Materials and Methods General Study Design The examinations used in this study consisted of FFDM and DBT and have been described in detail elsewhere [7]. The acquisition of all examinations was performed under protocols approved by the institutional review board (IRB) that included signed informed consent by the participant. The reading study consisted of, among others, an FFDM alone mode and a combined FFDM plus DBT mode. We consider these two modes in the present analysis. All reviews and ratings were performed on our specially designed workstation. The workstation (Dual Core Opteron, Processor 270, 2 GHz, and 6.00 GB of RAM; AMD) operates under Microsoft Window Server The workstation display consisted of two high-resolution (2,048 2,560), 8-bit gray-scale, portrait monitors at a nominal setting of 80 ftl. Two Dome C5i flat-panel monitors (Planar Systems) were used for image display. A management program determined the reading sessions for individual observers and the order of displayed cases within a session. All examinations were reviewed under the modes in question with a predetermined minimum delay of 30 days between consecutive modes. Radiologists typically completed the viewing and rating of the 125 examinations during three to four separate reading sessions. Cases The detailed description of the data set is provided elsewhere [7]. Each examination consisted of two views of one breast, either the right craniocaudal and right mediolateral oblique or the left craniocaudal and left mediolateral oblique. Ninety examinations were verified as negative or depicting benign findings, and 35 examinations had verified cancer [7]. All cancer-depicting examinations available to us at the time of the study design and a predetermined number of randomly selected negative examinations were selected for the study. We note that some of the 35 positive cases depicted more than one index lesion (abnormality) associated with verified cancer; hence, in total, there were 50 positive regions that were included in the freeresponse receiver operating characteristic (FROC) analysis. Radiologists rate breast tissue density into one of four BI-RADS categories established by the American College of Radiology: 1, almost entirely fatty (< 25% fibroglandular); 2, scattered fibroglandular densities (25 50% fibroglandular); 3, heterogeneously dense breast tissue (51 75% fibroglandular); and 4, extremely dense breast tissue (> 75% fibroglandular). Subjective density ratings were determined and assigned to each breast independently of the reader study during the performance of our case verification protocol. The BI-RADS breast tissue density ratings for the examinations used in this study were 22 of 125 (17.6%), 95 of 125 (76.0%) and eight of 125 (6.4%) for density BI-RADS 2, 3, and 4, respectively. As a part of a prestudy comprehensive verification protocol, the center coordinate of each depicted abnormality was marked and saved in a reference (truth) file [7]. Observers Eight board-certified, Mammography Quality Standards Act qualified radiologists with varying experience ranging from 3 to 35 years of reading mammography were selected for the study. Observers were unaware of the type of examinations used for this study in terms of case mix (negative vs positive), breast density distribution, and distribution of abnormalities within the image set or the specific aims of the study. Observers received an Instruction to Observers document to review before beginning the study, which has been described previously. Before the start of each mode, each observer was given specific examples and an interactive training session for familiarization with the workstation functionality under the study conditions as well as the computerized scoring form. Observers were given an opportunity to ask questions, and a staff member was available during the sessions to answer questions not related to the actual diagnosis. Case Interpretations Radiologists were asked specifically to independently review and rate each examination for the presence or absence of the abnormalities in question under each of the reading conditions. Namely, they had to assume that this was the initial screening examination (i.e., no priors) and there was only one imaged breast with two corresponding views. Under the FROC paradigm, the observer was instructed to mark the assessment of the center of any suspicious region using a computer mouse and then rate it (i.e., suspicion level). After marking the region, the type of abnormality in question was identified and two semicontinuous (0 100) rating scales (sliders) appeared for the likelihood of the presence or absence of the abnormality and the likelihood of the abnormality in question representing a cancer if it was actually present [7]. Multiple abnormalities (or suspected locations) could be marked and rated on the same examination (or image) as deemed appropriate. If no abnormality was detected, the reader could just click on the done button at the bottom of the display. At any time during the location-based interpretation (and marking/rating) of an examination, the observer could edit, remove, or add marks as deemed appropriate. On completion of the ratings of all suspected abnormalities, the observer was asked to provide a screening BI- RADS recommendation for the examination in question (i.e., 0 for recall, 1 for negative, or 2 for benign findings). The latter rating data were analyzed and the results reported [7]. Data Analysis We focus in this article on the detection, correct localization, and classification of 50 depicted abnormalities that were verified to be associated with cancer [7]. Consequently, for the primary analysis we considered only marks (reported locations) rated by the radiologists as having nonzero probability for being associated with a cancer. Each mark with a nonzero malignancy rating was classified as a true-positive if the marked center was sufficiently close to an abnormality depicting cancer determined by a preset acceptance radius and was of the same type (mass or calcification). The use of an acceptance radius (radius of the acceptance target) is required in all conventional localization-based FROC analyses, and it defines the maximum allowed distance between the center of the abnormality in question marked during the verification process and the radiologist s mark to be considered a correctly localized response. Figure 1 illustrates the concept of acceptance targets as used in FROC analyses. These targets are predetermined and are not shown or known to the observers during interpretation of the examinations of interest. If observers mark the center of a suspected abnormality within the target region, it is considered a hit (or a true-positive finding). Otherwise, the mark is considered a miss (or a false-positive finding).for the primary analysis the acceptance radius was set at 200 screen pixels on images with a maximum actual image matrix size of 1,800 by 2,048 pixels. All other marks with a nonzero rating for probability of malignancy were classified as false-positive identifications. We also assessed the consistency of our findings with the estimates of performance levels with respect to all (n = 98) abnormalities (benign and malignant) depicted in this data set. To assess the sensitivity of our results to the size of the acceptance radii, we considered smaller (100 pixels) and larger (300 pixels) radii than the one used in the primary analysis (200 pixels). The primary tool for performance assessment under the FROC paradigm is the FROC curve, which describes the trade-off between the number of false-positive marks per examination (FPR) and the fraction of true-positive findings (TPF) (Fig. 2). The two important characteristics of the FROC curves are the coordinates of the last operating point, namely, the maximum achieved FPR (FPR π ) and the maximum achieved TPF ( ). 738 AJR:196, March 2011

3 Fig. 1 Craniocaudal full-field digital mammogram in 71-year-old woman shows mass verified as positive for cancer with center of mass and acceptance target shown for illustration. Center mark and acceptance target are not shown to observers during interpretations. If observer marks anywhere within acceptance target that was identified as mass, the mark is considered hit (true-positive finding) in analysis. If mark, representing mark of perceived center of suspected mass, is outside acceptance target, observer s mark is considered miss (false-positive identification) in analysis. Fig. 2 Graph shows empirical free-response receiver operating characteristic performance curves for full-field digital mammography (FFDM) alone (circles) and FFDM plus digital breast tomosynthesis (DBT) combined (dashes) modes when rating data are pooled for all observers. TPF = truepositive fraction; FPR = false-positive results per examination. TPF These reflect the maximum number of correctly detected and localized abnormalities and the corresponding number of false-positive marks. The trade-off between TPF and FPR is guided by the ratings assigned to the reported marks in a manner similar to the traditional receiver operating characteristic (ROC) approach. We conducted an overall comparison of the empirical FROC curves using a summary index (Λ) [8]. This performance summary index accounts simultaneously for FPR π,, and the discriminative ability of the ratings (that determine the degree of bulging or curvature of the FROC curve). The index (Λ) is the FROC analog of the area between the ROC curve and a diagonal line (or the chance line with an area under the ROC of 0.5), but it has a different scale of measurement. Similar to its analog in ROC analysis, Λ can be interpreted as a measure of superiority of an imaging system, or a practice, over the guessing process (which is used as a lower benchmark). Numerically, Λ is equivalent to the area between the augmented empirical FROC curve (the analog of the ROC curve) and the guessing FROC curve (the analog of the diagonal line in the ROC). Finally, this index accounts for the magnitude of an adopted proximity criteria through a parameter that depends on the relative size of the acceptance target and density of the abnormalities in the sample. For a specific predetermined acceptance target, this parameter represents the ratio of the probability of accidently marking (by chance) a true abnormality by a single mark that is randomly placed on an image relative to the probability that a randomly placed mark does not accidentally hit any abnormality of interest on the entire ensemble of images. In our study, this parameter was equal to for an acceptance radius of 200 pixels. The statistical significance of observed differences was assessed using the nonparametric bootstrap approach. Resampled data sets were generated from the samples of subjects or cases with different numbers of cancer-depicting abnormalities and the sample of readers, where each of the samples was constructed independently. The Monte Carlo bootstrap CIs were estimated as the and percentiles of the bootstrap distribution on the basis of 10,000 replicates. We also supplemented the comparison of the performance levels of the techniques with the bootstrap p value for testing the null hypothesis of zero difference. Results Figure 2 shows empirical FROC curves for the FFDM alone and the combined (FFDM plus DBT) modes for the pooled data over all readers. Under the combined mode, radiologists as a group were able to detect more abnormalities associated with cancer (higher ); however, this increase in true-positive findings was obtained at the cost of increasing the number of false-positive marks per FPR examination (higher FPR π ). Beginning at an FPR of approximately 0.06, the FROC curve for FFDM plus DBT is higher than the FROC curve for FFDM alone, with a difference in the TPF that is increasing up to 0.11 (0.72 vs 0.83) at an FPR of 0.34 (which corresponds to the last experimentally ascertained [or empirical] data point for FFDM). We note that the first empirical operating point for the combined mode is at an FPR of Table 1 shows that each reader was able to detect, correctly localize, and characterize more abnormalities associated with cancer under the combined mode (higher for individual readers). Two radiologists (numbers 1 and 3) also had a smaller number of falsepositive findings (smaller FPR π ). For the radiologists who improved at the cost of increasing FPR π, improvements in TPFs were more substantial than increases in FPRs, resulting in overall higher performance levels for each of the radiologists (in terms of Λ) under the combined mode (difference in Λ ranging from 1.1 to 5.8, which corresponds to 5% and 34%, respectively). On average, radiologists achieved a 16% performance improvement under the combined mode (average difference in Λ, 3.3; p < 0.01; 95% CI, 1.4 to 5.4 or 7 26% improvement, correspondingly). Analysis of performance levels with respect to all abnormalities (cancer and benign) showed a similar pattern. Specifically, under the combined mode radiologists had a larger Λ, both individually (except for one radiologist) and on average. However, the average overall improvement was lower than for abnormalities AJR:196, March

4 TABLE 1: Computed Observer Performance Levels FPR π FPR π FFDM Alone FFDM + DBT Λ Reader Alone DBT Difference FFDM FFDM Average Note FFDM = full-field digital mammography, DBT = digital breast tomosynthesis, FPR π = estimated maximum false-positive rate, = estimated maximum true-positive fraction. Λ is the average improvement over the guessing process or area between augmented free-response receiver operating characteristic and guessing curves [8]. The maximum possible improvement over the guessing process that could be achieved by a perfect system is 28.9 for this data set. associated with verified cancers (9%, p < 0.05). The results after changing the size of the acceptance target were consistent with the results presented here. These results show performance superiority of the combined mode despite its tendency to yield overall a larger number of falsepositive findings. We note that the increase in the false-positive findings under the combined technique occurred in examinations depicting at least one true abnormality of interest. For examinations not depicting any true abnormalities of interest, radiologists reported a lower number of false-positive findings under the combined mode. Finally, in a previous breast-based analysis using a screening BI-RADS (binary) response, we commented on a small subset of cases with poor FFDM quality in this data set. The exclusion of the same set of cases from the analyses in this study did not change the results substantially, and the conclusions remain unchanged. Discussion Recent advances in digital imaging in general, and FFDM in particular, have led to the development of DBT systems for 3D breast examinations [9, 10]. The approach taken by the manufacturers is quite practical and may actually be relatively easy to implement on current and future digital systems that are being used routinely in radiology, particularly for breast imaging [1 3, 11, 12]. Previously [7], on the same examinations, we showed that in a specific screening setting using the breast-based screening BI-RADS ratings, the combined mode was superior in terms of a significantly lower recall rate for the patients without abnormalities of interest. In this study we showed performance superiority with respect to detection, localization, and characterization of multiple abnormalities (if any) per examination. Specifically, radiologists detected, correctly localized, and characterized (regardless of the assigned ratings) more abnormalities associated with verified cancer (on average six abnormalities in 35 cases with verified cancer) with the combined mode than with the FFDM alone mode. Although this improvement was associated with an overall increase in the number of false-positive identifications (on average eight false-positive marks per 125 examinations, or per examination), the magnitude of the improvement in detection of true abnormalities was much more substantial, leading to the increase in the overall performance level. Furthermore, the increase in the number of false-positive findings was associated with examinations depicting at least one actual abnormality of interest. In examinations not depicting any of the abnormalities of interest, the total number of false-positive findings was smaller under the combined mode compared with FFDM alone. In other words, under the combined mode we observed not only a reduction in the number of unnecessary recalls as previously shown [7] but also a reduction in the number of false findings in true-negative cases. The combined mode improvement achieved in this study may have been affected by the fact that we only presented single breast based imaging examinations, thereby eliminating bilateral comparisons. However, if there is an effect, it is likely to have a similar impact on both reading modes (FFDM alone and combined); therefore, the direction of the difference in performance levels we observed is likely to be preserved. In our data, the superiority of the combined mode in terms of TPF for the same FPR was preserved for operating points with an FPR as low as 0.06 (or a total of eight false-positive identifications per 125 examinations). The pooled FROC curve was lower for the combined mode at a small number of false-positive identifications (Fig. 2). This phenomenon was caused primarily by additional locations detected and marked under the combined mode that were not reported under the FFDM alone mode. Indeed, there were more locations that were identified and reported as highly suspicious (hence, contributing to the lower left corner of the FROC curve) under the combined mode but not under the FFDM alone mode than vice versa. Second, when restricted to the locations detected under both FFDM alone and combined modes, the latter showed actual improvement in ratings (primarily in terms of an increase for true-positive findings). We investigated the robustness of our conclusions with respect to several aspects of the analysis. First, we confirmed that the directions of the observed differences in performance levels were preserved for two additional (one smaller and one larger) acceptance radii. Second, because this is the first experimentally based presentation of our newly proposed FROC summary index of performance, we also corroborated the conclusion of our study with an analysis based on the jackknife FROC-2 (JAFROC-2) summary index combined with the corresponding DBM analysis [13]. Finally, we verified that similar advantages in performance under the combined mode exist for detection, localization, and characterization of either malignant or benign abnormalities. We note that our case mix included examinations with denser breasts than the average breast density distribution expected in the general screened population. This stems from the fact that most investigators believe that, if at all, DBT would have performance advantages in denser breasts; hence, our IRB-approved protocols focus primarily on recruitment of these women (e.g., younger age or those with known nonfatty breasts on the basis of prior examinations). 740 AJR:196, March 2011

5 In conclusion, this study showed improvements in observer performance levels for the combined reading mode compared with FFDM alone when observers rated the examinations under the FROC paradigm. Digital breast tomosynthesis: initial experience in 98 women with abnormal digital screening mammography. AJR 2007; 189: Reiser I, Nishikawa RM, Edwards AV, et al. Automated detection of microcalcification clusters for digi- 9. Rafferty EA. Digital mammography: novel applications. Radiol Clin North Am 2007; 45: Lewin JM, Niklason L. Advanced applications of digital mammography: tomosynthesis and contrastenhanced digital mammography. Semin Roentgenol References 1. Niklason LT, Christian BT, Niklason LE, et al. Digital tomosynthesis in breast imaging. Radiology 1997; 205: Niklason LT, Kopans DB, Hamberg LM. Digital breast imaging: tomosynthesis and digital subtraction mammography. Breast Dis 1998; 10: Smith A. Full-field breast tomosynthesis. Radiol Manage 2005; 27: Poplack SP, Tosteson TD, Kogel CH, Nagy HM. tal breast tomosynthesis using projection data only: a preliminary study. Med Phys 2008; 35: Good WF, Abrams GS, Catullo VJ, et al. Digital breast tomosynthesis: a pilot observer study. AJR 2008; 190: Gur D, Abrams GS, Chough DM, et al. Digital breast tomosynthesis observer performance study. AJR 2009; 193: Bandos AI, Rockette HE, Song T, Gur D. Area under the free-response ROC curve (FROC) and a related summary index. Biometrics 2009; 65: ; 42: Suryanarayanan S, Karellas A, Vedantham S, et al. Evaluation of linear and non-linear tomosynthetic reconstruction methods in digital mammography. Acad Radiol 2001; 8: Wu T, Moore RH, Rafferty EA, Kopans DB. A comparison of reconstruction algorithms for breast tomosynthesis. Med Phys 2004; 31: Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis and validation. Med Phys 2004; 31: AJR:196, March