Kelly H. Zou, Kemal Tuncali, Simon K. Warfield, Christian P. Zentai, Daniel Worku, Paul R. Morrison, Stuart G. Silverman

Transcription

1 Scientific Reports Three-dimensional Assessment of MR Imagingguided Percutaneous Cryotherapy Using Multiperformer Repeated Segmentations: The Value of Supervised Learning 1 Kelly H. Zou, Kemal Tuncali, Simon K. Warfield, Christian P. Zentai, Daniel Worku, Paul R. Morrison, Stuart G. Silverman Rationale and Objectives. Accurate and reproducible segmentations of two-dimensional images are an important prerequisite for assessing tumor ablations three dimensionally (3D). We evaluated whether supervised learning methods would improve multiperformer repeated segmentations of magnetic resonance images (MRI) obtained before and after MRI-guided cryotherapy of renal cell carcinoma. Materials and Methods. Three medical students independently performed five manual segmentations of a biopsy-proven renal cell carcinoma that was treated with percutaneous MRI-guided cryotherapy. Using pretreatment (T2-weighted fast recovery fast spin echo [FRFSE]) and posttreatment (T1-weighted, fat-suppressed, dynamically enhanced) MRIs, regions of tumor cryonecrosis were segmented. The same tasks were repeated after an experienced abdominal radiologist provided supervised learning. Segmentation sensitivity was compared with an estimated 3D-ground truth via voxel counts for regions of tumor, both before and after treatment, and for the regions of cryonecrosis. The sensitivity of each repeated segmentation was compared against the estimated ground truth using sensitivity, overlap index, and volume (ml). Results. Supervised learning significantly improved posttreatment segmentation sensitivity (P.03). With supervised learning, the ranges of the performance metrics over the segmentation performers were: pretreated tumor, sensitivity , overlap index , and volume ml; posttreated tumor, sensitivity , overlap index , and volume ml; in the ablation zone, sensitivity , overlap index , and volume ml. Conclusions. Supervised learning improved multiperformer repeated segmentations of MRIs obtained before and after MRI-guided percutaneous cryotherapy of renal cell carcinoma. These methods may prove useful in aiding the 3D assessment of percutaneous tumor ablations. Key Words. Cryotherapy; expectation-maximization (EM) algorithm; MR imaging; overlap index; renal cell carcinoma; sensitivity. AUR, 2005 Acad Radiol 2005; 12: From the Department of Radiology, Brigham and Women s Hospital, Harvard Medical School, Boston, MA (K.H.Z., K.T., S.K.W., C.P.Z., D.W., P.R.M., S.G.S.) and Department of Health Care Policy, Harvard Medical School, 180 Longwood Avenue, Boston, MA (K.H.Z.). Received August 2, 2004; revised December 9; accepted December 10. Partially supported by U.S. National Institutes of Health grants R01LM A1, R03HS , P01CA , R01EB000304, and P41RR13218, and grants from the Whitaker Foundation and Center for Integration of Medicine and Innovative Technology. Address correspondence to: K.H.Z. zou@bwh.harvard.edu. This article was presented at the session of the Radiology Alliance for Health Services Research (RAHSR) sponsored Association of University Radiologist (AUR), Annual Conference AUR, 2005 doi: /j.acra

2 Academic Radiology, Vol 12, No 4, April D MRI-GUIDED PERCUTANEOUS CRYOTHERAPY Image-guided percutaneous tumor ablations are promising alternative therapies for a variety of neoplasms, including the kidney (1 3). Magnetic resonance imaging (MRI)- guided cryotherapy is one such ablation technique (4 9). In general, to treat a tumor successfully, the ablation must include the tumor in its entirety, in three dimensions. Three-dimensional (3D) assessments of ablations can provide quantitative assessments of ablations and can assess if the treatments are complete (6). In order to create 3D images and quantitative metrics, two-dimensional (2D) images need to undergo a process called segmentation, which refers to outlining the region of interest, which is then used to create the 3D image. The validity of 3D assessments is directly dependent on segmentations that are accurate and reproducible. It is difficult to obtain an accurately segmented tumor (ie, one that represents the actual tumor volume). For example, distinguishing signal intensities resulting from tumor from those of surrounding normal tissue is difficult at the margin of tumor because of partial volume averaging. Similarly, determining the volume of necrosis induced by ablation requires differentiating signal intensities of necrotic tissue from nonnecrotic tumor and from unaffected surrounding tissue. In assessing ablations accurately, it is important to use segmentations that are as close as possible to the actual or true volume. The actual volume of a structure or region, hereafter referred to as the ground truth of the 3D volume cannot be determined directly. In this study, using an illustrative case of a renal cell carcinoma treated with MRI-guided percutaneous cryotherapy, we used repeated segmentations to estimate ground truth 3D volumes. Then, using an extensive statistical validation analysis and estimated ground truth 3D volumes as the gold standard, we show how supervised learning methods can be used to improve the sensitivity of multiperformer, repeated segmentations compared with a single set of manual segmentations (10 13). MATERIALS AND METHODS Cryotherapy Procedure and Imaging Protocols A 79-year-old man with biopsy-proven renal cell carcinoma underwent MRI-guided percutaneous cryotherapy under the following imaging protocol (14). Pretreatment imaging. MRI at 1.5T showed a 2.5-cm diameter tumor in the upper pole of the right kidney. Segmentations of the tumor were performed retrospectively using axial T2-weighted fast recovery fast spin echo images (repetition time/echo time [TR/TE]: 2,017 ms, ms; 5-mm slice thickness; 1-mm gap; 36-cm field of view [FOV]; matrix size). MRI-guided cryoablation. The procedure was performed in a vertically 0.5T open interventional MRI scanner (4 9). The patient was prepped in a prone position and maintained under general anesthesia. Under MRI guidance, three 2.4-mm diameter cryoneedles were placed into the tumor. Two 15-minute freezes (separated by a 10-minute thaw) were applied, during which ice formation in tissue (hereafter referred to as iceballs) was monitored by repeated MRI during therapy. Posttreatment imaging. 1.5T MRI at 24 hours demonstrated a region of cryonecrosis (ablation zone) that encompassed the tumor. Segmentations of the ablated tumor and ablation zone were performed retrospectively using intravenous gadolinium-enhanced axial 3D spoiled gradient recalled acquisition in a steady state (GRASS) T1-weighted images; flip angle; TR/TE: 6.7 ms, 1.9 ms; 2.5-mm slice thickness; 2.5-mm overlapping reconstruction; 40-cm FOV; matrix size). Repeated Multiple Manual Segmentations Manual segmentation. Repeated manual segmentations were performed independently using a 3D Slicer ( on an Ultra 10 Workstation (Sun Microsystems, Mountain View, CA). Structures were outlined for each 2D slice. Each anatomic structure was defined as the set of labeled voxels within the resulting 3D contour to compute the volume (ml) in the 3D Slicer (15). Note that these tasks are highly operator-dependent among these segmenters. The brightness and geometric features of the tumor were examined in the pretreatment segment and absence of uptake in the posttreatment images with recognition of differences in the look of the treated tumor versus treated normal tissue. Without supervised learning. This was a 15-minute session of reviewing the pretreatment and posttreatment MRIs with a board-certified abdominal imaging staff radiologist with 5 years of experience in performing percutaneous ablations and interpreting postablation MRIs. Three different medical students (segmenters) performed five manual segmentations each of the pretreatment tumors, posttreatment tumors, and the ablation zone using the previously specified MRIs on the segmentation workstation. With supervised learning. One month after the initial segmentations were completed, the segmenters had a dedicated teaching session provided by the same staff radiol- 445

3 ZOU ET AL Academic Radiology, Vol 12, No 4, April 2005 ogist. During this 1-hour session, the general anatomic structures and margins of the ablation zone and ablated tumor were reviewed, including those prone to volume averaging effects. Teaching interpretation of the ablation zone also included correlation with intraprocedural MRI of the iceball that forms the ablation zone. After this teaching session, the three segmenters performed again five manual segmentations each of the pretreatment and posttreatment tumor, as well as the region of cryonecrosis. The same staff radiologist was aware of the results from the gold standard when conducting the tutorial sessions. The main difference between the two session types was that the latter reviewed slice by slice what could be considered tumor and what was the necrosis. It more clearly established the demarcation rules in light of subtleties in the tissues, especially in postablation exam. All segmentations were done in a randomized order. The staff radiologist used the intraprocedural iceball images to better orient the segmentations during teaching. Unfortunately, it is generally difficult to obtain an accurately segmented target (ie, one that represents the actual tumor volume). For example, distinguishing signal intensities that are the result of tumor from those of surrounding normal tissue is difficult at the margin of tumor because of partial volume averaging. Similarly, determining the volume of necrosis induced by ablation requires differentiating signal intensities of necrotic tissue from nonnecrotic tumor and from unaffected surrounding tissue. Statistical Methods For simplicity, we assumed that the ith individual voxel (i 1,... I) in a region of interest belonged to either of two mutually exclusive sets (ie, background class versus target class, respectively), with m and n voxels (I m n), as determined by a corresponding unknown binary ground truth, G i, to be estimated. Furthermore, we denoted the segmenters by S k (k 1,... K), each performing independent segmentations with repetitions R l (l 1,... L). There were different segmentation learning conditions, labeled as T w (w 1,... W). Under each combination of (S k R l T w ), a binary segmentation decision, D i, was observed from the dichotomous choice in segmentation for the ith voxel, with a corresponding unknown ground truth, G i. In this example, the three medical students (K 3) each performed five repetitions (L 5) of manual segmentations in the target region of interest, including pretreatment tumor, posttreatment tumor, and region of cryonecrosis. Two conditions were employed (W 2), with and without having the performers undergone a supervised learning session provided by the same trained radiologist. Thus there was a total of combinations for each of the three target areas. Estimation of the 3D true target location. We estimated a 3D composite ground truth by an expectationmaximization algorithm. We applied our recently developed program named Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm (10,16,17). The 3D ground truth was estimated based on these segmenters independent repeated manual segmentations. We categorized their performance sensitivity (true positive fraction), in which true positive fraction was defined as the fraction of voxels in the segmentation under the estimated ground truth of the target. After convergence was reached to estimate the segmentation sensitivity using an iterative algorithm, a ground truth estimate on a [0, 1] continuum for each voxel was obtained, along with the estimated performance parameters for each segmentation repetition. Such a probabilistic ground truth was further dichotomized to derive a binary choice of the target versus the background voxels. See reference (10) for technical details on this STAPLE program for estimating the 3D ground truth, which has also extended the STAPLE algorithm using a Markov random fields (MRF) model by incorporating a spatial MRF prior distribution. Several combination methods were adopted using this expectation maximization (EM) algorithm. For example, with multiperformer repeated segmentations, we combined the repeated segmentations within the same segmenter first, here called intracombined ground truth (ICG), followed by combining these ICGs again between all performers, here called cross-combined ground truth. Alternatively, the segmentation results by segmenters over repetitions may be combined all together at once, called all-combined ground truth. Validation measures. The following statistical validation measures were calculated; their concepts were summarized previously (11 13). First, the segmentation sensitivity during the estimation of the ground truth was characterized by quality (12,18,19), which was estimated for each of the segmenter, repetition, and learning condition. When the fraction of the target relative to the entire region of interest is small, it is typical to observe a variable sensitivity. Second, Dice and Zijdenbos adopted a spatial similarity coefficient, here labeled as Z to represent an overlap index for the segmentation target (13,20 22). Excellent 446

4 Academic Radiology, Vol 12, No 4, April D MRI-GUIDED PERCUTANEOUS CRYOTHERAPY interaction between the effects because of the segmenter and supervised learning (23). For each target segmentation area of interest, the effects on sensitivity, overlap index, and volume including segmenter, repetition, and learning were tested via an F-test according to the analysis of variance. The associated P values were reported from these testing procedures. Because sensitivity and overlap index were both restricted in the range of 0 to 1, we transformed them using a logit-transformation, defined by logit (Z) ln{z/ (1-Z)}, where ln was the natural log function (based e) to improve the comparison using the F-test (13). RESULTS Figure 1. Region of cryonecrosis in normal tissue surrounding treated tumor after cryotherapy but before supervised learning: frequency of one expert s repeated manual segmentations of the zone (top left) and intracombined ground truth estimation of the zone (top right); after supervised learning: frequency (bottom left) and estimated ground truth (bottom right) of the same expert. agreement was considered when Z 70% (22). For the segmentation target and the corresponding estimated gold standard, here labeled A and B, their overlap index was simply defined as Z {2(A B)}/(A B) via voxel counts, where the symbol represents the spatial interception between the volumes of these two objects. See details and illustrative figures on this index published earlier (13). Usefully, this coefficient is in fact a special limiting case of the well-known reliability measure, the kappa statistic (18,19). It is a simple function of the Jaccard coefficient (21), essentially the intersection over the union of two regions spatially (13,22). We computed the 3D volumes, V. A coefficient of variation was used where CV (standard deviation)/mean. Variance component analysis. As described previously, we examined a total of K L W combinations of factors contributing toward the segmentation variability for segmenters S k (k 1,...,K), repetitions R l (l 1,...,L), and learning conditions T w (w 1,...,W). A two-way variance component model with an interaction term is employed with an We found that supervised learning significantly improved the sensitivity of segmentation for posttreatment tumor and posttreatment region of cryonecrosis (all P.003). Figure 1 shows the manual segmentations of posttreatment necrosis by a segmentation performer, along with the frequency of his decisions and the ICG, without and with learning. The 3D visualization of the pretreatment tumor is shown in Fig. 2. Moderate to high sensitivity was observed despite the small size of the target. In Table 1, we present the estimated sensitivity of each segmentation repetition by segmenter and by target region, with and without supervised learning to compare them with the ICG. These results were further combined using the cross-combined Figure 2. A three-dimensional visualization of the pretreatment tumor and the surrounding anatomy using the 3D Slicer. 447

5 ZOU ET AL Academic Radiology, Vol 12, No 4, April 2005 Table 1 Estimated Sensitivity ( ) Against Cross-Combined Ground Truth by Segmenter, Learning, and Target Segmenter Supervised Learning Pretreatment Posttreatment Posttreatment Region of Cryonecrosis S 1 Without With S 2 Without With S 3 Without With ground truth method over all three segmenters. We observed the following summary statistics with respect to segmentation sensitivity and the volume of the regions of interest. Without Supervised Learning For pretumor: sensitivity range , overlap index range , and volume range ml. For posttumor: sensitivity range , overlap index range , and volume range ml. In the region of cryonecrosis, sensitivity range , overlap index range , and volume range ml (Tables 1 3). With Supervised Learning For pretumor: sensitivity range , overlap index range , and volume range ml. For posttumor: sensitivity range , overlap index range , and volume range ml. In the ablation zone, sensitivity range , overlap index range , and volume range ml (see Tables 1 3). Figure 3. Transformed sensitivity on a logit scale against intracombination ground truth before (x axis) and after (y axis) supervised learning. Red: pretreatment tumor; green: posttreatment tumor; blue: posttreatment necrosis; triangle: Segmenter 1; dot: Segmenter 2; square: Segmenter 3. Figure 4. Transformed similarity index on a logit scale against intracombination ground truth before (x axis) and after (y axis) supervised learning. Red: pretreatment tumor; green: posttreatment tumor; Blue: posttreatment necrosis; triangle: Segmenter 1; dot: Segmenter 2; square: Segmenter

6 Academic Radiology, Vol 12, No 4, April D MRI-GUIDED PERCUTANEOUS CRYOTHERAPY Table 2 Estimated Zijdenbos s Similarity Coefficient (Z) Against Cross-Combined Ground Truth by Segmenter, Learning, and Target Segmenter Supervised Learning Pretreatment Posttreatment Posttreatment Region of Cryonecrosis S 1 Without With S 2 Without With S 3 Without With The mean logit-transformed sensitivity and mean logittransformed similarity based on ICG are displayed in Fig. 3 and 4, and in Table 2, respectively. Most metric values lie above a 45-degree line, indicating an improvement in qualities from supervised learning, particularly for posttreatment tumor segmentations. The variance component analyses confirmed that learning of the segmenters via supervised learning significantly improved the sensitivity of segmentations for posttreatment tumor and region of cryonecrosis (all P values.03, based on the F-tests), but the added benefit of supervised learning was not statistically significant for pretreatment tumor (closer to the 45-degree lines in Fig. 3). The interaction term between supervised learning and repeated segmentation was not significant, so learning appeared to be beneficial for all segmenters. Volumetric analysis showed that the learning helped reduce the variability. Finally, the repetition effect was not statistically significant, evidenced by all logit transformation of the coefficients of variation within 0.75 and 0.20 before and after learning, respectively. The segmenters did not show significant differences among them. However, the volume measures were variable among the three segmenters (Table 3). We did not find a statistical significant difference in segmentation sensitivity over all repetitions over time. DISCUSSION The goal of all ablations is to treat the entire volume of tumor such that the tumor is completely necrotic. This requires approaching the assessment of ablations in three dimensions. To assess tumor ablations in 3D, 2D images need to be segmented accurately. Our assumption was that the learning condition and repetition might have varied interactions with different segmenters. Thus we hypothesized a learning curve effect among the repetitions over time, requiring an expanded variance component model for volume. We demonstrated how medical students were trained to segment MRIs. We describe a tool that was used to assess how well they were trained. We derived a method Table 3 Estimated Volume (V, in ml) Against Cross-Combined Ground Truth by Segmenter, Learning, and Target Segmenter Supervised Learning Pretreatment Posttreatment Posttreatment Region of Cryonecrosis Without S 1 With Without S 2 With Without S 3 With Estimated ground truth Without With

7 ZOU ET AL Academic Radiology, Vol 12, No 4, April 2005 for establishing a reasonable approximation of the true volume of interest, referred to as the ground truth, to which repeated segmentations were compared. The tumor was easier to segment before the treatment. Interpretation of posttreatment imaging was more difficult because ablation blurred the margins of the tumor. Our results show that the sensitivity of segmenting posttreatment tumor and posttreatment region of cryonecrosis were significantly improved with a teaching session. In the postcontrast images, the tumor and ablated normal kidney tissue may have larger differences in signal intensity observed. The added benefit of learning was statistically significant in segmenting posttreatment areas. We recommend that medical students be trained for specific details of imaging findings pertaining to the pathology or posttreatment changes related to a procedure by an experienced staff radiologist. Our findings also suggested satisfactory but variable segmentation results among these students. We are now conducting a study to assess whether experienced radiologists who perform multiple segmentations on ablated tumors improve with repetition. There are several limitations to our study, however. First, sensitivity may depend on the size of the target in the region of interest (11 13). Furthermore, because intraprocedural images were used during the dedicated teaching session, observations with and without supervised learning were not strictly independent. Spatial dependence and homogeneity are typically present in image segmentation problems. Therefore, the assumption that the data at each voxel were independent might be an oversimplification. Postablation 3D imaging evaluation is a relatively new area with limited data. Ablative effects superimposed on tumors result in complex images. Margins of structures become blurred, and identification and interpretation of imaging findings thus become difficult. In the future, the above issues will be applied to other percutaneous MRI-guided tumor ablation evaluations based on our validation methods presented here with a sufficiently larger number of cases and with computer simulative methods. We will also conduct validation experiments to assess whether the output of an algorithm differs as much from individual observers, as the observers differ from each other, or the ground truth estimation via different combination strategies. Finally, more complicated analyses using a generalization to random effects models and other evaluative methods may also be conducted (11 13,23). REFERENCES 1. Gervais DA, McGovern FJ, Arellano RS, et al. Renal cell carcinoma: clinical experience and technical success with radio-frequency ablation of 42 tumors. Radiology 2003; 226: Lewin JS, Connell CF, Duerk JL, et al. Interactive MRI-guided radiofrequency interstitial thermal ablation of abdominal tumors: clinical trial for evaluation of safety and feasibility. J Magn Reson Imaging 1998; 8: Pavlovich CP, Walther MM, Choyke PL, et al. Percutaneous radio frequency ablation of small renal tumors: initial results. J Urol 2002; 167: Lu DS, Silverman SG, Raman SS. MR-guided therapy. Applications in the abdomen. Magn Res Imaging Clin North Am 1999; 7: Silverman SG, Tuncali SK, Adams DG, et al. MR imaging-guided percutaneous cryotherapy of liver tumors: Initial experience. Radiology 2000; 217: Silverman SG, Sun MRM, Tuncali K, et al. Three-dimensional assessment of MRI-guided percutaneous cryotherapy of liver metastases. AJR Am J Roentgenol 2004; 183: Shingleton WB, Sewell PE Jr. Percutaneous renal tumor cryoablation with magnetic resonance imaging guidance. J Urol 2001; 165: Shingleton WB, Sewell PE Jr. Percutaneous renal cryoablation of renal tumors in patients with von Hippel-Lindau disease. J Urol 2002; 167: Shingleton WB, Sewell PE Jr. Cryoablation of renal tumours in patients with solitary kidneys. BJU Int 2003; 92: Warfield SK, Zou KH, Wells WM III. Truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 2004; 23: Zou KH, Wells WM III, Kikinis R, et al. Three validation metrics for automated probabilistic image segmentation of brain tumors. Stat Med 2004; 23: Zou KH, Warfield SK, Fielding JR, et al. Statistical validation based on parametric receiver operating characteristic analysis of continuous classification data. Acad Radiol 2003; 10: Zou KH, Warfield SK, Bhratha A, et al. Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol 2004; 11: Zentai CP, Worku D, Tuncali K, et al. Validation of 3D-assessment of MR imaging-guided percutaneous cryotherapy of a soft-tissue metastasis. 18th International Congress and Exhibition CARS 2004, Computer Assisted Radiology and Surgery. Chicago, Il: June 23 26, Gering DT, Nabavi A, Kikinis R, et al. An integrated visualization system for surgical planning and guidance using image fusion and an open MR. J Magn Res Imaging 2001; 13: Dempster AP, Laird NM, Rubin DB. Maximum-likelihood from incomplete data via the {EM} algorithm. J R Stat Soc (Ser B) 1997; 39: Tanner MA. Tools for statistical inference: methods for the exploration of posterior distributions and likelihood functions. New York: Springer- Verlag, Agresti A. Categorical data analysis. New York: John Wiley & Sons, Fleiss JL. Statistical methods for rates and proportions. 2nd ed. New York: John Wiley & Sons, Dice LR. Measures of the amount of ecologic association between species. Ecology 1945; 26: Jaccard P. The distribution of flora in the alpine zone. New Phytologist 1912; 11: Zijdenbos AP, Dawant BM, Margolin RA, et al. Morphometric analysis of white matter lesions in MR images: method and validation. IEEE Trans Med Imaging 1994; 13: Pinheiro JC, Bates DM. Mixed-effects models in S and S-Plus. New York: Springer-Verlag,