A Validation Framework for Brain Tumor Segmentation 1

Transcription

1 Technical Report A Validation Framework for Brain Tumor Segmentation 1 Neculai Archip, PhD, Ferenc A. Jolesz, MD, Simon K. Warfield, PhD Rationale and Objectives. We introduce a validation framework for the segmentation of brain tumors from magnetic resonance (MR) images. A novel unsupervised semiautomatic brain tumor segmentation algorithm is also presented. Materials and Methods. The proposed framework consists of 1) T1-weighted MR images of patients with brain tumors, 2) segmentation of brain tumors performed by four independent experts, 3) segmentation of brain tumors generated by a semiautomatic algorithm, and 4) a software tool that estimates the performance of segmentation algorithms. Results. We demonstrate the validation of the novel segmentation algorithm within the proposed framework. We show its performance and compare it with existent segmentation. The image datasets and software are available at brain-tumor-repository.org/. Conclusions. We present an Internet resource that provides access to MR brain tumor image data and segmentation that can be openly used by the research community. Its purpose is to encourage the development and evaluation of segmentation methods by providing raw test and image data, human expert segmentation results, and methods for comparing segmentation results. Key Words. Brain tumor segmentation; imaging; repository; validation; STAPLE; spectral clustering. AUR, 2007 Brain tumor segmentation has long been recognized as a difficult problem. Many interactive and automated algorithms have been proposed for it for various types of clinical applications. It includes neurosurgery planning, radioand chemotherapy monitoring and drug discovery. Some representative work can be found in (1 8,11,13). Despite the variety of methods published, there is no general algorithm accepted to segment brain tumors from Acad Radiol 2007; 14: From the Harvard Medical School, Brigham and Women s Hospital, 75 Francis St, Boston, MA (N.A., F.A.J.), Computational Radiology Laboratory, Children s Hospital Boston, Harvard Medical School. Received October 13, 2006; accepted May 10, This investigation was supported in part by NSF ITR , NMSS Award #RG 3478A2/2, a research grant from CIMIT, and by NIH grants R03 EB006515, U41 RR019703, P01 CA067165, R01 RR021885, R03 CA126466, P30 HD018655, R01 HL074942, NIHR01 GM Address correspondence to: C.R.W. narchip@bwh.harvard.edu AUR, 2007 doi: /j.acra magnetic resonance (MR) images. The reasons are manifold: 1) it is difficult to validate the segmentation algorithms due to intra- and interoperator variability in outlining the tumors; 2) the ground truth is often difficult to define; 3) the segmentation methods are tested on different image datasets; and 4) no common brain tumors database is available that enables standard benchmarking and comparison between various types of segmentation techniques; 5) no consensus exists on metrics to use for validation of segmentation algorithms; and 6) large datasets of MR brain tumor images are often not available for researchers that design segmentation algorithms. We aim to address these issues by proposing a unified framework for validation available openly on the Internet. It consists of MR image datasets of patients with brain tumors from which four independent experts have performed tumor segmentation. Previous results of a semiautomatic method for brain tumor segmentation of the same datasets (6) are also available on this website. We also 1242

2 Academic Radiology, Vol 14, No 10, October 2007 A VALIDATION FRAMEWORK FOR BRAIN TUMOR SEGMENTATION Table 1 Details on the Current 10 Magnetic Resonance Imaging Datasets of Brain Tumors Proposed as Part of the Validation Framework Patient Number Tumor Location 1 Meningioma Left frontal 2 Meningioma Left parasellar 3 Meningioma Right parietal 4 Low-grade glioma Left frontal 5 Astrocytoma Right frontal 6 Low-grade glioma Right frontal 7 Astrocytoma Right frontal 8 Astrocytoma Left temporal 9 Astrocytoma Left frontotemporal 10 Low-grade glioma Left temporal include open source software called simultaneous truth and performance level estimation (STAPLE) (9). It estimates the performance of segmentation algorithms. We also demonstrate the validation of a novel brain tumor segmentation algorithm within this framework. Details are presented in the next sections. MATERIALS AND METHODS Brain Tumor Database The image repository consists of 10 MR image datasets of patients with brain tumors. Details on the pathology and location of tumors are presented in Table 1. The images are made available with institutional review board approval and are stored in DICOM format. Personal information about the patients is removed. Only image data and pathology information are provided. Each dataset consist of an Spoiled Gradient Recalled (SPGR) volume ( , mm), and pre- and post-contrast T1- and T2-weighted volumes ( , 115 mm) obtained with a GE 1.5 T MR imaging device (Signa; GE Medical Systems, Milwaukee, WI). Existent Segmentation Data Each MR image dataset has four different segmentations of tumor contours, performed by four independent experts. The segmentation results of a semiautomatic algorithm (4) applied to the same datasets are also included. Examples of segmentations are presented in Fig 1. The Performance Assessment of Image Segmentation Algorithms Characterizing the performance of image segmentation approaches has been a persistent challenge. Quantitative performance analysis is important because segmentation algorithms often have limited accuracy and precision. Interactive drawing of the desired segmentation by domain experts has often been the only acceptable approach, and yet suffers from intraexpert and interexpert variability and is time consuming and expensive to carry out. Automated algorithms have been sought in order to remove the variability introduced by experts, but automated algorithms must be assessed to ensure they are suitable for the task. Our group has previously introduced STAPLE (9). STAPLE takes a collection of segmentations of an image and computes simultaneously a probabilistic estimate of the true segmentation and a measure of the performance level represented by each segmentation. The source of each segmentation in the collection may be an appropriately trained human rater or raters, or it may be an automated segmentation algorithm. The algorithm is formulated as an instance of the expectation-maximization algorithm. In the formulation of this algorithm, the expert segmentation decision at each voxel is directly observable, the hidden true segmentation is a binary variable for each voxel, and the performance level, or quality, achieved by each segmentation is represented by sensitivity and specificity parameters. It is accepted and expands its use as a tool used in the validation of segmentation algorithms. Among other published works, it has been employed previously (1). Details are presented elsewhere (9). A Novel Spectral Clustering Algorithm for Brain Tumor Segmentation The existent segmentation data and images are used to validate a novel segmentation algorithm. The general scheme is presented in Fig 2. In this section, we introduce a novel algorithm for brain tumor segmentation, derived from spectral clustering theory. We demonstrate its validation in the proposed framework in the next section. The existent segmentation data and imaging are used to validate a novel segmentation algorithm. The general scheme is presented in Fig 2. In this section we introduce a novel algorithm for brain tumor segmentation, derived from spectral clustering theory. We demonstrate its validation in the proposed framework in the next section. The spectral clustering techniques are based on Gestalt laws of image perception which is seen as a meaningful organization of objects in a scene. Various factors that can con- 1243

3 ARCHIP ET AL Academic Radiology, Vol 14, No 10, October 2007 Figure 1. Example of brain tumor (a) and its segmentation performed by the four experts (b e). The variability of the segmentation can be noticed. (Fig 1 continues). tribute to this process, include grouping cues such as proximity and similarity. A great deal of research in computer vision over the last few decades has sought principled ways to operationalize these ideas. An essential component is the development of grouping methodologies that use these low-level cues to perform image segmentation. A recently developed idea is to cluster pixels (or other image elements) using pairwise affinities in the form of a normalized cut (NCut) (10). The basic idea is to use a new graph-theoretic criterion for measuring the goodness of an image partition. The maximization of this criterion can be formulated as a generalized Eigenvalue problem. An important drawback of this approach is the extreme computational demand because of the large matrices involved. Spectral methods for image segmentation use the eigenvectors and eigenvalues of a matrix derived from the pairwise affinities of pixels. These Eigenvectors induce an embedding of the pixels in a low-dimensional subspace wherein a simple central clustering method can be used to 1244

4 Academic Radiology, Vol 14, No 10, October 2007 A VALIDATION FRAMEWORK FOR BRAIN TUMOR SEGMENTATION Figure 1. (continued). do the final partitioning. Here the partitioning is based on the normalized cut (NCut) (10). Some mathematical notations are first introduced. Let I be the original image of size N N. The symmetric matrix S 僆 RN2 N2 denotes the weighted adjacency matrix for a graph G V, E with nodes V representing pixels and edges E whose weights capture the pairwise affinities between pixels. Let A and B represent a bipartition of V (ie, A 艛 B V and A 艚 B V). Let cut(a,b) denote the sum of the weights between A and B: cut共a,b兲 i僆a,j僆b Sij. The degree of the ith node is defined as di j Sij. The total connection from nodes in the set A to all nodes in the graph is de- 兺 兺 兺 noted by assoc共a,v兲 i僆a,j僆v Sij. The normalized cut between sets A and B is then given by: NCut(A, B) cut(a, B) assoc(a, V ) cut(a, B) assoc(b, V ). (1) The problem is to find A and B such that NCut(A,B) is minimized. Using elements of spectral graph theory, it is been shown (10) that an approximate solution may be obtained by thresholding the eigenvector (called the Fiedler vector) corresponding to the second eigenvalue 2 of the generalized eigenvalue problem: (D S)y Dy, where D 1245

5 ARCHIP ET AL Academic Radiology, Vol 14, No 10, October 2007 where X(i) is the spatial location of node i, and F(i) isa feature vector, based on intensity. The values maxf and minf are, respectively, the maximum and the minimum values of F for the all the pixels in the image. Finally, the similarity matrix S is normalized by S S/max(S) as proposed previously (11). A requirement of NCut is the number of clusters. Overcoming this problem is a challenging task. Unsupervised clustering based solely on Fiedler eigenvector is a potential solution. In the unsupervised clustering, the user does not need to explicitly specify the number of clusters. More details follow. Figure 1. (continued). is a diagonal matrix with entries D ii d i. Image segmentation is reduced to the problem of partitioning the set V into disjoint sets V i,..,v n, such that similarity among nodes in V i is high and similarity across V i and V j is low. Here we define the similarity function between two pixels i and j as: F(i) F(j) 2 maxf minf if X(i) X( j) 2 r S ij 1 X(i) X(j) 2 0 otherwise (2) Fiedler Eigenvector for Unsupervised Image Segmentation Instead of minimizing the NCut function (as suggested previously (10), a property of the Fiedler vector is used. Let each pixel be I j, j 1,...,N 2. Given the image pixels P (I 1,..,I N2 ) and the Fiedler vector V (v 1,.., v N2 ), we consider the permutation (i 1,..,i N2 ) that sorts the vector V: (v 1,..,v N2 ). By applying to the pixel vector P, the vector (I i1,...i in2 ) is obtained. It satisfies the property that?j 1,..,j k j p l j p 1 : S(Il, I l 1 ), where S is the similarity function defined in Eq 2. This property provides a way to cluster the original image according to the given similarity metric. In its new form, the vector components are grouped in compact blocks that represent the clusters. It becomes clear that the problem of clustering the initial matrix data is reduced to the problem of determining the sequences of maximum length having similar values in the vector obtained from the Fiedler Eigenvector. Calculating the eigenvectors for the Laplacian resulting from the whole image is computationally expensive; therefore, a grid is used to divide the whole image into smaller windows. We then apply the clustering algorithm to each of these smaller windows. A global criterion is used to regroup the cluster obtained in every grid cell. There is still no need to know or estimate the number of clusters, K, in advance. The introduced algorithm requires three parameters to be specified. The first parameter (DifPixels) specifies the maximum difference between two pixels to consider them as belonging to the same cluster. This parameter could be determined for specific applications using a priori information about the image content shown previously (12). However, for our experiments we used the same value ( 5) for all examples with good results. The second parameter (DifClusters) is the threshold value when two 1246

6 Academic Radiology, Vol 14, No 10, October 2007 A VALIDATION FRAMEWORK FOR BRAIN TUMOR SEGMENTATION Figure 2. The scheme demonstrating how to validate a novel brain tumor segmentation within the proposed framework. Simultaneous truth and performance level estimation will combine existent segmentation both generated by experts and segmentation algorithm and will assess their performances. clusters are grouped together during windowing. It can also be customized for a particular application but is kept constant in the images presented in this article ( 50). Both parameters are present in the majority of the standard segmentation algorithms because one needs to specify when two pixels are similar enough to be grouped in the same region. The third parameter is the window size and is also kept constant ( 10). We have also noticed that there are small differences in the clustering results when using different values for the window size. The technique is already proved to be efficient for ultrasound segmentation (13). This method generates a clustering of the MR image dataset. The cluster that corresponds to the brain tumor needs to be specified interactively by user. This is the only interactive step required by this novel algorithm. RESULTS The novel segmentation algorithm was validated using the proposed framework. For each T1w MR image scan in the repository, the algorithm is used to segment the brain tumor. Its results and the available segmentation (five sets for each dataset) are used as input for STAPLE (as presented in the Fig 2). Sensitivity and specificity are computed by STAPLE. A ground truth for the segmentation is also estimated. A typical example is shown in Fig 3. The complete results are presented in Table 2. On averages, its sensitivity is better than the fourth expert and the Kaus segmentation, whereas its specificity is better than experts one, two, and three (Fig. 4). Information about data images and algorithms can be found online at We are currently extending the repository with new MR image datasets and tumor segmentations. This framework is open to contributions in form of image data, manual, or automatic segmentations. DISCUSSION AND CONCLUSIONS Segmentations algorithms are commonly dependent on the application domain.their validation is acknowledged as a common problem. Common drawback in evaluation of segmentation algorithms include: small image datasets used, ground truth is difficult to determine, and segmentation algorithms usually are not tested on the same datasets. In a recent article (14), five main requirements for an evaluation framework are indicated: 1. specification of readily computable, effective, and meaningful metrics of efficacy; 2. real-life image data; 3. reference segmentations that can be used as surrogates of true segmentations (ground truth); 4. a few standard segmentations algorithms; and 1247

7 1248

8 Academic Radiology, Vol 14, No 10, October 2007 A VALIDATION FRAMEWORK FOR BRAIN TUMOR SEGMENTATION Table 2 The STAPLE Output on for the 10 Datasets Used Expert 1 Expert 2 Expert 3 Expert 4 Kaus Novel Method (Fiedler based) Patient 1 pj qj Patient 2 pj qj Patient 3 pj qj Patient 4 pj qj Patient 5 pj qj Patient 6 pj qj Patient 7 pj qj Patient 8 pj qj Patient 9 pj qj Patient 10 pj qj Average pj qj The segmentation of four experts, Kaus method, and a novel algorithm (Fiedler based) are used as input for simultaneous truth and performance level estimation. pj represent the true-positive fraction or sensitivity, while qj represent the true-negative fraction or specificity. 5. a software system that incorporates the evaluation methods and the standard segmentation algorithms. Our newly introduced framework addresses all the five points, for the specific problem of brain tumor segmentation on MR images. A unique repository with brain tumor T1w MR images is made available. We have proposed a benchmark for the validation of novel brain tumor images. To the best of our knowledge, we are the first to have created such a framework. We also demonstrated that the proposed framework can be used to validate novel segmentation algorithms. 4 Figure 3. Results of the novel segmentation algorithm and comparisons with existent segmentation data in the proposed database for the Patient 4. (a) Tumor region. (b) The four experts segmentation, (c) Kaus result, (d) the novel method (Fiedler based) result, and (e) the ground truth estimated by simultaneous truth and performance level estimation. 1249

9 ARCHIP ET AL Academic Radiology, Vol 14, No 10, October 2007 Figure 4. The performance of the new segmentation method (Fiedler-based), and comparison with existent segmentation data. (a) Sensitivity or true-positive fraction (p j ), and (b) specificity or false-positive fraction (q j ). A new algorithm for brain tumor segmentation is also introduced. Its validation is demonstrated based on the new framework. Its specificity (true negative fraction) is close to 1 for most of the cases. It indicates that the Fielder Eigenvector based algorithm is not producing undersegmentations for brain tumors. On the other hand, as indicated by the sensitivity (true positive fraction), it creates often oversegmentations. This aspect can be noticed also on Fig

10 Academic Radiology, Vol 14, No 10, October 2007 A VALIDATION FRAMEWORK FOR BRAIN TUMOR SEGMENTATION Contributions from the research community are welcomed, both in terms of data (images and segmentations) and for brain tumor segmentation software tools. Several clinical (including image-guided therapies) and drug discovery applications require accurate segmentation algorithms. A common strategy for validation of these technological developments is needed, and yet does not exist. Therefore we anticipate an important impact in the medical imaging research community of our proposed framework. REFERENCES 1. JE Cates, Whitaker RT, Jones GM. Case study: an evaluation of userassisted hierarchical watershed segmentation. Med Image Anal 2005; 9: Cuadra MB, Pollo C, Bardera A, et al. Atlas-based segmentation of pathological MR brain images using a model of lesion growth. IEEE Trans Med Imaging 2004; 23: Droske M, Meyer B, Rumpf M, et al. An adaptive level set method for interactive segmentation of intracranial tumors. Neurol Res 2005; 27: Kaus MR, Warfield SK, Nabavi A, et al. Automated segmentation of MR images of brain tumors. Radiology 2001; 218: Letteboer MM, Olsen OF, Dam EB, et al. Segmentation of tumors in magnetic resonance brain images using an interactive multiscale watershed algorithm. Acad Radiol 2004; 11: Mazzara GP, Velthuizen RP, Pearlman JL, et al. Brain tumor target volume determination for radiation treatment planning through automated MRI segmentation. Int J Radiat Oncol Biol Phys 2004; 59: Vaidyanathan M, Clarke LP, Hall LO, et al. Monitoring brain tumor response to therapy using mri segmentation. Magn Res Imaging 1997; 15: Xie K, Yang J, Zhang ZG, et al. Semi-automated brain tumor and edema segmentation using MRI. Eur J Radiol 2005; 56:1. 9. Warfield SK, Zou KH, Wells WM. Simultaneous truth and performance level estimation (staple): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 2004; 23: Shi J, Malik J. Normalized cuts and image segmentation. IEEE Trans Pattern Anal Machine Intell 2000; 22: Meila M, Shi J. Learning segmentation by random walks. Proceedings of Neural Inform Proc Sys 2000; Archip N, Erard PJ, Egmont-Petersen M, et al. A knowledge-based approach for automatic detection of spinal cord in CT images. IEEE Trans Med Imaging 2002; 21: Archip N, Rohling R, Cooperberg P, et al. Ultrasound image segmentation based on spectral clustering techniques. Ultrasound Med Biol 2005; 31: Udupa JK, Schmidt H, Leblanc VR, et al. A framework for evaluating image segmentation algorithms. Comp Med Imaging Graph 2006; 30: