DEEP LEARNING FOR BRAIN TUMOR SEGMENTATION MARC MORENO LOPEZ. B.S., Polytechnical University of Catalonia, Barcelona, 2015

Transcription

1 DEEP LEARNING FOR BRAIN TUMOR SEGMENTATION by MARC MORENO LOPEZ B.S., Polytechnical University of Catalonia, Barcelona, 2015 A thesis submitted to the Graduate Faculty of the University of Colorado Colorado Springs in partial fulfillment of the requirements for the degree of Master of Science Department of Computer Science 2017

2 This thesis for the Master of Science degree by Marc Moreno Lopez has been approved for the Department of Computer Science by Jonathan Ventura, Chair Terry Boult Jugal Kalita Date: ii

3 Moreno Lopez, Marc (M.S. Computer Science) Deep Learning For Brain Tumor Segmentation Thesis directed by Assistant Professor Jonathan Ventura ABSTRACT In this work, we present a novel method to segment brain tumors using deep learning. An accurate brain tumor segmentation is key for a patient to get the right treatment and for the doctor who must perform surgery. Due to the genetic differences that exist in different patients, even between the same kind of tumor, an accurate segmentation is crucial. To beat state-of-the-art methods, we want to use technology that has provided major breakthroughs in many different areas, including segmentation, deep learning, a new area of machine learning. It is a branch of machine learning that is attempting to model high level abstractions in data. We will be using Convolutional Neural Networks, CNNs, and we will evaluate the results that we obtain comparing our method against the best results obtained from the Brain Tumor Segmentation Challenge, BRATS. iii

4 DEDICATION To the two most important persons in my life: my father and my sister. Your support and love has been essential for me during all these years. Without you two I wouldn t have gone this far in life. One special dedication to my mother: my source of inspiration, my Guardian Angel and my motivation. Wherever you are I hope you are proud of me and of what I have become. And finally, to Ashley Morgan, my best friend, my love and the person with whom I want to share the rest of my life. I cannot thank you enough for all your support during these months and for always believing in me. iv

5 ACKNOWLEDGEMENTS First, a professional and personal acknowledgement to Dr. Jonathan Ventura from UCCS, whose guidance has been essential for me to achieve all my goals during my masters. I would like to thank him for believing in me. I would also like to thank him for giving me the opportunity to work on such a fascinating project and for all his support. I would also like to acknowledge Pete Balsells and his foundation, whose financial help has been key for me to complete my studies. A special thanks to Eva Vidal, whose wise advice helped me find my motivation and the strength to set forth on this adventure. And finally, thanks to all those teachers at UCCS who contributed to my education. Special mention to Dr. Jugal Kalita, whom I worked with this summer and I appreciate for giving me the opportunity to work with him. v

6 TABLE OF CONTENTS CHAPTER I. Introduction... 1 II. State of the art... 4 III. Background Genetics of cancer Database Convolutional Neural Network Convolutional layer Pooling Fully Connected Layer Fully Convolutional Network Dilation model MatConvNet IV. Project development Image patching Working with the whole image Binary classifier Elastic image deformation D model vi

7 V. Results VI. Conclusions VII. Future work References Appendices vii

8 LIST OF FIGURES FIGURE 1. Tumor-cut algorithm from Hamamci et al. [7], BRATS Random Forest Segmentation method from Zikic [13], BRATS Best BRATS results up-to-date [16] Segmentation using Shin regression model [17] Different modalities of MRI MRI with its correspondent label MRI for Real HG (left) and simulated HG (right) CNN general structure operator Fully Convolutional Network structure , 2 and 4 dilated convolutions Brain MRI with tumor and Gaussian probability density function Original MRI (left) and deformed MRI (right) D model user interface D model of the brain Net train Prediction results for the network Layout of the Unet MRI of a breast tumor (left) and MRI of a lung tumor (right) viii

9 LIST OF TABLES TABLE 1. Former configuration of the CNN Actual configuration of the CNN Precision Whole tumor Sensitivity Whole tumor Specificity Whole tumor Dice score Whole tumor Precision Tumor core Sensitivity Tumor core Specificity Tumor core Dice score Tumor core Best deep learning methods up-to-date Results from BRATS survey paper [16] ix

10 GLOSSARY BRATS: Brain Tumor Segmentation Challenge CNN: Convolutional Neural Network MLP: Multi-Layer Perceptron MNIST: Mixed National Institute of Standards and Technology MRI: Magnetic Resonance Imaging : Rectified Linear Unit WHO: World Health Organization x

11 CHAPTER I INTRODUCTION Cancer is one of the leading causes of death in the world. In the US, cancer is the 2 nd leading cause exceeded only by heart disease [1]. To put it in perspective, one out of every four deaths in the US is caused by cancer. Due to this high-death ratio, scientist all over the world have tried to find a cure for cancer. In this work, my intention is to find a faster and more efficient way to detect cancer in time. According to the National Cancer Institute [2], checking for cancer (or for conditions that may become cancer) in people who have no symptoms is called screening. Screening can help doctors find and treat several types of cancer early. Early detection is important because when abnormal tissue or cancer is found early, it may be easier to treat. By the time symptoms appear, cancer may have begun to spread and is harder to treat. Several screening tests have been shown to detect cancer early and to reduce the chance of dying from that cancer. But it is important to keep in mind that screening tests can have potential harms as well as benefits. Some screening tests may cause bleeding or other health problems. Screening tests can have false-positive results the test indicates that cancer may be present even though it is not. False-positive test results can cause anxiety and are usually followed by additional tests and procedures that also have potential harms. Screening tests can have false-negative results the test indicates that cancer is not present even though it is. False-negative test results may provide 1

12 false reassurance, leading to delays in diagnosis and possibly causing an individual to put off seeking medical care even if symptoms develop. Overdiagnosis is possible. This happens when a screening test correctly shows that a person has cancer, but the cancer is slow growing and would not have harmed that person in his or her lifetime. Treatment of such cancers is called overtreatment. In this work, we want to combine a screening method, such as Magnetic Resonance Imaging, MRI, with the latest technology on machine learning, deep learning. Deep Learning is a new area of Machine Learning research, which has been introduced with the objective of moving Machine Learning closer to one of its original goals: Artificial Intelligence. There have been some approaches using this method [3] [4], but they haven t presented an approach that can beat human performance. A correct segmentation is key for many reasons. The most important one is so that the patient can get the best possible treatment. An accurate tumor quantification is needed so that the patient gets the amount of treatment that he needs. A rightful segmentation is crucial too in life-threatening cases. These are cases where the tumor is next to or on top of one of the cerebellum, or to similar sensitive parts. Therefore, we need to do a correct segmentation, especially in the boundaries between tumor and edema. This last part is central when planning a brain tumor extraction. Doctors need to know what they are facing before performing any surgery. With all these questions in mind, we ask ourselves the following questions: 2

13 - Can we improve the state-of-the-art algorithms results with Convolutional Neural Networks? o Deep learning methods present an improvement over non-deep learning methods. - How can we deal with the scarcity of data? o Data augmentation by elastic deformation models can improve the performance even further and help us deal with the lack of data. - How do we introduce context in the network without losing output spatial resolution? o Dilated convolution is effective for brain tumor segmentation to introduce context without losing output spatial resolution. 3

14 CHAPTER II STATE OF THE ART The number of publications devoted to automated brain tumor segmentation has grown exponentially in the last decades. This fact only emphasizes the need to automate the process, and shows that research in that area is still a work in progress. Along with the advance of medical imaging, imaging modalities play an important role in the evaluation of patients with brain tumors and have a significant impact on patient care. Recent years, the emerging new imaging modalities, such as X-Ray, Ultrasonography, Computed Tomography (CT), Magneto Encephalo Graphy (MEG), Electro Encephalo Graphy (EEG), Positron Emission Tomography (PET), Single-Photon Emission Computed Tomography (SPECT), and Magnetic Resonance Imaging (MRI), not only show the detailed and complete aspects of brain tumors, but also help clinical doctors to study the mechanism of brain tumors to improve treatment. In the work by Liu et al. [5], they make a survey of state-of-the-art MRI-based brain tumor segmentation methods. They classify these methods into three categories: manual, semi-automatic, and fully automatic segmentations, based on the degree of required human interaction [6]. For semi-automatic brain tumor segmentation, it mainly consists of the user, interaction, and software computing. In the semi-automatic brain tumor methods, the user needs to input some parameters and is responsible for analyzing the visual information and providing feedback response for the software computing. 4

15 The software computing is targeted at the realization of brain tumor segmentation algorithms. The interaction oversees adjusting segmentation information between the user and the software computing. The semi-automatic brain tumor segmentation methods were divided into three main processes: initialization, feedback response, and evaluation. Although brain tumor semi-automatic segmentation methods can obtain better results than manual segmentation, it also comes into being different results from different experts or the same user at different times. Hence, fully automatic brain tumor segmentation methods were proposed. Figure 1. Tumor-cut algorithm from Hamamci et al. [7], BRATS 2012 For fully automatic brain tumor segmentation, the computer determines the segmentation of brain tumor without any human interaction. In general, a fully 5

16 automatic segmentation algorithm combines artificial intelligence and prior knowledge. With the development of machine learning algorithms that can simulate the intelligence of humans to learn effectively, the study of fully automatic brain tumor segmentations has become a popular research issue. The semi-automatic and fully automatic segmentation of tumor brain images are faced with great challenges due to usually exhibiting unclear and irregular boundaries with discontinuities and partial-volume effects for brain tumor images. They divide the current MRI-based brain tumor segmentation methods into three major categories: conventional methods, classification and clustering methods, and deformable model methods. Conventional brain tumor segmentation methods mainly include the use of standard image processing methods such as threshold-based methods [8] and region-based [9] methods. Threshold-based and region-based methods are commonly employed in two-dimensional image segmentation [10]. Classification and clustering methods are composed mainly by machine learning algorithms [11][12]. Machine learning provides an effective way to automate the analysis and diagnosis for medical images. It can potentially reduce the burden on radiologists in the practice of radiology, which can learn complex relationships or patterns from empirical data and make accurate decisions. Machine learning algorithms can be organized into different categories based on different principles. This method is classified into supervised learning, semi-supervised learning, and unsupervised learning algorithms based on the utilization of labels of training samples. 6

17 In fact, most of brain tumor segmentation algorithms are based on classification or clustering methods in the literature such as Fuzzy C-Means (FCM), k-means, Markov Random Fields (MRF), Bayes, Artificial Neural Networks (ANN), Support Vector Machines (SVM), Atlas-based, etc. Figure 2. Random Forest Segmentation method from Zikic [13], BRATS 2012 Due to the appearance of 3-D MRI data, the segmentation of these data has become a challenging problem [14] [15]. The challenge is to extract boundary elements belonging to the same structure and integrate these elements into a coherent and consistent model of the structure. Therefore, model-based segmentation methods including parametric and geometric deformable models were proposed to improve this problem. The widely-recognized potency of deformable models stems from their ability to segment images of anatomic structures by exploiting constraints derived from the image data together with a priori knowledge about the location, size, and shape of these structures. Deformable models can accommodate the often-significant variability of biological structures over time and across different individuals. Furthermore, deformable 7

18 models support highly intuitive interaction mechanisms that allow medical researchers and clinicians to bring their expertise to bear on the model-based image interpretation task when necessary. Figure 3. Best BRATS results up-to-date [16] Figure 4. Segmentation using Shin regression model [17] 8

19 CHAPTER III BACKGROUND 3.1. Genetics of cancer Cancer is a genetic disease [18]. That means that cancer is caused by certain changes to genes that control the way our cells function. These changes include mutations in the DNA that makes up our genes. Basically, cancer begins in the cells, which are the building blocks and foundation of the body. Normally, the human body forms new cells as it needs them, replacing old cells that die. However, this process can go wrong. New cells grow even when the body doesn t need them, and old cells don't die when they should. These extra cells can form a mass, which is called tumor. Tumors have two different typologies, benign and malignant, and their aggressiveness to the body differs immensely from one type to the other. Benign tumors aren't cancer while malignant ones are. Usually benign tumors follow an easy procedure to be extracted from the body and in some cases where they aren t life-threatening, there is no need to perform any surgery. On the other hand, malignant tumors are more difficult to treat and in some cases cells from malignant tumors can invade nearby tissues. They can also break away and spread to other parts of the body. Genetic changes that increase cancer risk can be inherited from our parents if the changes are present in germ cells, which are the reproductive cells of the body (eggs and sperm). Such changes, called germline changes, are found in every cell of the offspring. 9

20 Cancer-causing genetic changes can also be acquired during one s lifetime, as the result of errors that occur as cells divide during a person s lifetime or exposure to substances, such as certain chemicals in tobacco smoke, and radiation, such as ultraviolet rays from the sun, that damage DNA. Genetic changes that occur after conception are called somatic (or acquired) changes. They can arise at any time during a person s life. The number of cells in the body that carry such changes depends on when the changes occur during a person s lifetime. In general, cancer cells have more genetic changes than normal cells. But each person s cancer has a unique combination of genetic alterations. Some of these changes may be the result of cancer, rather than the cause. As the cancer continues to grow, additional changes will occur. Even within the same tumor, cancer cells may have different genetic changes. Cancer is not just one disease but many diseases. There are more than 100 different types of cancer. Most cancers are named for where they start. The spread of cancer from one part of the body to another is called metastasis. Symptoms and treatment depend on the cancer type and how advanced it is. Most treatment plans may include surgery, radiation and/or chemotherapy. Some may involve hormone therapy, immunotherapy or other types of biologic therapy, or stem cell transplantation. The medical name for a brain tumor is a glioma. A glioma is a type of tumor that starts in the brain or spine. It is called a glioma because it arises from glial 10

21 cells. Gliomas make up about 30% of all brain and central nervous system tumors and 80% of all malignant brain tumors. Of numerous grading systems in use, the most common is the World Health Organization (WHO) grading system for astrocytoma, under which tumors are graded from I (least advanced disease best prognosis) to IV (most advanced disease worst prognosis), based on observances made under the microscope. In general, grade I and grade II are benign brain tumor; grade III and grade IV are malignant brain tumor. Using this grading system, which is determined by the pathologic evolution of the tumor, gliomas can be classified into the following: - Low-grade gliomas (WHO grade II) are well-differentiated; these tend to exhibit benign tendencies and foreshadow a better prognosis for the patient. However, they have a uniform rate of recurrence and increase in grade over time, so can end up being classified as malignant. - High-grade gliomas (WHO grade III-IV) are undifferentiated; these are malignant and carry a worse prognosis Database One of the main challenges of this project was to find a reliable database that was fully labeled. Medical images are hard to find due to privacy issues. Thankfully we could find the BRATS [19] database from BRATS is a challenge that started in 2012 and has been done every year since. Because of their unpredictable appearance and shape, segmenting brain tumors from multi-modal imaging data is one of the most challenging tasks in medical image analysis. Although many different segmentation strategies have 11

22 been proposed in the literature, it is hard to compare existing methods because the validation datasets that are used differ widely in terms of input data (structural MR contrasts; perfusion or diffusion data;...), the type of lesion (primary or secondary tumors; solid or infiltratively growing), and the state of the disease (preor post-treatment). - T1-weighted MRI: image contrast is based predominantly on the T1 (longitudinal) relaxation time of tissue; tissue with short T1 relaxation time appears brighter (hyper intense). - T2-weighted MRI: image contrast is based predominantly on the T2 (transverse) relaxation time of tissue; tissue with long T2 relaxation time appears brighter (hyper intense). - FLAIR images: Fluid-Attenuated Inversion-Recovery MRI: bright signal of the CSF (cerebrospinal fluid) is suppressed which allows a better detection of small hyperintense lesions. - T1-weighted MRI after administration of contrast media: many tumors show signal enhancement after administration of contrast agent. 12

23 Figure 5. Different modalities of MRI The real data consists of multi-contrast MR scans of 30 glioma patients (both low-grade and high-grade, and both with and without resection) along with expert annotations for "active tumor" and "edema". For each patient, T1, T2, FLAIR, and post-gadolinium T1 MR images are available. All volumes were linearly co-registered to the T1 contrast image, skull stripped, and interpolated to 1mm isotropic resolution. No attempt was made to put the individual patients in a common reference space. Patients with high- and low-grade gliomas have file names "BRATS_HG" and "BRATS_LG", respectively. All images are stored as signed 16-bit integers, but only positive values are used. The manual segmentations (file names ending in "_truth.mha") have only three intensity levels: 13

24 - 0, for normal tissue/empty pixel - 1, for edema - 2, for active tumor When referring to normal tissue, both grey and white matter are included in this section. In figure 6 we can appreciate how a label is displayed using imshow(). White is for the active tumor; grey is for edema and black is for normal tissue or background. Figure 6. MRI with its correspondent label Edema is the medical term for swelling. Body parts swell from injury or inflammation. It can affect a small area or the entire body. Edema happens when your small blood vessels become "leaky" and release fluid into nearby tissues. That extra fluid builds up, which makes the tissue swell. The data also contains simulated images for 25 high-grade and 25 lowgrade glioma subjects. These simulated images closely follow the conventions used for the real data, except that their file names start with "SimBRATS"; and their 14

25 MR scans and ground truth segmentations are stored using unsigned 16 bit and unsigned 8 bit integers, respectively. Even though this is one of the most widely used brain tumor datasets, when observing all the data we can perceive vast differences in between the real data and the simulated data, and between HG and LG. Due to the perfection of the simulated data, it is really easy to differentiate between them. Another big difference is that in simulated data you can clearly see every different kind of tissue without a problem. On the other hand, in the real data, it can be difficult to differentiate between the different tissues Figure 7. MRI for Real HG (left) and simulated HG (right) 3.3. Convolutional Neural Network Convolutional Neural Networks (ConvNets or CNNs) are a subtype of Neural Network that have provided major breakthroughs in many areas regarding image recognition and classification [20]. LeNet was one of the very first convolutional neural networks which helped propel the field of Deep Learning. This pioneering work by Yann LeCun [21] was 15

26 named LeNet5 after many previous successful iterations since the year There have been several new architectures proposed in the recent years which are improvements over the LeNet, but they all use the main concepts from the LeNet. In the years from late 1990s to early 2010s convolutional neural network were in incubation. As more and more data and computing power became available, tasks that convolutional neural networks could tackle became more and more interesting. In 2012, AlexNet [22] paper was released. This new method was a huge step forward and made the scientific community aware of CNN potential. Another occurrence that contributed to the impact of CNN was the availability of millions of images. Figure 8. CNN general structure Before getting more into detail, I would like to define some key concepts related to neural networks: 16

27 - An artificial neuron is a mathematical function conceived as a model of biological neurons. Artificial neurons are the constitutive units in a neural network. - The activation function of a node defines the output of that node given an input or set of inputs. - A multilayer perceptron or MLP, is a mathematical function mapping some set of input values to output values. The function is formed by composing many simpler functions. We can think of each application of a different mathematical function as providing a new representation of the input. - A layer is an operation performed in an input that is part of the network. - A weight is a parameter obtained from the layers. - An epoch is a training iteration over the dataset. - The learning rate is a positive scalar determining the size of the step for each training iteration. - The batch size defines number of samples that are going to be propagated through the network. For instance, let's say you have 1050 training samples and you want to set up batch_size equal to 100. Algorithm takes first 100 samples (from 1st to 100th) from the training dataset and trains the network. Next it takes second 100 samples (from 101st to 200th) and trains the network again. 17

28 Convolutional layer The primary purpose of Convolution in case of a CNN is to extract features from the input image. Convolution preserves the spatial relationship between pixels by learning image features using small squares of input data. A filter slides over the input image (convolution operation) to produce a feature map. The convolution of another filter, over the same image gives a different feature map. It is important to note that the Convolution operation captures the local dependencies in the original image. In practice, a CNN learns the values of these filters on its own during the training process (although we still need to specify parameters such as number of filters, filter size, architecture of the network etc. before the training process). The more filters we have, the more image features are extracted and the better our network becomes at recognizing patterns in unseen images stands for Rectified Linear Unit and is a non-linear operation. replaces all negative pixel values in the feature map by zero. The purpose of is to introduce non-linearity in CNNs, since most of the real-world data we want CNNs to learn is non-linear. The operator can be expressed in the following way: f(x) = max {0, x} 18

29 Figure 9. operator Pooling Spatial Pooling reduces the dimensionality of each feature map and, at the same time, retains the most important information. Spatial Pooling can be of different types: max, average, sum etc. A pooling function replaces the output of the net at a certain location with a summary statistic of the nearby outputs. When doing Max Pooling, we define a spatial neighborhood and take the largest element from the rectified feature map within that window. Instead of taking the largest element we could also take the average or sum of all elements in that window. In practice, the most common kind of Pooling used is Max Pooling since it has been shown to work better. The function of pooling is to progressively reduce the spatial size of the input representation. To be more exact, pooling: 19

30 - Makes the input representations (feature dimension) smaller and more manageable. - Reduces the number of parameters and computations in the network, therefore, controlling overfitting. - Makes the network invariant to small transformations, distortions and translations in the input image. - Helps us arrive at an almost scale invariant representation of the image, also referred as is equivariant. This is very powerful since we can detect objects in an image no matter where they are located - One of the main disadvantages of pooling is losing output spatial resolution Fully Connected Layer The Fully Connected Layer is a Multi-Layer Perceptron (MLP) that uses an activation function in the output layer. An activation function of a node defines the output of that node given an input or set of inputs. The most common activation function used is a softmax function/classifier. The term Fully Connected implies that every neuron in the previous layer is connected to every neuron on the next layer. The output from the convolutional and pooling layers represent high-level features of the input image. The purpose of the Fully Connected layer is to use these features for classifying the input image into various classes based on the training dataset. 20

31 3.4. Fully Convolutional Network Fully convolutional networks [23] or FCN, are a rich class of models, of which modern classification CNN are a special case. Fully convolutional indicates that the neural network is composed of convolutional layers without any fullyconnected layers or MLP, usually found at the end of the network. A CNN with fully connected layers is just as end-to-end learnable as a fully convolutional one. The main difference is that the fully convolutional net is learning filters everywhere. Even the decision-making layers at the end of the network are filters. A fully convolutional network tries to learn representations and make decisions based on local spatial input. Appending a fully connected layer enables the network to learn using global information where the spatial arrangement of the input falls away and doesn t need to apply. Figure 10. Fully Convolutional Network structure 21

32 3.5. Dilation model In [24], Yu et al. develop a new convolutional network module that is specifically designed for dense prediction. They present a model that uses dilated convolutions. This model is designed to systematically aggregate multi-scale contextual information without losing resolution. All their work is based on the fact that dilated convolutions support exponential expansion of the receptive field without losing resolution or coverage. Figure 11. 1, 2 and 4 dilated convolutions 3.6. MatConvNet According to the authors and extracted from their manual, MatConvNet [25] is an implementation of Convolutional Neural Networks (CNNs) for MATLAB. The toolbox is designed with an emphasis on simplicity and flexibility. It exposes the building blocks of CNNs as easy-to-use MATLAB functions, providing routines for computing linear convolutions with filter banks, feature pooling, and many more. In this manner, MatConvNet allows fast prototyping of new CNN architectures; at 22

33 the same time, it supports efficient computation on CPU and GPU allowing to train complex models on large datasets such as ImageNet ILSVRC. We decided to choose MatConvNet because it is easy to work with, it is used by a large community, thus many discussion forums offer solutions for a big variety of problems. 23

34 CHAPTER IV METHOD When this project started, we decided to look for many alternatives and not to stick to the conventional libraries like Caffe, Theano or Tensorflow. Instead, we looked for a library that was compatible with MATLAB but still offered many resources. In [26] they have many libraries and after doing some research we decided to choose Rasmus Berg Palm Deep Learn Toolbox [27]. We did many tests on this library but after a while we decided to stop working with it, since it hadn t been updated since and it lacked a support community where we could look for answers when we didn t understand something. After this first approach we decided to work with MatConvNet. It presented all the characteristics that we were looking for. It is simple, efficient, and can run and learn state-of-the-art CNNs. On top of that, it has a great community support and many forums with the most common troubles. Since we tried many different approaches using MatConvNet, I think it is convenient to create different sections for each one of them Image patching One of the first implementations was a CNN with a lenet architecture, similar to our baseline network, but with pooling. Instead of working with the whole image, like we would do in a FCN architecture, the CNN worked with multiple patches extracted from each image. We used the network structure that was set for the Mixed National Institute of Standards and Technology (MNIST) [28] experiment and adapted it to our dataset. 24

35 This strategy has two main drawbacks. First, it is quite slow because the network must be run separately for each patch, and there is a lot of redundancy due to overlapping patches. Secondly, there is a trade-off between localization accuracy and the use of context. Larger patches require more max-pooling layers that reduce the localization accuracy, while small patches allow the network to see only little context. Since we didn t obtain good results we decided to start using the whole image instead of extracting patches from the image Working with the whole image After obtaining very deceiving results, we decided to change the network configuration and work with the whole image instead of with image patches. Layer # Type Configuration 1 Convolutional 5x5x1x18 2 Batch Normalization 3 4 Convolutional 5x5x8x16 5 Batch Normalization 6 7 Convolutional 5x5x16x32 8 Batch Normalization 9 10 Convolutional 1x1x32x2 Table 1. Former configuration of the CNN 25

36 Layer # Type Configuration Dilation 1 Convolutional 5x5x1x8 1 2 Batch Normalization 3 4 Convolutional 5x5x8x8 1 5 Batch Normalization 6 7 Convolutional 5x5x8x Batch Normalization 9 10 Convolutional 5x5x16x Batch Normalization Convolutional 5x5x32x Batch Normalization Convolutional 5x5x64x Convolutional 5x5x128x Convolutional 1x1x128x2 1 Table 2. Actual configuration of the CNN Binary classifier A binary classifier classifies the elements of a given set into two groups based on a classification rule. The classifier, basically, discerns if an element from the set has or not a qualitative property. To test the dataset with the binary classifier, we decided to consider 1 and 2 labels, edema and tumor respectively, in the same category. Hence, all 0 labels were considered a -1 and all labels 1 and 2 were considered label 1. The main problem with this approach is that the weights were calculated in an inappropriate fashion, since we have way more 0 labels than 1 and 2. 26

37 L2 regularization To improve performance of the binary classifier, a regularization layer was added to the network. This was done mainly for the size of the dataset. Since the dataset is small, adding the regularization layer can help with the results. L2 refers to the sum of squares of all the weights in the network, multiplied by a constant and added to the loss function. However, the results didn t improve and stayed more less the same Avoiding empty pixels Since the binary classifier struggled a little bit, especially due to the vast number of empty pixels, we decided to try a new approach. This was done to avoid considering pixels whose value is 0 or almost 0. By doing this, we can ignore those pixels when calculating the weights. Thus, we have the following configuration. Pixels whose value in the image is less than 2, are considered as 0 or ignore. All pixels whose label is 0 are considered 1 or background and all pixels whose labels are 1 or 2 are contemplated as 2 or foreground Weight initialization vs. random One crucial aspect to take into account, is the weight initialization for each layer [29]. Until this moment in the project, we had used a random weight initialization drawn from Gaussian distributions. Nevertheless, we decided to use a weight initialization. With the weight initialization, we have a proper initialization method that avoids reducing or magnifying the magnitudes of input 27

38 signals exponentially. This means that if the initialization properly scales the backward signal, then this is also the case for the forward signal; and vice versa Elastic image deformation Data augmentation is essential to teach the network the desired invariance and robustness properties, when only few training samples are available. In case of biomedical images, we need a feasible deformation of the tumor and the edema without affecting the shape of the brain. To do so, we adopt one of the measures that Ronneberger et al. use in the Unet paper, random elastic deformations. Random elastic deformations of the training samples seem to be the key concept to train a segmentation network with very few annotated images. We generate smooth deformations using random displacement vectors on a coarse 3 by 3 grid. The displacements are sampled from a Gaussian distribution with 10 pixels standard deviation. Per-pixel displacements are then computed using bicubic interpolation [30]. One crucial aspect when doing elastic deformations is to take into consideration the correct deformation of edema together with the tumor. Another consideration to take into account is that the brain cannot be deformed, since it must always preserve the same shape. Therefore, we decided to combine the elastic deformations with a Gaussian probability density function that would allow us to do the deformations without doing any anatomical mistakes. The mean and the covariance for the Gaussian are calculated using the labels. We finally create the Gaussian using these parameters and a multivariate normal probability density 28

39 function. Once we have constructed the Gaussian, we multiply it per-pixel with the random deformation map. Figure 12. Brain MRI with tumor and Gaussian probability density function Figure 13. Original MRI (left) and deformed MRI (right) 29

40 4.4. 3D model One of the main outcomes of this project is to create a 3D model of the brain with the segmented tumor inside. To do so, we first need to construct a 3D model of a healthy brain. To create the 3D model, a different database is used. The database [31] consists of 60 vertical MRI slices that form an entire human brain, taken sequentially. These MRI images include other tissues besides brain tissue. Therefore, a thresholding technique is applied to isolate brain tissue from the background. Following an image morphology operator is applied to remove most of the skull and the peripheral tissue. Since we might still have some residual regions which don t belong to brain tissue, we must apply blob analysis and stay with the largest region. In brain analysis, it is often useful to distinguish between the two main types of brain tissue, grey and white matter. By labeling every pixel between 0 and 70 as grey matter, and all pixels above 70 as white matter, we get a good differentiation between the two kinds of matter. Finally, to create the 3D model, we use a MATLAB tool for visualization of different intensity level in 3D data. We give unique color to each type of brain tissue and describe the boundaries of each volume with vertex and patches. 30

41 Figure 14. 3D model user interface Figure 15. 3D model of the brain 31

42 CHAPTER V RESULTS To test all the algorithms, we used two distinctive measure. First, a dataset was created using the middle slice for every subject from the Simulated High Glaucoma dataset. We decided to use only the middle slice to avoid unbalancing the data even more. If the MRI of the subject contains a tumor, we saw that around a 23% of the slices of the MRI contained tumor tissue. However, many of them contained a small quantity of tumor pixels. That is the main reason why we decided only the middle slice. Therefore, the dataset consists of 25 images. Using only a small portion of the database also allowed us to obtain preliminary results on the different algorithms, mentioned in chapter 4, without wasting too many resources and time. Second, to measure the performance of the algorithms, we use precision, recall (or sensitivity), specificity and the dice score (also called F1) [32]. Precision (also called positive predictive value) is the fraction of retrieved instances that are relevant, while recall (also known as sensitivity) is the fraction of relevant instances that are retrieved, therefore it measures the proportion of positives that are correctly identified as such. Specificity measures the proportion of negatives that are correctly identified as such. The dice score or F1, is a is a statistic used for comparing the similarity of two samples. It is calculated using the precision and recall parameters. For all the tests, the dataset was always split in 70% training, 10% validation and 20% testing. We stopped using the Rasmus Berg algorithm, mentioned in chapter 4, algorithm because it hadn t been updated in a long time and it had no support or a 32

43 community where you could find solutions for implementation problems. After this we started using MatConvNet. The first approach that we used, based on image patching and a lenet architecture, didn t present good results and that is why we decided to change the architecture to a FCN. One of the main challenges when we started using the baseline FCN for MatConvNet in this project was to obtain a reasonable result. At the beginning, due to the use of a uniform weight initialization, the precision, recall and sensitivity that we obtained wouldn t go up from a 0.0. This was discouraging because it didn t matter how many changes we made to the network, the results wouldn t improve at all. Thankfully, after applying a correct weight initialization, as mentioned in chapter , we finally obtained a precision and recall different from 0.0 and we could scale up from there. The first result that we obtained was a precision of 0.1, recall of 0.2 and a sensitivity of The results weren t good, but at least it gave us a place to start from and now we could see how the network behaved in different situations. To improve the results, we decided to work in different stages. First of all, since there are 4 different MRI modalities we decided to see which one adjusted the best to the network. Following, we decided to modify the network setup (batch_size, num_epochs and learning rate). We played with these parameters, in a logical way, trying to obtain the best results for our small database. Another approach that we tried was using all the slices from only one subject. Therefore using 181 images. The results indicated that this wasn t a good 33

44 approach, since we obtained a precision of 0.04, a recall of 0.44 and a sensitivity was almost 0. This approach didn t present good results for two main reasons. First, the number of slices containing tumor or edema are very few compared to the slices where all we have is brain and background. The second reason is that using only one of the subjects causes overtraining, since the tumor is always on the same side of the brain and it always has the same shape. As for the results where we try to avoid reading pixels whose value is 0 or close to 0, we followed a similar method to the one explained above, trying to find the most suitable group of images and parameters. After changing the weight initialization from normal/random to weight initialization, the same procedure was followed. Another approach that we tried for this weight initialization was to create a new database consisting of all the middle slices, a total of 56 for each subject, for 4 different subjects. Therefore, the new dataset consists of a total of 224 images. The results are similar to the ones that we were obtaining with the 25 images dataset (0.2 precision, 0.7 recall). Hence, we decided to keep working with the 25 images dataset, since it is faster to work with and gives us a good overview of how the net is working. After the proposal, we started using the elastic deformation model to train the network. We incorporated the deformation model in the training and validation datasets, leaving the testing images non-deformed. Every single time the network does a new epoch, the network is trained using a new batch of deformed images. Therefore, the network never sees the same image twice. We saw that the results improved, but it wasn t the improvement that we were expecting. After reviewing 34

45 the code, we saw a small error that was causing our network not to perform at its fullest capacity. Instead of creating the data as a singles type, we created it as an unsigned integer (uint8). After changing this, the network behavior improved a lot and so did our results. With this FCN configuration and the elastic image deformation model, the best results that we obtained were a precision of 0.66, a recall of 0.96, a specificity of 0.95 and a dice score of With this score, we saw that we were getting closer to human performance, which is a 0.88 dice score. Finally, we decided to make one last modification to our new network architecture using the dilation model. This model allowed us to improve even further and obtain the best results that we could obtain. Since this were our best results, we decided to test all the different parts of the dataset. We also decided to see how much of an improvement we got using the elastic image deformation model. All these tests were done using k-fold cross-validation, using k=5 for simulated images and k=4 for real images. All the test images are non-deformed images. The first four tables correspond to the whole tumor results and the four last tables correspond to the tumor core results. Type MRI HG Data Dilation Precision modality /LG Augmentation Simulated FLAIR HG No Yes Simulated FLAIR HG Yes Yes Simulated FLAIR HG No No Simulated FLAIR HG Yes No Simulated FLAIR LG No Yes Simulated FLAIR LG Yes Yes Real T2 HG No Yes Real T2 HG Yes Yes Table 3. Precision Whole tumor 35

46 Type MRI HG/ Data Dilation Recall / modality LG Augmentation Sensitivity Simulated FLAIR HG No Yes Simulated FLAIR HG Yes Yes Simulated FLAIR HG No No Simulated FLAIR HG Yes No Simulated FLAIR LG No Yes Simulated FLAIR LG Yes Yes Real T2 HG No Yes Real T2 HG Yes Yes Table 4. Sensitivity Whole tumor Type MRI modality HG Data /LG Augmentation Dilation Specificity Simulated FLAIR HG No Yes Simulated FLAIR HG Yes Yes Simulated FLAIR HG No No Simulated FLAIR HG Yes No Simulated FLAIR LG No Yes Simulated FLAIR LG Yes Yes Real T2 HG No Yes Real T2 HG Yes Yes Table 5. Specificity Whole tumor Type MRI modality HG/ Data LG Augmentation Dilation Dice Simulated FLAIR HG No Yes Simulated FLAIR HG Yes Yes Simulated FLAIR HG No No Simulated FLAIR HG Yes No Simulated FLAIR LG No Yes Simulated FLAIR LG Yes Yes Real T2 HG No Yes Real T2 HG Yes Yes Table 6. Dice score Whole tumor 36

47 Type MRI modality HG/ Data LG Augmentation Dilation Precision Simulated FLAIR HG Yes Yes Simulated FLAIR HG Yes No Simulated FLAIR LG Yes Yes Real T2 HG Yes Yes Table 7. Precision Tumor core Type MRI modality HG/ Data LG Augmentation Dilation Recall/ Sensitivity Simulated FLAIR HG Yes Yes Simulated FLAIR HG Yes No Simulated FLAIR LG Yes Yes Real T2 HG Yes Yes Table 8. Sensitivity Tumor core Type MRI modality HG/ Data LG Augmentation Dilation Specificity Simulated FLAIR HG Yes Yes Simulated FLAIR HG Yes No Simulated FLAIR LG Yes Yes Real T2 HG Yes Yes Table 9. Specificity Tumor core Type MRI modality HG/ Data LG Augmentation Dilation Dice Simulated FLAIR HG Yes Yes Simulated FLAIR HG Yes No Simulated FLAIR LG Yes Yes Real T2 HG Yes Yes Table 10. Dice score Tumor core 37

48 The results tables demonstrate that the final configuration of the network is the best one so far and that it improves the previous experiments. We have highlighted in bold the best results in each of the cases. In 3 out of the 4 tables, the best result corresponds to the dilation experiment with data augmentation In the next figure, we have included some of the segmented images that we obtained and the learning curve for a training/validation process. In the training figure, in the x axis in both graphs, we have the number of epochs. In the left graph, we are representing the accuracy which can go from 0 to 1. We want to get as close to 1 as possible. In the right graph, we are representing the error that we obtain, which can go from 0 to 1. We want to get as close to 0 as possible. 0.9 accuracy 0.9 objective train val train val epoch epoch Figure 16. Net train 38

49 Figure 17. Prediction results for the network Now we will proceed to compare our results against the state-of-the-art algorithms, to see where we stand. Note however, that we cannot compare to them directly since these algorithms have been evaluates using the BRATS online evaluation tool or using the on-site challenge database, which consists of 11 HG real patients, 4 LG real patients, 10 HG simulated patients and 5 LG simulated patients. 39

50 Author Model Level of user Performance (dice score) interaction Whole tumor Tumor core Human Rater Medical training and Manual experience Pereira et CNN with small filter Fully al. [4] for deeper automatic Kwon et al. [34] Havaei et al. [35] architecture Generative model that performs joint segmentation and registration Cascaded Twopathway CNNs for simultaneous local and global processing Semiautomatic Fully automatic Table 11. Best deep learning methods up-to-date These results have been extracted from [33]. They have been obtained using the challenge dataset of BRATS 2013 benchmark and all of them use deep learning methods. In the survey paper, the authors don t specify if the results are global or if they are just for a portion of the dataset. Whole Core Dice (in%) LG/HG LG/HG Zhao et al. [36] 82 78/ /68 Menze et al. [37] 78 81/ /59 Guo et al. [38] 74 71/ /67 Subbanna et al. [39] 75 55/ /75 Table 12. Results from BRATS survey paper [16] These results have been extracted from [16] and all of them use non-deep learning methods. Again, in the survey paper the authors don t specify how these results are obtained. We don t know if they are the best results for each algorithm, for simulated data, real data or a combination of both. 40

51 Comparing our results to both tables, and having in consideration that we couldn t evaluate our network in the same way, we believe that our network performs better than the non-deep learning algorithms except for Zhao et al., which seems to have a better performance than our network. However, we still must improve to reach the performance of deep learning methods. 41

52 CHAPTER VI CONCLUSIONS Results are encouraging and prove that we are on the correct path. We have a great performance in comparison to all the machine learning methods used in past editions of BRATS. When we started this project in 2015, we wanted to beat the state-of-the-art methods at that moment, which didn t include any deep learning methods. After finishing this project, we think our network could beat all the existing methods when we started the project. We cannot confirm this since the evaluation methods are different. When comparing our method to other deep learning methods, we see that we still have some improvement. We also need a big improvement until we reach human performance. Lack of medical data is one of the main problems that we have encountered in this project. To publish medical data, the three parts involved, doctor, institution and patient, must agree to publish it. Therefore, if you aren t working with a medical institution, it s difficult to obtain lots of medical data or even trustworthy data. However, with our data augmentation model using elastic image deformation, we overcome this scarcity of data. Finding the right network configuration to work with such a small amount of data was an enormous challenge. We went through many different networks and different configurations to try to find the most suitable network for our needs. The dilated convolution is effective for brain tumor segmentation to introduce context without losing output spatial resolution and it is an interesting direction to explore in the future. 42

53 Our network works better with simulated than with real images. This relates to the work that other people have done and it makes a lot of sense. Simulated data is uniform in size, conditions and shape. However, real images have many differences between them. This can be caused due to many reasons, like the patient moving slightly while taking the MRI. Brain tumor segmentation isn t an easy task. Due to the genetics of cancer, it remains being a task for which the doctor s help is needed. However, they don t have to do all by themselves, since with tools like the one that we have designed in this work, we can help them and we can contribute in the fight against cancer. 43

54 CHAPTER VII FUTURE WORK One of the main directions to explore to improve the results is trying new network architectures. One of the best CNN implementations for medical images so far has been Unet [29]. In this paper, Ronneberger et al. design a CNN shaped like a U, with two different paths, a contracting path (left side) and an expansive path (right side). For the contracting path, they use the repeated application of two 3x3 convolutions, followed by a operator and a 2x2 max pooling operator with a stride 2 for downsampling. Every time they downsample, the number of feature channels is doubled. On the other hand, for the expansive path, they apply an upsampling to the feature map prior to applying a 2x2 convolution that reduces by 2 the number of channels. Then a concatenation with the correspondingly cropped feature map is applied, followed by two 3x3 convolutions, each with their own operator. In the final layer a 1x1 convolution is applied to map the 64- component feature vector to the desired number of classes. In total, the network has 23 convolution layers. In Figure 18 we can see the layout of the Unet. 44

55 Figure 18. Layout of the Unet Another part of the future work is to adapt the algorithm so that it can detect other kinds of cancer beside brain tumors. Usually, brain tumors tend to form big clumps. Therefore, we need to look for a distinct kind of cancer that acts similar. The most similar cases are breast cancer and lung cancer since they have similar behavior to brain cancer. 45

56 Figure 19. MRI of a breast tumor (left) and MRI of a lung tumor (right) Another direction to explore is new methods of data augmentation. As we have stated during this thesis, incrementing the number of images is key. Finding a new reliable method to perform this is a must in the future. More complex approaches include a new network trained to create small deformations or working in a mathematical model for brain tumor deformations. 46