AALBORG UNIVERSITY, SEMCON, 21. DECEMBER

Transcription

1 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER Automatic algorithm for segmentation, registration, and fusion of digital fundus retinal images from patients suffering from diabetic retinopathy Mads Bo Christensen, Christian Bork Hardahl, Michael Munk Jakobsen, Toke Sonnenburg Ottesen, and Sille Petersen, Group 706d, Aalborg University Abstract Diabetic retinopathy is a complication of diabetes mellitus and is diagnosed from fundus retinal images. To evaluate the degree of retinopathy the specialist often receives two or more separate images of the retina. Clinical evaluation is done by hand and can be very time-consuming partly because of large overlaps between the images. [1] To reduce the workload the images can be fused. We present an automatic algorithm that fuses two retinal images based on registration of three main features: the fovea, optic disc, and blood vessel bifurcations and crosses. Method - The fovea and the optic disc were used as an estimate for an initial transformation of two images. An affine transformation was estimated iteratively by including coordinate pairs found by template matching vessel bifurcations and crosses from one image to another. The final transformation was determined by finding the iteration with the shortest average distance between the matched points. Results - 21 image pairs were used for testing. The fovea was found correctly in 88% of the images and the optic disc was found correctly in 86% of the images. In average, 19 matching cross points were found correctly for each image pair. The transformation was found with an average error of 70,5 pixels. A Spearman rank correlation test was used to show a correlation between the matched points and the result of the image fusion. Discussion - The presented algorithm is able to register and fuse two overlapping retinal images. The study shows a correlation between the total matched points and a successful fusion. Further work is necessary to increase the accuracy of the fusion. To decrease the distortion in the fused image more matching points has to be identified in the overlap. Index Terms Diabetic retinopathy, Image fusion, Image registration, Ophthalmology I. INTRODUCTION DIABETIC RETINOPATHY is the most common reason to blindness in the western part of the world. [2] Diabetic retinopathy is a complication of diabetes mellitus and in the year 2000 more than 171 million people worldwide suffered from diabetes and this number is rapidly increasing. The continuously increasing prevalence of diabetes is predicted to affect more than 366 million people in the year [2] 20% of people suffering from type 2 diabetes already suffer from retinopathy when diagnosed with diabetes mellitus, and almost every person suffering from diabetes has retinopathy after 20 years of disease. [3] Patients suffering from diabetic retinopathy are referred to a specialist of ophthalmology, who can use different diagnostic tools to evaluate the degree of retinopathy. To evaluate the degree of retinopathy in clinical practice the specialist counts lesions in two or more images taken with different angles to view a larger area of the retina. This will cause partly overlapping areas of the retina images and there will be a risk of counting the same lesion several times in these areas. In addition, the clinical evaluation process is done by hand, which contributes to a large and time-consuming workload for the ophthalmologists. [1] To reduce the workload for the ophthalmologists an automatic algorithm for fusion of two or more retinal images is of potential interest. This paper focuses on the development of an automated algorithm that fuses two retinal images from patients suffering from diabetic retinopathy. This could improve the screening process and less resources could be used on each patient obtaining the same results as today. So far, no articles have been published that fully automates the ophthalmologists work. A lot of work has been done to identify objects located in the retina such as the fovea [4], the optic disc [5] [6], and blood vessel bifurcations [7] [8]. [9] Studies have been developed that detects lesions such as hard exudates, cotton wool spots, microaneurysms, and haemorrhages in patients suffering from diabetic retinopathy [10] [11] [12] [13]. Identification of such features on the retina is an incitement to fuse two or more retinal images. These features could be used as guiding points to make the fusion as precise as possible. To fuse two retinal images four different transformation types have been applied in other studies: translation, similarity, affine, and second-order polynomial transformation. In this paper the affine transformation has been applied and developed since Laliberte et al indicated that it was the best solution for transformation of retinal images. [1] This paper is organized as follows; Section II describes the methods and materials applied. The methods used are segmentation, registration, and fusion. Section III presents the results obtained, and section IV discusses the results followed by a conclusion. II. METHODS & MATERIALS In all, 48 retinal image pairs (tagged image format, TIF) were available for this study. The images were obtained from a 60 degrees fundus photography captured with a Canon CF-60UV fundus camera using Kodak Ectachrome 64 color diapositive film at the Department of Ophthalmology, Aarhus University Hospital. The images were 24-bit RGB images and

2 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER were obtained with a resolution of 3120x2080 pixels. Each retinal image pair was evaluated for pathological features and divided into three groups according to the degree of retinopathy; mild (24), moderate (8) and severe (12). No inclusion criteria were defined due to the fact that the study group did not collect the images themselves. However, five image pairs were excluded due to optical artifacts that affect the image processing. second filter used a mask of 801x801 in order to estimate the background intensity and preserve the blood vessels and the fovea. The result of the second filter is shown in figure 2B. The overall purpose of this study was to fuse two fundus retinal images. The first image was defined as a source image and the second as a destination image. The source image was transformed and fused with the destination image. To do this the following methods were applied: Preprocessing of the images included extracting the green color band from the RGB images. Segmentation of the main retinal features (optic disc, fovea, blood vessels, bifurcation and cross points) in order to get adequate points to fuse the source and destination image. Registration of corresponding points between the source and the destination image to estimate a transformation between the images. Fusion of the images by using an estimated transformation. A. Preprocessing The first step was to reduce the amount of information in the digitized image. The green band was used due to better contrast between the blood vessels, fovea, the optic disc and the image background, see figure 1B. Fig. 2. (A) shows the result of the 31x31 median filter. (B) shows the gray-scaled image after filtering by a 801x801 mask. (C) shows the result of subtracting (A) from (B). (D) shows the center of mass of the detected fovea in the original gray-scale image. The two images were subtracted to isolate fovea resulting in an image containing the fovea and other small artifacts, see figure 2C. The image was converted into a binary image using a dynamic threshold function and some of the smaller remaining artifacts were removed with an opening operation. The center of mass of the largest remaining object was marked as the fovea, see figure 2D. Fig. 1. (A) shows the original RGB fundus retinal image. (B) shows the green color band extracted from (A). B. Feature extraction 1) Segmentation of the fovea: To determine an initial transformation between the images two known points in each image were required. Therefore, the fovea and the optic disc were segmented. To isolate the fovea from the rest of the retinal image the first step was to enhance the difference between the fovea and the background, which was conducted by histogram equalization. The background and the blood vessels were removed using two different median filters. The first filter used a mask of 31x31 pixels to estimate the background intensity and preserve the blood vessels. The median filtered image was subtracted from the gray-scale image, see figure 2A. The 2) Segmentation of the optic disc: The optic disc was segmented in order to obtain the second point necessary for the initial transformation. To enhance the contrast between the optic disc and the rest of the image a 31x31 median filter was applied to the gray-scaled image. The optic disc represented by bright pixels was separated using an automated threshold to convert the image into a binary image. Artifacts derived from lesions in the retina were also converted due to bright pixels. Usually the optic disc was the largest circular area in the retina. A measurement of the circularity was used to identify the optic disc. To evaluate the circularity of the different objects three criteria were defined, see figure 3. comparing the maximum and minimum diameter of the object comparing the areas using the formula π r 2 comparing the circumferences using the formula 2 π r For each object two areas and circumferences were calculated from the maximum and minimum diameters of the object respectively, see figure 3 left. The proportion between the diameters, the areas, and the circumferences were calculated and averaged. The closer the average of the three criteria equaled one, the more circular were the objects. The measurements

3 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER Fig. 3. The figure shows an overview of the three applied criteria. with a further increased contrast between the vessels and the background was obtained. To ensure that all information in the image besides the blood vessels were excluded, a threshold was applied to convert the image into a binary image, see figure 5. Artifacts smaller than 2500 pixels were removed. By dilating with a disc structuring element with a radius of four pixels small discontinuities in the blood vessel tree were connected. of circularity of three different objects are illustrated in figure 4A. Fig. 5. Binary image of blood vessels in the source image. The blood vessels were thinned to a width of one pixel to ease the extraction of the blood vessel coordinates, see figure 6A. The thinning operation resulted in a blood vessel tree containing unwanted spurs. To remove the spurs the image was pruned, see figure 6B. The amount of pruning was determined by evaluating the length of the spurs in the training data. From the pruned image it was possible to extract the coordinates from the segmented blood vessel tree, which were used as input to the bifurcation and cross detection. Fig. 4. (A) shows the numbers found when evaluating the two criteria. (B) shows the original gray-scale image with the optic disc identified (green dot). The circularity of objects in the binary image were evaluated and the objects with a circularity less than 0.75 were removed. The threshold was estimated empirically from the training data. The optic disc was not always the most circular object as seen in figure 4A, but usually the largest of the remaining. The size of the remaining objects was calculated and the center of mass of the largest object was found. The center of the optic disc is shown in 4B. 3) Segmentation of the blood vessels: Blood vessels were segmented as a step toward identifying bifurcations and cross points in the vessel tree. These point were to be used in the registration process. The gray-scale retinal image was filtered with a 71x71 median filter producing a blurry version of the image to estimate the image background. By subtracting this image from the original gray-scale image, an image Fig. 6. Thinned blood vessels (A) and pruned blood vessels (B) 4) Identification of blood vessel bifurcations and crosses: After extracting the blood vessel tree, the blood vessel crosses and bifurcations were identified. The starting point for registration of blood vessel control points was the thinned and pruned image shown in figure 6. A 3x3 mask was applied to the image in order to determine whether the pixels under the mask represented a single vessel, a vessel bifurcation, or a vessel cross, see figure 7. To optimize the computation the mask was only applied to the pixels containing blood vessels, i.e. the center of the mask was applied to a pixel with value 1.

4 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER Fig. 7. Different blood vessel responses from the mask applied on the binary, thinned, and pruned blood vessel tree. points in the destination image, see figure 10. First a 201x201 pixel template was created with the center point containing a bifurcation or cross point from the source image. The template was transformed (rotated, scaled, and skewed) to match the coordinates of the destination image by using the transfer function estimated from previous matched points. A mask response was calculated as the sum of all pixels with the value 1. If the mask response was 2 a vessel endpoint was detected. A value of 3 indicated an inner point of a vessel, while a mask response > 3 indicated a bifurcation or cross point. In the neighborhood of a bifurcation or cross point the mask response would in some cases be > 3 for several pixels next to the exact branch or cross point, see figure 8A and 8B. This resulted in several pixel markings of the same vessel bifurcation or cross. Fig. 8. (A) Illustration of a bifurcation point marked with several markings. (B) example of several markings of bifurcations. (C) Result of finding the center of mass of the group of markings. Therefore, the center of mass of each group of markings was found and used as the true bifurcation or cross point, see figure 8C. An example of detected bifurcation and cross points is shown in figure 9. The bifurcation and cross points was used in the transformation between the source and destination image. Fig. 10. Shows how a bifurcation or cross point from the source image was template matched in the destination image. A 501x501 window was opened in the destination image to reduce the template searce area. To remove black edges caused by rotation, scaling, and skew from the transformation the template was cropped to 101x101 pixels with the bifurcation or cross as center point. The template was matched in a 501x501 window opened in the destination image and centered at the transformation output point obtained by applying the transformation to the given bifurcation or cross point, see figure 10. This limits the template search area and reduces the probability of false positive classifications of the corresponding bifurcation or cross points. Only template matches above a threshold value of 0.8 (maximum correlation = 1) was included. The template matched point and the previous matches including the fovea and the optic disc were used to estimate a new transfer function. Initially a rigid transformation was estimated based on the location of the fovea and the optic disc and the angle between these points in the two image. The rigid transformation was of the form [ ] Xd = Y d [ cos(θ) sin(θ) sin(θ) cos(θ) ] [ ] Xs + t 1 Y s + t 2 where X s and Y s are the coordinates of the bifurcation point in the source image, t 1 and t 2 are the translation along the two axes and θ is the angular difference between the two images. After five iterations of matching templates correctly, and storing the matching points, the matched points and the coordinates from the optic discs and the foveas in the two images were used to estimate an affine transformation between the images. The affine transformation was of the form (1) Fig. 9. Detected bifurcation and cross points. X d = B 0x + B 1x X s + B 2x Y s + B 3x X s Y s (2) C. Registration 1) Template matching and estimation of the transfer function: Bifurcation and cross points detected in the source image were matched by template matching their corresponding Y d = B 0y + B 1y X s + B 2y Y s + B 3y X s Y s (3) where X d and Y d are the coordinates of the destination point corresponding to the source point X s and Y s. B ix and B iy are coefficents that determines the amount of translation, scaling, rotation, and skewing. The affine transformation

5 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER improves the results of the image fusion compared to the rigid transformation because of the B coefficents properties. A new affine transformation is estimated and saved iteratively for each new point matched. All of the matched points were later used to estimate the best output of the saved transfer functions. 2) Estimation of the best transfer function: To determine the best transfer function an error was calculated for each saved transfer function. The error was measured as the distance (D) between the template matched point (TMP) and the corresponding transformed bifurcation point (TBP), see figure 11. The distance between all the TBPs and their corresponding TMPs were calculated for each saved affine transfer function and an average distance was found. The affine transformation with the smallest average value was used as the best transfer function. The calculation was done under the assumption that the TMPs were matched correctly. Fig. 12. Illustration of the image transformation process. The coordinates of the source image was transformed to match the coordinates of the transformed source image by inserting the pixel value from the source image into the transformed source image. transformed source image. An illustration of the procedure is shown in figure 12. To fuse the transformed image with the destination image the size of the destination image was expanded to the same as the transformed image. The transformed image was then added to the destination image and their gray colors were averaged. The fusion process is illustrated in figure 13. Fig. 11. Error distance between the template matched point (TMP) and the transformed bifurcation point (TBP). D. Fusion The fusion of the destination image and the source image was based on the best affine transfer function. To fuse the two images the source image was transformed by the transfer function and added to the destination image. The transformation produced a displacement of the pixel coordinates for which reason the transformed source image was enlarged with the size of the translation from the transfer function. The greatest translation was found by transforming the corners of the image and using the maximum deviation x = max(t x [1, 1], T x [1, 3120]) (4) y = max(t y [1, 1], T y [2080, 1]) (5) where T x represents the transformation of the columns and T y represents the transformation of the rows. y represents the maximum translation in the rows of the image, and x represents the maximum translation of the columns of the image. To transform the source image a black image was created with the same size as the source image expanded by x and y from equations 4 and 5, see figure 12. Pixel values were transfered from the source image to the black image by using the transformation. In this way, the new image contained the Fig. 13. The transformed source image was added with the expanded destination image which results in a fused image. To remove the black edge in the fusion image a threshold was defined for which only colors above the threshold are shown in the fused image. The final fusion image is seen in figure 14. III. RESULTS The images used in the study were evaluated to exclude images containing optical errors. Figure 15 illustrates the number of images used through the study. The images without optical errors (43 image pairs) were randomly divided into two groups; one group of 22 image pairs for training, and one group of 21 image pairs for testing, see figure 15. The data was allocated into the two groups of equal size due to a relatively low number of observations compared to the variety of retinopathy represented in the images. The images were randomly assigned between the two groups to avoid allocation bias.

6 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER Fig. 14. Result of the image fusion between the transformed source image and the distination image. bifurcations or crosses. As seen in figure 15, 14 image pairs were able to be fused. 7 of the 21 image pairs could not be fused due to the missing segmentation of the fovea or the optic disc in these images. The results of all of the fusions can be seen in appendix 17 on page 9. The figure illustrates both successful fusions, averaged fusions, and fusions that are deformed. The results of the average error, and the number of matched points in each image pair were as follows. The average error of each fused image pair were measured as the pixel distance from the transformed points to the template matched points. In average, the error in every image pair was 70,5 pixels. The highest error was 166,7 pixels and the lowest 2,7 pixels. The average number of matched points found in every image pair was 19. The highest number was 42 and the lowest was 3. Due to the low number of observations available for this study it was not possible to assume that the data was normally distributed. To measure the linear associations between two independent variables a non parametric test was used to investigate the correlation between the number of template matches and the average error. In addition, the same test was used to find a correlation between the number of template matches and the result of the fusion. In this paper the Spearman rank correlation test was applied. The spearman coefficient was calculated using the following equation r s = 1 6 d 2 i n(n 2 1) (6) Fig. 15. Flow chart showing the number of images used through the study. The images used were randomized into training and test data. The images excluded due to optical errors or detection errors are also shown in the boxes to the right. where r s is the spearman coefficient, d is the difference between each rank, and n is the number of items or individuals being ranked. The first test showed a r s = with a p value = , which indicates that there is a significant negative correlation between the number of template matches and the average pixel error, see figure 16. As shown in figure 15, 21 image pairs (42 separate images) were tested. The results of the segmentation of the optic disc, fovea and blood vessel bifurcations and crosses were as follows: The optic disc was segmented correctly in 36 images and incorrectly in 6 images, which gives a total of 86% correct segmentations. The fovea was segmented correctly in 37 images and incorrectly in 5 images. The total correct segmentations of the fovea was 88%. In average, the number of localized blood vessel bifurcations or crosses were 48 in each image. The highest number was 110 and the lowest was 3 blood vessel Fig. 16. The graph shows a correlation between the number of matched points and error. The second test showed a r s = with a p value = , which means that there is a significant negative correlation between number of template matches and image fusion.

7 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER IV. DISCUSSION In this study, an automatic algorithm was used to localize the three main retinal features (the fovea, the optic disc, and the blood vessel bifurcation points) and to fuse two fundus retinal images. Methods for preprocessing, segmentation of the features, registration of bifurcation points, estimation of an affine transformation and fusion was presented. The applied methods have a number of advantages and disadvantages that will be discussed and compared to previous published work. A total of 48 image pairs were available for the study, see figure image pairs were excluded from the study because of optical artifacts. The remaining 43 pairs were randomly divided into two groups; 22 image pairs were used for training and 21 for testing. The fovea and the optic disc could not be detected in 5 (12 %) and 6 (14 %) images respectively. This reduces the number of images that could be used for testing of the image fusion to 14 image pairs. This means that only 66,6 % of the images allocated as test images could be used for testing the algorithm. To decrease the number of excluded images different precautions could be made in future studies. The capturing of the images could be improved by recapturing images with optical artifacts in order not to exclude any patients. If capturing of the images could be controlled a model of the fundus retina could be developed in order to compensate for image distortion in the periphery of the images. The fovea was recognized in 88 % of the images and the optic disc in 86 % of the images. The methods proposed by Sinthanayothin et al for recognizing the optic disc and the fovea showed a recognition degree of 99.1%, and 80.4% respectively. Compared to the presented algorithm in this study the recognition of the optic disc could be improved by implementing this method. The recognition of the fovea is higher for the presented algorithm than proposed by Sinthanayothin et al [4] In some images the fovea is not detected due to low contrast between the image background and the area of the fovea. This implies that more robust features of the fovea are needed to segment fovea in these special cases. The detection of blood vessel bifurcation points resulted in an average of 48 bifurcations in each image, ranging from a minimum of 2 bifurcations to a maximum of 110. The results imply that the segmentation was inconsistent and should be improved in order to ensure smaller deviation from the mean number of detected bifurcations. By using a threshold that states the minimum number of matched points in an image pair it could be ensured that enough matches is found to make a successful fusion. Small blood vessels often have a higher intensity, which causes removal of these when a threshold is applied to the binary vessel image. To solve this problem the images could be analyzed using different thresholds with the purpose of enhancing vessels with different intensities and then adding the results. However, this would increase the computational burden. The results showed a correlation between the number of matched points and the average error caused by the best iteration of the transformation. This means that if the segmentation of the blood vessel bifurcations is improved it would improve the overall transformation and result in a more consistent image fusion. In images with low contrast between the blood vessels and the image background an alternative to bifurcations should be used for template matching. For instance lesions could be used as matching points. Another approach could be to detect the size of the overlap between the images and insert match points along the edge of the overlap to reduce perimeter distortion. This method could be used as a supplement to the previous matched points. Evaluation of the result of the fusions was found to be correlated with the number of matched points in the images. This emphasizes that the number of detected bifurcations and crosses needs to be increased to improve the image fusion. It can be seen in figure 16 that for some image pairs with the same number of matched points the averaged errors are significantly different. This indicates a tendency between a good fusion and the localization of the matched points in the overlapping area. However, this has to be further investigated to obtain a conclusion. It is concluded that the presented algorithm was able to register and fuse two overlapping retinal images. However, to decrease the distortion in the fused images further processing work has to be done. The identification of the fovea and the optic disc has to be optimized to decrease the number of excluded images. The segmentation of blood vessel bifurcations and crosses has to be improved in order to increase the number of matched points, which would increase the number of successful fusions. REFERENCES [1] F. Laliberte, L. Gagnon, and Y. Sheng, Registration and fusion of retinal images - an evaluation study, IEEE Transactions on Medical Imaging, pp , [2] S. Wild, G. Roglic, A. Green, R. Sicree, and H. King, Global prevalence of diabetes, Diabetes Care, vol. 27, pp , [3] T. Schroeder, Basisbog i Medicin og Kirurgi. Munksgaard, [4] C. Sinthanayothin, J. F. Williamson, T. H. Cook, E. Mensah, and S. Lal, Automated detection of diabetic retinopathy on digital fundus images. diabetic medicine, Diabetic Medicine, pp , [5] S. El-Allali and S. Brown, Active contours technique in retinal image identification of the optic disk boundary, Department of Computer Science and Engineering, University of South California, [6] A. Pinz, S. Bernogger, P. Datlinger, and A. Kruger, Mapping the human retina, IEEE Transactions on Medical Imaging, pp , [7] D. Paulus, S. Chastel, and T. Feldmann, Vessel segmentation in retinal images, Institut fur Computervisualistik, Universitet Koblens Landau, [8] M. E. Martinez-Perez, A. D. Hughes, A. V. Stanton, S. A. Thom, N. Chapman, A. A. Bharath, and K. H. Parker, Retinal vascular tree morphology: A semi-automatic quantification, IEEE Transactions on biomedical engineering, vol. 49, no. 8, [9] Patton, Aslan, MacGillivray, Deary, Dhillon, Eikelboom, Yogesan, and Constable, Retinal image analysis: Concepts, applications and potential, Progress in Retinal and Eye Research, vol. 12, pp , [10] A. Singalavanija, J. Supokavej, P. Bamroongsuk, C. Sinthanayothin, Phoojaruenchanachai, and V. Kongbunkiat, Feasibility study on computer.aided screening for diabetic retinopathy, Jpn. J. Opthalmol., vol. 50, pp , [11] A. D. Fleming, S. P. N. K. A. Goatman, G. J. Williams, J. A. Olson, and P. F. Sharp, Automated detection of exudates for diabetic retinopathy screening, Phys. Med. Biol., vol. 52, pp , 2007.

8 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER [12] M. Larsen, T. Gondolf, J. Godt, M. S. Jensen, N. V. Hartvig, H. Lund- Andersen, and N. Larsen, Assesment of automated screening treatmentrequiring diabetic retinopathy, Current Eye Research, vol. 32, no. 4, pp , [13] H. J. Jelinek, M. J. Cree, D. Worsley, A. Luckie, and P. Nixon, An automated microaneurysm detector as a tool for identification of diabetic retinopathy in rural optometric practice, Clinical and Experimental Optometry, vol. 85, no. 5, pp , 2006.

9 AALBORG UNIVERSITY, SEMCON, 21. DECEMBER Fig. 17. The figure shows all the results of the fusions. For every image pair the number of matched points, the average error, and the classification of the fusion are shown.