Object-Level Saliency Detection Combining the Contrast and Spatial Compactness Hypothesis

Transcription

1 Object-Level Saliency Detection Combining the Contrast and Spatial Compactness Hypothesis Chi Zhang 1, Weiqiang Wang 1, 2, and Xiaoqian Liu 1 1 School of Computer and Control Engineering, University of Chinese Academy of Sciences, Beijing, China 2 Key Lab of intelligence information Processing, Institute of Computing Technology, CAS, Beijing, China Abstract - Object-level saliency detection is an important branch of visual saliency. Most previous methods are based on the contrast hypothesis which regards the regions presenting high contrast in a certain context as salient. Although the contrast hypothesis is valid in many cases, it can t handle some difficult cases, especially when the salient object is large. To make up for the deficiencies of contrast hypothesis, we incorporate a novel spatial compactness hypothesis which can effectively handle those tough cases. In addition, we propose a unified framework which integrates multiple saliency maps generated on different feature maps built on different hypotheses. Our algorithm can automatically select saliency maps of high quality according to the quality evaluation score defined in this paper. The experimental results demonstrate that each key component of our method contributes to the final performance and the full version of our method outperforms all state-of-the-art methods on the most popular dataset. Keywords: Saliency Detection, Salient Object Detection 1 Introduction Human vision system is capable to effectively preselect the data of potential interest from a complex scene for further processing. The computer vision systems will be much more efficient if we endow them with such an excellent ability. To this end, a lot of efforts have been conducted on bottom-up saliency detection. In early years, saliency detection methods [1][2][3][4] are usually biologically inspired, they aim to produce consistent predictive fixations with human vision systems and the evaluation datasets are eye movement datasets which record the information of fixations. We call this category of saliency detection as fixation-level saliency detection. In recent years, a new trend that aims to uniformly highlight the whole salient object becomes popular because of its various applications in object based tasks, such as object recognition [5], image cropping [6], and content-aware image resizing [7], etc. This kind of work is evaluated on the datasets with salient objects manually labeled, so we call it as object-level saliency detection. In this paper, our research mainly focuses on the object-level saliency detection. Due to the lack of high-level knowledge, every bottomup detection method is established on some effective hypotheses inspired by the characteristics of salient objects or background. Among all hypotheses, the contrast hypothesis is the most widely used. It takes the region presenting high contrast in a local or global context as salient. The contrast hypothesis is effective in most cases, because the salient object is usually distinct from its surroundings. So most existing object-level saliency detection methods, either explicitly or implicitly make use of this hypothesis. They usually implement it by measuring the difference between the current region and its surroundings in pixel-level [9], patch-level [10], super-pixel-level [12], the sliding window-level [18] or partial combination of them [11]. However, the contrast hypothesis is not omnipotent, because there are some tough cases it can t handle well. As shown in Fig.2, due to intrinsic flaw of the contrast hypothesis, the pure contrast hypothesis based methods (FT-CVPR09 [9], CA- CVPR10 [10] and RC-CVPR11 [12]) are usually fail to highlight the interior region of large salient object and the salient objects sharing similar appearance with the background, meanwhile, they can t suppress the highly textured background regions which also present high contrast. To cover the shortage of contrast hypothesis, we incorporate an effective spatial compactness hypothesis which believes that the salient regions should be spatially more compact than the background regions. In other words, a region is salient if its appearance-similar regions are nearby, and it is less salient if there are many appearance-similar regions far away from it. In this paper, we intuitively implement the spatial compactness hypothesis by computing the reciprocal of the sum of appearance weighted distances from the current region to all the other regions. It is fortunate that the spatial compactness can usually handle those tough cases described above. However, same like the contrast hypothesis, the spatial compactness hypothesis is not always valid because sometimes the background regions may be composed of some isolated regions which also show great spatial compactness or there are multiple similar salient objects far away from each other. At this moment, the contrast hypothesis is needed to make up for the deficiencies of the spatial compactness. So these two hypotheses could nicely complement each other and by combining the contrast hypothesis and spatial compactness hypothesis, our method is able to effectively handle most cases. Previous work which is the most relevant to ours is [14]. To our best knowledge, [14] is the first method that employs both contrast hypothesis and spatial

2 compactness hypothesis, though it considers the spatial compactness hypothesis as one type of contrast hypothesis. It firstly over-segments the image into some super-pixels, and each super-pixel is represented by the mean color value of the pixels within the super-pixel in Lab space. Then the color uniqueness (contrast) value and color distribution (spatial compactness) value of each super-pixel are computed. Finally, the color uniqueness values and color distribution values are combined and the saliency value is assigned to each super-pixel after a smoothing procedure. This method achieved improvement over the contrast hypothesis based state-of-the-art method [12]. However, in experiments, we find that the salient object sometimes presents salient only in individual color channels but not in the whole color space, so it is not wise to simply represent each super-pixel by the mean color value or only compute the contrast and spatial compactness values in the whole color space. In this paper, we compute the contrast and spatial compactness values not only in the whole color space but also in individual color channels. To fuse multiple saliency maps generated on different feature maps with different hypotheses into the final saliency map, we propose a unified framework that can automatically select saliency maps of high quality according to the quality evaluation score defined in this paper. The evaluation score is computed by combining three novel heuristic rules which are proposed according to the characteristics of salient object. What s more, it is convenient to add new features and hypotheses into this unified framework, e.g. we can easily generalize our method to video by incorporating some proprietary cues of video such as motion information. 2 The proposed method 2.1 Overview The whole procedure of our method is listed as follows: firstly, the input image is converted into CIE Lab color space and split into uniform patches. The compact representations of patches in each color channel are obtained by PCA; then, the contrast saliency values and spatial compactness saliency values are computed for each color channel and the whole color space; next, the multiple saliency maps are refined by incorporating an image segmentation component. Finally, the quality score of each saliency map is computed and some saliency maps are selected and fused into the final saliency map. In the following, we will elaborate each step of our method. 2.2 Patch representation For an input color image I, we first convert its color space into CIE Lab and then partition it into some nonoverlapping square patches size of (we set to 8 in our implementation). For each patch at row and column,,, it is represented as three vectors,, where is formed by concatenating the pixel values in patch on color channel and the length of each vector is. Correspondingly, image I is represented as three patch matrices, To eliminate the dimension with noises and make the following computation more efficient, we employ principal component analysis (PCA) to reduce the dimension of the patch representation and obtain the compact patch matrices, is the compact vector in low-dimensional space corresponding to. In addition, the spatial position of patch is denoted by the vector in the following formulations. 2.3 Saliency maps generation We introduce the details of how to generate saliency maps based on the contrast hypothesis and spatial compactness hypothesis in this subsection. It is interesting that the computation of both the contrast value and spatial compactness value can be formulated into a unified weighted linear combination form like the following formulation, (1) where is the main factor and is the weight. The position dissimilarity between patches and appearance dissimilarity between patches alternately act as the main factor and weight. Concretely, for the contrast hypothesis, the appearance dissimilarity acts as the main factor, while the position dissimilarity is used to compute weight. They swap roles for the spatial compactness hypothesis. In this paper, the computation methods of those two kinds of dissimilarity are different. The appearance dissimilarity is measured by the Euclidean distance defined in Eqn. (2), while the position dissimilarity is computed via infinite norm distance defined in Eqn. (3), ( ), (2) ( ). (3) With the appearance dissimilarity in individual color channels, we can obtain the appearance dissimilarity in the whole color space easily by adding them together, namely, ( )= ( ). We add the whole color space Lab to the color channel set, namely, in the following,. When position dissimilarity and appearance dissimilarity act as weights, both of them will be feed into a Gaussian function, which are respectively defined in Eqn. (4) and (5), ( ) ( ( ) ( ( ) ), (4) ( ) ), (5)

3 where and are the normalization factors to ensure ( ) and ( ) equal 1. And is an important factor that controls the influence of weights. The bigger the is, the smaller the influence of weights will be. So it is very important to choose an appropriate. Because ( ) and ( ) vary greatly with different images, it is difficult to select a fixed suiting all the images. So we employ adaptive and defined as, ( ), (6) ( ), (7) in which, and are constant factors. We set both of them to 0.35 in this paper Contrast value computation Similar to [14] [19], we obtain the contrast value by computing the sum of spatially weighted dissimilarities between the current patch and all other patches. Concretely, the contrast value of patch in color channel is defined as, ( ) ( ), (8) where ( ) denotes the appearance dissimilarity between patch and in color channel defined in Eqn. (2) and ( ) is the position weight defined in (4) Spatial compactness value computation In this paper, the spatial compactness is defined as the reciprocal of the sum of appearance weighted position dissimilarities between the current patch and all the other patches. So the spatial compactness value of patch in color channel is defined as, ( ( ) ( )), (9) where ( ) denotes the position dissimilarity between patch and defined in Eqn. (3) and ( ) is the appearance weight defined in Eqn. (5) Saliency value assignment To reduce the impact of noise and improve the homogeneity of saliency map, we employ a weighted sum of the contrast values or spatial compactness values of the patches in a square neighborhood of the current patch as the saliency value defined as,, (10) in which, denotes the color channel, and denotes the hypothesis we employ, and controls the size of square neighborhood, in our method =7. Finally, we normalize the obtained saliency maps to [0, 1] via a linear stretch defined as, ( ) ( ) ( ) ( ) ( ). (11) By now, we obtain eight patch-level saliency maps generated on different color channel with different hypotheses. Based on these patch-level saliency maps, the pixel-level saliency maps can be easily obtained through assigning saliency value of the patch to the pixels within it. Then we get eight pixel-level saliency maps, they are,,,,,,,. 2.4 Saliency maps refinement The obtained saliency maps are generated on the unit of patch, which may suffer from the blur of salient object boundary, because object boundaries may fall into different patches when an image is partitioned into the patches of the same size and shape without considering visual content. Therefore, these saliency maps may not achieve accurate saliency values near the object boundaries. So, we incorporate an image segmentation module to refine our saliency maps. The original image is over-segmented into a group of regions via the mean shift method implemented in the EDISON system [8], we set the min area of each region to 1000 in our implementation. Then the pixel-level saliency value in each region is given by the mean of pixel-level saliency values in it, i.e.,, (12) where denotes the region which the pixel at location belongs to, and is the total number of pixels in region. The linear stretch normalization is performed to each refined saliency map as Eqn. (11). 2.5 Selection and fusion of saliency maps The last but not least step is to fuse the multiple saliency maps obtained in preceding steps into the final saliency map. As shown in Fig.1, not all the obtained saliency maps are able to highlight the salient object uniformly and accurately, some saliency maps even mistakenly highlight the background regions. So it is not wise to fuse all those saliency maps into final result, the saliency maps of high quality should be picked out and those of poor quality should be discarded. Based on the above thoughts, we propose a fusion framework which can automatically pick out saliency maps of high quality according to the quality evaluation score defined in the following. In the framework, we first compute the evaluation score of each saliency map according to three

4 Figure 1. Refined saliency maps of each individual color channel and the whole color space built on both contrast hypothesis and spatial compactness hypothesis. In the first row from left to right: original image,,, and. In the second row from left to right: ground-truth,,, and. The evalutation scores of the first four saliency maps respectively are 1.83, 2.24, 1.79 and The evalutation scores of the last four saliency maps respectively are 2.82, 2.82, 2.24 and The selected saliency maps are,, and novel heuristic rules we proposed. Then, the saliency map whose evaluation score is higher than a pre-defined threshold will be selected. Finally, we fuse those selected saliency maps in a linear weighted sum form, and the weight is the corresponding evaluation score Definition of the evaluation scores According to the characteristics of salient object, we propose three novel heuristic rules to evaluate the quality of a saliency map, i.e., ratio of center-surround, distribution compactness and saliency variance. The final evaluation score is computed based on those three rules to measure the extent that how accurately and uniformly the saliency map highlights the salient object. Based on the fact that salient objects usually lie near the center of an image and the finding that human eyes also tend to fixate on the center when viewing scenes [20], we choose the ratio of center-surround as one assessment rule. The ratio of center-surround is defined as the ratio of the sum of saliency values of the central region to that of the surrounding region, and in our implementation it is formulated as, ( ),(13) where and denote the width and height of the saliency map respectively. The high ratio of center-surround reflects high quality of a saliency map. However, salient objects don t always lie in the center of image, so we need other rules to enhance the reliability of the final evaluation score. Because the salient objects are spatially compact as described by the spatial compactness hypothesis, in the saliency map of high quality the pixels with high saliency values should be spatially close as well. So we take the distribution compactness as the second assessment rule defined as, ( ) ( ( ( ) )), (14), (15), (16), (17) where the is the normalization factor, and denote the coordinates and of the gravity center, and are the width and height of the saliency map. A lot of observations in the experiments show that the saliency map which is well consistent with the ground truth usually has a high pixel variance, so we employ the pixel variance of saliency map as the last assessment rule defined as, where ( ), (18), (19) is the mean value of the saliency map. With the above three assessment rules, we define the final quality evaluation score as, ( ) ( ) ( ) ( ), (20)

5 Original FT- CVPR09 CA- CVPR10 LIU- PAMI11 RC- CVPR11 BS-TIP12 LR- CVPR12 SF- CVPR12 GS- ECCV12 CC-OURS Groud- Truth Figure 2. Three examples of qualitative comparison results between our method (CC-OURS) and other 8 state-of-the-art methods (FT-CVPR09 [9], CA-CVPR10 [10], LIU-PAMI11 [11], RC-CVPR11 [12], BS-TIP12 [13], LR-CVPR12 [14], SF- CVPR12 [15] and GS-ECCV12 [16]). where, and are the normalization factors, which are respectively the maximum of ( ), ( ) and ( ), i.e., ( ), (21) ( ), (22) ( ). (23) Combination of saliency maps After computing the quality evaluation score of each saliency map, we combine those selected saliency maps in a linear weighted sum form, which is formulated as, ( ( )), (24) ( ( )) { ( ) ( ) ( ), (25) ( ), (26) where is an adaptive threshold defined in Eqn. (26). The saliency map will be discarded, if its quality evaluation score is smaller than. Finally, we normalize the final saliency map as Eqn. (11). It is worth mentioning that, with the novel fusion framework, we can easily integrating new features and hypotheses into our method. 3 Experiments and results We test our method on a publicly available dataset which contains 1000 images along with ground-truth (GT) in the form of human-labeled masks for salient objects provided by Achanta et al. [9]. We compare the proposed method (CC- OURS) with 8 state-of-the-art object-level saliency detection methods. We select these methods according to: popularity (FT-CVPR09 [9], CA-CVPR10 [10], LIU-PAMI11 [11] and RC-CVPR11 [12]) and recency (BS-TIP12 [13], LR-CVPR12 [14], SF-CVPR12 [15] and GS-ECCV12 [16]). The saliency maps of FT-CVPR09 [9] and RC-CVPR11 [12] are provided by [12]. The saliency maps of LIU-PAMI11 [11] are generated with the parameters reported in their paper using the code provided by the author. The saliency maps of other methods are obtained from the corresponding authors homepages. In our experiments, both qualitative and quantitative comparisons are performed. Three examples of qualitative comparison results are shown in Fig.2. These three examples were selected as the most difficult stimuli by [17], because they represent three typical tough cases of the objectlevel saliency detection task, including large salient object, textured background and low contrast between salient object and background. It can be seen that our method consistently produces saliency maps close to ground-truth while the other methods don t perform well in at least one case. So our method is more robust than the rest 8 state-of-the-art methods and can handle the three tough cases well. In quantitative comparison, similar to [9], we employ the recall-precision curve to evaluate our method. With saliency value in the range [0,255], we binarize the saliency maps with varying thresholds from 0 to 255 and the recalls and precisions are obtained by comparing the binary results with the groundtruth. Concretely, the precision and recall under a certain threshold are defined as,, (27), (28) where is the binary mask under threshold of the image, is the corresponding ground-truth and is the number of test images. The recall-precision curves of 8 stateof-the-art methods and our method are shown in Fig.3 (a) (b). We can see that our method outperforms all the other 8 methods. Our method can predict the most salient regions accurately with a maximum precision about 0.96, meanwhile it keeps high precisions when the recalls are high (achieve a precision over 0.9 when the recall is 0.8 and a precision over 0.8 when the recall is 0.9). In addition, we also evaluate the incomplete versions of our method. Each of them eliminates

6 one key component from the full version of our method. They are the version without contrast hypothesis, the version without spatial compactness hypothesis and the version without the novel fusion framework. The last one is implemented by only computing the contrast values and spatial compactness values in the whole color space and simply adding the saliency maps together without weights. The evaluation results are shown in Fig.3 (c). We can see that the proposed fusion framework play an important role in our method, relying on it the two incomplete versions respectively only employing one hypothesis both achieve satisfactory results. The results also show that compared with the version without spatial compactness hypothesis, the version without contrast hypothesis achieves higher precisions especially when the recalls are high. This is mainly because the spatial compactness hypothesis can usually highlight the whole salient object even when the salient object is large or doesn t present high contrast and suppress the background even if the background is textured while the contrast hypothesis isn t good at these tough cases. So the spatial compactness hypothesis is a nice complement to contrast hypothesis. By integrating all these three key components, the full version of our method achieves the optimal performance. (c) Figure 3. (a)(b) The recall-precision curves of 8 state-of-theart methods and our method. (c) Evaluation of the methods without contrast hypothesis, without spatial compactness hypothesis, without thefusion framework and the full version of our method. Figure 4. The recall-precision curves of the proposed method and the results generated by manually selecting the saliency maps of high quality in the fusion stage. (a) (b) Figure 5. The failure cases of our method. The first column is the original image, the second column is the saliency map of our method, the third column is the ground-truth.

7 4 Conclusions In this paper, we propose an object-level saliency detection method which integrates the popular contrast hypothesis and the effective spatial compactness hypothesis. The experimental results show that our method achieves satisfactory performance and outperforms all state-of-the-art methods in both qualitative and quantitative comparisons. In addition, we propose a fusion framework of multiple saliency maps that can automatically select saliency maps of high quality. The experimental results demonstrate that this fusion framework is very effective and plays an important role in our method. However, all though the proposed three assessment rules could help pick out the saliency maps of high quality accurately, as shown in Fig.4, there is still room for improvement comparing with manual selection. So we are supposed to find some new assessment rules to make the automatic selection more accurate. As shown in Fig.5, though the contrast hypothesis and spatial compactness hypothesis can handle most cases via cooperation, there are still some tough cases that neither of those two hypotheses can effectively deal with. So, to improve our method, we can incorporate some other effective hypotheses into our framework. As introduced before, we will try to make our method be utilized in video by incorporating some proprietary cues of video. 5 Acknowledgement This work was supported by the National Natural Science Foundation of China under Grant No , No , No References [1] L. Itti, C. Koch, and E. Niebur, A model of saliencybased visual attention for rapid scene analysis, TPAMI, vol. 20, no. 11, pp , [2] N.D.B. Bruce and J.K. Tsotsos, Saliency based on information maximization, in NIPS, [3] X. Hou and L. Zhang, Saliency detection: A spectral residual approach, in CVPR, 2007, pp [4] J. Harel, C. Koch, and P. Perona, Graph-based visual saliency, in NIPS, [5] U. Rutishauser, D. Walther, C. Koch, and P. Perona, Is bottom-up attention useful for object recognition?, in CVPR, [6] A. Santella, M. Agrawala, D. DeCarlo, D. Salesin, and M. Cohen, Gaze-based interaction for semi-automatic photo cropping, in Proceedings of the SIGCHI conference on Human Factors, [7] H. Wu, Y.S. Wang, K.C. Feng, T.T. Wong, T.Y. Lee, and P.A. Heng, Resizing by symmetry-summarization, ACM Trans. Graph., vol. 29, no. 6, pp. 159:1 159:10, [8] D. Comaniciu and P. Meer, Mean shift: A robust approach toward feature space analysis, TPAMI, vol. 24, no. 5, pp , [9] R. Achanta, S. Hemami, F. Estrada, and Sabine Ssstrunk, Frequency-tuned salient region detection, in CVPR, [10] Stas Goferman, Lihi Zelnik-Manor, and Ayellet Tal, Context-aware saliency detection, in CVPR, [11] T. Liu, Z.J. Yuan, J. Sun, J.D. Wang, N.N. Zheng, X.O. Tang, and H.Y. Shum, Learning to detect a salient object, TPAMI, vol. 33, no. 2, pp , [12] M.M. Cheng, G.X. Zhang, N.J. Mitra, X.L Huang, and S.M Hu, Global contrast based salient region detection, in CVPR, [13] Yulin Xie, Huchuan Lu, and Ming-Hsuan Yang, Bayesian saliency via low and mid level cues, TIP, vol. 22, no. 2, pp , [14] F. Perazzi, P. Krahenbul, Y. Pritch and A. Hornung, Saliency filters: contrast based filtering for salient region detection, in: CVPR, [15] X.H Shen and Y. Wu, A unified approach to salient object detection via low rank matrix recovery, in CVPR, [16] Y.C. Wei, F. Wen, W.J. Zhu, and J. Sun, Geodesic Saliency Using Background Priors, in ECCV, [17] Ali Borji, Dicky N. Sihite, and Laurent Itti, Salient Object Detection: A Benchmark, in ECCV, [18] Esa Rahtu, Juho Kannala, Mikko Salo, and Janne Heikkila, Segmenting salient objects from images and videos, in ECCV, [19] L.J. Duan, C.P Wu, J. Miao, L.Y. Qing and Y. Fu Visual Saliency Detection by Spatially Weighted Dissimilarity, in CVPR, [20] Benjamin W. Tatler, The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions, Journal of Vision, vol. 7, no. 14, pp. 1 17, 2007.