INCORPORATING VISUAL ATTENTION MODELS INTO IMAGE QUALITY METRICS

Size: px

Start display at page:

Download "INCORPORATING VISUAL ATTENTION MODELS INTO IMAGE QUALITY METRICS"

Ellen Stephens
5 years ago
Views:

1 INCORPORATING VISUAL ATTENTION MODELS INTO IMAGE QUALITY METRICS Welington Y.L. Akamine and Mylène C.Q. Farias, Member, IEEE Department of Computer Science University of Brasília (UnB), Brasília, DF, , Brazil ABSTRACT A recent development in the area of image quality consists of trying to incorporate aspects of visual attention in the metric design. This is generally achieved by combining distortion maps and saliency maps. Although a good number of saliency-inspired image quality metrics have been proposed, results are not yet conclusive. Some researchers have reported that the incorporation of visual attention increases the performance of quality metrics, while others have reported no or very little improvement. In this work, we investigate the benefit of incorporating saliency maps obtained with computational visual attention models into three image quality metrics (SSIM, PSNR, and MSE). More specifically, we compare the performance of quality metrics using saliency maps obtained with computational visual attention models versus quality metrics using subjective saliency maps. Results show that performance is linked to not only the precision of the saliency map, but also to the type of distortion. I. INTRODUCTION AND MOTIVATION Objective visual quality metrics can be classified as data metrics, which measure the fidelity of the signal without considering its content, or picture metrics, which estimate quality considering the visual information contained in the data. Customarily, quality measurements in the area of image processing have been largely limited to a few data metrics, such as mean absolute error (MAE), mean square error (MSE), and peak signal-to-noise ratio (PSNR), supplemented by limited subjective evaluation. Although over the years data metrics have been widely criticized for not correlating well with perceived quality measurements, it has been shown that such metrics can predict subjective ratings with reasonable accuracy as long as the comparisons are made with the same content, the same technique, or the same type of distortions. One of the major reasons why these simple metrics do not generally perform as desired is because they do not This work was supported in part by Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) Brazil, in part by Fundação de Empreendimentos Científicos e Tecnológicos (Finatec), and in part by University of Brasília (UnB) ProIC/DPP. incorporate any human visual system (HVS) features in their computation. It has been discovered that, in the primary visual cortex of mammals, an image is not represented in the pixel domain, but in a rather different manner. Unfortunately, the measurements produced by metrics like MSE or PSNR are simply based on a pixel to pixel comparison of the data, without considering what is the content and the relationships among pixels in an image (or frames). In the past few years, a big effort in the scientific community has been devoted to the development of better image and video quality metrics that incorporate HVS features (i.e. picture metrics) and, therefore, correlate better with the human perception of quality [1][2]. A recent development in the area of image quality consists of trying to incorporate aspects of visual attention in the design of visual quality metrics, mostly using the assumption that visual distortions appearing in less salient areas might be less visible and, therefore, less annoying [3], [4]. This research area is still in its infancy and results obtained by different groups are not yet conclusive, as pointed out by Engelke et al [5]. Some researchers have reported that the incorporation of saliency maps increases the performance of quality metrics, while others have reported no or very little improvement. Among the works that have reported some improvement, most use subjective saliency maps, i.e. saliency maps generated from eye-tracking data obtained experimentally [6] [7]. This raises the question of how the metric performance is affected by the precision of the saliency map and the integration model. Another open question is how the distortion type affects the saliency map and, consequently, the metric performance. In this work, we investigate the benefit of incorporating saliency maps obtained with computational visual attention models into three image quality metrics (SSIM, PSNR, and MSE). More specifically, we compare the performance of quality metrics using saliency maps obtained with computational visual attention models versus quality metrics using subjective saliency maps [6]. We also discuss the influence of different types of distortion on the saliency maps (objective and subjective) and the performance of the tested metrics.

(a) Original image Caps (b) Itti s saliency map of Caps (c) Achanta s saliency map of Caps (d) Subjective saliency map of Caps (e) Original image Rapids (f) Itti s saliency map of Rapids (g) Achanta

Saliency maps corresponding to the images Caps and Rapids, obtained using computational attention models by Itti [8] and Achanta et al [9], and subjective saliency maps from TUD LIVE Eye Tracking

VISUAL ATTENTION AND IMAGE QUALITY When observing a scene, the human eye typically filters the large amount of visual information available on the scene and attend to selected areas [8].

The selection of fixations is based on the visual properties of the scene.

Visual attention is, therefore, a feature of the human visual system that has the goal of reducing the complexity of scene analysis.

These two mechanisms are known as bottom-up and top-down attention selection.

It is fast and short lasting, being performed as a response to low-level features that are perceived as visually salient, standing out from the background of the scene.

It is slower than bottom-up attention, requiring a voluntary effort. In this work, we use two bottom-up visual attention computational models and a database of subjective saliency maps.

2 (a) Original image Caps (b) Itti s saliency map of Caps (c) Achanta s saliency map of Caps (d) Subjective saliency map of Caps (e) Original image Rapids (f) Itti s saliency map of Rapids (g) Achanta s saliency map of Rapids (h) Subjective saliency map of Rapids Fig. 1. Saliency maps corresponding to the images Caps and Rapids, obtained using computational attention models by Itti [8] and Achanta et al [9], and subjective saliency maps from TUD LIVE Eye Tracking database [10]. II. VISUAL ATTENTION AND IMAGE QUALITY When observing a scene, the human eye typically filters the large amount of visual information available on the scene and attend to selected areas [8]. Oculo-motor mechanisms allow the gaze of attention to either hold on a particular location (fixation) or to shift to another location when sufficient information has already been collected (saccades). The selection of fixations is based on the visual properties of the scene. Priority is given to areas with a high concentration of information, minimizing the amount of data to be processed by the brain while maximizing the quality of the collected information. Visual attention is, therefore, a feature of the human visual system that has the goal of reducing the complexity of scene analysis. It can be divided in two mechanisms that, combined, define which areas of the scene are to be considered relevant and, therefore, should be attended. These two mechanisms are known as bottom-up and top-down attention selection. The bottom-up mechanism is an automated selection that is controlled mostly by the signal, independent of the task being performed. It is fast and short lasting, being performed as a response to low-level features that are perceived as visually salient, standing out from the background of the scene. The top-down mechanism is controlled by higher cognitive factors and external influences, such as semantic information, viewing task, personal preferences, and context. It is slower than bottom-up attention, requiring a voluntary effort. In this work, we use two bottom-up visual attention computational models and a database of subjective saliency maps. The first computational model is the visual attention proposed by Itti [8] and implemented by Walther and Koch[11]. Itti s model analyzes the images by looking for the most relevant visual properties: color, intensity, and orientation. The algorithm creates maps corresponding to these three properties and combines these maps to predict probabilities of fixations in specific areas of the image. Using a winner-take-all approach, predicted fixations are selected as the most salient points in the image. In order to combine the attention information with the objective quality metrics, a saliency map is needed, instead of the individual (ordered) fixation points. To obtain such a map, we filter the set of predicted fixation points with a Gaussian filter. The second visual attention computational model used in this work was proposed by Achanta et al [9]. To obtain a saliency map, the model uses a contrast determination filter operating at various scales. First, the algorithm calculates the local contrast of a sub-region with respect to its neighboring regions at various scales (varying sizes). Then, the distance between the average feature vector of this sub-region and the average feature vector of its neighboring regions is calculated. Using these feature vectors, feature maps at a given scale are calculated. These individual maps are combined to produce the final saliency map. For comparison purposes, we also used the subjective saliency maps from the TUD LIVE Eye Tracking database [10], collected in a subjective experiment performed at the Technical University of Delft. The experiment used an eyetracker equipment and twenty-nine source images of the LIVE image quality assessment database [12]. The saliency maps used in this work were derived from the eye-tracking data. In Fig. 1, the images Caps and Rapids are depicted, along with their saliency maps. Columns 2 and 3 of Fig. 1 correspond to the saliency maps obtained using Itti s and Achanta s computational model, respectively. Column 4 of Fig. 1 corresponds to the subjective saliency maps from the

3 TUD LIVE Eye Tracking database. In the saliency maps, higher (lighter) luminance values correspond to a higher saliency, while lower (darker) values correspond to points where a low attention was predicted. Notice that all saliency maps are able to capture (at least) the most salient area of the images Caps and Rapids, which correspond to the caps and the boat, respectively. Nevertheless, the three maps show lots of differences. For example, there seems to be less salient areas in the subjective maps. Also, the maps obtained with Achanta s model have less uniform areas. Without further processing, Itti s model seem to provide a saliency map that is closer to the subjective maps. III. INCORPORATING SALIENCY INTO IMAGE QUALITY METRICS In the last few years, many works have incorporated saliency maps into image quality metrics using simple combination rules. In this work, our goal is to investigate the benefit of incorporating saliency maps obtained using computational models. We incorporate the information from the saliency map into three different image quality (fidelity) metrics: Mean Square Error (MSE), Peak signal-to-noise ratio (PSNR), and Structural Similarity (SSIM) [13]. The MSE metric is calculated using the following equation: M N x=1 y=1 MSE = (I o(x,y) I t (x,y)) 2, (1) M N where I o (x,y) is the reference image pixel, I t (x,y) is the test image pixel, and M N is the size of the images. PSNR is calculated using the following equation PSNR = 20log MAX i MSE, (2) where MAX i is the highest intensity level of the pixels. For colored images, the total MSE and PSNR is calculated by taking the average of the MSE and PSNR for the three color planes. The third metric, SSIM, is a more complex and robust fullreference (FR) image quality metric. The general equation for SSIM is: (2µ o µ t +C 1 )(2σ ot +C 2 ) SSIM(I o,i t ) = (µ 2 o +µ2 t +C 1)(σo 2 +σ2 t +C 2), (3) where µ is the average intensity, σ is the standard deviation, andσ ot is the covariance between the original image (I o ) and the test image (I t ). The variables C 1, C 2, C 3 are control constants used to avoid problems when the denominator reaches values close to zero. In order to combine the saliency and quality models, we used an approach similar to the approach used by Liu and Heynderickx [7]. Since all the metrics considered generate distortion maps that have the same size as the saliency maps, the points in the saliency maps are used as weights for the distortion maps generated by the quality metrics. In the case of MSE, the integration model is obtained by multiplying each pixel in the MSE-distortion map by the corresponding pixel in the saliency map. In other words, the modified saliency-based MSE (SalMSE) is given by: M N x=1 y=1 SalMSE = MSE(x,y) Sal(x,y)) M N x=1 y=1 Sal(x,y), (4) where MSE(x, y) is the MSE-distortion map, Sal(x, y) is the saliency map pixel, and x and y are the horizontal and vertical coordinates. In the same way, the modified saliency-based PSNR (SalPSNR) is given by the following equation. SalPSNR = 20log MAX i SalMSE (5) and the modified saliency-based SSIM (SalSSIM) formula is given by M N x=1 y=1 SalSSIM = (SSIM map(x,y) Sal(x,y)) M N x=1 y=1 Sal(x,y), where SSIM map (x,y) is the SSIM distortion map. IV. SIMULATION RESULTS To study the effect of integrating visual information into image quality metrics, we tested the quality metrics with and without incorporating saliency maps. We used the images of the LIVE image quality assessment database [12], which contains the following degradations: JPEG, JPEG2000, Gaussian Blur, Fast Fading, and White Noise. In Fig. 2, we show images with examples of very strong degradations with the purpose of exemplifying each type of distortion. In Tables I, II, III, IV, and V, we present the Pearson and Spearman correlation coefficients obtained for the quality metrics described in the previous sections and tested on images with JPEG, JPEG2000, Gaussian Blur, Fast Fading, and White Noise degradations, respectively. The tables show the correlation values for the metrics with no saliency map ( none : MSE, PSNR, and SSIM) and with the incorporation of the three saliency maps considered in this work (Itti s and Achanta s computational attention models and the subjective saliency maps from TUD). The columns 4, 6, and 8 in each table show the differences in correlation values obtained by incorporating the saliency map into the quality metric. A positive difference means that an improvement in performance was obtained, while a negative value means a deterioration in the performance. Differences below are, probably, not significant. Notice that the Spearman correlation values are always higher than the Pearson correlation values. The Pearson correlation is more sensitive to a linear relationship between the two variables, while the Spearman correlation emphasizes the monotonicity of the relationship between the variables. (6)

(a) Original (b) JPEG (c) JPEG2000 (d) Gaussian blur (e) fast fading (f) white noise Fig. 2. Sample images illustrating degradations of LIVE image quality assessment database [12].

The results in Tables I-V show that the performance of metrics with incorporated saliency maps depends on the type of saliency map used and also on the type of distortion.

4 (a) Original (b) JPEG (c) JPEG2000 (d) Gaussian blur (e) fast fading (f) white noise Fig. 2. Sample images illustrating degradations of LIVE image quality assessment database [12]. Therefore, we may conclude that the relationship between the objective and subjective scores is monotonic, but not necessarily linear. The results in Tables I-V show that the performance of metrics with incorporated saliency maps depends on the type of saliency map used and also on the type of distortion. For example, in the case of JPEG2000 distortions, the SSIM performance only improved (slightly) when subjective saliency maps were used. For Gaussian Blur and Fast Fading distortions, the addition of saliency maps improved the SSIM performance, with Achanta s model representing the highest gain, followed by the subjective saliency maps. For JPEG distortions, the SSIM performance did not improved at all. With the addition of the saliency maps, the correlation coefficients corresponding to MSE and PSNR improved for all tested distortions, except for White Noise. For JPEG and JPEG200 distortions, the addition of the subjective saliency maps incurred in the highest difference in performance. For Gaussian Blur and Fast Fading, in most cases the subjective saliency maps incurred in the highest correlation values, followed by Achanta s model. We can also notice that the positive correlation difference values for MSE and PSNR are always higher than those for SSIM. This is expected, since MSE and PSNR are extremely simple metrics with a lot of room for improvement. It is worth pointing out that for the degradation White Noise, none of the metrics showed a significant improvement in performance with the addition of saliency maps, either subjective or objective. This is in agreement with other studies that show that the importance of saliency maps depends on the type of distortion [5]. As can be seen in Fig. 2(f), noise completely changes the saliency areas in an image. V. CONCLUSIONS AND FUTURE WORK In this work, we have investigated the benefit of incorporating saliency maps obtained with computational visual attention models. More specifically, we have compared the performance of quality metrics using saliency maps obtained with computational visual attention models versus quality metrics using subjective saliency maps. We have also discussed the influence of different types of distortion on the saliency maps (objective and subjective) and the performance of the tested metrics. Our results show that the saliency information has improved the results of the two simple fidelity metrics tested. For the more complex metric, SSIM, the addition of saliency has not always represented an improvement in performance. The results have also shown that subjective saliency maps have incurred in a slightly higher improvement in performance than saliency maps generated by computational models. Nevertheless, the results depend on the distortion type, with White Noise and JPEG distortions presenting the lowest levels of improvement. Further studies are needed in order to understand how visual attention affects quality and interacts with the different types of distortions. More importantly, better ways of incorporating visual attention to the design of quality metrics are urgently needed. VI. REFERENCES [1] Zhou Wang and Alan C Bovik. Mean squared error: Love it or leave it? IEEE Signal Processing Magazine, 1(January):98 117, [2] S. Chikkerur, V. Sundaram, M. Reisslein, and L.J. Karam. Objective video quality assessment methods: A classification, review, and performance comparison. Broadcasting, IEEE Transactions on, 57(2): , june 2011.

5 Table I. Pearson and Spearman correlation coefficient values between objective (SSIM, PSNR, and MSE) and subjective measures, for images with JPEG distortions. JPEG Map SSIM no saliency PSNR no saliency MSE no saliency Pearson CC none Achanta Itti Subjective Spearman CC none Achanta Itti Subjective Table II. Pearson and Spearman correlation coefficient values between objective (SSIM, PSNR, and MSE) and subjective measures, for images with JPEG2000 distortions. JPEG2000 Map SSIM no saliency PSNR no saliency MSE no saliency Pearson CC none Achanta Itti Subjective Spearman CC none Achanta Itti Subjective Table III. Pearson and Spearman correlation coefficient values between objective (SSIM, PSNR, and MSE) and subjective measures, for images with Gaussian Blur distortions. Gaussian Blur Map SSIM no saliency PSNR no saliency MSE no saliency Pearson CC none Achanta Itti Subjective Spearman CC none Achanta Itti Subjective Table IV. Pearson and Spearman correlation coefficient values between objective (SSIM, PSNR, and MSE) and subjective measures, for images with Fast Fading distortions. Fast Fading Map SSIM no saliency PSNR no saliency MSE no saliency Pearson CC none Achanta Itti Subjective Spearman CC none Achanta Itti Subjective

6 Table V. Pearson and Spearman correlation coefficient values between objective (SSIM, PSNR, and MSE) and subjective measures, for images with White Noise distortions. White Noise Map SSIM no saliency PSNR no saliency MSE no saliency Pearson CC none Achanta Itti Subjective Spearman CC none Achanta Itti Subjective [3] C Oprea, I Pirnog, C Paleologu, and M Udrea. Perceptual Video Quality Assessment Based on Salient Region Detection. In Telecommunications, AICT 09. Fifth Advanced International Conference on, pages , May [4] J. Redi, Hantao Liu, P. Gastaldo, R. Zunino, and I. Heynderickx. How to apply spatial saliency into objective metrics for jpeg compressed images? In Image Processing (ICIP), th IEEE International Conference on, pages , nov [5] U. Engelke, H. Kaprykowsky, H.-J. Zepernick, and P. Ndjiki-Nya. Visual attention in quality assessment. Signal Processing Magazine, IEEE, 28(6):50 59, nov [6] Hantao Liu and Ingrid Heynderickx. Studying the added value of visual attention in objective image quality metrics based on eye movement data. [7] Hantao Liu and I. Heynderickx. Studying the added value of visual attention in objective image quality metrics based on eye movement data. In Image Processing (ICIP), th IEEE International Conference on, pages , nov [8] Laurent Itti and Christof Koch. Computational modelling of visual attention. Nature Reviews Neuroscience, 2(3): , [9] Radhakrishna Achanta, Francisco Estrada, Patricia Wils, and Sabine Susstrunk. Salient region detection and segmentation. In Antonios Gasteratos, Markus Vincze, and John Tsotsos, editors, Computer Vision Systems, volume 5008 of Lecture Notes in Computer Science, pages Springer Berlin / Heidelberg, [10] H. Liu and I. Heynderickx. Tud image quality database: Eye-tracking release 1, [11] Dirk Walther and Christof Koch. Modeling attention to salient proto-objects. Neural Networks, 19: , [12] H.R. Sheikh, Z.Wang, L. Cormack, and A.C. Bovik. Live image quality assessment database release 2. [13] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): , 2004.

Computational Cognitive Science. The Visual Processing Pipeline. The Visual Processing Pipeline. Lecture 15: Visual Attention.

Computational Cognitive Science. The Visual Processing Pipeline. The Visual Processing Pipeline. Lecture 15: Visual Attention. Lecture 15: Visual Attention School of Informatics University of Edinburgh keller@inf.ed.ac.uk November 11, 2016 1 2 3 Reading: Itti et al. (1998). 1 2 When we view an image, we actually see this: The