What we see is most likely to be what matters: Visual attention and applications

Transcription

1 What we see is most likely to be what matters: Visual attention and applications O. Le Meur P. Le Callet November 9,

2 Introduction Yarbus [Yarbus, 1967] demonstrated how eye movements changed depending on the question asked of the subject. 1 No question asked 2 Judge economic status 3 What were they doing before the visitor arrived? 4 What clothes are they wearing? 5 Where are they? 6 How long is it since the visitor has seen the family? 7 Estimate how long the unexpected visitor had been away from the family. Each recording lasted 3 minutes. 2

3 1 Introduction 2 3 Hierarchical models Statistical models Some examples 4 Building a ground truth... Metrics Limitations 5 Retargeting Compression Others 6 3

4 1 Introduction

5 For the computational modelling, two `schools' can be considered: One based on the assumption that there is an unique saliency map [Koch et al., 1985][Li, 2002]: Denition (saliency map) A topographic representation that combines the information from the individual feature maps into one global measure of conspicuity. This map can be modulated by a higher-level feedback. A comfortable view for the computational modelling... Our dierent senses Computer Saliency map Memory Eye Movements 5

6 For the computational modelling, two `schools' can be considered: There exist multiple saliency maps (distributed throughout the visual areas) [Tsotsos et al., 1995]). Many candidate locations for a saliency map: Primary visual cortex[li, 2002] Lateral IntraParietal area (LIP) [Kusunoki et al., 2000] Medial Temporal cortex [Treue et al., 2006] `At each level, saliency can thus be used as a gain control mechanism to spatially gate relevant information for the next processing level. From [Van Rullen, 2003]. Priority map [Fecteau et al., 2006] 6 From [Van Rullen, 2003].

7 Hierarchical models Statistical models Some examples 1 Introduction 2 3 Hierarchical models Statistical models Some examples

8 Hierarchical models Hierarchical models Statistical models Some examples Itti's model [Itti et al., 1998], probably the most known... Based on the Koch and Ullman's scheme Hierarchical decomposition (Gaussian) Early visual features extraction in a massively parallel manner Center-surround operations Pooling of the feature maps to form the saliency map 8

9 Hierarchical models Hierarchical models Statistical models Some examples Le Meur's model [Le Meur et al., 2006], an extension of Itti's model... Based on the Koch and Ullman's scheme Light adaptation and Contrast Sensitivity Function Hierarchical and oriented decomposition (Fourier spectrum) Early visual features extraction in a massively parallel manner Center-surround operations on each oriented subband Enhanced pooling [Le Meur et al., 2007] of the feature maps to form the saliency map Other models in the same vein: [Marat et al., 2009], [Bur et al., 2007]... 9

10 Statistical models Hierarchical models Statistical models Some examples Statistical models are based on a probabilistic framework taken their origin in the information theory. Denition (Self-information) Self-information is a measure of the amount information provided by an event. For a discrete X r.v dened by A = {x 1,..., x N } and by a pdf, the amount of information of the event X = x i is given by: I (X = x i ) = log 2 p(x = x i ), bit/symbol Properties if p(x = x i ) < p(x = x j ) then I (X = x i ) > I (X = x j ) p(x = x i ) 0, I (X = x i ) + The saliency of visual content could be deduced from the self-information measure. Self-information rareness, surprise, contrast... 10

11 Statistical models Hierarchical models Statistical models Some examples First model resting on this approach has been proposed in 2003 [Oliva et al., 2003]: S(x) = 1 p(v l (x)), where v l is a feature vector (48 dimensions), the pdf is computed over the whole image. Bruce in 2004 [Bruce, 2004] and 2009 [Bruce et al., 2009] modied the previous approach by using the self-information locally on independent coecients (projection on a given basis). From [Bruce et al., 2009] Other models in the same vein: [Mancas et al., 2006], [Zhang, 2008]. 11

12 Some examples Hierarchical models Statistical models Some examples (a) Original (b) Itti (c) Le Meur (d) Bruce (e) Zhang 12

13 Building a ground truth... Metrics Limitations 1 Introduction Building a ground truth... Metrics Limitations

14 Ground truth Building a ground truth... Metrics Limitations (a) SM,FD (b) SM,FN (c) Data A good review of parsing algorithm in [Salvucci et al.,2000]. On the web (for natural images): Bruce's database Le Meur's database (28 color pictures) Rajashekar's database 14

15 Metrics Building a ground truth... Metrics Limitations Dierent methods are used to assess the degree of similarity between the ground truth and the prediction: Saliency-map-based method: (a) Original (b) Exp. SM (c) Predicted SM ROC (Receiver Operating Characteristic). Each pixel is labeled as xated or not. Several thresholds are used (AUC (Area Under Curve)). The higher the AUC, the better is the prediction, with 0.50 indicating random performance and 1.00 denoting perfect performance. KL-Divergence, Linear correlation coecient... 15

16 Metrics Building a ground truth... Metrics Limitations Fixation point-based method: NSS (Normalized Scanpath salience) gives the degree of correspondence between human xation locations and predicted saliency maps [Parkhurst et al.,2002],[peters et al., 2005]. 1 Each saliency map is normalized to have zero mean and one unit standard deviation. 2 Extraction of the predicted saliency at a given human xation point. 3 Average of the previous values. From [Peters et al., 2005] NSS = 0: random performance; NSS >> 0: correspondence between human xation locations and the predicted salient points: NSS << 0: anti-correspondence. 16

17 Limitations Building a ground truth... Metrics Limitations Mostly inherent to the building of the ground truth and to the experimental setting... 1 Several parameters can have a signicant impact on the results: the task subjects performed. What is the question we should asked? the nature of the stimuli viewed the apparatus used to record the eye movements a signicant central bias cognitive constraints (higher-level goals, prior knowledge, expectations...) 2 Has the xation the same meaning? 17

18 Limitations and perspectives Building a ground truth... Metrics Limitations The assessment of computational model mainly rests on the analysis of individual xations. But... Is there a focal-ambient dichotomy? [Unema et al., 2005][Follet et al., 2009]. Does every xation convey the same processing? Would it be possible to categorize xations as attentional xations, semantic xations...? A promising solution to disentangle dierent processes is called the EFRP (Eye-Fixation-Related Potentials) [Baccino et al., 2005]. 18 Courtesy of T. Baccino. Measuring ERP (Event-Related Potential) and EM conjointly to track the cognitive processes.

19 Retargeting Compression Others 1 Introduction Retargeting Compression Others 6 19

20 Retargeting Compression Others Retargeting (or reframing): Principle [Fan et al.,2003][chamaret et al., 2008]: Examples: From [Le Meur et al., 2006]. 20

21 Retargeting Compression Others Video compression: To allocate more bit rate to the salient areas than to others (adaptive quantization). Below, the macroblock cost distribution for a H.264 coding and for a saliency-based H.264 coding: A spatial blur is applied to the input frames such that the non regions of interest are strongly blurred [Itti, 2004]. From [Itti, 2004]. 21

22 Retargeting Compression Others Quality assessment: to weight the distortion of an area by its level of interest to have a better prediction of the quality score[ninassi et al., 2007]. (a) 1 impaired patch Structured document evaluation: (b) 3 impaired patches (a) Original images from web sites (b) Mouse-tracking map (From [Mancas, 2009]) Others: robot navigation, super-resolution, advertising... 22

23 1 Introduction

24 How to model a network of saliency/priority maps? How do they cooperate? How to evaluate accurately a computational model? Do the visual xations convey the same information? Eye movements are the results of a multiple source guidance [Henderson, 2003]. 24

25 24 [Baccino et al., 2005] T. Baccino, Y. Manunta. Eye-xation-related potentials: insight into parafoveal processing. Journal Of Psychophysiology, 19(3), pp , [Bruce, 2004] N.D.B. Bruce. Image Analysis through local information measures. International Conference on Pattern Recognition, [Bruce et al., 2009] N.D.B. Bruce, J.K. Tsotsos. Saliency, attention and visual search: an information theoretic approach. Journal of Vision, 9(3), pp. 1-24, [Bur et al., 2007] A. Bur, H. Hügli. Dynamic visual attention: competitive versus motion priority scheme. Proc. ICVS Workshop on Computational Attention &, [Chamaret et al., 2008] C. Chamaret, O. Le Meur. Attention-based video reframing: validation using eye-tracking. ICPR, [Henderson, 2003] J. M. Henderson, Human gaze control during real-world scene perception, trends in cognitive sciences, Vol. 7, N. 11, [Itti et al., 1998] L. Itti, C., Koch, E., Niebur. A model for saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Analysis and Machine Intelligence, 20, pp , [Itti, 2004] L. Itti. Automatic Foveation for Video Compression Using a Neurobiological Model of Visual Attention, IEEE Transactions on Image Processing, 13(10), pp , [Kusunoki et al., 2000] M., Kusunoki, J. Gottlieb, M.E. Goldberg. The lateral intraparietal area as a salience map: The representation of abrupt onset, stimulus motion, and task relevance. Vision Research, 40(10), pp , 2000.

26 [Koch et al., 1985] C. Koch, S. Ullman. shifts in selective visual attention: towards the underlying neural circuitry. Human Neurobiology 4, pp , [Fan et al.,2003] X. Fan, X. Xie, W. Ma, H. Zhang, H. Zhou. Visual attention based image browsing on mobile devices, ICME, pp , [Fecteau et al., 2006] J.H. Fecteau, D. P. Munoz. Salience, relevance, and ring: a priority map for target selection, Trends in cognitive sciences, 10, pp , [Follet et al., 2009] B. Follet, O. Le Meur, T. Baccino. Relationship between coarse-to-ne process and ambient-to-focal visual xations. ECEM, [Le Meur et al., 2007] O. Le Meur, P. Le Callet, and D. Barba. Predicting visual xations on video based on low-level visual features. Vision Research, 47(19), pp , [Le Meur et al., 2006] O. Le Meur, P. Le Callet, D. Barba and D. Thoreau. A coherent computational approach to model the bottom-up visual attention. IEEE Trans. on Pattern Analysis and Machine Intelligence, 28(5), pp , [Li, 2002] Z. Li. A saliency map in primary visual cortex. Trends Cognitive Sciences 6(1), pp. 9-16, [Mancas et al., 2006] M. Mancas, C. Mancas-Thillou, B. Gosselin, B. Macq. A rarity-based visual attention map - application to texture description. ICIP, [Mancas, 2009] M. Mancas. Relative Inuence of Bottom-Up and Top-Down Attention. Attention in Cognitive Systems, Lecture Notes in Computer Science,

27 [Marat et al., 2009] S. Marat, T. Ho Phuoc, L. Granjon, N. Guyader, D. Pellerin, A. Guerin-Dugue. Modelling Spatio-Temporal Saliency to Predict Gaze Direction for Short Videos, International Journal of Computer Vision, 82(3), pp , [Ninassi et al., 2007] A. Ninassi, O. Le Meur, P. Le Callet, D. Barba. Does where you gaze on an image aect your perception of quality? Applying to image quality metric, ICIP [Oliva et al., 2003] A. Oliva, A. Torralba, M. S. Castelhano, and J. M. Henderson. Top-down control of visual attention in object detection. ICIP, vol. 1, pp , [Parkhurst et al.,2002] D. Parkhurst, K. Law, E. Niebur. Modelling the role of salience in the allocation of overt visual attention. Vision Research, 42(1), pp , [Peters et al., 2005] R. Peters, A. Iyer, L. Itti, C. Koch. Components of bottom-up gaze allocation in natural images. Vision Research, [Salvucci et al.,2000] D.D. Salvucci, J.H. Goldberg. Identifying xations and saccades in eye-tracking protocols. ETRA, pp , [Treue et al., 2006] S. Treue, J. C. Martinez-Trujillo. Visual search and single-cell electrophysiology of attention: Area MT, from sensation to perception. Visual Cognition, 14(4), pp , [Tsotsos et al., 1995] J.K., Tsotsos, S. Culhane, W., Wai, Y., Lai, N., Davis, F., Nuo. Modeling visual attention via selective tuning, Articial Intelligence 78(1-2), pp ,

28 [Unema et al., 2005] P. Unema, S. Pannasch, M. Joos and B. Velichkovsky. Time course of information processing during scene perception: the relationship between saccade amplitude and xation duration. Visual Cognition, 12(3), pp , [Van Rullen, 2003] R. Van Rullen. Saliency and spike timing in the ventral visual pathway. Journal Of Physiology, [Yarbus, 1967] A. L. Yarbus. Eye Movements and vision. New York: Plenum, [Zhang, 2008] L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W. Cottrell. SUN: A Bayesian framework for saliency using natural statistics. Journal of Vision, vol. 8, no. 7, pp. 1-20,