Adaptive visual attention model

Transcription

1 H. Hügli, A. Bur, Adaptive Visual Attention Model, Proceedings of Iage and Vision Coputing New Zealand 2007, pp , Hailton, New Zealand, Deceber Adaptive visual attention odel H. Hügli and A. Bur Institute of Microtechnology, University of Neuchâtel, CH-2000 Neuchâtel, Switzerland Eail: Abstract Visual attention, defined as the ability of a biological or artificial vision syste to rapidly detect potentially relevant parts of a visual scene, provides a general purpose solution for low level feature detection in a vision architecture. Well considered for its universal detection behaviour, the general odel of visual attention is suited for any environent but inferior to dedicated feature detectors in ore specific environents. The goal of the developent presented in this paper is to reedy this disadvantage by providing an adaptive visual attention odel that, after its autoatic tuning to a given environent during a learning phase, perfors siilarly well as a dedicated feature detector. The paper proposes the structure of an adaptive visual attention odel derived fro the saliency visual attention odel. The adaptive odel is characterized by paraeters that act at several feature detection levels. A procedure for autoatic tuning the paraeters by learning fro exaples is proposed. The experiental exaples provided show the feature selection capacity of the generic visual attention odel. The proposed adaptive visual attention odel represents a frae for further developents and iproveents in adaptive visual attention. Keywords: Coputer vision, visual attention, adaptive odel, low-level vision, feature learning, unsupervised learning Introduction Visual attention (VA) is the ability of a vision syste, be it biological or artificial, to rapidly detect potentially relevant parts of a visual scene, on which higher level vision tasks, such as object recognition, can focus. A odel of visual attention that encopasses this ability can thus be used as a universal feature detector that can be used in a generic way in all kind of environents. Well considered for its universal detection behaviour, the generic odel of visual attention is suited for any environent but inferior to dedicated feature detectors in ore specific environents. The goal of the developent presented in this paper is to reedy this disadvantage by providing an adaptive visual attention odel that, after its autoatic tuning to a given environent during a learning phase, perfors siilarly well as a dedicated feature detector. Nuerous coputational odels of visual attention have been suggested during the last two decades. For a ore coplete overview on existing coputational odels of visual attention, the reader is referred to []. The saliency odel of visual attention introduced in [2], is now widely used [3], naely in a nuber of coputer vision applications, including iage copression, chroaticity iage segentation, and object tracking in dynaic environents [4]. Despite this success, the general odel of visual attention is not in a position to copete with other feature extraction systes that were designed specifically for a given environent. The idea of aking the VA odel adaptable was already recognized in previous works. For instance in [5], the authors consider the possibility to change the weights of feature aps by a general leaning procedure, that was however liited to the supervised learning. Also, odel adaptation was investigated for the purpose of cobining both botto-up and topdown control strategies in the visual transforation operation. The research led to the developent of a ore sophisticated dual syste that provides an arbitration echanis between the two control flows [6] [7]. A sipler adaptive visual attention odel is presented and considered in the present paper, which consists of a single paraetric VA processing structure. Depending on the purpose of use, the odel adaptation can be perfored in a supervised learning phase but also in the ore difficult case of unsupervised learning. The presented adaptive VA relies widely on the saliency odel of VA, which basics is recalled in section 2 and which adaptation is treated in section 3. Section 4 is devoted to the paraeter learning procedure in the supervised and also the unsupervised learning scenario, including a special consideration for robust training. Finally, siple VA experients perfored with the generic VA odel and presented in section 5, show that in presence of video sequences 233

2 of very different nature, the aps provided differ enough and justify odel adaptation. 2 Saliency odel of VA This section describes the classical saliency odel of visual attention and identifies eans of adaptation. 2. Conspicuity and saliency aps The saliency odel of visual attention transfors each iage of a video strea into a saliency ap, a scalar ap that accounts for the interest distribution in the original iage. It is based on three ajor principles: Visual attention acts on a ultifeatured input; Saliency of locations is influenced by the surrounding context; The saliency of locations is represented on a scalar saliency ap. Several works have dealt with the realization of this odel i.e. [3]. In this paper, the saliency ap results fro 3 cues (intensity, chroaticity and orientation) and the cues ste fro 7 features. Elsewhere [8], additional features like depth or otion are also considered. The different steps of the odel are illustrated in figure and recalled below. First, several features are extracted fro the scene by coputing the so-called feature aps fro an RGB color iage. The features are: (a) Intensity feature F, (b) Chroatic features coprising the two color opponency coponents red-green F 2, and blueyellow F 2,2 (c) Local orientation features for the four orientations 0, 45, 90 and 35, naed respectively F 3, to F 3,4. In a second step, each feature ap is transfored into its conspicuity ap: the ultiscale analysis decoposes each feature F,j in a set of coponents F,j,k for resolution levels k=...6; the centre-surround echanis produces the ultiscale conspicuity aps M,j,k to be cobined, in a copetitive way, into a single feature conspicuity ap C,j in accordance with: K, j = N(M, j, k ) k = C () where N( ) is a noralization function that siulates both intra-ap copetition and inter-ap copetition aong the different scale aps. In the third step, using the sae copetitive ap integration schee as above, the features are then grouped, according to their nature, into the three cues intensity, chroaticity and orientation. Forally, the cue conspicuity aps are thus: C = N ( C, j ) (2) j J In the final step of the attention odel, the cue conspicuity aps are integrated, by using the schee as above, into a saliency ap S, defined as: 3 S = N( C ) (3) = That is, the scalar ap that accounts for the final interest distribution over the iage. F C, ap fusion level j video feature extraction feature aps F 2, F 2,2 F 3, F 3,2 F 3,3 F 3,4 ultiscale center-surround analysis ap fusion level k C 2, C 2,2 C 3, C 3,2 C 3,3 C 3,4 C C 2 C 3 intensity chroaticity orientation ap fusion level S ap fusion level j saliency Figure : Saliency odel of visual attention 2.2 Map fusion The process of ap fusion ipleents two basic operations: - a noralization of otherwise unscaled inforation channels of arbitrary units - a copetition schee that prootes channels of higher interest and deotes channels of lesser interest The process has been widely studied and several ethods were proposed which forally represent variants for the ipleentation of the N(C) function appearing in equations above [5]. In paper [9], the authors show the superiority of a universal noralization. This ipleentation will therefore be used here in the experiental part. 2.3 Spots of attention The saliency ap S provided by the odel of VA is often not the final result of the iage analysis process. Instead, spots of attention are often used for further needs. They are defined as the iage locations of the saliency axia, forally: X={x i S(x i ) is a local axiu} Alternatively, rearranging the spots in order of decreasing saliency value, we keep usually the set of the N ost iportant spots: X N = {x i S(x i ) is a local axiu, S(x i ) S(x i+ ), i=..n} 234

3 3 Adaptive odel of visual attention 3. Adaptive ap fusion For providing adaptation in the odel, we need a echanis that changes the relative contribution to the final saliency ap of the various features considered. Thus, a feature prootion and deotion echanis is therefore introduced. The basic idea is to odify the contribution to the fused ap of each ap contribution N(C ) by an adaptation weight a, where 0 a such that the adaptive fusion rule that coputes the adaptive saliency S ad becoes: 3 S = a N( C ) (4) ad = 3.2 Paraetrized odel of VA Introducing the adaptive ap fusion process proposed in equation (4) in the visual attention odel of figure, we obtain a odel paraetrized by the weights a. Grouping all paraeters a in atrix A, the view of the new adaptive VA odel is the transfor of a video iage into a saliency ap controlled by the set A={ a } of paraeters. video Adaptive fusion level k VA fusion level j A= a a 2... a 6 a 2 a a a 34 a a 346 a 2 a 22 a 3 a 32 a 33 a 34 level of detail of the features. It requires odification of equation by adding weights a,j,k relative to the ultiscale analysis of the single features. The general expression of A is as illustrated in figure Adaptation principle Adaptation consists in the odification of the paraeters A in order for the odel to fulfil optially the VA task. In the case considered here where adaptation to a given specific visual environent is required, a general solution consisting in arbitrary paraeter odifications and statistical goal evaluation is very tedious and not really feasible. Therefore, we propose an adaptation principle which tends to adjust the weight of each feature in proportion of its average contribution to the successful detection of the environent spots of interest. Given Y={y i }, a set of I spots of interest of the environent, we copute the ean contribution of each conspicuity ap to the final saliency as I b = an( C( yi )) I i= (5) with I i= a = With these contributions, and applying now above defined adaptation principle, it results that adaptation requires odifying the single weights ust tend to the ean percentile contribution: b a (6) b i= fusion level saliency a a 2 a 3 4 Model adaptation The basic idea consists in teaching the adaptive VA syste to produce the expected spots of attention. Two fundaental different approaches are supervised learning and unsupervised learning. Figure 2: Adaptive odel of visual attention Now, consider that adaptation can be introduced at several levels of detail of the feature space, as illustrated in figure 2. A siplest adaptation schea could consider only the cue level, and adaptation is thus restricted to fusion level, and the set of paraeters A would therefore include only paraeters a, a 2 and a 3. Introducing adaptation at the fusion level j, the feature fusion level would require the odification of equation 2 by introduction of ultiplying paraeters a,j,. Finally a full adaptable odel of visual attention would add adaptation at fusion level k, offering also adaptation of the spatial 4. Supervised training In the supervised training approach, a set Y={y} of spots of interest is provided by the supervisor and the scene of interest is analysed by the generic VA odel, providing saliency ap S, as well as all interediate conspicuity aps C. Applying equation 5 to the respective Y and C perits to copute the different channel contributions b which are then used to odify the adaptation paraeters after equation 6, for instance with an adaptation rule: 235

4 Δa b = α a (7) b i= where α, 0<α<, is a coefficient that controls the aount of relative odification of the paraeters. A new step of saliency ap coputation with the odified adaptive VA odel together with the subsequent odification of A ust then be perfored. This process is finally repeated until a stable A is obtained: the learnt adaptive VA odel is thus available. 4.2 Unsupervised training The unsupervised learning approach is solved according to the reinforceent learning paradig: The unbiased syste's response is used as a first solution which is then iproved by reinforcing it. Accordingly, the generic VA is used to produce the first spots of attention X N. These spots are then used like the spots of interest Y in previous approach and an iterative procedure is started siilar to previous description, except that now, the spots of interest Y will be updated at each iteration with the new coputed spots of attention X N. 4.3 Robust unsupervised training The idea is to qualify the spots obtained by the VA odel and reinforce only the best. For instance, by a procedure [4] that, aong all spots of attention detected in a aniated sequence of iages, keeps only the spots which have longer ter existence and discards spots which appear only teporarily in few fraes. 5 Experients In order to support the ideas explained in this paper, we perfored a series of exploratory experients for easuring the response of a generic VA odel in presence of different scenes and illustrating its potential of adaptation. The analysis refers to three short video sequences of 250 fraes and 0 s duration illustrated in figure 3. A generic VA odel using a conventional linear and long ter noralization N(C) schee [9] was used to produce spots of attention fro which the 6 first fro each frae were kept and their conspicuity channel contribution evaluated. The related ean contribution is reported in table. a) football sequence b) indoor sequence c) traffic sequence Figure 3: Exaple fraes of three video sequences and the related 6 spots of attention; the cheese diagra indicates the spot percentile contribution to saliency of intensity (black), chroaticity (red) and orientation (grey) The results reveal clear contribution doinance of chroaticity in the football sequence, orientation doinance in the indoor sequence and a shared doinance of orientation and chroaticity in the traffic sequence. These results are in close agreeent to the subjective analysis of the respective scenes, with the strong green-red contrasted features of the football sequence, the orientation doinated indoor sequence, and less obvious doinance of a single cue in the traffic sequence. Considering the analysis at the next level, i.e. the fusion level j, we report the relative contributions of the chroaticity channels in table 2 and the orientation channels in table 3. For the chroaticity channels, the results show the clear doinance of the red-green channel (b 2, ) in both the football and the traffic sequences, whereas in the indoor sequence, the two channels are ore balanced. Table : ean contribution b (bold) and percentile contribution to the saliency ap b intensity b 2 chroaticity b 3 orientation football % 83.2% 0.7% indoor % 6.8% 89.5% traffic % 5.6% 46.7% 236

5 Table 2: ean contribution b 2,j (bold) and ean percentile contribution to the chroaticity ap b 2, b 2,2 (R-G) (B-Y) football % 9.8% indoor % 34.0% traffic % 9.% Table 3: ean contribution b 3,j (bold) and ean percentile contribution to the orientation ap b 3, (0 ) b 3,2 (45 ) b 3,3 (90 ) b 3,4 (35 ) football % 6.6% 44.7% 4.7% indoor % 8.8% 37.6% 47.2% traffic % 24.0% 2.9% 28.5% For the four orientation channels reported in table 3, the results are only significant for the two sequences doinated by orientation, i.e. the indoor and traffic sequences (see table ). Coparing the two, it appears that the first has a strong doinance of horizontal (90 ) and oblique (35 ) orientation while the second is characterized by a rather hoogeneous distribution of the orientations. These results clearly illustrate that the VA odel not only provides saliency aps and spots of attention, but provides internal conspicuity aps that can be exploited to provide inforation about the contribution and significance of single features. Given the iportant differences of contributing channels in the different sequences, it is expected that the proposed adaptation tunes the VA procedure for enhanced feature detection. 6 Conclusions There is a real interest for an adaptive visual attention (VA) odel that can be autoatically tuned to a given environent. This paper proposes a siple, paraetrized VA odel that can be tuned to perfor specifically for a given environent. The presented adaptive VA relies widely on the saliency odel of VA which adaptation is obtained by weighting of the fusion procedure at the cue level, the feature level and possibly also at the scale level. Supervised learning was considered, but, given the siple learning schee, unsupervised learning is also possible. Special consideration is given to a robust training scenario. Finally, exploratory VA experients perfored with the generic VA odel show that in presence of video sequences of very different nature, the aps provided differ enough and justify odel adaptation. Therefore, the proposed adaptive visual attention odel represents a challenging frae for further investigations and expected iproveents in adaptive visual attention. 7 Acknowledgeents This work is partially supported by the Swiss National Science Foundation under grant FN References [] D. Heinke, G.W. Huphreys, Coputational Models of Visual Selective Attention: A Review., in: G. Houghton (Ed.), Connectionist Models in Psychology, Psychology Press, Hove. England pp [2] C. Koch and S. Ullan. Shifts in selective visual attention: Towards the underlying neural circuitry. Huan Neurobiology, Vol. 4, pp , 985. [3] L. Itti and Ch. Koch. A coparison of feature cobination strategies for saliencybased visual attention systes Proc. SPIE, Vol. 3644, pp , 999. [4] N. Ouerhani, H. Hügli, G. Gruener and A. Codourey, A Visual Attention-Based Approach for Autoatic Landark Selection and Recognition, Lecture Notes in Coputer Science, Springer-Verlag, Vol. 3368, [5] L. Itti, Ch. Koch, and E. Niebur. A odel of saliency-based visual attention for rapid scene analysis. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Vol. 20, No., pp , 998. [6] S. Frintrop. VOCUS: A Visual Attention Syste for Object Detection and Goal- Directed Search Lecture Notes in Artificial Intelligence, Vol-3899, Springer Verlag, [7] B. Rasolzadeh, M. Björkan, J.-O. Eklundh, An attentional syste cobining top-down and botto-up influences, Proc. 2nd Int. Cognitive Vision Workshop, ECCV 2006 Workshop, [8] A. Bur, P. Wurtz, R. Müri, H. Hügli, Motion integration in visual attention odels for predicting siple dynaic scenes, Proc. SPIE, Vol , Feb, 2007 [9] N. Ouerhani, T. Jost, A. Bur, H. Hügli, Cue Noralisation Schees in Saliencybased Visual Attention Models, Proc. Second International Cognitive Vision Workshop, Graz (A), May 3,