Computational Saliency Models Cheston Tan, Sharat Chikkerur

Transcription

1 Computational Salieny Models Cheston Tan, Sharat Chikkerur

2 Outline Salieny 101 Bottom up Salieny Model Itti, Koh and Neibur, A model of salieny-based visual attention for rapid sene analysis. IEEE PAMI, 2011, 98 Itti and Koh, A salieny-based searh mehanism for overt and overt shifts of visual attention, Vision Researh, 00 Contextual Guidane Model : Bottom up + Top Down A. Torralba, A. Oliva, M. Castelhano and J. M. Henderson, Contextual guidane of attention and eye movements in real-world senes: the role of global features in objet searh. Psyhologial Review, 06 A. Torralba, Modeling global sene fators in attention", JOSA, 207, 03 A. Torralba, "Contextual Priming for Objet Detetion", IJCV, 532, 03 Summary Demo Comparison of bottom up salieny models

3 Salieny 101 What is salieny? But first, what is Attention? Biologial visual system proess omplex senes serially despite parallel omputation Speifi parts of the sene are attended by overt or overt attention eye movements. What drives attention? Salieny! Bottom up salieny Driven by sene features, Fast! Top down salieny Driven by volitional ontrol, Slow Biologial Evidene Believed to be loated in posterior pareital ortex,v4 Spike modulation observed in V1,V2,V4 Luk et al., 97; Reynolds et al 00

4 Bottom up salieny

5 Top down salieny- Spatial Modulation Torralba

6 Top down salieny-feature modulation Navalpakkam and Itti

7 Computational Salieny Model Bottom up salieny Intuition: Unusual/Salient items should draw our attention and be easy to searh for. Unusual targets? :A target whose features are outliers to the loal distribution of features. How do we detet outliers? Expliit statistial Model, Ruth Rosenholtz et al., Torralba et al. Approximate estimation with enter surround filters, Itti et al. Top down salieny Intuition: Searhing is task oriented Task priors hange relevane of loations and features How are the priors manifested? Modulation : additive boosting Gain ontrol: multipliative boosting/supression Luk et. al, 97

8 Outline Salieny 101 Bottom up Itti, Koh and Neibur, A model of salieny-based visual attention for rapid sene analysis. IEEE PAMI, 2011, 98 Itti and Koh, A salieny-based searh mehanism for overt and overt shifts of visual attention, Vision Researh, 00 Top Down A. Torralba, A. Oliva, M. Castelhano and J. M. Henderson, Contextual guidane of attention and eye movements in real-world senes: the role of global features in objet searh. Psyhologial Review, 06 A. Torralba, Modeling global sene fators in attention", JOSA, 207, 03 A. Torralba, "Contextual Priming for Objet Detetion", IJCV, 532, 03 Summary Demo Comparison of bottom up salieny models

9 Itti and Koh Algorithm Feature maps Compute strength of individual features Conspiuity maps Compute salieny of individual features through enter surround Salieny maps Combines salieny from different features Inhibition of return Models overt attention Itti et al.

10 Example Itti et al.

11 Feature maps,,, 2 2, 2 2, 2 σ σ σ σ Y B G R b g r g r Y g r b B b r g G b g r R *, *, G I I G I I I I I I b g r I σ σ σ σ σ θ θ σ *, Gabor I O Itti et al.

12 Example Features Red/Green Blue/Yellow Intensity Orientation0

13 Center surround and normalization {3,4}, {2,3,4},,,,,,,, + δ δ θ θ θ s s O O s O s B s Y Y B s BY s R s G G R s RG s I I s I

14 Example: Center surround Red/Green Blue/Yellow Intensity Orientation0

15 Example: Conspiuity maps [ ] θ,θ,,,, s O O s BY s RG C s I I s s s Ν N Ν Ν Ν Itti et al.

16 Example: Conspiuity maps

17 Salieny maps S 1 3 N I + N C + N O Itti et al.

18 IOR: Inhibition of return Attention shifts are modeled using IAF neurons Saliene map feeds into a WTA neural network Attention is first shifted to the most salient loation The region is onsequently suppressed, and attention is shifted to the next most salient loation The FOA is shifted in simulated time to model human attention mehanism Itti et al.

19 Example: IOR

20 Time line: Bottom-up attention Koh, Ullman 1985 Itti, Koh, Niebur 1998 Itti and Koh 2001 V. Navalpakkam and Itti, 2005 D.Walter And Koh,2006 Courtesy, D.Walter

21 Outline Salieny 101 Bottom up Itti, Koh and Neibur, A model of salieny-based visual attention for rapid sene analysis. IEEE PAMI, 2011, 98 Itti and Koh, A salieny-based searh mehanism for overt and overt shifts of visual attention, Vision Researh, 00 Top Down A. Torralba, A. Oliva, M. Castelhano and J. M. Henderson, Contextual guidane of attention and eye movements in real-world senes: the role of global features in objet searh. Psyhologial Review, 06 A. Torralba, Modeling global sene fators in attention", JOSA, 207, 03 A. Torralba, "Contextual Priming for Objet Detetion", IJCV, 532, 03 Summary Demo Comparison of bottom up salieny models

22 Contextual Guidane model Salieny and global-ontext features omputed in parallel, feedforward manner Searh task exerts top-down ontrol

23 Contextual Guidane model Salieny map modulated by ontextual information Probability of target presene by integration of task onstraints and global and loal image information

24 Contextual Guidane model

25 Statistial approah to salieny Statistially distinguishable from bakground Loations differing from neighboring regions more informative Rare image features more likely to be objets

26 Statistial approah to salieny Eah olor hannel passed through bank of multisale oriented filters e.g. Steerable pyramid to extrat loal features Model distribution of features using multivariate powerexponential distribution Normalization onstant, k Exponent α Mean η Covariane

27 Statistial approah to salieny Eah olor hannel passed through bank of multisale oriented filters e.g. Steerable pyramid to extrat loal features Model distribution of features using multivariate powerexponential distribution Normalization onstant, k Exponent α Mean η Covariane Gaussian

28 Statistial approah to salieny

29 Comparison

30 Comparison

31 Comparison

32 Comparison

33 Comparison

34 Comparison

35 Comparison

36 Comparison

37 Comparison

38 Comparison

39 Comparison

40 Comparison

41 Comparison

42 Comparison

43 Summary Salieny is the underlying mehanism that drives attention Salieny: bottom-up or top-down Bottom up: feature driven Top down: Task driven Computing bottom-up Detets outliers in feature spae Itti et al. algorithm- uses enter surround filters Torralba et al. expliitly model statistis of features Rosenholtz gaussian modeling of features Comparison

44 Thank You

45 Biologial Plausibility Pop-out Searh time/false positives is independent of the number of distrators Searh Searh time inreases linearly with the number of distrators Performs better than humans!