Adding Shape to Saliency: A Computational Model of Shape Contrast

Transcription

1 Adding Shape to Saliency: A Computational Model of Shape Contrast Yupei Chen 1, Chen-Ping Yu 2, Gregory Zelinsky 1,2 Department of Psychology 1, Department of Computer Science 2 Stony Brook University THE EYE COG LAB EYE MOVEMENTS AND VISUAL COGNITON

2 Attention Used to prioritize information in space Computationally modeled as Priority Maps (Zelinsky & Bisley, 2015) Top-down, goal-driven: Target Maps (Zelinsky, 2008) Bottom-up, stimulus-driven: Saliency Maps

3 Saliency The bottom-up, image-based distinctiveness of an object (Itti & Koch, 2001) The contrast between the features of objects and the features of their neighbors There are many physical features (color, intensity, orientation, )

4 Saliency Map Itti & Koch (1998) Fine scales Coarse scales - = Center-surround contrast Itti & Koch, 1998

5 MIT300 Saliency Benchmark

6 MIT300 Saliency Benchmark

7 GBVS Model Graph-Based Visual Saliency (Harel, Koch & Perona, 2006) Compute individual feature maps and combine them to form a master map Define Markov chains over image maps, and treat the equilibrium distribution over map locations as saliency values

8 What features guide attention? Undoubted features - Color - Orientation - Motion - Size Probable features - Shape - Depth - Closure - Wolfe, & Horowitz, (2004). What attributes guide the deployment of visual attention and how do they do it?. Nature reviews neuroscience

9 Study Goals Promote shape to a certain basic feature Show how shape contrast can be computed and added into a saliency map model Describe a Proto-Object Model that uses shape contrast to better predict fixations in the context of scene viewing

10 Adding Shape to Itti-Koch Shape

11 Methods Define guidance as Immediate Target Fixations (ITFs), the proportion of trials in which the first fixated object was the target Proportion of First Fixated Object Chance Target Distractors Yang & Zelinsky, 2009

12 Methods Define guidance as Immediate Target Fixations (ITFs), the proportion of trials in which the first fixated object was the target Targets and distractors had the same features (same color, intensity, size, and orientation histograms), and only differed with respect to shape

13 0 Target 1 Distractors 2 Target-Distractor Shape Similarity 3 4 Rotation Sample Display

14 Methods Shape Similarity: 5 levels Rotation: 3 levels Object size: 4.7, square Between-object distance: > 2 Set size: 10 Object distance from center: > 4.7 Target present: 100% Distractors: Homogenous

15 Methods Shape Similarity: 5 levels Rotation: 3 levels Object size: 4.7, square Between-object distance: > 2 Set size: 10 Object distance from center: > 4.7 Target present: 100% Distractors: Homogenous

16 Procedure N = 15 5 x 3 conditions 30 trials/condition Randomly interleaved Fixation Time Fixate & Respond 35 47

17

18

19

20

21

22

23 Predictions Recorded eye movements (EyeLink 1000, tower) and calculated the proportion of Immediate Target Fixations (ITFs) for each condition If shape is not a guiding feature of attention, then ITF should be at chance (i.e. ITF = 0.1, given a set size of 10) If ITF is significantly above chance, this would be evidence that shape is guiding attention in this experiment

24 Results Immediate Fixations Immediate Fixations n.s. n.s. * * * * * * * * * * * * * Condition

25 Example Saliency Maps Itti-Koch Saliency Map Original Display GBVS Saliency Map

26 Results Models vs. Human

27 Shape Distance Align two objects by their centroids Calculate their Jaccard Index A B

28 Shape Distance Align two objects by their centroids Calculate their Jaccard Index A B

29 Shape Distance Align two objects by their centroids Calculate their Jaccard Index A B

30 Shape Distance Align two objects by their centroids Calculate their Jaccard Index A B = J(A,B)

31 Shape Distance Align two objects by their centroids Calculate their Jaccard Index Shape Distance = 1- Jaccard Index A B J(A,B) = Shape Distance =

32 Target Shape Distance for the 15 targetdistractor pairs Distractors

33 Adding Shape to Itti-Koch Itti-Koch Saliency Map Original Display Itti-Koch + Shape Saliency Map Shape Feature Map

34 Adding Shape to Itti-Koch Itti-Koch Saliency Map Original Display Itti-Koch + Shape Saliency Map Shape Feature Map

35 Adding Shape to GBVS GBVS Saliency Map Original Display GBVS + Shape Saliency Map Shape Feature Map

36 Results Models vs. Human Immediate Target Fixations Models Human

37 Results Models vs. Human Immediate Target Fixations Models 1 / Time-to-Target (1/s) Human Human

38 Results Models vs. Human Immediate Target Fixations Models 1 / Time-to-Target (1/s) Human Human

39 Scene Viewing Proto-objects

40 Scene Viewing Proto-objects Proto-objects (POs) are pre-attentively available mid-level representations of visual shape Quantified using our PO model (Yu, Samaras, & Zelinsky, 2014, JoV) Superpixel Segmentation Clustering and Merging

41 Scene Viewing Proto-objects Small PO surrounded by larger POs having very different shapes Compute shape contrast between every PO and its neighbors.

42 Scene Viewing Proto-objects Small PO surrounded by larger POs having very different shapes Compute shape contrast between every PO and its neighbors.

43 Scene Viewing Proto-objects Itti-Koch Saliency Map Original Display Itti-Koch + Shape Saliency Map Shape Feature Map

44 Scene Viewing Methods 384 images of scenes from SUN09 12 subjects freely viewed each for 3 seconds, and each viewing was followed by a memory test Scene 3s Patch Present or absent? Time

45 Example Saliency Map I-K + Shape Saliency Map I-K Saliency Map Red dots: Human Fixations AUC: Area under ROC curve

46 Example Saliency Map I-K + Shape Saliency Map I-K Saliency Map Red dots: Human Fixations AUC: Area under ROC curve

47 Scene Viewing Model Evaluation AUC (Area Under ROC Curve) NSS (Normalized Scanpath Saliency) Our Dataset (SUN09) AUC NSS Itti-Koch I-K + Shape

48 Scene Viewing Model Evaluation AUC (Area Under ROC Curve) NSS (Normalized Scanpath Saliency) MIT ICCV Dataset AUC NSS Itti-Koch I-K + Shape

49 Conclusions Shape contrast, like other basic features, is preattentively available to guide behavior. Our Shape+Saliency Model outperformed shapeless versions of saliency models, and it did this for both simple and complex stimuli. Our work argues for the existence of a mid-level visual representation of shape that is preattentively available to guide behavior. Our broad goal is to quantify this representation and its influence on behavior in terms of a computationally explicit Proto-Object Model.

50 Future Plans More experiments using simple stimuli to study shape saliency (i.e., distractor heterogeneity) Tease apart the roles of shape contrast from rotation and size contrast in guiding behavior. Build a saliency model based on a proto-object representation rather than features at the V1 level

51 Thank You! Thanks to everyone in the Eye Cog Lab! Gregory Zelinsky Hossein Adeli Chen-Ping Yu Justin Maxfield THE EYE COG LAB EYE MOVEMENTS AND VISUAL COGNITON