Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition

Transcription

1 Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition Stefan Mathe, Cristian Sminchisescu Presented by Mit Shah

2 Motivation Current Computer Vision Annotations subjectively defined Intermediate levels of computation?? 2

3 Motivation Lack of large scale datasets that provide recordings of the workings of the human visual system 3

4 Previous Work... Study of Gaze patterns in Humans A person browsing reddit with the F-shaped pattern 4

5 Previous Work... Study of Gaze patterns in Humans Inter-observer consistency 5

6 Previous Work... Study of Gaze patterns in Humans Inter-observer consistency Bottom-up Features 6

7 Previous Work... Study of Gaze patterns in Humans Inter-observer consistency Bottom-up Features Human Fixations 7

8 Previous Work... Study of Gaze patterns in Humans Inter-observer consistency Bottom-up Features Human Fixations Models of saliency 8

9 Previous Work... Study of Gaze patterns in Humans Inter-observer consistency Bottom-up Features Human Fixations Models of saliency Uses of Saliency maps Action Recognition Object Localization Scene Classification 9

10 Previous Work... Study of Gaze patterns in Humans Inter-observer consistency Bottom-up Features Human Fixations Models of saliency Uses of Saliency maps Previous data sets At most few hundred videos recorded under free viewing conditions 10

11 Contributions... (1) Extended existing large scale datasets Hollywood-2 and UCF Sports 11

12 Contributions... (2) Dynamic consistency and alignment measures AOI Markov Dynamics Temporal AOI Alignment 12

13 Contributions... (3) Training an End-to-End automatic visual action recognition system 13

14 Data Collection... Hollywood-2 Movie Dataset 12 classes 69 movies 823/884 split 487k frames 20 hr Largest and Most challenging dataset Answering phone, driving a car, eating, fighting, etc. 14

15 Data Collection... UCF Sports Action Dataset Broadcast of television channels 150 videos covering 9 sports action classes Diving, golf swinging, kicking, etc.. 15

16 Data Collection... Extending the two data sets Many other Specifications Timings/Durations & Breaks SMI iview X HiSpeed 1250 Tower-Mounted Eye Tracker Context Recognition 19 Humans ed d i v Di 3 into ks s Ta Action Recognition Free Viewing TASKS Recording Environment Recording Protocol 16

17 Static & Dynamic Consistency Action Recognition by Humans Goal & Importance Human errors Co Occurring Actions False Positives Mislabeling Videos 17

18 Static Consistency Among Subjects How well the regions fixated by human subjects agree on a frame by frame basis? Evaluation Protocol 18

19 Static Consistency Among Subjects 19

20 The Influence of Task on Eye Movements SA \ {s} Derive Saliency Maps Predict Fixations of Subject s na prediction scores na Times SA Derive Saliency Maps Evaluate average prediction score for s in SB nb prediction scores Hypothesis Independent 2-sample T-test with unequal variances p-value >= 0.5? 20

21 The Influence of Task on Eye Movements Results - 21

22 Dynamic Consistency Among Subjects Spatial distribution - highly consistent Significant consistency in the order also?? Automatic Discovery of AOIs & 2 metrics AOI Markov dynamics Temporal AOI alignment 22

23 Scanpath representation Human fixations - tightly clustered Assigning to closest AOI Trace the scan path 23

24 Automatically Finding AOIs Clustering the fixations of all subjects in a frame Start K-Means with 1 cluster Successively Increase until the sum of squared errors drops below a threshold Each fixation assigned to the closest AOI at the time of creation Link centroids from successive frames into tracks Each resulting track becomes an AOI 24

25 Automatically Finding AOIs. 25

26 AOI Markov Dynamics Transitions of human visual attention between AOIs by.. n ma ion u H xat f i Fi ring St Fixated at AOI time t-1 Probability of Transitioning to AOI time t 26

27 Temporal AOI Alignment Longest Common Subsequence?? Able to handle gaps and missing elements 27

28 Evaluation Pipeline Interest Point Operator Descriptor Input: A video Output: A set of spatio-temporal coordinates Spacetime generalization of the HoG & MBH from optical flow Visual Dictionary Cluster descriptors into 4000 Visual words using K-means Classifiers RBF-2 kernel and Multiple Kernel Learning (MKL) framework 28

29 Human Fixation Studies Human vs. Computer Vision Operators Fixations as interest point detector Findings Low correlation Why?? 29

30 Impact of Human Saliency Maps for Computer Visual Action Recognition Saliency maps encoding only the weak surface structure of fixations (no time ordering), can be used to boost the accuracy of contemporary methods 30

31 Saliency Map Prediction Static Features Motion Features AUC & Spatial KL Divergence 31

32 Automatic Visual Action Recognition 32

33 Conclusions Combining Human + Computer Vision Extending Dataset Evaluating Static & Dynamic Consistency Human Fixations -> Saliency Maps End-to-End Action Recognition System 33

34 Thanks! 34