The Visual World as Seen by Neurons and Machines. Aaron Walsman, Akanksha Saran

Transcription

1 The Visual World as Seen by Neurons and Machines Aaron Walsman, Akanksha Saran

2 PPA dataset What does the data encode?

3 Clustering

4 Exemplars clustered together Very few clusters had same category images S1 S2 S3 S4 S6 S7 S8 S9 S10 Left Right

5 Left PPA

6

7

8

9 s6 - gulch, nightclub, volleyball court outdoor

10 Right PPA

11 s9 - nunnery

12 s10 - elevator interior

13

14 s6 - elevator indoor, nightclub cosine - 17, 45

15 Clustering in fmri space Average Distances to exemplar cluster S1 S2 S3 S4 S6 S7 S8 S9 S10 Avg LH RH

16 Clustering in fmri space Average Ranking S1 S2 S3 S4 S6 S7 S8 S9 S10 LH RH

17 Clustering after PCA Not really helping! Average Ranking S1 S2 S3 S4 S6 S7 S8 S9 S10 LH RH

18 How do we Compare? User 1 Left User 1 Right

19 Standard Correlation Measures Left vs Right, Same User Left vs Left, Different User Right vs Right, Different User Kendall s Tau Left vs Right, Same User Left vs Left, Different User Right vs Right, Different User Pearson

20 Kendall s Tau Results Left vs Left, Different User Right vs Right, Different User RGB8 vs Left RGB8 vs Right RGB8 Histogram Left vs Left, Different User Right vs Right, Different User RGB16 vs Left RGB16 vs Right RGB16 Histogram

21 Kendall s Tau Results Left vs Left, Different User Right vs Right, Different User RGB vs Left RGB vs Right RGB Histogram Left vs Left, Different User Right vs Right, Different User RGB4 vs Left RGB4 vs Right RGB4 Histogram

22 Kendall s Tau Results Left vs Left, Different User Right vs Right, Different User BOW vs Left BOW vs Right Bag of Words Left vs Left, Different User Right vs Right, Different User GIST vs Left GIST vs Right GIST

23 Kendall s Tau Results Left vs Left, Different User Right vs Right, Different User OB vs Left OB vs Right Object Bank Left vs Left, Different User Right vs Right, Different User PHOG vs Left PHOG vs Right PHOG

24 Kendall s Tau Results Left vs Left, Different User Right vs Right, Different User LBP vs Left LBP vs Right LBP Histogram Left vs Left, Different User Right vs Right, Different User Harris vs Left Harris vs Right Harris Corner Density

25 Kendall s Tau (all pairwise comparisons)?

26 Which is closer? A B

27 Which is closer? A B

28 Which is closer? A B

29 Kendall s Tau (all pairwise comparisons)?

30 Why is this bad??

31 Is there a better way? Similar? Dissimilar

32 Overlap of Top N Images Given two rankings, what percentage of the top N images overlap? N = 2, %50 N = 3, %66.7 N = 4, %50?

33 How do we choose N? 24 User Vote for best bottom N Vote for best top N Vary N and look for the maximum difference from chance. 15 User

34 What changes? Left vs Right, Same User Left vs Left, Different User Right vs Right, Different User Kendall s Tau Left vs Right, Same User Left vs Left, Different User Right vs Right, Different User Overlap Left vs Right, Same User Left vs Left, Different User Right vs Right, Different User Pearson

35 Overlap Results Left vs Left, Different User Right vs Right, Different User RGB vs Left RGB vs Right RGB Histogram Left vs Left, Different User Right vs Right, Different User RGB4 vs Left RGB4 vs Right RGB4 Histogram

36 Overlap Results Left vs Left, Different User Right vs Right, Different User RGB8 vs Left RGB8 vs Right RGB8 Histogram Left vs Left, Different User Right vs Right, Different User RGB16 vs Left RGB16 vs Right RGB16 Histogram

37 Overlap Results Left vs Left, Different User Right vs Right, Different User BOW vs Left BOW vs Right Bag of Words Left vs Left, Different User Right vs Right, Different User GIST vs Left GIST vs Right GIST

38 Overlap Results Left vs Left, Different User Right vs Right, Different User LBP vs Left LBP vs Right LBP Histogram Left vs Left, Different User Right vs Right, Different User Harris vs Left Harris vs Right Harris Corner Density

39 Overlap Results Left vs Left, Different User Right vs Right, Different User Object Bank vs Left Object Bank vs Right Object Bank Left vs Left, Different User Right vs Right, Different User PHOG vs Left PHOG vs Right PHOG

40 Overlap Summary: What we would like: Bring cross-user values closer to single user values Increase Variance in feature responses What we got: Cross-user values get closer, but variance is stubborn.

41 Also... Modeling bottom-up attention in V4

42 Gallant Lab Natural Movie 4T fmri Dataset training images 120 minutes of short video 15 Hz over 7200 timepoints 8100 testing images 9 minutes 15 Hz over 540 timepoints

43 Training data

44 Validation Data

45 Videos at 15 fps uo&feature=youtu.be One fmri data feature per second - temporal latency

46 What visual features are modeling bottom-up attention in V4? What visual features closest to fmri similarity matrices - Color - Luminance ratio - Hog - Global image features : gist

47 Color S1 S2 S3 Avg Left V Right V

48 Luminance S1 S2 S3 Avg Left V Right V

49 PHOG S1 S2 S3 Avg Left V Right V

50 GIST S1 S2 S3 Avg Left V Right V

51 Takeaway.. global features have stronger representation in the left V4 region color and luminance still better than others in explaining passive viewing MT strong indicator of motion

52 Thanks!