Categories. Represent/store visual objects in terms of categories. What are categories? Why do we need categories?

Transcription

1 Represen'ng Objects

2 Categories Represent/store visual objects in terms of categories. What are categories? Why do we need categories? Grouping of objects into sets where sets are called categories!

3 Categories What is a category? (class, concept) A set of objects/things..what sets are valid? A probability distribu'on? What determines what belongs to a category? With a category comes the ability to judge in principle whether new things are part of it. How does this work? Are categories in the world or in our head? How do we determine categories computa'onally?

4 Categories Believed to fundamental in language, predic'on, inference, thinking. But have you ever asked why do we need categories in our mind.. Grouping helps us to interpolate and extrapolate GENERALIZATION.

5 Classical View of Categories Dates back to Plato & Aristotle 1. Categories are defined by a list of properties shared by all elements in a category 2. Category membership is binary 3. Every member in the category is equal

6 Problems with Classical View Humans don t do this! People don t rely on abstract definitions / lists of shared properties (Wittgenstein 1953, Rosch 1973) e.g. define the properties shared by all games e.g. are curtains furniture? Are olives fruit? Typicality e.g. Chicken -> bird, but bird -> eagle, pigeon, etc. Language-dependent Le Bus vs. L Autocar in French Doesn t work even in human-defined domains e.g. Is Pluto a planet?

7 Prototype Theories Not really a theory, i.e., not too specific. States that categories have structure. Prototypes. Example of a specific prototype theory: a sta's'cal model based on proper'es. Gaussian distribu'on; weighted combina'on of proper'es: eg., Bird Proper'es: flies, sings, lays eggs, is small, nests in trees, eats insects. All are true of a robin, but maybe only some need to be true (eg., a chicken).

8 Categories in Computer Vision Started with Faces Extended to Pedestrians and Cars.. Caltech- 101 and Caltech- 256 were the first datasets to have more than three categories. The names of 101 categories were generated by flipping through Webster Collegiate Dic>onary, picking subset of categories that were associated with a drawing.

9 PASCAL VOC Basic Categories. Focus on the problem of detec'ng objects in these 20 categories: Person, Sheep, Cat, Dog, Horse, Bird, Cow, Aeroplane, Bus, Bicycle, Car, Train, Motorbike, Boat, Chair, Couch, Table, Bocle, Poced- Plant, TV.

10 Bias with Seman'c Categories Con'nues

11 Imagenet Built upon hierarchical structure of WordNet 80,000 categories: images per category

12 Hierarchical Structure Collect Images at All Levels

13 Imagenet

14 Fine- Grained Categoriza'on

15 Ques'ons from Computer Vision Are seman'c categories the right categories to work with? How do dogs and cats get their categories? How are the categories represented in human brain? Are all animals are close to each other and all vehicles close to each other? Is there one hierarchy to work with or does our human brain has mul'ple hierarchy? Is there hierarchy at all? Do categories have to interact with each other?

16 Interes'ng Ques'ons 1 Neuron per Category? Or 1 Neuron per instance? Classical vs. Prototypical: If classical is true then different areas of brain processing different categories is fine. If prototypical theory then probably more con'nuous space?

17 Haxby et al.

18 Huth et al.

19 Goals Early papers in brain studies concluded different processing regions for different kind of objects. Faces Body Parts Movements Given there are 80K categories to represent, having different areas for each category seems unreasonable Haxby et al. (2001) further suggests a more distributed representa'on. More likely that categories are organized in a con'nuous space But what is this con'nuous space?

20 Goal Discover the con'nuous space in human brain for representa'on of categories. Similar categories have similar neural signatures.. No discrete step. The distance in brain space might be similar or propor'onal to seman'c similarity between categories. Some work has proposed the dimensions along which the categories are organized but no explicit represent of con'nuous space Bus Car

21 Assump'ons Visual world is organized via categories Categories are Seman'c Categories are arranged in a Hierarchy

22 Experiment

23 Devil is in details Labeled 1364 categories only by humans 341 superordinate propagated via wordnet! Creates connec'ons within the data and hence forcing a con'nuous space and other dimensions in the con'nuous space? Specifically, no'ce the entailing part!!

24 Experiment

25 Objects and Structures Outdoor Scenes Non Human Biological

26 Social Se7ngs

27 Seman'c Space Category Weight Matrix (N x V) N = Number of Categories V= Number of Voxels Use PCA to recover Seman'c Space (dimensions to organize categories) PCA ensures categories with similar cor'cal voxels will project nearby in reduced dimensions

28 Seman'c Space via PCA 1705 PCs But only first few meaningful rest represent variance due to noise in the data. PCs meaningful since it represents more variance in the data than s'muli matrix

29

30 Lets look at the Principal Components!

31 Things That MOVE

32 Social Interac'ons

33 Civiliza'on (people, man- made, vehicles) Vs. Nature (Non- human Animals)

34 Biological Vs. Non- Biological

35

36 Interes'ng Observa'ons PPA/RSC/TOS Scene Area: Contrast between place categories and non- place categories is not captured. Sta'c vs. Movies Object Size also not captured

37

38 Map from high- dimension to low- dimensional space

39 Project Voxel Weights onto 2-4 PCs

40 Animal, Human - IT Succulus

41 FFA Faces: surrounded by human, animals

42 Medial Occipitotemporal cortex: vehicle, landscape

43 Are these really smooth spa'ally? Pair of Voxels are projected in 4PCs and then correla'on between them. Weighted by distance in brain space.

44 Good for Predic'on? Few or Lot of Category Selec've Regions?

45 Discussion WordNet based Experiment Construc'on.. Only rare categories S'mulus Correla'ons: Mouth vs. Talking Conceptual vs. Visual Use audio/nlp to differen'ate Movies vs. Sta'c Images

46 Discussion Most vision approaches s'll treat as discrete recogni'on problems Cat: Yes/No Dog: Yes/No Acributes: Shared seman'cs across categories