What Helps Where And Why? Semantic Relatedness for Knowledge Transfer

Transcription

1 What Helps Where And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 1,2 Michael Stark 1,2 György Szarvas 1 Iryna Gurevych 1 Bernt Schiele 1,2 1 Department of Computer Science, TU Darmstadt 2 MPI Informatics, Saarbrücken

2 Knowledge transfer for zero-shot object class recognition Unseen class (no training images) Giant panda? Attributes for knowledge transfer Describing using attributes [Farhadi et al., CVPR `09 & `10] Group classes by attributes [Lampert et al., CVPR `09] animal four legged mammal Manual supervision: Attribute labels white paw Manual supervision: Object class-attribute associations CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 2

3 Knowledge transfer for zero-shot object class recognition Unseen Attributes class for knowledge transfer (no Replace training manual images) supervision Giant panda? by semantic relatedness mined from language resources WordNet Attributes for knowledge transfer Describing using attributes [Farhadi et al., CVPR `09 & `10] Group classes by attributes [Lampert et al., CVPR `09] animal four legged mammal Unsupervised Transfer Manual supervision: Attribute labels white paw Manual supervision: Object class-attribute associations CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 3

4 Attribute-based model [Lampert et al., CVPR `09] Known training classes Class-attribute associations Attributes classifiers spots white ocean [Lampert et al., CVPR `09] Supervised: manual (human judges) Class-attribute associations Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 4

5 Attribute-based model [Lampert et al., CVPR `09] Known training classes Class-attribute associations Attribute classifiers spots white ocean Word Net semantic relatedness from language [Lampert et al., CVPR `09] Supervised: manual (human judges) Class-attribute associations Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 5

6 Direct similarity-based model Known training classes Class-attribute associations Attribute classifiers spots white ocean Word Net semantic relatedness from language Class-attribute associations Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 6

7 Direct similarity-based model Known training classes Classifier per class Dalmatian polar bear killer whale Word Net semantic relatedness from language Class-attribute associations Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 7

8 Direct similarity-based model Known training classes Classifier per class Dalmatian polar bear killer whale Word Net semantic relatedness from language most similar classes Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 8

9 Outline Models for visual knowledge transfer Semantic relatedness measures Language resources WordNet Wikipedia WWW Image search Respective state-of-the-art measures Evaluation Conclusion CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 9

10 Semantic Relatedness Measures WordNet Lin measure [Budanitsky & Hirst, CL `06] Wikipedia Explicit Semantic Analysis [Gabrilovich & MarkovitchI, IJCAI `07] Word Wide Web Hitcount (Dice coeffient) [Kilgarriff & Grefenstette, CL `03] Image search Visually more relevant Hitcount (Dice coeffient) horse WordNet [Fellbaum, MIT press `98] animal mammal entity elephant car vehicle bike CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 10

11 Semantic Relatedness Measures WordNet Lin measure [Budanitsky & Hirst, CL `06] Wikipedia Explicit Semantic Analysis [Gabrilovich & MarkovitchI, IJCAI `07] Word Wide Web Hitcount (Dice coeffient) [Kilgarriff & Grefenstette, CL `03] Image search Visually more relevant Hitcount (Dice coeffient) horse WordNet [Fellbaum, MIT press `98] animal mammal entity elephant car vehicle bike CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 11

12 Semantic Relatedness Measures WordNet Lin measure [Budanitsky & Hirst, CL `06] Wikipedia Explicit Semantic Analysis [Gabrilovich & MarkovitchI, IJCAI `07] Word Wide Web Hitcount (Dice coeffient) [Kilgarriff & Grefenstette, CL `03] Image search Visually more relevant Hitcount (Dice coeffient) Farm A farm is an area A of farm lan is an area of lan the training of horses. the training of horses. Hoof A hoof is the tip of A a hoof toe is the tip of a toe Rear hooves of a horse Rear hooves of a horse Most evem tped ungulat Tusk Article horse elephant Farm 3 0 Hoof 2 1 Tusk 0 4 Farm Hoof Tusk Tusks are long teeth, Tusks u are long teeth, u Elephants and narwhals Elephants and narwhals CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 12

13 Semantic Relatedness Measures WordNet Lin measure [Budanitsky & Hirst, CL `06] Wikipedia Explicit Semantic Analysis [Gabrilovich & MarkovitchI, IJCAI `07] Word Wide Web Hitcount (Dice coeffient) [Kilgarriff & Grefenstette, CL `03] Image search Visually more relevant Hitcount (Dice coeffient) Farm A farm is an area A of farm lan is an area of lan the training of horses. the training of horses. Hoof A hoof is the tip of A a hoof toe is the tip of a toe Rear hooves of a horse Rear hooves of a horse Most evem tped ungulat Tusk Article horse elephant Farm 3 0 Hoof 2 1 Tusk 0 4 Farm Hoof Tusk Tusks are long teeth, Tusks u are long teeth, u Elephants and narwhals Elephants and narwhals cosine CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 13

14 Semantic Relatedness Measures WordNet Lin measure [Budanitsky & Hirst, CL `06] Wikipedia Explicit Semantic Analysis [Gabrilovich & MarkovitchI, IJCAI `07] Word Wide Web Hitcount (Dice coeffient) [Kilgarriff & Grefenstette, CL `03] Image search Visually more relevant Hitcount (Dice coeffient) CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 14

15 Semantic Relatedness Measures WordNet Lin measure [Budanitsky & Hirst, CL `06] Wikipedia Explicit Semantic Analysis [Gabrilovich & MarkovitchI, IJCAI `07] Word Wide Web Hitcount (Dice coeffient) [Kilgarriff & Grefenstette, CL `03] Image search Visually more relevant Hitcount (Dice coeffient) web search We watched a horse race yesterday. [..] Tomorrow we go in the zoo to look at the baby elephant. Incidental co-occurence image search [ lahierophant/ /] the dance of the horse and elephant Terms refer to same entity (the image) CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 15

16 Outline Models for visual knowledge transfer Semantic relatedness measures Evaluation Attributes Querying class-attribute associations Mining attributes Direct similarity Attribute-based vs. direct similarity Conclusion CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 16

17 Experimental Setup Animals with attributes dataset [Lampert et al., CVPR `09] 40 training, 10 test classes (disjoint) images total Downsampled to 92 training images per class Manual associations to 85 attributes Image classification SVM: Histogram intersection kernel Area under ROC curve (AUC) - chance level: 50% Mean over all 10 test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 17

18 Performance of supervised approach CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 18

19 Querying class-attribute association Performance of queried association Encouraging Below manual supervision Image search Based on image related text Wikipedia Robust resource Yahoo Web Very noisy resource WordNet Path length poor indicator of class-attribute associations Querying: abbreviation agile Manual supervision: detailed description having a high degree of physical coordination CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 19

20 Querying class-attribute association Performance of queried association Encouraging Below manual supervision Image search (Yahoo Img, Flickr) Based on image related text Wikipedia Robust resource (definition texts) Yahoo Web Very noisy resource WordNet Path length poor indicator of class-attribute associations image search the dance of the horse and elephant CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 20

21 Querying class-attribute association Performance of queried association Encouraging Below manual supervision Image search (Yahoo Img, Flickr) Based on image related text Wikipedia Robust resource (definition text) Yahoo Web Very noisy resource WordNet Path length poor indicator of class-attribute associations Noise: While he watched a horse race the leg of his chair broke. CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 21

22 Querying class-attribute association Performance of queried association Encouraging Below manual supervision Image search (Yahoo Img, Flickr) Based on image related text Wikipedia Robust resource (definition text) Yahoo Web Very noisy resource WordNet Path length poor indicator of class-attribute associations entity mammal horse elephant tooth tusk CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 22

23 Mining attributes Known training classes Class-attribute associations Attribute terms spots white ocean Word Net semantic relatedness from language Class-attribute associations Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 23

24 Mining attributes Known training classes Class-attribute associations Attribute terms Class-attribute associations???? Word Net semantic relatedness from language Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 24

25 Mining attributes Known training classes Class-attribute associations Part Attribute attributes Leg classifiers of a dog semantic relatedness leg paw flipper from language Word Net Word Net Class-attribute associations Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 25

26 Mining attributes Additional measure: Holonym patterns Only part attributes Hit Counts of Patterns [Berland & Charniak, ACL 1999] cow s leg leg of a cow Dice coefficient web search While he watched a horse race the leg of his chair broke. Incidental co-occurence holonym patterns Leg of the horse One term likely part of other term CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 26

27 Mining attributes Best: Yahoo Holonyms Close to manual attributes Tailored towards part attributes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 27

28 Mining attributes Best: Yahoo Holonyms Close to manual attributes Tailored towards part attributes Performance drop Reduced diversity Only part attributes Specialized terms E.g. pilus (=hair) Coverage problem: Image search, Wikipedia CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 28

29 Outline Models for visual knowledge transfer Semantic relatedness measures Evaluation Attributes Querying class-attribute associations Mining attributes Direct similarity Attribute-based vs. direct similarity Conclusion CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 29

30 Direct similarity-based model Known training classes Classifier per class Dalmatian polar bear killer whale Word Net semantic relatedness from language most similar classes Unseen test classes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 30

31 Direct Similarity Nearly all very good On par with manual supervision attribute model (black) Clearly better than any mined attribute-associations result Why? Five most related classes Ranking of semantic relatedness reliable Similar between methods CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 31

32 mean AUC (in %) Attributes vs. direct similarity Extending the test set Add images From known classes As negatives More realistic setting Results Direct similarity drop in performance (orange curve) Attribute models generalize well 70 attributes: manually defined attributes: queried mined associations attributes: mined attributes 65 direct similarity 0 10, , Number of additional training class images in test set CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 32

33 Outline Models for visual knowledge transfer Semantic relatedness measures Evaluation Conclusion CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 33

34 Conclusion Supervision replaced with semantic relatedness Direct similarity better than attributes on par with supervised approach Attributes: generalizes better Semantic relatedness measures Overall best Yahoo image with hit count Holonym patterns for web search Improvement Limited to part attributes CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 34

35 mean AUC (in %) Conclusion Supervision replaced with semantic relatedness Direct similarity better than attributes on par with supervised approach Attributes: generalize better Semantic relatedness measures Overall best Yahoo image with hit count Holonym patterns for web search Improvement Limited to part attributes attributes: manually defined attributes: queried mined associations attributes: mined attributes direct similarity Number of additional training class images in test set CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 35

36 mean AUC (in %) Conclusion Supervision replaced with semantic relatedness Direct similarity better than attributes on par with supervised approach Attributes: generalize better Semantic relatedness measures Overall best Yahoo image with hit count Holonym patterns for web search Improvement Limited to part attributes WordNet poor for object-attributes associations Number of additional training class images in test set patterns: dog s leg leg of the dogs attributes: manually defined attributes: queried mined associations attributes: mined attributes direct similarity CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 36

37 Thank you! Further supervision for closing the semantic gap? See us at our poster (A2, Atrium)! Software? CVPR 2010 What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 37