CONSTRAINED SEMI-SUPERVISED LEARNING ATTRIBUTES USING ATTRIBUTES AND COMPARATIVE

Transcription

1 CONSTRAINED SEMI-SUPERVISED LEARNING USING ATTRIBUTES AND COMPARATIVE ATTRIBUTES Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta The Robotics Institute Carnegie Mellon University

2 SUPERVISION BIG-DATA ACTIVE LEARNING SUPERVISED [Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al., ICCV 2007] [von [Qi et Ahn al., and CVPR Dabbish, 2008] [Joshi 2004], et [Russell al., CVPR et 2009] al., IJCV [Siddiquie 2009] and Gupta, CVPR 2010] 2

3 SUPERVISION UNSUPERVISED ACTIVE LEARNING SUPERVISED [Russell et al., CVPR 2006] [Kang et al., ICCV 2011] 3

4 UNSUPERVISED SUPERVISION SEMI-SUPERVISED ACTIVE LEARNING SUPERVISED [Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009] 4

5 UNSUPERVISED SUPERVISION SEMI-SUPERVISED ACTIVE LEARNING SUPERVISED BOOTSTRAPPING Labeled Seed Examples Train Models Unlabeled Data Select Candidates Amphitheatre Amphitheatre Add to Labeled Set [Efron and Tibshirani, 1993] 5

6 UNSUPERVISED SUPERVISION SEMI-SUPERVISED ACTIVE LEARNING SUPERVISED BOOTSTRAPPING Labeled Seed Examples Retrain Models Unlabeled Data Select Candidates Amphitheatre Amphitheatre Add to Labeled Set [Curran et al., PACL 2007] 6

7 UNSUPERVISED SUPERVISION SEMI-SUPERVISED ACTIVE LEARNING SUPERVISED BOOTSTRAPPING Labeled Seed Examples Retrain Models Unlabeled Data Select Candidates Amphitheatre Amphitheatre Add to Labeled Set 25 th Iteration [Curran et al., PACL 2007] 7

8 UNSUPERVISED SUPERVISION SEMI-SUPERVISED ACTIVE LEARNING SUPERVISED BOOTSTRAPPING Labeled Seed Examples Retrain Models Unlabeled Data Select Candidates Amphitheatre Amphitheatre + Auditorium Add to Labeled Set 25 th Iteration Semantic Drift [Curran et al., PACL 2007] 8

9 UNSUPERVISED SUPERVISION SEMI-SUPERVISED ACTIVE LEARNING SUPERVISED GRAPH-BASED METHODS [Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009] 9

10 OUR APPROACH Amphitheatre Amphitheatre Auditorium Auditorium [Carlson et al., NAACL HLT Workshop on SSL for NLP 2009] 10

11 Amphitheatre OUR APPROACH Joint Learning Amphitheatre Auditorium Auditorium Share Data [Carlson et al., NAACL HLT Workshop on SSL for NLP 2009] 11

12 [Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009] BINARY ATTRIBUTES (BA) Conference Room Banquet Hall

13 BINARY ATTRIBUTES (BA) Amphitheatre Auditorium [Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009] 13

14 BINARY ATTRIBUTES (BA) Tables and Chairs Large Seating Capacity Conference Room Indoor Banquet Hall Amphitheatre Man-made Auditorium [Patterson and Hays, CVPR 2012]

15 Auditorium 15

16 Auditorium 16

17 Auditorium 17

18 SHARING VIA DISSIMILARITY Amphitheatre Auditorium Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] 18

19 Amphitheatre Auditorium? 19

20 Amphitheatre Auditorium 20

21 Amphitheatre Auditorium 21

22 DISSIMILARITY COMPARATIVE ATTRIBUTES Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] 22

23 DISSIMILARITY COMPARATIVE ATTRIBUTES Has Larger Circular Structures Features GIST RGB (Tiny Image) Line Histogram of: Length Orientation LAB histogram [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008] 23

24 DISSIMILARITY COMPARATIVE ATTRIBUTES Has Larger Circular Structures Classifier Boosted Decision Tree [Hoiem et al., IJCV 2007] Has Larger Circular Structures or [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] 24

25 COMPARATIVE ATTRIBUTES Is More Open Amphitheatre > Barn Amphitheatre > Conference Room Desert > Barn Has Taller Structures Church (Outdoor) > Cemetery Barn > Cemetery [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] 25

26 Labeled Seed Examples Amphitheatre Selected Candidates Bootstrapping Amphitheatre Auditorium Auditorium 26

27 Labeled Seed Examples Amphitheatre Bootstrapping Amphitheatre Selected Candidates Our Approach (Constrained Bootstrapping) Amphitheatre Attributes Indoor Has Seat Rows Auditorium Auditorium Comparative Attributes Auditorium Has Larger Circular Structures 27

28 more space larger structures indoor has grass Bedroom Banquet bedroom banquet hall Scene Classifiers Unlabeled Data Labeled Data Attribute Classifiers has more space has larger structures Training Pairwise Data Comparative Attribute Classifiers [Gupta and Davis, ECCV 2008] Conference Room Banquet Hall Promoted Instances 28

29 Iteration 40 Iteration 1 Seed Examples Conference Room Introspection 29

30 Our Approach BA Constraints Bootstrapping Seed Images Amphitheatre 30

31 Our Approach BA Constraints Bootstrapping Seed Images Bridge 31

32 EXPERIMENTAL EVALUATION 32

33 CONTROL EXPERIMENTS # Images (SUN Database) 15 Scene Classes 2 (seed) Black-box Binary Attributes (BA) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset 18,000 (Distractors) (9,500) Test Set 50 SUN Database : [Xiao et al., CVPR 2010] 33

34 CONTROL EXPERIMENTS 34

35 CONTROL EXPERIMENTS 35

36 CONTROL EXPERIMENTS 36

37 CONTROL EXPERIMENTS 37

38 \\ CONTROL EXPERIMENTS 38

39 \\ CONTROL EXPERIMENTS Eigen Functions: [Fergus et al., NIPS 2009] 39

40 Iterations Seed Images Banquet Hall

41 CONTROL EXPERIMENTS # Images (SUN Database) 15 Scene Classes 2 (seed) Black-box Binary Attributes (BA) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset 18,000 (Distractors) (9,500) Test Set 50 SUN Database : [Xiao et al., CVPR 2010] 41

42 CO-TRAINING (SMALL SCALE) # Images (SUN Database) 15 Scene Classes 2 (seed) Black-box Binary Attributes (BA) Black-box Comparative Attributes (CA) 15x2 (seed) 15x2 (seed) Unlabeled Dataset 18,000 (Distractors) (9,500) Test Set 50 SUN Database : [Xiao et al., CVPR 2010] 42

43 Our Approach Iteration-60 Iteration-1 Bootstrapping Iteration-60 Iteration-1 Seed Images Bedroom 43

44 SCENE CLASSIFICATION Eigen Functions: [Fergus et al., NIPS 2009] 44

45 CO-TRAINING (LARGE SCALE) 15 Scene Categories 25 Seed images / category Unlabeled Set 1Million (SUN Database + ImageNet) >95% distractors Improve 12 out of 15 scene classifiers SUN Database: [Xiao et al., CVPR 2010] ImageNet: [Deng et al., CVPR 2009] 45

46 CONCLUSION Sharing via Dissimilarities Amphitheatre Auditorium Has Larger Circular Structures Constrained Bootstrapping Labeled Seed Examples Amphitheatre Bootstrapping Amphitheatre Constrained Bootstrapping Amphitheatre Auditorium Auditorium Auditorium 46

47 more space larger structures indoor has grass Bedroom Banquet bedroom banquet hall Scene Classifiers Unlabeled Data Labeled Data Attribute Classifiers has more space has larger structures Training Pairwise Data Comparative Attribute Classifiers Conference Room Banquet Hall Promoted Instances 47