Domain Generalization and Adaptation using Low Rank Exemplar Classifiers

Size: px

Start display at page:

Download "Domain Generalization and Adaptation using Low Rank Exemplar Classifiers"

Hector Watkins
5 years ago
Views:

1 Domain Generalization and Adaptation using Low Rank Exemplar Classifiers 报告人 : 李文苏黎世联邦理工学院计算机视觉实验室李文苏黎世联邦理工学院 9/20/2017 1

2 Outline Problems Domain Adaptation and Domain Generalization Low Rank Exemplar Classifiers Low Rank Exemplar Classifiers (LRE-SVMs and LRE-LSSVMs) Domain Generalization and Adaptation Experiments Domain Generalization Domain Adaptation Summary 李文苏黎世联邦理工学院 9/20/2017 2

3 Outline Problems Domain Adaptation and Domain Generalization Low Rank Exemplar Classifiers Low Rank Exemplar Classifiers (LRE-SVMs and LRE-LSSVMs) Domain Generalization and Adaptation Experiments Domain Generalization Domain Adaptation Summary 李文苏黎世联邦理工学院 9/20/2017 3

4 Domain Adaptation: Examples Examples Web <-> Consumer Sketch <-> Photo Synthetic <-> Real Fall <-> Winter K. Saenko, B. Kulis, M. Fritz and T. Darrell. Adapting Visual Category Models to New Domains. In ECCV, T. Kim M. Cha, H. Kim, J. Lee J. Kim. Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. In ICCV 2017 Placeholder X. Peng, K. for Saenko. organisational Synthetic unit to Real name Adaptation / logo with Generative Correlation Alignment Networks. Arxiv , (edit J. in Hoffman, slide master D. Wang, via View F. Yu, > T. Slide Darrel. Master ) FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation. Arxiv , 李文苏黎世联邦理工学院 9/20/

5 Visual Recognition System Visual Recognition System Labeled Training Images Learning Test Images Independently and identically distributed (i.i.d. assumption) 李文苏黎世联邦理工学院 9/20/2017 5

6 Visual Recognition System Visual Recognition System Labeled Training Images Learning Test Images Independently and identically distributed (i.i.d. assumption) i.i.d assumption may not always hold Data collection bias is inevitable The real-world test data changes easily 李文苏黎世联邦理工学院 9/20/2017 6

7 Visual Recognition System Visual Recognition System Labeled Training Images Learning Test Images Independently and identically distributed (i.i.d. assumption) i.i.d assumption may not always hold Data collection bias is inevitable The real-world visual data varies a lot 李文苏黎世联邦理工学院 9/20/2017 7

8 Visual Recognition System Visual Recognition System Labeled Training Images Learning Test Images Independently and identically distributed (i.i.d. assumption) i.i.d assumption may not always hold Data collection bias is inevitable The real-world visual data varies a lot 李文苏黎世联邦理工学院 9/20/2017 8

9 Dataset Bias Name that dataset! Caltech-101 LabelMe MSRC COIL-100 UIUC ImageNet Tiny 15 Scenes Corel Caltech-256 PASCAL 07 SUN09 (edit A. Torralba, in slide master and A. via Efros. View Unbiased > Slide Look Master ) at Dataset Bias. In CVPR 李文苏黎世联邦理工学院 9/20/2017 9

10 Dataset Bias Name that dataset! Caltech LabelMe 7 MSRC 3 COIL UIUC 2 ImageNet 5 Tiny Images 4 15 Scenes 9 Corel 10 Caltech PASCAL 07 6 SUN09 8 Current vision datasets contains their own biases, regardless of their semantic categories. (edit A. Torralba, in slide master and A. via Efros. View Unbiased > Slide Look Master ) at Dataset Bias. In CVPR 李文苏黎世联邦理工学院 9/20/

11 Visual Recognition System Visual Recognition System Labeled Training Images Learning Test Images Independently and identically distributed (i.i.d. assumption) i.i.d assumption may not always hold Data collection bias is inevitable The real-world visual data varies a lot 李文苏黎世联邦理工学院 9/20/

12 Dataset Bias Real-world objects vary a lot (edit A. Torralba, in slide master and A. via Efros. View Unbiased > Slide Look Master ) at Dataset Bias. In CVPR 李文苏黎世联邦理工学院 9/20/

13 Dataset Bias Impact of Dataset Bias Cross-dataset Classification Performance Car Classification S L P I C M drop S % L % P % I % C % M % (edit A. Torralba, in slide master and A. via Efros. View Unbiased > Slide Look Master ) at Dataset Bias. In CVPR 李文苏黎世联邦理工学院 9/20/

14 Dataset Bias Impact of Dataset Bias Cross-dataset Classification Performance Car Classification S L P I C M drop S % L % P % I % C % M % Dataset bias harms the crossdataset classification performance. (edit A. Torralba, in slide master and A. via Efros. View Unbiased > Slide Look Master ) at Dataset Bias. In CVPR 李文苏黎世联邦理工学院 9/20/

15 Visual Recognition System Visual Recognition System Labeled Training Images Learning Test Images Independently and identically distributed (i.i.d. assumption) i.i.d assumption may not always hold Data collection bias is inevitable The real-world visual data varies a lot Re-collect data? Cost money and time! 李文苏黎世联邦理工学院 9/20/

16 Domain Adaptation: Problem Description Domain Adaptation bike: cup: Source Domain P s (x, y) P s (x, y) P t (x, y) Target Domain P t (x, y) 李文苏黎世联邦理工学院 9/20/

17 Domain Adaptation: Problem Description (Unsupervised) Domain Adaptation n s + Source domain: *(x s i, y s i ) i=1 Target domain: *x t n i t i=1 + Data distribution mismatch: P s (x s, y s ) P t (x t, y t ) or P s (x s ) P t (x t ) Feature space and label space are consistent: x s, x t R D y s, y t L 李文苏黎世联邦理工学院 9/20/

18 Domain Adaptation: Problem Description (Unsupervised) Domain Adaptation n s + Source domain: *(x s i, y s i ) i=1 Target domain: *x t n i t i=1 + Data distribution mismatch: P s (x s, y s ) P t (x t, y t ) or P s (x s ) P t (x t ) Feature space and label space are consistent: x s, x t R D y s, y t L Related Concepts: Transfer Learning Label space are different (cross-task) Heterogeneous Domain Adaptation Feature space are different (cross-feature) 李文苏黎世联邦理工学院 9/20/

19 Domain Adaptation: Related Works Feature-Level Methods Strategy: Traditional methods P s (x s ) P t (x t ) P s (g(x s )) P t (x t ) For example, TCA, SGF, GFK, SA, DIP, based on subspace and manifold principles. 李文苏黎世联邦理工学院 9/20/

20 Domain Adaptation: Related Works Feature-Level Methods Strategy: Traditional methods P s (x s ) P t (x t ) P s (g(x s )) P t (x t ) For example, TCA, SGF, GFK, SA, DIP, based on subspace and manifold principles. CNN based methods Top-down: DAN, JAN, GRL, DRCN, Bottom-Up: AdaBN, AutoDIAL source target CNN cls loss DA loss 李文苏黎世联邦理工学院 9/20/

21 Domain Adaptation: Related Works Feature-Level Methods Strategy: Traditional methods P s (x s ) P t (x t ) P s (g(x s )) P t (x t ) For example, TCA, SGF, GFK, SA, DIP, based on subspace and manifold principles. CNN based methods Top-down: DAN, JAN, GRL, DRCN, Bottom-Up: AdaBN, AutoDIAL source target CNN loss Domain Adaptive Normalization 李文苏黎世联邦理工学院 9/20/

22 Domain Adaptation: Related Works Feature-Level Methods Strategy: P s (x s ) P t (x t ) P s (g(x s )) P t (x t ) Traditional methods For example, TCA, SGF, GFK, SA, DIP, based on subspace and manifold principles. CNN based methods Top-down: DAN, JAN, GRL, DRCN, Bottom-Up: AdaBN, AutoDIAL Image-Level CycleGAN, DiscoGAN, DualGAN, UNIT 李文苏黎世联邦理工学院 9/20/

23 Domain Adaptation: Related Works Instance-Level Methods Strategy: Methods KMM, DA-SVM Deep Methods Transductive DA, Associative DA P s (x s ) P t (x t ) g(x s )P s (x s ) P t (x t ) 李文苏黎世联邦理工学院 9/20/

24 Domain Generalization What if we do not know about target domain? Multi-source domain generalization source-1 source target source-2? source-m Domain Adaptation Domain Generalization K. Muandet, D. Balduzzi, B. Scholkopf. Domain generalization via invariant feature representation. In ICML, 李文苏黎世联邦理工学院 9/20/

25 Domain Generalization Problems Real world domains are not distinctive source-1 source-2? source-m 李文苏黎世联邦理工学院 9/20/

26 Domain Generalization Problems A more common case, single but diverse source domain source-1 source-2? source domain source-m 李文苏黎世联邦理工学院 9/20/

27 Domain Generalization Latent Domain Discovery Partition one source domain into multiple latent domains source domain latent-1 latent-2 latent-3 J. Hoffman, B. Kulis, T. Darrell, K. Saenko. Discovering Latent Domains For Multisource Domain Adaptation. In ECCV (edit in slide B. Gong, master K. via Grauman, View > and Slide F. Sha. Master ) Reshaping Visual Datasets for Domain Adaptation. In NIPS 2013 李文苏黎世联邦理工学院 9/20/

28 Domain Generalization Latent Domain Discovery Partition one source domain into multiple latent domains source domain latent-1 latent-2 latent-3 How many latent domains? Non-trivial to disentangle correlated variances J. Hoffman, B. Kulis, T. Darrell, K. Saenko. Discovering Latent Domains For Multisource Domain Adaptation. In ECCV (edit in slide B. Gong, master K. via Grauman, View > and Slide F. Sha. Master ) Reshaping Visual Datasets for Domain Adaptation. In NIPS 2013 李文苏黎世联邦理工学院 9/20/

29 Outline Problems Domain Adaptation and Domain Generalization Low Rank Exemplar Classifiers Low Rank Exemplar Classifiers (LRE-SVMs and LRE-LSSVMs) Domain Generalization and Adaptation Experiments Domain Generalization Domain Adaptation Evolving Domain Adaptation Conclusions and Future Work 李文苏黎世联邦理工学院 9/20/

30 Low-Rank Exemplar Classifiers for Domain Generalization One-stage Approach source-1 source source-2 source-m domain adaptation domain generalization 李文苏黎世联邦理工学院 9/20/

31 Low-Rank Exemplar Classifiers for Domain Generalization One-stage Approach source domain adaptation domain generalization 李文苏黎世联邦理工学院 9/20/

32 Exemplar SVMs (E-SVMs) SVM vs E-SVMs Each exemplar SVM is trained using one positive samples and all negative samples SVM E-SVM 1 E-SVM 2 T. Malisiewicz, A. Gupta, A. Efros. Ensemble of Exemplar-SVMs for Object Detection and Beyond. In ICCV 2011 李文苏黎世联邦理工学院 9/20/

33 Exemplar SVMs (E-SVMs) SVM vs E-SVMs Each exemplar SVM is trained using one positive samples and all negative samples bike, clean background front viewpoint bike, indoor background front viewpoint SVM E-SVM 1 E-SVM 2 李文苏黎世联邦理工学院 9/20/

34 LRE-SVMs for Domain Generalization Ensemble Exemplar Classifiers Example Exemplar Classifiers Test Sample 李文苏黎世联邦理工学院 9/20/

35 LRE-SVMs for Domain Generalization Ensemble Exemplar Classifiers Example Sensitive to noise Exemplar Classifiers Test Sample 李文苏黎世联邦理工学院 9/20/

36 Exemplar SVMs (E-SVMs) Learning Objective Training data where loss is defined as Unify all exemplar SVMs 李文苏黎世联邦理工学院 9/20/

37 Low Rank Exemplar SVMs (LRE-SVMs) Low Rank Property Using n E-SVMs to predict n positive training samples 李文苏黎世联邦理工学院 9/20/

38 Low Rank Exemplar SVMs (LRE-SVMs) Low Rank Property Using n E-SVMs to predict n positive training samples G W R n n is low-rank 李文苏黎世联邦理工学院 9/20/

39 Low Rank Exemplar SVMs (LRE-SVMs) Objective Adding a nuclear-norm based regularizer on G(W) Where G W = g ij 李文苏黎世联邦理工学院 9/20/

40 Optimization Introducing an intermediate variable Alternating optimization Fix W, Update F Fix F, Update W Main computational cost! Placeholder J. Cai, for E. organisational J. Cands, and Z. unit Shen. name A singular / logo value thresholding algorithm for matrix completion, (edit in slide In SIAM master Journal via View on Optimization, > Slide Master ) vol. 20, no. 4, pp , 2010 李文苏黎世联邦理工学院 9/20/

41 Low Rank Exemplar LS-SVMs (LRE-LSSVMs) Using least square SVM as the base classifier Loss function is changed Closed form solution 李文苏黎世联邦理工学院 9/20/

42 Low Rank Exemplar LS-SVMs (LRE-LSSVMs) A fast solution exemplar n+m Cost only O((n + m) 2 ) 李文苏黎世联邦理工学院 9/20/

43 Outline Problems Domain Adaptation and Domain Generalization Low Rank Exemplar Classifiers Low Rank Exemplar Classifiers (LRE-SVMs and LRE-LSSVMs) Domain Generalization and Adaptation Experiments Domain Generalization Domain Adaptation Evolving Domain Adaptation Conclusions and Future Work 李文苏黎世联邦理工学院 9/20/

44 LRE-SVMs for Domain Generalization Ensemble Exemplar Classifiers Example Exemplar Classifiers Test Sample 李文苏黎世联邦理工学院 9/20/

45 LRE-SVMs for Domain Adaptation Ensemble Exemplar Classifiers Similarity between E-SVM training data and the target domain f(mmd) Source Domain Target Domain 李文苏黎世联邦理工学院 9/20/

46 LRE-SVMs for Domain Adaptation Learn A Unified Classifier where f is the prediction score obtained using ensemble E-SVMs Placeholder L. Duan, for D. organisational Xu, and I. W. Tsang. unit name Domain / logo adaptation from multiple sources: A domain-dependent regularization approach. (edit in In slide T-NNLS, master vol. via 23, View no. 3, > Slide pp , Master ) March 2012 李文苏黎世联邦理工学院 9/20/

47 Outline Problems Domain Adaptation and Domain Generalization Low Rank Exemplar Classifiers Low Rank Exemplar Classifiers (LRE-SVMs and LRE-LSSVMs) Domain Generalization and Adaptation Experiments Domain Generalization Domain Adaptation Conclusions and Future Work 李文苏黎世联邦理工学院 9/20/

Datasets Action Recognition IXMAS dataset: 5 actions from 5 different view points Mix several viewpoints, and leave remaining s for test Object Recognition

48 Datasets Action Recognition IXMAS dataset: 5 actions from 5 different view points Mix several viewpoints, and leave remaining s for test Object Recognition Office-Caltech Dataset: 10 classes from 4 domains, Caltech-256, Amazon, DSLR, Webcam Mix several domains, and leave remaining's for test 李文苏黎世联邦理工学院 9/20/

49 Domain Generalization Action Recognition Results (0,1)->(2,3,4) (2,3,4)->(0,1) (0,1,2,3)->4 SVM E-SVMs E-LSSVMs LRE-SVMs LRE-LSSVMs Summary Ensemble exemplar classifiers help domain generalization LRE-SVMs and LRE-LSSVMs improves E-SVMs and E-LSSVMs. 李文苏黎世联邦理工学院 9/20/

50 Domain Generalization Experimental Comparisons 李文苏黎世联邦理工学院 9/20/

51 LRE-SVMs v.s. LRE-LSSVMs Training Time IXMAS, (0,1) -> (2,3,4) Training Time (seconds) LRE-SVMS LRE-LSSVMs Summary LRE-LSSVM is more than 80 times faster than LRE-SVMs 李文苏黎世联邦理工学院 9/20/

52 Low Rank Regularizer Visualization on the prediction matrix G(W) IXMAS, (0,1) -> (2,3,4) Clustering viewpoints, actors 李文苏黎世联邦理工学院 9/20/

53 Low Rank Regularizer Visualization on the prediction matrix G(W) IXMAS, (0,1) -> (2,3,4) Clustering viewpoints, actors 李文苏黎世联邦理工学院 9/20/

54 Domain Adaptation Action Recognition Using target domain information can further improves the accuracy (0,1)->(2,3,4) (2,3,4)->(0,1) (0,1,2,3)->4 SVM LRE-LSSVMs LRE-LSSVMs-DA 李文苏黎世联邦理工学院 9/20/

55 Domain Adaptation: Object Recognition Comparison with deep domain adaptation approaches 图表标题 (A,C)->(D,W) (D,W)->(A,C) (C,D,W)->A SVM Ours DAN DAN+Ours GRL GRL+Ours 李文苏黎世联邦理工学院 9/20/

56 Outline Problems Domain Adaptation and Domain Generalization Low Rank Exemplar Classifiers Low Rank Exemplar Classifiers (LRE-SVMs and LRE-LSSVMs) Domain Generalization and Adaptation Experiments Domain Generalization Domain Adaptation Evolving Domain Adaptation Summary 李文苏黎世联邦理工学院 9/20/

57 Summary Summary Exploiting inner-domain structure helps cross-domain generalization ability LRE-SVMs and LRE-LSSVMs Learn locality domain property based on E-SVMs Exploit the low-rank structure of prediction matrix Complementary to current CNN based (global) domain adaptation methods. 李文苏黎世联邦理工学院 9/20/

58 Future Trends Increasingly important in deep learning era large scale training data => high cost on annotation new principles: GRL, AdaBN/AutoDIAL, JAN, CycleGAN New issues: domain generalization, transfer learning, heterogeneous domain adaptation Task-oriented: synthetic to real adaptation, semantic segmentation, video recognition, image captioning, VQA 李文苏黎世联邦理工学院 9/20/

59 Collaborators and References Collaborators Zheng Xu Li Niu Dengxin Dai Dong Xu Luc Van Gool References 1) W. Li, Z. Xu, D. Xu, D. Dai, and L. Van Gool. Domain Generalization and Adaptation using Low Rank Exemplar SVMs. In T-PAMI, 2017 (In Press) 2) Z. Xu, W. Li, L. Niu, and D. Xu. Exploiting Low-rank Structure from Latent Domains for Domain Generalization. In ECCV 李文苏黎世联邦理工学院 9/20/

60 WebVision Challenge Learning from Web Data Flickr Images Google Images, Learning Representation CNN Categorization Detection Segmentation Dataset 1,000 categories, 2.4 million images, 50,000 validation images Challenge WebVision classification track Transfer learning track $10,000 cash prize! 李文苏黎世联邦理工学院 9/20/

61 Thank you! 欢迎提问和指正! Wen Li - Learning from Web Data 62

DMAIN adaptation techniques, which aim to reduce the

DMAIN adaptation techniques, which aim to reduce the Domain Generalization and Adaptation using Low Rank Exemplar SVMs Wen Li, Zheng Xu, Dong Xu, Dengxin Dai, and Luc Van Gool Abstract Domain adaptation between diverse source and target domains is a challenging