Probabilistic Evaluation of Saliency Models

Transcription

1 Matthias Kümmerer Matthias Bethge Centre for Integrative Neuroscience, University of Tübingen, Germany October 8,

2 Introduction Model evaluation Modelling Saliency Maps 2 Matthias Ku mmerer, Matthias Bethge

3 mit.saliency.edu 3

4 How to judge saliency model performance? 4

5 Common intuition: Higher saliency corresponds to more fixations 5

6 Common intuition: Higher saliency corresponds to more fixations Saliency is operationalised by measuring fixation densities: p(x, y) 5

7 History of Probablistic Modelling in Saliency Vincent et al: Do we look at lights? Vis.Cog Barthelmé et al: Modelling fixation locations using spatial point processes, JoV 2013 Kümmerer et al: Information-theoretic model comparison unifies saliency metrics, PNAS 2015 Kümmerer et al: DeepGaze I, ICLR Worshop

8 Information theory provides a principled and accepted way to assess how well a model predicts the true density 7

9 The average log-likelihood of a model is N i 1 N log ˆp(x i, y i I i ) for fixations (x i, y i ) on images I i and a model ˆp(x, y I ) 8

10 The average log-likelihood of a model is N i 1 N log ˆp(x i, y i I i ) for fixations (x i, y i ) on images I i and a model ˆp(x, y I ) Information gain: Average log-likelihood relative to baseline IG(ˆp p bl ) = 1 N N log ˆp(x i, y i I i ) 1 N i N log p bl (x i, y i ) i 8

11 The average log-likelihood of a model is N i 1 N log ˆp(x i, y i I i ) for fixations (x i, y i ) on images I i and a model ˆp(x, y I ) Information gain: Average log-likelihood relative to baseline IG(ˆp p bl ) = 1 N N log ˆp(x i, y i I i ) 1 N i N log p bl (x i, y i ) i Interpretation: In a game of 20 questions, how many questions does the model save compared to baseline when trying to find the location of a fixation 8

12 Information Gain [bit/fix] Model A Model B Model C 9

13 Information Gain [bit/fix] Model A Model B Model C Gold Standard 9

14 Information Gain [bit/fix] Model A Model B Model C Gold Standard 100% 50% 42% 21% information gain explained 9

15 10

16 Saliency map Blurring Nonlinearity Center bias 11

17 Resolving the metric inconsistencies. relative performance information gain explained information gain explained AUC wrt. uniform AUC wrt. center bias i.b. D KL f.b. D KL wrt. center bias f.b. D KL wrt. uniform NSS CC Kümmerer et al., PNAS

18 Resolving the metric inconsistencies. relative performance relative performance information gain explained information gain explained information gain explained AUC wrt. uniform AUC wrt. center bias i.b. D KL f.b. D KL wrt. center bias f.b. D KL wrt. uniform NSS CC Kümmerer et al., PNAS

19 DeepGaze 13

20 Deep Gaze II: Model architecture VGG features (fixed parameters) readout network (learned parameters) DeepGaze I: ICLR Workshop 2015 DeepGaze II: arxiv 2016 deepgaze.bethgelab.org blur + softmax 14

21 Deep Gaze: Performance information gain explained [%] Baseline IttiKoch Kienzle CovSal HouZhang SUN, orig GBVS IttiKoch2 Context Aware Torralba Judd SUN, optim RARE AIM BMS edn Deep Gaze I Deep Gaze II Gold standard 15

22 Analysing Probabilistic Models stimulus with ground truth 16

23 Analysing Probabilistic Models stimulus with ground truth model predictions 16

24 Analysing Probabilistic Models stimulus with ground truth model predictions 30 fixation counts for image areas # fixations

25 Analysing Probabilistic Models 17

26 Analysing Probabilistic Models 17

27 Analysing Probabilistic Models 17

28 Analysing Probabilistic Models 17

29 Introduction Model evaluation Modelling Analysing Probabilistic Models 18 Matthias Ku mmerer, Matthias Bethge

30 Analysing Probabilistic Models p_dg(left)-p_mlc(left)= SUN.scale015 AIM Torralba Kienzle MLC Baseline Nonparametric CovSal IttiKoch HouZhang SUN.scale06386 GBVS.ittikoch GBVS.gbvs BMS ContextAware RARE edn Judd DeepGaze2 DeepGaze1 gold_standard Mean-Luminance-Contrast DeepGaze2 19

31 Analysing Probabilistic Models 20

32 Saliency Benchmarking Define submission format; encourage people to hand in probabilistic models Have a principled way to use different saliency maps for different metrics, e.g. with/without centre bias (AUC vs sauc) match empirical saliency histogramm for CC Evaluate information gain / information gain explained Publish the centre bias What to do about classical models? 21

33 Summary Phrasing saliency models probabilistically allows to resolve the inconsistency between different metrics, making benchmarking more interpretable (PNAS 2015) By using probabilistic modeling and optimizing for information gain, we were able to improve the state-of-the-art in fixation prediction (ICLR Workshop 2015; arxiv 2016) Probabilistic modelling gives us new analysis techniques to quantify where and how models fail, and to visualize the limitations of our datasets relative performance information gain explained [%] Baseline IttiKoch information gain explained Kienzle CovSal HouZhang SUN, orig model predictions GBVS IttiKoch2 # fixations Context Aware Torralba 0 Judd SUN, optim RARE AIM BMS edn Deep Gaze I Deep Gaze II Gold standard fixation counts for image areas 22

34 Thank you Matthias Bethge Tom Wallis Lucas Theis Bethgelab 23