Lecture 20: Emotion Recognition

Transcription

1 Lecture 20: Emotion Recognition

2

3

4

5 Goals How do humans recognize facial expressions? Review some properties/constraints of human expression processing Discuss automatic facial expression recognition Walk through offline experiment Illustrate some of the issues in emotion recognition How to get training data Computational challenges Revisit contextual influences Are we going to find emotion where we typically looking? Context effects and multimodal approaches

6 Human face processing 3 Some basic questions What are facial expressions of emotion and what are they for? Are there specific centres in the brain dedicated to emotion perception? Are different emotions processed in different ways? Adapted Jonathan from Gratch Roger Newport, Nottingham

7 Why bother recognizing expressions Survival Social survival

8 Faces are Special Face perception may be most developed visual perceptual skill in humans Infants prefer to look at faces from shortly after birth (Morton and Johnson 1991) Most people spend more time looking at faces than any other type of object We seem to have capacity to perceive the unique identity of an unlimited number of different faces Adapted Jonathan from Gratch Roger Newport, Nottingham

9 Faces are Special Adapted Jonathan from Gratch Roger Newport, Nottingham

10 Human faces designed for rapid communication 9 Expressions elicited when there is an emotional stimulus and an audience present Our interpretation of another s emotion modulates our behaviour and vice versa The ability to recognise emotion expressions appears very early first few days (neonates) Four- to six-month seven months can distinguish between expressions of happiness, sadness, and surprise show preferences for facial expressions of happiness over neutral and angry expressions can distinguish among expressions of fear, anger, surprise, happiness, and sadness Adapted Jonathan from Gratch Roger Newport, Nottingham

11 Quick experiment I will show a bunch of pictures Left-hand side of class: Raise your hand when you find the face Right-hand side of class: Raise your hand when you find the gloves

12 Recognition an automatic processes - fear & threat Angry faces are detected much more rapidly than faces depicting nonthreatening expressions Attention is driven by fear Adapted Jonathan from Gratch Roger Newport, Nottingham

13 Faces are Special The seem to be evolved to support efficient communication Adapted Jonathan from Gratch Roger Newport, Nottingham

14 Social Gaze

15 Social Gaze

16 Human faces designed for rapid communication e.g. Human eye is unique amongst all animals in allowing rapid communication of attention and intent Humans have the whitest sclera Humans have the widest percentage of exposed sclera Exposed portion of eye most elongated in horizontal direction Allows efficient communication of gaze direction Kobayashi and Kohshima (2001)

17 Human faces designed for rapid communication e.g. Human eye is unique amongst all animals in allowing rapid communication of attention and intent Humans have the whitest sclera Humans have the widest percentage of exposed sclera Exposed portion of eye most elongated in horizontal direction Contrast this with Fridlund s claim that no evolutionary advantage to showing our emotions Allows efficient communication of gaze direction Kobayashi and Kohshima (2001)

18 Faces seem to be processed automatically (System 1) e.g., Fear and the amygdala Evidence from animal, neuropsychological and imaging studies suggest the amygdala is of primary importance in the recognition of fear. Adapted Jonathan from Gratch Roger Newport, Nottingham

19 Fear and the amygdala - evidence from animal studies Bilateral amygdala removal: reduces levels of aggression and fear in rats and monkeys facial expressions and vocalisations become less expressive impairs fear conditioning Adapted Jonathan from Gratch Roger Newport, Nottingham

20 Fear and the amygdala - evidence from human neuropsychology Bilateral amygdala damage reduces recognition of fear-inducing stimuli reduces recognition of fear in others reduces ability to express fear Does NOT affect ability to recognise faces or to know what fear is 13 Adapted Jonathan from Gratch Roger Newport, Nottingham

21 Hadjihhani and de Gelder, 2003 Adolphs and Tranel, 2003 Fear condition (and amygdala activation) strongest when face present Adapted Jonathan from Gratch Roger Newport, Nottingham

22 Emotions - the importance of the eyes Whalen et al., 2004 % signal change from fix. The amygdala is responsive to large eye whites in fear (and surprise) expressions Adapted Jonathan from Gratch Roger Newport, Nottingham

23 Amygdala damage changes where we look SM is patient w/ amygdala damage SM s eye fixation (or lack of it)

24 Emotions - amygdalae are not simply eye detectors The amygdalae are not just eye detectors - may direct attention to relevant stimuli - a biological relevance detector Some easy to tell from eyes Others from mouths Hybrid Faces from Vuilleumier, We need more than just the eyes to determine emotional and social relevance

25 Other basic emotions - implicated brain regions Green = neutral Red = anger Purple = fear Yellow - happy Blue = sad Duvernoy 1991, but see Linquist et al 2012

26 Automatic expression analysis

27 Desired Functionality Human visual system = good reference point Desired properties: Works on images of people of any sex, age, and ethnicity Robust to variation in lighting Insensitive to hair style changes, glasses, facial hair, occlusions Can deal with rigid head motions Is real-time Classifies expressions into multiple emotion categories Learns the range of emotional expression by a particular person

28 Automatic Analysis Overview Three basic problems need to be solved: Detect the face Detect face pose Classify the emotional expression

29 Face detection slides Jonathan adapted Gratch from P. Viola

30 Face detection Basic idea: slide a window across image and evaluate a face model at every location slides Jonathan adapted Gratch from P. Viola

31 Challenges of face detection Sliding window detector must evaluate tens of thousands of location/scale combinations Faces are rare: 0 10 per image For computational efficiency, we should try to spend as little time as possible on the non-face windows A megapixel image has ~10 6 pixels and a comparable number of candidate face locations To avoid having a false positive in every image image, our false positive rate has to be less than 10-6 slides Jonathan adapted Gratch from P. Viola

32 The Viola/Jones Face Detector A seminal approach to real-time object detection Training is slow, but detection is very fast Key ideas Integral images for fast feature evaluation Boosting for feature selection Attentional cascade for fast rejection of non-face windows P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. CVPR P. Viola and M. Jones. Robust real-time face detection. IJCV 57(2), slides Jonathan adapted Gratch from P. Viola

33 Extract facial pose Template-based approaches Active Appearance Models most common Objects represented by paramaterized model of shape and texture (grey-level appearance) Models are fit to new images Model parameters tell us head orientation and location of facial landmarks

34 Extract facial pose

35 Frame-by-frame vs. Dynamics Cohn et al. Displacement of each point calculated relative to the first frame The displacement of feature points between the initial and peak frames used for classification You d expect dynamics to work best but not currently

36 Classifying emotional expression Joy Positive CLNF Tadas B SHORE

37 Classifying emotional expression Several basic problems: Choose a representation language for emotion What are our labels? Getting good labeled data of ground truth How do we know if someone Happy/Positive/Positive+Active/etc.? Choosing a classification/learning mechanism Need algorithm to label novel images Usually involves machine learning

38

39 What is our language of emotion? Fill in the blank Basic emotions Dimensional models 6 These categories used by most automated methods

40 Defining set of categories/labels 6 Facial Action Coding System

41 Ground Truth (what s his emotion?) Ideas? Self report What were you feeling? 6 Observers This what I asked you to do Experimental design Induce an emotion

42 How do we get good coders? Train them? Get lots of them and use (wisdom of crowd)? Pick emotionally smart people?

43 Social Intelligence Test The Reading the Mind in the Eyes Test Revised Version: A Study with Normal Adults, and Adults with Asperger Syndrome or High-functioning Autism. Baron-Cohen, Wheelwright, Hill, Raste and Plumb

44 Social Intelligence Test Reading the Mind in the Eyes Designed to assess mind reading ability (theory of mind) Aimed at distinguishing people with theory-of-mind deficits, especially Autism Not correlated with IQ Negatively correlated with Autism Spectrum Quotient (AQ) Negatively correlated w/ ability to annotate facial expressions Typical scores Autistic/High-Functioning Autistic Adults (mean score 22) Normal Students (mean 28), Males (27), Females (29) Class? 25.6 avg, Max 32 (Sandeep), Min 19 The Reading the Mind in the Eyes Test Revised Version: A Study with Normal Adults, and Adults with Asperger Syndrome or High-functioning Autism. Baron-Cohen, Wheelwright, Hill, Raste and Plumb

45 Fill in the Blank Skeptical Dumbfounded Surprised Surprised Curious Baffled Sarcastic Incredulous Joking Confused Flirtatious Indissoluble Entertained Disgusted Joking Distressed Shocked Incredulous Happy Content Satisfied Funny Interested Amused Seductive Bored Shy Lost Flirtatious Annoyed Fear Amused Shocked Joyful Aghast Exciting Horrified Amused Aghast Playful Imploring Joyous Angry Confused Hostile Angry Distrusting Doubting

46 Valence Other schemes Basic emotions Angry, Happy, Surprised, Disgusted, Sad, Afraid, Neutral Valence: Valence: Valence: Basic:Surprised (3) Basic:? Basic: Happy (4) Valence: -2.2 Basic: Disgusted (3) Valence: -0.8 Valence: -1.5 Valence: -2.7 Basic: Neutral (5) Basic: Surprised (3) Basic:? Valence: 0.8 Basic: Happy (3)

47 Classification Several basic problems: Getting good labeled data of ground truth Defining a set of categories/classes How do we know if our labels are any good? One common approach Rater reliability

48 Inter-rater agreement In statistics, inter-rater reliability, inter-rater agreement, or concordance is the degree of agreement among raters. It measures how much homogeneity there is in the ratings given by judges. It is useful in refining the tools given to human judges, for example by determining if an annotation scheme is appropriate for measuring a particular variable. If various raters do not agree, either the measure is defective or the raters need to be re-trained. Krippendorff s alpha measures the amount of noise in the data, controlling for amount of agreement one would expect by chance Rules of Thumb: α >.8 consider good agreement..8 > α >.67 tentative Rarely get alpha s this high for behavioral data

49 Steps for machine learning Got good agreement? YES? We can use to train a classifier OK but what if we have 2 million faces?» Crowd source?

50

51 Steps for machine learning Got good agreement? NO? Just go for it anyway Use mean of bunch of people (wisdom of crowd) Use other learning approach

52

53 Competitive?

54

55 End Product

56 How did we do? Happy Content Satisfied Funny Interested Amused 3

57 5 Seductive Bored Shy Lost Flirtatious Annoyed

58 Veteran Recounting his PTSD story: I needed a place to live. I was undergoing depression. They knew it. And they pretty much put me in a building where I was the only one living there. Really depressed in a desert environment. Incredulous Confused Indissoluble Disgusted Distressed Incredulous 4

59 Angry Confused Hostile Angry Distrusting Doubting Amused Joyful Exciting Amused Playful Joyous 7 8 Filling out a Questionnaire Just stole in prisoner s dilemma

60 Challenges for emotion recognition Context Very difficult for machines (and even people) to infer anything from single naturalistic image

61 Approach: Brunswik s Lens Model Step 4: Calculate relationship between behavior and received affect Sucks if we Use Sender Real Signal Machine (and Data human) Recognition 4 Great if use Acted Data Anger ENCODING 5 23 But people Feel they are Pretty good At recognition 24 DECODING Receiver Received Anger

62 Context effects Perception: What do I see? People pretty good a judging affect Want to get computers to be as good or better than people

63 Issue: people s judgments shaped by context Properties of Receiver Culture mood Mind-reading skills Noise: Lighting, Angel of regard. Properties of Sender Culture Impression mgmt. How can computer take advantage of context Other information Task Other modalities

64 East Asian Western Cognitive biases: e.g., culture People from different cultures attend to different facial features These systematical impact the interpretation of facial expression Blais,, et al. (2008). Culture shapes how we look at faces. PLoS ONE, 3(8) Khooshabeh, et al. (2010). Does culture affect the perception of emotion in virtual faces? 7th Symposium on Applied Perception in Graphics and Visualization, 165

65 Other modalities

66 Other modalities

67 (Aviezer et al. 2008)

68 Behavior of other actors Perceived fear of the target

69 Next lecture Multi-modal recognition

70 Summary How do humans recognize facial expressions? Review some properties/constraints of human expression processing Discuss automatic facial expression recognition Walk through offline experiment Illustrate some of the issues in emotion recognition How to get training data Computational challenges Revisit contextual influences Are we going to find emotion where we typically looking? Context effects and multimodal approaches