Modeling Affect in Dialog Systems CSCI 599: Special Topics. Justin Garten 3/6/13

Transcription

1 Modeling Affect in Dialog Systems CSCI 599: Special Topics Justin Garten 3/6/13

2 What is Affect?! No fixed definition! Emotion, in a broad sense! Expressions Non-verbal signals! Mood Timescale of days rather than minutes! Physiological states Heart rate, hormonal levels, etc! Attitudes Disengagement, uncertainty! Reflexes Surprise, focus on motion 2

3 Why should we care?! Applications: Training! Cultural training! Negotiation Education/tutoring! Empathy with students! Recognize and respond to difficulties Advertising and customer support! Recognize frustration and either respond or escalate to a human! Respond to customer state and improve offers Entertainment! Better game characters! More immersive environments! Beyond simulation, generally improved agents 3

4 So, what is emotion?! Strong Feeling Just a little question-begging! Distinct set of physiological states Which states?! The affective aspect of consciousness Still feels like we re going in circles! God s punishment for disobedience (St Augustine) Difficult to build into a computational system 4

5 Theories of Emotion! Discrete models Innate responses Fixed scripts (expressions, responses) Universal (generally in humans, sometimes assumed in other primates) Ekman! Happiness, sadness, anger, fear, disgust, surprise, (contempt) Tomkins! Interest-excitement, enjoyment-joy, surprise-startle, distress-anguish, anger-rage, fear-terror! Shame-humiliation! Dismell and disgust Some evidence for this:! Facial expressions in congenitally blind! Cultural universality (disputed) 5

6 Dimensional Models of Emotion! Dimensional models Emotions are not distinct circuits, but labels applied to activations of various physiological circuits 2-factor model: arousal and valence 3-factor model: pleasure, arousal, dominance (PAD) Evidence here as well 6

7 Adaptive effects of Emotions! Mobilization of physiological resources Fear -> adrenaline! Social signaling Distress Happiness Trust and social coordination! Focus cognitive resources Adjust processing times Impact depth/breadth of search Bring moves into focus Reason is, and ought to be, only the slave of the passions David Hume 7

8 Appraisal theory! Emotion as a result of cognitive processes Set of basic evaluations! Desirability! Expectedness! Controllability! Causal Attribution Emotion as a result (either direct or indirect) of these evaluations! Agnostic as to discrete versus dimensional models Different versions of appraisal theory can fit either or both! Combination of response and coping Emotions trigger certain action potentials which can act either on the world or on the emotional state 8

9 Example of Appraisal Model! Smith and Lazarus

10 Uses in Dialog Systems! Modeling agents Better simulation Better agents! Modeling humans Affective prediction (i.e. if the user is already angry, a request for them to repeat themselves due to an ASR problem will likely make it worse)! Communication and social signaling Use of affective states and signals as a way to clarify and enrich communication Relevant and appropriate non-verbal behaviors Input to language generation Factor in language understanding 10

11 NLG! Word choice He bumped me versus He shoved me! Sentence focus She took the car versus The car is gone! Syntax Complex/simple Active/passive Clause structure, subject/object distance, etc! Errors Possibly increase errors in situations of higher arousal! Length Short/long utterances! Formality, use of modifiers, force, etc 11

12 NLG (cont)! Major problem: all of these factors vary based on (among other things): Personality Culture Training Interpersonal context Local dialogue context 12

13 NLU! Identification of speech acts! Identification of intent Nice job.! Reference resolution and disambiguation Focal points in environment Salient points for agent! Cognitive resources available for processing An angry agent is less likely to pause and consider alternative meanings! Sentiment analysis Major area of study with broad current applications 13

14 Dialogue Tasks! Negotiation scenarios! Multi-agent situations Focus (affect as both a cue and signal) Mediation Balancing conversational goals! Social grounding Emotional states as features of the world But, more generally, influencing the basic grounding process! Social planning If the person you re talking to is angry at you, it might be the wrong moment to ask for a favor 14

15 Pros and cons of affective agents (first pass)! Pros: More realistic Greater adaptive potential Can operate in a wider range of tasks! Cons: More complicated May actively annoy users May set user expectations too high Can obscure the underlying structure of the agent, slowing user adaptation 15

16 Emotion and Adaptation (EMA)! Based on appraisal theory (Lazarus and Smith) Emotions arise from appraisals of agent-environment relation! Process model: Cognitive assessments Consequences for cognition and behavior Affect of assessments on subsequent assessments! Includes coping responses to emotions For example, sadness over a loss might lead to disengagement from a prior goal 16

17 Appraisals in EMA! Perspective Viewpoint from which the event is judged! Desirability What is the utility of the event (from the given perspective)! Likelihood How probable is the outcome of the event! Causal Attribution Who deserves credit or blame! Temporal Status (past, present, future)! Controllability Can the outcome be altered by actions controlled by the agent! Changeability 17 Can the outcome be altered by some other agent

18 Mapping appraisals to emotions From Marsella and Gratch 2009, EMA: A process model of appraisal dynamics 18

19 Coping behaviors! Examples: 19 Actions! Select an action for execution Planning! Form an intention to perform an act Seek instrumental support! Ask another agent with power over the outcome to intervene Procrastination! Wait for something to change Positive/negative reinterpretation! Increase/decrease the utility of a side effect of an event Acceptance! Drop a threatened intention Mental disengagement! Lower the utility of a desired state Etc.

20 EMA s influence on dialogue! Reference resolution Filter reference resolution by salient features of the environment (defined by appraisal values) Self-centered (agent appraises in terms of it s own interests)! Initiative and direction EMA can cue the dialogue manager on important issues to discuss! NLG Help the dialogue manager to determine speech acts! Suggestion versus order Focus for utterance Words used to describe situation! Match affect and coping strategy In a blame-shifting situation, say They rammed us rather than We collided (Fleischman and Hovy, 2002) 20

21 Dialogue s influence on EMA! Bring key events into focus Topics of discussion will come up for appraisal! Set the context for coping options The dialogue manager will lay out the situation in which EMA will, for example, evaluate coping potential! What moves are available in the present context! Theory of mind The dialogue context will introduce key players. The emotion manager can then evaluate events from those agents perspectives 21

22 Simple Example: Two Bears Scenario! Basic setup (Uncle Justin s Storytime, 2013): Two bears:! A big bear named Barney! A little bear named Paco One ball between them! Originally, Paco has the ball! Simplified appraisal model: Expectedness Valence Causal attribution Control 22

23 Two Bears (cont)! Event: Barney takes the ball from Paco! This event is passed to Paco and the four appraisal processes begin evaluation All appraisals running in parallel. Order is a result of relative speed! First appraisal: Expectedness Paco did not expect this event Evaluation: Expectedness = low Emotion:! Surprise! Second appraisal: Valence Having the ball had a positive utility, so this event is negative Evaluation: Valence = low [Expectedness = low] Emotion:! Fear 23

24 Two Bears (cont)! Third appraisal: Causal attribution Barney is judged responsible for stealing the ball Evaluation: Cause = Barney [Expectedness=low, Valence=low] Emotion:! Anger! Fourth appraisal: Control Barney is a much bigger bear than Paco Evaluation: Control = low [Expectedness=low, Valence=low, Cause=Barney] Emotion:! Sadness 24

25 Two Bears (cont)! Coping At this phase, several coping behaviors are possible, depending on the structure of the scenario! Given that Barney the larger bear: Paco coping options:! Disengage ( I never liked that ball anyway )! Negotiate (if that option is available in the dialogue context)! Alternative: Momma bear (Svetlana) present Paco additional coping options:! Seek instrumental support (cry, directly ask Svetlana for help, etc) 25

26 Two Bears (cont)! Consider the possibilities in terms of this emotional sequence as a series of non-verbal behaviors: Surprise (eyes open, orient towards event) Fear (shift backwards) Anger (orient towards Barney, aggressive posture) Sadness (frown, gaze downward, orient away, bitter tears)! Effectively, the appraisal process gives us a rich emotional display sequence without being forced to model each pattern individually 26

27 An Alternative Approach: VirtualHuman System! Emotion OCC model Appraisal based on character s subjective appraisal rules! Mood PAD model (Mehrabian)! Contrast to EMA incorporation of mood as emotional tendencies! Personality Big 5 model! Mapping to influences on PAD (Mehrabian 1996) 27

28 Other potential alternatives! Variations in underlying model Dimensional rather than discrete! Pattern matching Add appraisal coded responses to NPCEditor! More shallow integration Infer non-verbal cues from dialogue and information state! Physiological models Actively model physiological subsystems (hormones, blood pressure, etc) 28

29 Non-verbal behavior! Types: Expression Gaze Gestures Posture Orientation Nodding! Internally generated or in response to user behavior Affect Mirroring Mimicry 29

30 Levels of modeling! Complete pre-animated sequences Works for scripted responses! Mixed pre-animated sequences Fixed elements, combined on the fly Think of the unit-based TTS systems discussed last week! Full dynamic control Facial Action Coding System (FACS)! Allows for control of facial expression down to individual muscles Kinematic control for gestures 30

31 Example: Backchannel Selection! Sensitive Artificial Listener! Adjusted frequency and selection of backchannels Values based on human studies! For example: higher levels of extroversion generally correlate with more head nods and more active movements! Used a 2-factor model of personality Extroversion Neuroticism! Tested against videos of predicted and inappropriate backchannels for the personality type! Very mixed results in testing. Why? 31

32 Difficulties in affective representations! Determining personality from backchannel frequency would be a difficult task for humans In general, keep in mind how often humans misinterpret one another. That s the baseline we re working against.! Difficulties when working with a limited model A two factor personality model blends a great many things together, possibly muddying interpretation! Phrasing of questionnaires Evaluation is a difficult task in itself 32

33 UNC_DISE_ADAPT! Tutoring scenario Uncertainty (original system) Disengagement (tested addition)! System provided variations in response based on these factors! Limits Kappa 0.55 Difficulties in determining student state from narrow channels! Results emphasize the potential and weaknesses of adding affective components When adding the module for disengagement:! Disengaged students saw improved performance! Engaged students performance declined relative to the original system 33

34 Usefulness of Wizard of Oz! Given the number of pieces involved in any affective dialogue system, this provides a relatively simple way to test hypotheses and frameworks! Advantages: Allows you to factor out the performance of component pieces Provides a baseline for the evaluation of those pieces once the system is built Potentially faster results! Test multiple hypotheses to focus later development! Disadvantages: May allow you to ignore real limitations in system components until too late in the design phase Human capabilities may blur results if they account for signals that may not be available to final system 34

35 Evaluating Affect! At the most basic level, there are a number of complicating factors: Cultural differences Multiple dimensions in any situation! How to separate out factors in a user s response? Difficulties in separating out different aspects of system performance! Users might be responding to the quality of the animation rather than the performance of the system Age related differences! Some older users treating the systems as human versus treating them as tools! Even the survey phrasing is difficult Different labels for a given emotion! Can we even treat emotions as natural kinds? Generally vague language around affect 35

36 Example: Simple Social Signals! Impact of expression on social cooperation! Study using an iterated Prisoner s Dilemma (de Melo 2012)! Agent strategy and behavior is identical (tit-for-tat)! A virtual agent is presented which either emotions based on results Cooperative :! guilt when it defects and the human cooperates! joy with mutual cooperation! anger when the user defects and it cooperates! sadness with mutual defection Individualistic :! joy when it defects and human cooperates! sadness when user defects! neutral at mutual cooperation 36

37 Example: Simple Social Signals! Impact of expression on social cooperation! Study using an iterated Prisoner s Dilemma (de Melo 2012)! Agent strategy and behavior is identical (tit-for-tat)! A virtual agent is presented which either emotions based on results Cooperative :! guilt when it defects and the human cooperates! joy with mutual cooperation! anger when the user defects and it cooperates! sadness with mutual defection Individualistic :! joy when it defects and human cooperates! sadness when user defects! neutral at mutual cooperation 36

38 37

39 Results! Users responded significantly better to cooperative agents! Especially noticeably in the middle rounds! Same expression is interpreted in completely different ways depending on context! The exact mechanism of what s taking place is still unclear Mental model Contagion Etc 38

40 Affective Signaling! Are emotions true signals? Mixed evidence (Russell et al 03, Feldman-Barrett) Different channels provide different clues (and different studies provide different answers):! Prosodic cues tend to be most reliable for arousal, more context-dependent on valence! Facial cues have medium reliability on valence and arousal (but the reliability declines as you add specificity) Smiles seem to be the most universal but, even there, meaning is highly context-dependent! Semantic clues highly content dependent 39

41 Affective Signaling (cont)! How do strategic displays fit in? Cultural differences on which emotions are appropriate to display In strongly hierarchical situations, affective displays may be highly constrained towards superiors Bluffing in poker! Leakage models People mask their true emotions, but signals still leak Ekman and micro-expressions! Mixed evidence! Noisy channel True signals, but obscured by noise source(s)! Issues Many-to-one mapping (many emotions might have the same expression) One-to-many (one emotion has many possible signals) 40

42 Affect and Social Reasoning! We gain information about individuals based not just on their actions, but also on their reaction to other people and events! Affect, though not a perfect channel, is still an essential component of that interaction Causality not critical here! Building a model of another agent Observations:! Actions in the world Including speech acts! Emotional signals Hidden variables:! Goals! Beliefs 41

43 Affect and Social Reasoning (cont)! Inference and model updates based on observations Observe behavior, see if congruent with modeled goals Observe emotional signals! Use social signals as clue about the state of the world Agent frightened and running away, might be a threat in opposite direction! Use model of agent s beliefs and goals to predict Actions Emotional displays! Gets very complex, especially once you start considering strategic versus true displays 42

44 Affect and Social Reasoning in Dialogue! Generally combined with a richer information state Some model of the social situation! Results of utterances and moves Possibly STRIPS-like planning! Possible moves: Information gathering! Raise a topic to see their response So, I heard that Ronaldo went to see Taylor Swift last night... Empathy building! Attempts to build solidarity! Improve social relationships False signals! Manipulation! Comfort 43

45 When to Use Affect?! So, dealing with affect is a lot of work, when should we bother?! Details of scenario Situational requirements User population! Resources available Programmers, writers, artists, time! Test cycle These systems are finicky; you really want to be able to test prior to deployment! Still limited off-the-shelf options for multimodal and affective dialog, but situation likely to improve Better default models Better tools 44

46 Questions? 45