Affective Computing: Emotions in human-computer. interaction. emotions.usc.edu

Transcription

1 Affective Computing: Emotions in human-computer interaction emotions.usc.edu

2 Affective Computing The field of study concerned with understanding, recognizing and utilizing human emotions in the design of computational systems. Motivated by the fact that emotion pervades human life emotions motivate behavior, they promote social bonds between people and between people and artifacts emotional expressions play an important role in forecasting human mental state and future actions. Technology is less efficient if it perturbs human emotions; more efficient if it engages with them productively; more attractive if it appeals to human emotions; and often it is primarily concerned with enabling humans to experience particular emotions (notably happiness) Term coined in Roz Picard s 97 book 2

3 Affective Computing A community of practice International Conferences Professional society 3

4 An illustration of artificial emotions Interviewing a virtual rape victim Patrick Kenny, Thomas D. Parsons, Jonathan Gratch, Albert A. Rizzo. Virtual Patients for Clinical Therapist Skills Training. International Conference on Intelligent Virtual Agents, Paris, France, September

5 Emotion alters social interaction Cognition: alters cognitive processes People behave differently than rational models Quicker to act aggressively (DeSteno et al 2000/2004) Underestimate risk (Lerner & Keltner 2000/2001) IS THIS FOR GOOD REASON AND CAN THIS INFORM ARTIFICIAL INTELLIGENCE? Physical: alters physiological processes Prepare aggressive responses (Keltner & Haidt 1999) Characteristic bodily displays (Spoor&Kelly04, Parkinson01, Ekman) CAN COMPUTERS USES THESE CUES TO SEE INTO USER S MIND? Social: alters the behavior of others Anger elicits fear (even subliminal presentation) (Dimberg&Ohman 1996) Negotiators concede more to angry partner (van Kleef et al. 2007) CAN COMPUTERS USES THESE BEHAVIORS TO INFLUENCE USER? 5

6 Unified Understanding Scientific Questions Cognitive Central questions: what are the cognitive functions of emotion? how can they be modeled computationally? Can these functions be abstracted and inform AI Physical what is the link between cognition and behavior? how is emotion physically manifest? Can we infer emotional state from behavior? Social how do emotional displays impact the cognition and behavior of users? Can synthetic emotions similarly impact users 6

7 Cognitive How does emotion bias cognition? Might this be functional?

8 Anger Lerner&Tiedens06: Portrait of the angry decision maker 8

9 9

10 Questions concerning emotional cognitions Sources of emotion What environmental and cognitive factors trigger an emotion Can computer predict human s emotional response to its own actions or actions of others? Can computer predict that the user is trying to provoke it? Consequences of emotion How do different emotions alter decision-making Can computer predict decision-bias given emotional state Can computer infer emotional state from decision-bias? Functions of emotion Do these biases serve a function? Do the produce better decisions? Are these functions missing from AI systems? Can they be added? 10

11 Illustration Explaining Invasive Ant Scenarios Using Emotion Modeling Joshua Wainer, Dusan Jan, and MinHee Kwon In this paper we presented an emotion-inspired model for simulating local ant behavior and showed that computer simulations confirm the hypothesis presented by biologists. Comparisons of foraging success in simulations between ants of different species show that foraging success is not related with frequency of aggression, but is largely dependent on who initiates the interactions. While the Argentine ants are also numerically dominant in the introduced areas, these behavioral tendencies affect their success as an invader to a greater degree. The fact that their numbers are high further helps them to explore space more effectively as well as encounter and displace ants of native species more frequently at food sources. 11

12 Physical Linking the Cognitive to the Physical (Emotional Leakage?)

13 Mind reading Well, Guy Cuny is the editor of the technology website, news wireless... 13

14 Mind reading 14

15 15

16 E.G. Recognizing Human Nonverbal Behavior Gratch, Morency, Wang Gaze (Morency et al 2007) Facial Expression (Bartlett et al 2006) Gesture (Demirdjian et al 2007) Speech Prosody (Ward; Narayanan) 16

17 Facial Mimicry CERT

18 Questions concerning emotional behaviors Emotional cues What behaviors are diagnostic of human emotion Speech Gesture Expression Posture Physiological markers What are the key features Dynamics, Can these be recognized Learn to predict emotion from data Importance of context/cognitive models in prediction Can these be synthesized Emotional speech, gesture, expression

19 Illustration Emotion recognition using neutral acoustic speech models Carlos Busso, Emily Mower, Takayuki Hidehira Since emotional speech can be regarded as a variation of neutral (nonemotional) speech, it is expected that a robust neutral speech model can be useful in contrasting different emotions expressed in speech. The study presented explores this idea by creating acoustic models trained with spectral features, measured from the emotionally-neutral TIMIT corpus. This method obtained accuracy levels of up to 78% and 65%, respectively. These results suggest that well-trained neutral acoustic models can be effectively used as a front-end for emotion recognition, and once trained with MFB, it may work well regardless of the channel characteristics..

20 Social Social effects of emotional behaviors

21 Prior Findings Real emotional displays can influence users Trust Persuasion Contagion Control Dynamics matter Intra-agent dynamics: Facial expression dynamics impact willingness to cooperate in ultimatum game (Krumhuber et al. 2007) Inter-agent dynamics: Timing/contingency of feedback 21

22 Social impact of displayed emotion Elicits adaptive social responses from others Anger Elicits fear-related responses (even subliminal presentation) (Dimberg&Ohman96) Serves as demand for someone to change course of interaction (Emde, Gaensbaur&Harmon76) Diverts blame or averts someone else s anger or disapproval (cf. Biglan et al., 1985; Clark, Pataki and Carver, 1996). Distress elicits sympathy (Eisenberg et al89) Joy foster social interaction (Haviland&Lelwica87) 22

23 Transactional Emotions Emotion as a dynamic and continuously readjusted relational alignment to the social and physical environment (Parkinson08) Video Video Video *complements Jacqueline Nadel

24 Rapport Agent Architecture Speaker Stereo Camera Microphone stereo images audio signal Watson [1] Laun [2] head postures & body moves voice features Nod, Shake Look left, Look right Response Look up, Look down Posture shift left Uses Posture mapping shift right rules to generate Silence listening behaviors in Normal response speech to extracted Loud speech features Backchannel opportunity point Rising intonation Silence Raised loudness Backchannel joint Speaker shifts posture angles Game Engine gaze up/straight BML [3] head nod head nod mimic SmartBody [4] [1] Morency, L.-P., Sidner, C., Lee, C., & Darrell, T. (2005). Contextual Recognition of Head Gestures. The 7th International Conference on Multimodal Interactions, Torento, Italy. [2] Lamothe, F. and M. Morales (2006). Response Behavior. Marina del Rey, CA, University of Southern California: Technical Report ICT TR [3] Kopp, S., Krenn, B., Marsella, S., Marshall, A., Pelachaud, C., Pirker, H., et al. (2006). Towards a common framework for multimodal generation in ECAs: The behavior markup language. The Intelligent Virtual Agents, Marina del Rey, CA.. [4] Kallmann, M., & Marsella, S. (2005). Hierarchical Motion Controllers for Real-Time Autonomous Virtual Humans. The 5th International Working Conference on Intelligent Vir-tual Agents, Kos, Greece.

25 Face-to-face Responsive Non-contingent

26 Findings Timing of Virtual Human feedback crucial Poor timing produces disfluency, lower rapport (Gratch et al IVA07) Individual differences in how virtual humans impact users Shy subjects heavily influenced by virtual human nonverbal behavior Extroverts less sensitive (Kang et al AAMAS08) Agreeable subjects like agreeable agents (Kang et al AAMAS08) In progress Cultural differences in rapport-eliciting signals (w/ McNeil Lab) Iraqi arabic vs. Mexican vs. American Impact of immediacy cues on learning (Ning Wang) Assessing story recall immediately and after 3 days Impact of immediacy cues on self-disclosure (Sin-hwa Kang) 26

27 Questions concerning emotion influence What behaviors induce a change in the observer Emotional contagion Empathy and counter-empathy Can synthetic displays have same (more) effect Hyper realism 27

28 Illustration The Emotion Mirror: Recognizing Emotion in Speech and Displaying the Emotion in an Avatar Abe Kazemzadeh, Samuel Kim, and Yoonji Kim This paper describes the Emotion Mirror, a demonstration that recognizes emotion in a user's utterance and displays the emotion through the facial expressions of an animated avatar that repeats what the user has said. The demonstration is discussed in terms of system architecture and component design. In addition to the technical details of the system we also look at general engineering and theoretical issues that pertain to the demo system. Finally, describe our experience in creating the system to highlight what worked and what did not. 28