Intelligent User Interfaces affective computing Emotion Models and Affective Computing OCC emotion model Emotion recognition from bio-signals Research of the R. Picard s Affective Computing Group Empathic Feedback Human Affect Affect Emotions, moods, preferences, attitudes,... Also social behavior Affect in humans doesn t generally interfere with effective functioning Under normal conditions, it serves important functions, facilitating effective functioning Abnormal conditions emotional disorders depression, anxiety, mania A.Ortony 1
The OCC Model of Emotions theory of emotion OCC refers to the authors A. Ortony, G.L. Clore,A. Collins Models the antecedents of emotions The cognitive structure of emotions Emotion types are characterized as by their antecedent conditions Emotions are the result of appraising events, agents, and objects Very influential in the computer science community Ref.: A. Ortony, G.L. Clore, and A. Collins, 1988. The Cognitive Structure of Emotions. Cambridge University Press, Cambridge. Positive vs. Negative Affect course granularity Negative affect Increased attention to local events ( here and now ) is useful for immediate threat detection Positive affect Increased attention to global features ( out there ) encourages curiosity and exploratory behaviors A.Ortony 2
Positive vs. Negative Affect examples Happiness Happy-for Gloating Hope Relief Satisfaction Pride Admiration Gratification Gratitude Love /Liking Positive Sadness Resentment Pity Fear Disappointment Fears-confirmed Shame Reproach Remorse Anger Negative Hate/Disliking A.Ortony The OCC Model goals (for events) EVENTS, AGENTS, OR OBJECTS appraised in terms of norms/standards (for agents actions) tastes/attitudes (for objects) desirability praiseworthiness appealingness joy, distress hope, fear relief disappointment anger gratitude gratification remorse pride shame admiration reproach love hate etc. GOAL-BASED EMOTIONS etc. COMPOUND EMOTIONS etc. NORM-BASED EMOTIONS etc. TASTE-BASED EMOTIONS A.Ortony 3
Appraisal Rules formal representation (1) joy(l,f,i,s) if % emotion type wants(l,f,des,s) and % goal holds(f,s) and % belief I = Des. % intensity happy-for(l1,l2,f,i,s) if % emotion type likes(l1,l2,app,s) and % attitude joy(l2,l1,f,des,s) and % belief % (hyp. emotion of L2) log-combination(app,des,i). % intensity Appraisal Rules formal representation (2) angry-at(l1,l2,a,i,s) if % emotion type holds(did(a,l2),s) and % belief causes(a,f,s0) and % belief precedes(s0,s) and % formal condition blameworthy(a,praise,l1) and % standard wants(l1,non-f,des,s) and % goal log-combination(praise,des,i). % intensity 4
Modeling Behavioral Concomitants A. Ortony How do emotions affect behavior? Emotion-behavior linkage has little constraints Emotion response tendencies Strongly influenced by intensity of emotion Emotion Response Tendencies Emotion Response-Tendencies expressive informationprocessing coping somatic flushing behavioral fist-clenching communicative attentional obsessing evaluative despising emotion-oriented problem-oriented preventing recurrence gestural pacing facial scowling verbal swearing self-regulating calming down other-modulating distressing offender planfulness heat A.Ortony 5
Two Examples for Negative Emotions Sample values for Fear emotions % $ ' # &! " A.Ortony Sample values for Anger emotions Expressive Information- Processing $! "! " #! " Coping A.Ortony 6
Example for Positive Emotion Sample values for Joy emotions % Expressive $ & ' &( ' (! " # Information- Processing # Coping # # A.Ortony Emotion Recognition from Bio-signals physiological data assessment with the ProComp device EMG BVP GSR EMG: Electromyography EEG: Electroencephalography EKG: Electrocardiography BVP: Blood Volume Pulse GSR: Galvanic Skin Response Respiration Temperature sensors 7
Physiological Signals summary + ) (+ ', + + ) ( $ * *! "!" Inferring Emotions from Bio-signals P. Lang s 2-dimensional emotion model high Arousal low neg. Valence % - % some named emotions in the arousal-valence space pos. Lang s two dimensions Valence - positive or negative dimension of feeling Arousal - degree of intensity of emotional response Biometric measures Skin conductivity increases with arousal and stress Heart rate increases with negatively valenced emotions Note introverts reach a higher level of emotional arousal than extroverts Ref.: Lang, P. 1995. The emotion probe: Studies of motivation and attention. American Psychologist 50(5):372 385. 8
Signals for Anger and Grief R.Picard Affective Speech vocal effects associated with five emotions Emotion Fear Anger Sadness Happiness Disgust Speech rate much faster slightly faster slightly slower faster or slower very much slower Pitch average very much higher very much higher slightly lower much higher very much lower Pitch range much wider much wider slightly narrower much wider slightly wider Intensity normal higher lower higher lower Pitch changes normal abrupt on stressed syllables downward inflections smooth upward inflections wide downward terminal inflections Ref.: I. R. Murray, J. L. Arnott, 1995. Implementation and testing of a system for producing emotion-by-rule in synthetic speech. Speech Communication (16), 369-390. 9
Autonomic Specificity implications for emotion theory (some) emotions can be distinguished by their associated pattern of autonomic nervous system activity. (Some) emotions have autonomic signatures. Issues Emotions are very short-lived (some seconds) ANS activity is imposed on ongoing internal nervous system activity and in complex contexts (attention/orientation, social interaction) Results Anger, fear, sadness produce higher heart rate than disgust Anger produces higher finger temperature than fear Anger and fear produce higher heart rate than happiness Fear and disgust produce higher skin conductance than happiness Ref.: Levenson, R.W., 2003. Autonomic specificity and emotion. Handbook of Affective Sciences, Ed. by R.J. Davidson, K.R. Scherer, H.H. Goldsmith, Oxford University Press. Which Emotion Model Should We Use? OCC emotion model vs. Ekman emotion model vs. valence/arousal model OCC emotion model (A. Ortony) 22 (24) emotion types: happiness, sadness, happy-for, resentment, gloating, pity, hope, fear,, pride, shame,, gratitude, anger, P. Ekman emotion model 6 basic emotions (universal across cultures): fear, anger, sadness, happiness, disgust, surprise Valence/arousal model (P. Lang) Emotions characterized by 2 dimensions Ref.: Ekman, P., 1992. An argument for basic emotions. Cognition and Emotion 6 (3-4): 169-200. 10
Representing Emotion Influences sigmoidal non-linearity Function describes a variety of natural phenomena Non-linearity of emotion (R. Picard) The same event happening twice (e.g. some music) does not always make you twice as happy (non-linearity) Hearing the same piece of music does not necessarily make you always happy (no time invariance) input x output y Function accounts for activation thresholds Too little activation no emotion is produced Too much activation emotion intensity cannot increase indefinitely (saturation) Steepness of sigmoid slope Aggressive personalities move faster from mild anger to loosing their temper Monitoring ANS Activity for HCI Evaluation short time vs. longer period measurements Two paradigms Measuring short-time (<5sec.) ANS changes in response to specific interface events Comparisons of ANS readings across longer periods of time (>5min.) under different circumstances Short-time ANS measurements Can be used to evaluate specific interface events moment-to-moment assessment of user s emotion Assumes tightly controlled experimental conditions Longer period ANS measurements Can be used to estimate overall impact of interface Works with less tight experimental controls (closer to real world situations and environments) 11
Monitoring ANS Activity for HCI Evaluation media quality (1) Video Study of Wilson and Sasse 00 Monitor ANS activity to assess user cost of different levels of multimedia quality videoconferencing tools Subjects see two recorded interviews at 5-25-5 fps / 25-5-25 fps each frame rate for 5 min 75% of subjects show significant increase in GSR/HR/BVP at 5 fps indicating stress Only 16% noticed that the frame rate had changed! Ref.: Wilson, G.M., Sasse, M.A., 2000. Listen to your heart rate:counting the cost of media quality. In: Affective Interactions Towards a New Generation of Computer Interfaces, Ed. by A. Paiva, Springer, 9-20. Monitoring ANS Activity for HCI Evaluation media quality (2) Web page design Study of Ward and Marsden 03 Monitor physiological responses while performing a task with well-designed vs. poorly-designed web pages poorly organized, many pull-down menus, many pop-ups Poorly-designed web pages cause more stress SC, HR, finger blood volume Ref.: Ward, R.D., Marsden P.H., 2003. Physiological responses to different web page designs. International Journal of Human-Computer Studies 59, 199-212. 12
Affective Computing Rosalind Picard, MIT Media Lab affective computing, computing that relates to, arises from, or deliberately influences emotions. Machines that recognize and respond to user emotion Ref.: R. Picard, 1997. Affective Computing. The MIT Press, Cambridge, MA. This character barges into your office when you re busy. He doesn t apologize, doesn t introduce himself, and doesn t notice you are annoyed. He offers you useless advice. You express more annoyance. He ignores it. He continues to be unhelpful. The clarity of your emotional expression escalates. He ignores it. (this goes on) Finally you have to tell him explicitly go away He winks, and does a little dance before exiting. 13
Affective Computing why should a computer recognize emotions? Human-human communication Based on efficient grounding mechanisms including the ability to recognize interlocutors emotions frustration, confusion, Humans may react appropriately upon detection of an interlocutor s emotion clarification upon confusion Human-computer communication Computers typically lack ability to recognize user emotions Ignoring users emotions causes users frustration Recognizing and responding to users (often) negative emotions may improve users interaction experience Affective Computing research map 14
Affective Computing provides context for learning about when to interrupt how to reduce frustration how to increase interest What Can be Recognized? examples Expressions, behaviors Flared nostrils, tightened lips, a quick sharp gesture, skin conductivity=high So probably she is angry Situation, reasoning, stereotypes That was an important goal to her and Bob just thwarted it So she probably feels angry toward Bob 15
What Can be Recognized? emotions give rise to changes that can be sensed Distance Sensing Up-close Sensing Internal Sensing Face Voice, prosody, linguistic style Gestures, posture, movement, behavior Temperature Respiration Pupillary Dilation Skin conductivity ECG, EEG, Blood pressure Hormones Neurotransmitters Wearable Affect Communicators affective computing applications Wearable skin conductivity communicator Sensing Processing Expression 16
Communicating Frustration affective computing applications Things to communicate frustration Detecting Stress (1) affective computing applications Detect driver stress Simultaneously examine physiology and behavior for recognizing level of stress: up to 96% accurate, across 12 drivers (Healey and Picard, ICPR 00) Ref.: Affective Computing website: http://affect.media.mit.edu/ 17
Detecting Stress (2) affective computing Detect driver stress Siren heard Stress is evident for this person when Driving through city ( City ) Turning around a toll booth ( Turn ) Hearing siren Detecting Interest (1) affective computing applications Detect and learn states of boredom vs. interest Sit upright Lean Forward Slump Back Side Lean 18
Detecting Interest (2) affective computing applications Detect and learn states of boredom vs. interest Results on kids not in training data (Mota and Picard 03) 9-state posture recognition: 89-97% accurate High/low interest, taking a break: 69-83% accurate Ref.: Affective Computing website: http://affect.media.mit.edu/ Demo of Mirroring affective computing 19
Relational Agents (1) affective computing Relational Agents Computational artifacts designed to build and maintain long-term, social-emotional relationships with their users Relational Agents (2) affective computing applications Long-term social-emotional relationship Trust, caring, liking, respect, alliance, warmth Sensitivity to your affective state To support relational functions Empathy and understanding To support interactional functions When not to bother (annoyance) Ability to read subtle, non-verbal training cues Approval / disapproval Ref.: Affective Computing website: http://affect.media.mit.edu/ 20
Relational Agents (3) affective computing applications Study Exercise behavior One month duration 99 subjects Relational Agents Demo 21
Relational Agents (4) affective computing applications Results "Laura and I respect each other." (p <.001) "Laura and I trust one another." (p <.001) "I feel Laura cares about me..." (p <.001) "I feel Laura appreciates me." (p =.009) "I believe Laura likes me." (p <.001) Liking of Laura. (p =.007) Desire to continue working with Laura. (p =.001) Four out of five have seen colleagues hurling abuse at their PCs Three quarters admit that they swear at their computers Nearly half of all people working with computers feel frustrated or stressed because of IT problems A quarter of all under-25-year-olds admit they have kicked their computer (Mori survey in UK, 1250 users) 22
Affective Computing giving empathic response How can machines respond more intelligently given sensing of users affective state? Empathic behavior a form of social intelligence Affect-Support Agent affective computing Strategy Recognize a situation as frustrating (achieved by system design, inserted delays) Is the user willing to talk? If so: Practice active listening with empathy and sympathy Sorry to hear your experience wasn t better This computer apologizes to you for Evaluation Test Affect-Support Agent with 70 subjects with two control conditions: ignore and vent Ref.: Klein, J., Moon, Y., Picard, R., 2002. This computer responds to user frustration: Theory, design, and results. Interacting with Computers. 14, 119-140. 23
Affect-Support Agent active listening Affect-Support Agent empathy 24
Affect-Support Agent empathy Affect-Support Agent venting 25
Results affect-support agent Subjects receiving affect-support showed a significant behavioral effect of reduced frustration compared with both the ignore and vent control groups (p < 0.01) Subjects played the game longer although it was not required Result holds across age, gender, gaming experience Question affect-support agent Why not measure the physiological effect rather than behavioral effect of the computer s empathic behavior directly during the interaction? We designed a mathematical quiz game We also developed an Empathic Companion (EC) system that recognizes user emotion in real-time Ref.: Prendinger, H., Mori, J., Ishizuka, M., 2005. Using human physiology to evaluate subtle expressivity of a virtual quizmaster in a mathematical game. International Journal of Human Computer Studies. Vol. 62, No. 2, 231-245. Ref.: Prendinger, H., Ishizuka, M., 2005. The Empathic Companion: A character-based interface that addresses users affective states. International Journal of Applied Artificial Intelligence, Vol. 19(3-4), 267-285. 26
Can an Embodied Agent Undo User Frustration? experimental study Aim of study Show that an embodied agent may improve users experience (= reduce frustration) Method Measure skin conductance as an index of user stress or frustration objective assessment of user experience Questionnaire users subjective assessment Method experimental study Application domain A simple mathematical quiz game Instruction Addition/subtraction task (short-term memory load) Solve 30 quizzes correctly and as fast as possible Frustration is deliberately caused by delay (in 6 out 30 quizzes) Prototype version of game (possibility of bugs) Subjects 20 university students JPY 1,000 for participation, JPY 5,000 for best score 27
Experimental Setup mathematical quiz game Instruction mathematical quiz game question Add 5 numbers and subtract the i-th number (i < 5) 1 + 3 + 8 + 5 + 4 = [21] 21] E.g.: subtract the 2 nd number? timer 28
Instruction mathematical quiz game answer Select the correct answer by clicking the radio button next to the number Then the character tells whether answer is correct It is correct. (polite language) Delays mathematical quiz game Delay is designed to frustrate (stress) the user Risk of giving incorrect answer Cannot finish the game quickly sometimes delay occurs here (6 14 sec.) 29
Two Versions of the Game independent variables affective vs. non-affective Affective Version Non-Affective Version Character expresses happiness (sorriness) for correct (wrong) answer Character shows empathy (when delay occurs) Character expresses affect both verbally and non-verbally Character may reduce user stress (skin conductance) Description Hypotheses Character does not show affective response Character ignores occurrence of delay Character has no significant effect on user emotion (SC) Agent Response for incorrect answer Affective feedback Verbal expression of sorriness Hanging shoulder gesture Non-affective feedback Wrong. No non-verbal emotion expression I am sorry. It is wrong. (hyper-polite language) 30
Agent Response when delay occurs Affective feedback ( Empathy ) Verbal apology Agent shows non-verbal sign of apology Non-affective feedback Agent ignores the occurrence of delay I am sorry for the delay. (polite language) Skin Conductance During Delay BVP could not be taken reliably BVP Shima agent suddenly freezes DELAY segment Shima agent apologizes RESPONSE- TO-DELAY segment The screenshot shows GSR/BVP data from one user GSR delay start delay end Biograph Software (Thought Technologies) 31
Results impact of empathy Main Hypothesis (Empathy): Skin conductance is lower when the agent shows empathy after a delay occurred, than when the agent does not show empathy. DELAY segment RESPONSE- TO-DELAY segment Non-affective version: mean = 0.08 Affective version: mean = 0.14 t-test (assuming unequal variance) t(16)= 2.47; p =.025 mean values of SC (BVP could not be taken reliably) Note: 9 subjects in each version, data of 2 subjects discarded Results impact of affective response to question score Affective Feedback Hypothesis: When the agent tells whether the subject s answer is right or wrong, skin conductance is lower in the affective version than in the non-affective version Hypothesis is not supported Score Hypothesis: Subjects interacting with the affective version score better. Hypothesis is not supported 32
Results perception of game experience Question I experienced the quiz as difficult. I was frustrated with the delays. I enjoyed playing the quiz game. Nonaffective version 7.5 5.2 6.6 Affective version 5.4 4.2 7.2 Ratings: from 1 ( disagreement ) to 10 ( agreement ) Subjects tend to rate the affective version as less difficult (p = 0.1) Can an Agent Undo User Frustration? summary Main positive results An embodied agent verbally and non-verbally expressing empathy may significantly decrease user frustration and stress An agent with affective behavior (including the display of empathy) may have a positive impact on the user s perception of task difficulty Main negative result The affective agent has no impact on task performance 33