AFFECTIVE COMPUTING. Affective Computing. Introduction. Guoying Zhao 1 / 69

Transcription

1 Affective Computing Introduction Guoying Zhao 1 / 69

2 Your Staff Prof. Guoying Zhao - guoying.zhao@oulu.fi - office: TS302 - phone: Thur. 3-4pm Dr. Xiaobai Li (Assistant Lecturer) - xiaobai.li@oulu.fi, TS321 Mr. Henglin Shi (Teaching Assistant) - henglin.shi@oulu.fi, TS317 Ms. Yante Li (Teaching Assistant) - yante.li@oulu.fi, TS334 Mr. Tuomas Varanka (Teaching Assistant) - tuomas.varanka@student.oulu.fi, TS333 Guoying Zhao 2 / 69

3 Objectives Basic understanding of affective computing Psychological theory Recognition of emotions Synthesizing emotions Applications Guoying Zhao 3 / 69 3

4 Prerequisites Computer engineering Basic Data Structures Programming skills using Python (Python 2.7) Guoying Zhao 4 / 69 4

5 Bird s Eye View of the Course Lecture 1: Course Overview. Introduction to Affective Computing. Sep. 5 (10-12, PR119) Lecture 2: Emotion theory and modeling (Dr. Xiaobai Li). Sep. 6 (12-14, PR119) Lecture 3: Emotion recognition from facial expressions. Sep. 7 (8-10, PR119) Lecture 4: Emotion recognition from speech (Mr. Henglin Shi) Sep. 12 (10-12 Room PR104) Lecture 5: Fusion of multiple modalities (Ms. Yante Li) Sep. 19 (10-12, PR104) Lecture 6: Synthesis of emotional behaviors (Dr. Xiaobai Li) Sep. 26 (10-12, PR104) Lecture 7: Crowdsourcing techniques for affective computing. Oct. 3 (10-12, PR104) Lecture 8: Research related to Affective Computing. Oct. 10 (10-12, PR104) Exam: Oct. 25: Thu 16:15-19:15; 2 nd Exam: Dec. 3: Mon 16:15-19:15. Lecture room: PR119, PR104 Guoying Zhao 5 / 69

6 E1.1, Group1: Facial expression recognition. Sep. 13 (12-14, MA336) E1.1, Group2: Facial expression recognition. Sep. 11 (12-14, TS137) E1.2, Group1: Facial expression recognition. Sep. 20 E1.2, Group2: Facial expression recognition. Sep. 18 E2.1, Group1 : Emotional speech recognition. Sep. 27 E2.1, Group2 : Emotional speech recognition. Sep. 25 E2.2, Group1: Emotional speech recognition. Oct. 4 E2.2, Group2: Emotional speech recognition. Oct. 2 E3.1, Group1: Fusion of multi-modalities for emotion recognition. Oct. 9 E3.1, Group2: Fusion of multi-modalities for emotion recognition. Oct. 11 E3.2, Group1 : Fusion of multi-modalities for emotion recognition. Oct. 16 E3.2, Group2 : Fusion of multi-modalities for emotion recognition. Oct. 18 & Exercise room: MA336 (Group 1) & TS137 (Group 2) Guoying Zhao 6 / 69

7 Course studying Studying this course consists of three parts: following the lecture self-study with the distributed materials doing the assigned exercises (mandatory) Guoying Zhao 7 / 69

8 Grading Study materials: lecture slides, pre-collected code, tools and databases The course consists of lectures and exercises. The final grade is based on the final exam while the exercises need to be passed before exam. Course webpage: Guoying Zhao 8 / 69

9 Feedback New course feedback system: The feedback can be given from the first date of the course until three weeks after the last date. Guoying Zhao 9 / 69

10 Affective Computing: Machines with Emotional Intelligence Guoying Zhao 10 / 69

11 Pioneers Roz Picard introduced the term Affective Computing in 1995 Her book with that title published in 1997 Originally defined as computing that relates to, arises from, and deliberately influences emotion Guoying Zhao 11 / 69

12 A community of practice: HUMAINE Association, 2007 Represented by a professional society Now, Association for the Advancement of Affective Computing a professional, world-wide association for researchers in Affective Computing, Emotions and Human-Machine Interaction International Conference ACII (bi-annual International Conference, started in 2005) International Journal (started in 2010) Guoying Zhao 12 / 69

13 Affective Computing An Interdisciplinary field of research: computer science, psychology, and cognitive science. Research and develop systems that recognize, stimulate and simulate human affect including: How affective sensing can inform machine-understanding of people How affect influences human-computer and human-robot interaction and machine usability How to make computers more human-like The ethics of giving machines emotional capabilities Covers but is not limited to the topics involving: Psychology and behavior as they relate to affective computing Behavior generation and user interaction. Guoying Zhao 13 / 69

14 Motivations and Goals Research shows that human intelligence is not independent of emotion. Emotion and cognitive functions are inextricably integrated into the human brain. Automatic assessment of human emotional/affective state. Creating a bridge between highly emotional human and emotionally challenged computer systems/electronic devices - Systems capable of responding emotionally. The central issues in affective computing are representation, detection, and classification of users emotions and synthesis of emotions for response. Norman, D.A. (1981). Twelve issues for cognitive science Picard, R., & Klein, J. (2002). Computers that recognize and respond to user emotion: Theoretical and practical implications. Guoying Taleb, T.; Bottazzi, Zhao D.; Nasser, N.;, "A Novel Middleware Solution 14 / 69to Improve Ubiquitous Healthcare Systems Aided by Affective Information,"

15 Dacher Keltner (professor of psychology at University of California, Berkeley) helped filmmakers understand emotions for the Pixar movie Inside Out.

16 Affective Computing Research Affective computing can be related to other computing disciplines such as Artificial Intelligence (AI), Virtual Reality (VR) and Human Computer interaction (HCI). Questions need to be Answered? What is an affective state (typically feelings, moods, sentiments etc.)? Which human communicative signals convey information about affective state? How various kinds of affective information can be combined to optimize inferences about affective states? How to apply affective information to designing systems? The research areas of affective computing visualized by MIT (2001) Guoying Zhao 16 / 69 M. Pantic, N. Sebe, J. F. Cohn, and T. Huang, Affective multimodal human-computer interaction. In ACM International Conference on Multimedia (MM).

17 Steps towards affective computing research We first need to define what we mean when we use the word emotion. Second, we need an emotion model that gives us the possibility to differentiate between emotional states. In addition, we need a classification scheme that uses specific features from an underlying (input) signal to recognize the user s emotions. The emotion model has to fit together with the classification scheme used by the emotion recognizer. Guoying Zhao 17 / 69 R. Sharma, V. Pavlovic, and T. Huang. Toward multimodal human-computer interface. In Proceedings of the IEEE, 1998.

18 How Emotion/Affection is Modeled? In affective computing, affect is often seen as another kind of information discrete or continuous units or states internal to an individual that can be transmitted in a loss-free manner from people to computational systems and back. Affection description perspectives Discrete Emotion Description Happiness, fear, sadness, surprise, disgust, anger Dimensional Description valence, arousal, dominance Valence: the pleasantness of the stimulus Arousal: the intensity of emotion provoked by the stimulus Dominance: the degree of control exerted by the stimulus Guoying Zhao 18 / 69

19 Mind-Read > Act > Persuade hmm Roz looks busy. Its probably not a good time to bring this up Analysis of nonverbal cues Inference and reasoning about mental states Modify one s actions Persuade others

20 Skills of Emotional Intelligence Recognizing emotions Having emotions Expressing emotions Regulating emotions Utilizing emotions if have emotion Guoying Zhao 20 / 69

21 Robotic Computer Research Areas - Recognizing another s emotions - Expressing emotions - Handling another s emotions and showing emotions Affective and Cognitive Decision Making - Regulating and utilizing emotions - Affect as a self-adapting control system Affect changes the operating characteristics of other three domains (cognition, motivation, behavior) Guoying Zhao 21 / 69

22 Guoying Zhao 22 / 69 Some typical recent research topics (from TAC) Games/Entertainment computing Emotion Recognition in: Speech: Responses to victory and defeat Emotion in natural speech Affective music player Depression detection Boredom detection Face Modeling Recognizing expressions Intensity analysis Modeling emotional influences on Physiology decision-making Inferring response to music via EEG Modeling factors that elicit emotions Detecting stress from skin conductance Applications Text Opinions in twitter; blogs Health: detection and shaping Emotocons Synthesis Emotional speech Emotional facial expressions Games/entertainment: detection and shaping; synthesis/realism Education: detections; shaping Behavioral Science

23 Emotional computer Including facial expression, speech and gesture analysis and facial expression and speech synthesis. Guoying Zhao 23 / 69

24 Computer recognizes emotions Guoying Zhao 24 / 69

25 Affection detection sources: Speech/Vocal expression Facial Expression Affection Detection and Recognition Bio-signals (Psychological sensors, Wearable sensors) Brain Signal, skin temperature, blood pressure, heart rate, respiration rate Gesture Limbic movements Text Rafael A. Calvo, Sidney D'Mello, "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications Guoying Leon, E.; Zhao Clarke, G.; Sepulveda, F.; Callaghan, V., "Optimised attribute selection for emotion 25 / 69 classification using physiological signals

26 Affection Detection and Recognition Affection recognition modalities Unimodal Multimodal provide a more natural style for communication Rafael A. Calvo, Sidney D Mello, Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Guoying Applications. Zhao 26 / 69 Zhihong Zeng; Pantic, M.; Roisman, G.I.; Huang, T.S.; "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions,"

27 Speech in affective computing Affective messages in speech Emotions, moods, feelings Intentional, elicited Lexical (explicit) Affective words, stress, etc. Paralinguistic (implicit) vocal signals beyond the basic verbal message convey meaning beyond the words and grammar used Speech prosody voice quality vocalizations Guoying Zhao 27 / 69

28 Affection Recognition Method Voice / Speech Paralinguistic Features of Speech how is it said? Prosodic features (e.g., pitch-related feature, energy-related features, and speech rate) Spectral features (e.g., MFCC - Mel-frequency cepstral coefficient and cepstral features) Spectral tilt, LFPC (Log Frequency Power Coefficients) F0 (fundamental frequency of speech), Long-term spectrum Speech disfluencies (e.g., filler and silence pauses) Context information (e.g., subject, gender, and turn-level features representing local and global aspects of the dialogue) Nonlinguistic vocalizations (e.g., laughs and cries, decode other affective signals such as stress, depression, boredom, and excitement) Guoying Zhao 28 / 69 Rafael A. Calvo, Sidney D'Mello, "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications, IEEE Transactions on Affective Computing, 1(1): 18-37, 2010

29 Acted Corpora Bored? Interested? Sad? Angry? Happy? confident frustrated friendly anxious encouraging

30 Acted Corpora happy sad angry confident frustrated friendly interested anxious bored encouraging

31 Affection Recognition Method Speech Recognition Architecture Sadness, anger, and fear are the emotions that are more easily recognized through voice than disgust. Speech Signal Pre-processing Feature Extraction Classification Audio recordings collected in call centers and, meetings, Wizard of Oz scenarios interviews and other dialogue systems Classified Result M. Pantic, N. Sebe, J. F. Cohn, and T. Huang. Affective multimodal human-computer interaction. In ACM International Conference on Multimedia (MM), Guoying Rafael A. Calvo, Zhao Sidney D'Mello, "Affect Detection: An Interdisciplinary 31 / 69 Review of Models, Methods, and Their Applications

32 Segmentation Emotion recognition from speech Active speech level, voiced/unvoiced, phonemes (/p/, /ʊ/, /ʃ/), etc. Speech feature extraction Acoustic features Spectral features and other transformations Glottal source parameterization Formant analysis Prosodic features Pitch/F0, Loudness/Intensity, Rhythm/Duration Voice quality Guoying Zhao 32 / 69

33 Emotion recognition from speech Machine learning tools used frequently Feature selection and transformations Sequential floating search (SFS), principal component analysis (PCA), nonlinear manifold modeling, etc. Classifiers Linear discriminant analysis (LDA), k-nearest neighbors (knn), support vector machines (SVM), hidden markov models (HMM), neural networks (NN) Validation and bias Cross-validation, structural risk minimization (SRM), etc. Guoying Zhao 33 / 69

34 Facial Expression Guoying Zhao 34 / 69

35 Facial Expression Recognition Mug Shot Dynamic Information Intensity Action Units Prototypic Emotional Expressions Psychological studies have suggested that the facial motion is fundamental to Guoying Zhao the recognition of facial expressions. 35 / 69

36 Emotion recognition from facial expressions Feature extraction Appearance features Local binary pattern Local binary pattern from three orthogonal planes Gabor, HOG, SIFT(Scale-invariant feature transform), etc. Shape features Distance between facial landmarks, angle, coordinates of facial landmarks, etc. To locate facial landmarks using ASM, AAM, CLM, DRMF(Discriminative Response Map Fitting), etc. Deep learning features CNN, RNN Guoying Zhao 36 / 69

37 Affection Recognition Method Facial Expression Example: Active Appearance Model (AAM) (AAM) based system which uses AAMs to detect and track the face and extract visual features. Guoying Zhao 37 / 69

38 Example: LBP-TOP (a) Non-overlapping blocks(9 x 8) (b) Overlapping blocks (4 x 3, overlap size = 10) (a) Block volumes (b) LBP features (c) Concatenated features for one block volume from three orthogonal planes with the appearance and motion Guoying Zhao 38 / 69 Features in each block volume

39 Guoying Zhao 39 / 69

40 Physiological Signals / Bio-Signals Physiological signals derived from e.g., Central Nervous System (CNS) and Peripheral Nervous System (PNS) of human body. Fear for example increases heartbeat and respiration rates, causes palm sweating, etc. Physiological information used: GSR - Galvanic Skin Resistance RESSP - Respiration BVP - Blood Pressure Skin Temperature Electrocardiography (ECG) Electrodermal activity (EDA) Electromyogram (EMG) Electroencephalogram (EEG) Guoying Zhao 40 / 69

41 Electroencephalogram (EEG), Electrocardiography (ECG), Electrodermal activity (EDA), Electromyogram (EMG) ECG: recording of the electrical activity of the heart EEG: recording of electrical activity along the scalp EDA: to measure the electrical conductance of skin, which varies depending on the amount of sweatinduced moisture on the skin. Guoying Zhao 41 / 69 EMG: for evaluating and recording the electrical activity produced by skeletal muscles.

42 Gesture / Body Motion Pantic et al. s survey shows that gesture and body motion information is an important modality for human affect recognition. Combination of face and gesture is more accurate than facial expression alone. Two categories of Body Motion based affect recognition Stylized The entirety of the movement encodes a particular emotion. Non-stylized More natural - knocking door, lifting hand, walking etc. Example: Applying SOSPDF (shape of signal probability density function) feature description framework in captured 3D human motion data Guoying Zhao 42 / 69

43 Frequently used Modeling Techniques Fuzzy Logic Neural Networks (NN) Hybrid: Fuzzy + NN Tree augmented Naïve Bayes Hidden Markov Models (HMM) K-Nearest Neighbors (KNN) Linear Discriminant Analysis (LDA) Support Vector Machines (SVM) Gaussian Mixture Models (GMM) Discriminant Function Analysis (DFA) Sequential Forward Floating Search (SFFS) Deep learning Guoying Zhao 43 / 69

44 Computer expresses emotions Guoying Zhao 44 / 69

45 Facial expression synthesis Interact with virtual agents: help desks, museum guides, online shopping, and distance learning We would like such virtual characters to behave realistically. This involves understanding the signals we use in non-verbal communication, and enabling the virtual character to display such signals. Such understanding can be achieved through the analysis of data collected from humans interacting with each other or with computers. This data can then be used to teach the virtual characters to express themselves in an appropriate manner. Guoying Zhao 45 / 69

46 Facial expression synthesis Computer Laboratory of University of Cambridge Synthesized affective gestures: Guoying Zhao 46 / 69

47 Emotional speech synthesis The overall goal of the speech synthesis research community is to create natural sounding synthetic speech. One way synthesized speech benefits from emotions is by delivering certain content in the right emotion (e.g. good news are delivered in a happy voice), therefore making the speech and the content more believable. Emotions can make the interaction with the computer more natural because the system reacts in ways that the user expects. Emotional speech synthesis is a step in this direction. Guoying Zhao 47 / 69

48 Effects of mood on decision making (Lerner & Keltner 2000, 2001, 2004) A person s mood at the time of decision-making also influences issue interpretation and his or her choice under risk. Happiness Optimistic about judgments of future events Anger Optimistic judgments of future events, Risk-Seeking choices Fear Pessimistic judgments of future events, Risk-Aversive choices Guoying Zhao 48 / 69 Sadness Pessimistic about the present and make a choice to change the current sad state. Reverse Endowment Effect

49 Applications In the security sector affective behavioural cues play a crucial role in establishing or detracting from credibility In the medical sector, affective behavioural cues are a direct means to identify when specific mental processes are occurring Neurology (in studies on dependence between emotion dysfunction or impairment and brain lesions) Psychiatry (in studies on schizophrenia and mood disorders) Dialog/Automatic call center Environment to reduce user/customer frustration Academic learning Human Computer Interaction (HCI) Zhihong Guoying Zeng; Zhao Pantic, M.; Roisman, G.I.; Huang, T.S.; 49 /, 69 "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," Pattern Analysis and Machine Intelligence

50 Context Aware Multimodal Affection Analysis Based Smart Learning Environment Guoying Zhao 50 / 69

51 Output Face Analysis Voice Analysis Posture Analysis Multimedia Note Reading Behavior Report Lesson Length Suggestion System Controller Class Efficiency Report Hardware Calibration Manager Physiology Analysis Decision Support System Multimodal Affect Input Application System Architecture Parameter Adjustment

52 Driver Emotion Aware Multiple Agent Controlled Automatic Vehicle

53 Basic Emotions Stress Level Complex Emotions Feature Estimator Alert the Driver Driving Aid Agent Speed, ABS, Traction Control Audio Linguistic / Non-linguistic Feature Detector Facial Expression Feature Detector Navigation Agent Route Selection Bio-signals Actions Steering Movement Interaction with Gas / Break Paddle... Safety Agent Inter agent communication to aid decision making Notify in case of Emergency Music, Climate Control Seat Pressure Feature Detector Affective Multimedia Agent 53

54 Concerned Issues Privacy concerns I do not want the outside world to know what goes through my mind Ethical concerns Robot nurse or caregivers capable of effective feedback Risk of misuse of the technology In the hand of impostors Computers start to make emotionally distorted, harmful decisions (In the Terminator films, computers become self-aware and try to exterminate human life) Complex technology Effectiveness is still questionable, risk of false interpretation Guoying Zhao 54 / 69

55 Future Research Directions So far Context has been overlooked in most Affection Computing researches Collaboration among Affection researchers from different disciplines Fast real-time processing Multimodal detection and recognition to achieve higher accuracy Systems that can model conscious and subconscious user behaviour Guoying Rafael A. Zhao Calvo, Sidney D'Mello, "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their 55 / 69 Applications. IEEE Transactions on Affective Computing, 1(1): 18-37, 2010

56 Databases

57 Facial expression: JAFFE (Easy) Static facial expression images 213 images of 7 facial expressions 10 Japanese females Protocol Leave-one-subject-out Leave-one-sample-out Guoying Zhao 57 / 69

58 Dynamic facial expression Cohn-Kanade (Medium) 372 facial expression sequences Six facial expressions (Anger, Disgust, Fear, Happiness, Sadness and Surprise) Protocol Leave-one-subject-out Guoying Zhao 58 / 69

59 MAHNOB (Hard) Spontaneous facial expression Video Nine emotion keywords Valence and arousal mapping Selected session: 267 samples from 14 subjects Protocol Leave-one-subject-out Valence & Arousal dimension Table. The emotion keywords are mapped into three levels on arousal and valence Arousal classes Emotional keywords Calm sadness, disgust, neutral Medium arousal joy and happiness, amusement Excited/Activated surprise, fear, anger, anxiety Valence classes Emotional keywords Unpleasant fear, anger, disgust, sadness, anxiety Neutral valence surprise, neutral joy and happiness, amusement Guoying Zhao Pleasant 59 / 69

60 AFEW (Hard) Dynamic temporal facial expression data Close to the real-world condition Seven facial expressions Audio & Visual More difficult Protocol 10-fold-cross-validation Guoying Zhao 60 / 69

61 Emotional speech databases Available speech databases MediaTeam emotional speech corpus (easy) Finnish, acted (stereotypical), single modal, basic emotions (4-7 discrete classes), proprietary Hytke (challenging) Finnish, spontaneous (emphasized), multimodal, SAM scale (dimensional annotation: valence, activation) Speech, facial video, heart rate (RR), eye tracking, posture Guoying Zhao 61 / 69

62 Available biosignal databases DEAP ( Electroencephalogram (EEG) and peripheral physiological signals (also some facial videos) Dimensional annotation (self-report and online ratings) Database descriptions me.html ac_special_issue_2011.pdf Guoying Zhao 62 / 69

63 Emotion recognition from multi-modal Feature extraction Video/image: LBP, LBP-TOP, Shape etc. Speech: Acoustic features, Prosodic features etc. Bio-signal: EEG: spectral power (SP) feature, spectral power difference (SPD) features, etc. Fusion Decision-level on classifiers SUM, MAX, PRODUCT, MIN, MEAN, etc. Feature-level Multiple kernel learning (MKL), Canonical correlation analysis (CCA), etc. Guoying Zhao 63 / 69

64 Multi-modal Databases AFEW (challenging) Guoying Zhao 64 / 69 Available databases Spontaneous, multimodal, seven categorized emotions Facial video & Speech 325 video clips with seven emotions (surprise, sadness, neutral, sadness, neutral, happiness, fear, disgust and anger) Hytke (challenging) 24 females Finnish, spontaneous (emphasized), multimodal, SAM scale (dimensional annotation: valence and activation) Speech, facial video, heart rate, eye tracking, posture

65 Available databases Multimodal databases MAHNOB (Challenging) Spontaneous, multimodal, dimensional annotation: valence and arousal Facial video, EEG signal (32 channels), ECG, respiration amplitude and skin temperature The physiological signals are stored using BDF/MAT format, which is readable by EEGLAB, Matlab, EDFB browser, etc. (We provide BDF and mat format for physiological signals) Guoying Zhao 65 / 69

66 Face detection Facial expression tools _face_detection/py_face_detection.html Appearance features: LBP/LBP-TOP Landmark Detection for geometric features Discriminative Response Map Fitting (DRMF): / ASM, AAM Guoying Zhao 66 / 69

67 Praat Speech tools (GPL licenses) Phonetics toolbox/software, for prosody and general speech analysis Voicebox MATLAB toolbox for speech processing, feature extraction etc. Guoying Zhao 67 / 69

68 EEGLAB Biosignal tools (GPL licenses) Graphical EEG signal processing environment/toolbox for MATLAB BioSig MATLAB Toolbox for biosignal processing, feature extraction etc. Guoying Zhao 68 / 69

69 Fusion Guoying Zhao 69 / 69 Fusion and Classification Decision-level on classifiers SUM, MAX, PRODUCT, MIN, MEAN, etc. Feature-level Classifiers: SVM Multiple kernel learning (MKL), Canonical correlation analysis (CCA), etc. Libsvm HMM CRF KNN