Face Analysis for Emotional Interfaces

Transcription

1 Face Analysis for Emotional Interfaces Matti Pietikäinen, Guoying Zhao Center Machine Vision and Signal Analysis, University of Oulu, Finland Human faces contain lots of information Identity Demographic information (e.g., gender, age, race/ethnicity...) Emotions ( happy, sad angry, surprise, etc.) Direction of attention (head pose, gaze direction) Visual speech (+ audio speech) Health (e.g., pain, autism, mental disorders,...) Even unvisible information (e.g., heart rate, micro-expressions) Face information is very important for developing emotionally intelligent systems to various applications 1

2 Applications Education: online learning Health and medicine: autism, emotional well-being HCI: emotional chatpots, human-robot interaction Security/safety: interogation, border control, safe driving Job interviews, video conferences User/customer experience analysis AI with emotional intelligence Jiaojiao: The bank lobby manager robot was deployed in Chinese banks to introduce financial products and help customers. 2

3 Affective human-robot interaction Röning J, Holappa J, Kellokumpu V, Tikanmäki A & Pietikäinen M (2014) Minotaurus: A system for affective human-robot interaction in smart environments. Cognitive Computation 6(4):

4 A perceptual interface for face analysis 4

5 Examples of key results from our research Image and video description with local binary patterns (LBP) Benchmark of LBP variants and deep texture descriptors Face description with LBPs Facial expression recognition Pain intensity analysis from facial expressions Group-level happiness intensity analysis Recognition and spotting of spontaneous facial microexpressions Remote heart rate measurement from video data Face anti-spoofing by detecting pulse from videos Visual speech analysis and animation Multi-modal emotion analysis Local Binary Pattern and Contrast operators Ojala T, Pietikäinen M & Harwood D (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recognition 29: ,283 citations in Google Scholar ( ) - Most cited paper of the Pattern Recognition journal! An example of computing LBP and C in a 3x3 neighborhood: example thresholded weights Pattern = LBP = = 241 C = ( )/5 - (5+2+1)/3 = 4.7 Important properties: LBP is invariant to any monotonic gray level change computational simplicity 5

6 Multiscale LBP Ojala T, Pietikäinen M & Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with Local Binary Patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7): (an early version at ECCV 2000). - 10,645 citations in Google Scholar - Most cited paper of the top-ranking IEEE PAMI journal since the most cited Finnish paper in ICT area published since arbitrary circular neighborhoods - uniform patterns - multiple scales - rotation invariance - gray scale variance as contrast measure Median Robust LBP Liu L, Lao S, Fieguth P, Guo Y, Wang X & Pietikäinen M (2016) Median robust extended local binary pattern for texture classification. IEEE Transactions on Image Processing 25(3):

7 Liu L, Fieguth P, Guo Y, Wang X & Pietikäinen M (2017) Local binary features for texture classification: Taxonomy and experimental study. Pattern Recognition, 62: Table from: Liu L, Fieguth P, Wang X, Pietikäinen M & Hu D (2016) Evaluation of LBP and deep texture descriptors with a new robustness benchmark. In: Computer Vision, ECCV 2016 Proceedings, Lecture Notes in Computer Science, 9907:

8 LBP vs. CNN: Handcrafted features vs. handcrafted architectures LBP: + excellent performance of some recent variants (e.g., L. Liu et al., IEEE TIP 2016) + computational efficiency -> without GPU computations + robustness to some image degradations, e.g. rotations, noise (e.g., MRELBP) + used in various applications (e.g., biometrics, medical image analysis, motion analysis) - most variants cannot be trained - not so effective for macro-textures and large appearance changes CNN: + effective also for macro-textures and large appearance changes + able to learn texture descriptors at multiple scales from data + end-to-end training of filters and of classification of filter responses - need massive amounts of training data (not available in many applications) - computationally very expensive -> need GPU processing - not so robust to image rotations and noise - currently limited use in real-world applications What next?: Combining strengths of LBP and CNN descriptors moderate amounts of training data (as required by many applications) more insensitivity to image trasformations and degradations compactness and efficiency -> can be used e.g. in mobile/wearable devices with strict low-power constraints How to achieve these goals? A Survey of Recent Advances in Texture Representation Li Liu, Jie Chen, Paul Fieguth, Guoying Zhao, Rama Chellappa, Matti Pietikäinen (Submitted on 31 Jan 2018) Texture is a fundamental characteristic of many types of images, and texture representation is one of the essential and challenging problems in computer vision and pattern recognition which has attracted extensive research attention. Since 2000, texture representations based on Bag of Words (BoW) and on Convolutional Neural Networks (CNNs) have been extensively studied with impressive performance. Given this period of remarkable evolution, this paper aims to present a comprehensive survey of advances in texture representation over the last two decades. More than 200 major publications are cited in this survey covering different aspects of the research, which includes (i) problem description; (ii) recent advances in the broad categories of BoW-based, CNN-based and attribute-based methods; and (iii) evaluation issues, specifically benchmark datasets and state of the art results. In retrospect of what has been achieved so far, the survey discusses open challenges and directions for future research. Comments: 27 pages and 29 figures, submitted to IJCV Subjects: Computer Vision and Pattern Recognition (cs.cv); Learning (cs.lg) MSC classes: 68T10 Cite as: arxiv: [cs.cv] (or arxiv: v1 [cs.cv] for this version) 8

9 Spatiotemporal LBP Zhao G & Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6): ,741 citations in Google Scholar Face description with LBP Ahonen T, Hadid A & Pietikäinen M (2006) Face description with local binary patterns: application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(12): (an early version published at ECCV 2004) - 4,474 citations in Google Scholar (ECCV paper: 2,377 citations) - 3rd most cited paper of PAMI journal since ECCV paper was awarded with Koenderink Prize 2014 for fundamental contributions in computer vision 9

10 Adopted from a slide by Anil K. Jain 2014 Taigman et al. DeepFace Facial expression recognition 10

11 Zhao G & Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6): Facial expression recognition in determining the emotional state of the face Regardless of the identity of the face (a) Non-overlapping blocks(9 x 8) (b) Overlapping blocks (4 x 3, overlap size= 10) (a) Block volumes (b) LBP features (c) Concatenated features for one block volume from three orthogonal planes with the appearance and motion 11

12 Demos for acted facial expression recognition Guo Y, Zhao G & Pietikäinen M (2016) Dynamic facial expression recognition with atlas construction and sparse representation. IEEE Transactions on Image Processing 25(5): Towards spontaneous expressions 12

13 Dealing with facial occlusion Real-world challenge (Occlusion) Existed occlusion detections (on static images) Huang X, Zhao G, Zheng W & Pietikäinen M (2012) Towards a dynamic expression recognition under facial occlusion. Pattern Recognition Letters 33(16): Dealing with multiple views Huang X, Zhao G & Pietikainen M (2013) Emotion recognition from facial images with arbitrary views. BMVC

14 3D action unit detection Lip Stretcher Lip Funneler Lip Corner Puller Lip Corner Depressor Chin Raiser Lip Tightener Bayramoglu N, Zhao G & Pietikainen M (2013) CS-3DLBP and geometry based person independent 3D facial action unit detection. ICB Pain Intensity Estimation 14

15 A pain scale measures a patient's pain intensity or other features. Automatic pain intensity estimation has received increased attention recently Wide applications in health care, ranging from monitoring patients in intensive care units to assessment of chronic lower back pain Figure. An example of sequences from the UNBC-MacMaster Shoulder Pain Expression Archive dataset, with four video-level and two frame-level pain ratings. Overview of the proposed second pooling framework. We take frame level pain intensity estimation as an example We propose a 2 nd order standardized moment average pooling technique to capsule both low-level information from raw features and the mid-level ones from the local descriptors and form the image representations. Relevance Vector Regression (RVR) is used to learn the mapping from image representations to the frame level intensity (PSPI scale). Table. Pain intensity estimation on the UNBC-MacMaster Shoulder Pain Expression Archive dataset. The Mean Squared Error (MSE) and the Pearson Product-moment Correlation Coefficient (PCC) are used as the quantitative Methods MSE PCC PTS DCT LBP PTS/DCT/LBP_average PTS/DCT/LBP_RVR Ours Hong X, Zhao G, Zafeiriou S, Pantic M & Pietikäinen M (2016) Capturing Correlations of Local Features for Image Representation. Neurocomputing 184:

16 Pain intensity estimation with RCNN Zhou J, Hong X, Su F & Zhao G (2016) Recurrent convolutional neural network regression for continuous pain intensity estimation in video. Proc. CVPR Workshops. Traditional static methods prefer to extract features from frames separately in a video, unstable changes and peaks among adjacent frames. We propose a real-time regression framework based on the recurrent convolutional neural network (RCNN) for automatic frame-level pain intensity estimation Group-level happiness intensity analysis Huang X, Dhali A, Zhao G, Goecke R & Pietikäinen M (2015) Riesz-based volume local binary pattern and a novel group expression model for group happiness intensity analysis. Proc. the British Machine Vision Conference (BMVC 2015), Swansea, UK, 13 p. 16

17 Group-level happiness intensity analysis Background Large pool of data containing multiple people Crowd analysis and family relationships Purpose Infer emotional intensity of a group of people Issues Feature extraction Group expression model (GEM) Table. Comparison of various GEM approaches with HOG, LBP, LPQ and RVLBP (Mean absolute error) Methods Feature HOG LBP LPQ RVLBP GEM avg GEM w GEM LDA GEM CCRF

18 Micro-expression (ME) analysis The first paper on spontaneous micro-expressions: Pfister T, Li X, Zhao G & Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. Proc. International Conference on Computer Vision (ICCV 2011), Barcelona, Spain, Q: Micro-expression Paul Ekman Micro-expression rapid involuntary facial expressions reveal suppressed affect reveal contradictions between facial expressions and the emotional state enable recognition of suppressed emotions. Contempt micro-expression Micro-expression spotting and recognition

19 Attribute Microexpressions Macroexpressions Duration < 0,5 s 0,5-4 s Facial movements Subtle Clearly visible Emotion Repressed Expressed Yan, W.J., Wu, Q., Liang, J., Chen, Y.H., Fu, X.: How fast are the leaked facial expressions: The duration of microexpressions. Journal of Nonverbal Behavior 37(4), (2013) Idea Read hidden emotions in faces 19

20 Potential applications Reveal lies Our method Interrogation Price negotiation Find right price SMIC: Micro-expression Database 20

21 Researcher High speed camera Emotional movie clips Participant Movie to induce Negative emotion: 1/10 Speed 21

22 SMIC database Other databases: Micro-expression recognition 22

23 Our research reported in MIT Technology Review, Nov. 2015, based on our article at Machine Vision Algorithm Learns to Recognize Hidden Facial Expressions Microexpressions reveal your deepest emotions, even when you are trying to hide them. Now a machine vision algorithm has learned to spot them, with wide-ranging applications from law enforcement to psychological analysis. Final version of the arxiv report: Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G & Pietikäinen M (2017) Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting. and recognition methods. IEEE Transactions on Affective Computing, in press (available online). First framework: An example of a facial micro-expression (top-left) being interpolated through graph embedding (top-right); the result from which spatiotemporal local texture descriptors are extracted (bottom-right), enabling recognition using multiple kernel learning. 23

24 New Framework Micro-expression spotting 24

25 Framework Demo for Spotting facial movements: 25

26 Heart-rate measurement from videos Color-based method on a face video Ambient light ,5 93,5 92, Tiny signal; Many noise factors; Eulerian Video Magnification for Revealing Subtle Changes in the World" -- Wu et al

27 Heart rate measurment from videos (remotely!) Li X, Chen J, Zhao G & Pietikäinen M (2014) Remote heart rate measurement from face videos under realistic situations. Proc. International Conference on Computer Vision and Pattern Recognition (CVPR 2014). Demo with MAHNOB database Heart rate variability contains information about emotions and identity 27

28 Generalized face anti-spoofing BY DETECTING pulse FROM FACE VIDEOS Xiaobai Li, Jukka Komulainen, Guoying Zhao, Pong-chi Yuen, and Matti Pietikäinen CMVS, University of Oulu, Finland Li X, Komulainen J, Zhao G, Yuen PC & Pietikäinen M (2016) Generalized face anti-spoofing by detecting pulse from face videos. Proc. International Conference on Pattern Recognition (ICPR 2016) Application: face antispoofing for biometrics Without countermeasures biometric systems are vulnerable to spoofing attacks Recognition algorithms try to identify/verify instead of checking whether the biometric input is genuine Especially true for face biometrics falsifying biometric data is easy Social media (Facebook, etc.) Hard to hide your face in public Paper prints, video displays, masks 28

29 THE PROBLEM 3) Mask Attack: o 3D printing o Depth o Eye movements REAL-F 3DMAD Visual speech recognition Zhao G, Barnard M & Pietikäinen M (2009). Lipreading with local spatiotemporal descriptors. IEEE Transactions on Multimedia 11(7): Visual speech information plays an important role in speech recognition under noisy conditions or for listeners with hearing impairment. A human listener can use visual cues, such as lip and tongue movements, to enhance the level of speech understanding. The process of using visual modality is often referred to as lipreading which is to make sense of what someone is saying by watching the movement of his lips. McGurk effect [McGurk and MacDonald 1976] demonstrates that inconsistency between audio and visual information can result in perceptual confusion. 29

30 Features in each block volume. Mouth movement representation. Demo for visual speech recognition 30

31 Zhou Z, Hong X, Zhao G & Pietikäinen M (2014) A compact representation of visual speech data using latent variables. IEEE Transactions on Pattern Analysis and Machine Intelligence 36(1): Using generative latent variable model (GLVM) to model the interspeaker variations of visual appearances and those caused by uttering Speech emotion analysis Speech emotion analysis deals with methods to analyze vocal behavior as a marker of affect (e.g., emotions, moods, and stress) Traditionally only audio speech has been used, but according to psychological studies also lip-patterns carry information about emotions Our understanding of visual speech falls short of our understanding of the acoustic aspects of speech Integrating audio and visual speech would be highly desirable Our past research has focused on visual speech recognition, but could form a basis for emotion analysis 31

32 Visual speech animation Zhou Z, Zhao G & Pietikäinen M (2012) An image-based visual speech animation system. IEEE Trans. Circuits and Systems for Video Technology 22(10): Example application area: Interaction with a social robot 32

33 3D visual speech from video sequences Musti U, Ouni S, Zhou Z & Pietikäinen M (2014) 3D visual speech animation from image sequences. Proc. The Ninth Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP 2014), Bangalore, India. 3D facial shape animation 33

34 Teeth and tongue information Examples Emotions are missing! 34

35 Multimodal emotion analysis a central part of human communication. a multimodal procedure: speech, facial expressions, gestures, characteristics have a key role in human-computer interactions (HCI). Multi-modal emotion recognition Huang X, Kortelainen J, Zhao G, Li X, Moilanen A, Seppänen T & Pietikäinen M (2016) Multi-modal emotion analysis from facial expressions and electroencephalogram. Computer Vision and Image Understanding 147:

36 Multimodal emotion recognition by combining the complementary information from facial expressions and EEG. 1) emotion recognition from expressions in long evoked videos; 2) extraction and selection of spectral power and spectral power difference features for EEG analysis; 3) fusion of facial expressions and EEG for valence and arousal recognition on challenging MAHNOB-HCI database; 4) human test for perceiving the emotions and analysis for the effects of gender on expressing and perceiving emotions. While the subjects watched the videos, facial expressions and EEG were recorded. 36

37 Results Some future challenges 37

38 Facial behaviour analysis in-the-wild 75 Facial Action Unit (AU) coding in-the-wild 76 38

39 Continuous dimensional facial behavior analysis in-the-wild 77 Multi-modal recognition of emotions expressions speech physiological signals gestures Happy Happy Angry Most of the current methods work well for near-frontal faces in proper illumination conditions -> robust descriptors for changing conditions still needed Recognition of subtle spontaneous macro/micro-expressions from continuous video data is a great challenge Facial expressions are not culturally universal, also gender and facial age affect emotion decoding Joint use of different modalities (vision, speech, physiological signals (e.g., heart rate, temperature, respiration, galvanic skin response, blood pressure, EEG) Collaboration with experts from different disciplines, e.g., cognitive sciences, psychology, medicine 39

40 Role of emotions in computer-mediated interaction A new project (& Univ. of Helsinki and Aalto Univ.): Quantifying Human Experience for Increased Intelligence Within Work Teams and in the Customer Interface Research challenges Machine-centric -> Human-centric Perception -> Cognition Rational -> Emotional Dr. Harry Shum, Executive vice president of Microsoft's Artificial Intelligence (AI) and Research group (In interview by Business Management Review, China): Get 10 billion USD profit with his new team by bringing emotions to AI in next 3-5 years. Other companies: Apple bought Emotient (2016): Prof. Marian Bartlett (UC San Diego) Affectiva: Prof. Rosalind Picard, co-founder Facebook Facereader: Netherlands Google Realeyes: UK Affecto: Finland 40

41 Summary Machines with some emotional intelligence are emerging Various methods for emotion analysis were introduced Recognition of identity, emotions, and speech are needed for affective interaction Talking face avatars are used for computer s (robot s) response, but should include emotions Micro-expressions and heart rate provide useful invisible information There are still many challenges to make major breakthroughs for real-world applications in-the-wild Multimodal data is needed Thanks! 41