Distributed Multisensory Signals Acquisition and Analysis in Dyadic Interactions

Transcription

1 Distributed Multisensory Signals Acquisition and Analysis in Dyadic Interactions Ashish Tawari Cuong Tran Anup Doshi Thorsten Zander Max Planck Institute for Intelligent Systems, Department Empirical Inference, Tuebingen, Germany Mohan M. Trivedi Copyright is held by the author/owner(s). CHI 12, May 5 10, 2012, Austin, Texas, USA. ACM /12/05. Abstract Human-machine interaction could be enhanced by providing information about the user s state, allowing for automated adaption of the system. Such context-aware system, however, should be able to deal with spontaneous and subtle user behavior. The artificial intelligence behind such systems, hence, also needs to deal with spontaneous behavior data for training as well as evaluation. Although harder to collect and annotate, spontaneous behavior data are preferable to posed as they are representative of real world behavior. Towards this end, we have designed a distributed testbed for multisensory signals acquisition while facilitating spontaneous interactions. We recorded audio-visual as well as physiological signals from 6 pairs of subjects while they were playing a bluffing dice game against each other. In this paper, we introduce the collected database and provide our preliminary results of bluff detection based on spatio-temporal face image signal analysis. Author Keywords Multimodal database; emotion recognition; deception detection; facial expression analysis. ACM Classification Keywords I.5.m [PATTERN RECOGNITION]: Miscellaneous;

2 Introduction The human-computer paradigm suggests that user interfaces of the future need to perceive subtleties and changes in the user s behavior and to initiate interactions based on this information rather than simply responding to the user s commands. The future human-centered multimodal HCI will change the ways in which we interact with computer systems. The key component for the design of context-sensitive systems is the ability to recognize and generate social signals and social behaviors - in order to become more effective and more efficient. Among humans, social interactions involve explicit information (i.e. intentionally sending a message) as well as implicit information. Such information might also be relevant for a more intuitive HCI. For example, information into users affective state is critical for a good evaluation of their experience when interacting with the systems. An estimation of affect is of paramount importance for recreational systems such as games the whole point is to have fun and be engaged. Successful games carefully balance frustration, accomplishment, delight, pleasure, etc. to deliver fun. Direct measures of affect that do not interrupt the flow of games would be extraordinarily useful. Consequently, integrating information on aspects of user state into HCI could lead to a more natural way of interaction between human and machine. Such information can broadly be divided into three modalities: audio, visual and physiological. Among these modalities physiological signals are often neglected, since they can not be sensed all the times. Yet, the research in psychophysiology has produced strong evidence that a range of somatic and physiological measurements including pupillary diameter, heart rate, galvanic skin response, temperature, respiration rate, brain signals such as electroencephalogram (EEG) have shown in part or group high correlation with affective states like arousal [1] as well as cognitive states [11]. Active research in this field along with the recent advent of non-intrusive sensors and wearable computers, which promises less invasive physiological sensing [8], are pushing hard to bring technologies out of the lab and into society and onto the market. On the other hand, speech and vision being the primary senses for human expression and perception, significant research effort has been focused on developing intelligent systems with audio and video interfaces [6]. Moreover, the ease of availability of non-contact and non-intrusive sensors has encouraged researchers both in academia and industry to pursue design and development of the intelligent systems using visual and auditory channels. One of the aims of the intelligent systems is to perform automatic analysis of human behaviors. Tawari and Trivedi use audio modality for affect recognition in real world environment as well as in controlled studio settings [9]. Doshi and Trivedi [2] proposes attention estimation using a Bayesian framework which incorporates vision based gaze estimation (looking the driver) and visual saliency maps (looking the environment) as well as cognitive models that affects relationship between gaze and attention. Importance of such intelligent systems in HCI is undisputed. Hence to understand user state and behavior, HCI should incorporate one or more above mention modalities. The first step in automatic analysis of human behavior is the development of a database. Although harder to collect and annotate, spontaneous behavior data are preferred to posed/acted as they are representative of real world behavior. In more recent years, number of efforts have been put to develop such databases. However, databases

3 with all the three modalities are lacking. One such notable effort is of Soleymani et. al. [7]. Authors have captured synchronized recording of audio-visual and physiological signals in spontaneous behavior with the goal of affect recognition research. Emotions, however, are induced in the participants using affective stimuli. In this paper, we present a novel testbed for investigating temporal dynamics of multisensory signals in spontaneous HCI settings and the collected database along with the preliminary study using visual signal. To the best of our knowledge, our database is the first database, collected in spontaneous dyadic interactions between two human players, which has all the three modalities. Distributed Multi-modal Multi-sensory Testbed It is well recognized that interpreting the mix of audio-visual signals is essential in human-human communication. Hence, it seems natural to strive for a multimodal intelligent system with ability to perceive, analyze and respond to their surroundings in a way that is seamless to humans. There exists a vast literature on multimodal interfaces [3] also because of their many advantages: they prevent errors, bring robustness to the interface, help the user to correct errors or recover from them more easily, bring more bandwidth to the communication, and add alternative communication methods to different situations and environments. The artificial intelligence behind such interfaces, however, needs to deal with spontaneous behavior data for training as well as evaluation to understand their suitability in real world environment. Towards this end, we introduce a novel distributed testbed for investigating Temporal Dynamics of multi-sensory Signals (TDSS) in spontaneous dyadic interactions among humans. The testbed consists of two separate rooms with each room having similar equipments and capability to synchronously acquire multisensory signals. The players can interact with video-conferencing like setup where they can see each other s face and hear sound. Figure 1 shows the various components of the testbed. The testbed is capable of capturing multimodal data that includes audiovisual and physiological signals. Video feeds consist of player s face and upper body. We used three cameras as shown in Figure 1 along with eye tracker systems. Audio signal includes two audio channels per player in close (attached to the player s body) and far field settings. The physiological signal consists of high-density EEG (Brainproducts ActiCap) to capture brain s spontaneous electrical activity along with electrooculogram (EOG), electromyogram (EMG) to record physiologic properties of muscles, Electrocardiogram (ECG) to record physiologic properties of heart and galvanic skin response (GSR) signals. A summery of database characteristics is given in Table 1. Investigating Bluffing With the ambitious goal of mind reading, which plays key role in realistic interactions, we explore deception behavior in this first study utilizing our TDSS testbed. We adapted a German drinking game called Mäxchen, which included states dedicated to the decision of the player whether to bluff or to quit [4]. In the experiment, two players were situated in the two rooms of the distributed testbed. Starting with an account of 20 points, each player rolled two dices indicating a two-digit number in alternation. The player in turn needed a higher number than his predecessor, otherwise, he/she had to bluff or to quit. However, bluffing was more risky as its detection by the opponent

4 Modalities 2 channels per subjects (far and near Audio field) (sampling rate - 48kHz; resolution - 32 bits/sample) 3 video feeds (one for face and two for upper body) (frame rate Video fps; resolution ), Eye Gaze (30Hz) 64-channel high-density electroencephalogram (EEG), Electromyogram (EMG), Electrocardiogram (ECG), Physiological Electrooculogram (EOG) and Galvanic skin response (GSR) - 500Hz) Participants and Sessions No. of participants 12 No. of participants 2 per session Session 4 hours per session length Table 1: Multimodal Synchronized Database Content Summery Figure 1: Distributed testbed for multisensory signal acquisition in social interactive setting.

5 Bluffing game A typical trial had the following time line: player A pressed a button to roll his dice, the result appeared on screen 2 seconds later. He had to wait for an auditory go-signal to announce his true or alleged number or to quit. Responses were given verbally, synchronously with a button press to cue the time of response. After player A s response player B had to decide whether to accuse A of having bluffed or to accept the number and to roll the dice in his turn. Feature Type Geometric (per frame) Appearance (per frame) Statistical (per sequence) Extracted feature 2-D image coordinates of 49 facial landmark locations corresponding to eyebrows, eyes, nose and mouth regions and their first derivative over time Gabor filter (8 orientations and 5 spacial frequencies) magnitude response at 49 facial landmark locations and their first derivative over time Histograms (5-bin) of features extracted from each frame over the whole sequence Table 2: The list of feature extracted from video sequence for the bluffing classification experiment cost 2 points, while quitting only cost 1 point. If a bluff was wrongly accused, the accuser lose 2 points. A game was won when the opponent had lost all 20 points. Each pair of subjects played eight games. Twelve paid subjects were invited for the study. For motivation, subjects got extra monetary bonus for each won game. Facial dynamics in bluffing In this preliminary study, our goal is to classify trials into the categories Bluff, Truth or Quit based on visual signals. In a typical experiment, one player rolled the dice 250 times; of that 140 are truths (non-bluff s), 50 are quits and 60 are bluffs. Facial feature extraction and classification For Face analysis, we calculate spatio-temporal features. Figure 2 shows an overview of the system. From each video sequence, we extract three types of features: Geometric based (facial landmarks), appearance based (Gabor filter response) and statistical (histograms over sequence of frames). From each frame, we automatically extract 66 facial landmark using deformable model fitting [5] and align the face such that the centers of the eyes are roughly 50 pixels apart and are horizontally aligned. Table 2 provides the list of all the relevant features extracted from the aligned face. The calculated features are then utilized for classification tasks. We used the discriminative relevance vector machine (RVM) classifier, which is based on sparse Bayesian learning, developed by Michael Tipping [10]. The algorithm is a Bayesian counterpart to the popular support vector machines and is used to train a classifier that translates a given feature vector into a class membership probability which can then be thresholded to determine a true positive and false positive rate for classification tasks. Figure 2: Facial feature extraction and training Results In this study, we investigate two different classification tasks - Bluff Vs Truth and Bluff Vs Quit - using video data from one session. The video data corresponds to time where the user makes decision (truth, bluff or quit). The player presses a button before announcing his/her decision. We explored different durations around this button press and the best performance corresponding to 0.5 sec prior sec past the button press is reported in the paper. We also compare two configurations - full face and left/right half face. Using 5 fold cross validation approach, we generated average Receiver Operating Curve (ROC). Figure 3 shows ROC for the classification between Bluff Vs Truth. The area under curve (AUC) for full face condition is 69% while for half face condition is 58%. Figure 4 shows ROC for Bluff Vs Quit classification. In this case, we have AUC of 79% and 75% for full and left half face respectively. An AUC of 50% indicates chance level performace. Clearly, the performace using full face is quite above chance level for both the tasks. A higher AUC for Bluff Vs Quit classification can also be attributed to the fact that while quitting a player does not speak any

6 Figure 3: ROC curve of classification task - Truth Vs Bluff. Red curve shows performance using full face and blue shows performance using left half face. Figure 4: ROC curve of classification task - Bluff Vs Quit. Red curve shows performance using full face and blue shows performance using left half face. number. It s also important to note the drop in AUC using half face specially for Bluff Vs Truth classification. This can be attributed to the fact that mircoexpressions duing bluffing are subtle and detection of bluffing will be benefitted using full face which can capture any asymmetry in facial geometry as well. Concluding remarks In this paper, we presented a novel testbed capable of acquiring distributed multisensory data in time-synchronized fashion while facilitating communication in dyadic interactions. In the preliminary study of bluff detection, our analysis suggest that facial dynamics does provide useful information. While analysis is based one subject s data and more subjects must be included before making any conclusion, the findings reported is very encouraging. Our focus in this paper, however, is to introduce the testbed and dataset which we believe is the unique contribution. The Presence of a natural and spontaneous human behavior multimodal database will certainly benefit studies involving user understanding, which in turn is essential for effective HCI. Our continuing and longer term goal is to study multimodal system and towards this end, we will analyze other available modalities namely - physiological and audio. References [1] J. Cacioppo, G. G. Berntson, J. T. Larsen, K. M. Poehlmann, and T. A. Ito. The psychophysiology of emotion. In M. Lewis and J. Haviland-Jones, editors, Handbook of Emotions, pages [2] A. Doshi and M. Trivedi. Attention estimation by simultaneous observation of viewer and view. In Comp Vision and Pattern Recog, pages 21 27, [3] A. Jaimes and N. Sebe. Multimodal human-computer interaction: A survey. Computer Vision and Image Understanding, 108(1-2): , [4] J. Reissland and T. O. Zander. Automated detection of bluffing in a gamerevealing a complex covert user state with a passive bci. In Proc. of the Human Factors and Ergonomics Society Europe Chapter, [5] J. Saragih, S. Lucey, and J. Cohn. Face alignment through subspace constrained mean-shifts. In Int. Conf. on Computer Vision, pages , [6] S. Shivappa, M. M. Trivedi, and B. Rao. Audio-visual information fusion in human computer interfaces and intelligent environments: A survey. Proceedings of the IEEE, 98(10): , October [7] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic. A multi-modal affective database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing, 99, [8] T. Starner. The challenges of wearable computing: Part 1. Micro, IEEE, 21(4):44 52, jul/aug [9] A. Tawari and M. M. Trivedi. Speech emotion analysis in noisy real-world environment. Int. Conf. on Pattern Recognition, pages , [10] M. E. Tipping. Sparse bayesian learning and the relevance vector machine. J. Mach. Learn. Res., 1: , September [11] T. O. Zander and C. Kothe. Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general. Journal of Neural Engineering, 8(2), 2011.