EMOTION RECOGNITION FROM USERS EEG SIGNALS WITH THE HELP OF STIMULUS VIDEOS

Transcription

1 EMOTION RECOGNITION FROM USERS EEG SIGNALS WITH THE HELP OF STIMULUS VIDEOS Yachen Zhu, Shangfei Wang School of Computer Science and Technology University of Science and Technology of China 2327, Hefei, Anhui Qiang Ji Department of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute Troy, NY ABSTRACT In this paper, we propose a novel approach to recognize users emotions from electroencephalogram (EEG) signals by using stimulus videos as privileged, which is only available during training. Firstly, five frequency features are extracted from each channel of the EEG signals, and several audio/visual features are extracted from video stimulus. Secondly, features are selected by statistical analyses. Then, a new EEG feature space is constructed using Canonical Correlation Analysis under the help of video content. Finally, a support vector machine is adopted as the classifier on the constructed EEG feature space. Experimental results on two benchmark databases demonstrate that video content, as the context, can improve the emotion recognition performance when employed as privileged. Index Terms emotion recognition, videos, EEG, privileged, CCA 1. INTRODUCTION The ability to understand human emotions is desirable for human-computer interaction. Users emotions can be detected from voice, visual behavior and physiological signals. To the best of our knowledge, one of the earliest study on emotional recognition is conducted by AJ Fridlund et. al. [1], who applied pattern recognition to classify emotions from physiological features. Since then, thousands of papers have been published. Early research mainly recognize deliberately displayed and exaggerated emotions from single modality. Recently, an increasing number of studies are reported to recognize natural and spontaneous emotions from multi- is the corresponding author. This work has been supported by National Program 863 (28AA1Z122), the National Science Foundation of China (Grant No , ), project from Anhui Science and Technology Agency (116c858) and the Fundamental Research Funds for the Central Universities. modalities. A detailed survey of emotion recognition can be find in [2][3][4]. Although present research recognizes users emotion from multi-modalities, including face, voice, and physiological signals, and achieves great progress, there still exist many challenges [3]. One of them is context, such as the environment, observed subject and current task. Apart from users spontaneous response, the context, in which the emotional behavior is displayed, is very important [3] for emotion recognition. Specifically, for analyzing the emotions from video-induced EEG signals, the stimulus video, could provide important assistance in building the video-based context modal for emotion recognition. Therefore, in this paper, we propose a new emotion recognition approach to classify emotions from EEG signals with the help of the stimulus videos. The users emotional response, such as EEG, is available during both training and testing, and it is called as available. The context, such as the stimulus, is often available during training, but is not available during testing. This kind of is called as privileged [5], which may be exploited to find better feature s- pace or construct better classifier for the available, and therefore improve the recognition performance using available. In this paper, we recognize emotion from electroencephalograph (EEG) signals, which are collected when subjects watch emotion-inducing videos. The stimuli videos, are regarded as privileged. First, five frequency features are extracted from each channel of the EEG signals, and several audio/visual features are extracted from video stimulus. Second, statistical analyses are constructed to explore the relations between emotional tags and EEG/video features. Third, a new EEG feature space is constructed with the help of video content using Canonical Correlation Analysis (CCA). Finally, the Support Vector Machine (SVM) is trained on the new EEG feature space. Experimental results on two benchmark databases demonstrate that our approach outperforms the ones of merely using EEG fea-

2 tures, even several fusion methods of previous work, in both valence and arousal spaces. This shows the model can benefit from the physical-physiological-related space constructed by CCA. The outline of this paper is organized as follows. In section 2, we propose a framework of emotion recognition and introduced the method in each step. Section 3 presents the experimental results on dataset from [6] and MAHNOB-HCI, as well the analyses and comparison with current work. Section 4 gives the conclusion. 2. METHODOLOGY Fig. 1 shows the framework of our emotion recognition approach. Firstly, features of EEG and video are extracted. Then, statistical analyses for hypothesis testing are conducted to check whether there exists significant difference in every feature between the two groups of emotions. After this step, relations between EEG signals and video content are exploited using CCA. Then, EEG features are transformed according to this relationship and used for training classifiers. The video features are privileged used to optimize the EEG features during training. Fig. 1. Framework of recognizing emotions with privileged by exploiting relations between physical and physiological spaces Visual-Audio features We extract both visual and audio features from videos. For visual features, lighting, color and motion are powerful tools to establish the mood of a scene and affect the emotions of the viewer according to cinematograph and psychology. Thus, three features, named lighting key, color energy and visual excitement are extracted from video clips. The details of features can be found in [1]. For audio features, thirty one commonly used audio features are extracted from video, including average energy, average energy intensity, spectrum flux, Zero Crossing Rate (ZCR), standard deviation of ZCR, 12 Mel-frequency Cepstral Coefficients (MFCCs), log energy of MFCC, and the standard deviations of the above 13 M- FCCs [11], are extracted by using PRAAT (V5.2.35) [12] Relations between EEG / video features and user s emotions We explore the relations between EEG / video features and user s emotions with the approach in [6]. After feature extraction, we conduct statistical analyses for hypothesis testing to analyze whether there exists significant difference in every feature between the two groups of emotional tags. The null hypothesis H means the median difference between positive and negative valence (or high and low arousal) for a feature is zero. The alternative hypothesis H1 is that the median d- ifference between positive and negative valence (or high and low arousal) for a feature is not zero. We may reject the null hypothesis when the P-value is less than the significant level. The procedures are described as follows: First, normality test is performed on each feature. If the feature is not normally distributed, Kolmogorov-Smirnov test is used. Otherwise, homogeneity of the features is tested. If the variance is homogeneous, T-test with homogeneity of variance is performed; otherwise, T-test with inhomogeneity of variance is performed. In our study, P-value threshold is set to.5 to capture more features that could represent the nature of EEG signals or video contents. This procedure can also be used as feature selection Feature extraction EEG features First, noise mitigation is carried out. HEGO and VEOG are removed and a bandpass filter with a lower cutoff frequency of.3 Hz and a higher cutoff frequency of 45 Hz is used to remove DC drifts and suppress the 5Hz power line interference [7, 8]. Then the power spectrum (PS) is calculated and divided into five segments [9], which are the delta (-4 Hz), theta (4-8 Hz), alpha (8-13 Hz), beta (13-3 Hz) and gamma (3-45 Hz) frequency bands. The ratio of the power in each frequency band to the overall power is extracted as the feature Constructing a new EEG feature space After feature selection, we construct a new EEG feature space with the help of video content using CCA. The main idea of CCA is to assess the relationship between metric independent variables and multiple dependent measures. In the our approach, denote the original training features of video and EEG by V and E. The goal of CCA is to find a and b, and use them to construct canonical components F and G F = Va, (1) G = Eb, (2)

3 where F and G are corresponding canonical components which have the highest Pearson Correlation Coefficients, which is actually obtained for training. Here we use the approach described in [13]. The relationship between V, E, a, b are the follows: (V T V) 1 V T E(E T E) 1 E T Va = β 2 a (3) (E T E) 1 E T V(V T V) 1 V T Eb = β 2 b (4) where β = a T V T Eb. Thus, β can be solved as the max eigenvalue of (V T V) 1 V T E(E T E) 1 E T V and a is the corresponding eigenvector. In the similar way, b is the eigenvector of (E T E) 1 E T V(V T V) 1 V T E. With this calculation, the relationship between E and V is specified to b, which could be used for training and testing. Moreover, the relationship between these two spaces could also be described as follows: F = 1 β V(VT V) 1 V T G, (5) G = 1 β E(ET E) 1 E T F, (6) After exploiting this relation, we adopt it in training and testing phases. In training phase, G is employed for training the model. And in testing phase, let E t be the testing EEG features since video features has been used as privileged, transfer E t into the video-related space E t by 2.4. Classifier and emotion recognition E t = E b. (7) A SVM classifier is employed recognize users emotions from EEG signals. During training, both EEG signals and video stimulus are available. Therefore, we obtain the EEG feature space using G of Eq. 2. During testing, only EEG signals are available, we obtain the EEG feature space using E t in Eq EXPERIMENTS 3.1. Experimental conditions To validate the performance of our approach, we conducted experiments on two benchmark databases, one is collected by Wang et. al. [6], the other is MAHNOB-HCI [14]. Wang et. al. collected users 197 EEG responses to 92 video stimulus. Users emotional self-assessment are fivescale evaluations (i.e. -2, -1,, 1, 2) for both valence and arousal. In our work, we divide them into two groups based on whether they are higher than zero or not. Finally, in arousal space, 149 EEG recordings are high, and 48 recordings are low; 7 videos are high and 22 are low. In valence space, 77 EEG recordings are positive, and 12 are negative; 3 video clips are positive and 62 are negative. MAHNOB-HCI is a multi-modal database for emotion recognition and implicit tagging. It includes the physiological signals from 27 participants correspond to 2 videos. Subjects emotional self-assessment are nine-scale evaluations, from 1 to 9, for both valence and arousal. In our work, we defined the ratings as positive or high if they are larger than 5, otherwise, we define them as negative or low. Thus, we get 533 EEG segments corresponding to 2 stimulus videos. For valence, there are 289 positive and 244 negative EEG segments, as well as, 7 positive and 13 negative videos. For arousal, there are 268 high and 265 low EEG segments, as well as 1 high and 1 low video stimulus. To validate the effectiveness of our proposed emotion recognition approach, we conduct three experiments: emotion recognition from EEG signals only, emotion recognition from EEG signals after Principle Component Analysis (PCA), and emotion recognition from EEG signals with the help of video content. The first experiment recognize emotions from the selected EEG features as described in Section 2.1.1; the second experiment recognize emotions from the EEG features which are transferred using PCA from the selected EEG features; The third one recognize emotions from the new EEG feature space, which is built with the help of video content using CCA. Furthermore, to test the generalized ability of our approach, leave-one-video out cross-validations are performed to simulate the EEG response to a unknown video. In addition, in the training procedure of each fold, a model selection procedure is carried out for tuning the best candidate hyper-parameters of the model Experimental Results and Analyses Experimental results for feature selections Fig. 2 shows the frequency distribution of selected EEG features in both datasets. Fig. 2(a) and Fig. 2(c) shows the selected features in valence space. We find that selected features main occurs in frontal area, which indicates that frontal area highly correlate with humans emotions. Fig. 2(b) and Fig. 2(d) illustrate the area that are active for arousal response. It could be discovered that features mainly gather in occipital area, which means this area is highly relevant to the excitement of humans emotions [15]. For video features, we also calculated the mean selected frequencies in all folds of cross-validation. Fig. 3 shows the frequencies of selected video features in both datasets. Fig. 3(a) and Fig. 3(c) shows the selected frequencies in valence space. For visual features, even though only three of them are extracted, two of them are selected, which means visual stimulus strongly influence emotions on valence [16]. For audio features, MFCCs are discriminating features [16], since a majority of them are with high selected frequencies. In arousal space, Fig. 3(b) and Fig. 3(d) illustrate the features that are highly related to the assessment. It could be discovered that the frequencies of video features are lower than most of au-

4 (a) (b) (c) (d) Fig. 2. (a) The distribution of EEG features with significant difference on valence in [6]. (b) The distribution of EEG features with significant difference on arousal in [6]. (c) The distribution of EEG features with significant difference on valence in MAHNOB-HCI. (d) The distribution of EEG features with significant difference on arousal in MAHNOB-HCI (a) (b) (c) (d) Fig. 3. (a) The selected frequencies of video features with significant difference on valence in [6]. (b) The selected frequencies of video features with significant difference on arousal in [6]. (c) The selected frequencies of video with significant difference on valence in MAHNOB-HCI. (d) The selected frequencies of video features with significant difference on arousal in MAHNOB- HCI. dio features. This further confirms MFCC features are important for emotion recognition. Moreover, none of the visual features are selected in 3(b). One possible reason is that the number of visual features is three, which is much smaller compared to 31 audio features Valence Recognition Results Table 2. Emotion recognition results on MAHNOB-HCI in valence space EEG EEG + PCA Video as privileged Positive Negative Positive Negative Positive Negative Positive Negative Accuracy 55.72% 54.3% 58.16% F1-score The emotional valence recognition results on the two databases are shown in Table 1 and Table 2. From these t- wo tables, we can find follows: First, the method of obtaining video as privileged outperforms the ones of merely using EEG features, i.e., with and without PCA; Second, the single-modality method using PCA shows a negative influence on emotion recognition. In the EEG only method, when PCA processed EEG features are used for training, all the three parameters, ie., accuracy, F1-score and, are slightly decreased by 2.53%,.488 and.31 respectively. However, when video features are employed as privileged through C- CA, these parameters are increased by 3.5%,.764 and.584, respectively. Table 2 shows the recognition results on MAHNOB-HCI. Compared with EEG only method, when video features are obtained as privileged, these parameters are increased by 2.46%,.492 and.268. Similar with the situation of the collected data, when PCA were employed before training SVM, these values are slightly reduced by 1.69%,.144 and.143. The improvement of using privileged indicates the calibration from merging video features by CCA, which means the video features is benefit in recognizing emotions by physiological response in valence space. Moreover, the reduction of performance of using PCA might be caused by the loss of since part of the unfaithful features have been removed in feature selection. These two phenomenon shows the validity of obtaining video features as privileged by CCA in valence space Arousal recognition results The emotion recognition results on arousal are shown in Table 3 and Table 4. From these two tables, similar phenomenon could be discovered: First, obtaining video as privileged in-

5 Table 1. Emotion recognition results on the data of [6] in valence space EEG EEG + PCA Video as privileged Independent featurelevel fusion in [6] fusion in [6] level fusion in [6] Decision-level Dependent feature- Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Positive Negative Accuracy % 76.65% 78.68% 77.66% 76.65% F1-score Table 3. Emotion recognition results on the data of [6] in arousal space EEG EEG + PCA Video as privileged Independent featurelevel fusion in [6] fusion in [6] level fusion in [6] Decision-level Dependent feature- High Low High Low High Low High Low High Low High Low High Low Accuracy % % F1-score Table 4. Emotion recognition results on MAHNOB-HCI in arousal space EEG EEG + PCA Video as privileged High Low High Low High Low High Low Accuracy 6.23% 55.53% 61.35% F1-score formation could improve the performance comparing with the ones of merely using single modality; Second, the singlemodal method using PCA also shows a negative influence on emotion recognition. The recognition results on the collected data in arousal s- pace are shown in Table 3. In the EEG only method, when PCA processed EEG features are used for training, the F1- score and are decreased by.563 and.282. When video features are employed as privileged through CCA, even though the accuracy is reduced by 5.7%, the F1-score and are increased by.339 and.159. This is due to the fact that in both of the EEGonly results, a majority of low samples are misclassified. Table 4 presents the recognition results on MAHNOB-HCI. Compared with EEG only method, when video features are obtained as privileged, the accuracy is slightly increased by.75%. The F1-score and average is increased by.531 and.119. Similar with the experimental results on Wang et al. s data, when PCA were employed before training SVM, these values are reduced by 4.,.397 and.467. The improvement of using privileged also indicates the video features is benefit for recognizing e- motions by physiological response in arousal space. From the recognition results on valence and arousal, it is clear that the approach of obtaining video features as privileged through CCA has a better performance than that of merely using EEG features, which is higher than that of using PCA after feature selection phase Comparison with related works There exist one related work using the database collected by Wang et al.[6], and another related work using MAHNOB- HCI database [17]. Since the feature extraction and selection methods used in this work are different from those used in [17], but are the same as those used in [6], we only compare our work with [6], as shown in Table 1 and Table Valence From Table 1, we can find that: compared with the results of three fusion methods proposed in [6], our method performs slightly better than the dependent feature-level fusion, which also builds the relationship between EEG and video features. We get the same in accuracy,.82 higher in F1-score and.47 higher in average. Even though the increase is not distinct, considering the features we used for recognition are only EEG features, the method we use utilize less knowledge than the fusion methods and still reaches similar performance Arousal Compared with the results of three fusion methods in Table 3, our method performs better than the decision-level fusion and dependent feature-level fusion with.14/.828 higher in F1-score and.51/.344 in average. The decision-level fusion and dependent feature-level fusion proposed in [6] use EEG signals and video content during both

6 training and testing. While, our method only use video content in training. It further demonstrate our proposed method can recognize emotion effectively by employing stimulus video during training. 4. CONCLUSION In this paper, we proposed a novel emotion recognition approach with privileged by exploiting relations between EEG signals and stimulus videos. Specifically, CCA is used to construct the relationship between EEG and video features, which could help to calibrate the feature by mapping them into a highly context-aware space. The experimental results on two benchmark databases verify the effectiveness of our approach. With the growing availability of built-in sensors, we believe emotional recognition with the help of users spontaneous responses will attract more and more attention. Thus, in the future, we will expand this study to other modalities such as user s expressions and gestures to broaden the way of applications. Besides, we only build relationships between EEG and video features implicitly through SVM in this work. We shall discover and employ such relations explicitly to improve EEG classification in the future. References [1] AJ Fridlund and Carroll E Izard, Electromyographic studies of facial expressions of emotions and patterns of emotions, Social psychophysiology: A sourcebook, pp , [2] Moataz El Ayadi, Mohamed S Kamel, and Fakhri Karray, Survey on speech emotion recognition: Features, classification schemes, and databases, Pattern Recognition, vol. 44, no. 3, pp , 211. [3] Zhihong Zeng, Maja Pantic, Glenn I Roisman, and Thomas S Huang, A survey of affect recognition methods: Audio, visual, and spontaneous expressions, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 31, no. 1, pp , 29. [4] Hatice Gunes, Björn Schuller, Maja Pantic, and Roddy Cowie, Emotion representation, analysis and synthesis in continuous space: A survey, in Automatic Face & Gesture Recognition and Workshops (FG 211), 211 IEEE International Conference on. IEEE, 211, p- p [5] Vladimir Vapnik and Akshay Vashist, A new learning paradigm: Learning using privileged, Neural Networks, vol. 22, no. 5, pp , 29. [6] Shangfei Wang, Yachen Zhu, Guobing Wu, and Qiang Ji, Hybrid video emotional tagging using users eeg and video content, Multimedia Tools and Applications, pp. 1 27, 213. [7] Sander Koelstra, Christian Muhl, and Ioannis Patras, Eeg analysis for implicit tagging of video data, in Affective Computing and Intelligent Interaction and Workshops, 29. ACII 29. 3rd International Conference on. IEEE, 29, pp [8] Sander Koelstra, Ashkan Yazdani, Mohammad Soleymani, Christian Mhl, Jong Seok Lee, Anton Nijholt, Thierry Pun, Touradj Ebrahimi, and Ioannis Patras, Single trial classification of eeg and peripheral physiological signals for recognition of emotions induced by music videos, Brain Informatics, pp. 89 1, 21. [9] Zhong Ji and Shuren Qin, Detection of eeg basic rhythm feature by using band relative intensity ratio (brir), in Acoustics, Speech, and Signal Processing, 23. Proceedings.(ICASSP 3). 23 IEEE International Conference on. IEEE, 23, vol. 6, pp. VI 429. [1] Hee Lin Wang and Loong Fah Cheong, Affective understanding in film, Circuits and Systems for Video Technology, IEEE Transactions on, vol. 16, no. 6, pp , 26. [11] Molau Sirko, Pitz Michael, Schlter Ralf, and Ney Hermann, Computing mel-frequency cepstral coefficients on the power spectrum, in Acoustics, Speech, and Signal Processing, 21. Proceedings. (ICASSP 1). 21 IEEE International Conference on, 21, vol. 1, pp vol.1. [12] Paul Boersma, Praat, a system for doing phonetics by computer, Glot international, vol. 5, no. 9/1, pp , 22. [13] David Weenink, Canonical correlation analysis, in IFA Proceedings, 23, vol. 25, pp [14] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, A multimodal database for affect recognition and implicit tagging, Affective Computing, IEEE Transactions on, vol. 3, no. 1, pp , 212. [15] Pierre Krolak-Salmon, Marie-Anne Hénaff, Alain Vighetto, Olivier Bertrand, and François Mauguière, Early amygdala reaction to fear spreading in occipital, temporal, and frontal cortex: a depth electrode erp study in human, Neuron, vol. 42, no. 4, pp , 24. [16] Min Xu, Jesse S Jin, Suhuai Luo, and Lingyu Duan, Hierarchical movie affective content analysis based on arousal and valence features, in Proceedings of the 16th ACM international conference on Multimedia. ACM, 28, pp [17] Sander Koelstra and Ioannis Patras, Fusion of facial expressions and eeg for implicit affective tagging, Image and Vision Computing, 212.