A Pilot Study on Emotion Recognition System Using Electroencephalography (EEG) Signals

A Pilot Study on Emotion Recognition System Using Electroencephalography (EEG) Signals 1 B.K.N.Jyothirmai, 2 A.Narendra Babu & 3 B.Chaitanya Lakireddy Balireddy College of Engineering E-mail : 1 buragaddajyothi@gmail.com, 2 narendraalur@gmail.com, 3 chaitanya.bhavaraju@lbrce.ac.in Abstract - Establishing a new communication channel for physically immobilized people to interact with the outside world through their brain waves is one of the active research areas in Brain Computer Interface (BCI). The main objectives of this research holds with three important aspects, (i) To review the previous works on human emotion detection using Electroencephalogram (EEG) (ii) To design an audiovisual induction based data acquisition protocol for data collection. (iii) To propose the new Statistical analysis based features for emotion recognition. The commercial video clips were used for evoking the emotions namely Happy, Fear, Relax and Memory related. 64 channel EEG/ERP system was used to record/collect the data. A comparison of Power spectrum, P300 latency, Skewness was used to distinguish the onset of different emotions. Significant change in P300 latency and Skewness were observed with a statistical significance of 0.05. Keywords - BCI (Brain Computer Interface), Emotion Recognition, ERP (Event Related Potential). I. INTRODUCTION The main aim of this study is to evaluate different Human emotions through EEG signal and to receive the information about the internal changes of brain state based on some salient features of EEG signal. Nowadays, an increasing number of people spend more time interacting with a computer than with other humans. In addition, more and more people communicate with each other through computers. The problem when communicating through or with computers is that computers usually are not able to show or to submit any emotional reactions. Making this communication more human-like could improve collaboration when computers are involved. One of the most affect-limited communication forms is the communication via email. The only ways to express emotions in an email are emoticons which can easily be misinterpreted and lead to misunderstandings. Emotion is one of the most important features of humans. However different subjects have different emotional experiences that are influenced by their experiences. Identifying Human emotion is important in facilitating communication and interaction between individuals and emotion interaction between humans and machines is of the important issue in Human Machine Interaction today [1]. Without the ability of emotions processing, computers cannot communicate with humans. In recent years, Human computer interactions are focused on the means to empower computer to understand Human emotions. Researchers are trying to realize man-machine interfaces with an emotion understanding capability which requires to develop a reliable emotion recognition system [2]. Human emotion recognition has been one of the most interesting topics for researchers in the past few decades. There are a number of algorithms for recognizing emotions. The main problem of such algorithms is a lack of accuracy. Research is needed to be carried out to evaluate different algorithms. They can be divided into three main categories. The first method works on the analysis of facial expressions or change tone of speech [3-7].However, these techniques might be more prone to deception, and these signals are not continuously available. Recognizing the emotions from just the facial expressions is probably not accurate enough. For a computer to truly understand the emotional state of a human, other measurements have to be made. The second method focuses on periphery physiological signals. These signals changing in different emotional states with the changes of autonomic nervous system in the periphery, such as electro-cardiogram (ECG), skin conductance (SC), respiration, and pulse [8]-[9]. These signals occur continuously and are hard to conceal hence give complex information for estimating emotional states. The third method focuses on brain signals such as electroencephalography (EEG), electrocorticography (ECoG), and functional magnetic resonance imaging (fmri). Among these brain signals, Only the EEG signals are directly connected to the scalp and reading the onset changes in brain activity to give more reliable information about the emotional state changes [10].EEG is the measurement of the electrical activity of the brain by recording from electrodes on the scalp. Emotions are thought to be related with the activity in brain areas that direct our attention, motivate our behaviour and determine the significance of what is 30

going on around us. It is very difficult to keep a subject in one specific mental state due to environmental effects because emotion occurs very differently according to the situation, personality, growth and environment etc[11]. The EEG mainly detects the signal of task performed by the specific brain region. These signals vary from one state to another. A lot of works have been proposed to find the relation between the changes in signals and changes in emotions. [12], also used EEG signals to read a person s emotion. They extracted cross correlation coefficients between the EEG activities from different locations, to find information about the correlation between the EEG signals from different locations. The activity in specific EEG bands on specific locations on the scalp was used as features to describe the EEG data. [13], systematically compares three kinds of existing EEG features for emotion classification, and introduces an efficient feature smoothing method for removing the noise unrelated to emotion task. [14], proposed an emotion simulation experiment using statistical features for classifying five different emotions. The studies of associations between EEG activity and emotions have been received much attention [15]-[16]. Emotions can be induced in different ways: visual (images/pictures) [17], audio (songs/sounds) [18], audiovisual (video clips). The properties of dynamic audio visual stimulus makes videos seem one of the most effective ways to elicit emotions. In this paper, we analyze the EEG signals using different Signal Processing techniques and to extract different Statistical features to identify the changes and to detect the specific emotional state. Our pilot study is in line with all the previous studies to bring out better correlation in the extracted features. II METHODOLOGY/ DATA ACQUISITION AND METHODS This section describes the procedure for acquisition of EEG signals under emotion stimulation experiment. Subjects: A pilot panel study is conducted on 10 healthy female subjects (with mean age of 22.4 years, variance =0.518, Stddev=0.72).All subjects were informed about the aim and scope of the study and gave written informed consent according to the declaration of undergoing experiment. The participants have no history of physical or mental illness and they are not currently taking drugs or medication to affect their EEG. Experimental Setup: EEG signals were measured with 64 surface electrodes which were fixed at the standard positions on the scalp according to International 10-20 System. All the electrode impedances were kept below 5kΩ and made up of Ag/Ag-cl. The recording of EEG Signal has been done through 64 Channel EEG/ERP System, Compumedic Company (NeuroScan) and the impedance is kept below 5KΩ. EEG Data Acquisition: We have designed an audio-visual induction based protocol for eliciting emotions in this study because of its strong correlation between induced emotional states and physiological responses. All the video clips are collected from internet.[19] reported on a movie set, which included two clips for each of the target emotional state. In this project, we developed a set of four video clips that could be used to induce four emotional states such as Happy, Fear, Relax and Memory Related. All the signals are collected without much discomfort to the subjects. Fig. 1. Process of the Experiment First, before the experiment is started, the purpose of the study was clearly explained to the participant. The Participants were requested to minimize their body movement to reduce the appearance of relevant artifacts in the EEG recordings, and concentrate on the emotional stimuli. The subjects were informed that between each movie clips they would be prompted to answer the questions about the emotions they experienced under self assessment section. In this Section we posted some questions to subjects regarding corresponding Emotions: 1. Have you seen these video clips in an earlier period? 2. What emotion did you experience from this video clip? 3. Is this test making you to use your memory to answer these questions? Etc Procedure: The audio-visual stimulus protocol design of our experiment is shown in Figure1. The Participants were asked to relax (Eyes Closed) to record the Baseline signal which we take it as Reference Signal to recognize different emotional states. Then, four different video clips are presented one after to evoke different emotions. In between each emotional stimulus (video clips), a blank screen is displayed for 5sec to bring the subject to 31

their normal state and to experience a calm mind. The orders of the emotional video clips are changed in a random order for each of the subjects. Signal Analysis: The EEG signals are analyzed using several methods including signal pre processing, feature extraction etc. The recorded EEG signals are usually contaminated with noises (due to power line fluctuations and external interferences) and artifacts (Eye blinks (EOG), Muscular Activity (EMG), heart rate (ECG)). The EEG signals are pre processed to identify the artifacts that may be present in the signal. First the EEG Signals were down sampled to a sampling rate of 100Hz to reduce the burden of computation. In this work, Basic FIR filter is used with a cut off frequency of 0-30 Hz for removing the noises. This filter attenuates the EEG activity in order to improve the spatial resolution of the recorded signal. Artifacts are removed by using Independent Component Analysis (ICA).The overview of Emotion Recognition system using EEG Signals is shown in the Fig2. order to obtain further information from that signal that is not readily available in the raw signal. We have considered four emotions (Happy, Fear, Relax, and Memory Related) of ten persons and three kinds of features are extracted to assess the association between Skewness, Event Related Potential (ERP) P300 Latency, and Power Spectrum. A. Skewness: Skewness is the 3 rd moment of the data distribution. Skewness shows the degree of asymmetry in a distribution (away from normal Gaussian distribution) of EEG signal. Below Equation used to extract skewness from the EEG data. It has the purpose of verifying and calculating the data symmetry, indicating the probability of variable distribution The value of skewness is defined as: Skew = ne ins x 3 ieeg xeeg n e n s / n e ins x ieeg e x n n s 2 EEG 3 (1) Fig. 2. Emotion Recognition System Overview ICA algorithms have proven capable of isolating both artifactual and neurally generated EEG sources whose EEG contributions, across the training data, are maximally independent of one another. Independent Component Analysis (ICA) is now widely used in EEG research community to detect and remove eye, muscle and line noise artifacts. After the ICA analysis, total study has been created for whole ten subjects. Having created the study structure and designated the design parameters, the time frequency analysis can be implemented for every participant, condition and channel in just one step. Then we can extract the features such as Power spectrum which computes the power spectrum (FFT) for each channel, Event related potential (ERP) P300 Latency which computes the time domain ERP s for each channel, and Skewness. Feature Extraction: Features are characteristics of a signal that are able to distinguish between different activities. Extracting the more prominent statistical features from EEG Signal is highly inevitable for efficient detection of emotions. We have used FFT transform to extract the features. Mathematical transformations are applied to signals in Where a signal xieeg contains B. Event Related Potential (ERP) : i ( ne ns) points Event related potentials (ERP) are small changes in the electrical activity of the brain recorded by an EEG and triggered by some internal (cognitive tasks) or external (stimuli) event. Positive and Negative potential changes are commonly labeled with either P (for Positive) or N (for Negative) and a number corresponding to the time of their appearance relative to the event. P300: The Subject is instructed to respond to the infrequently or target stimulus and not to the frequently presented or standard stimulus.p300 wave only occurs if the subject is actively engaged in the task of detecting the targets C. Power spectrum feature: Fast Fourier Transform (FFT) is used in EEG Signal Processing soon after its introduction. Even today; this method remains the most widespread method in EEG Signal Processing. Power spectrum can be analyzed to characterize the perturbations in the oscillatory dynamics of ongoing EEG [20]-[21]. Power Spectrum across all Frequency Bands extracted from EEG signals Performs well on distinguishing emotions. Power spectra give which frequencies contain he signals power. The answer is in the form of distribution of power values as a function of frequency. 32

III. RESULTS AND DISCUSSION Skewness Plot: amplitude increases in response to stimuli considered relevant for the task by subjects, it could have occurred that amplitudes in response to stimuli were greater than the neutral ones. Power Spectrum Results: Fig.3. Bar plot of all signals showing Overall Skewness From Fig 3, it is observed that the deviations in different emotional states for 10 different subjects are specified in the above averaged plot. The bar Plot shows negative skewness in case of Baseline which means when the deviations from the mean are greater in one direction this statistic will deviate from zero in the direction of the larger deviations.hence the bar plot shows positive skewness in case of Happy, Relax and negative skewness in case of Fear and Memory state from which the shape of the EEG signal can be detected. P300 Latency: Table I: All subjects Averaged Mean latency with Standard Deviation for all Conditions (a) (b) Latency Values are observed from the Event Related Potentials (ERP) of the particular signal. ERP is the measured brain response that is directed result of a cognitive event.p300 is the positive ERP that occurs at 300ms when a subject has onset response of a particular event. In our study, from the above Table, it is observed that there is some significant shift to that 300ms latency for each of the condition with respect to the Baseline Signal. The Objective is to distract, hindering the subjects from easily making emotional categorizations of the stimuli. Common characteristic of these experiments was that the objective of studying emotion was explicit in stimuli themselves. Moreover, after each stimulus subjects were asked to categorize it as emotional or neutral. As P300 (c) (d) Fig. 4 (a): Averaged Channel Power Spectrum for 33

Baseline Vs Happy; (b): Averaged Channel Power Spectrum for Baseline Vs Fear ; (c): Averaged Channel Power Spectrum for Baseline Vs Relax ; (d): Averaged Channel Power Spectrum for Baseline Vs Memory Related From the above Power Spectrum Plots Fig.4, it is observed that we can get the ten subjects averaged Power spectrum signal for each of the emotional state. From this, it is known that the frequency components of particular mental states for different subjects are lie in the same range and vary with different mental states which is very effective to detect the different states for unknown signals. In Summary, it is observed that Power Spectrum across all Frequency Bands extracted from EEG signals performs well on distinguishing different emotional states and the emotion specific feature is mainly related to high frequency band rather than low frequency band. From Fig.4 (a), it is inferred that highest peak occurs between 13-15Hz as it indicates β-band and the signal shows more attentive towards the task. From Fig.4 (b), it is showed that it is in 13-15Hz range. It is in β-band these waves dominate our normal waking state of consciousness when attention is directed towards cognitive tasks and outside world. From Fig.4(c), it is similar to the baseline condition but not with eyes closed condition. Relaxation can be done by showing the nature clips to make the subject into normal condition. In this signal, Peak Occurs at 8Hz (αwave) to show the resting state of the brain. From Fig.4(d), peak fall under θ and α bands which is 7-9Hz Since θ-band acts as our gateway to learning and memory and α-waves also aid overall mental conditions, mind integration with learning memory. [22] Suggested that frontal brain electrical activity was associated with the experience of positive and Negative emotions, the studies of associations between EEG asymmetry and emotions have been received much attention. Two Way ANOVA: Analysis of variance (ANOVA) is a collection of statistical models used to analyze the differences between group means and their associated procedures. In this study, we conducted a two-way anova between conditions analysis of variance (ANOVA) with reference to Baseline Signal For all Happy, Fear and Memory states, we observe the significant value is p<0.05, F>Fc which means there is statistically significant difference between all the conditions with the reference signal. Hence concluded that the differences between condition Means are likely due to condition change and not likely due to the Independent variable (IV) manipulation. For Relax Condition, p=0.05and F<Fc, which means there is statistically significant difference with some change in error as relax state resembles the baseline signal. Hence concluded that the differences between condition Means are likely due to condition change. Human-computer interaction research often involves experiments with human participants to test one or more hypotheses. Hence ANOVA result is reported as an F- statistic and its associated degrees of freedom and p- value. Fig.5. ANOVA Results IV. CONCLUSION In this study, a group of 10 healthy female volunteers are participated to induce the emotions. Subjects were all right handed females between the age of 21-24 years. All subjects had normal or corrected vision and none of them had neurological disorders. Statistical features such as Spectral power, P300 latency and skewness were extracted from the EEG signal. We observe some sort of correlation in these features to detect different emotional states and the preliminary results presented in this study address the human emotions using reduced set of EEG channels(cz,fz,pz).from these features, the significant deviations among the pre specified states for different subjects are observed which help to detect mental states. There is a need to reduce the study time to provide comfort to the subjects. We perform this study with limited number (10) of subjects and with a mean age group of 22.4 years. Future Scope: This study can be extended with the mixed population of subjects, varied age group, gender discrimination and event related test conditions. Other than the three statistical features extracted in this study we can increase the number of features based on different techniques of signal processing to get the better results with a more accurate correlation in the subjects. V. REFERENCES [1] R. Picard, Affective Computing, MIT press, 2000. [2] M.Murugappan, Human Emotion Classification using Wavelet Transform and KNN, International Conference on Pattern Analysis and Intelligent Robotics, 2011. 34

[3] V. Petrushin, Emotion in speech: recognition and application to call centers, in: Proceedings of the Artificial Networks in Engineering Conference, pp.7-10, 1999. [4] M. Black, Y. Yacoob, Recognizing facial expressions in image sequences using local parameterized models of image motion, Int. J. Comput. Vis. 23 48, 25 (1997). [5] K. Anderson, P. McOwan, A real-time automated system for the recognition of human facial expressions, IEEE Trans. Syst. Man Cybern. B Cybern. 96 105, 36 (2006). [6] Ira Cohen et.al, Emotion Recognition from Facial Expressions using Multilevel HMM. [7] EIAyadi M, Kamel MS, KarrayF, Survey on speech Emotion Recognition: Features, Classification Schemes and Databases, Pattern Recognition; 44: 572-587, 2011. [8] J. Wagner, J. Kim, From physiological signals to emotions: Implementing and comparing selected methods for feature extraction and classification, in: Proceedings of IEEE International Conference on Multimedia and Expo, pp.940-943, 2005. [9] K. Kim, S. Bang, S. Kim, Emotion recognition system using short-term monitoring of physiological signals, Med. Biol. Eng. Comput.,419 427, 42 (2004). [10] P. Petrantonakis, L. Hadjileontiadis, A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-based emotion recognition, IEEE Trans. Inf. Technol. Biomed., 737 746, 15 (2011). [11] Monira Islam, Tazrin Ahmed, Sheikh Shanaz Mostafa, Human Emotion Recognition Using Frequency and Statistical Measures of EEG Signal, IEEE,2013. [12] T.Musha, Y.Terasaki, H.A.Haque, and G.A. Ivanitsky, Feature Extraction from EEGs associated with emotions, Springer, Research and Development in Intelligent Systems, Proceedings of AI-2007 The 27 th SGAI International Conference on Innovative Techniques and Application of Artificial Intelligent, 2007. [13] Xiao-Wei Wang, Dan-Nie, Bao-Liang Lu, Emotional State Classification using Machine Learning Approach, NeuroComputing, 94-106,129(2014). [14] Chai Tong Yuen,Woo San San, Mohamed Rizon, Classification of Human emotions from EEG Signals using Statistical features and Neural Network,International Journal of Integrated Engineering. [15] D.Nie, Emotion Recognition based on EEG(Master s Thesis),Shanghai Jiao Tong University,Shanghai,2012. [16] D.Nie, X.Wand,L.shi,B.Lu, EEG based Emotion recognition during watching movies : Proceedings of International IEEE Conference on Neural Engineering,pp.667-670,2011. [17] Danny,O.B.(2008) Automated Artifact detection in brain stream,technical Report,1-8. [18] Jonghwa.K,Elisabeth.A,(2008) Emotion Recognition based on Physiological changes in musical listening, IEEE Transactions on Pattern Analysis and Machine Intelligence,pp.1-17, 30(2012). [19] P.Phillipot,Inducing and Assessing differentiated emotion feeling states in the laboratory,cogn.emot,171-193, 7(1993). [20] D. Sammler, M. Griguttsch, T.Fritz, S.Koelsch, Music and Emotion: electro physiological correlates of the processing of pleasant and unpleasant music, Psychophysiology,293-304, 44(2007) [21] Y.P. Lin, C.H. Wang, T.P. Jung, T.L. Wu, S.K.Jeng, J.R.Duann, J.H.Chen, EEG based Emotion recognition in music listening, IEEE Transactions on Biomedical Engineering,1798-1806, 57(2010). [22] R. Davidson, N. Fox, Asymmetrical brain activity discriminates between positive and negative affective stimuli in human infants, Science,1235,218 (1982). 35