Construction of the EEG Emotion Judgment System Using Concept Base of EEG Features

Transcription

1 Int'l Conf. Artificial Intelligence ICAI' Construction of the EEG Emotion Judgment System Using Concept Base of EEG Features Mayo Morimoto 1, Misako Imono 2, Seiji Tsuchiya 2, and Hirokazu Watabe 2 1 Department of Information and Computer Science, Graduate School of Science and Engineering, Doshisha University, Kyo-Tanabe City, Kyoto, Japan 2 Department of Intelligent Information Engineering and Science, Faculty of Science and Engineering, Doshisha University, Kyo-Tanabe City, Kyoto, Japan Abstract - For a robot converse naturally with a human, it must be able to accurately gauge the emotional state of the person. Techniques for estimating emotions of a person from facial expressions, intonation and speech content have been proposed. This paper presents a technique for judging the emotion of a person using EEGs. The system of judging the emotion from EEGs is called EEG Emotion Judgment System, and constructed Concept Base for reducing noise in this paper. Accuracy of emotion judgment using EEG features of all subjects was 57.6% and using leave-one-out cross validation was 30.8%. Although performance accuracy remains low, continued development is required through further development of methods for both reducing different variety of noise mixed in with EEGs. Keywords: EEG, judging emotion, concept base Input (EEG) Output (Emotion) EEG Emotion Judgment System Spectrum Analysis Normalization (Linear and Non-Linear) Judging emotion from EEGs EEG features KB 1 Introduction For a robot to converse naturally with a human, it must be able to accurately gauge the emotional state of the person. Techniques for estimating emotions of a person from facial expressions intonation and speech content have been proposed. Currently, EEGs are attracted attention for the tool of estimating emotion. EEGs are the electrical signals from brain, and control people s expression. It has advantage for possibility of reading without direct processing. Calculation of Degree of Association Association Mechanism EEG features Cocept Base This paper presents a technique for judging the emotion of a person by differing between EEGs. The system of judging the emotion from EEGs is called EEG Emotion Judgment System. The noises are easily included in EEGs when judging emotions and are difficult to remove out. This system considered for noise by constructed Concept Base to reduce the influence of noises. 2 Overview of Proposed Technique The objective of this technique was to read the emotions of a conversation partner from EEGs. Figure 1 shows outline of the proposed technique of emotion judgment from EEGs which is called EEG Emotion Judgment System. Fig.1 EEG Emotion Judgment System EEGs acquired from the subject, and are used as source EEGs. Emotions of the subject are acquired simultaneously as source EEGs. Emotions have been assigned to spectrum analysis of the source EEGs and are performed every 1.28 second. The EEG features are determined to θ waves (4.0 Hz to 8.0 Hz), α waves (8.0 Hz to 13.0 Hz), and β waves (13.0 Hz to 30.0 Hz) which showed at Figure 2.

2 486 Int'l Conf. Artificial Intelligence ICAI'15 θ α β Japanese film for approximately two hours. During watching film, subjects were asked to gauge the emotions each felt by speech in the film, and source EEGs were acquired simultaneously. Scene in the film were frozen for each of speech in the film, and the subject were asked what emotion they felt at that time watching the scene. Fig.2 Spectrum analysis of the source EEGs Emotion is judged from EEG features by an Association Mechanism. The EEG features association was realized by using the huge Concept Base [1, 2] which was automatically built from EEG features. A method to calculate the Degree of Association [3, 4, 5] evaluate the relationship between EEG features. Concept Base and Degree of Association are used at natural language processing, however, the structure of EEG features and the word was similar so applied the technique at EEGs. Hereafter, this Concept Base and the calculation method are called the Association Mechanism. Emotions judged in thus study pleasure, anger, sadness, and no emotion. 3 Acquisition of Source EEGs and Emotions EEGs were measured at 14 locations that their positions conforming to the International system showed at Figure 3 [6]. Left Right Fig.4 Image of Electroencephalography Eighteen subjects (nine males and nine females) were used, and viewing was divided into four sessions to reduce the physical burden to subjects. Before and after the film, EEGs of open-eye and closed-eye states were measured for approximately one minute each for use in normalization of EEG features. 4 Normalization of EEG Features EEGs show changes in voltage intensity over time within an individual, and base voltage intensity differs among individuals. For this reason, the possibility of misjudgment exists because those values differ greatly even among EEGs with similar waveforms. To solve this problem, linear normalization and non-linear normalization were performed. 4.1 Linear Normalization This was performed to take into account for voltage intensity of EEGs varies over time depending on the subject. Since the eyes were open while viewing the film, linear normalization was performed to acquire EEGs by both before and after experiment based on EEG features from the eyeopen state. EEG features Linear_al ij, obtained by linear normalization of first EEG feature alij at a certain point in time during the experiment, and is expressed by Formula 1: q1 q2 q2 q1 Linear _ alij = alij + l + q2 l + q2 p2 p1 p2 p1 2 (1) Fig.3 14 locations to measure EEG conforming to the International system Subjects fitted with an electroencephalography [7] caps, showed at Figure 4. The subjects were asked to watch 4.2 Non-linear Normalization This was performed to take into account for the difference among individuals in base voltage intensity.

3 Int'l Conf. Artificial Intelligence ICAI' Non-linear normalized values were obtained by using Formula 4.2. f(x) is the EEG features after non-linear normalization has been applied, x is the EEG features applied in non-linear normalization, x min is the minimum EEG features of individual, and x max is the maximum EEG features of individual. As a result, EEG features with large values are compressed and EEG features with small value are expanded by non-liner normalization. Thus, the degree of voltage intensity of an individual s EEGs is solved. ( x xmin ) ( x x ) log f ( x) = (2) log max min 5 EEG Features Knowledge Base EEG features Knowledge Base is data base constructed by EEGs and emotions. EEG features of 42 represented by three bandwidths which obtain from 14 locations are assumed to one EEG data, and each are matched with emotions, either anger, sadness, no-emotion, nor pleasure. EEG features Knowledge Base is consisted with each knowledge base of each emotion. EEG features Knowledge Base containing 2887 EEGs obtained by excluding outliers and noise from the total of 5670 EEGs. The emotions of the 2887 EEGs are comprised 541 anger features, 726 sadness features, 1226 noemotion features, and 394 pleasure features. The image of EEG features Knowledge Base is showed in Figure 5. EEG Features Knowledge Base Anger 541 data No-emotions 1226 data 42 EEG features Sadness 726 data Pleasure 394 data pleasure of Association is used for the semantics expansion, and it expresses the relationship between one EEG feature and another by a numeric value. The methods of a Concept Base and a Degree of Association were proposed in the field of the natural language processing, and the research results are apply to EEGs written in this paper. 6.1 Concept Base of EEG Features First of all, a Concept Base of words is explained that is in the field of the natural language processing. The research results are applied to EEG in this paper. A Concept Base is a large-scale database that is constructed both manually and automatically using words from multiple electronic dictionaries. The entry word in dictionary used as concepts and independent words in the explanation under the concept are used as an attributes. In the current research, a Concept Base containing approximately 90,000 concepts, in which auto-refining processing has been done after the base had been manually constructed. In this processing, attributes considered by the standpoint of human sensibility, that the inappropriate attributes were deleted and necessary attributes were added. In the Concept Base, Concept A is expressed by Attribute a i indicating the features and the meaning of the concept in relation to a Weight w i, denoting how important an Attribute a i is in expressing the meaning of Concept A. Assuming that the number of attributes of Concept A is N, Concept A is expressed by Formula 3 at below. Here, the Attribute a i are called Primary Attribute of Concept A. {( a w ), ( a, w ),,( )} A = 1, L a N, w N (3) By the reason of Primary Attribute a i of Concept A is defined as the concepts in the Concept Base, attributes can be similarly elucidated from a i. The Attributes a ij of a i are called the Second Attributes of Concept A. Attribute a i is defined by a ij, and also defined as the concepts, so a ij is Primary Attribute of a i. Thus, Concept Base can be connecting to N- dimension. Figure 6 shows the elements of the Concept train expanded as far as the Secondary Attributes. Fig.5 Image of EEG Features Knowledge Base 6 Association Mechanism The Association Mechanism consists of the Concept Base and the Degree of Association. The Concept Base generates semantics from a certain EEG features. The Degree

4 488 Int'l Conf. Artificial Intelligence ICAI'15 train Fig.6 Example of demonstrating the Concept train expanding as far as Secondary Attributes In this study, a Concept Base was made by using source EEGs instead of electronic dictionaries. In fact, EEG features are used instead of words. θ α β θ α β Secondary Attributes concept (locomotive, 0.21) (train, 0.36) 14 locations Fp1 Fp2 Pz μV 2000 kinds (a 1j, w 1j) Max 1000 μv α, θ wave Fp1 Fp2 Pz (locomotive, 0.21) (train, 0.36) (railroad, 0.21) β wave (a 2j, w 2j) (streetcar, 0.23) (a i, w i) Primary Attributes Max 500 μv 0.025μV 2000 kinds (a i1, w i1) (a i2, w i2) Fig.7 Process for conceptualization of EEG features (a ij, w ij) The voltage value in each location and each bandwidth is considered to be an each part of the word respectively. However, the granularity of the voltage value is more detailed than that of the word. Therefore, the voltage value of the certain scope is treated as the same as showing in Figure 7. As a result, the number of EEG features are controlled, similar to words are controlled by synonym. The concrete method is to delimit the θ waves and α wave by 0.05μV, and delimit β waves by 0.025μV. Furthermore, different numerical values in each divided group are allocated same part and bandwidth. This numerical value is treated as a word, and part and bandwidth are new information than in natural language processing. As a result, EEG features can be conceptualized similar to a Concept Base of words. In the present research of natural language processing, 20,252 concepts were used in a Concept Base. 6.2 Weight Weight is performed by the method of TF IDF. TF IDF is popularly used in the field of natural language processing for searching information. Weight W (A, B) of Concept A for attribute B, is calculated as follows: D W ( A, B) = tf ( B) log2 (4) df ( B) Concept A is one EEG feature of 42 EEG features, and remained 41 EEG features are considered as an attribute. EEG features in the same group, considered in figure 7 are also added to an attributes. As the premises, tf(b) express the frequency that come out within all attribute of Concept A. D is a number of concepts stored in EEG features Concept Base, and df(f) is a number of concepts to have Concept A included in each attribute. idf is calculated to divide D by df(b), having logarithm as 2 for bottom. Thus, tf idf is calculated as multiple tf by idf. The image of concept and attribute are showed at Figure 8. EEG EEG Concept Attribute Fig. 8 Image of concept and attribute 6.3 Calculation of Degree of Association for EEG Features First of all, a Calculation of Degree of Association is explained that is in the field if the natural language processing, calculating by words Degree of Match by Weight Ratio Degree of Match by Weight Ratio is calculated by total value of each 42 EEG feature s degree of match. Regard input EEG as Concept A, and EEG in EEG features Knowledge Base as Concept B. Each EEG features of

5 Int'l Conf. Artificial Intelligence ICAI' Concepts are regarded as A and B, and attribute are defined as a i and b i. Weight of A and B are defined as u i and v i. If the numbers of attributes are L and M respectively to the concepts (L M), they can be expressed as follows: {( a, u ), ( a, u ),, ( )} A ' = L a L, u L (5) {( b, v ), ( b, v ),, ( )} B ' = L b M, v M (6) Electrodes part p, frequency f, and EEG features e are defined by a L and b M, showed by follows: ( p, f e ) a L = L L, L (7) ( p, f e ) b M = M M, M (8) Due to this, Degree Match by Weight Ratio DoM(A, B) of Concept A and B is calculated as follows: DoM ( A', B' ) ( S A' na' + SB' nb' ) min( ui, vi ) 2 max( u, v ) = (9) i i S A ' = u i S B ' = v i (10) a i = b j a i = b j L M n A ' = u i= 1 1 n B ' = v j (11) j= 1 ( u,v ) ( ui v j ) ( u > v ) ui min i j = (12) v j i j ( u,v ) ( ui > v j ) ( u v ) ui max i j = (13) v j i j a i =b j is expressed when attribute matches. S A is the total weight of a i when a i = b j matched, and S B is the total of b j when a i =b j matched. n A and n B is the total weight of Concept A and B. Thus, S A /n A is a ratio of weight that matched to attribute from look from Concept A, and S B /n B is a ratio of weight that matched to attribute from look from Concept B. Therefore, (S A /n A + S B /n B )/2 express average of S A /n A and S B /n B. Degree of Match by Weight Ratio is calculated by considering ratio of coincidence by attribute and weight Degree of Association by Weight Ratio This paper applied Degree of Association by Weight Ratio, considers the coincidence of attribute and weight. For calculation, input EEG data and EEG features Knowledge Base data, such as para-concept A and B, use para-concept A for standard to fix the row. Para-concept is a one block of concept, so EEG features in this block becomes first attribute. Then, sort the attribute in para-concept B for making the total of the degree of match largest between para-concept A. The attribute and weight of para-concept B is defined as (b xi, v xi ). The electrode part p xi, frequency band f xi, and EEG features e xi are defined by b xi. {( b, v ), ( b, v ), L, ( b v )} B' = X X1 X 2 X 2 X 42, X 42 (14) ( p, f e ) b Xi = Xi Xi, Xi (15) In this study, EEG features of 42 represented by three bandwidths which obtain from 14 locations are assumed to the attributes, and are used by the calculation of the Degree of Association. 7 Calculation Experiment 7.1 Experiment Method The method of evaluation was a leave-one-out crossvalidation, a technique which involves using one data extracted as the validation set and the remaining observations are as the training set for comparison. In this study, EEG features Knowledge Base containing 2887 EEGs obtained by excluding outliers and noise from the total of 5670 EEGs. The emotions of the 2887 EEGs are comprised 541 anger features, 726 sadness features, 1226 no-emotion features, and 394 pleasure features in this study. The evaluation was performed for two proposed technique. The difference of two techniques is using different EEG features Knowledge Base. First proposed technique uses EEG features Knowledge Base include all EEG features of 18 subjects. Comparison to first technique, second proposed technique uses EEG features Knowledge Base, excluding one subject that used for evaluation. That is to say, EEG features Knowledge Base is constructed as the number of the subjects. Thus, second proposed technique used 18 EEG features Knowledge Base for the evaluation. Image of EEG features Knowledge Base of proposed technique showed in Figure 9. Proposed Technique1 Input EEG of subject A Proposed Technique2 Input EEG of subject A EEG features Knowledge Base of all subjects KB Without Subject A EEG features Knowledge Base For each subjects KB Without Subject B Fig.9 Image of EEG features Knowledge Base...

6 490 Int'l Conf. Artificial Intelligence ICAI' Evaluation of Accuracy The result of the emotion judgment form EEGs are showed in Figure 10. another as calculating a numeric value. The methods of a Concept Base and a Degree of Association were proposed in the field of the natural language processing, and their research result were applied to EEGs. As a result, accuracy of EEG Emotion Judgment System used EEG features Knowledge Base of all subjects was 57.6%, and accuracy of using EEG features Knowledge Base of each subject is 30.8%. As comparison, accuracy of emotion judgment at random was 25.0%. By the result, we believe that the proposed method was effective. 10 Ackowledgements This research has been partially supported by the Ministry of Education, Science, Sports, and Culture, Grantin-Aid for Scientific Research (Young Scientists (B), ). Fig.10 Result of the emotion judgment from EEGs Accuracy of emotion judgment from EEGs using the Association Mechanism was 30.8%. As a comparison, accuracy used EEG features Knowledge Base that includes EEGs of all the subjects was 57.6%, accuracy of using EEG features Knowledgebase of each subject is 30.8%, and accuracy of emotion judgment at random was 25.0%.. 8 Discussion Accuracy that includes EEGs of all the subjects was a highest accuracy in the evaluation. The reason is EEGs of same subject as validation sets are in EEG features Knowledge Base so it influenced to the judgment of emotion. Then, the accuracy using EEG features Knowledge Base of each subject was 30.8%, and was higher than 25% at the random. From these result, we believe that the proposed method is effective 9 Conclusion Authors are conducting research for new emotion judgment system by using EEG features. Especially, authors focus on constructing EEG features Concept Base and using the Degree of Association, to reduce noise in the EEGs. EEGs acquired from the subject are used as source EEGs. Spectrum analysis of the source EEGs have assigned emotion flags, and performed every 1.28 second. This determined as the EEG features. Emotion is judged from EEG features by an Association Mechanism. The Association Mechanism consist the Concept Base and the Degree of Association. The Concept Base generates semantics from a certain EEG features, and the Degree of Association uses the result of the semantic expansion. The Degree of Association expresses the relationship between one EEG features and 11 References [1] K. Kojima, H. Watabe, T. Kawaoka, A Method of a Concept-base Construction for an Association System: Deciding Attribute Weights Based on the Degree of Attribute Reliability, Journal of Natural Language Processing, Vol.9, No.5, pp , [2] N. Okumura, E. Yoshimura, H. Watabe, and T. Kawaoka, An Association Method Using Concept-Base, KES2007/WIRN2007, Part I, LNAI4692, pp , [3] H. Watabe and T. Kawaoka, The Degree of Association between Concepts using the Chain of Concepts, Proc. Of SMC2001, pp , [4] T. Hirose, H. Watabe, and T. Kawaoka, Automatic Refinement Method of Concept-base Considering the Rule between Concepts and Frequency of Appearance as an Attribute, Technical Report of the Institue of Electronics, Information and Communication Engineers, NLC , pp , [5] H. Watabe, E. Yoshimura, and S. Tsuchiya, Degree of Association between Documents using Association Mechanism, KES2009, LNAI5711, pp [6] J.Clin, Guideline thirteen: guidelines for standard electrode position nomenclature, American Electroencephalographic Society, Neurophysiol 11, pp , [7] T. Musha, Y. Terasaki, H. A. Haque, G. A. Iranitsky Feature extraction from EEG associated with emotions, Art Life Robotics, pp