ADVANCES in NATURAL and APPLIED SCIENCES

Transcription

1 ADVANCES in NATURAL and APPLIED SCIENCES ISSN: Published BYAENSI Publication EISSN: Special 11(5): pages Open Access Journal Emotion Recognition based on Eye Movement 1 R. S. Soundariya and 2 R. Renuga 1 Department of Computer Science and Engineering Coimbatore Institute of Technology Coimbatore, Tamilnadu, India. 2 Department of Computer Science and Engineering Coimbatore Institute of Technology Coimbatore, Tamilnadu, India. Received 28 January 2017; Accepted 22 April 2017; Available online 1 May 2017 Address For Correspondence: R. S. Soundariya, Department of Computer Science and Engineering Coimbatore Institute of Technology Coimbatore, Tamilnadu, India. rssoundariya@gmail. com Copyright 2017 by authors and American-Eurasian Network for ScientificInformation (AENSI Publication). This work is licensed under the Creative Commons Attribution International License (CC BY). ABSTRACT Emotions play a vibrant role in life as the human reaction or response to a system is purely based on the nature of his emotion, may it be the response to a computer system or to fellow mates. The need and significance of automatic emotion recognition have grown with the emergent role of human computer interface applications and the development of AI based companions or selfassistance system. The development of AI and machine learning systems has paved a brighter way for the optimistic yet accurate emotion recognizing systems. Emotion recognition can be done from any form of response from a person such as text, speech, facial expression or gesture. The proposed system introduces an emotion recognition system, based on human eye movement using electrooculography (EOG) signals. Based on EOG signals emotions are classified as happy, sad, angry, fear and pleasant. Multi-class Support Vector Machines is used for classifying the processed Electrooculography signals and for feature extraction ICA (Independent Component Analysis) is used. Using these techniques, human emotions are recognized and inputted in an augmented reality (AR) system where the humans can interact or respond to the system. KEYWORDS: Emotion recognition, Eye movement analysis, Electrooculography INTRODUCTION There are several traditional approaches contemplate on recognizing emotions from text, facial expressions, speech and gesture [1]. Psychologically, it is substantiated that emotions can be recognized from eye movements. Eye movements can be captured with the help of eye trackers and cameras. Various signals such as EEG, EOG can also be used to measure eye movements. Researches concentrate on EEG signals to find out human emotions, as EEG can be used to track and record brain wave patterns [2]. In the same way, EOG signals can be used to measure the corneo-retinal standing potential which subsists among the forward-facing and the backwardfacing position of the human eye. The Electrooculography system reflects the human eyes as electric dipoles, in view of positive pole at the cornea and its negative pole at the retina, assuming a stable Corneo-Retinal potential change, the eye is the origin of a steady electric potential field. The resulting electrical signal that can be obtained from this field is called Electrooculogram (EOG). If the eye passages from the central position towards the periphery, the retina approaches one side while the cornea approaches the opposite side. This transformation in dipole direction causes a transformation in electric potential field and results in Electrooculogram (EOG) signal. By analysing these deviations, the movements of eyes can be monitored. Two sets of skin electrodes positioned at the opposed sides of the eye and an additional reference electrode is located anywhere on the fore head area, which produces two signal components - a horizontal movement and a vertical movement which corresponds to two directional movements of the eyes. The signal amplitude shown by EOG systems varies from 5-20 micro-volts per degree ToCite ThisArticle: R. S. Soundariya and R. Renuga., Emotion Recognition based on Eye Movement. Advances in Natural and Applied Sciences. 11(5); Pages: 38-43

2 39 R. S. Soundariya and R. Renuga., 2017/Advances in Natural and Applied Sciences. 11(5) Special 2017, Pages: and necessary frequency content lies between 0 and 30 Hz. The proposed work concentrates on identifying human emotions from eye movement that are captured using EOG signals. Related Work: A. Sentiment Analysis: Sentiment classification has been a long standing Natural Language Processing problem with both supervised and unsupervised machine learning based methodologies prevailing for the task. Sentiment Analysis is ordinarily performed for finding the sentiment polarity as positive versus negative of user generated short texts and sentences [3]. The most famous usage of sentiment analysis is sarcasm detection on twitter [4]. Apart from texts it is also shown that sentiment analysis has been performed on certain cognitive features [5]. All these works have been focused on determining the polarity of user generated text but did not identify emotion recognition using sentiment analysis. B. Eye Movement Analysis: Eye movement patterns can be captured in many ways. In order to sense visual fixations of how an observer recognizes an entity, prototypes like Markov process can be used [6]. An emergent number of research works were carried on eye movement analysis using video-based eye trackers to capture eye movement patterns. Each tracker has changeable annotation precision [7]. A method has been proposed to identify subjectivity extraction through anticipation and homing, with the use of eye tracking [8]. All the existing methodologies have concentrated on various eye trackers to record eye movements. The proposed work uses eye movement data that are captured using EOG. C. Emotion Recognition: Eye tracking technology has been used in recent times for sentiment analysis and annotation related research [9]. Traditional emotion recognition systems evaluate emotions based on facial expression and speech. Psychologically, it is proved that human emotions can be found out using eye movements. Researches have been done on emotion recognition using EEG signals [10]. This work concentrates on emotion recognition using EOG signals. D. EOG Applications: The electrooculography (EOG) based HCI system is used for providing information about human eye activity by detecting changes in eye position [11]. EOG systems are widely used in HCI system. A driver drowsiness revealing system was developed using EOG, measures the physiological EOG signal, processes the signal, and sends it to an Android phone through Bluetooth communication [12][13]. Devices such as television, wheelchair can be controlled using an EOG based system [14]. Such system serves as a communication medium for the disabled people, especially when there are some physical restrictions that prevent them from the usage of HCI devices. A grouping method for EOG based HCI system has been suggested for sensing eye movements in eight directions [15]. Eye movement features such as saccades, fixations and blinks can be measured using Electrooculography. These revisions reveal that EOG is an economical measurement procedure for recording eye movements. All the above applications uses EOG in such a way that it acts as an interface and controls the applications, our approach is to use EOG as a source of information for emotion recognition. E. Valence - Arousal model: In the field of Emotion research and affective sciences, Emotional classification is done based on different emotional models. Valence Arousal model is one such model, used to categorize emotions based on valence and arousal state. The term Valence refers to the feel of the person that transforms from positive to negative, pleasant to unpleasant. The term Arousal refers to amount of anticipation transforming from calm to excited. Valence can be considered in two different manner intrinsic attractiveness and aversiveness (Positive and Negative). The level of arousal can be expressed as high or low. High arousal refers to the high energetic state and low arousal refers to state of tiredness or immobility. The combination of these four factors gives different emotions. Such that positive/high arousal (happy), positive/low arousal (pleasant), negative/high arousal (anger and fear), negative/low arousal (sad). An enormous range of researchers had concentrated in this model for emotion recognition systems. A multi classification system was propsed for sentiment analysis in blog posts [19]. An emotional control system was proposed as a therapy system, which prompts different type of music for different emotions of the subject [18]. Thus, from the literature survey it is examined that Electrooculography has been used for an extensive array of applications. The proposed work targets at building an emotion recognition system by means of human eye movements which are captured using Electrooculography.

3 40 R. S. Soundariya and R. Renuga., 2017/Advances in Natural and Applied Sciences. 11(5) Special 2017, Pages: Methodology: The work is presented for making a system which is able to detect human emotions through EOG signals. The methodology is organised as follows: an overview of the architecture of eye-based emotion recognition, EOG signal processing, Feature extraction and selection, Classification. Finally Multiclass SVM is used for classifying five different emotions. A. Emotion recognition architecture: Figure1. shows the overall architecture for eye based emotion detection. [11] Records the eye movement for a set of eight routine computer users aged between 23 years and 31 years, who reports 6 to 14 hours of use per day. They had used a commercial EOG device, the Mobi8, from Twente Medical Systems International (TMSI). The proposed methodology uses the horizontal and vertical signals which were mentioned in [11] Fig. 1: Overall architecture for emotion recognition B. EOG signal processing: EOG signals may be degraded with noise from different sources such as housing power line, electrodes or wires and also physical deeds may also cause the electrodes or wires to lose contact. To ascertain the appropriate methods for noise removal, two different algorithms are equated: a low-pass filter and median filter. By Visual investigation, it is found that median filter shows improved performance than low-pass filter. C. Feature extraction and Feature selection: Features like mean, variance, standard deviation, skewness, auto-correlation, Kurtosis, maximum, summation are calculated for the horizontal and vertical EOG signals of all the participants. Independent Component Analysis (ICA) is used as a redundancy technique for Feature selection. ICA splits the de-noised signal into individual components and calculates the individual mean value for all participants. D. Classification: Classification is done after extracting the features that produce the resulting output as happy, pleasant, sad, fear and angry. A linear Multiclass Support Vector Machine (SVM) is used as a classification technique. The extension of traditional Support Vector machines is called as Multiclass SVM, which can be obtained by coalescing multiple two-class SVM. Support vector machine kernel classifier is used to order the emotions based on positive and negative value of arousal and valence features [10]. It uses a hyper plane that splits up the datasets into suitable classes. Primarily binary classification is done, which splits up the data into two groups as positive (0) and negative (1) classes. Based on the Valence Arousal model, positive classes are classified into two emotions: happy and pleasant and the negative classes are classified into three emotions: sad, angry and fear. Result Analysis: The various results that are obtained during each stage are summarized below:

4 41 R. S. Soundariya and R. Renuga., 2017/Advances in Natural and Applied Sciences. 11(5) Special 2017, Pages: Fig. 2: Input EOG signal Fig. 3: Signal after noise removal Fig. 4: Feature selection using ICA

5 42 R. S. Soundariya and R. Renuga., 2017/Advances in Natural and Applied Sciences. 11(5) Special 2017, Pages: Fig. 5: Difference between the denoised signal using ICA (blue) and the denoised signal without using ICA (red) Fig. 6: Emotion Recognition (Message received in Mobile) Thus, the above results show the various processing stages of the given EOG signal. SVM is used for classification, as its performance is better as compared to other classifiers in emotion detection. Emotion recognition for the given input signal is done using the Valence Arousal model. The following table summarizes the valence and arousal value for various emotions: Table I: Valence and Arousal Valence Arousal Range Emotion positive high 6.62 and 7.15 happy positive low 3.36 and 4.47 pleasant negative high 7.94 and 8.35 angry negative low 3.51and 4.75 sad negative high 7.94 and 8.35 fear Applicability Scenario: This system can be used in Personal Assistance system where the system can automate or react accordingly based on the emotion of the user. This has a very big impact as it adds a human nuance to the AI powered computer systems. Apart from this it can be used in various social applications.

6 43 R. S. Soundariya and R. Renuga., 2017/Advances in Natural and Applied Sciences. 11(5) Special 2017, Pages: Conclusion And Future Work: Thus the overall architecture of this system deals with a system where the emotion of a person is captured using an optimum method which is advantageous over the existing systems with complex structures. And also the optimality of this system nowhere complements the accuracy. SVM can be used as a classification technique as it is effective in handling composite datasets. The proposed system uses Multiclass SVM to classify the user in dissimilar emotions such as happy, sad, angry, fear and pleasant. In near future this system can be used in many forms which may play a vital role in the sociality of the human computer interactions and paves a whole new way for the understanding of the human computer interactions. REFERENCES 1. Maria Guarnera, Zira Hichy, Maura I. Cascio, Stefano Carrubba, Facial Expressions and Ability to Recognize Emotions From Eyes or Mouth in Children, Europe's Journal of Psychology, 11(2). 2. Yifei Lu, Wei-Long Zheng, Binbin Li, Bao-Liang Lu, Combining Eye Movements and EEG to Enhance Emotion Recognition, IJCAI 15,24 th International Conference on Artificial Intelligence, Buenos Aires, Argentina, pp: Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow and Rebecca Passonneau, Sentiment analysis of twitter data. In Proceedings of the Workshop on Languages in Social Media, pp: Bouazizi, M. and T. Otsuki Ohtsuki, "A Pattern-Based Approach for Sarcasm Detection on Twitter," in IEEE Access, 4: Mishra, A., D. Kanojia, S. Nagar, K. Dey, P. Bhattacharyya, Leveraging Cognitive Features for Sentiment Analysis in Proceedings of the 20 th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pp: Hacisalihzade, S.S., L.W. Stark and J.S. Allen, Visual perception and sequences of eye movement fixations: a stochastic modelling approach, IEEE Trans. Systems, Man and Cybernetics, 22(3): Hayhoe, M.M. and D.H. Ballard, Eye movements in natural behavior, Trends in Cognitive Sciences, 9: Abhijit Mishra, Aditya Joshi and Pushpak Bhattacharyya, A cognitive study of subjectivity extraction in sentiment annotation, pp: Vidas Raudonis, Evaluation of Human Emotion from Eye Motions International Journal of Advanced Computer Science and Applications (IJACSA), 4: Prashant Lahane, Shrutika Lokannavar, Apurva Gangur de, Poonam Bhosale and Pooja Chidre, EEG Based Emotion Recognition System, International Journal of Computer Science and Information Technologies, 5(6): Bulling, A., J.A. Ward, H. Gellersen and G. Troster, "Eye Movement Analysis for Activity Recognition Using Electrooculography," in IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(4): Sahayadhas, A., K. Sundaraj and M. Murugappan, "Detecting driver drowsiness based on sensors: a review,"sensors, 12: Ma, Z., B.C. Li and Z. Yan, "Wearable driver drowsiness detection using electrooculography signal," 2016 IEEE Topical Conference on Wireless Sensors and Sensor Networks (WiSNet), Austin, TX, pp: Ali Marjaninejad, Sabalan Daneshvar, A low-cost real time wheelchair navigation system using Electrooculography The 22nd Iranian Conference on Electrical Engineering (ICEE 2014). 15. Shang-Lin, W., et al., "Controlling a human-computer interface system with a novel classification method that uses electrooculography signals Biomedical Engineering, IEEE Transactions on, 60: Md. Shazzad Hossain, Kristie Huda and Mohiuddin Ahmad, Command the Computer with Your Eye An Electrooculography Based Approach, IEEE Transactions on, 60: Huda, Md. Shazzad Hossain, Recognition of Reading Activity from the Saccadic Samples of Electrooculography Data 1st International Conference on Electrical & Electronic Engineering (ICEEE), Uma, M. and S.S. Sridhar, A feasibility study for developing an emotional control system through brain computer interface,international Conference on Human Computer Interactions (ICHCI), Chennai, pp: Paltoglou, G. and M. Thelwall, Seeing Stars of Valence and Arousal in Blog Posts,"in IEEE Transactions on Affective Computing, 4(1): , doi: /T-AFFC