CHENNAI - INDIA Emotion based E-learning System using Physiological Signals School of Engineering, Vels University, Chennai
Outline Introduction Existing Research works on Emotion Recognition Research Methodology Results Emotion based e-learning system GUI Future Work MATLAB tools Publications
Introduction Emotions Mental and physiological state associated with feelings, thoughts and behavior Learnt in diverse areas like Psychology, Cognitive Science, Philosophy and Computer Science, However there hasn t been an universally accepted definition or categorization of emotional states Highly subjective and an integral part of any communication, learning, perception and decision making
Introduction (Cont.) Need of Emotion Recognition System Intelligent machine interfaces for smooth interaction between machines and human Human Computer Intelligent Interaction (HCII) for natural interaction between human and machines Improves mutual empathy Smart classrooms, Computer based Training, Medical applications
Introduction (Cont.) Emotion Recognition Methods SELF REPORT methods Questionnaires, Ratings and descriptions provided by the subject Participant biased
Introduction (Cont.) Emotion Recognition Methods (Cont.) OBSERVOR (BEHAVIOUR) methods Facial, Vocal and gesture cues Dependant on external circumstances and prone to social masking
Introduction (Cont.) Emotion Recognition Methods (Cont.) NEURO-PHYSIOLOGICAL signal processing methods Physiological response of the Central Nervous System (CNS) and Autonomous Nervous System (ANS) Electroencephalogram (EEG), Electrocardiogram (ECG), Electromyogram (EMG), Galvanic Skin Response (GSR), Skin Temperature (ST), Skin Conductance (SC) etc., Complex, but provides the TRUE emotional state of the person.
Introduction (Cont.) Applications of Emotion recognition systems Robots Dialogue systems Computer based learning Smart Classroom Therapists for ASD Medical Doctors
Introduction (Cont.) E-learning systems Feedback mechanisms in existing e-learning systems help to study at the pace of the user. They take into account only the understanding on the subject and not the state such as fatigue, emotions etc.,
Introduction (Cont.) Emotion based E-learning systems Increases the receptiveness and productivity of the user Suggests appropriate action to be taken depending on the emotional state of the learner
Existing research works on Emotion Recognition REFERENCES BIOSIGNALS NO OF SUBJECTS NO OF EMOTIONS EMOTIONAL STIMULI CLASSIFICATION RATE (%) (Picard, Vyzas et al. 2001) EMG, BVP, SC, RR 1 8 Personalized Imagery 81(User Dependent) (Lisetti and Nasoz 2004) GSR, HR, ST 29 6 Movies 72(User Dependent) 75(User Dependent) 84(User Dependent) (Lan and Ji-hua 2006) ECG, ST, SC, RR 60 3 Movies 85.3 (User Dependent) (Maaoui and Pruski 2008) BVP, EMG, SC, ST, RR 25 6 Visual (IAPS) 88(User Dependent) (Jonghwa and Ande 2008) EMG, ECG, SC, RR 3 (22 trials) MIT database 4 Music 95(User Dependent) 70(User Independent) (Kim and André 2009) EMG, ECG, SC, RR 3 (22 trials) MIT database 4 Music 95% (user dependant) 70% (user independent) (Kim and André 2009) EMG, ECG, SC, RR 3 (22 trials) MIT database 4 Music 91% (user dependant)
Existing research works on Emotion Recognition (Cont.) REFERENCES BIOSIGNALS NO OF SUBJECTS NO OF EMOTIONS EMOTIONAL STIMULI CLASSIFICATION RATE (%) (Chuan-Yu, Jun-Ying et al. 2010) ECG, RR, GSR, BVP 11 3 Movies 90.6% and 90.2% (user independent) (Maaoui and Pruski 2010) BVP, EMG, ST, SC, RR 10 6 Visual (IAPS) 90(User Dependent) and 45(User independent) (Gouizi, Reguig et al. 2011) (Valenza, Lanata et al. 2012) 4 6 Visual (IAPS) 85% (user dependent) EDA, ECG, RR 35 2 Visual (IAPS) 90% (user independant) (Vanny, Park et al. 2013) BVP, ST, SC 4 4 Visual (IAPS) 100% for fear and joy, 60% for disgust and neutral (User dependent) (Chang, Chang et al. 2013) ECG, GSR, BVP 11 4 Movies 89.2% (user independant)
Existing research works on Emotion Recognition (Cont.) Emotion classification rate varies from 45% to 100% for the different research works. They cannot be compared as they vary in, Number of subjects Type of elicitation Type of physiological data Number and placement of electrodes Type of analysis (user dependency) Though there is no standardization, some of the significant advances are IAPS, IADS database for emotion induction Development of AuDB database (4 emotional states) Statistical features for emotion recognition (Picard et al., 2001)
Existing research works on Emotion Recognition (Cont.) Challenges Emotion database of physiological signals Emotional states must be elicited internally The sensors should be less intrusive and at the same time capture the emotional changes Methodology Complex, non-linear and non stationary nature of physiological signals Subject dependence of emotions Reliability
Research Methodology Step 1 Development of Emotional data base (ECG, EMG) Step 2 Preprocessing Step 3 Feature Extraction (Linear and non-linear methods) Step 4 Fusion of emotional features derived from ECG and EMG Step5 Emotion Classification Step 6 Development of GUI
Maximum Classification Accuracy (%) Results Performance of Emotion Recogntition System 90 82.54 78.47 80 70 60 50 40 30 20 10 0 64.62 ECG EMG ECG and EMG
Emotion based E-Learning system Detects emotional state from ECG and EMG signals Suggests the activity based on the detected emotional state User can recheck the emotion again before proceeding further EMOTIONAL STATE Neutral Happiness Sadness Fear Surprise Disgust SUGGESTION Start Lessons Start Lessons Listen to Music Listen to Music Calm Down Play a game
GUI
Future Work On-line system Develop efficient algorithms to capture the emotional states More Automation Integrate with mobile apps, robots and other personalized devices
MATLAB TOOLS Signal processing Filter Design for pre-processing Feature Extraction algorithms Wavelets, Fourier Transform, Hilbert Huang Transform, Empirical Mode Decomposition, Hurst exponent Machine Learning Pattern Recognition KNN, Regression Tree, Bayesian Classifier Confusion Matrix SIMULINK Development of GUI
Publications International Journals Jerritta S, M Murugappan, Sazali Yacob, Khairunizam Wan ECG based Emotion Recognition System using Empirical Mode Decomposition and Discrete Fourier Transform, Journal: Expert Systems by Wiley Publishers (IF : 0.733) Jerritta S, Murugappan M, Khairunizam Wan, Sazali Yaacob, "Emotion recognition from Facial EMG signals using Higher Order Statistics and Principal Component Analysis", Journal of Chinese Institute of Engineers (JCIE)(IF : 0.295). Jerritta S, Murugappan M, Khairunizam Wan, Sazali Yaacob, "Classification of emotional states from electrocardiogram signals: a non-linear approach based on Hurst" Biomedical engineering online, May 2013. (IF:1.61). Jerritta S, Murugappan M, Khairunizam Wan, Sazali Yaacob, Frequency study of Facial electromyography signals with respect to emotion recognition, Biomedical engineering/ Biomedzinische technik, Degruter, (IF: 1.157) Dec 2013. Arun S, Sundaraj Kenneth, Murugappan M (2012) Hypovigilance detection using energy of Electrocardiogram signals, Journal of Scientific & Industrial Research, 71(12), 794-799. (ISI Impact Factor 0.505). Arun Sahayadhas, Kenneth Sundaraj and Murugappan Murugappan (2012) Detecting Driver Drowsiness Based on Sensors: A Review, Sensors, 12, 16937-16953. (ISI Impact Factor 1.953). Arun Sahayadhas, Kenneth Sundaraj and Murugappan Murugappan, (2013) "Drowsiness detection during different times of day using multiple features", Australasian Physical & Engineering Sciences in Medicine 36(2), 243-250 (ISI Impact Factor 0.885). Arun Sahayadhas, Kenneth Sundaraj M Murugappan, Electromyogram signal based hypovigilance detection, Biomedical Research 2014; 25(3), ISI Impact Factor 0.177). Arun Sahayadhas, Kenneth Sundaraj M Murugappan, Rajkumar Palaniappan, A Physiological Measures-Based Method for Detecting Inattention in Drivers Using Machine Learning Approach, Biocybernetics and Biomedical Engineering, Accepted for publication, ISI Impact Factor 0.157
Publications International Conferences Jerritta S, M Murugappan, R Nagarajan, Kahirunizam Wan, Physiological Signals Based Human Emotion Recognition: A Review, IEEE Colloquium on Signal Processing and Applications 2011, 3-5 March 2011, Penang, Malaysia. Jerritta S, M Murugappan, R Nagarajan, Khairunizam Wan, Investigation on different Emotion Elicitation Methods for Human Computer Interactaion, International Conference on Robotics Automation System 2011, 23-24 May 2011, Terungannu, Malaysia. Jerritta S, M Murugappan, R Nagarajan, Khairunizam Wan, Digital Filtering based Pre-processing of Electrocardiogram (ECG) Signals for Human Emotion Recognition, Malaysian Technical Universities International Conference in Engineering, 2011. M Murugappan, NQI Baharuddin, S Jerritta, DWT and MFCC based human emotional speech classification using LDA, International Conference on Biomedical Engineering (ICoBE), 2012. Jerritta S, M Murugappan, Khairunizam Wan, Sazali Bin Yaacob, Emotion Recognition from ECG using Hilbert Huang Transform, in IEEE STUDENT 2012, Oct 6-9, 2012. Jerritta S, M Murugappan, Khairunizam Wan, Sazali Bin Yaacob, Emotion detection from QRS complex of ECG signals using Hurst Exponent for different age groups, Submitted to IEEE 3rd Workshop on Affective Brain-Computer Interfaces, Geneva, Swizerland, Sept 2,2013. Arun S., Murugappan, M., & Sundaraj, K. (2011). Hypovigilance warning system: A review on driver alerting techniques. In Control and System Graduate Research Colloquium (ICSGRC), 2011 IEEE (pp. 65-69). Arun, S., Sundaraj, K., & Murugappan, M. (2012). Driver inattention detection: A review. In Sustainable Utilization and Development in Engineering and Technology (STUDENT), 2012 IEEE. (pp. 1-6)