Development of a Domestic Device to Measure and Analyze Physiological Signals for Emotion Recognition

Transcription

1 Development of a Domestic Device to Measure and Analyze Physiological Signals for Emotion Recognition Chun-Ju Hou 1, Min-Wei Huang 2, I-Chung Hung 1, Jia-Ying Zhou 3, Yen-Ting Chen 1* 1 Department of Electrical Engineering, Southern Taiwan University of Science and Technology, Tainan, Taiwan 2 Department of Psychiatry, Chiayi Branch, Taichung Veterans General Hospital, Chia-Yi, Taiwan 3. Asia University, Taichung, Taiwan Abstract We used electrocardiography, electromyography, photoplethysmography, galvanic skin response, and skin temperature to analyze emotions in two participant groups. We presented emotionally triggering images (featuring happiness, sadness, fear, and disgust) to 23 male participants while simultaneously measuring neurophysiological signals. The average age of the participants was 22.91±2.15. Participants with HAM-D scores higher than 7 were considered to be depression-prone. Of the 23 participants, 5 had HAM-D scores higher than 7. We obtained 46 characteristic factors for the participants in the two groups, and subjected the data to a nonparametric Wilcoxon two-sample test. The resulting accuracy rate for the emotion disgust was the highest of the four emotions tested. Index Terms Physiology Signals, Emotion Recognition, Depression, Signal Processing I. INTRODUCTION An emotional response comprises a subjective feeling and an accompanying physiological reaction, which are often elicited by an external trigger. A variety of emotions have been described and categorized, and these vary according to different scholars and academic theories. There are three main theories of emotion in the field of physiological psychology: the James Langes theory [1], which considers physiological responses to be the cause of emotional responses, the Cannon and Bard theory [2], which states that physiological and emotional responses occur simultaneously, and the Schachter Singer theory [3], which positions cognitive and physiological factors as the deciding factors of emotional responses. Although the above theories differ in their approach and assessment of the causes of emotional responses, they agree that there is a close relationship between emotional and physiological responses. Measurement of physiological signals is an essential component of clinical diagnosis and academic research. Some physiological signals (such as heart rate, blood pressure, and body temperature) can be easily measured, while others may be more difficult to obtain outside of a clinical environment [4]. Physiological signals may be affected by illness or anomalies in the body [5]; however, they can also be affected by the emotions of the test participants. A number of studies have used physiological signals to analyze emotions [6] [11]. New technological initiatives aim to improve interactive measurements of user emotions, with the hope that these products will improve human-computer interactions. Domestic devices capable of analyzing the physiological and emotional states of individuals may lead to improved awareness and maintenance of mental health. For instance, a person who is prone to depression may benefit from such a system. The purpose of this study was to integrate hardware and software to develop a domestic device for measuring physiological signals and analyzing emotions. A. System Overview II. MATERIALS AND METHODS We developed a domestic device that collected measurements via electrocardiography (ECG), electromyography (EMG), photoplethysmography (PPG), galvanic skin response (GSR), and skin temperature (SKT), and then used the characteristics of these physiological signals, along with a categorization system for emotions, to distinguish whether an individual was prone to depression. The device was composed of three main modules: (1) a 253

2 physiological signal acquisition module, which amplified and filtered physiological signals; (2) a digital signal processing module, which converted, processed, and analyzed digital signals; and (3) an external storage and display module, which facilitated the storage of the data on SD cards and displayed the results on an LCD monitor (Figure 1). Fig 1: System module diagram B. Characteristics and Measurements of Physiological Signals Our domestic device measured ECG, EMG, PPG, GSR, and SKT. Brief explanations regarding each physiological signal are as follows: (1) ECG is used to measure the electrical activity of the heart over time. A complete ECG signal should contain P waves, QRS complexes, and T waves, which comprise the various intervals of a heartbeat (see Figure 2 below). Fig 2: ECG diagram [12] (2) EMG is used to measure electrical activity in skeletal muscles. (3) PPG is used to measure changes in peripheral microvascular blood volume. A complete microvascular blood 254

3 Skin conductance(µs) PPG Amplitude ISSN: flow waveform is shown in Figure 3. Time(sec) Fig 3: Micro vascular blood flow waveform (4) GSR is used to measure the electrical conductance of the skin. The electrical conductance of the skin can vary with moisture levels, cuticle thickness, chemical substances, and so on, although the level of moisture on the skin has the greatest impact on electrical conductance levels [13] (Figure 4). Time(sec) Fig 4: GSR waveform [14] (5) SKT is used to measure skin temperature. The acceptable range of body temperature for humans varies depending on the site at which the measurement is taken. Body temperature can also be affected by physiological responses, environmental temperature, and seasonal changes. The frequency range and signal strength of the five physiological signals measured by the domestic device is shown in Table 1. Table.1 Frequency ranges and signal strengths of physiological signals Frequency Range Signal Strength ECG 0.01~250Hz 0.5~4mV EMG 10~1000Hz 0~10mV PPG 0~30Hz Varies with organ measured GSR 0.01~1Hz 1~500kΩ SKT 0~0.1Hz 32~40 The five physiological signals were measured using a variety of equipment attached to various sites on the body: (1) ECGs were measured using silver-chloride electrode patches attached to the left hand, right hand, and right foot in the pattern indicated by Einthoven s triangle [14] (Figure 5). 255

4 Fig 5: Silver-chloride electrode patches (2) EMGs were measured using electrode patches attached to the right trapezius muscle, as emotions can cause contractile responses in the trapezius muscles. Fig 6: Electrode patches for measuring EMGs (3) To measure PPGs, we used the CNY70, which was manufactured by Vishay Semiconductors (PA,USA). The CNY70 contains an infrared LED and a phototransistor. The CNY70 was attached to the top joint of the right forefinger. Fig 7: CNY70 (4) To measure GSRs, we attached two copper plates (both squares with sides 1.5 cm in length) to the second joints of the middle and ring fingers [13]. Fig 8: Copper plates used to measure GSRs 256

5 (5) To measure SKTs, we used 503ET-3H thermistors manufactured by Semitec. 503ET-3H thermistors were chosen because they have a higher sensitivity and faster reaction time than traditional thermistors. The thermistors were attached to the top joint of the right little finger. C. Hardware Specifications Fig 9: 503ET-3H thermistors As most physiological signals are weak (at the millivolt level) and prone to noise interference, we found it necessary to amplify the collected signals and to filter out noise prior to our analysis. In accordance with Nyquist principles, all sample frequencies were set to 1024Hz to prevent distortion. Figure 10 contains a diagram of the analog circuit structure: Fig 10: Analog circuit structure diagram (1) Pre-amplifiers: We used the AD620 instrumentation amplifier, manufactured by Analog Devices, to pre-amplify the ECG and EMG signals. The AD620 is composed of two non-inverting amplifiers and one differential amplifier. It has an input impedance of 10GΩ and a common-mode rejection ratio (CMRR) of 93dB. (2) Driven right leg circuit: for the ECG, we used the Lead I method of measuring analog signals. However, instead of a wire leading from the right leg to the earth, as specified by Einthoven, we attached a DRL to the right leg of the participants to reduce common-mode noise and interference from electrical power lines. For safety reasons, we created a parallel connection between Rf and Ro to limit the flow of electricity into the human body. Abnormally large voltage levels were neutralized by an earth lead running through the operational amplifier, as shown in Figure

6 Fig 11: Driven right leg circuit diagram [14] (3) Band-pass filters: to filter the ECG and EMG signals, we used four-pole Butterworth high pass filters and four-pole Butterworth low pass filters in a circuit designed using the Sallen Key topology [15]. To filter the PPG signals, we used a two-pole low pass filter (cutoff frequency 15Hz) in a circuit with a design similar to the one described above. We used the low pass filter to remove 60Hz interference from electrical power lines and movement interference caused by movement of the fingers. To counter the DC bias of the CNY70 used to measure the PPGs, we designed a passive high pass filter with a cutoff frequency of 1Hz. To filter the GSR and SKT signals, we used a two-pole Butterworth low pass filter designed according to the Sallen Key unity-gain method [15]. The cutoff frequency of the low pass filter was 25Hz. Again, we used the low pass filter to remove 60Hz interference from electrical power lines and movement interference from movement of the fingers. (4) Notch filters: we used notch filters to filter out interference from electrical power lines when measuring ECG and EMG. The frequency of Taiwanese electrical power lines is 60Hz, which falls within the range of ECG frequencies (0.5Hz-250Hz) and EMG frequencies (10Hz-1000Hz). Therefore, it was necessary to filter out the interference from the electrical power lines. (5) Post amplifiers: after amplification and filtering, we passed the ECG and EMG signals through a series of post amplifiers. The post amplifiers were used to adjust the amplitudes of the analog signals to optimal levels for digital processing. (6) Isolation amplifiers: to prevent electrocution, we used isolation amplifiers, manufactured by Texas Instruments (ISO124), to isolate signals when measuring ECGs, EMGs, and GSRs. The ISO124 protected against 1500Vrms, provided anti-interference against 60Hz signals at 140dB, and had a maximum nonlinearity of 0.01% and output voltage of ±10V. (7) Amplifiers: as GSR and SKT signals are weak, we found it necessary to increase the amplitude to optimal levels for digital processing. We used OPA333 amplifiers; manufactured by Texas Instruments, which have low drift (10µV), low zero-drift (0.05µV/ C), low operating voltage (1.8V 5.5V), and rail-to-rail operation. The digital circuit structure of our domestic device is shown in the diagram below: 258

7 Fig 12: Digital circuit structure The digital circuit included circuits for the microprocessor, the driver for the Secure Digital card (SD card), the driver for the LCD, the Universal Asynchronous Receiver/Transmitter (UART), and the driver for the music player. We used the MSP430F5438A microprocessor, manufactured by Texas Instruments. This 16-bit, low power consumption, mixed-signal microprocessor chip was chosen mostly for its multi-channel A/D converter. The A/D converter of the MSP430F5438A resulted in a resolution of 0.8mV and sample rate of 200-ksps. The firmware program on the microprocessor was written using the IAR Embedded Workbench development tool produced from IAR Systems. SD cards are a type of storage device with flash memory technology, and are widely used in electronic products such as cameras, cell phones, and music players. There are two file format types for SD cards (FAT16 and FAT32) and two communication modes (SD mode and Serial Peripheral Interface (SPI) mode). The SD card chosen for this study used the FAT32 file format and the SPI communication mode. The connections between the SD card and the MSP430F5438A microprocessor are shown in Figure V 50k 9. DAT2/NC 1. DAT3/CS 2. CMD/DI 3. VSS1 4. VDD 5. CLK/SCK 6. VSS2 7. DAT0/DO 8. DAT1/IRQ CS SIMO CLK SOMI MSP430 D. Characteristic Factors Fig 13: Connection diagram between SD card and microprocessor After removal of baseline drift and low pass filter, the physiological signals were also under the following processes before feature extractions: (1) ECG: Heart Rate, the Mean of RR interval(mean), Standard Deviation of Normal to Normal(SDNN), the Coefficient of Variation(CV) from Normal to Normal Interval(NNI), the Number of pairs of adjacent NN intervals differing by more than 50 ms in the entire recording(nn50), the NN50 count divided by the total number of all NN intervals(pnn50) [16], Standard derivation of successive RRIs(SDSD), the root means of squared successive RRI's RRIS (RMSSD) [17]. (2) EMG: Electrical Activity(EA), Root Mean Square(RMS) [18]. (3) PPG: Heart Rate, the Mean of RR interval(mean), Standard Deviation of Normal to Normal(SDNN), the 259

8 Coefficient of Variation(CV) from Normal to Normal Interval(NNI), the Number of pairs of adjacent NN intervals differing by more than 50 ms in the entire recording(nn50), the NN50 count divided by the total number of all NN intervals(pnn50), Standard derivation of successive RRIs(SDSD), the root means of squared successive RRI's RRIS (RMSSD), PPG Mean, PPG Energy, PPG Time Duration, PPG Bandwidth, PPG Bandwidth Product, PPG Dimensionality, PPG Rise Time, PPG Fail Time, Pluse Height Max, Pluse Height Min, Cardiac Period [19]-[20]. (4) GSR: GSR Mean, GSR Energy, GSR Bandwidth, GSR Dimensionality, GSR Time Duration, GSR Time Bandwidth Product [20], GSR Peak Rise Time Sum, GSR Peak Amplitude Sum, GSR Half Recovery Sum [21], GSR Peak Energy Sum, GSR First Derivative Average [22], GSR % Decay [23], Root Mean Square of GSR, GSR Number of Peaks [19]. (5) SKT: the Mean of RR interval (Mean), Root Mean Square(RMS), Standard Deviation(SD). E. Experiment Design and Statistical Analysis To test whether our device could detect changes in emotional response based on changes in physiological signals, we asked a total of 23 test participants (five of whom were prone to depression) to watch a short film containing emotion-triggering images from the International Affective Picture System (IAPS). The physiological signals of the participants were collected simultaneously during the course of the film. The data were then subject to nonparametric analysis and discriminant analysis. All statistical analyses were conducted using SPSS III. RESULTS A total of 23 adults, all male, participated in this experiment. The average age of the participants was 22.91±2.15. Before completing the experiment, all participants were asked to answer questions taken from the Hamilton Depression Rating Scale (HAM-D). The questions were read aloud by a researcher, who was also responsible for determining the HAM-D scores of the participants based on their answers. Participants with a HAM-D score higher than 7 were considered to be depression-prone. Of the 23 participants, five had HAM-D scores higher than 7. Using the physiological signals collected from the participants during the course of the experiment, we evaluated a total of 46 characteristic factors, as shown in Table 2. We calculated the means and standard deviations of the characteristic factors. The characteristic factors were then subjected to a nonparametric Wilcoxon two-sample test (Mann Whitney U test). Table 2: Table of Physiological Signal Factors Physiological signals ECG EMG PPG SKT GSR Heart Rate NN50(times) SDSD EA RMS Heart Rate CV(%) SDSD PPG_Energy PPG_Bandwidth Product Pluse Height Min Cardiac Period Mean RMS SD GSR Mean GSR RMS GSR Time Bandwidth Product GSR Peak Rise Time Sum GSR Half Recovery Sum Characteristic Factors Mean pnn50(%) RMSSD Mean NN50(times) RMSSD PPG_Time Duration PPG_Dimensionality Pluse Height Max GSR Time Duration GSR First Derivative Average GSR Dimensionality GSR Peak Amplitude Sum GSR Number of Peaks SDNN CV(%) SDNN pnn50(%) PPG_Mean PPG_Bandwidth PPG_Rise Time PPG_Fall Time GSR Bandwidth GSR Energy GSR % Decay GSR Peak Energy Sum 260

9 IV. DISCUSSION A study [6] conducted in 2004 used a support vector machine method to evaluate emotions from physiological data. The authors reported a maximum accuracy rate of 78.4%. A similar study [7] in 2005 used three methods: linear discriminant analysis, k-nearest neighbor, and multilayer perceptron, and produced a maximum accuracy rate for predicting emotions of 92%. An additional study [16] in 2006 used canonical correlation analysis, and produced a maximum accuracy rate of 85.3%. A study [17] conducted in 2008 used an adaptive neuro-fuzzy inference system, and produced a maximum accuracy rate of 79.3%. Finally, a study [18] conducted in 2011 used four methods: support vector machines, artificial neural networks, a neuro-fuzzy system, and a random forest system, to produce a maximum accuracy rate of 84.3%. Our analysis of the differences between Group 1 and Group 2 for the emotion happy resulted in an original classification accuracy rate of 87%, and a cross validation classification accuracy rate of 87%. Both classifications had relatively high levels of accuracy. Our analysis of the differences between Group 1 and Group 2 for the emotion sad resulted in an original classification accuracy rate of 91.3%, and a cross validation classification accuracy rate of 82.6%. Both classifications had relatively high levels of accuracy. Our analysis of the differences between Group 1 and Group 2 for the emotion fear resulted in an original classification accuracy rate of 87%, and a cross-validation classification accuracy rate of 82.6%. Both classifications had relatively high levels of accuracy. Our analysis of the differences between Group 1 and Group 2 for the emotion disgust resulted in an original classification accuracy rate of 100%, and a cross-validation classification accuracy rate of 95.7%. Both classifications had relatively high levels of accuracy. The accuracy rate for the emotion disgust was the highest of the four emotions tested. V. CONCLUSIONS We successfully conducted calculations for 46 characteristic factors in the participant group. However, we found that the variance in the standard deviations for some characteristic factors was quite large. This was mostly because of large variance between the physiological signals of the participants. While discriminant analysis for each group showed high accuracy rates for the original classifications, the accuracy rates for cross-validation classifications were lower than expected. This is likely because of the large variance between participants. Our data show a clear difference between the physiological signals of participants that were prone to depression and those that were not prone to depression. These results support the notion that there is a correlation between emotions and physiological signals. The number of participants in both Group 1 and Group 2 were small. This small sample size could have affected the classification and efficiency of our analysis. In the future, we hope to collaborate with hospitals to further examine the differences between physiological signals in depressed and non-depressed individuals. REFERENCES [1] W. James, What is an Emotion?, Mind, vol. 9, no. 34, pp , [2] W. B. Cannon, The James-Lange Theory of Emotion: A Critical Examination and an Alternative Theory, The American Journal of Psychology, vol. 39, pp , [3] S. Schachter, J. E. Singer, Cognitive, Social, and Physiological Determinants of Emotional State, Psychological Review, vol. 69, no. 5, pp , [4] S. S. Mader, Understanding Human Anatomy & Physiology, 4 edition, McGraw-Hill, [5] M. J. Goldman, Principles of Clinical Electrocardiography, 12th edition, Lange Medical Pubns, [6] K. H. Kim, S. W. Bang, S. R. Kim, Emotion recognition system using short-term monitoring of physiological signals, Medical & Biological Engineering & Computing, vol. 42, pp , [7] J. Wagner, et al, From physiological signals to emotions: implementing and comparing selected methods for feature extraction and classification, Multimedia and Expo, pp , [8] P. Rainville, A. Bechara, N. Naqvi, A. R. Damasio, Basic emotions are associated with distinct patterns of cardiorespiratory activity, International Journal of Psychophysiology, vol. 61, no. 1, pp. 5-18, [9] F. Canento, A. Fred, H. Silva, H. Gamboa, A. Lourenco, Multimodal biosignal sensor data handling for emotion recognition, Proceedings of the IEEE Sensors Conference, pp ,

10 [10] C. D. Katsis, N. S. Katertsidis, D. I. Fotiadis, An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders, Biomedical Signal Processing and Control, vol. 6, no. 3, pp , [11] D. Giakoumis, D. Tzovaras, K. Moustakas, Automatic recognition of boredom in video games using novel biosignal moment-based features, IEEE Transactions on Affective Computing, vol. 2, no. 3, pp , [12] G. S. Wagner, Marriott's practical electrocardiography, 10th edition, Lippincott Williams & Wilkins, [13] B. Wolfram, Electrodermal Activity, Plenum Press, [14] J. G. Webster, Medical Instrumentation: Application and Design, John Wiley & Sons, [15] R. P. Sallen, E. L. Key, A Practical Method of Designing RC Active Filters, IRE Transactions on Circuit Theory, vol. 2, no. 1, pp , [16] T.Y. Chen, et al, Cardiac autonomic functions derived from short-term heart rate variability recordings associated with no diagnostic results of treadmill exercise testing, Int Heart J., vol. 51, no. 2, pp , [17] R. K. Sharma, Y. P. S. Balhara, R. Sagar, K. K. Deepak, M. Mehta, Heart rate variability study of childhood anxiety disorders, Journal of Cardiovascular Disease Research, vol. 2, no. 2, pp , [18] R. Merletti, M. Knaflitz, C. J. De Luca, Electrically evoked myoelectric signals, CRC Critical Reviews in Biomedical Engineering, vol. 19, pp , [19] R. R. Singh, S. Conjeti, and R. Banerjee, An approach for real-time stress-trend detection using physiological signals in wearable computing systems for automotive drivers, th International IEEE Conference on Intelligent Transportation Systems, pp , Washington, DC, USA, [20] C. S. Lessard, Signal Processing of Random Physiological Signals, Morgan & Claypool, [21] S. Schmidt, H. Walach, Electro dermal Activity (EDA) State-of-the-Art Measurement and Techniques for Parapsychological Purposes, The Jour. of Parapsychology, vol. 64, pp , [22] J. A. Healey, R.W. Picard, Detecting Stress During Real-World Driving Tasks Using Physiological Sensors, IEEE Trans. Intelligent Transportation Systems, vol. 6, no. 2, pp , [23] M. Soleymani, G. Chanel, J.J.M. Kierkels, T. Pun, Affective Ranking of Movie Scenes Using Physiological Signals and Content Analysis, Proceedings of the 2nd ACM workshop on Multimedia semantics 2008, pp , [24] L. Li, J. H. Chen, Emotion recognition using physiological signals, Advances in Artificial Reality and Tele-existence Lecture Notes in Computer Science, vol. 4282, pp , [25] C. D. Katsis, N. Katertsidis, G. Ganiatsas, D. I. Fotiadis, Toward emotion recognition in car-racing drivers: a biosignal processing approach, IEEE Transactions on Systems, Man, And Cybernetics, Part A:Systems and Humans, vol. 38, pp , [26] F. Canento, A. Fred, H. Silva, H. Gamboa, A. Lourenco, Multimodal biosignal sensor data handling for emotion recognition, Proceedings of the IEEE Sensors Conference, pp ,