Design of Intelligent Emotion Feedback to Assist Users Regulate Emotions: Framework and Principles

Transcription

1 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) Design of Intelligent Emotion to Assist Users Regulate Emotions: Framework and Principles Donghai Wang, James G. Budd Georgia Institute of Technology Atlanta, GA, USA Yu Hao Cornell University Ithaca, NY, USA Abstract Positive environmental emotion feedback is important to influence the brain and behaviors. By measuring emotional signals and providing affective neurofeedback, people can be better aware of their emotional state in real time. However, such direct mapping does not necessarily motivate people s emotion regulation effort. We introduce two levels of emotion feedback: an augmentation level that indicates direct feedback mapping and an intervention level which means feedback output is dynamically adapted with the regulation process. For the purpose of emotion regulation, this research summarizes the framework of emotion feedback design by adding new components that involve feature wrapping, mapping to output representation and interactive interface representation. By this means, the concept of intelligent emotion feedback is illustrated that not only enhances emotion regulation motivation but also considers subject and trial variability based on individual calibration and learning. An affective Brain-computer Interface technique is used to design the prototype among alternatives. Experimental tests and model simulation are planned for further evaluation. Self regulation & Automatic regulation Affect/reflect Fig. 1. The relationship between people and the environment Fig. 1 shows how people interact with the external environment and how our internal mechanism receives and responds to the external stimuli. Generally, people do something in the environment and if the behavior results in some consequence, the individual will receive the feedback of his/her behavior via environmental output. People will evaluate the results and the consequences themselves as stimuli effect their emotions. These form big and small loops of emotion feedback and regulation. Keywords affective brain-computer interface; emotion regulation; intelligent emotion feedback; design framework; environment; human-computer interaction The key methods of motivating people s emotion regulation are 1) augmenting their emotion awareness by affective neurofeedback, and 2) attracting their attention to positive and meaningful stimuli. How attention selects the stimuli and responds to them, how perception perceives them and how people interpret them will directly affect and strengthen the emotion [4]. I. INTRODUCTION Emotion regulation is an important research topic in psychotherapy. Emotion regulation refers to the processes by which we influence which emotions we have, when we have them, and how we experience and express them [1]. Without proper regulation, people are under risk of being mentally stressed especially with heavy work pressure and unexpected things occurring. How we regulate our emotions matters: Our wellbeing is inextricably linked to our emotions [2]. A failing of emotion regulation can be attributable to two reasons: one is that people do not typically recognize their emotional state, and the other is that without positive environment feedback, people are not able to motivate self-regulation [3]. In achieving the first method affective neurofeedback, recognizing and classifying emotion is an important first step. For more than two decades, researchers have been studying and developing Brain-Computer Interfaces (BCIs) that measure minute changes in brain signals [5]. Studies of different brain signals as well as other physiological signals produced during emotional experience have led to emotional state classifiers [6][7][8][9]. A growing interest in emotional (or affective ) BCIs has led to groundbreaking research in emotion communication [10]. In [8], researchers discovered that EEG signals were the best among other physiological measures at determining the arousal level of an emotion, but not the specific valence coordinate. People can get positive or negative feedback from the environment about their interactions and behaviors. Sometimes such feedback is timely and comprehensible so that they can use it to properly adjust their behavior. However, sometimes the feedback is not timely and does not provide positive modification. Positive environment feedback can influence them and help them adjust their behavior. Therefore, feedback from the environment is an important factor that can assist people to regulate their emotion state /15/$ IEEE Environment While this pioneering work in detecting and classifying emotional states is critical to the new field of affective BCI, there has been minimal attention focused on how this emotional information is displayed or communicated to users via 938

2 environmental dynamic interaction, even less attention on how to use the display to motivate emotion regulation. Without accurate, intuitive, and interactive displays, emotional response can be misdirected. In this paper, the concept of a feedback device and the environment are interchangeable. The goal of this study is to explore and propose the concept of meaningful and intelligent emotional feedback/display in the environment, controlled by brain signals, for people regulating emotions. In trying to achieve this goal, this paper mainly addresses two problems. The first one is to find the best mechanisms to deliver emotion information that will augment people s real-time emotional awareness. The second one is to find the best mechanisms to deliver emotion feedback as stimuli that will help people to reach a desired emotional state and facilitate emotion regulation. By examining the preliminary results of our previous experiment [11], we found these two issues should be studied separately. More detailed discussion is in the following sections. The remainder of this paper offers the following: a) Two typical design prototypes and other alternatives; b) The preliminary results from one experiment and subjects feedback; c) The intelligent emotion feedback design framework. II. DESIGN AND EXPERIMENT Since the encephalogram (EEG cap) interface is a quick tool to measure human brain activity, a commercial EEG device the Emotiv EPOC was selected. Reference [6] has conducted experiments using the Emotiv EPOC headset to record EEG signals while participants were watching emotional movies. Emotiv EPOC is a portable dry EEG system and it provides classified emotion arousal signals with an API for further development. Once an operative tangible system has been built, and empirical studies show the feasibility of this approach, the same principles could be applied on more accurate and portable EEG devices with more advanced hardware in the future. As shown in [12], where brain signals were captured and classified to influence changing lights projected on landmarks, environmental objects can display emotional state. Ambient displays [13] could indicate emotion by changing the colors of lights projected on the ceiling or walls of a room by mapping the colors to emotional states. Wearable objects, such as the bracelet studied in previous work [11], can display colors that depend on emotional state that could also serve as emotional indicators. A. Visual We designed a color-coded emotion feedback system based on EEG measurements in previous research [11]. The aim of this prototype was to test the affective output abstracted by the Emotiv EPOC. The experiment is shown in Fig. 2. The basic components of this prototype include: 1) the Emotiv EPOC device to detect brain signals; 2) a computer to transmit the brain signal Fig. 2. Experiment setting to an Arduino board; 3) an Arduino board to connect and display LEDs. A wearable wristband with an integral array of LEDs was designed to provide continuous visual feedback within the user's immediate cone of vision. The arousal level that Emotiv EPOC gives is the data range from 0 to 1, and in this prototype it is evenly divided by 7 intervals. Because the wristband color is a set of 7 colors rainbow colors, each color maps one interval. As people are familiar with the rainbow color orders, from red (warm color) to purple (cold color), it is natural to map the decline of the arousal level. Two groups of participants regulated emotion with and without feedback after they watched video clips with high arousal level such as traffic flows that agitated them. The subjects emotion baseline was collected by asking them to watch extreme sports and doing nothing to regulate emotions. During the experiment, we asked the participants to try to stay still as possible, so results were not affected by muscle movement. B. Motion Another prototype is a windmill that rotates along with the emotion regulation process. When the emotion arousal level is lower than a preset threshold, the windmill rotates on a dial plate indicating the regulation effort is successful. In Fig. 3, the prototype mechanism is illustrated. When the subject is wearing the Emotiv headset, the EEG signal is configured to generate an arousal level ranging from high level to low level. After filtering, the data is transmitted to the signal that controls the Arduino, which controls the pattern representation/reduction process. Here if we choose the threshold option, we set a value and define the motion based on the threshold. Then the Arduino controls the servo to let the windmill rotate, the speed of which is determined by Arduino control signal. A dial records the total time of windmill rotation. EEG signal High Low Emotiv EEG cap LED Fliter Windmill Video Servo Arduino Environment Arduino Dial Arduino control signal 1 0 Fig. 3. Motion feedback /15/$ IEEE 939

3 C. Design Alternatives The design of the emotion display involves two components. The first is how to map data to the output form, as discrete or continuous mapping. The other component is to map the output interface form, as a static visual element or a moving element. We propose a matrix of design alternatives shown in Table I, among them, two experimental schemes A and B are chosen as examples to be designed. Scheme A fits people who are sensitive to visual (color and bright) stimuli and the discrete changes provide alert effects. In scheme B, the controlled motion is a physical game-like feedback that can only be taken into account when achieving a level, which enhances the interaction experience. Other combinations could be explored in the future as options to fit into a range of appropriate application scenarios. III. PRELIMINARY RESULTS AND SUBJECT FEEDBACK Currently we have the preliminary results for experiment A. There were 24 participants in this study - 12 females and 12 males, ranging from 20 to 35. The valid subject number is 17. For each video, the system calculates the mean value and standard deviation value of the excitement level. It plots on the graph how the wristband color changes over time. Each dot represents the color of each unit time. The data collection is 8 numbers per seconds. Usually the number will remain the same for 7 to 8 dots. For the sake of comparing the effect of regulation by using feedback, it also needs to set some character term value. The first value is the calm value. In this study analysis, the calm value is set by the time percentage of dark blue and purple shown on the wristband, which is the arousal value less than 2/7. The excited value is set by the time percentage of red and orange shown on the wristband, which is the arousal level value greater than 5/7. If the absolute value of the calm state when regulating with feedback minus the calm state when regulating without feedback is greater than 15%, it is a significant difference that means this subject is sensitive to using the feedback to regulate emotions. To establish a baseline, the system also needs to know whether the subjects successfully regulate emotions to a calm state. So if the mean value of regulation tasks is less than the mean value of non-regulation tasks, it says the regulation is successful. Comparing the two groups with feedback and without feedback regulating emotion tasks, there were 11 out of 17 (64.7%) subjects that could do better with visual emotional feedback. In summary, subjects can regulate their emotions better when using the emotional feedback. Generally, the sub- Data mapping to output: α. Discrete; β. Continuous; γ. Rate of change; δ. Threshold; ε. Discrete & continuous TABLE I. MATRIX OF DESIGN ALTERNATIVES Experiment scheme A: α & a Experiment scheme B: δ & c. Interactive interface: a. color and bright; b. size and quantity c. motion d. sound e. smell jects reported that they believe in the feedback to some extent and made an effort to make adjustment. They felt it is a useful indicator and tool to help them be aware of their emotional state and act accordingly. Subject 6 is an example of a feedback/better emotion regulation case that is shown in Fig. 4. When the subject regulated emotion by receiving the emotion feedback, he was aware of his state and was able to adjust his emotional state. So the time of the excited state (green, yellow and orange) did not last very long. While under conditions without feedback, he was not aware of what state he was in. The excited state (yellow, orange and red) lasted quite a long time. After the experiment, each participant was interviewed. First we asked questions concerning each subject s personal emotional character. Second, we asked each participant questions about their feelings of using this device. We instructed them to think like this: imagine there is a mind mirror, what do you expect to see in this mirror, how would you like to make use of it, and what kind of format do you prefer? Three subjects mentioned the design of an emotion watch, which is easy to use and more intuitive. They also mentioned that the wearable product would be very effective, especially compared with the cellphone app. A wearable device that specifically provides emotion feedback and works as a training tool would have many benefits. People would have a better sense of its existence and the capability of the device. One subject proposed a simpler model of the emotion regulation, which is in line with the threshold feedback. There is only one red LED light which lights up when it detects users in a positive emotional state. Users will want to light up the light, to recover to a better emotional state. Another subject proposed nonlinear changing colors. For example, the system stops changing color or possibly uses another feedback format when the excitement level reaches a certain threshold. Some subjects mentioned a positive enhancement strategy. There are two methods to realize regulation instruction that come with the feedback. The first one is instruction on how to get rid of current bad emotions like asking people to take a deep breath and instruct them to do something else to divert attention. The second one is to provide positive stimulus that attracts the attention like playing the light music or showing his or her favorite color or turning on the comedy automatically, etc. Fig. 4. The effect of feedback on human brain signal /15/$ IEEE 940

4 IV. DISCUSSION A. The Augumentation Level and Intervention Level To help address issues related to emotion regulation, two levels of emotion feedback implementation requirements are proposed: the augmentation level and the intervention level. The augmentation level refers to the feedback loop designed to enhance the awareness of people s current emotional state; the intervention level refers to the feedback and interaction loop designed to motivate people s self-regulation effort. The intervention level is based on the augmentation level and both requires emotion recognition. In the subject study, we found visual feedback has a stronger augmentation effect compared to the non-feedback condition; however, merely reflecting truly emotional states seems not to be a good strategy to help people regulate emotion to a desired state. Moreover, due to individual differences, a uniform system cannot adjust the feedback output according to different emotional thresholds, so the regulation is not suitable for all people. For the intervention level, to motivate people s selfregulation effort, displaying the environmental feedback via emotion regulation is important. Based on the process model of emotion regulation [1][14][15], the appraisal theories [16] assume people automatically and implicitly appraise everything. The cognitive evaluation of events forms an essential part of the emotional experience. One strategy to achieve the intervention level requirement is gamification. Game elements (e.g. social relatedness, tangible interaction, and competition [17]) may not only motivate the user to continue an interaction playing but motivate the user to achieve the underlying goals of the game behavioral training or emotion regulation [18]. As perceiving is doing [19], games also give instructions and hints, directing people to accomplish tasks and to modulate their attention. This provides the user a sense of control over their goals and healthcare [20]. Fig. 5 shows an example of a situation where the environment and feedback direct the user s attention and behavior. When it comes to the feedback itself at the intervention level, the feedback stimulus has emotion attributes that in turn impact people s emotional experience and help alter current emotional state in real-time. With continuous interaction with emotional feedback, people gradually reach a desired emotion state. The concept we illustrate here is intelligent emotionfeedback. The feedback does not necessarily reflect real emo- Gamification/ visualization Motivation influences brain & behavior Motivation Fig. 5. The positive effect of environment on emotion tion, rather, it adjusts feedback output to adapt with subjects response to the feedback. The output is based on real-time training sets of previous performance and features; moreover, its calibration solves the trial-to-trial and subject-to-subject variability in brain data of emotions [21]. If the user is nervous about receiving notification of agitated emotional feedback, the feedback should change to a calm condition that helps evoke a desired emotional state. In another case, when people have been trained to use the feedback for a period of time, the feedback scale should be recalibrated again to accommodate interaction more effectively. B. The Emotion Design Framework In this section, we examine the concept of intellligent emotion feedback more thoroughly by adding new components into the emotion feedback design framework. The models are used for abstracting the relationship of components in the system, and for future algorithm/mathematical model and simulation analysis. Apart from existing components, such as physiological signal detection and emotion pattern recognition, the model involves several new components highlighted in blue as shown in Fig. 6: 1) Feature wrapping: Use feature wrapping to calibrate non-linear data range so that the data that map to output representation take into account individual differences. 2) Mapping to output representation: The issue is to determine the most informative aspects of an arousal pattern to provide feedback after the classified emotion pattern. This requires learning and adapting with the regulation performance. Reinforcement learning is one of algorithms good to apply here. 3) Interactive interface representation: The issue is about how to choose the most effective method and media of representing arousal feedback suitable for application scenarios. C. The Intelligent Emotion Algorithm Reinforcement learning and other machine learning techniques have been used in pattern recognition, but have not been extensively applied in the intelligent feedback domain. Reinforcement learning is concerned with how agents ought to take actions in an environment to maximize the notion of cumulative reward. Here, the emotion state S of people is controlled by designing a policy to automatically Regulation Sensory system Output Input Interactive interface representation Output Input Map pin g to ou tput representation Physiological measures Feature wrapping Pattern/emotion brain signals recognition Environment emotion display Fig. 6. The conceptual model of emotion feedback regulation /15/$ IEEE 941

5 TABLE II. PARAMETERS OF REINFORCEMENT LEARNING Parameter State value function V Policy π State s Actions a Rewards r Probability p Discount factor γ 0> γ <1 Definition Function mapping states to real numbers, V: s reals output environment input, π: s a Happy, anger, calm, melancholy, etc., s: s a reals LED color change, rotating windwill speed change User-defined, r: s a r Regulation rate, the conversion between states achieve positive visual feedback, i.e. LED color and tree rotating speed. Mathematically, the goal g of the algorithm is learning to choose actions a that maximize the cumulative reward r: π 2 t ( ) (0) γ (1) γ (2)... γ ( ) t = 0 V s = r + r + r + = r t (1a,b) with the changing state s as time increases s(0) s(1) s(2) s(3)... a(0) a(1) a(2) a(3) r(0) r(1) r(2) r(3) The parameters in reinforcement learning are defined in Table II. To give an intuitive application example of reinforcement learning in regulating emotion, the decision process for an emotion-unstable melancholy patient is modeled as shown in Fig. 7(a). The patient has three types of emotions: exciting, calm, and melancholy. Two actions of lighting red or blue LEDs are given with state transition possibility numbers on arcs. The reward shown inside any state represents the reward received upon entering that state. This result in Fig. 7(b) indicates choosing the best policy for exciting the patient, with which the LED color can be chosen for the patient s current state. The figures below illustrate the simple application of reinforcement learning and emotion regulation based on LED and motion feedback together with the following discussions. 1/2 2/3 s 3: Calm +5 1/2 1/3 s 1: Exciting +10 1/3... 1/ (a) Decision process : red LED : blue LED... s 2: Melancholy +0 Fig. 7. Simple example for emotion-unstable melancholy patient 1/3 s 1: Exciting +10 s 2: Melancholy +0 s 3: Calm +5 (b) LED color choose Fig. 8 (a) shows the conceptual model for this specific design scenario. The key elements of reinforcement learning are shown in Fig. 8 (b): agent (human) interacts continually with its environment (LED); Agent has access to performance measure (EEG signal), and is not told how it should behave; Performance measure (LED color) depends on sequence of actions chosen (determined by a policy); Not everything is known to the agent in advance (human self-regulation). Another typical example is using the windmill rotating speed as shown in Fig. 9. D. Experimental Design and Data Anslysis Issues In the experiment, apart from one control group (emotion regulation without feedback), a yoked control group is necessary. In the yoked control, the subjects will receive the emotion feedback which is actually from another person. This control group study will test if it is in fact the feedback that motivates the emotion regulation effort. Add the examination of the interpretation of the feedback meaning. One idea is to let another person observe the visual emotional feedback and rate it. The other idea is to produce the experiment result right after the experiment and show the regulation comparison to the subject and ask the subjects what they think about this result. The successful regulation rate is not very high. The subjects were not aware of how to regulate emotion in a short period of time in the lab setting. Each subject had his/her own approach to calm down, either effectively or not. So the training session should provide the subjects an instruction list on how to regulate emotions while watching the video. The data analysis as stated in the method part, was to define successful regulation when the calm rate of regulation with feedback is greater than non-regulation tasks. It did not set a threshold on how much it should be less because the subject were not well trained in regulating emotions. The self-regulation: appraisal, automatic regulation self-regulation: appraisal, automatic regulation LED colour 1. EEG is measured by cap 2. EEG signal is mapped to LED color by a policy (a) Conceptual model Fig. 8. Visual feedback Windmill rotating speed 1. EEG is measured by cap 2. EEG signal is mapped to speed by a policy (a) Conceptual model State Reward Fig. 9. Motion feedback State Reward LED EEG signal (b) Principle Windmill EEG signal (b) Principle Action Action /15/$ IEEE 942

6 Device TABLE III. FUNCTION REQUIREMENT Augmentation People can learn to identify their arousal states and interpret real-time emotion feedback. Machine can learn to identify arousal states of people. Intervention level People can learn to regulate their emotions (by their neurofeedback). High level People can improve their working performance by emotion regulation training. This device helps the regulation of emotional states. result is based on the setting. Also, for analysis convenience, it is set to the calm value of each video clip as the percentage time when the excitement level was below 2/7 in which case the wrist band was showing dark blue and purple. There might be other ways to get the calm values, in which cases the result may differ. E. Function Requirement for the Validity of Effectiveness Table III is the summary of the function requirement for the validity of effectiveness when designing for emotion feedback. Other requirements can be used if necessary. Some questions need to be studied in perspective to reach these two levels: How well are the physiologically measurable effects correlated with the actual emotional states of people (with anxiety disorder)? How does feedback reflect people's response to the feedback? Do the changes in feedback provide more impact than the feedback states alone? What is the best way to represent emotions as real-time feedback for people with special needs? V. CONCLUSION AND FUTURE WORK This paper introduces and summarizes three novel components about the design of an emotion feedback system framework then proposes the idea of intelligent emotion feedback and methods to achieve it. This paper proposes a framework and principles that help guide future designs and experiments. This paper summarizes two problems to achieve a positive environmental emotion feedback in two levels: augmentation level and intervention level. The affective Brain-computer Interface technique is used to design the affective neurofeedback and experimental prototype. The preliminary experimental results verify the possibility of using LED visual feedback to enhance the emotion awareness while other methods should be explored to facilitate regulating human emotion. This system would be most beneficial for people who feel anxious and have difficulty regulating their emotions under stressful conditions. Future work will focus on three parts: first, in the product prototype development, more forms of design alternatives would be tested based on the Mapping to Output Representation and Interactive Interface Representation. Second, simulation experiments will be tested for the intelligent emotion feedback performance based on the algorithm model. Moreover, apart from reinforcement learning, other algorithms should be explored to further develop this system. Third, more extensive subject tests including normal people and patients with emotion regulation deficit should be conducted with intelligent emotion feedback algorithm and effectiveness tests. REFERENCES [1] J. J. Gross, "The emerging field of emotion regulation: an integrative review," Review of General Psychology, vol. 2, no. 3, pp , Sep [2] J. J. Gross, "Emotion regulation: Affective, cognitive, and social consequences," Psychophysiology, vol. 39, no. 3, pp , May [3] A. Gyurak et al., "Explicit and implicit emotion regulation: a dualprocess framework," Cognition and Emotion, vol. 25, no. 3, pp , Feb [4] M. Corbetta et al., "Control of goal-directed and stimulus-driven attention in the brain," Nature Reviews Neuroscience, vol. 3, no. 3, pp , Mar [5] J. R. Wolpaw et al., "Brain computer interfaces for communication and control," Clinical Neurophysiology, vol. 113, no. 6, pp , June [6] C. Mühl et al., "Modality-specific affective responses and their implications for affective BCI," 5th Int. Brain-Comput. Interface Conf., Graz, Austria, 2011, pp [7] S. Makeig et al., "First demonstration of a musical emotion BCI," in Affective Computing and Intell. Interaction, Berlin/Heidelberg, Germany, 2011, pp [8] S. Koelstra et al., "Deap: A database for emotion analysis; using physiological signals," IEEE Trans. Affective Computing, vol. 3, pp , Apr [9] X. Kong and Y. Yang, "Measuring emotions in interactive contexts," IEEE 10th Int. Conf. on Comput.-Aided Ind. Design & Conceptual Design, Wenzhou, China, 2009, pp [10] Y. Liu et al., Real-time EEG-based emotion recognition and its application, Trans. Computational Sci. XII, vol. 6670, pp , [11] Y. Hao, et al., "A visual feedback design based on a brain-computer interface to assist users regulate their emotional state," in CHI'14 Extended Abstracts on Factors in Computing Systems ACM, Toronto, Canada, 2014, pp [12] P. Ganapati, Winter Olympics to demo lighting controlled by thoughts, Wired Magazine, Feb 4, [13] J. Mankoff et al., "Heuristic evaluation of ambient displays," in Proc. of SIGCHI Conf. on Factors in Computing Syst., NY, US, 2003, pp [14] Handbook of Emotion Regulation, 1st, the Guilford Press, 2009, pp [15] J. J. Gross, "Antecedent-and response-focused emotion regulation: divergent consequences for experience, expression, and physiology," J. of Personality and Social Psychology, vol. 74, no. 1, pp , Jan [16] K. R. Scherer et al., Appraisal Processes in Emotion: Theory, Methods, Research. NY, US: Oxford University Press, [17] R. Garris et al., "Games, motivation, and learning: A research and practice model," Simulation & Gaming, vol. 33, no. 4, pp , Dec [18] U. Ritterfeld et al., Serious Games: Mechanisms and Effects. NY, US, Routledge, [19] J.-F. Lepage and H. Théoret, "The mirror neuron system: grasping others actions from birth?," Developmental Sci., vol. 10, no. 5, pp , Jul [20] S. McCallum, "Gamification and serious games for personalized health," Stud. Health Technol. Inform., vol. 177, pp , [21] K.-R. Müller et al., "Machine learning for real-time single-trial EEGanalysis: from brain computer interfacing to mental state monitoring," J. of Neuroscience Methods, vol. 167, no. 1, pp , Jan /15/$ IEEE 943