Emotional Responses to Thermally Augmented Audiovisual Stimuli

Transcription

1 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Emotional Responses to Thermally Augmented Audiovisual Stimuli Stefan Greiner Quality & Usability Lab Berlin Institute of Technology, Berlin Jan-Niklas Antons Quality & Usability Lab Berlin Institute of Technology, Berlin Ina Wechsung Quality & Usability Lab Berlin Institute of Technology, Berlin Abstract Physiological reactions to certain emotions make us feel warm or cold, thus there are strong bounds between emotions and temperature. However, within HCI research, using thermal information to evoke emotion related experiences is still rare. For the research presented in this paper a kidney belt with thermal actuators was developed in order to investigate the emotional responses to thermally augmented audiovisual stimuli (movie clips). Participants then continuously evaluated their emotional state via a tablet application on the dimensions arousal and valence. The results indicate that thermal stimulation has significant effects on the emotional experience of subjects. While emotions with negative valence are more prone to be affected on the arousal dimension, emotions with positive valence are significantly influenced on the valence dimension. I. INTRODUCTION Emotions are experienced not only in the mind, but also in the body [1]. This is important since emotions are not only the feelings we have when we are sad, happy or in love but also serve our organism with quick programs to react in survival essential moments. Studies in the field of Human Computer Interaction that map the relationship between thermal stimuli and emotions are rare although thermal stimulation could be seen as a novel and appealing way to convey information and evoke affective experiences [2], [3]. Not only trends like multidimensional cinema environments could benefit from such a mapping but also more subtle human-computer interaction scenarios like basic thermal feedback for mobile or stationary computing applications. With the development and emergence of smart and wearable devices such as smart watches and the like, also physiological measures like pulse and skin conductivity are already in the possible scope to gather over time. With this data, emotional states of the user can be tracked and for example be transmitted to a desktop computer or a smartphone in order to give appropriate thermal feedback attuned with the emotion of the user, thus creating a new way of emotional user experience. The study at hand takes first steps towards a systematic emotion temperature mapping. To measure thermal effects onto a set of four emotions, subjects wore a modified kidney belt with thermal actuators sewed in, that was synchronized with a set of emotional movie clips. Via an app, they then continuously evaluated the emotional experience while they were watching the clips. The following sections describe the applied set of methods in more detail. II. METHODS 1) Stimulus Material: The stimulus material presented to the subjects should evoke emotions. There is a semi standardized set of commercially available movies, validated by several other authors [4] [7]. Four classified emotions (anger, sadness, happiness and fear) according to the Russell Circumplex Model of Affect [8] are represented by the list of movie clips shown in Table I. Previous research has identified the movies as a valid set of stimuli for inducing the four emotions [7]. 2) Thermal Harness: In order to induce thermal stimuli during the movie clips a thermal harness was developed. Figure 1 shows the device. Three Peltier actuators were used to provide thermal stimuli. The cables, supplying the Peltier substrates with direct current were isolated. To additionally ensure the safety of the subjects a plexiglas protection around the elements were cut into shape. This protection also served as a handy solution to affix the elements by sewing them onto the harness. Fig. 1. Thermal Harness to induce discrete thermal stimuli The Peltier elements were mounted on a modified kidney belt close to the body, ensuring direct skin contact without causing any discomfort from the device itself. Since spatial acuity of the lower back is very low, using three Peltier devices mounted across the kidney belt has the advantage that the three separate stimuli create a band of heat. This in turn helps to create a unified experience, leading to suitable method for inducing emotions. To stabilize and control the temperature of each Peltier Device, a LM335 temperature sensor is sticked with thermally conductive glue on each of the devices backside. An Arduino hardware interface was used to control the Peltier elements and to synchronize the thermal stimuli with correspondent scenes within the movie clips. Greiner, S.; Antons, J.-N.; Wechsung, I., "Emotional responses to thermally augmented audiovisual stimuli," Quality of Multimedia Experience (QoMEX), 2015 Seventh International Workshop on, vol., no., pp.1,6, May 2015 doi: /QoMEX

2 Three thermal conditions were used warm (+ 6 C from actual skin temperature) cold (- 6 C from actual skin temperature) neutral (control condition = no presentation of thermal stimulus) Movie Clip 1. Stimulus 2. Stimulus 3. Stimulus Cry Freedom (Anger) Gandhi (Anger) Witness (Anger) Officer2 (Happiness) Harry & Sally (Happiness) On golden Pond (Happiness) Killing Fields (Sadness) Officer1 (Sadness) Champ (Sadness) Shining (Fear) Silence of the lambs (Fear) TABLE I. Halloween (Fear) Stand still! (first sentence of police officer when peaceful demonstrants approaching police barrier) Police officer is very arroused and orders to march an indian off. Teenager provokes man putting ice cream in his face Man surprises woman in fabric hall conversation on fake orgasm Woman is dancing happily besides a lake Farewell situation under tears Man sees his friend hanged up Last sentence of dying father to his son Child plays in lobby, when tennis ball is approaching Death's head hawkmoth is landing on some mandrels Man get shot by police officer Rage attack of police officer (because indian keeps burning his identity papers) Man is very angry and steps out off his coach Jealously looking girl Woman starts to mimic an orgasm Woman surveillancing the dancing woman Man leaves under rain and is worried about his kids Man realizes that his friend is dead Boy cries because his father just died Boy follows the direction of the tennis ball FBI agent screams at suspect to put his hands up. Last scene. Police officer aims with gun at little boy and shoots him down. Angry police officer shouts at indian that he should stop burning his ID (after teenager throws away his hat) man gets very angry (Happy End) Man carries woman Elderly woman looks pleased Enjoyment of reunion of two women Man cries under rain Man cries with his dead friend Boy tries to find support from people standing around Boy comes to an open door and asks if his mum is in there Help screams in the cellar Woman climbs up stairs Woman opens door Dead body is approaching woman PRESENTED MOVIE CLIPS WITH CORRESPONDING THERMAL STIMULUS TIMES There were twelve different movie clips, where respectively three represent one common emotion. That made it possible to present each thermal condition to all four emotions. However, since the story lines of the movie clips have different chronological sequences and the respective emotions were not evenly represented during an entire movie clip, a way to make the clips comparable had to be found. Thus, in terms of comparability the decision was made to present three discrete thermal stimuli at three emotional exposed moments within the movie clips (see Table I for thermal stimulus times). Each of the three thermal stimuli, three seconds long, was kept at the same condition (either warm or cold or neutral) within one movie clip. A. Continuous Emotion Measurement App The idea was to develop an application which allows participants a continuous evaluation of their emotional state by moving their finger on a tablet screen. Having the tablet fixed on the subjects chair, it represents an easy way to give continuous ratings on two dimensions. To achieve an intuitive use, that doesnt require much of eye and head movements towards the tablet, the axes of the two dimensions were masked with a small strip of adhesive tape (see Figure 2) in order to give the user haptic feedback regarding the current input area. The X axis of the app represented the valence rating, with the endpoints unangenehm (engl. unpleasant ) at the left end and angenehm (engl. pleasant ) at the right end. The Y axis represented the arousal dimension with the endpoints Fig. 2. Topview onto the tablet/app aufgeregt (engl. excited ) at the upper end and ruhig (engl. calm ) at the lower end. The endpoints, although explained in detail with the handouts given before the experiment were displayed at the endpoints of each dimension. B. Participants Part of the participants were recruited with the help of an online recruiting tool from our University, the other part were recruited using online announcements in diverse platforms. As compensation for their willingness to participate, people got either some sweets or course credit. Participants were required to possess normal sight and hearing abilities; preferably they should have experience with touch based devices. Since the tablet for continuous measurement was mounted at the right armrest of the participants chair, only right handed people were considered for the experiment. In total, 28 people took part at the experiment, of which 13 were male and 15 were female. The average age was 27 +/- 3,63 years. C. Procedure After welcoming the participants, they were shortly introduced to the context of the study and sat down at the table, next to the experimenters table. As soon as they signed the informed consent, possible questions due to the introductory questionnaire were clarified. After that their emotional state was assessed with the Self-Assessment-Manikin-Scales [9]. Subsequent to that, the participants were led to the chair where they first belt on the thermal harness. Since the device was belt on with direct skin contact, this was done by the subjects themselves to avoid any discomfort. Then the correct position of the harness was checked by the investigator. As soon as subjects felt comfortable to begin, light was switched off and the presentation of the thermally augmented movie clips started.

3 Training To get subjects used to the handling of the app, the first movie clip was only for training purposes. This training clip was presented without thermal stimulation. After watching the training clip and having interacted with the app, possible questions were clarified and subjects were instructed that with the next movie the actual evaluation starts. Main Experiment - Presentation of Movie Clips The order in which the movie clips were presented to the subjects was randomized. But it was ensured that movie clips showing the same emotion were not presented subsequently. After each movie clip, participants were asked to answer a few questions regarding the emotion felt during the movie clip and further questions regarding the thermal stimulation such as Please decribe the experienced emotion during the just watched clip with a few words and Would you rather connect cold or warmth with the just mentioned emotion?. Once they had answered the questions they were presented with a neutral movie clip (train ride from train conductors perspective Bahnmitfahrt ). Subjects were instructed to relax while watching these neutral clips and no measurements were taken. End of Experiment After watching and evaluating all movie clips, subjects removed the thermal harness by themselves. After that there was time for the participants to put additional questions onto the experiment. Subsequently to that, subjects were thanked for the participation and were led to the exit. Fig. 3. Mean values over entire movie clips with different thermal conditions III. RESULTS Each subject was presented with all thermal conditions combined with all emotions, that made it possible to run a repeated measures analysis of variance. For the analysis the dependent variable was either the mean value from the arousal or from the valence measurements taken while they watched the videos. The independent variables were the thermal conditions and emotions. Sidak corrected post hoc test were calculated for the pairwise comparison of the emotions. A. Discrimination of emotions Figure 3 shows the mean values for each condition over the entire movie clips. The arrangement of the four emotions within the arousal/valence dimensions is in accordance with Russel s Circumplex Model of Affect [8]. Thermal conditions of each emotion have slightly different mean values but are still grouped together. Corresponding mean values and standard deviation for each emotion and thermal condition are listed in Table II. The main effect between the emotions is significant for valence, Fvalence(3, 151) = , p <.001 as well as for arousal, Farousal (3, 152) = 19.08, p <.001. The Sidak corrected post hoc tests for main effects shown in Table II present the significance levels of arousal and valence onto the four emotions. Significant results are marked in bold. TABLE II. P-VALUES OF PAIRED COMPARISONS (SIDAK-CORRECTED) OF AROUSAL AND VALENCE RATINGS OVER ENTIRE MOVIE CLIPS. SIGNIFICANT DIFFERENCES ARE MARKED IN BOLD (SIGNIFICANT ON 1% LEVEL) mean value of the seven seconds after the first thermal stimulus from the ones seven seconds before the first thermal stimulus. Thus, the interpretation of the position of the barplots is that if the mean value is positive on the y axis, there is a tendency to pleasant (valence) and calm (arousal) states and if negative there is a tendency to unpleasant (valence) and excited (arousal) evaluations. All of the conditions were then tested within a variance analysis. The plots show significant results with a parentheses and a star above corresponding conditions. B. Subjective Evaluation The Figures 4-6 show the differences of the time intervals seven seconds directly before and after the first thermal stimulus. The differences were calculated by subtracting the

4 Anger Figure 4 shows the bar plots of each thermal condition on the two dimensions for anger. Fig. 5. Differences in ratings for happiness seven seconds before and after first thermal stimulus Fig. 4. Differences in ratings for anger seven seconds before and after first thermal stimulus Both, the valence (Mcold = -8.59, SDcold = 21.82, Mwarm = -.40, SDwarm = 19.10, Mneutral = , SDneutral = 37.02) and arousal mean values (Mcold = -1.15, SDcold = 11.10, Mwarm = -5.08, SDwarm = 11.06, Mneutral = , SDneutral = 24.10) stay in the negative area. For valence no main effect was observed Fvalence(2, 47.88) = 1.10, p =.343. For the arousal ratings there is a significant main effect, Farousal(2, 38.12) = 4.41, p =.019. Post-hoc test with paired comparison showed that the cold condition was significantly different from the neutral condition (p=0.017). All of the thermal conditions remain in the negative area. That means, all conditions show a tendency to excitement. The cold condition has the effect that people evaluate and experience the situation less exciting than without a thermal stimulus. Sadness Figure 6 shows the bar plots of each thermal condition on the two dimensions for sadness. Happiness Figure 5 shows the bar plots of each thermal condition on the two dimensions for happiness. Starting with the valence ratings for happiness, the results show that the cold condition compared to the neutral and warm condition have an opposite sign. Whereas in the cold condition, subjects tended to experience an unpleasant situation (Mcold = -8.68, SDcold = 29.54) the mean values of the warm (Mwarm = 5.38, SDwarm = 16.63) and the neutral (neutral = 11.64, SDneutral = 15.85) condition both show a tendency to pleasantness. The arousal ratings, except for the neutral condition (neutral = 1.87, SDneutral = 24.67), remain in the negative area (Mcold = -8.68, SDcold = 29.54, Mwarm = 5.38, SDwarm = 16.63). The ANOVA results indicate a main effect on the valence scale (Fvalence(2, 41.44) = 6.97, p =.002). The post-hoc tests showed that the cold condition resulted in significantly lower valence values compared to the warm condition (p=0.014) and the neutral condition (p=0.002). For arousal, no significant differences were observed (Farousal(2, 39.68) = 1.46, p =.244). Fig. 6. Differences in ratings for sadness seven seconds before and after first thermal stimulus All of the mean values on the valence dimension show a negative difference rating (Mcold = , SDcold = 17.23, Mwarm = -6.41, SDwarm = 21.60, Mneutral = -9.50, SDneutral = 17.14). For arousal the mean value within the cold condition (Mcold = 4.21, SDcold = 16.16) has a positive sign while in the warm and neutral conditions both mean values have a negative sign (Mwarm = , SDwarm = 14.99, Mneutral = -8.29, SDneutral = 25.11) For valence no main effect between the thermal condition was observed (Fvalence(2, 57) =.41, p <.668). For arousal the analysis indicated a significant effect

5 (Farousal(2, 58.97) = 3.48, p =.037). The post hoc tests showed a significant difference between the cold and neutral condition (p=0.0497). As stated before, that difference also bears a change from positive sign of the cold condition (tendency to calmness) to a negative sign in the control condition (tendency to excitement). Hence, like with the happiness ratings, the thermal stimulus converts the evaluation to the opposite direction in this case for the arousal. IV. DISCUSSION Discrimination of Emotions To discuss thermal augmentation of the movie clips or respectively the emotions intended to induce, it is important to first ensure that the emotion induction actually took place within the subjects. The comparison of the mean values over the entire movie clips in Figure 3 shows, that except for anger and fear there is a high degree of discrimination of the arousal/valence ratings. Whereas fear in general has a higher amplitude in the overall rating, anger has still similar arousal/valence relations in the rating. To get a clearer picture a variance analysis was calculated to see if there are significant differences within the emotions and corresponding arousal and valence values. Table II shows that some of the emotions have non significant differences either at the arousal or the valence scale. However each emotion has at least one dimension where it is significantly different to the rest of the emotions. For example even if the mean arousal value of sadness and happiness is not significantly different from each other, the relation and combination with the valence value makes the unique characterization and discrimination of the emotion possible. Subjective Evaluation Except fear, all other emotions show significant differences in their arousal or valence ratings provoked through thermal stimulation. For none of the emotions the thermal stimuli had an effect on both dimensions, it was either on the arousal or on the valence ratings. Thus, anger was more prone on the arousal rating showing the effect that with cold stimulus people rate the anger situation with a pronounced tendency to calmness. Happiness was clearly more affected on the valence rating with the thermal augmentation. Here again, the cold condition had significant effects, one time compared to the warm stimuli and to the control condition. Both times, cold stimuli led subjects to experience the situation in a more unpleasant way. Within the sadness movie clips, again the cold condition had a significant effect. Within this emotion, the arousal ratings were significantly affected. The cold compared to the neutral control condition had the effect that subjects experienced the situation with a tendency to calmness compared with the neutral condition where they felt more excited. Noticeable is that the cold stimuli seem to have a bigger effect in general. There is just one result where the warm condition is significantly different to another thermal condition (happiness), but then only compared to the cold stimuli and not to the control condition. This on one hand could have a psychophysiological explanation, since the ratio of cold to warm receptors is 30:1 and cold fibers have a higher conduction velocity but has also be discussed in connection with the corresponding emotional experience. The external thermal stimulation might also have a compensation effect in that subjects attributed their inner physiological homoestatic processes in contrary objection to the external presented thermal stimulus. That means, that if subjects felt warm because of the rising pulse or other thermoregulatory processes that cause heat, they interpreted the external thermal stimulus as either a compensation (in case of the cold stimuli) or as an intensifier of the inner corporal thermal feeling (in case of warm stimuli). That would actually explain why in general the cold stimulus led subjects to a calmer state and not to a more aroused one. This is in accordance with the report of one subject: She stated that even when she got warm stimuli, she would rather connect a cold feeling to it (fear) and another subject that reported that the cold condition actually felt comfortable since it compensated the hot feeling coming from inside. V. CONCLUSION The results of the study at hand show that thermal augmentation of audiovisual stimuli has effects on the emotional experience. Within the subjective evaluation, especially the cold stimuli had a significant influence onto the emotional responses. While emotions with negative valence (anger and sadness) were more affected on the arousal scale, happiness was significantly influenced by cold stimuli on the valence dimension. The continuous evaluation of emotional experience with the help of the tablet application developed within the study at hand stands for a valid method. Subjects reported that it was intuitive to use and didn t distract from the actual target stimuli. Emotion induction via movie clips always comes with a strong fluctuation regarding the movie plot and not evenly distributed emotional situations. To balance and get better comparability over different plots, it is crucial to define key moments where thermal augmentation takes place. However, in terms of future application of thermal augmentation in cinemas for example, it would also be interesting to further investigate the results at hand with continuous thermal stimulation. REFERENCES [1] W. James, What is an Emotion? Wilder Publications, [2] H. Fujita and K. Nishimoto, Lovelet: a heartwarming communication tool for intimate people by constantly conveying situation data, in CHI 04 extended abstracts on Human factors in computing systems. ACM, 2004, pp [3] K. Salminen, V. Surakka, J. Raisamo, J. Lylykangas, J. Pystynen, R. Raisamo, K. Mäkelä, and T. Ahmaniemi, Emotional responses to thermal stimuli, in Proceedings of the 13th international conference on multimodal interfaces. ACM, 2011, pp [4] P. Philippot, Inducing and assessing differentiated emotion-feeling states in the laboratory, Cognition & Emotion, vol. 7, no. 2, pp , [5] J. J. Gross and R. W. Levenson, Emotion elicitation using films, Cognition & Emotion, vol. 9, no. 1, pp , [6] D. Hagemann, E. Naumann, S. Maier, G. Becker, A. Lürken, and D. Bartussek, The assessment of affective reactivity using films: Validity, reliability and sex differences, Personality and Individual Differences, vol. 26, no. 4, pp , 1999.

6 [7] R. Schleicher, Emotionen, kontinuierliche selbstbewertung und psychophysiologische inklusive okulomotorische veränderungen, Ph.D. dissertation, [8] J. A. Russell and L. F. Barrett, Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant. Journal of personality and social psychology, vol. 76, no. 5, p. 805, [9] M. M. Bradley and P. J. Lang, Measuring emotion: the self-assessment manikin and the semantic differential, Journal of behavior therapy and experimental psychiatry, vol. 25, no. 1, pp , 1994.