Visualization of emotional rating scales

Transcription

1 Visualization of emotional rating scales Stephanie Greer UC Berkeley CS ABSTRACT Visual ratings scales have the potential to be a powerful tool for recording emotional information and quantifying it over time. However, little is known about how people perceive the emotional content of arbitrary visual elements and furthermore whether the alignment or misalignment of this visual emotional content could effect the use of an emotional rating scale. This study addressed this by way of two human subjects experiments designed to first, collect data on emotional content of visual elements (Experiment 1) and then assess the use of ratings scales based on this information (Experiment 2). This research has implications for the usability and development of computer based emotional rating scales that could be useful tools for selfunderstanding of mood and mood improvement. Author Keywords Emotion; Mood; Rating scales; Color; Motion INTRODUCTION Quantifying emotional subjective states is important for psychological research that seeks to understand how emotional states change overtime or in response to certain stimuli. Given the inherently subjective nature of emotion, it is difficult to develop systems for recording and quantifying emotion that can be used consistently across different individuals or within individuals across time without extensive training or effort. This issue is of particular relevance for a new wave of self-tracking tools that allow users to record their momentary emotional states using their computer or mobile phone in order to better understand their emotional life. This type of tracking is similar to a method called Experience Sample Monitoring which has been useful for psychology research. A few preliminary investigations into self-tracking of mood states in this way indicates that it is a useful tool for self reflection and may additionally benefit people with mental illness related to mood disruption[2,3]. Although these studies indicate that some form of mood tracking is useful for emotional reflection, there is very little research into how the visual elements of the mood rating system can aid in (or detract from) either the accuracy/precision of the ratings or the easy of use of the rating device. This is of particular interest for developing rating systems for mobile phones given their capabilities for rich visual content. An important question when considering the aid of visual information in emotional ratings is whether visual elements have an inherent and consistent emotional tone. Although it is not immediately obvious that this would be the case phrases like feeling blue or bright and cheering indicate there may be common associations between visual information (e.g. color) and emotional states. Recent research from Palmer et. al. [4] supports this conclusion in that they were able to identify consistent and reliable associations between colors and emotional facial expressions as well as emotional labels. Their research indicated that levels of saturation, brightness and, to a lesser extent, hue all showed consistent patterns of emotional mapping. Specifically, higher levels of saturation and brightness were associated with happiness (as opposed to sadness and anger). There was also a similar relationship along the red-green axes of hue as well as the blue-yellow axes. Interestingly, although the hues in this experiment were rated with consistent emotional tone, this did not necessarily align with consistent color preferences that were assessed in a separate study [5]. In other words the interpretation of an emotion in a color (even a positive one) does not necessarily lead to liking that color. Given these consistent associations between attributes of color and attributes of emotion, it may be possible to use these naturally occurring associations to enhance the perception and use of emotional rating scales. Furthermore, there maybe other visual elements (e.g. types of motion or size) that could have similarly consistent emotional

2 Figure 1. Affective circumflex model of emotion. This image was taken from Posner et al. 2005[1]. associations that could also be utilized in emotional rating scales. Before considering how visual information maybe utilize in emotional ratings it is important to fist define the framework for encoding emotional information. There have been a variety of psychological frameworks proposed over the years but perhaps the two most influential are 1) the basic emotion model and 2) the affective circumplex model[1]. The basic emotion model stipulates that there are a set of discrete basic emotions (most commonly defined by these six: happy, sad, fear, anger, disgust, surprise) that can be independently expressed at different intensities. Alternatively, the affective circumplex (Fig. 1) model stipulates that emotional states can very along the dimensions of valence (running from negative to positive) and activation or energy (running from low to high). While both models are actively used, the affective circumplex model is arguably better supported by the biological and neurological characterization of emotion [1] and furthermore has some logistical benefits since it only requires ratings of two easy to understand detentions rather than a variety of emotional names which are not always interpretated similarly across people. While the affective circumplex was developed in order to provide a conceptual framework for understanding emotion (rather than as a survey technique), recent research form the computer human interaction literature has investigated the use of the 2d grid of the valance and arousal axes of the model as a direct interface for emotional recordings [6,7]. Given, the utility of the affective circumplex model along with the context of it s use in related research, this paper will focus on the use of the affective circumplex model for characterizing emotion. Taken together this indicates that there is both a need for more research into the emotional associations of visual elements as well as research into how to apply this information to make more intuitive and visually rich emotional rating scales. This study is designed to address this need by way of two human subjects experiments implemented through online surveys. In Experiement 1 information will be collected on the emotional nature of abstract color patches as well as abstract animations. Experiment 2 will build upon this information by incorporating these visual elements into emotional rating scales and testing the hypothesis that congruent visual-emotional mappings lead to better and easier to use rating scales as compared to incongruent visual-emotional mappings. EXPERIMENT 1 The goal of experiment 1 is to gain further insight into the emotional nature of visual information. This experiment builds off of research from Palmer and colleges[4] by extending similar color patch ratings of basic emotions to ratings based on affective dimensions of valence and energy. Furthermore it adds to this research by assessing the emotional nature of abstract animations. Methods Instructions: At the start of the study (on the first page of the experiment website), participants were given an overview of the experiment procedures as well as basic definitions of positivity/negativity in emotion and high/low energy in emotion. Additionally, each page that required a valence rating included the instructions: Click on the square that you find more emotionally positive (e.g. excited, happy, pleasant or calm) rather than negative (e.g. anxious, sad, upset or boring). If both squares are the same in terms of positivity click 'A and B have the same positivity'. Similarly, each page that required an energy rating included the instructions Click on the square that you find has more energy or fits with energetic emotions (e.g. excited, anxious, elated or angry) rather than less energy (e.g. boring, pleasant, calm or sad). If both

3 squares are the same in terms of energy click 'A and B have the same energy. These instructions were used for both the color ratings and the animation ratings. All valence ratings were always assessed first followed by all of the energy ratings. Everyone rated the same color and animation parings, however the order of the trials was randomized for each participant and the side that the stimuli appeared on (left or right) was also randomized on every trial. Color patch ratings: Participants rated a total 42 color pairs in terms of both valence and energy. On each trial two color patches were shown and the participant had to select the color patch that was either more positive (for valance ratings) or the one that has higher energy (for energy ratings). If the participant did not perceive a difference in valance or energy they could also indicated that using a third option. See Figure 2 for example trials of color ratings. The color stimuli ranged according to four possible hues (red: 345, green: 160, blue: 197 and yellow: 50), three levels of saturation (25%, 50% and 75%) and three levels of brightness (25%, 50% and 75%). The hues were selected based on the Natural Color System as in previous related studies[4,5]. In order to limit the combinatorial number of comparisons, hue comparisons were always assessed with the same saturation and brightness levels and differences in saturation and brightness were always assessed within the same hue. For example the left of Figure 2 illustrates a hue discrimination trial where saturation and brightness are constant and the right of Figure 2 illustrates a brightness discrimination trial with the blue hue held constant.! Figure 2. Color patch ratings for hue discrimination (left) and brightness discrimination (right). Animation ratings: Participants rated a total 18 animation pairs in terms of both valence and energy. Animations were created in javascript using D3. Each animation consisted of a set of white circles within a white outlined box (see Figure 3). The circles were animated by transitioning from a random starting position to another randomly selected position within the box. The animations/circles could vary on four dimensions: 1) number of circles in the box 2) Spread of the circles away form the middle 3) speed of the transitions and 4) size of the circles. In order to limit the combinatorial number of comparisons, only one dimension was assessed at a time. For example if spread was being assessed (as in Figure 3 right) then number, speed and size would be held constant. Four levels of each dimension were tested.! Figure 3. Animation ratings for spread discrimination (left) and size discrimination (right). Participants: Eight subjects participated in experiment 1. Participants were recruited through e- mails and the class wiki. Participation was anonymous and all parts of the experiment were conducted online. Note that one participants color ratings for energy were not included due to missing data. Data analysis: For each subject, trials were binned according to factors of interest. For color patches, factors of interest included the four hues (from hue comparison trials), saturation (collapsed across hues from saturation comparison trials), and brightness (collapsed across hues from brightness comparison trials). For animations, factors of interest included number, spread, speed and size of the animations. Within each of these bins percentage of each choice (higher levels, lower levels or same level) was calculated for each subject and then the average was taken across participants. Importantly this type of analysis does not allow for detection of nonlinear relationships between visual factors and emotional factors. Given the small number of slices (e.g. only 3 levels of saturation and brightness) taken across each dimension, it would be difficult to make any conclusions about nonlinear trends. Results All of the results from experiment 1 are displayed in Figure 4. First, looking at hue (Figure 4; left), ratings indicated that on average a higher percentage of the green patches (as compared to all other hues) were rated as positive while on average a higher percentage of red patches were rated as less positive

4 Figure 4. Results of experiment 1. Bars indicate the average (across subjects) of the percentage of trials rated in the direction indicated on the x-axes. (as compared to all other colors). In further examination, direct comparisons of green and red hues revealed that green was consistently rated as more positive than red (over all subjects there was only one trial when a red patch was rated more positive than a green patch). For the blue and yellow hues, there was little indication of consistent valence ratings in either the positive or negative direction. Additionally, none of the hues showed consistent energy ratings in either the high energy or low energy direction. Taken together this data indicates that in this sample of subjects green was considered more positive, red was considered less positive and all hues had similar energy levels. Looking next at saturation and brightness (Figure 4; center) both of these elements were highly consistently associated with more positivity with little indication of a difference between these two items. For energy, again both saturation and brightness were rated as associated with higher energy ratings, however, a substantially higher proportion of high saturation patches were rated as high energy as compared to high brightness patches. Taken together this data indicates that both higher saturation and brightness are considered as having more positivity and more energy, however, saturation may be a better indicator of energy as compared to brightness. Finally, looking at the animation elements, each element did seem to have some signal related to valance as well as energy. Specifically, higher number, wider spread, faster speed and smaller size were more likely to be rated as more positive and as having higher energy. While there was signal in both valence and energy, these ratings appeared to be more reliably related to energy than to valence. Furthermore, spread and speed had more consistent ratings (particularly for energy) as compared to number and size. EXPERIMENT 2 The goal of experiment 2 was to assess whether visual elements that have emotional interpretations (as identified in experiment 1) can either enhance emotional rating scales (when emotional rating and visual elements are consistent with each other) or detract from emotional ratings scales (when emotional and visual elements are inconsistent with each other). A second goal of experiment 2 was to evaluate how adding visual animations, which varied along emotional rating scales, compared to more traditional rating scales.

5 Methods Instructions: At the start of the study (on the first page of the experiment website), participants were given an overview of the experiment procedures as well as basic definitions of positivity/negativity in emotion and high/low energy in emotion. This page informed subjects that they would be seeing short video clips and they would asked to rate the emotional content of the video using a few different rating scales. Color based Rating scales: First two grid based rating scales were defined based on both previous methods for mood rating systems (See Mood Map description[7]) as well as the color emotion ratings from experiment 1. Both scales displayed a two dimensional plane with valence (negative to positive) labeled on the x-axes and energy (low to high) labeled on the y-axes as in the affective circumplex (Fig. 1) as well as previous studies[7]. The two scales differed in the way that color gradients varied along the axes (See figure 5). The first scale (grid 1) was designed to be the best aligned with the emotion ratings from experiment 1. Therefore this scale varied in hue along the x-axes (valance axes) from red (negative) to green (positive) and varied in saturation along the y-axes (energy). This is because red-green hue showed the strongest association with valance and saturation showed the strongest association with energy in experiment 1. The second scale (grid 2) was also designed to be the aligned with the emotion ratings form experiment 1 but this time with the weaker elements. Therefore this scale varied in hue along the x-axes (valance axes) from blue (weakly negative) to yellow (weakly positive) and varied in brightness along the y-axes (energy). This is because blue-yellow hue showed the weaker association with Figure5. Color rating grids that show consistent mappings between emotion ratings in experiment 1 and scale labels. Grid 1 has stronger alignment with emotion compared to Grid 2. Figure 6. Color rating grids that show inverted mappings between emotion ratings in experiment 1 and scale labels. valance and brightness showed the weaker association with energy in experiment 1. Importantly, both scales were aligned with the emotional information form experiment 1, however, grid 1 was aligned with the stronger elements while grid 2 was aligned with the weaker elements. In order to gain greater insight into the effects of visual elements that are miss-aligned with the emotional information (and therefore better understand the influence of visual elements on emotional ratings) a second set of grid based scales was used in a separate group of people that used inverted color-emotion parings. These scales were the same as grid 1 and grid 2 except that the colors were inverted along the y-axes and flipped along the x-axes (See figure 6). Animation based Rating scale: In order to investigate the utility of using animation based visual elements for emotional rating scales, another type of rating scale was developed and used along side the grid based scales. The animation based scale (Figure 7) included two sliders that corresponded to valence (above) and energy (below). As the sliders were set, the properties of the animation would change to reflect the emotional rating as determined by experiment 1. In order to reduce the complexity of these stimuli, not all of the animation elements were used. Hue (ranging from red to green) varied only along the valence dimension, speed varied only along the energy dimension and saturation as well as spread varied according to a combination of valence and energy. This meant that the spread and saturation was the highest for states of high positive energy and lowest for states of low negative energy. This demonstrated in figure 7 by looking across the four states of each combination of valence and energy.

6 Figure 7. Examples of four states of the animation rating scales. Videos and trials: In experiment 2 participants viewed and rated eight short video clips using the scales explained above for valence and energy. The videos were taken from a database of 9,700 movie clips that have been rated and ranked in terms valence and energy levels previously using crowd sourcing[8]. From these videos eight were selected such that two came from each of four categories 1) the top 10% of highest energy and lowest valence 2) the top 10% of highest energy and highest valence 3) the top 10% of lowest energy and lowest valence 4) the top 10% of lowest energy and highest valence. On each trial the participant would first see a video clip and then make two back to back ratings of that clip. The first rating would be made using either grid 1 or grid 2 (depending on the trial) and the second rating would always be made using the animation rating scale. Separate groups of participants were used to assess the aligned set of grid scales (i.e. those shown in figure 5) and the inverted set of grid scales (i.e. those shown in figure 6). Trial order was pseudo-randomized across participants. At the end of the all of the video trials, participants were asked to indicate on a ten point scale how easy it was to use each rating scale. Participants: Eleven subjects participated in experiment 2. Six of these subjects were included in the group with the aligned color grid rating scales and five were in the group shown the inverted color grid rating scales. Some, but not all subjects participated in both experiment 1 and 2. Participants were recruited through s and the class wiki. Participation was anonymous and all parts of the experiment were conducted online. Data analysis: first, in order to visualize how the scales were used, individual ratings of each movie across each participant were plotted from each ratings scale and color coded according to movie type (Figure 8; left). The means from each scale were also plotted and analyzed to see if there were any consistent shifts in ratings that could b attributed to the use of the scale (Figure 8; middle). Finally, in order to investigate the combined valance and energy ratings, for each movie rating the distance from the center point was calculated and averaged across movies and participants (Figure 8; right). Since the movie clips were designed to evoke more extreme emotional ratings (according to the aggregate database rating) farther distance from the center might be considered an indication of a more effective use of the scale. Results Results of the rating outcomes from experiment 2 are displayed in Figure 8. Overall the ratings on all of the scales showed high coherence with the expected ratings of the videos based on the video s previously identified valence and energy ranking. This was true both for the individual trial ratings (Figure 8; left) as well as the mean ratings (Figure 8; middle). There was one exception to this which is that the low valence & high energy movies were rated as having less energy than expected. However, this is probably due to differences in rater perception rather than differences in scale usage considering all other categories were correctly rated and all scales showed similar agreement. There were no systematic differences in any of the ratings scales (either qualitatively or quantitatively) that could be identified by the individual ratings or the mean ratings. When looking at the metrics of distance of the ratings from the center, there did seem to be a tendency to rate movies farther from the center using the aligned version of grid 1 (i.e. the grid most consistent with the emotional ratings in experiment 2) as compared to either grid 2 or the inverted versions of the grids. However, it should be noted that this was only a numeric difference and it was not statistically significant (assessed using a t- test of distances).

7 Figure 8. Results of experiment 2. Left plot shows all individual ratings for each movie trial on each scale. The points are color coded according to the type of movie shown (based on previous database ratings). Green=high valance & high energy; purple=high valence & low energy; blue = low valence & low energy; red = high valence & high energy. When considering the difficulty ratings for the rating scales, there was no difference in ratings for grid 1 between the aligned and inverted scales. However, for grid 2, the group with the inverted scale rated this scale as significantly more difficult to use as compared to the group with the well aligned scale (Figure 9). The animation based rating scales were consistently rated as more difficult to use as compared to the grid scales regardless of subject group. Taken together this data indicates that while visual elements of emotional rating scales may not affect the use of the scale in a systematic way (especially when linguistic cues are also used), the visual elements can effect the difficulty of the scale for the user experience. Figure 9. difficulty ratings for all emotional scales. DISCUSSION The data collected in experiment 1 revealed several dimensions of visual elements that have consistent emotional interpretations. Specifically, red vs. green hue varied in terms of valence (red being more negative and green being more positive) while saturation and brightness of color were strongly associated with higher levels of positivity as well as higher levels of emotional energy. Interestingly, hue carried the least amount of consistent emotional information as compared to saturation in brightness. This may be an important consideration for visualizations of sentiment analysis that try to convey emotional words with color information. Often these visualizations will rely heavily on hue (likely because it is a useful tool for encoding categorical information such as categories of emotions). However, this research as well as previous studies of color-emotion associations suggest that the other dimensions of color may be more evocative of emotional tone. Experiment 1 also revealed consistent associations between the animated circle stimuli and both emotional valence and emotional energy. The results for energy are perhaps not surprising given the strong associate between energy and intensity of physical motion in the word. However, the valence results were somewhat unexpected. One possibility is that there was a carry over effect between rating systems and subjects were therefore rating higher positivity and higher energy in a similar way. In

8 follow-up experiments it would be better to introduce more randomization into the order of the ratings as well as the direction of the response (i.e. randomly alternate between asking which animation is more positive? and which animation is more negative? ) in order to mitigate against this bias. In experiment 2 there was little evidence that the coloration of the grids influenced the emotion ratings in a systematic way. There are a few possible explanations for this. The most likely explanation is that the language based processing of the grid given by the axes labels (i.e. the direct mapping to high/low valance and energy on the grid) was too string to be influenced by the more subtle cues given by the color information. If the axes labels had been removed or if the instructions indicated that color should be taken into account when making the ratings then this would have lead to much more dramatic differences. It is also possible that there was an effect of the coloration of the grid on the ratings but there were simply too few subjects to detect the small effect reliably. There was however some indication that the coloration of the grid effected it s use since the inverted version of grid 2 was rated as significantly harder to use that the aligned version of this scale. Reaction times (i.e. the duration of time need to make the ratings) on each rating scale would have also been an interesting measure of usability in this experiment. Future research could investigate these issues in more detail to try and further understand the effects of visual color elements on emotional ratings. While the animated scale was used equally effectively as the grid rating systems for encoding the information, it was consistently rated as being more difficult to use than the grid ratings. This is most likely due to the unfamiliarity of this type of stimulus while the 2-D axes system was likely very familiar to the participants of this experiment. There may have also been more individual differences in how people perceived these stimuli that could have lead to difficulty of use. It would be interesting to consider how this system would compare in a population that was less familiar with axes systems or in an experiment were the animation elements could be customized to individual emotional perception. FUTURE WORK Building on experiment 1, future work in this area could explore the emotional nature of a wider set of animations as well as objects or cartoons. This could include variations in the type of motion (e.g. linear vs. waving) or in how the physics of the elements interact (e.g. whether the objects would collide or move passively). This could help further the understanding of how visual elements are perceived emotionally and therefore how to display emotional content in a visual way. One application of the visual rating scales assessed in experiment 2 is for use in mood-tracking applications where people can repeatedly record their own mood over time. In the experiment presented here, each person only used the rating scales a few times and they did not rate the same stimuli repeatedly. It would be interesting to know whether bigger differences between scales would arise when the scales were used repeatedly in this way. Another interesting line of work would be to assess whether differences in the visual nature of rating scales (including animated scales) lead to differences in ability to remember or reflect on mood. For example if the scale shows a bright saturated green as reflecting a happy state, it may be easier to recall and associate that state with the scale than if the scale displayed dark, unsaturated red as a happy state. Finally, emotionally aligned colored or animated scales could also be useful when reporting the mood data back to the user, for example, rather than showing the user a mood history displayed directly on the axes, the history could be displayed through a series of color patches which may make it easier to understand the gist of mood patterns. ACKNOWLEDGMENTS I would like to thank everyone who participated in my experiment for making this study possible. I would also like to thank Pablo Paredes for sharing a series of references from the Human Computer Interaction literature. REFERENCES 1. Posner J, Russell JA, Peterson BS (2005) The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev Psychopathol 17:

9 2. Drake G, Csipke E, Wykes T (2013) Assessing your mood online: acceptability and use of Moodscope. Psychol Med 43: van der Krieke L, Wunderink L, Emerencia AC, de Jonge P, Sytema S (2013) E-Mental Health Self- Management for Psychotic Disorders: State of the Art and Future Perspectives. Psychiatr Serv. 4. Palmer SE, Schloss KB, Xu Z, Prado-Leon LR (2013) Music-color associations are mediated by emotion. Proc Natl Acad Sci U S A 110: Palmer SE, Schloss KB (2010) An ecological valence theory of human color preference. Proc Natl Acad Sci U S A 107: Daniel McDuff AK, Ashish Kapoor, Asta Roseway, and Mary Czerwinski (2012) AffectAura: An Intelligent System for Emotional Memory. ACM. 7. Margaret E Morris1 PQK, MFA; Todd K Leen2, PhD; Ethan E Gorenstein3, PhD; Farzin Guilak1, MS; Michael Labhard1, MD; William Deleeuw1, B (2010) Mobile Therapy: Case Study Evaluations of a Cell Phone Application for Emotional Self- Awareness. J Med Internet Res Y. Baveye J-NB, E. Dellandréa, L. Chen and C. Chamaret (2013) A Large Video Database for Computational Models of Induced Emotion. Affective Computing and Intelligent Interaction