Emotionally Augmented Storytelling Agent

Transcription

1 Emotionally Augmented Storytelling Agent The Effects of Dimensional Emotion Modeling for Agent Behavior Control Sangyoon Lee 1(&), Andrew E. Johnson 2, Jason Leigh 2, Luc Renambot 2, Steve Jones 3, and Barbara Di Eugenio 2 1 Connecticut College, New London, CT, USA james.lee@conncoll.edu 2 College of Engineering, University of Illinois at Chicago, Chicago, IL, USA {ajohnson,spiff,renambot,bdieugen}@uic.edu 3 College of Liberal Arts and Sciences, University of Illinois at Chicago, Chicago, IL, USA sjones@uic.edu Abstract. The study presented in this paper focuses on a dimensional theory to augment agent nonverbal behavior including emotional facial expression and head gestures to evaluate subtle differences in fine-grained conditions in the context of emotional storytelling. The result of a user study in which participants rated perceived naturalness for seven different conditions showed significantly higher preference for the augmented facial expression whereas the head gesture model received mixed ratings: significant preference in high arousal cases (happy) but not significant in low arousal cases (sad). Keywords: Virtual humans Intelligent agents Dimensional theory PAD 1 Introduction Recently, many research efforts toward natural and affective virtual agent capabilities have been undertaken [1]. Several studies have shown that a naturally behaving agent capable of interaction becomes more effective. For example Burgoon and Hoobler pointed out that more than 50 % of social meaning in our communication is carried via nonverbal cues [2]. Although positive effects of emotional agent behavior have been established, it is still challenging to design a computational model for such behaviors. Dimensional theories of emotion have motivated many prior studies to build a computational model to assess an agent s emotional state and provide a framework to map human emotion in one or more dimensions. To this end the Pleasure-Arousal-Dominance (PAD) model [3] provides well-formed computational foundations. The significance of the PAD model is that continuous changes in emotional state can support a smooth transition between discrete emotion eliciting stimuli. In particular, we investigate the capability of a dimensional model to control agent behavior in the context of storytelling. Given our interests, we have limited our exploration to evaluate the agent s ability to augment emotional behavior for stories Springer International Publishing Switzerland 2015 W.-P. Brinkman et al. (Eds.): IVA 2015, LNAI 9238, pp , DOI: / _53

2 484 S. Lee et al. that dominantly elicit two categorical emotions, happy and sad. The presented model is compared to our previous system that shows a fixed intensity of facial expression and head gesture. 2 Approach: System Model The nonverbal behavior processor (NVBP) in the system is composed of three components: the Affect Analyzer, the Emotion Processor, and the Gesture Processor. The Affect Analyzer takes an utterance and executes part-of-speech tagging, affect extraction for words based on WordNet-Affect, and structural analysis to revise word level affect. The gesture predictor uses basic rules adapted from McClave [4] and a data-driven model trained with the SEMAINE video corpus to generate head gesture events. The NVB events are encoded for each word, fed to the speech synthesizer, and the synthesizer then synchronizes NVB events as an agent speaks. The Emotion Processor takes an affect type event, then processes affect to update the agent s mood with the PAD model; finally, it computes the intensity of an emotional facial expression. PAD vector values are adapted from prior work [5], and the layered model is revised based on Gebhard and Kipp [6]. This final step is an augmentation that may increase or decrease the intensity of facial expression. The computational equations are shown below: X E evaluated ¼ e þ c t kk e T þ c a Eactive M updated ¼ T þ E evaluated þ X E active k E intensity ¼ e intensity E evaluatedk þ RðeÞ kk e ð1þ ð2þ ð3þ Equation (1) describes how an initial PAD vector, e, is evaluated. c t and c a are constant factors toward trait and active emotion. The length of e is denoted by e. Trait vector T and currently active emotions E active, and mood M, are applied to skew the initial vector. Equation (2) shows how the mood is updated based on newly added affect. It is a sum of trait, evaluated new emotion, and all active emotions in PAD. Equation (3) presents augmentation of new emotion to be sent to the facial expression synthesizer. The auxiliary effect, R(e), accounts for repetition of the similar emotion. When a gesture type event arrives from the speech synthesizer, the Gesture Processor refers to the arousal value of the current mood to compute the intensity of head gesture. For example, when one feels very depressed (low arousal), one may not show much head movement, whereas one may show intense gestures when the arousal value is high. The augmentation ranges from 25 % to 200 % of the baseline intensity.

3 Emotionally Augmented Storytelling Agent Evaluation To evaluate whether the presented model can increase the perceived naturalness of emotional facial expressions and head gestures, we designed seven conditions (Table 1). The intensity of facial expression and head gesture ranges from 0.0 to 1.0 as a weight value in our system to map it from no expression/gesture to the strongest expression/gesture. We chose a half intensity for control conditions to avoid extreme cases: intensity 0.0 or 1.0. We assume this as a moderate level of behavior for the system that does not have a computational model to reflect an agent s emotional state dynamically on agent s emotional behavior. A 10 % fixed intensity of a head gesture condition is added to compare very subtle head gesture with the presented model. The study included three stories from psychological literature and blogs. Each story is a personal experience in the past [7, 8]. The system detected 14 emotion-eliciting words/phrases, 12 happy and 2 sad, and parsed 32 head gestures, 10 nods and 22 shakes in the first story (happy story). For the second story (sad story) a total of 12 emotion-eliciting words/phrases including 10 sad, 1 fear, and 1 surprise and a total of 44 head gestures, 23 nods and 21 shakes were processed. The results were fed to the system to create videos of an agent. The audio track was excluded to avoid the bias caused by audio cues. Instead, subtitles were embedded in the videos. A total of 24 participants (18 male, mean age with SD 11.15) were recruited in this study. The study was composed of three sessions: one training session and two test sessions. During each session, participants (1) read one of the written stories, (2) selected a primary emotion among the six basic emotions that they might feel if they were telling the story, (3) drew a graph depicting the intensity of the primary emotion, and (4) reviewed a video showing the seven conditions and rated them (Fig. 1). The rating was measured as a number of check marks that a participant gave to each condition when one or more agents seemed more realistic than others. Rating Results. We performed the Wilcoxon signed-rank test to compare ratings between variables: combined cross-variable analysis for model vs. control group in both facial expression and head gesture. The model combined condition in story I for both facial expression (C1, C2, and C3) and head gesture (C1 and C4) was most preferred; its differences were significant (p <.05) except in the comparison with Control IV(C3 and C6) case for head gesture. The model combined condition in story II for facial expression (C1, C2, and C3) was significantly higher than control groups (p <.05), whereas the head gesture case (C1 and C4) was not significant. Facial expression Table 1. Combinational behavior model conditions used in the study Head gesture Model (varied Control III (fixed half Model (varied C1 C2 C3 Control I (fixed half C4 C5 C6 Control II (Control I w/abrupt transition) n/a C7 n/a Control IV (fixed 10 %

4 486 S. Lee et al. Fig. 1. A participant is reviewing virtual agents video on a large display system. Seven identical agents are telling the same story in sync with different conditions. In summary, the presented model that can emotionally augment nonverbal behavior received significantly higher preference than the control conditions for facial expression. However, the head gesture model showed mixed results. 4 Conclusions We designed a PAD space model to compute a virtual agent s emotional state to drive its behavior including facial expressions and head gestures. The presented model uses a shallow parsing of a surface text to extract emotion-eliciting stimuli and generates augmented behavior according to the agent s mood. The result of a user study confirmed that the facial expression model received a significantly higher rating than all control conditions that use fixed intensity of facial expression. However, our head gesture model showed mixed results. We found significant preference for the head gesture model in high arousal cases whereas we obtained meaningful but not significant ratings in low arousal cases. The difficulty of recognizing subtle gestures in low arousal cases may have contributed to the inconsistent rating as some participants noted naturalness of lessened head gesture in a sad/depressed mood. However, more focused future study is required to verify this interpretation. References 1. Lee, J., Marsella, S.C.: Predicting speaker head nods and the effects of affective information. IEEE Trans. Multimedia 12(6), (2010) 2. Burgoon, J.K., Hoobler, G.D.: Nonverbal Signals. In: Knapp, M.L., Daly, J.A. (eds.) Handbook of interpersonal communication, pp SAGE Publications, Inc., Thousand Oaks (2002) 3. Mehrabian, A.: Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in Temperament. Curr. Psychol. 14, (1996) 4. McClave, E.Z.: Linguistic functions of head movements in the context of speech. J. Pragmat. 32(7), (2000) 5. Zhang, S., Wu, Z., Meng, H.M., Cai, L.: Facial expression synthesis based on emotion dimensions for affective talking avatar. In: Nishida, T., Jain, L.C., Faucher, C. (eds.) Modeling Machine Emotions for Realizing Intelligence. SIST, vol. 1, pp Springer, Heidelberg (2010)

5 Emotionally Augmented Storytelling Agent Gebhard, P., Kipp, K.H.: Are computer-generated emotions and moods plausible to humans? In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA LNCS (LNAI), vol. 4133, pp Springer, Heidelberg (2006) 7. DiMarco, L.: My minxy mouse encounter: Disney World. favorite-travel-memories 8. Ossorio, P.G.: Clinical topics: A seminar in Descriptive Psychology. Linguistic Research Institute, Whittier (1976)