Perceived Realism 1 Running Head: PERCEIVED REALISM OF DYNAMIC FACIAL EXPRESSIONS Perceived Realism of Dynamic Facial Expressions of Emotion Optimal Durations for the Presentation of Emotional Onsets and Offsets Holger Hoffmann, Harald C. Traue, Franziska Bachmayr and Henrik Kessler University Clinic for Psychosomatic Medicine and Psychotherapy, Medical Psychology Section, Ulm, Germany Correspondence concerning this article should be addressed to Holger Hoffmann, University Clinic for Psychosomatic Medicine and Psychotherapy, Medical Psychology Section, Am Hochsträß 8, D-89081 Ulm, Germany, or on e-mail to holger.hoffmann@uni-ulm.de. This research was supported in part by grants from the Transregional Collaborative Research Centre SFB/TRR 62 "Companion-Technology for Cognitive Technical Systems" funded by the German Research Foundation (DFG).
Perceived Realism 2 Abstract The presentation of facial displays of emotions is an important method in emotion recognition studies in various basic and applied settings. This study intends to make a methodological contribution and investigates the perceived realism of dynamic facial expressions for six emotions (fear, sadness, anger, happiness, disgust, and surprise). We presented dynamic displays of faces evolving from a neutral to an emotional expression (onsets) and faces evolving from an emotional expression to a neutral one (offsets). Participants rated the perceived realism of stimuli of different durations (240 3040 ms) and adjusted the duration of each sequence until they perceived it as maximally realistic. Durations perceived as most realistic are reported for each emotion, providing an important basis for the construction of dynamic facial stimuli for future research.
Perceived Realism 3 Perceived Realism of Dynamic Facial Expressions of Emotion Optimal Durations for the Presentation of Emotional Onsets and Offsets The presentation of facial displays of emotions is an important method in emotion recognition studies used in basic as well as clinical research. Research in this area was propelled by the introduction of standardized stimuli, primarily the Japanese and Caucasian Facial Expressions of Emotion image set (JACFEE; Matsumoto & Ekman, 1988) and the portrait pictures of facial affect that were produced using Ekman s Facial Action Coding System (FACS; Ekman & Friesen, 1978). Although facial expressions do not necessarily reflect an internal affective state (Kappas, 2003) especially if they are posed as in the case of the JACFEE pictures for the purpose of this article we refer to an emotional facial expression when the face looks to an observer at least as if it is displaying an emotion. One of the major drawbacks of research on facial expressions is that static displays are used in most studies (e.g. Calder et al., 2003; Hall et al., 2004) which are clearly different from reallife conditions and therefore lack ecological validity (Carroll & Russell, 1997). The use of dynamic facial displays of emotion that evolve from a neutral to an emotional expression over time, and vice versa, offers a solution to this problem. The time the full-blown expression stays on the face is called apex, and the change back to a neutral expression is defined as offset. In addition to increased ecological validity, the use of dynamic stimuli offers additional advantages. First, recent neuroimaging research (Kilts, Egan, Gideon, Ely, & Hoffman, 2003; LaBar, Crupain, Voyvodic, & McCarthy, 2003; Sato, Kochiyama, Yoshikawa, Naito, & Matsumura, 2004) and computer modelling studies (Haxby, Hoffman, & Gobbini, 2002) have shown that more brain areas are active in reaction to dynamic than to static facial emotions, indicating that the use of dynamic stimuli may provide additional information that helps to elucidate underlying recognition mechanisms. Second, there is empirical evidence showing that participants recognize emotions better when displayed
Perceived Realism 4 dynamically than when displayed as static or multi-static pictures (Ambadar, Schooler, & Cohn, 2005; Harwood, Hall, & Shinkfield, 1999; Wehrle, Kaiser, Schmidt, & Scherer, 2000; Weyers, Muhlberger, Hefele, & Pauli, 2006). Ecological validity may become problematic, however, when morphed sequences are used. For example, unnaturally slow morphed sequences have been used in one study (Blair, Colledge, Murray, & Mitchell, 2001), but the perceived realism of those stimuli has never been assessed (LaBar et al., 2003). Although the importance of dynamic information in facial-emotion recognition was pointed out by Bassili (1978) and Ekman & Friesen (1982) several decades ago, detailed studies on the perceived realism of facial expressions with different durations have only been conducted recently. To our knowledge, only three studies to date report on the perceived realism of different durations for different facial expressions (Kamachi et al., 2001; Sato & Yoshikawa, 2004; Pollick, Hill, Calder, & Paterson, 2003). We will compare the results of those three studies with our own data in the discussion. The major aim of the present study is to determine at which duration morphed dynamic emotional expressions have to be shown in order to be perceived as realistic. Although the nature of this investigation is exploratory rather than hypothesis-driven, we assume that distinct emotions need to be displayed at different durations to appear realistic (see Kamachi et al.,2001, Sato & Yoshikawa,2004, and Pollick et al.,2003). This study was designed to measure the perception of the temporal characteristics of both onsets and offsets of facial displays of emotion. Participants were shown dynamic displays of faces either evolving from a neutral to an emotional expression (onsets) or faces evolving back from an emotional expression to a neutral one (offset) and asked to adjust the duration of sequences until they perceived it as most realistic.
Perceived Realism 5 Method Participants A total of n=124 volunteers participated in this experiment. The onset condition was rated by n=84 participants (aged 19-61 years, M=21.93 years, 71% female) and n=40 participants (aged 19-56 years, M = 24.35 years, 53% female) rated the offset condition. Written consent from participants was obtained before the experiment, which was approved by the University ethics board. Stimuli Sequences used in this experiment were generated using static images from the JACFEE image set (Matsumoto & Ekman, 1988). In this image set, fifty-six different actors portray one of six emotions (anger, disgust, fear, happiness, sadness, surprise), half of them are male and half are female, half are of Japanese and half of Caucasian origin. Throughout our experiments, this balance was maintained in our sub-sets of stimuli in order to enhance ecological validity. Every actor displays only one emotional and one neutral expression. Except for happiness, which was photographed while the actors were spontaneously smiling, all emotional expressions were posed. Various studies have shown the reliability and validity of the JACFEE set in displaying the intended emotions (e.g., Biehl et al. 1997). Morph sequences were generated using the FEMT (Facial Expression Morphing Tool; Kessler et al., 2005). This newly developed software uses state-of-the-art morphing algorithms to produce intermediate frames between two images. To optimize the production of facial morphs, additional techniques were implemented. First, to minimize distracting facial information, sequences were generated using multiple layers in order to morph only the important features of the face. Second, the use of multiple layers and special smoothing algorithms allowed us to create realistic transitions from closed to open mouths. Third,
Perceived Realism 6 transition control allowed us to apply different rates of warping and color blending for different facial areas across the sequence. Since there are no empirical data available for the actual transition of facial features in emotional expressions, we heuristically applied global warping with an S-curve and color blending was done in a linear manner. The FEMT generates video files of which the duration is specified by the number of frames morphed between the first and the last image and by the number of frames per second (in this study constantly 25 frames s -1 ). Since each frame was constantly presented for 40 ms and the duration of sequences differed, the degree of morphing changed accordingly. Regardless of the absolute length of the video clips, with a frame rate of 25 s -1 all sequences were perceived as fluid. All stimuli used in this study were in color. Morph generation was done on a regular Win2K/XP system. We selected pictures for the six emotions (anger, disgust, fear, happiness, sadness and surprise) from the JACFEE set; each picture was portrayed by eight different actors. This resulted in a total of 48 different actors portraying the emotions. A neutral picture of each actor was also selected. Next we generated emotional sequences, in which the expression changed from a neutral face to an emotional face (onset) or from an emotional expression to a neutral one (offset), using both pictures from the JACFEE set, and synthesized intermediate images using FEMT. Sequences were generated using various durations and consisted of 6 to 76 frames (by an increment of 5 frames), using a frame rate of 25 frames s -1. We thus obtained 15 video clips for each actor, lasting between 240 ms and 3040 ms (in steps of 200 ms), for a total number of 720 sequences (48 actors 15 video clips) for each condition. We used a 6 (emotions: anger, disgust, fear, happiness, sadness, surprise) 2 (actor s sex) 2 (actor s ethnicity: Caucasian, Japanese) 2 (participant s sex) mixed design (linear mixed model considering the dependency structure of the data). Apparatus
Perceived Realism 7 The experiment was run using our own computer software written in Delphi 6.0 on a Win2k/XP system. The stimuli were presented on 19 -TFT monitors, using a resolution of 800 600 pixels and 24 bit colour depth. The viewing distance was approximately 60 cm, which corresponds to a viewing angle of 11.4 horizontally and 15.6 vertically (260 335 pixels). Procedure Participants were tested individually using our laboratory software, each session lasting approximately 15 minutes. Subjects saw 48 emotion sequences (8 actors for each emotion) showing the development from a neutral to an emotional face for the onset sample or showing the fading from an emotional to a neutral face for the offset sample. They were instructed to adjust the duration of each sequence until they perceived the sequence as maximally realistic. This was done by repeatedly pressing faster/slower buttons. Participants could use six different buttons to control the action: four to change the duration of the presented video clip (in steps of +/- 200/600 ms), one to repeat the current sequence and one to select the current time frame when they had decided it was maximally realistic. Since in a preliminary test, subjects could barely differentiate between two sequences with longer durations (e.g. 1.8 and 2.0 seconds), we decided to present sequences in steps of 200 ms. Each emotion sequence was embedded in a prior 1000 ms neutral (onset) or emotional (offset) face and a 300 ms full-blown emotional (onset) or neutral (offset) face afterwards. Sequences were presented in random order throughout all emotions. The duration of the first sequence displayed for every trial was also randomized (240-3040 ms). The name of the emotion displayed was written at the top of the screen as the sequence unfolded. Furthermore, participants were allowed to repeat a sequence as often as they wanted, until they were confident that the duration was as realistic as possible.
Perceived Realism 8 Results Onset condition Due to the distribution of the selected time frames, values were transformed logarithmically to be used in a linear mixed model. An analysis of variance revealed that the duration selected as most realistic differed significantly across emotions, F(5,3923) = 121.81, p < 0.001. The main effect participant s sex was also significant (F(1,82) = 4.24, p <.05), as well as the interaction between participant s sex and type of emotion, F(5,3923) = 7.05, p < 0.001. The main effect actor s sex was also significant (F(1,3923) = 4.36, p <.05), but the main effect of actor s ethnicity was not. Other interactions were not significant either. Table 1 shows descriptive statistics for the preferred duration of each emotion and the estimated values ( model estimates ) based upon our statistical model. Sadness was considered most realistic when shown for relatively long durations, followed by anger, happiness and disgust; surprise and fear were seen as most realistic when shown for relatively short durations (see also Figure 1, left side). Further, male participants (M = 806 ms, SD = 571 ms) tended to perceive the sequences as more realistic when presented at shorter durations than female participants (M = 947 ms, SD = 598 ms), especially for the emotions disgust, fear and sadness (p <.05). Offset condition The analysis of the selected time frames for each emotion revealed that preferred durations again differed across emotions, (F(5,1855) = 35.20, p < 0.001), whereas there was only a trend for participant s sex (F(1,38) = 3.43, p =.072). No significant effects were found for actor s sex or ethnicity. No significant interaction effects were found either. Table 1 shows descriptive statistics and the model estimates for each emotion, indicating which offset duration was perceived as most realistic by participants. The most realistic offset duration for
Perceived Realism 9 sadness was longer than for all other emotions. Again, surprise was perceived as being most realistic when presented for a shorter duration than all other emotions. The most realistic offset durations for anger, disgust, fear, and happiness ranged between 1200 and 1700 ms (see also Figure 1, right side). [Table 1 to be included here] [Figure 1 to be included here] When comparing the perceived realism for offset and onset condition, a 2 2 analysis of variance (ANOVA) with type of presentation condition (onset, offset) and participant s sex showed that conditions differed significantly (F(1,5948) = 975.41, p <.0001). The main effect of participant s sex was also significant (F(1,5948) = 135.72, p <.0001), as well as the interaction between participant s sex and condition (F(1,5948) = 18.75, p <.0001). The means show that female participants chose longer durations on average, and this was particularly the case in the offset condition. Discussion This study contributes new data to the question of how long dynamic facial displays of emotion have to be shown in order to be perceived as realistic. Such empirically obtained time frames allow the creation of realistic dynamic facial expressions. The results confirm that facial displays have different onset durations at which they are perceived as most realistic. Durations vary from 480 ms to 1120 ms according to the estimation of our statistical model. Sadness (860 1120 ms), surprise (500-530 ms) and fear (480-650 ms) are the emotions for which the longest and the shortest onset durations, respectively, are perceived as being realistic. The most realistic onset durations for anger, disgust, and happiness range between 670 ms and 890 ms. We suggest that these temporal characteristics should be taken into account when presenting dynamic facial expression stimuli. As with onsets, preferred offset durations differed across emotions. Surprise (850-1110 ms) and sadness (1440-1600 ms) the
Perceived Realism 10 two displays with the shortest and longest onset durations also had the shortest and longest offset durations. Offset durations for happiness, disgust, fear, and anger ranged between 1040 ms and 1510 ms. The mean offset duration was significantly longer than the mean onset duration across all emotions presented. Participants may have chosen longer durations for offsets than for onsets, because real-life emotion expressions may appear more quickly than they fade away. Our study went beyond the earlier work of Kamachi et al. (2001) and Sato and Yoshikawa (2004) in several aspects. We used six emotions and drew upon a reliable, valid and widely used stimulus set (JACFEE, see below) to enhance ecological validity. Yet our data are comparable to those from other studies conducted with different methods. In line with our own findings, Kamachi et al. (2001) reported that there were specific durations for different facial emotion film clips at which recognition rates were highest. The durations for sadness (3.4 s) and surprise (.2 s) that they reported as optimal for the recognition of an emotion, were in the same relative range as in our study. The duration for anger (.9 s) was almost exactly the same. Moreover, participants easily perceived a facial expression as surprise when it was shown for a brief duration. Sato and Yoshikawa (2004) asked participants to rate the naturalness of the onset of dynamic facial expressions of six emotions presented in different time frames. Although they offered only four possible durations (255, 510, 1020 and 2040 ms), their findings are comparable to ours. Surprise and, to a lesser extent, fear were rated most natural when displayed briefly (255 ms), whereas sadness was rated most natural when shown for long durations (1020 ms). An interesting study on real dynamic facial expressions was conducted by Pollick et al. (2003). They recorded posed facial expressions of actors with 3-D point-light techniques and reported mean durations for the onsets of anger, happiness, sadness and surprise. Supporting our findings, they also found surprise (858 ms) to be the emotion with the shortest
Perceived Realism 11 duration, sadness (1067 ms) with the longest duration, and anger (933 ms) to lie in the intermediate range. Happiness (1100 ms), on the contrary, was displayed longer than our subjects perceived it to be realistic (888 ms). Interestingly, Pollick et al. (2003) reported a higher variability of durations between actors than between emotions. Since in our stimulus set all sequences were morphed and therefore had no temporal variability between posers, we could not contribute data to that issue. With this high variability in mind, one should be cautious providing time data for emotion expressions across different actors. For instance, actors gender was a significant main effect in our study. The detailed interplay between emotion expression and the displaying person remains an open issue for further research. The gender difference found in this study was an unexpected surprise. Since the study was not designed to clarify this issue, interpretations of this effect are speculative, and further studies should provide more detailed information. However, as differences were significant, we decided, to provide data for optimal time frames separated by gender. Limitations of the Study Since all participants were of Caucasian origin, the issue of possible cultural differences remains open. Sato and Yoshikawa (2004) tested Asian subjects and their results were similar to ours, but this question could only be investigated further in cross-cultural studies. Some methodological concerns remain. Although heuristically using the S-curve warping procedure, morphed sequences do not fully reflect real-life conditions. We are currently extracting the optical flow of the most important facial areas by analyzing facial emotions from videos. Research has shown, for example, that different facial movements in the expression of happiness have specific temporal characteristics (Schmidt, Cohn, & Tian, 2003). Our morphing algorithms will include this information to create sequences that develop in a non-linear manner in the future. The realism of these morphs will then need to be
Perceived Realism 12 tested again. Additionally, longer sequences theoretically include a larger amount of information because they have more frames. The confound between sequence duration and amount of information can not be ruled out with our data. Another potential limitation (offset condition) concerns the limited range of durations that subjects could choose for the film clips. Almost all displays of emotions (except surprise) were often perceived as realistic at 3040 ms. Therefore it is not certain that the chosen range (240 3040 ms) of the different durations was optimal for this experiment. Future research should test longer durations with a larger sample of participants in order to confirm that the offset characteristics reported here can be suggested as optimal durations. The last, and probably most profound criticism concerns the rationale behind our methodological approach. We implicitly assume that the durations perceived as being most realistic by our subjects do in fact mirror real-life dynamic facial expressions of emotions. From an epistemological standpoint, however, there is no necessary link between the perception and production of facial expressions. This is further complicated by the fact, that our stimuli were artificially morphed sequences based on real human faces. We do not know whether the morphed (and thus intrinsically unnatural ) stimuli had any effects on the judgements of our subjects. The so-called uncanny valley effect (Mori, 1970) could for instance have influenced subjects ratings of realism: Artificial agents (robots, facial avatars, etc.) are perceived as being increasingly realistic the more natural they look. Paradoxically, when being very close to a natural appearance they are perceived as uncanny and unrealistic. Since our subjects only made relative judgements, this problem cannot be solved with the present design.
Perceived Realism 13 References Ambadar, Z., Schooler, J. W., & Cohn J. F. (2005). Deciphering the enigmatic face: the importance of facial dynamics in interpreting subtle facial expressions. Psychological Science, 16(5), 403-410. Bassili, J. N. (1978). Facial motion in the perception of faces and of emotional expression. Journal of Experimental Psychology: Human Perception and Performance, 4(3) 373-379. Biehl, M., Matsumoto, D., Ekman, P., Hearn, V., Heider, K., Kudoh, T., & Ton, V. (1997). Matsumoto and Ekman s Japanese and Caucasian Facial Expressions of Emotion (JACFEE): Reliability Data and Cross-National Differences. Journal of Nonverbal Behavior, 21, 2 21. Blair, R. J., Colledge, E., Murray, L., & Mitchell, D. G. (2001). A selective impairment in the processing of sad and fearful expressions in children with psychopathic tendencies. Journal of Abnormal Child Psychology, 29(6), 491-498. Calder, A. J., Keane, J., Manly, T., Sprengelmeyer, R., Scott, S., Nimmo-Smith, I., & Young, A. W. (2003). Facial expression recognition across the adult life span. Neuropsychologia, 41(2), 195-202. Carroll, J. M., & Russell, J. A. (1997). Facial expressions in Hollywood's portrayal of emotion. Journal of Personality & Social Psychology, 72(1), 164-176. Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System. Palo Alto: Consulting Psychologists Press. Ekman, P., & Friesen, W. V. (1982). Felt, false, and miserable smiles. Journal of Nonverbal Behavior, 6(4), 238-252. Hall, J. A., & Matsumoto, D. (2004). Gender differences in judgments of multiple emotions from facial expressions. Emotion, 4(2), 201-206.
Perceived Realism 14 Harwood, N. K., Hall, L. J., & Shinkfield, A. J. (1999). Recognition of facial emotional expressions from moving and static displays by individuals with mental retardation. American Journal of Mental Retardation, 04(3), 270-278. Haxby, J. V., Hoffman, E. A., & Gobbini, M. I. (2002). Human neural systems for face recognition and social communication. Biological Psychiatry, 51(1), 59-67. Kamachi, M., Bruce, V., Mukaida, S., Gyoba, J., Yoshikawa, S., & Akamatsu, S. (2001). Dynamic properties influence the perception of facial expressions. Perception, 30(7), 875-887. Kappas, A. (2003). What facial activity can and cannot tell us about emotions. In M. Katsikitis (Ed.), The human face: Measurement and meaning (pp. 215-234). Dordrecht: Kluwer Academic Publishers. Kessler, H., Hoffmann, H., Bayerl, P., Neumann, H., Basic, A., Deighton, R. M., & Traue, H. C. (2005). Die Messung von Emotionserkennung mittels Computer-Morphing: Neue Methoden für Forschung und Klinik [Measuring emotion recognition with computer morphing: New methods for research and clinical practice]. Nervenheilkunde, 24, 611-614. Kilts, C. D., Egan, G., Gideon, D. A., Ely, T. D., & Hoffman, J. M. (2003). Dissociable neural pathways are involved in the recognition of emotion in static and dynamic facial expressions. Neuroimage, 18(1), 156-168. LaBar, K. S., Crupain, M. J., Voyvodic, J. T., & McCarthy, G. (2003). Dynamic perception of facial affect and identity in the human brain. Cerebral Cortex, 13(10), 1023-1033. Matsumoto, D., & Ekman, P. (1988). Japanese and Caucasian Facial Expressions of Emotion (JACFEE) and Neutral Faces (JACNeuF). [Slides]: Dr. Paul Ekman, Department of Psychiatry, University of California, San Francisco, 401 Parnassus, San Francisco, CA 94143-0984.
Perceived Realism 15 Mori, M. (1970). The uncanny valley. Energy, 7(4), 33-35. Pollick, F. E., Hill, H., Calder, A., & Paterson, H. (2003). Recognising facial expression from spatially and temporally modified movements. Perception, 32(7), 813-826. Sato, W., Kochiyama, T., Yoshikawa, S., Naito, E., & Matsumura, M. (2004). Enhanced neural activity in response to dynamic facial expressions of emotion: an fmri study. Cognitive Brain Research, 20(1), 81-91. Sato, W., & Yoshikawa, S. (2004). The dynamic aspects of emotional facial expressions. Cognition and Emotion, 18(5), 701-710. Schmidt, K. L., Cohn, J. F., & Tian, Y. (2003). Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biological Psychology, 65(1), 49-66. Wehrle, T., Kaiser, S., Schmidt, S., & Scherer, K. R. (2000). Studying the dynamics of emotional expression using synthesized facial muscle movements. Journal of Personality and Social Psychology, 78(1), 105-119. Weyers, P., Muhlberger, A., Hefele, C., & Pauli, P. (2006). Electromyographic responses to static and dynamic avatar emotional facial expressions. Psychophysiology, 43(5), 450-453.
Perceived Realism 16 Table 1 Descriptive statistics for the durations perceived as being realistic for the onset and offset condition Onset Condition Offset Condition Emotion n M (SD) in ms Mdn in ms Model Estimates n M (SD) in ms Mdn in ms Model Estimates Male participants Surprise 24 647 (419) 528 504 (14) 19 1002 (414) 915 845 (36) Fear 24 611 (356) 515 483 (21) 19 1283 (566) 1190 1077 (53) Happiness 24 857 (419) 653 707 (9) 19 1279 (597) 1340 1036 (38) Disgust 24 797 (351) 703 670 (24) 19 1227 (567) 1040 1040 (18) Anger 24 896 (448) 740 735 (29) 19 1458 (606) 1540 1236 (35) Sadness 24 1029 (456) 903 862 (17) 19 1681 (676) 1615 1437 (52) Female participants Surprise 60 658 (377) 540 528 (15) 21 1338 (725) 1090 1114 (47) Fear 60 822 (431) 728 650 (29) 21 1705 (683) 1815 1481 (73) Happiness 60 901 (353) 915 761 (9) 21 1615 (728) 1715 1374 (50) Disgust 60 1032 (389) 1015 892 (32) 21 1559 (723) 1440 1335 (23) Anger 60 997 (419) 953 829 (32) 21 1714 (670) 1690 1510 (42) Sadness 60 1270 (419) 1240 1122 (22) 21 1838 (723) 1765 1604 (58) Note. Mean values can be used as optimized time frames for the presentation of emotional sequences (all values in ms). Model estimates are calculated values for each duration (details in the methods section). Standard deviation in parentheses.
Perceived Realism 17 Figure 1 Box plots of the selected durations for each emotion for male and female participants Note. The x-axis displays the different categories of emotion, the y-axis shows the different durations between 240 and 3040 ms. The graph on the left side shows the results for emotional onsets, the graph on the right side shows the results of emotional offsets. Bars represent minimal and maximal values.