Misinterpretation of facial expression:a cross-cultural study

Psychiatry and Clinical Neurosciences (1999), 53, 45 50 Regular Article Misinterpretation of facial expression:a cross-cultural study TOSHIKI SHIOIRI, md, phd, 1,3 TOSHIYUKI SOMEYA, md, phd, 2 DAIGA HELMESTE, phd 3 AND SIU WA TANG, md, phd 3 1 Department of Psychiatry, Shiga University of Medical Science, Otsu, 2 Department of Psychiatry, Niigata University School of Medicine, Niigata, Japan and 3 Department of Psychiatry and Human Behavior, University of California, Irvine, California, USA Abstract Key words Accurately recognizing facial emotional expressions is important in psychiatrist-versus-patient interactions. This might be difficult when the physician and patients are from different cultures. More than two decades of research on facial expressions have documented the universality of the emotions of anger, contempt, disgust, fear, happiness, sadness, and surprise. In contrast, some research data supported the concept that there are significant cultural differences in the judgment of emotion. In this pilot study, the recognition of emotional facial expressions in 123 Japanese subjects was evaluated using the Japanese and Caucasian Facial Expression of Emotion (JACFEE) photos. The results indicated that Japanese subjects experienced difficulties in recognizing some emotional facial expressions and misunderstood others as depicted by the posers, when compared to previous studies using American subjects. Interestingly, the sex and cultural background of the poser did not appear to influence the accuracy of recognition. The data suggest that in this young Japanese sample, judgment of certain emotional facial expressions was significantly different from the Americans. Further exploration in this area is warranted due to its importance in cross-cultural clinician patient interactions. cross-cultural difference, emotion, facial expression, recognition. INTRODUCTION Observation of facial expression is an essential part of a general psychiatric examination. Accurately recognizing or interpreting a patient s facial emotional expression is also important in the patient therapist interaction. While there exist individual differences in the recognition of another person s facial expression, with some observers being more sophisticated than others, most might think that misjudgment of facial expression should occur more often when the individual is from another culture. Substantial research has documented the universality of several emotional expressions. 1 These expressions include anger, contempt, disgust, fear, Correspondence address: Siu Wa Tang, md, phd, Department of Psychiatry and Human Behavior, North Campus Research Lab (ZOT 1681), University of California, Irvine, CA 92697, USA. Email: <swtang@uci.edu> Received 6 July 1998; accepted 27 August 1998. happiness, sadness, and surprise. While these expressions are universal, cross-cultural differences in levels of recognition 2 and intensity attributed to these emotional expressions 3 appear to exist. Additionally, recent studies suggest that disease states may affect facial expression recognition. 4 6 Thus it is important to establish normal baselines and to be aware of potential variations which contribute to the normal range of responses. To study cultural differences in facial expression recognition requires a different methodology than studying similarities. Matsumoto outlined such requirements: (i) raters from different cultures must view the same expressions of posers; (ii) the facial expressions presented must meet criteria for validly and reliably portraying the universal emotions; (iii) each poser must appear only once; and (iv) facial expressions presented must include posers of more than one race. 2 While some earlier sets failed to meet these criteria, 7 15 the Japanese and Caucasian Facial Expressions of Emotion (JACFEE) fulfills these requirements. 16

46 T. Shioiri et al. Previous studies have reported cross-cultural differences in facial expression recognition using the JACFEE. 2,3,17 This facial emotion expression photo set constructed by Matsumoto and Ekman includes details on ratings by Euro-American observers and a small Japanese study group, 16 which has also been published separately in Biehl et al. 17 Because their data demonstrated high consensus ratings by Euro-American raters, we accept that the photo set depicts standardized emotions recognizable by most Euro-American individuals. The observations made by Matsumoto and Ekman and Biehl et al., that Japanese subjects have lower correct identification of certain emotional facial expressions than American subjects, was intriguing. 16,17 We therefore decided to examine the utilization of this instrument in a larger sample of young Japanese living in Otsu, Japan. Otsu is a suburban city not exposed to foreign travelers as much as the larger cities of Tokyo or Osaka. It is assumed that while young Japanese might have had exposure to foreign facial expressions through the media (television, movies, newspapers and magazine) or travel, their long-term residence in Japan makes them a good sample for studying cross-cultural differences in facial expression recognition. SUBJECTS AND METHODS Instruments Matsumoto and Ekman s (1988) JACFEE 16 photo set was used for testing the subjects. In the JACFEE photo set, more than 100 people were photographed repeatedly while attempting to show each of the seven universal emotions. Briefly, the Caucasians in the photo set consisted of Euro-Americans; the Japanese in the photo set consisted of both Japanese Americans and Japanese nationals living in the US. All were recruited from the student populations of the San Francisco Bay area. Models were instructed to activate certain facial muscles associated with the seven universal emotions rather than to pose a particular emotion, except for the photos of happiness, when shots were taken with the models spontaneously smiling. Neutral poses were shot at the beginning of the session, prior to the expression poses. A pool of photos were selected for possible inclusion, which ensured that the expressions were exactly those requested with no extraneous muscle movements. A final pool of photos was then selected to ensure that each poser only contributed one photo in the final set, which comprises 56 photos, including eight photos each of anger, contempt, disgust, fear, happiness, sadness, and surprise. Four photos of each emotion depict posers of either Japanese or Caucasian descent (two men, two women). Subjects The 123 Japanese subjects in our study consisted of 42 male and 81 female students with a mean age (± SD) of 19.8 ± 2.3 years, drawn from the Shiga University of Medical Science in Otsu, Japan. Otsu is a small suburban city with a population of 250 000, near Kyoto. Students at Shiga University of Medical Science were recruited from all over Japan. All subjects were born and raised in Japan, as were their parents. All were college undergraduates. We regard this sample as a good representation of contemporary young educated Japanese. We then referenced our data to the results reported earlier by Matsumoto and Ekman and Biehl et al. 3,16,17 Procedure and statistical analysis Each slide was shown for 10 s to the group of subjects, in a random order. The answer sheet provided a choice of seven emotion terms: anger, contempt, disgust, fear, happiness, sadness, and surprise. Each emotion term was also presented on a nine-point (0 8) intensity scale, labelled neutral, 0; weak, 1; moderate, 4; to strong, 8. The subjects selected the one word that they felt best described the emotion expressed in each slide. They also rated each slide for emotional intensity as described here. The data reported by Matsumoto and Ekman and Biehl et al. were used for statistical comparison. 3,16,17 Statistical analysis was performed by analysis of variance (ANOVA), and multiple comparison with the Student-Newman-Keuls procedure, unpaired t- test, and the Chi-squared test. All of the data were tabled and analysed using the SPSS statistical package. RESULTS Our results (mean ± SD) showing the percentage of individuals who correctly identified each emotional expression, are summarized in Table 1. Table 1 also includes a comparison with the data of Matsumoto and Ekman and Biehl et al. 16,17 For the results in Table 1, the mean scores of correct judgment of expressed emotions were tested using a 2 (judge culture) 2 (intended emotion term versus others) Chi-squared test, and computed separately for each photo. A P value of less than

Misinterpretation of facial expression 47 Table 1. Comparison of the mean percentage of correct identification for each of seven emotions Our results Results of Matsumoto and Ekman and Biehl et al. 16,17 Japanese American Chi-squared* Emotion Poser culture (n = 123) (n = 271) P value Anger Caucasian 67.9 ± 25.3 85.3 ± 9.3 P < 0.001 Japanese 44.7 ± 16.6 85.9 ± 5.2 P < 0.001 Contempt Caucasian 54.5 ± 7.2 63.8 ± 1.9 n.s. Japanese 55.9 ± 17.7 61.3 ± 10.2 n.s. Disgust Caucasian 63.4 ± 0.8 79.3 ± 8.3 P < 0.005 Japanese 63.9 ± 3.8 82.6 ± 6.6 P < 0.005 Fear Caucasian 43.9 ± 5.0 77.6 ± 5.8 P < 0.001 Japanese 45.1 ± 10.9 80.9 ± 7.9 P < 0.001 Happiness Caucasian 91.1 ± 18.5 97.5 ± 2.6 n.s. Japanese 98.4 ± 1.0 97.8 ± 0.6 n.s. Sadness Caucasian 71.4 ± 28.2 91.8 ± 3.2 P < 0.001 Japanese 61.8 ± 17.2 91.2 ± 2.1 P < 0.001 Surprise Caucasian 94.5 ± 2.4 92.6 ± 4.8 n.s. Japanese 83.9 ± 17.8 90.9 ± 9.4 n.s. * The Chi-squared test was used to compare between our subjects and the subjects of Matsumoto and Ekman and Biehl et al. As for this analysis, a P value of less than 0.005 was regarded as statistically significant. In the studies of Matsumoto and Ekman and Biehl et al., the subjects were 271 US-born American and Japanese nationals (data of Japanese subjects not shown in table). 16,17 The American sample included 128 men (mean age: 19.9) and 143 women (mean age: 19.8), and was recruited from universities in the San Francisco Bay area; all were at least third generation Americans and none were of Asian descent. 16,17 0.005 was regarded as statistically significant. Matsumoto and Ekman s data showed a significantly higher percentage of American observers identifying the expressed emotions correctly (30 of the 56 emotional expressions presented, with only one contempt photo (no. 15, Japanese female poser), being scored significantly higher by Japanese observers). Compared to American observers in the Matsumoto and Ekman and Biehl et al. data, 16,17 our Japanese observers appeared to experience particular difficulties with the emotions of anger, disgust, fear, and sadness, regardless of poser culture or sex. There were almost no significant differences between our and Matsumoto and Ekman s Japanese subjects (data not shown in Table 1). A four-factor ANOVA was computed on the scores, using the observer culture (2) as betweensubjects factors, and poser culture (2), poser sex (2), and emotion (7) as within-subjects factors. We found two main significant effects (culture: d.f. = 1, F = 51.5, P < 0.001; emotion: d.f. = 6, F = 18.3, P < 0.001) and a twoway interaction (culture emotion: d.f. = 6, F = 3.81, P < 0.003). Two one-way ANOVA, one each for American and Japanese subjects, were computed on the average percentage of correct identifications across all eight photos for each emotion, using emotion as the independent variable. Both were significant (Japanese subjects: F(6,49) = 11.47, P < 0.0001. American subjects: F(6,49) = 29.5, P < 0.0001). When the multiple comparison with the Student-NewmanKeuls procedure was applied (< 0.05 as statistically significant), the data indicated that the order of the emotions, in terms of least easily identified to most easily identified, were: Japanese: Fear = Contempt = Anger = Disgust = Sadness < Surprise = Happiness, and American: Contempt < Fear = Disgust = Anger < Sadness = Surprise = Happiness. Thus, the ordering of the difficulty or ease in identifying the correct emotion was different for the two cultures. Two two-way ANOVA were computed on these scores, one each for American and Japanese subjects, using poser culture and poser sex as the independent variables. None of the effects for both groups were significant, indicating that the ratings did not differ as a function of poser culture or sex.

48 T. Shioiri et al. The mean intensity ratings for each emotional expression and a statistical comparison of our data (mean ± SD) with that reported by Matsumoto and Ekman 3 are shown in Table 2. A four-factor ANOVA was computed on these scores, using judge culture (2) as between-subjects factors, and poser culture (2), poser sex (2), and emotion (7) as within-subjects factors. We found two main significant effects (judge culture: d.f. = 1, F = 6.27, P < 0.02; emotion: d.f. = 6, F = 48.2, P < 0.001) and two two-way interactions (emotion poser culture: d.f. = 6, F = 2.71, P < 0.03; emotion poser sex: d.f. = 6, F = 3.27, P < 0.01). In investigating the relationship between accuracy and judgments of intensity, we found that the mean intensity rating scores were significantly different in most instances, when Americans were compared to Japanese raters (Table 2). Some statistically different intensity ratings were also observed between our Japanese sample and the Matsumoto Ekman Japanese group, but there was no significant trend for one group to score lower than the other group, when all emotions were considered (data not shown in Table 2). Figure 1 depicts the types of emotions that our Japanese raters selected for each of the seven emotion categories presented to them during testing. There were significant errors in the correct recognition of emotions, except for the emotions of surprise and happiness. To summarize, our results show that (i) sadness and anger were misunderstood by some Japanese raters as disgust; (ii) fear was regarded as surprise at a high rate; and (iii) the subjects showed a tendency to rate disgust as contempt or anger, and contempt as disgust or anger. Moreover, 10% of the subjects mistook contempt as happiness. DISCUSSION In the present pilot study, we tested the recognition of emotional facial expressions in a 123-Japanesestudent sample using the JACFEE. The JACFEE is an instrument designed for cross-cultural evaluation of emotional facial expression. As shown in the previous section, there are some very interesting observations in the present study. We found a significant difference in facial emotional expression recognition between our larger Japanese sample and previously reported American observer Table 2. Comparison of the mean intensity ratings for each of seven emotions Our results Matsumoto and Ekman s results 3 Japanese American Unpaired t-test Emotion Poser culture (n = 53 122)* (n = 124) P value, t values Anger Caucasian 5.63 ± 1.73 6.61 ± 0.84 P < 0.0001, t = 5.67 Japanese 4.47 ± 1.72 6.18 ± 1.10 P < 0.0001, t = 9.31 Disgust Caucasian 5.69 ± 1.66 6.06 ± 1.24 P = 0.0485, t = 1.98 Japanese 5.70 ± 1.74 5.97 ± 1.15 n.s., t = 1.44 Happiness Caucasian 5.27 ± 1.64 6.29 ± 1.14 P < 0.0001, t = 5.67 Japanese 5.96 ± 1.59 6.67 ± 1.02 P < 0.0001, t = 4.18 Sadness Caucasian 4.52 ± 1.57 4.66 ± 1.31 n.s., t = 0.76 Japanese 4.01 ± 1.54 4.73 ± 1.44 P = 0.0002, t = 3.79 Surprise Caucasian 4.87 ± 1.77 5.59 ± 1.23 P = 0.0003, t = 3.71 Japanese 4.31 ± 1.59 5.44 ± 1.20 P < 0.0001, t = 6.30 Contempt Caucasian 3.97 ± 1.83 Japanese 4.09 ± 1.81 Fear Caucasian 5.32 ± 1.60 Japanese 5.44 ± 1.51 * For intensity ratings, the sample size for our Japanese observers is not fixed for each of the 56 photos although we chose only persons who recognized the emotion correctly to compare to Matsumoto and Ekman s results. 3 Data not reported in Matsumoto and Ekman (1989). 3 In Matsumoto and Ekman s study, the American sample included 124 (80 male, 44 female) US-born college students recruited from the University of California, Berkeley, excluding those of Japanese, Chinese, or Korean ancestry. 3 There was no mention of the subjects ages in their study. 3

Misinterpretation of facial expression 49 Figure 1. Summary of emotional facial recognition evaluation by our Japanese sample using the Japanese and Caucasian Facial Expressions of Emotion (JACFEE) photo set. Data are summarized as the percentage of subjects who listed the stated emotion. For example, for the eight photos depicting the emotion of surprise, 89% of 123 Japanese observers regarded the emotion as surprise, 4.5% as fear, 2.1% as sadness, 1.5% as contempt, 1.1% as happiness, 0. 8% as anger, and 0. 3% as disgust. ratings. 2,3,17 While facial expressions of surprise and happiness were well understood by both Japanese and American subjects, anger, contempt, disgust, fear and sadness were not always well recognized by Japanese compared to American test subjects. In general, Japanese subjects scored more poorly on these slides than American subjects. On only one of eight slides for contempt, did the Japanese subjects score better than the American comparison group. Our Japanese study data, having a larger sample size (n = 123) compared to the earlier Japanese (n = 44) sample of Matsumoto and Ekman, as well as involving subjects from a different area of Japan (Otsu vs Osaka), lends support to the concept that these test differences represent Japanese norms, which are independent of sample size and location. These data raise some interesting questions requiring further exploration. Differences in the agreement levels across emotions may arise because of cultural differences in the meanings of the emotional expressions, or because of differences in the frequency of their expression. 17 It is possible that commonly used American and Japanese facial expression prototypes do not completely overlap. Second, judgments may be affected by the complexity of the facial components involved in the expressions, and which may overlap to some extent among emotions. 17 For example, there may be an element of surprise in the facial expression of fear. As shown in Fig. 1, surprise was substituted for fear in a significant percentage of our ratings. As another example, the contempt slides depicted individuals who were smirking. The slight curvature of the mouth in a smirk appeared to be misunderstood in 10% of our Japanese subjects as a smile or happiness. Japanese subjects also gave lower emotional intensity ratings for the facial expression photos compared to American raters, confirming previous observations. 3 The smaller percentage of correct identification and lower intensity ratings in our Japanese sample compared to the American data of Matsumoto and Ekman, may be related to a number of factors. Cultural differences in the perception of emotion may be related to cultural differences in

50 T. Shioiri et al. emotional expression. Data have shown that Japanese more than Americans mask the expression of negative emotions in the presence of an authority figure. One hypothesis for interpretation of lower intensity ratings by the Japanese subjects, is that Japanese may discount the amount of emotion that they see. Alternatively, the opposite may be true. Emotions may be expressed more intensely in the Japanese culture. Thus when judging the same expression, they may interpret it as less intense than the Americans did. A third interpretation is that intensity ratings are influenced by confidence in the ratings, so that low confidence would produce low intensity ratings. These possibilities require further exploration. These investigations underscore the importance of controlling for cultural issues if ratings are to be used in clinical investigation of diseases, such as in the investigation of Huntington s disease gene carriers and recognition of disgust. 4 While there is consensus that recognition of the basic emotions of happiness, sadness, surprise, disgust, anger, and fear are universal, culturally dependent variations in the normal population may occur. Thus, we conclude that Japanese university students judge emotional facial expressions using different criteria compared to Euro-American subjects, and we suggest the possibility of cultural factors in the recognition of these universal emotional expressions. In the present preliminary study, certainly, it is possible that between American and Japanese subjects the differences in sex ratio and academic background may influence our results and that there may be differences in emotional condition at the experiment among subjects. However, these data are important in the general psychiatric examination setting, because cultural differences in facial expression and facial expression recognition may confound psychiatric evaluation. Many psychiatric hospitals nowadays have staff from foreign countries. Further understanding of cultural differences should aid diagnosis and contribute to understanding the baseline norm from which psychiatric evaluation is determined, particularly in cross-cultural settings. REFERENCES 1. Ekman P. Strong evidence for universals in facial expressions: A reply to Russell s mistaken critique. Psychol. Bull. 1994; 115: 268 287. 2. Matsumoto D. American-Japanese cultural differences in the recognition of universal facial expressions. J. Crosscult. Psychol. 1992; 23: 72 84. 3. Matsumoto D, Ekman P. American Japanese cultural differences in intensity ratings of facial expressions of emotion. Motivation Emotion 1989; 13: 143 157. 4. Gray JM, Young AW, Barker WA, Curtis A, Gibson D. Impaired recognition of disgust in Huntington s disease gene carriers. Brain 1997; 120: 2029 2038. 5. Rösler A, Lanquillon S, Dippel O, Braune HJ. Impairment of facial recognition in patients with right cerebral infarcts quantified by computer aided morphing. J. Neurol. Neurosurg. Psychiatry 1997; 62: 261 264. 6. Phillips MIL, Young AW, Senior C et al. A specific neural substrate for perceiving facial expressions of disgust. Nature 1997; 389: 495 498. 7. Eiland R, Richardson D. The influence of race, sex, and age on judgments of emotion portrayed in photographs. Comm. Monogr. 1976; 43: 167 175. 8. Ekman P, Friesen W. Pictures of Facial Affect. Consulting Psychologists Press, Palo Alto, 1976. 9. Kilbride JE, Yarczower M. Recognition of happy and sad facial expressions among Baganda and US children. J. Crosscult. Psychol. 1976; 7: 181 194. 10. Kilbride JE, Yarczower M. Recognition and imitation of facial expressions: A cross-cultural comparison between Zambia and the United States. J. Crosscult. Psychol. 1980; 11: 281 296. 11. Kilbride JE, Yarczower M. Ethnic bias in the recognition of facial expressions. J. Nonverb. Behav. 1983; 8: 27 41. 12. Schlosberg H. Three dimensions of emotion. Psychol. Rev. 1954; 61: 81 88. 13. Fois-Wittman D. The judgment of facial expression. J. Exp. Psychol. 1930; 13: 113 151. 14. Shimoda K, Argyle M, Ricci Bitti P. The intercultural recognition of facial expressions of emotion. Eur. J. Soc. Psychol. 1978; 8: 169 179. 15. Wolfgang A, Cohen M. Sensitivity of Canadians, Latin Americans, Ethiopians and Israelis to interracial facial expressions of emotions. Int. J. Intercult. Relations 1988; 12: 1 13. 16. Matsumoto D, Ekman P. Japanese and Caucasian facial expressions of emotion (JACFEE) (Slides). San Francisco, CA. Intercultural and Emotion Research Laboratory, Department of Psychology, San Francisco State University, 1988. 17. Biehl M, Matsumoto D, Ekman P et al. Matsumoto and Ekman s Japanese and Caucasian facial expressions of emotion (JACFEE): Reliability data and cross-national differences. J. Nonverb. Behav. 1997; 21: 3 21.