Face Emotions and Short Surveys during Automotive Tasks

Transcription

1 Face Emotions and Short Surveys during Automotive Tasks LEE QUINTANAR, PETE TRUJILLO, AND JEREMY WATSON March 2016 J.D. Power A Global Marketing Information Company jdpower.com

2 Introduction Facial expressions are a daily occurrence in the communication of human emotions. In 1872, Charles Darwin described facial expressions as signals of specific emotions (Darwin, 1872/1998), which was later tested by Paul Ekman and Wallace Friesen (Ekman & Friesen, 1987). Their team conducted a cross-cultural study, demonstrating that interpretations of facial expressions appear to be universal across cultures. While minor cultural variations appeared in the ratings of emotional intensity, agreement in emotional evidence was found to be high across cultures. Moreover, a method called the Facial Action Coding System (FACS) was designed to help classify human facial movements by their appearance on the face, based on a system originally developed by a Swedish anatomist (Hjortsjö, 1969). Ekman, Friesen, and Hager later published a significant update to FACS (Ekman et al. 2002), with variants of the system emerging in modern technologies for computer-based detection of facial expressions. The purpose of our research is to better understand the relationship between human facial expressions face emotions and consumer attitudes toward products and services. J.D. Power s Voice of the Customer research measures customer satisfaction with products and services based on consumer surveys. Survey research asks respondents what they think or how they feel about products/services and then extrapolates that data to actual attitudes that impact consumer behavior. Biometrics is an alternative to infer attitudes from observations of bodily behaviors corresponding to human emotions. Specifically, the research evaluated how human facial expressions would compare with survey responses in order to measure attitudes and behaviors. Early research by Cacioppo, Quintanar, Petty, and Snyder (1981, 1979) evaluated the relationship among facial expressions, emotions, and attitudes, and an expansion of this assessment using modern computer technologies seemed promising. Biometrics is an alternative to infer attitudes from observations of bodily behaviors that correspond to human emotions. Biometrics takes physical information from a human body and makes it quantifiable. Biometrics technologies are becoming increasingly refined. A variety of bodily measurements can be evaluated, including facial expressions, heart rate, eye tracking, pupil dilation, galvanic skin response (arousal), and voice modulations. How effective is biometrics in assessing emotions and attitudes? The focus of the research in this study is on facial expressions as an assessment of respondent s attitudes. The automation of methods to recognize the emotional content of facial expressions has been evolving in parallel with psychological research. Research conducted by Bartlett et al. (2003, 2006) began with prototypes for automatic recognition of spontaneous facial actions. Littlewort et al. (2006) explored the dynamics of facial expression extracted automatically from video. Pantic and Bartlett (2007) went further with a machine analysis of facial expressions. Wu, Bartlett, and Movellan (2010) applied the use of Gabor motion energy filters in the recognition of facial expressions. Computerized methods improved when 1

3 Littlewort, Whitehill, Wu, Fasel, Frank, Movellan, and Bartlett (2011) developed the Computer Expression Recognition Toolbox (CERT) which served as an end-to-end system for fully automated facial expression recognition that operates in real time. To access the affective nature of facial expressions, the points on the face are scanned by computer analysis to recognize emotions. The results of this J.D. Power study are expected to help researchers better understand how facial expressions/face emotions can accurately assess consumer attitudes and, in turn, predict their behaviors. Research Design Methodology: The research paradigm consisted of a website evaluation scenario in which participants evaluated a site s usability. Three 2015 automotive websites (Honda, Kia, and Hyundai) were presented to participants in a randomized order. Asian automakers were chosen to reduce preference bias that might emerge with European or U.S.-based automakers. Participants were asked to use each website s Build a Car price tool and afterward complete a short survey to rate Appearance, Navigation, Speed, and their Overall satisfaction. A webcam captured videos of participants faces as they used the Build a Car tool. Although eye-tracking information was also collected, it was not used in this analysis. The research paradigm was a website evaluation scenario in which webcam face videos were collected while using the website s Build a Car tool. Figure 1: Build a Car Websites Were Utilized in Research Procedure Note: Participants completed short surveys after using Build a Car websites. Facial encoding: The technology platform we used for the emotions recognition of facial expressions was the imotions Biometric Research Platform (Release 5, 2015). Detailed specifications can be found on the imotions product website (2016-a), which includes additional resources, e.g., a facial emotions publications list (2016-b) and a guide for facial emotions analysis (2016-c). The underlying technology involved three steps: (1) face detection; (2) feature detection; and (3) classification. The position, orientation, and information encompassing key facial features were input into classification algorithms that translated the features into emotional states and affective metrics. These technologies rested 2

4 on methods of image processing, edge detection, Gabor filters, and statistical comparisons with normative databases provided by facial expression software engines. This can be imagined as a kind of invisible virtual mesh covering the face of a respondent: whenever the face moves or changes expressions, the face model adapts, follows, and classifies emotions. The face video collected during each Build a Car website session was analyzed using the imotions biometrics platform, which was also used to set up the PC-based experimental procedure, sequence and timing of stimulus events, baseline gray screen, website presentation, and online survey questions and ratings. Different baselines screens were also tested but yielded no differences from a gray screen for establishing an emotions baseline. The imotions platform digitizes facial expressions to measure emotions by using encoding algorithms based on the FACET scoring methods. Facial encoding scans the various points on a face and then interprets the patterns into measurable emotion events. Face emotion index scores were collected simultaneously for the following nine indices: Joy, Anger, Surprise, Fear, Sadness, Disgust, Contempt, Confusion, and Frustration. Overall sentiment scores were also collected for generalized positive and negative facial expressions. Sample: The sample consisted of 30 college-educated participants, 50% male and 50% female. Ages ranged from 21 to 67 (37 mean), with a racial mix of 79% white and 7% each of black, Asian, and Hispanic. The average length of each session was 35 minutes. Figure 2: Facial Encoding for Emotions Webcam Facial encoding scans the various points on a face and then interprets the patterns into measurable emotion events. Eye Tracker Note: Webcam face videos were collected and computer analyzed for emotions. 3

5 Analysis Face emotions from video recordings: On average, face videos were recorded for 35 minutes for each of the 30 participants at a frame rate of 30 frames per second. Each frame was computer analyzed for the 11 emotions listed above, which resulted in a data set consisting of nearly 2 million observations (records): 35 min x (30 frames/sec) x 30 participants = 1.9 million. Consequently, proper data aggregation and transformation methods were required for effective data analysis. There is a need to reduce big data into a measurable Emotion Index. Figure 3: Data Example Biometric measures require big data solutions for analysis. An Emotion Index was calculated by deriving a percentage of emotion change. Note: Biometric measures require big data solutions for analysis. J.D. Power Emotion Index: Emotion Index scores were calculated by deriving the percentage of emotion change between the highest and lowest ranges for each participant across all conditions. For each face emotion, a minimum and maximum were determined for each person across all conditions (within-subjects). An Emotion Index score was derived by (a) converting to positive integers; (b) calculating the percentage between minimum and maximum; and (c) converting to a 1,000-point scale. This indexing approach created a percent Emotion Index (from zero to 1,000) based on the range of distance between within-subject minimum and maximum end points. This was found to be the best and most emotion-sensitive data transform method among the alternative methods, such as difference scores from baseline; threshold binary scores; square roots; and logarithm scores. Raw data scores were low and variable across participants, while other transforms changed the scale and/or data distribution. 4

6 Results Core Emotional Dimensions A factor analysis of the 11 face emotions was performed using a principal components analysis with orthogonal varimax rotation (see Kim & Mueller, 1978a; 1978b), followed by an oblique Procrustes rotation (SAS ROTATE=PROMAX), with the varimax output as the target matrix. An oblique rotation method was used because the simultaneous emotion measurements were expected to be interrelated with each other. The number of factors retained was determined based on the solution that best satisfied the following criteria: the percentage of variance explained by each factor; the outcome of a scree test; the size of the eigenvalue differences between factors; the number of high loadings on each factor; the perseverance of factors over each of the possible rotations; and the meaningfulness of the factor structures over different rotations. As shown in Figure 4, a rotation of three factors, accounting for 93.1% of the variance, was selected as representing the best estimate of the primary emotional judgmental dimensions utilized during the automotive task evaluation. After examining the pattern of factor loadings, these factors were labeled Enjoyment, Dislike, and Perplexed. Factor scores were also calculated for use in later analyses. This method of collapsing the data matrix by stringing out across conditions to assess dimensionality and derive factor scores for group comparisons has been found useful in prior precedent (Quintanar, 1982; Osgood, May, and Miron, 1975). Figure 4: Factor Analysis of Core Emotional Dimensions Three factors were found as primary emotional dimensions utilized during this automotive task evaluation: Enjoyment, Dislike, and Perplexed. Note: Factor loadings are multiplied by 100, rounded to integers, and those > 40 are flagged by asterisks. Scree plots and eigenvalues indicate three primary factors. 5

7 Face Emotions Analysis of Variance (ANOVA repeated measures) was used to compare automaker websites on face emotions. The strongest emotions appeared during the first 2 minutes of the Build a Car website sessions. Kia s car build had higher levels of Confusion and Disgust and lower levels of Joy. An analysis of factor scores also showed that Kia had higher Perplexed and lower Enjoyment scores. Figure 5: Face Emotions during Initial Impressions Honda Hyundai Kia Confusion Disgust Joy Note: Kia s car build had higher levels of Confusion and Disgust and lower levels of Joy. The level of Confusion in Kia s Build a Car evaluation persisted throughout the remainder of the Web session after the initial impressions. Surprise also emerged at higher levels during Kia s car build session. Figure 6: Face Emotions during Latter Session Honda Hyundai Kia Confusion Surprise Note: Confusion in Kia s car build persisted throughout the remainder of the Web session. 6

8 An analysis of emotion indices is also available on a second-by-second basis, which can be useful for comparing emotions when key events occur during a session. You can also observe fluctuations and overall slope. An example with Confusion is shown in Figure 7. Figure 7: Confusion across 10 Minutes of Using Build a Car Tool Note: Confusion in car builds is shown in second-by-second plots. Survey Ratings An analysis of short survey ratings after each Build a Car session found that the Honda car build scored highest in Navigation, Speed, and Overall Satisfaction. Hyundai was a close second. Kia s car build scored lowest in Appearance, Navigation, and Speed. As shown in Figure 8, these ratings are consistent with the face emotion results. Figure 8: Ranking Car Builds by Survey Ratings Note: Highest ratings are marked by green and lowest by orange. 7

9 When website satisfaction ratings were divided into low/high categories, it was found as expected that Appearance, Navigation, and Speed were highest when Overall satisfaction was high (see Figure 9). Figure 9: Attributes during Low/High Satisfaction Note: Appearance, Navigation & Speed receive highest ratings when satisfaction is high. Face Emotions and Survey Ratings How do face emotions relate to participant evaluations and satisfaction levels? Face emotions were found to be aligned, that is, there is a directional relationship with survey ratings. Negative face emotions were at higher levels when satisfaction ratings were lower. Findings show that face emotions were aligned with survey ratings. Negative face emotions were higher when satisfaction ratings were lower. Figure 10: Face Emotions during Low/High Satisfaction Ratings Note: Face emotions are aligned with ratings, with negative emotions higher when satisfaction ratings are lower. 8

10 A correlational analysis also showed that negative emotions increased as survey ratings decreased. Why were negative emotions more prominent than positive? It may be that the nature of this task seemed more like work rather than fun for this website evaluation. Conclusions Overall, face emotions were an accurate measure of participant reactions during the Build a Car website sessions as measured by the J.D. Power Emotion Index percent (zero-1,000). Initial impressions (first 2 minutes) showed higher levels of Confusion (a mean index score of 745) and Disgust (609), and lower levels of Joy (237) in Kia s car build. During the latter part of the website session, higher Confusion (739) persisted throughout Kia s car build, with Surprise emerging (444). Honda and Hyundai didn t have these issues. Factor analysis of the 11 emotions revealed three core underlying emotional dimensions used by participants during this automotive task: (1) Enjoyment; (2) Dislike; and (3) Perplexed. Further analysis showed that Kia scored highest on Perplexed and lowest on Enjoyment (more confusion) during initial impressions. Correlations were found between attribute ratings and emotions. Negative face emotions were high when overall website satisfaction was low. These results are corroborated by Kia s recent 2015 decision to dismiss their Web design firm in order to pursue a better redesign. These findings are also supported by the J. D. Power manufacturer website evaluation studies. Overall, Honda and Hyundai car builds were straightforward and allowed you to easily build and explore car options. The Kia car build was pretty (nice photos and car views), but more complicated, harder to search and navigate. Overall, face emotions were an accurate measure of participant reactions during website sessions as measured by the Emotion Index. Moreover, participants initial impressions seemed impacted when a large pop-up panel window appeared first thing in Kia s car build and required a ZIP code to continue. Although the Hyundai car build also asked for a ZIP code, it was done via a small pop-up panel described as needed for the latest rebates and prices. Honda didn t ask for a ZIP code until the end of their car build and only as an optional item. Future Research and Applications There are many applications of this research in providing nonobtrusive evaluations of human emotions to predict consumer behavior and attitudes toward products and services. Customer video feeds can be used to provide evaluations of consumer reactions in automotive, retail, travel, hospitality, or similar environments. Face emotions can be gathered from video sources such as webcams for digital-comfortable consumers (e.g., Millennials) to leave video-based service feedback or product reviews. There might also be optional security-based applications to assess strongly polarized emotions. Moreover, it s possible to do a census of branches/facilities to assess and improve customer service. 9

11 There is an abundance of video opportunities for recognizing face emotions so much so that questions of privacy and legal permissions to record and process such video for evaluation of personal emotions may be necessary. One strategy might be to obtain approval for recordings similar to what is currently done when contacting a call center and a request for approval to record for the purpose of improving customer service is asked up front. Future research is expected to investigate more thoroughly the core judgmental dimensions used in product evaluation to assess how they persist across various industries. Empirical assessments of emotion indexing methods and data aggregation strategies are also important for hardened research paradigms and tools. Other opportunities can include utilizing scenarios that elicit stronger emotions about products and services. Future research is also expected to delve deeper into the comparison of face emotions to survey-measured attitudes with special attention given to the persistence of these feelings and their predictive nature on consumer behaviors. For example, are face emotions more transitory and reflective of the moment rather than more enduring attitudes in predicting consumer behavior? Perhaps face emotions are additive and with consistent reactions contribute to the formation of enduring persistent attitudes. Perhaps facial expressions are inherent in the processing of emotions and always involved with attitudes at all levels. There are many research opportunities available for evaluating more effective ways to blend biometrics, consumer attitudes, and the prediction of consumer behavior. We see many applications for nonobtrusive evaluations of emotions to predict consumer behavior and attitudes toward products and services. Authors Lee Quintanar, Ph.D., Director, Marketing Science, J.D. Power Pete Trujillo, Senior Manager, J.D. Power Jeremy Watson, Ph.D., Senior Statistician, J.D. Power References J. D. Power 2015 Biometrics Research Study SM Bartlett, M.S., Littlewort, G., Braathen, B., Sejnowski, T.J., & Movellan, J.R. (2003). A prototype for automatic recognition of spontaneous facial actions. In S. Becker & S. Thrun & K. Obermayer, (Eds.) Advances in Neural Information Processing Systems, Vol 15, p , MIT Press. Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek,C., Fasel, I., Movellan, J.R. (2006). Automatic Recognition of Facial Actions in Spontaneous Expressions. Journal of Multimedia 1(6) p Cacioppo, J. T., Quintanar, L. R., Petty, R. E., & Snyder, C. W. (1981). Electroencephalographic, facial EMG, and cardiac changes during equivocal and less equivocal attitudinal processing [Abstract]. Psychophysiology, 18,

12 Cacioppo, J. T., Quintanar, L. R., Petty, R. E., & Snyder, C. W. (1979). Changes in cardiac and facial EMG activity during the forewarning, anticipation, and presentation of proattitudinal, counterattitudinal, and neutral communications [Abstract]. Psychophysiology, 16, 194. Darwin, Charles. (1998). The Expression of The Emotions In Man And Animals. New York: Philosophical Library. (original work published in 1872). Ekman, P., et al. (1987). Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion. Journal of Personality & Social Psychology, 53(4), Ekman, P., Friesen, W. V., & Hager, J. C. (Eds.). (2002). Facial Action Coding System [e-book]. Salt Lake City, UT: Research Nexus.Ekman, P., Friesen, W. V., & O Sullivan, M. (1988). Smiles when lying. Journal of Personality and Social Psychology, 54, Hjorztsjö, C. H. (1969). Man's face and mimic language. Studentlitteratur, Lund, Sweden. imotions (2016a). Biometric Research Platform. Product website: imotions (2016b). Publications resources for Facial Expressions Analysis. Retrieved from imotions (2016c). Facial Expression Analysis: Everything you need to know to elevate your research with emotion analytics - The Definitive Guide. Retrieved from Kim, J., & Mueller, C. W. Introduction to factor analysis. (Sage University Paper Series on Quantitative Applications in the Social Sciences, ). Beverly Hills and London: Sage Publications, 1978a. Kim, J., S Mueller, C. W. Factor analysis: statistical methods and practical issues. (Sage University Paper Series on Quantitative Applications in the Social Sciences, ). Beverly Hills and London: Sage Publications, 1978b. Littlewort, G., Bartlett, M., Fasel, I., Susskind, J., and Movellan, J. (2006). Dynamics of facial expression extracted automatically from video. Image and Vision Computing 24(6), p Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, and Bartlett M (2011). The Computer Expression Recognition Toolbox (CERT). Proc. IEEE International Conference on Automatic Face and Gesture Recognition. Osgood, C. E., May, W. H., & Miron, M. S. Cross-cultural universals of affective meaning. Chicago, Illinois: University of Illinois Press, Pantic M. & Bartlett, M.S. (2007). Machine Analysis of Facial Expressions, in K. Delac & M. Grgic, Eds., Face Recognition, Vienna, Austria: I-Tech Education and Publishing, 2007, pp Quintanar, L. R. (1982). The interactive computer as a social stimulus in computer-managed instruction: a theoretical and empirical analysis of the social psychological processes evoked during human-computer interaction (Doctoral dissertation). The University of Notre Dame, Notre Dame, Indiana. 11

13 Wu, T., Bartlett, M.S., and Movellan, J. (2010). Facial expression recognition using Gabor motion energy filters. IEEE CVPR workshop on Computer Vision and Pattern Recognition for Human Communicative Behavior Analysis. 12