Affective User Interface Design that Endows Mobile Phones with Emotional Expressions

Transcription

1 Affective User Interface Design that Endows Mobile Phones with Emotional Expressions Chia-Yu Hsu*, John Kar-kin Zao*, Ming-Chuen Chuang** * Department of Computer Science, National Chiao Tung University, cyuhsu.cs98g@nctu.edu.tw, jkzao@csie.nctu.edu.tw ** Institute of Applied Arts, National Chiao Tung University, cmimg@cc.nctu.edu.tw Abstract: This paper proposes an affective interface for mobile phone which makes user feel like interacting with an individual who has personality. Mobile phone -among all electronic devices- is the one that we now rely on and interact with the most. However, the thinking of humanity and emotional engagement with the device had yet being explored. By designing a set of different emotion featured elements for the interface, users are able to intuitively understand and therefore better respond to the devices' information and status. Analyzed the in-depth interviews we concluded an affective emotional interaction model and its design principles which offer more emotional engagement for interaction. Based on the principles, we explored the design possibilities for the interface and tested it. The study intends to structure an affective emotional interaction and its design criteria for mobile phones to gain certain personalities that lead to users' stronger comprehensibility, more emotional attachment to mobile phone, and better behavioral engagement in interaction. Key words: human-machine interactions, affective interfaces, affective design, interaction, affective expressions, mobile phone, users' emotion 1. Introduction People nowadays rely more on their mobile phones and the interactions with them have become more intimate and crucial. In particular, smart phones have brought more innovative and human-centered technology with development. As machines become more intellectual, some new notions started to emerge, such ideas include Norman s Emotional Design [1] and Picard s Affective Computing [2].These have suggested that a machine should be able to express emotion in order to fully perform its intelligence, and thus being effectively utilized. If users are having emotional attachments with their mobile phones, they may be more careful while using and more willing to repair them when broken, therefore the turnover of the product will be suspended and product loyalty will be raised [3]. As a result, it is crucial for designers to raise the consumer s affection of the products [4]. Besides, when technological product like computer become a society member of our lives, they may help us to acknowledging technology and to construct interactions that encourage us to learn [5]. The research assumed that applying emotional interaction features to mobile phones can enhance the user s willingness and frequency of use. Following the idea of Living object by Jordan [6] and Norman s introspection of people s lack of emotional attachment with their cell phones, we aimed to propose an affective interaction for mobile phones. Through in-depth interviews, we analyzed the key factors of how human emotionally attached to living or non-living objects and which of these factors can be implemented into mobile phones. Based on these 1

2 factors, we identified two aspects: Emotion and behavior. We then constructed the new interaction interface for mobile phone which consists of an Emotion Model and a Behavior Architecture. Eventually we tested the prototype with users and evaluated the comprehensibility and consistency of the new interaction interface. 2. User Study The user study has two phases. The first part- the in-depth interview with mobile phone users - is to understand what are the factors making people interact emotionally with their daily objects. In the second part, we used the KJ method and introduced some principles which can be applied to mobile phones based on the interview data. Due to such affective products are still rare and hard to imagine, the daily objects also includes interviewee s pets. Series of one-on-one, in-depth interviews were conducted. The brief of the interview guide lines were as below: 1. What is the role of interviewee s mobile phone in his/her daily life, and how much emotion he/she gets out of their phone? 2. What is the object in the interviewee s life that he/she affectively interacts with the most? Be it either living or non-living. 3. We asked for more detail of the interaction with the object, including reason of having the object, type of interactions, emotional changes along interactions, scenarios of the interactions, the behaviors during the interactions, etc. 4. We asked interviewees to imagine a new mobile phone that can do all mentioned interactions, and asked for their initial thoughts and opinions about such a product. 5. We asked interviewee s opinions on the balance between having affective interactions with phones and keeping the usability or functionality of the phones. 2.1 Participants We recruited seven participants (three female) between 20~50 years old. All have previous experience with mobile phones. Their jobs were diverse, and their phones ranged from regular phones to smart phones. All participants have affective attachments to living or non-living objects (e.g., pets, cars, dolls) in their daily lives. 2.2 In-depth Interview The interview conducted was semi-structural. Through verbatim method, we analyzed the emotional interactions between people and living and non-living objects. From the result, we realized that non-living objects can potentially arouse user s emotional projection, just as living things. However, since mobile phones are functional products in daily life, the functionality and usability are the most crucial factors that make users love the devices. Emotional connections are rare in this relationship. As a result, most of the interviewees claimed that they could hardly imagine the phones become living objects that they might put more emotional into. In addition, negative or complicated emotional behaviors may lead to user s distaste, while adding of Artificial Intelligence may lead to the user s fear of use. 2.3 KJ Method Though interviewees responded rather rationally claiming they have no emotional connection with their phones, we did find some emotional interaction according their narratives. We extracted the larger intentions behind some subconscious behaviors, and used the KJ Method as tool to sort-out the criteria and key points for affective interaction between people and their mobile phones. We used the KJ Method to categorize users needs from former interviews, and organized the essential factors that people emotionally interact with targets. These contain: target s active behavior; feeling of achievement by 2

3 taking care of the target; feeling of owning each other; feeling of being relied on; feeling of being accompanied; accumulated affection through times; intimacy and friendliness; eagerness for simple interaction; target s instant emotional feedback; pleasure of teasing/ playing with the target; learning because of the target; owner s mental projection on target; developed specific behaviors with target; and personification of the target. After synthesizing the information into a few key points by the KJ Method, we sorted the points to identify design possibilities (Figure 1). We used design controllability as a y-axis in which the higher means the easier to control by design; the lower means more depending on individual s development, yet some of which can be stimulated by means like imitating life or personalities, etc. The x-axis represents the time, the more towards to Instance means short-time or immediate reaction in interaction; the Constancy side means the long-staying characteristic of interaction. Figure.1 Analysis from KJ method 3. Affective User Interface Design Based on the conclusion of interviews, we start to design the affective interface for mobile phone. We tried to fully interpret those key points with higher design controllability and extend their potentials in order to cover those cannot be achieved directly through design. 3.1 Design Feature From the study of users, we found all the key factors can be categorized as either emotion-wise or behaviorwise. As a result, we proposed a system featuring a Behavior Architecture that generates the phone s contextual behaviors, and an Emotion Model which gives the phone sophisticated personality. While considering the interaction between user and phone, we included the basic functional behaviors which people are already used to nowadays, and also take into account the new behaviors that may happen under an emotional interaction scenario. Based on the result from KJ Method, we tried to define the simplest behaviors yet generate most of the emotional connection. We concluded the interaction behavior architecture between user and mobile phone. The architecture comprises levels from actor, functional/emotional, action/reaction, to purpose. Behaviors can be seen as a complete set responding to the architecture and can be altered according to different scenarios (Figure 2). 3

4 The semantic expressions user make to the phone includes functional commands such as Yes, No, Previous/ Select Left, Next/ Select Right ; contextual feedbacks including encourage/discourage the phone s suggestion; and emotional interaction including Comfort, Pat, Tickle, or Ignore. Phone needs to express intellectual responses including Agree/Disagree, Suggest Previous/ Next, Notify ; active expression such as Emotions, Playful (get attention) ; and emotional responses Like/Dislike, Comforted, and Dodge. We see the whole screen as a poseable body and designed the expression according to human instinctive gestures. We also mirrored the gestures for user to phone to phone to user in order to make the two sets similar. This helps users to easily associate expressions with corresponding gestures, and vice versa. Figure.2 Behavior Architecture We use dimension models as emotion model for our prototype, and it comprises two dimensions- Arousal and Valence. Different emotion states are represented as coordinates in the model. We can categorized the four basic emotions in this model as joyful, angry, sad and relaxed (Figure 3). During interaction, different action from the user will trigger different movement of coordinates in both Arousal and Valance that leads to the change of emotion status. By changing the emotion status, the corresponding behaviors may change as well. The research also proposed corresponding expressions for both behaviors and emotions that derived from this behavior architecture. Figure.3 Emotion Model 3.2 Design Principle The idea of the Affective User Interface is to design certain features that trigger users to associate with daily communication, such as facial expression, languages, biological features and body languages, in order to comprehend the meaning. 4

5 Starting from the present technology of mobile phone, the interface can be defined as the composition of several Elements, i.e. video, audio, motion, etc., and each Element comprises both behavioral and emotional attributes (Figure 4). More sophisticated and subtle semantics and be expressed thorough manipulating the sequence and combination of the Elements. The design of the affective interface in this research is based on the following principles: (1) Use sensuous effects to imply certain emotion or to emphasize the semantic, e.g., higher color saturation and warm hue implies more positive emotion status, or longer vibration and sound implies higher excitement. (2) Manipulate life-like features for user s better understanding of emotion changes, e.g., simulating heartbeats, breaths, etc. (3) Imitate body-languages for user s better understanding of specific semantic, e.g., considering the area of the screen as movable body, the distortion of the screen can leads to certain understanding of body language. Following the design principles, we developed the structure of Elements in relationship with Emotions and Behaviors. On the other hand, the way user communicates with the mobile phone also needs to be designed corresponding to the device s language in a coherent and intuitive manner. We adapted the following principles while designing the gesture of user s: (1) Coherence of bi-directional communication: similar meaning expressed by both device and user, such as user s Yes and phone s Recommend, should be consistent. This helps user to intuitively adopt the language without being confused. (2) Intuitive gestures: searching from daily gestures human do to others or their pets, such as beating for suppressing and bullying, or caress for comforting and showing affection. Figure.4 Compositions of Elements 4. Design Result From the earlier study, we concluded that both behavior and emotion are needed to achieve better affectiveness of an object. After exploring the suitable emotion model and the emotion architecture for mobile phone, we then further developed the rules of the emotion model and the visualization of the behaviors as one prototype. 4.1 Emotion Interface Design The interface design in this research takes the entire screen as designable area and all visual effects apply to the whole screen. To imply one of the two emotional dimensions - Arousal, the design manipulates the frequency of sparkling screen to make user think of breathing or heartbeat thus to have a sense of its arousal degree. As for Valence, we used different degree of screen saturation to imply the dimension; the higher saturation means the higher Valence. For better feasibility while implementing on prototype, we set 5 different degrees of both Arousal 5

6 and Valence, which creates the matrix of 25 different emotions. In responding to the matrix, we also set 5 different degrees of sparkling frequency and screen color saturation (Figure 5). Figure.5 Emotion Model and Attributes In order to make the emotional expression more convincing, the visual effects on screen exists across different work modes of the phone, i.e. desktop page, in-application, locked mode, etc., and will vary with time, like human s emotion does. However, in order to avoid disturb, the visual effect of sparkling screen, which implies the Arousal of emotion, should be deactivated while user is using the device. From the initial test of these emotional features, we discovered that the simply manipulating the sparkling frequency and color saturation wasn t making the display legible enough to tell the different emotion, therefore we introduced different Hue overlaying the entire screen, which is based on mankind s universal impression to colors, to emphasize different emotional status. Such as dark red masking implies anger; and the rainbow color on screen implies happiness. Table 1 shows an example of design by applying the proposed emotion model. Table 1. Emotional Expression 4.2 Behavior Interface Design 6

7 In the animation movie: Aladdin(1992), the animator made the carpet into some human-like or pet-like character by distorting the shape of itself and it was successfully affective. Norman once emphasized that people tend to project their own emotion and belief on anything [1], therefore consumers would gain more fun and pleasure from the product s being personated. Other research also indicated that a life-like or a Living Object is one of types that create meaningful user-product relationship [7], and it s also the factor influencing user s pleasure and preference. As a result, we tried to personate the phone while considering the design of the interface. While designing the behavior, we find it easy to associate the upper corners of the screen to the hands of the device, which can be used to do waving in order to express notifying ; The upper half of the screen can be seen as the head or the upper-body of the device which can do directional swinging, nodding or shaking head; the middle part of the screen can be thought as the body or the belly of the device, which can imitate the pose that some pets do to invite people s caress or the dodging movement while being punished; etc. (Table 2). Table 2. Semantic Behaviors 4.3 User Gesture Design The design of the control method focuses on gesture-control and multi-touch instead of other means existed in the market because of the following: (1) Advanced and mature built-in motion sensors such as accelerometer had been widely adopted on mobile phone to achieve intuitive gesture control. (2) Nowadays touch screen become a standard for mobile phones, physical keypad is no longer considered necessary; also, considering users may not be accustomed to speaking to machine and voicing may be inconvenient in many situations, voice-control is not included in our design yet it can be further developed as alternative. As a result, we don t put keypad and voicecontrol in our design consideration. (3) By designing the gesture control, we can create better consistency between 7

8 user s behavior and the movement that the device is expressing while with similar meaning. However, the gesture and multi-touch control design for the prototype should not be understood as replacement but alternatives to conventional means, thus user can chose between existing means for more specific function or the corresponding and intuitive language (Table 3). Table 3. User Gesture 5. Evaluation To evaluate the application of this interface design, we experimented with prototype and tested the usability. 5.1 Participants There are ten participants in this experiment. The range of age is between 20~50 years old, and all have previous experience with mobile phones. Considering about that gender may effects demand and preference for high-tech products, we recruited five males and five females. Considering participants between using smart phone and regular phones may have different acceptance of the new design, both smart phone users and regular phone users are included in the test, 5 participants for each of the group. Table 4 shows the information of participants. Table 4. The Information of Participants 8

9 5.2 Experimental Procedure In this experiment, we tested how users comprehend the interface and gesture design. We asked participants to watch each video clip of mobile phone s behavior and emotion expression, and then participants matched the clips with different meaning according to their comprehension. for evaluating the result, we used a five-point Likert scale to see how user understand and agree with them. For the gesture test, we demoed each gesture user interact with the mobile phone, and interpret their command. Then, we used a five-point Likert scale to find out if it was intuitive and understandable. 5.3 Results Further discussion base on the results as below: (1) Confusion Matrix: Table 5 is the confusion matrix of how did user comprehend mobile phone s expression. Table 5. Confusion Matrix We can tell from this matrix that most participants understand the expression we designed for mobile phones. In this testing of ten participants, only one confused Dislike/Frown with Beaten/Dodge and one confused Playful/Naughty with Beaten/Dodge. As for the emotional expressions, two participants confused the Neutral with Relaxed. Other than these 5 items were successfully comprehended. (2) Evaluating the expressions, participants ranked the degree of engagement for each item from 1 to 5, and then we calculated the average and standard deviation for each item. The result is shown in Table 6. Table 6. Evaluation of emotional attachment of mobile phone s expressions 9

10 Among the semantic behavior expressions, Table6 shows that Clip #2 (Comforted/Satisfied), #4 (Dislike/Frown), #5 (Playful/Naughty), and #7 (Beaten/Dodge) fall relatively low between 3 (Fair) and 4 (comprehensive) than others, and the deviations are also greater. The design of these behaviors has room for improvement, and some of them may be customized in order to fit user s preference and habits. Besides those with lower grades, the rest were comprehensive and can be seen as effective ways for semantic expressions. As for emotional expressions, Clip #14 (Joyful) and #15 (Sad) have higher rank in comprehensibility and lowest deviations, therefore these two are proven universal and successful; Clip #13 (Angry) has slightly lower ranks yet still comprehensive and low deviation; Clip #12 (Neutral) has low rank and largest deviation and, according to interviewees, easily got confused with the implication of Clip #16 (Relaxed). This is perhaps due to the different degrees of color saturation are too subtle and not easy to distinguish. Therefore, greater difference between saturations, color overlaying or more contrastive sparkling rate may be considered adding to the visual for better legibility. 10

11 (3) Evaluation of behavioral engagement of user gestures: participants ranked the degree of intuitiveness for each item from 1 to 5, and then we calculated the average and standard deviation for each item. The result is shown in Table 7. Table 7. Behavioral engagement of user gestures Among the designs of user gestures, Gesture #2 (No/Cancel) has lowest deviation, which means the least controversial gesture design; Gesture #3 and #4 (Left, Right) are also less controversial, yet with average 3.8, they can be improved in design; Gesture #5 (Comfort/Caress) and #6 (Pat/Bully) have high intuitiveness yet also high deviations, this may have to do with interviewees behavioral habits therefore can be customized for better usability; Gesture #1 (Yes/Okay) and #7 (Tease/Tickle) are the most controversial even though the average intuitiveness are still above 3 (Fair), the design of these two gesture may also need adjustment. The key idea of designing the gestures is to provide consistency between gestures and device s behaviors. For instance, the movement of the phone s expressing Left, Right should be similar to user s gesturing Left, Right in order to cause least confusion. Another important idea is that the design of these gestures is not to replace the existing means of operation but to provide intuitive alternatives for better interaction. Also, the gestures alone cannot cover the entire complicity of user s commands. Thus, even though some participants still prefer the traditional ways of operation, it doesn t degrade the value of the gestures for being optional alternatives. 6. Discussion Most of the expression designed are comprehensive with little controversy expect 4 expressions- Comforted/Satisfied, Dislike/Frown, Playfu/Naughty, and Beaten/Dodge. This should results from participants own understanding of certain body-languages or different cultural context. These controversial expressions can be customized or to allow user set up their own preference in order to meet user s own understanding. As for emotional expressions, the design for Joyful, Sad, Angry are comprehensive, yet the design for Neutral and Relaxed are relatively confusing. Unlike the other three emotions which overlay certain color or grayscale on the entire screen, the expression of Neutral and Relaxed used merely sparkling and color saturation, this should be the reason of less legibility. However, the average grades of the two were still above Fair meaning still understandable. Using greater contrasts in sparkling rate and saturation may help this issue, or other type of visual effect can be further developed for these two emotions. 11

12 7. Conclusion Through user interviews, we concluded the following design principles for the affective interface: (1) the design of expressions, either emotional or behavioral, should not decrease the usability or functionality of mobile phone. (2) Consistency and comprehensibility should be carefully examined while designing either emotional expression or semantic behavior, and the meaning should be delivered correctly to the general. (3) The emotional expressions and the behaviors should be extendable to different phone working status in order to fully evoke user s emotional projection, i.e. extend the affective expressions across locked mode, in-application mode, initial desktop, etc. (4) The dimensional system- Arousal and Valence- adopted as emotion model in this design prototype can be further developed for its internal changing logic, and the design of expressions should explicitly convey the corresponding emotion. Based on the design principles, we designed the interface prototype and tested it with mobile phone users. From the evaluation tests, we proved the applicability of the principles and from which the interaction model developed. The evaluation shows that the affective interface we designed can provide user the experience of higher comprehensibility, emotional attachment of mobile phone s expression, and behavioral engagement of user gestures. We expect the design can improve the emotional connection between users and their phones therefore bringing more joy to the interaction. The interface elements and features constructed in this research can be further developed in terms of orders and compositions in favor of conveying more sophisticated and precise meaning. However, some behavior design should take individual difference into account and may allow customization to achieve better comprehension. To sum up, the design prototype and the design principles concluded in the research can be valuable references while developing an affective interface for mobile phone. 8. References [1] Norman, D. A. (2004) Emotional Design: Why We Love (or Hate) Everyday Things, New York: Basic Books. [2] Picard, R. W. (1997) Affective Computing, The MIT Press. [3] Mugge, Ruth, Hendrik N. J. Schifferstein, and Jan P. L. Schoormans (2006) A Longitudinal Study on Product Attachment and Its Determinants, European Advances in Consumer Research, Vol. 7, Eds. Karin M. Ekstrom and Helene Brembeck, Duluth, MN: Association for Consumer Research. [4] Van Hinte, E. (1997) Eternally yours: visions on product endurance, Rotterdam NL: 010 Publishers. [5] Marakas, G., Johnson, R. and Palmer, J.W. (2000) A Theoretical Model of Differential Social Attributions Toward Computing Technology: When the Metaphor Becomes the Model, International Journal of Human- Computer Studies. [6] Jordan, P. W. (2000) Designing Pleasurable Products: An Introduction to the New Human Factors, London: Taylor & Francis. [7] Katja Battarbee and Tuuli Mattelmaki. (2002) Advertising and Promotion: An Integrated Marketing Communication Perspective, Proceeding of Design and Emotion Conference. 12