Social Robots and Human-Robot Interaction Ana Paiva Lecture 5. Social Robots Recognising and Understanding the others

Transcription

1 Social Robots and Human-Robot Interaction Ana Paiva Lecture 5. Social Robots Recognising and Understanding the others

2 Our goal Build Social Intelligence d) e) f)

3 Scenarios we are interested.. Build Social Intelligence d) e) f) Focus on the Interaction

4 The problem Based on the limited perception of a robot, how to build technology to understand the social situation and the user s (and other agents ) affective, social, motivational and informational states, in order to respond in a socially appropriate manner.

5 Machine sensing and automatic understanding of human behaviors

6 Action Analysis User Analysis Interaction Analysis

7 Scientific and Engineering Issues* Which types of messages are communicated by behavioral/ social signals? This question is related to psychological issues pertaining to the nature of behavioral signals and the best way to interpret them. Which human communicative cues convey information about a certain type of behavioral signals? This issue shapes the choice of different modalities to be included into an automatic analyzer of human behavioral signals. How are various kinds of evidence to be combined to optimize inferences about shown behavioral signals? This question is related to issues such as how to distinguish between different types of messages, how best to integrate information across modalities, and what to take into account in order to realize context-aware interpretations. *Pantic, M., Pentland, A., Nijholt, A., & Huang, T. S. (2007). Human computing and machine understanding of human behavior: a survey. In Artifical Intelligence for Human Computing (pp ). Springer Berlin Heidelberg.

8 Which types of messages are communicated by behavioral/social signals? * affective/attitudinal states (e.g. fear, joy, inattention, stress); manipulators (actions used to act on objects in the environment or selfmanipulative actions like scratching and lip biting), emblems (culture-specific interactive signals like wink or thumbs up), illustrators (actions accompanying speech such as finger pointing and raised eyebrows), regulators (conversational mediators such as the exchange of a look, palm pointing, head nods and smiles). *Pantic, M., Pentland, A., Nijholt, A., & Huang, T. S. (2007). Human computing and machine understanding of human behavior: a survey. In Artifical Intelligence for Human Computing (pp ). Springer Berlin Heidelberg.

9 Human Sensing* Sensing human behavioral signals including facial expressions, body gestures, nonlinguistic vocalizations, and vocal intonations, is essential in the human judgment of behavioral cues and involves a number of tasks. Face: face detection and location, head and face tracking, eye-gaze tracking, and facial expression analysis. Body: body detection and tracking, hand tracking, recognition of postures, gestures and activity. Vocal nonlinguistic signals: estimation of auditory features such as pitch, intensity, and speech rate, and recognition of nonlinguistic vocalizations like laughs, cries, sighs, and coughs. *Pantic, M., Pentland, A., Nijholt, A., & Huang, T. S. (2007). Human computing and machine understanding of human behavior: a survey. In Artifical Intelligence for Human Computing (pp ). Springer Berlin Heidelberg.

10 Beyond the four W s Who? (Who the user is?) Where? (Where the user is?) What? (What is the current task of the user? ) How? (How the information is passed on? Which behavioral signals have been displayed?) When? (What is the timing of displayed behavioral signals with respect to changes in the environment? Are there any co-occurrences of the signals?) Why? (What may be the user s reasons to display the observed cues?)

11 Faces Gestures Actions

12 Social Signals Five types of Nonverbal behaviour (Ekman & Friesen 1969) Emblems gestures directly translated to words (e.g. peace sign) different meanings across cultures e.g. giving the finger Illustrators/Iconic gestures gestures that acompanies speech to make it vivid, visual, or empathic e.g illustrating the size of something, throwing a ball also include some facial expressions

13 Social Signals Five types of Nonverbal behaviour Regulators nonverbal behaviours used to coordinate conversation head nodes looking at/orienting the body towards someone Self-adaptor nervous behaviours that release nervous energy touching face, tug hair, bite lips uncounscious (elicit suspicion) Displays of emotion face, voice, body, touch

14 Looking at faces Automatic face analysis: - face detection & recognition (detect who and where the user is); - facial expression recognition (detect how the user is feeling); - Gaze analysis

15 Face detection & recognition One way of identifying users is through the use of tools and algorithms for person identification. These may involve the following steps: - Extract Low-level Features: For each face image low-level features are extracted (for example normalized pixel values, image gradient directions) these vectors to form a large feature vector F(I). - Computing Visual Traits: For each extracted feature vector F(I), the output of n trait is calculated based on classifiers Ci=1...n in order to produce a trait vector C(I) for the face. These classifiers may be focused on attributes such as gender, age, and race, which provide strong cues about a person s identity. - Perform Verification: To decide if face matches one already in the system (calculating if the new user is he same person, can be done by comparing their trait vectors using a final classifier D.

16 Recognizing Facial Expressions A typical approach to expression recognition is to: Goal: to categorize input samples into a number of classes (attitudes, or emotions) Approach: apply standard pattern recognition procedures to train a classifier (Yang et al., 2007) for the detection.

17 Expressions through the Face

18 Facial Expressions What does a facial expression show? The internal physical state of a person An indication of what he/she is going to do next The plans, expectations and memory. The emotional state.

19 Facial Expressions of Markers of Emotional Expression Emotion expressions of emotion tend to last a couple of seconds (smile with enjoyment - 10 seconds) polite smile without emotion (excepcionally brief ¼ second or it can be enforced during long periods of time) facial expressions of emotion involve involuntary muscle actions that people cannot deliberately produce/supress

20 Facial Expression of Emotion Markers of Emotional Expression e.g. Duchenne smile Non Duchenne Duchenne

21 Facial Expression of Emotion Markers of Emotional Expression e.g. Duchenne smile cheek raiser Lip corner puller Non Duchenne Duchenne

22 First test of the universal hypothesis 3000 photos of different people 6 basic emotions Universality of Facial Expressions

23 Universality of Facial Expressions First test of the universal hypothesis 3000 photos of different people 6 basic emotions Anger Fear Disgust Surprise Happyness Sadness

24 Universality of Facial Expressions: Ekman s studies First test of the universal hypothesis photos showed to participants across multiple countries Japan, Brasil, Argentina, Chile, U.S participants were asked to select the emotion term that better matched the emotion being displayed from a list of 6 terms participants achieved accuracy rates of 80-90% In all countries

25 Universality of Facial Critics to this first experiment Expressions Participants had seen U.S. television and movies and might have learned american labels for the expressions

26 Second experiment Universality of Facial Expressions: Ekman s studies Ekman travelled to Papua, new Guinea lived 6 months with people of the Fore tribe did not see any movie or magazine did not speak english minimal exposure to westerners Two tasks were designed to evaluate the universality hypothesis

27 Universality of Facial Expressions Fore participants judging western photos Adults Children U.S students judging Fore expressions Anger Disgust Fear Happiness Sadness Surprise

28 Universality of Facial Expressions Other studies about universality of facial expressions further confirmed these results Ekman, 1984, 1993 Elfenbein & Ambady 2002,2003 Izard 1971, 1994

29 FACS The Facial Action Coding System (FACS) is a human-observer-based system designed to detect subtle changes in facial features. Viewing videotaped facial behaviour in slow motion, trained observers can manually FACS code all possible facial displays, which are referred to as action units (AU).

30 AUs (Action Units ) in FACS

31 Techniques for automatically detecting AUs in faces Most current work on automated facial expression analysis attempt to recognize a small set of prototypic expressions, such as joy and fear (those are trained using pictures and a classifier is built for the purpose). Image analysis techniques Extraction of motion information by computing the difference in image intensity between successive frames in an image sequence Extraction of edges and lines from facial images for detecting furrows and wrinkles Problem: prototypic expressions, occur infrequently, and human emotions and intentions are communicated more often by changes in one or two discrete features.

32 Recognizing Facial Expressions: what s in a face?

33 Recognizing Poses and Poses are static configurations of a person s body (arms, legs) for example, a Stop when interacting with a robot. Other gestures, like for example, go away or come with me are motion gestures and defined through a set of motion patterns. Gestures

34 Recognizing Poses and Gestures As with facial expressions, the recognition of these gestures involves several steps. One of the steps involves pose analysis. Techniques such as neural networks can be used, where a network is trained with data annotated with the gestures we want to identify.

35 Human actions usually involve human-object interactions, where we can see articulated motions along complex temporal structures. Identifying Human Actions Actions are spatio-temporal patterns. Issues in action recognition: - the extraction and representation of suitable spatiotemporal features, and - the modeling and learning of dynamical patterns.

36 Human Actions Analysis Input: - Video-based methods (more used) - Multi- camera motion capture (MoCap) systems: can produce accurate 3D joint positions, (yet special equipment and very expensive). And it is still a challenging problem to develop a marker-free motion capturing system using regular video sensors. - Depth cameras can be used for motion capturing, with some reasonable results (although there is noise occlusion occurs). - Because of the difference in the motion data quality, the action recognition methods designed for MoCap data might not be suitable for depth cameras.

37 Actions

38 Social Signals Engagement & Attention Social Attitudes (e.g. dominance, conflict) Personality Intention and Plans Emotional States (anger, happiness, etc) Confusion, Stress Informational state (Knowledge levels)

39 Data sets for Social Signals recognition To train the classifiers for specific types of attitudes or emotion recognition that we want the robot to do, data has to be captured and annotated for that attitude/emotion. In general, annotation of the data, both for posed and spontaneous data, is usually done separately for each attitude assuming independency between them. For example, the classification emotion classes the annotation can be done in two dimensions: valence and arousal. As such, annotation can be done in: positive activation+positive evaluation; positive activation+negative evaluation; negative activation+negative evaluation; negative activation+positive evaluation; and neutral (close to the centre of the 2D emotional space). Once a corpus of annotated data is created for a certain attitude, then a classifier can be built with the features extracted.

40 Kinect sensor and Software Development Kit (SDK) Kinect is an array of sensors, including a camera and a depth sensor. In addition to the raw depth image, Kinect extracts a 3D virtual skeleton of the body. These capabilities, packed in an affordable and compact device, is ideal to use as a sensor for social robots, in particular for gesture analysis

41 Cases Studied

42 Case Studied 1: icat, the Affec2ve Chess Player

43 Perceiving the User: applica2ondependent user s affec2ve states and expressions User s states related to the game and the social interac2on with the icat Valence of feeling Interest towards the icat Engagement with the icat (Poggi, 2007) Willingness to interact and to maintain the interac2on with the icat

44 The Inter-ACT corpus Mul2modal data 4 cameras; contextual informa2on Affect annota2on experiment Levels of valence of feeling and interest towards the icat Ginevra Castellano, Iolanda Leite, André Pereira, Carlos Mar6nho, Ana Paiva, Peter W. McOwan: Mul6modal Affect Modeling and Recogni6on for Empathic Robot Companions. I. J. Humanoid Robo6cs 10(1) (2013)

45 The Inter-ACT corpus Inter-ACT (INTErac2ng with Robots- Affect Context Task) corpus: 156 thin-slices of the interac2on between 8 children and the icat robot

46 Expression detec2on: a smile detector prototype Behaviour baseline Trained model Extract smile indicators - Lips bounding box ratio - Lips corners distance - Mouth bounding box image norm Smile detector - Based on SVMs (LibSVM, Chang and Lin, 2001) - Kernel type: Radial Basis Function Trained with 512 samples Probabilit y of smile 5 subjects, leave-one-subject-out cross valida2on Accuracy: %

47 icat at work

48 Case Study 2

49 Case studied 2*: Scenario: A robot that plays a Tangram game with a user Goal: Maintain the user engaged. So, the robot needs to recognise engagement. Engagement is the process by which two (or more) participants establish, maintain and end their perceived connection during interactions they jointly undertake *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

50 Case studied 2: Initial Human Engagement Study In order to be able to create a classifier that detects engagement an initial study was conducted. A study of human engagement behavior in which pairs of humans sat across an L-shaped table from each other and prepared canape ś together. Each of four sessions involved an experimenter (confederate) and two study participants and lasted about minutes. *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

51 Case studied 2: Initial Human Engagement Study - In the first half of each session, the experimenter instructed the study participant in how to make several different kinds of canape ś using combinations of the different kinds of crackers, spreads and toppings arrayed on the table. - The experimenter then left the room and was replaced by a second study participant, who was then taught to make canape ś by the first participant.1 - All sessions were videotaped using two cameras. *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

52 Case studied 2: Initial Human Engagement Study, - Analysis of the videotapes (looking at engagement maintenance process) Analysis - During the periods of maintained engagement, the researchers coded where each person was looking at each moment. *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

53 Case studied 2: Initial Human Engagement Study, Analysis *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

54 Case studied 2: Development of the Social - Using the data captured, a system was developed to detect engagement with the robot. Robot *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

55 Case studied 2: Development of the Social Robot *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

56 Case studied 2: Development of the Social Robot *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

57 Case studied 2: Development of the Social Robot *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in humanrobot interaction. In Human-Robot Interaction (HRI), th ACM/ IEEE International Conference on (pp ). IEEE.

58 Case studied 2: Initial Human Engagement Study *Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010, March). Recognizing engagement in human-robot interaction. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

59 Case Study 3

60 Case studied 3*: Scenario: A robot that works in a shopping mall helping people. Goal: understand the users (customers of the shopping mall), be able to remember previous interactions, and understand their actions and moving patterns. *Glas, D. F., Wada, K., Shiomi, M., Kanda, T., Ishiguro, H., & Hagita, N. (2013, March). Personal service: a robot that greets people individually based on observed behavior patterns. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction (pp ). IEEE Press.

61 Case studied 3: Step one: data collection To obtain data about customers, a human tracking system was set up and two pan-tiltzoom video cameras in the entrance of a shopping mall. The cameras monitored two sets of sliding doors, and their video feeds were processed by a server running OKAO Vision facerecognition software *Glas, D. F., Wada, K., Shiomi, M., Kanda, T., Ishiguro, H., & Hagita, N. (2013, March). Personal service: a robot that greets people individually based on observed behavior patterns. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction (pp ). IEEE Press.

62 Case studied 3: *Glas, D. F., Wada, K., Shiomi, M., Kanda, T., Ishiguro, H., & Hagita, N. (2013, March). Personal service: a robot that greets people individually based on observed behavior patterns. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction (pp ). IEEE Press.

63 Case studied 3: Yet, because people interact with the robot in groups (crowds) a study of the type of movement they perform in a shopping mall will give information about the possible actions of the users; Yet, safe navigating among people is often studied from the perspective of constructing a collision free path, and not about the behaviour of the crowd itself. *Glas, D. F., Wada, K., Shiomi, M., Kanda, T., Ishiguro, H., & Hagita, N. (2013, March). Personal service: a robot that greets people individually based on observed behavior patterns. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction (pp ). IEEE Press.

64 Case studied 3*: Approach: Simulate the crowd and use that to predict and understand what is actually happening in the real world. *Kidokoro, H., Kanda, T., Brscic, D., & Shiomi, M. (2013, March). Will I bother here?-a robot anticipating its influence on pedestrian walking comfort. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

65 Case studied 3: Data Collection To calibrate the robot influence model, i.e. determine the ratio of persons belonging to each behavior category and the parameters of each behavior, pedestrian data in the same target environment was collected. The robot Robovie-II (120 cm high and 40 cm in diameter, maximum velocity 750 mm/s) roams back and forth on predefined paths in the corridor and hallway and observed people s responses, which were recorded. When a pedestrian stopped in front of the robot, a human operator halted the robot and waited until the pedestrian left. We collected trajectories and videos of pedestrians and the robot. *Kidokoro, H., Kanda, T., Brscic, D., & Shiomi, M. (2013, March). Will I bother here?-a robot anticipating its influence on pedestrian walking comfort. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

66 Case studied 3: Analysis 1. Coding: 1115 trajectories were collected and a human coder classified all trajectories into the four behavior categories by looking at videos and trajectories. Another coder conducted a validation coding for 10% randomly chosen cases. Their classifications were very consistent (Cohen's kappa coefficient was 0.991). 2. Analysis: Features 1. D interact and D observe from trajectories in "stop to interact" and "stop to observe" categories; in average people in these categories stopped at a distance of 0.92 m (S.D. is 0.25) and 1.51 m (S.D. is 1.18) from the robot, respectively. 2. "slow down to look" category, the change of the velocity was analized and found that their velocity around the robot was 76% of the average within 3.0 m from the robot; 3. D notice was considered to be 10 m, as an upper limit on the distance where significant interactions can occur. *Kidokoro, H., Kanda, T., Brscic, D., & Shiomi, M. (2013, March). Will I bother here?-a robot anticipating its influence on pedestrian walking comfort. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

67 Case studied 3: Predicting the comfort walking level in users in a shopping mall with a robot The data was used to predict walking comfort for users from trajectories. The factors that were considered to have significant influence were: Distance: Pedestrian models typically use (the inverse of) distance to compute the influence of other pedestrians. It was considered that it would be more comfortable for a pedestrian if distances to nearby persons are larger. Velocity: People prefer to walk with constant velocity when possible, so the change of velocity forced by congestion would decrease the walking comfort. We therefore considered the amount of change of velocity. Density: People could perceive congestion, i.e. high density of people in a narrow space as less comfortable, so we used the number of nearby persons as a possible factor. Preferred velocity: One can also argue that deviation from the desired velocity could harm comfort. *Kidokoro, H., Kanda, T., Brscic, D., & Shiomi, M. (2013, March). Will I bother here?-a robot anticipating its influence on pedestrian walking comfort. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

68 *Kidokoro, H., Kanda, T., Brscic, D., & Shiomi, M. (2013, March). Will I bother here?-a robot anticipating its influence on pedestrian walking comfort. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE. Case studied 3:

69 Case studied 3: *Kidokoro, H., Kanda, T., Brscic, D., & Shiomi, M. (2013, March). Will I bother here?-a robot anticipating its influence on pedestrian walking comfort. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

70 Case Studies 3: Results *Kidokoro, H., Kanda, T., Brscic, D., & Shiomi, M. (2013, March). Will I bother here?-a robot anticipating its influence on pedestrian walking comfort. In Human-Robot Interaction (HRI), th ACM/IEEE International Conference on (pp ). IEEE.

71 Commercial Tools for Facial and Body Expressions Analysis There are several multi-platform face recognition, identification and facial feature detection tools and are used for HRI. Examples OKAO* (by OMRON) Affdex** (by Affectiva) * **

72 OKAO OKAO Vision software suite from Omron Corporation, detects faces in images and can determine the person s gender and approximate age, or verify his or her identity from a database of faces. Omron used about 10,000 images of human faces some with spontaneous smiles, some with posed smiles, and others sporting different expressions to train the software to evaluate smiles. The smile -software is dedicated to facialexpression detection and analysis. Omron s smile-measurement software picks up the hallmarks of a smile (e.g. narrowed eyes, open mouth, creases around the mouth, and wrinkles turning downward around the eyes) and uses an algorithm to assess the extent of the smile and rate it on a percentage scale. *

73 Example of use of OKAO Functions of OKAO: 1. -Face Detection 2. -Face Recognition 3. -Gender Estimation 4. -Age Estimation 5. -Expression Estimation 6. -Facial Pose Estimation 7. -Gaze Estimation 8. -Blink Estimation 9. -Hand Detection 10.-Human Body Detection *

74 * An advert of OKAO

75 Affdex** by Affectiva Affdex measures emotional responses by capturing a user's facial expressions through an existing webcam, in real time. Tracking gestures and key points on the subject's face, Affdex is able to analyze subtle movements and correlate them with complex emotional and cognitive states. The Affdex system can identify and follow dozens of precise locations on an individual's face. The muscular micro-shifts of every smile, yawn, or moment of confusion are captured and reflected in the data. **

76 Example of use of Affdex by Affectiva **

77 What perception do your systems need?

78 Discussion