EMBODYING GESTURE PROCESSING A Computational Cognitive Model for Humanoid Virtual Agents

Size: px

Start display at page:

Download "EMBODYING GESTURE PROCESSING A Computational Cognitive Model for Humanoid Virtual Agents"

Anne Harrison
6 years ago
Views:

1 EMBODYING GESTURE PROCESSING A Computational Cognitive Model for Humanoid Virtual Agents Amir Sadeghipour PhD Sociable Agents Group Bielefeld University, Germany Supervisor: Prof. Stefan Kopp

2 Intro Investigating human social behavior

3 Intro Investigating human social behavior Psychology, Neurobiology, Linguistic & Philosophy

4 Intro Investigating human social behavior Psychology, Neurobiology, Linguistic & Philosophy Computational cognitive model of social behavior for humanoid virtual agents

5 Outline Goal: Gesture Processing Computational Cognitive Model Results Conclusion Outlook

6 Outline Goal: Gesture Processing Computational Cognitive Model Results Conclusion Outlook

7 Goal Motor Cognition encompasses all processes involved in the production and comprehension of one s own and others actions

8 Goal Motor Cognition encompasses all processes involved in the production and comprehension of one s own and others actions Planning Understanding Anticipating Generating Recognizing Perceiving

9 Goal

10 Goal Grounded cognition: Brain s modal simulation underlies cognition

11 Goal Grounded cognition: Brain s modal simulation underlies cognition Simulation: The reenactment of the neural representation networks, which arises during perceiving and acting

12 Goal Grounded cognition: Brain s modal simulation underlies cognition Simulation: The reenactment of the neural representation networks, which arises during perceiving and acting Embodied cognition: Cognition is grounded in the same neural representations that underlie perception, action and imagination

13 Goal Requirements dimensions Perception Generation

14 Goal Requirements dimensions Semantics Perception Generation Motor Capabilities

15 Goal Requirements dimensions Semantics Perception Understanding Perception Generation Motor Capabilities

16 Goal Requirements dimensions Semantics Intention Generation Perception Generation Motor Capabilities

17 Goal Requirements dimensions Semantics Perception Generation Perception Generation Motor Capabilities

18 Goal Requirements dimensions Semantics Perception Generation Perception Mimicry Generation Alignment Priming Motor Capabilities

19 Goal Requirements dimensions Semantics Perception Generation Perception Mimicry Generation Alignment Priming Motor Capabilities

20 Goal Requirements dimensions Semantics Simulation Emulation Perception Generation Perception Imitation Mimicry Generation Alignment Priming Motor Capabilities

21 Goal Requirements dimensions Semantics Simulation Emulation Perception Generation Perception Imitation Mimicry Generation Alignment Priming Motor Capabilities

22 Outline Goal: Gesture Processing Computational Cognitive Model Results Conclusion Outlook

23 Model The overall model Sensors Perception Shared Motor Knowledge Generation Actuators

24 Model The overall model Sensors Perception Shared Motor Knowledge Generation Actuators

25 Model The overall model > Shared Motor Knowledge Shared Motor Knowledge Forward Models Motor schemas Motor Programs Motor Commands Inverse Models

26 Model The overall model > Shared Motor Knowledge Shared Motor Knowledge Forward Models Motor schemas Motor Programs Motor Commands Inverse Models

27 Model The overall model > Shared Motor Knowledge > Gesture Representation Motor Schemas Motor Programs Motor Commands

28 Model The overall model > Shared Motor Knowledge > Gesture Representation Motor Schemas Motor Programs waving1 waving2 circle1 Motor Commands

29 Model The overall model > Shared Motor Knowledge > Gesture Representation Motor Schemas waving circle Motor Programs waving1 waving2 circle1 Motor Commands

30 Model The overall model > Shared Motor Knowledge Shared Motor Knowledge Forward Models Motor schemas Motor Programs Motor Commands Inverse Models

31 Model The overall model > Shared Motor Knowledge > Forward Models Forward Models Shared Motor Knowledge Motor schemas Motor Programs Motor Commands Inverse Models

32 Model Forward models P (h o) =αp (o h)p (h)

33 Model Forward models P (h o) =αp (o h)p (h) P T (h o) := 1 T T αp t (o t h)p t (h) t=1

34 Model Forward models P (h o) =αp (o h)p (h) P T (h o) := 1 T T t=1 αp t (o t h)p t (h) P T (h o) := 1 T T αp t (o t h)p t 1 (h o t 1 ) t=1

35 Model Forward models: Bottom-up perception MS MP MC l MC r

36 Model Forward models: Bottom-up perception MS MP P T (mc o) := 1 T T αp t (o t mc)p t 1 (mc o) t=1 MC l MC r

37 Model Forward models: Bottom-up perception MS P T (mp o l,o r ):= 1 T T t=1 i {l,r} mc MC i αp t (o t,i mc)p (mc mp)p t 1 (mp o l,o r ) MP P T (mc o) := 1 T T αp t (o t mc)p t 1 (mc o) t=1 MC l MC r

38 Model Forward models: Bottom-up perception P T (ms o l,o r ):= 1 T T αp t (o t mc)p (mc mp)p (mp ms)p t 1 (ms o l,o r ) t=1 i {l,r} mc MC i mp MP MS P T (mp o l,o r ):= 1 T T t=1 i {l,r} mc MC i αp t (o t,i mc)p (mc mp)p t 1 (mp o l,o r ) MP P T (mc o) := 1 T T αp t (o t mc)p t 1 (mc o) t=1 MC l MC r

39 Model Forward models: Bottom-up perception P T (ms o l,o r ):= 1 T T αp t (o t mc)p (mc mp)p (mp ms)p t 1 (ms o l,o r ) t=1 i {l,r} mc MC i mp MP MS P T (mp o l,o r ):= 1 T T t=1 i {l,r} mc MC i αp t (o t,i mc)p (mc mp)p t 1 (mp o l,o r ) MP P T (mc o) := 1 T T αp t (o t mc)p t 1 (mc o) t=1 MC l MC r

40 Model Forward models: Bottom-up perception MS MP P T (mc o) := 1 T T αp t (o t mc)p t 1 (mc o) t=1 MC l MC r

41 Model Forward models: Bottom-up & Top-down P T (ms o l,o r ):= 1 T T αp t (o t mc)p (mc mp)p (mp ms)p t 1 (ms o l,o r ) t=1 i {l,r} mc MC i mp MP MS P T (mp o l,o r ):= 1 T T t=1 i {l,r} mc MC i αp t (o t,i mc)p (mc mp)p t 1 (mp o l,o r ) MP P T (mc o) := 1 T T αp t (o t mc)p t 1 (mc o) t=1 MC l MC r

42 Model Forward models: Top-down belief guidance MS P t (mp MS):= ms MS αp (mp ms)p t (ms)p t 1 (mp MS) MP P t (mc MP):= αp (mc mp)p (mp ms)p t (ms)p t 1 (mc MP) ms MS mp MP MC l MC r

43 Model The overall model > Shared Motor Knowledge Shared Motor Knowledge Forward Models Motor schemas Motor Programs Motor Commands Inverse Models

44 Model The overall model > Shared Motor Knowledge > Inverse Models Forward Models Shared Motor Knowledge Motor schemas Motor Programs Motor Commands Inverse Models

45 Model The overall model > Shared Motor Knowledge > Inverse Models Forward Models Shared Motor Knowledge Motor schemas Motor Programs Motor Commands Inverse Models

46 Model Inverse models: Learning motor skills through SOM

47 Model The overall model > Shared Motor Knowledge Shared Motor Knowledge Forward Models Motor schemas Motor Programs Motor Commands Inverse Models

48 Model The overall model Sensors Perception Shared Motor Knowledge Generation Actuators

49 Model Integration of perception and generation through neural activation

50 Model Integration of perception and generation activation(m, t) = 1, m is being performed at t P t (m), m is being observed with probability P at t decrease(m), otherwise

51 Model Integration of perception and generation activation(m, t) = 1, m is being performed at t P t (m), m is being observed with probability P at t decrease(m), otherwise Perception (t) Generation (t) Perception (t-1) Generation (t-1)

52 Model Integration of perception and generation activation(m, t) = 1, m is being performed at t P t (m), m is being observed with probability P at t decrease(m), otherwise Perception (t) Generation (t) Perception (t-1) Self-Priming Generation (t-1)

53 Model Integration of perception and generation activation(m, t) = 1, m is being performed at t P t (m), m is being observed with probability P at t decrease(m), otherwise Perception (t) Generation (t) Perception (t-1) Self-Priming Motor resonance (Priming, Alignment) Generation (t-1)

54 Model Integration of perception and generation activation(m, t) = 1, m is being performed at t P t (m), m is being observed with probability P at t decrease(m), otherwise Perception (t) Generation (t) Perception (t-1) Self-Priming Motor resonance (Priming, Alignment) Generation (t-1) Perceptual resonance

55 Model Integration of perception and generation activation(m, t) = 1, m is being performed at t P t (m), m is being observed with probability P at t decrease(m), otherwise Perception (t) Generation (t) Perception (t-1) Self-Priming Motor resonance (Priming, Alignment) Generation (t-1) Perceptual resonance Self-Priming

56 Results Goal: Gesture Processing Computational Cognitive Model Results Conclusion Outlook

57 Results Demo1: Recognition & Generation

58 Results Demo2: Imitation learning

59 Results Demo2: Recognizing

60 Results Motor resonance = Embodied recognition Bottom-Up

61 Results Motor resonance = Embodied recognition Bottom-Up Top-Down + Bottom-Up

62 Results Perception Generation Perception Generation Generation Perception

63 Results Goal: Gesture Processing Computational Cognitive Model Results Conclusion Outlook

64 Conclusion A computational cognitive model for humanoid virtual agents, to perceive, recognize (embodied), generate and learn hand gestures The integration of perception and generation at different levels of abstraction, which accounts for different social capabilities and characteristics.

65 Results Goal: Gesture Processing Computational Cognitive Model Results Conclusion Outlook

66 Outlook The overall model Sensors Perception Shared Motor Knowledge Generation Actuators

67 Outlook The overall model Shared Motor Knowledge Sensors Perception MS Generation Actuators MP MC

68 Outlook Higher level? Shared Motor Knowledge? Sensors Perception MS Generation Actuators MP MC

69 Outlook Cross-modal interaction Shared Multimodal Knowledge Sensors Perception MS MP? Generation Actuators MC Motor Visual

70 Outlook Representation of concepts Shared Multimodal Knowledge Sensors MS MP?? Perception Generation Actuators MC Motor Visual Language

71 Thanks!

M.Sc. in Cognitive Systems. Model Curriculum

M.Sc. in Cognitive Systems. Model Curriculum M.Sc. in Cognitive Systems Model Curriculum April 2014 Version 1.0 School of Informatics University of Skövde Sweden Contents 1 CORE COURSES...1 2 ELECTIVE COURSES...1 3 OUTLINE COURSE SYLLABI...2 Page