Affective Game Engines: Motivation & Requirements Eva Hudlicka Psychometrix Associates Blacksburg, VA hudlicka@ieee.org psychometrixassociates.com DigiPen Institute of Technology February 20, 2009 1
Outline Why we need Affective Game Engines? How can Affective Computing help? What are some of the functionalities needed? What might an Affective Engine look like Conclusions 2
Where We Are Now Tremendous advances in gaming technologies Focused primarily on: Physical realism of game characters & game environments Complexity & performance of simulations & networking Today s games still limited in: Affective realism game characters Social complexity & realism of their interactions Ability to adapt to player s state 3
In Terms of the Full Potential of Gaming.. We Are About Here 4
To Achieve the next big leap..in engagement & effectiveness Need to enhance social & affective complexity & realism of: Game characters Their interaction with each other and the players Game narrative as a whole Especially true for serious games: Education Training of affective & social skills Rehabilitation & therapy 5
So How Do We Get Here? We Are Here 6
How Do We Get There? Add a lot more affect Affect-focused game design Affective gaming Add it easily - develop the right tools Affective game engines Draw on research & technology base in affective computing Recognition Modeling Expression 7
Why Emotions? 8
Emotion is a Key Factor in both Play & Work Mediates motivation Influences memory & learning Key role in decision-making & problem-solving Central factor in engagement Key factor in serious games 9
Future Games Need To Recognize & adapt to players emotions Understand players affective profiles WE NEED DEVELOPMENT Increase affective complexity & autonomy of game characters TOOLS TO MAKE ALL THIS Increase visual affective EASY realism of game characters & player avatars and maybe even fun?) Increase affective complexity of the entire game experience 10
Need for Affective Game Engines Game development greatly enhanced by game engines Existing engines support construction of: Visually-realistic characters Visually realistic physical environments 3-D objects But so far, no engines support affective gaming 11
Affective Game Engines: Requirements What functionalities should affective game engines support? Emotion sensing & recognition Affective modeling of players Game characters w/ more affective & social realism Affective models guide character behavior Affective expression in game characters 12
Relevant Disciplines AI AI is more than RBS, A*, FSA Symbolic agent architectures Probabilistic reasoning (Bayesian belief nets) Learning Affective Computing Affective Computing Emotion recognition Affective modeling Emotion expression 13
Outline Why we need Affective Game Engines? How can Affective Computing help? What are some of the functionalities needed? What might an Affective Engine look like Conclusions 14
Overview of Affective Computing MAX (Becker, Prendinger et al.) Breazeal De Rosis Hudlicka 15
Intelligent Affective Game Characters Affective Virtual Character - Max Becker et al., 2005 16
Outline Why we need Affective Game Engines? How can Affective Computing help? What are some of the functionalities needed? What might an Affective Engine look like Conclusions 17
Affective Game Engines Should Help Design Games That. Recognize player emotion Adapt to player s emotions Have more realistic game characters Affective & Social realism React with appropriate emotion Express the emotion in a believable manner Whose behavior is influenced by the emotion 18
Emotion Recognition & Expression in Games Much recent progress in basic emotion recognition (fear, anger, joy, sadness) Multi-modal approaches approach human recognition rates Audio-visual; Audio-visual-physiological Recognition of spontaneous emotions Recognition of complex emotions (e.g., embarrassment) 19
Challenges in Game Contexts Different emotions in different game categories FPS vs. social games (Sims) vs. serious games for training / therapy Different platforms VR vs. Wii vs. Nintendo vs. iphone Non-intrusive sensors Noisy data Player movement Lighting conditions 20
Cohn 2006 21
Emotion Signatures: Modalities & Time Emotions occur across multiple modalities Expressive / Behavioral Cognitive Physiological Experiential Different emotions have distinct multimodal signatures 22
Emotion Effects on Behavior Facial expression Gestures Emotion Posture Behavior Blah blah blah 23
Trigger: Progress toward a goal hindered esp. by other agent Anger Cognitive: Focus attention (very strong effect) Assign blame to the perceived causal agent (typically another agent) Overestimate chances of own success Try alternate strategies www.firstpeople.us Physiological mobilize and sustain high energy levels: Higher diastolic blood pressure (than fear) Greater peripheral resistance (than fear) Larger increase in heart rate (than disgust) Larger heart rate acceleration (than happiness) Larger increase in finger temperature (than fear) Behavioral: Eagerness to act Fight & aggression Social: prevent (or facilitate) aggression 24
Emotion Signatures: Modalities & Time Temporal & spatial congruence makes emotion recognition easier Multimodal approaches more successful Visual (face) + audio (speech) Physiological (arousal) + visual 25
Emotion Signatures: Modalities & Time Requirements for temporal & spatial congruence make emotion expression more difficult Facial expression, tone, words, gestures All must agree to be believable across temporal intervals 26
Semantic Primitives for Recognition & Expression Identify primitives for each channel to facilitate recognition & expression Associate each emotion with a specific configuration of primitives Some channels have established primitives Facial expressions (established & mature) Speech (generic signal properties) Posture (emerging) Movement (being adopted from choreography - Laban) 27
Facial Action Coding Units Basic emotions reflected in the face via characteristic configurations of facial muscles Shape of lips Shape of eyebrows Narrowing of eyes Raising cheeks Facial action coding system (FACS) provides codes for these configurations (Ekman & Friesen, 1978) Action Units (AU s) correspond to individual muscle positions & movements All possible expressions can be analyzed in terms of AU s 28
FACS-Coded Facial Expression Movellan et al. http://mplab.ucsd.edu/ 29
Modality, Channel & Sensor Selection Criteria Which emotions need to be recognized / expressed? What are their signatures along each channel What semantic primitives are available Facial action units Speech? Posture? Physiological data (arousal, EEG) What sensors are available Non-intrusive Generate adequately clean raw data Appropriate for context & platform 30
Sensors for Gaming Emsense: EEG, heart rate, respiration, head motion, temperature Neurosky Emotiv Systems EEG & facial movements 31
Non-Intrusive GSR Sensing 32 Picard et al., 2008
Computational Affective Modeling Agent architectures for game characters Control NPC behavior More affective & social realism More believable characters & their interactions More engaging & effective games Affective User Models Models of player affective profiles Support recognition of player s affective state Support adaptation to player s state 33
Affective Agent Architectures Enable game characters to: React to evolving situations in game React to other characters in game React to player s state and behavior by dynamically generating appropriate emotions which influence decision-making & behavior and by supporting their realistic display 34
Affective Agent Architectures Control Game Character Behavior Generation of Emotions (via cognitive appraisal) Effects of Emotions (on cognition & behavior) Stimuli Agent Architecture Emotions 35
MAMID Architecture: Semantics & Data Flow Cues Attention Situation Assessment Expectation Generator Affective state & emotions: Valence: Negative Happiness: Low Fear: High Cues: State of the world Growling dog approaching Situations: Perceived state Aggressive dog Affect Appraiser Expectations: Expected state Dog will bite me Goal Manager Goals: Desired state Avoid being bitten Action Selection Actions: to accomplish goals Climb a tree Actions 36
How Difficult Is This? Depends on game complexity game type.. Which emotions are necessary? What features of the game context are available to trigger an emotion? Simple games may not need much Sophisticated social games & serious games need: More emotions Real-time generation of appropriate emotion Realistic influence of emotion on perception + cognition Real-time expression of appropriate emotion More realistic affective dynamics 37
How Do We Do It? Black-box models Stimulus ---> Emotion Simple but clunky - does not generalize Process models Explicit models of some underlying processes Emotion generation Emotion effects on Perception Decision-making Behavior Expression 38
Black Box Models Directly map stimuli onto emotions: Character gains points ---> Happy Character loses points ---> Sad Character outsmarted ---> Angry Character ridiculed -->????? Ooops! No rule for that one Now what? 39
Process Models Model underlying mechanisms of emotion generation Cognitive appraisal Stimuli ---> Appraisal Dimensions ---> Emotion More general - more extensible Stimuli Goals Situations Expectations Cognitive Appraisal Emotions 40
STIMULI Appraisal Variables FEAR Novelty Valence Goal relevance Agency Outcome probability Goal congruence Urgency Coping potential high low high other high low v. high low Norms 41
Affective User Models Represent player s affective profile What makes them: Happy? Frustrated? Bored? Engaged? Angry? How is this manifested in THIS player? How do we find this out? Learning algorithms Baseline data Training period How do we represent the info Augmented state transition diagrams 42
Outline Why we need Affective Game Engines? How can Affective Computing help? What are some of the functionalities needed? What might an Affective Engine look like Conclusions 43
So What Does All This Suggest for the Affective Game Engine Requirements? 44
Emotion Sensing & Recognition Affective User Modeling Central Shared Emotion Knowledge-Base Modelling of Game Character Emotions Emotion Expression In Game Characters 45
Shared Emotion Knowledge-Base Generic knowledge about emotions Triggers, influences, manifestations Affective model of player Idiosyncratic triggers, influences, manifestations Affective models of game characters As above Start with basic emotions Include states relevant for gaming / training Surprise, boredom, engagement, flow Progress to social & more complex emotions 46
Possible KB Structure Generic Schema - Organized in an inheritance hierarchy - Different modules access different slots - Not all slots needed by each modules Instantiated Schema 47
Outline Why we need Affective Game Engines? How can Affective Computing help? What are some of the functionalities needed? What might an Affective Engine look like Conclusions 48
Summary Games need more focus on emotion Affective game engines would help develop affective games Recognition & generation of arbitrary emotions still difficult.. Esp. in noisy, realistic contexts But we re getting there 49
Conclusions Is it too early for affective game engines? Different features grouped in different tools analogous to today s engines Concrete requirements support systematic design of affective games and the tools for their development Facilitate development of better games Provide platforms for research in AI, Aff Comp, Psychology, 50
Thank you Eva Hudlicka Psychometrix Associates Blacksburg, VA hudlicka@ieee.org psychometrixassociates.com 51