Empirical Cognitive Modeling

Transcription

1 Empirical Cognitive Modeling Tanja Schultz Felix Putze Dominic Heger Lecture Cognitive Modeling (SS 2012) 1/49

2 Outline What are empirical cognitive modeling and recognition of human cognitive and affective states? General architecture for empirical cognitive modeling Training data generation Example: Emotion recognition from bio-potential signals Interplay of empirical modeling and formal cognitive models 2/49

3 What is Empirical Cognitive Modeling? Empirical Science Knowledge must be based on data/information from observations or experiments Empirical Cognitive Modeling is data-driven approach of learning a model of a user state from empirical data (e.g. sensory data) User States are results of a person s mental processes at a particular point of time Mental or physiological configurations of a human Reflecting universal patterns of affect or cognition Usually have behavioral implications 3/49

4 Human Cognitive States Cognitive States States of information processing Examples: Workload Relax -> load -> overload Stress, flow Activities Listening, reading, contemplating, Many others: Constitutions and personality Intentions, motivations 4/49 Four antique temperaments: choleric; melancholic; sanguine; phlegmatic (wikimedia)

5 Human Affective States Affective States States of the experiencing of feeling or emotion Emotions Discrete basic emotion models: Anger, fear, sadness, surprise, joy, disgust, Dimensional models: Valence, arousal, dominance Appraisal models Mood Dimensional models: Good/bad, awake/tired, nervous/calm Boredom, anxiety Disorders: maniac, major depressive disorder, 5/49 Basic emotions by Paul Ekman

6 Goals Assessment of the current state of a person is important aspect to enrich human machine interaction (adaptive interfaces) Human needs are widely ignored in today s human machine interaction Natural communication and implicit interfaces (machine integrates into human interaction styles) Entertaining interaction (e.g. gaming) Empathic behavior (e.g. humanoid robots) Intelligent assistants ARMAR III thinkinghead.edu.au Social robotics 6/49

7 Other Applications Evaluation of usability and product quality Self assessments by questionnaires are often difficult Get quantitative physiological measures Medical/Psychological use Communication for locked in patients, people with autism, Diagnosis New (neuro-)psychological insights Others: Virtual realities Smart surveillance Edutainment, 7/49

8 How to assess cognitive/affective states? How do humans do that? Theory of mind : The ability to make inferences about others mental states (Premack & Woodruff, 1978) Methods Dialog, questionnaires How are you doing, today? Difficult to interpret, easy to fool Context, environment Driver might be tired after 10 hours in the car Strongly situation dependent Biosignals and their correlates Known correlations between physiological measures and user states 8/49

9 Modalities Biosignals are messages that originate from physical (or chemical) actions of the human body Biosignals and correlates for user state recognition Brain activity (EEG, fmri, NIRS, ) Eye activity (EOG, eye tracking) Heart rate (ECG, PPG) Muscle activity (EMG) Speech (Prosody, choice of words, ) Skin conductance (EDA) Body temperature Vision (Facial expressions, posture, gesture, motion, ) More details on biosignals are presented in our winter term lecture Biosignale und Benutzerschnittstellen 9/49

10 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Integrating User States in cognitive models 10/49

11 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Integrating User States in cognitive models 11/49

12 Taxonomy of Biosignals Measurable Biosignals and their correlates Mechanical Thermal Electrical Chemical Acoustic Gestures Warmth Body sounds Mimics Motion Brain Eyes Muscles Heart Nonverbalarticulation Speech 12/49

13 Multimodality Multimodal user state recognition systems use information from more than one input modality Pro: Con: More information (higher recognition accuracy) Redundant information (higher robustness) Continuously available stream of input data (many signals are not emitted all the time, e.g. speech) Higher system complexity Need synchronization of input data streams Need suitable fusion scheme More data needs to be processed (cpu, memory) 13/49

14 Acquisition of Physiological Signals: EDA Rise Electrodermal Acticity (EDA) = Skin conductance (SC) Mostly dependent on activity of sweat glants Related to physical activity and physiological arousal Reacts to stimuli with shard rises of the signal EDA Amplitude Latency Half Life 50% Decay 37% 14/49

15 Acquisition of Physiological Signals: BVP Blood Volume Pressure (BVP): Measures change of blood volume in a extremity (e.g. finger, ear) Signal measured using a photoplethysmyograph Based on optical properties of blood Send red light into the tissue, capture the reflected light Can be used to deduce heart rate (frequency) and blood pressure (amplitude) 15/49

16 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Integrating User States in cognitive models 16/49

17 Pre-Processing - Framing Cut stream of digital input data into suitable portions (frames) for recognition Design frames to represent smallest information entity of the recognition system Frame length depending on modality and features to be extracted Time duration for stable features (not too short and not too long) Few milliseconds for phoneme parts in speech recognition Multiple seconds for variabilities in skin conductance Overlapping Compensate small slopes of window functions Higher temporal resolution 17/49

18 Preprocessing - Artifacts Recorded sensor data is often highly contaminated with noise, artifacts and outliers Artifact filtering techniques suitable for modality and feature extraction Drop data if it is not within the range of physiological possible values (e.g. EEG data larger than 150 μv) Decompose the signal by source separation techniques (e.g. Independent Component Analysis) to identify artifact components and reconstruct the signal without those components (e.g. EEG eye artifacts) 18/49

19 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Integrating User States in cognitive models 19/49

20 Feature extraction Extract relevant information from the raw data and represent them in a certain vector space Statistical properties of the signal portion Mean, variance Histogram values Exploit sensor and task specific domain knowledge E.g. 2 nd derivative of a fitted 2 nd order polynomial x 1... x 4 x 2 x 3 20/49

21 Feature extraction Often the following techniques are applied within feature extraction: Filtering / Filter banks Finite Impulse Response, Infinite Impulse Response, Spectral transformation FFT, Wavelets, Feature selection or dimensionality reduction Sequential Forward Selection, Principal Component Analysis, Filtering based on mutual information, correlation, Normalization Z-scores, Linear scaling within a certain value range, 21/49

22 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Integrating User States in cognitive models 22/49

23 Pattern Recognition Classification Mapping between feature space and discrete number of classes Techniques: Naïve Bayes, SVM, ANN, DT, Example: Recognition of discrete emotions Regression Model for functional description of the features Continuous output variables Techniques: Ordinary least squares, generalized linear regression, Example: Recognition of continuous workload Reinforcement learning Find suitable actions to take in a given situation in order to maximize a reward We will have a complete lecture on this topic 23/49

24 Sequence modeling User state can have temporal dynamics (e.g. facial expressions in videos) Model dynamic progression of elementary patterns Techniques: Hidden Markov Models (HMMs) States, observations, state transitions Dynamic Bayesian Nets (DBNs) Directed Acyclic Graphs Edges represent conditional probabilities Tracking algorithms (Kalman filters, Particle filters, ) Recursively apply system model Update state estimation by (noisy) observations 24/49

25 Pattern Recognition Taxonomy of pattern recognition Algorithm examples: Artificial Neural Nets (ANN) Decision Trees (DT) Gaussian Mixture Models (GMM) Generalized Linear Regression (GLR) Support Vector Machine (SVM) using RBF kernel 25/49

26 Which pattern recognition algorithm to use? Comparisons of different general purpose recognition algorithms often show similar performances Differences in flexibility are equally important Adaptability Amount of training data needed Training convergence Computational costs (training and recognition) More details of the different pattern recognition algorithms: Pattern recognition lecture (Prof. Beyerer) Machine learning lectures (Prof. Dillmann, Prof. Zöllner) Bishop: Pattern Recognition and Machine Learning Duda, Hart, Stork: Pattern Classification 26/49

27 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Integrating User States in cognitive models 27/49

28 Sensor Fusion Features usually have different stochastic distributions Few features can dominate, e.g. with high values and variance Need to be normalized to be comparable Data from different sensors are not received at the same time Different sampling rates Mapping of corresponding features Interpolation of features Unsynchronized recording Use synchronization channel (marker) Timestamps synchronized by Network time protocol Fusion level Data level: Combine raw data Feature level: Concatenate feature vectors Decision level: Combine recognition results 28/49

29 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Output: Recognized user state 29/49

30 Training Data Acquisition To train (statistical) models supervised recognition algorithms need labeled data Often: There is no data like more data But equally important: High data quality Typically recorded sensor data has noise, artifacts, outliers, incomplete data, Clever design of system setup and sensors can help Normalization and signal processing (e.g. ICA for EEG eye artifacts) Good coverage of all situations in the training data Training situations should be as similar as possible to run-time situations Representative training data Problem: Real world training data is rare and hard to collect and acted data is unnatual Emotion Elicitation 30/49

31 Elicitation of User states Approach: Induce cognitive/affective states using user study experiments Example: International Affective Picture System (IAPS) for emotion elicitation + Large data collection (pictures, videos, sounds) + Freely available for research purposes + Evaluated using questionnaires + Simple experimental setup Unrealistic experimental situation People are differently affected (ground truth?) Ambiguities in stimuli Fixation Relaxation 2 sec 6 sec 15 sec 31/49

32 Wizard of Oz Paradigm Collect experimental data for building a new system for Human-Machine-Interaction Chicken and egg problem: Enrollment of a new system that can not be built (trained), yet Wizard of Oz Experiment for data collection: Users thinks to interact with an autonomous system A human operator (wizard) controls the system The users are not conscious about this fact Pros: Cons: The recorded interactions can be used as training data Gain insights on human behavior in certain interactions Hard to design Often still differences to real world scenarios Time-consuming (only possible to collect small amounts of data) 32/49

33 Generating Ground Truth For training (supervised) statistical models, we need to assign ground truth labels to the data A-Priori assignment E.g. Setting is designed to create the emotion anger label it as such What if we are wrong about the effect of our treatment? Rating by Judges Have (naïve or expert) judges watch/listen to the data and rate it Relies on humans as experts Judges cannot peek insight a subject s mind Rating by the Subjects Have subjects label the data themselves Yields their true subjective emotions Approach 1: During the experiment, e.g. think-aloud (intrusive) Approach 2: Label afterwards (memory might betray subjects) 33/49

34 More on experiment design and evaluation in our winter-term lecture Design and Evaluation of innovative User Interfaces 34/49

35 Example: Multimodal Emotion Recognition K. Takahashi (2004): Remarks on emotion recognition from bio-potential signals Emotion recognition from multimodal biosignals Discrete emotions: joy, anger, sadness, fear, and relax Modalities: Frontal brain activity using an EEG head band Pulse at earlobe by Photoplethysmography (PPG) Skin conductance at finger tips (EDA) 35/49

36 Emotion recognition from bio-potential signals Data collection: Audio-visual stimuli presented at a PC 10 film clips (evaluated to be emotional) 12 subjects, age: years Feature extraction: Data: EDA, PPG: Time domain signals EEG: Three frequency bands Calculated features for each modality: Six statistical features: mean, standard deviation, means of the absolute value of the 1. and 2. differences, Feature fusion, normalized by their standard deviation 36/49

37 Emotion recognition from bio-potential signals Recognition: Support vector machines for discriminating the 5 emotions Leave one out cross-validation Average recognition rate is rather low (41.7%) but far above chance level The multimodal system using all three sensors gives best accuracies for joy, anger, and fear 37/49

38 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Integrating User States in cognitive models 38/49

39 Recognition of Cognitive and Affective States Up to now: Recognition of cognitive and affective states is based on interpretation of low-level biosignal features (pure bottom-up approach) Problems of the presented general architecture What is an adequate system reaction to e.g. a certain emotion of a user? Can we recognize all cognitive/affective states in the described way? Appraisal based emotions? Multiple user states can co-occur at one point of time Same biosignals can have different meaning in different contexts Same user states can emit different biosignals in different contexts Ignores intentions/goals/tasks/ that might influence user s perception We need to model influences of additional background knowledge (top-down) Cognitive Modeling 39/49

40 Combination of Empirical and Formal Models Combine Empirical Model with knowledge contained in formal cognitive model Augment recognition with information from formal cognitive model Augment formal model with live information from empiric recognizer Results in feedback loop: Prediction Formal Model Empirical Model Observation Way to integrate a validated a-priori knowledge with state-ofthe-art empirical models 40/49

41 Integration of Arousal in Cognitive Architecture Update formal cognitive model of arousal using recognized arousal from electrodermal activity Arousal describes reactivity to stimuli e.g. lethargic calm engaged - hyperactive Arousal has impact on memory (e.g. Yerkes-Dodson law) Correlation between electro-dermal activity and arousal Adaptation of the (ACT-R) memory model match the current recognized arousal state of the person 41/49

42 Acquisition of Physiological Signals: EDA Rise Electrodermal Acticity (EDA) = Skin conductance (SC) Mostly dependent on activity of sweat glants Related to physical activity and physiological arousal Reacts to stimuli with shard rises of the signal EDA Amplitude Latency Half Life 50% Decay 37% 42/49

43 Arousal Impact on Memory Experiment by Kleinsmith and Kaplan (1964) Pared associations task: Present 8 emotional words together with a number Arousal determined by electrodermal activity (EDA) for each word Subjects should recall pared associations of the 3 words with highest arousal 3 words with lowest arousal Recall at 2min, 20 min, 45 min, 1 day, 1 week Surprising results Low arousal: Recall rate drops High arousal: Initial blocking Words are better recalled after some time Experiment has been criticized as EDA might have measured not only arousal effects (Mather, 2007) 43/49

44 Activation of memory in ACT-R model Remember ACT-R memory consists of chunks Each chunk has base-level activation Activation equation has no parameter for arousal Resulting activation (d=0.05) Equation doe not model is not high arousal curve adequately 44/49

45 Adaptation of Activation Equation to match Adaptation of the base-level activation equation Replace decay parameter d by Linear function of arousal at time of encoding a => d d ( 1 a/ a ) 0 i.e. decay becomes growth a h ( a a ) n s h The term causes initial blocking for high arousal levels 45/49

46 Resulting Activation Resulting curves For typical parameters Scaling factor Nominal value High arousal threshold 46/49

47 General Architecture for User State Recognition Input: Multimodal digital sensor data Pre-processing Feature extraction Recognition Sensor Fusion Integrating User States in cognitive models 47/49

48 Wrap up Cognitive and affective states recognition can be involved but important for many human-computer-interaction systems General architecture Empirical Cognitive Modeling Connect empirical and formal Cognitive Modeling Empirical modeling is exciting research area We have lots of interesting Bachelor and Master theses 48/49