Subsystem synchronization and multimodal behavioral organization

Size: px

Start display at page:

Download "Subsystem synchronization and multimodal behavioral organization"

Dominick Neal
6 years ago
Views:

1 Synchronization Workshop HUMAINE/SCAS Summer School 2006, Genova Subsystem synchronization and multimodal behavioral organization Klaus R. Scherer Swiss Center for Affective Sciences

2 What is an emotion?

4 What is an emotion?

5 A definition of emotion Emotion can be reasonably considered as an episode of massive, synchronized recruitment of mental and somatic resources allowing to adapt to or cope with a stimulus event subjectively appraised as being highly pertinent to the needs, goals, and values of the individual

6 EVENT Emotion is synchronisation

7 Wh-Questions to be raised Why should synchronization occur? Who gets synchronized? Where does this happen? When is synchronization initiated? What is the organizing principle? How is synchronization and multimodal behavioral organization accomplished? What are its effects on interpersonal perception and communication?

8 Why should synchronization occur? Emotion facilitates adaptation to important events by preparing appropriate action emotion is elicited by organized appraisal efferent effects of appraisal occur in a coordinated, time-locked fashion multiple systems are coordinated to allow efficient action production of patterned signals for communication requires multimodal organization

9 Who gets synchronized? All behavioral subsystems: cognition (information processing) motivation (proactive and reactive) support system (NES, ANS) motor execution (SNS) supervisory and monitoring systems social interactive systems

10 Where does this happen? At different levels of behavioral organisation: electrical activity of brain cell assemblies discharge of neurotransmitters and hormones ANS and SNS activity expressive signal patterns instrumental motor behavior interactive interpersonal behavior

11 When is synchronization initiated? Serial volleys of sync impulses with different impact strength and pervasiveness brief affect burst onset of a major emotion episode effects of recursive re-appraisals effects of regulation decisions

12 What is the organizing principle? Push vs. Pull mechanisms Push: biologically determined externalization of naturally occurring internal processes of the organism, particularly information processing and behavioral preparation Push: socioculturally defined templates for instrumental behavior or communicative acts, controlled by strict production rules and contextual expectations

13 Push effects Production principle: Evolutionary continuous automatic response patterning. Relatively strong interindividual differences in the expressive patterns (as the underlying biological processes are dependent on both the idiosyncratic nature of the individual and the specific nature of the situation) Sync mechanism: automatic elicitation of simultaneous, coordinated activation of different neuroendocrine and neuro-motor systems in the service of immediate protective adaptation

14 Push effects Examples for pure push effects: animal expressions infant grunts affect bursts sudden, uncontrolled emotions

15 Evolutionary origin of affect expression Darwin confirmed A.K. Anderson, J. Susskind, A. Cusi, W. Grabski Facing danger: Facial expression form and sensory function

16 Pull effects Production principles: socioculturally determined norms or moulds concerning the signal characteristics required by the socially shared codes for the communication of internal states and behavioral intentions, including linguistic rules for the encoding of syntactic, semantic, and pragmatic aspects of meaning. Characterized by a high degree of symbolization and conventionalization, and thus comparatively few and small individual differences. Sync mechanism: controlled elicitation of simultaneous, coordinated activation of different neuromotor systems in the service of consciously represented action goals or communicative signals.

17 Pull effects Examples for pure pull effects: sound symbolism symbolic coding systems, language conventionalized expression rules mimicking push effects constraints of communication channels

18 How is synchronization accomplished? duration automatic control effortful push pull reflex motor acts: freezing affect bursts: gasps, snorts involuntary speech acts interjections simulated emotion expression planned verbal utterances consciously prepared motor display rehearsed emotional advocacy speech culturally stereoptyped signals or messages hostile interpersonal stance

19 How is synchronization accomplished? Tonic changes in different modalities a change in baseline for a relatively lengthy period of time, with possibly rapid onset but slow offset. Relationships between modalities can be described through covariation principles (e.g., correlation, regression) Phasic bursts coordinated across different modalities are generally indicative of meaningful configuration effects, as found in complex signals

20 Tonic vs. phasic sync Covariation vs. configuration effects Tonic Covariation Phasic Configuration

21 Differential coding of parametric features Different features of prosody can be coded differentially Scherer, Ladd, & Silverman (1984) have shown that F0 range is coded continuously whereas many intonation contours are coded configurationally Covariation Configuration Challenging z scores WH question Y/N question Levels of F0 Range emphatic involved aroused annoyed Final fall Final rise

22 Toward a multimodal definition of units of analysis Face Gestures Body Voice Speech Body position units (defined by static support patterns) Vocal action units (defined by voice production mechanisms) Speech acts Segmental Facial action units (defined by muscle units) Gestural action units (defined by hand-arm movement articulators) Suprasegmental Variable durations and velocities from onset over apex to offset Variable durations and velocities from onset over apex to offset Durations and extent Continuous changes in acoustic parameters Variations in speed, rhythm, etc.

23 Multimodal analysis of actor portrayals Scherer, K. R. & H. Ellgring (in press a). Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? Emotion. Scherer, K. R. &, H. Ellgring (in press b). Multimodal Expression of Emotion: Affect Programs or Componential Appraisal Patterns? Emotion. Out of a 14 (Emotion) x 6 (Actor) x 2 (Sex of Actor) x 2 (Sentence) x 2 (Scenario) factorial design, yielding a total of 1,344 samples 224 were chosen on the basis of naturalness and recognizability ratings

24 Examples of emotion portrayals produced by professional actors 9 actors/speakers 2 «sentences" hätt sandik prong nju ventsie fi gött leich jean kill gos terr actrice 2 acteur 3 acteur 4 actrice 7 acteur 8 actrice 9 actrice 10 actrice 11 acteur 12 cold anger (irrit) hot anger (rage) sadness (trist) despair (desp) anxiéty (anx) panic fear (paniq) calm joy (joie) elated joy (exalt) Banse & Scherer, 1996

25 Variables of the multimodal analysis FACE Facial action units AU1 Inner brow raiser AU2 Outer brow raiser AU4 Brow lowerer AU5 Upper lid raiser AU6 Cheek raiser AU7 Lid tightener AU9 Nose wrinkler AU10 Upper lip raiser AU11 Nasolabial furrow AU12 Lip corner puller AU13 Cheek puffer AU14 Dimpler AU15 Lip corner depressor AU17 Chin raiser AU20 Lip stretcher AU22 Lip funneler AU23 Lip tightener AU25 Lips part AU26 Jaw drops AU27 Mouth stretches AU41 Lids droop GESTURES Upper body Away from camera Collapsed Shoulders Up Backward Forward Head Downward Backward Turned sideways Bent sideways Arms Lateralized hand/arm movements Stretched out frontal Stretched out sideways Crossed in front of chest Crossed in front of belly Before belly Stemmed to hips Hands Fist(s) Opening/closing Hand(s) sideways Self-manipulator Illustrator Emblem Pointing VOICE Fundamental frequency: mean standard deviation 25th and 75th percentile Energy: mean Speech rate duration of articulation periods duration of voiced periods Spectral energy distribution Hammarberg index; slope above 1000 Hz % voiced energy up to 500 Hz % voiced energy up to 1000 Hz. Dichotomization: z >.5 = high 1 z < -.5 = low 1 else 0

26 Multimodal Agitation cluster

27 Multimodal Joyful surprise cluster

28 Multimodal Resignation cluster

29 Thank you for your attention! Klaus R. Scherer Neural Network #1

Computational models of emotion

HUMAINE Plenary Newcastle, May 24-27, 2004 Computational models of emotion Klaus R. Scherer University of Posing the problem: Three types of computational models Appraisal cirteria Integration rules Sequential