Health Behavior Informatics

Transcription

1 Health Behavior Informatics Louis-Philippe (LP) Morency CMU Multimodal Communication and Machine Learning Laboratory [MultiComp Lab] PhD students: Chaitanya Ahuja, AmirAli BagherZadeh, Volkan Cirik, Sayan Ghosh, and Alexandria Vail Master students: Torsten Wörtwein and Paul Liang Postdoctoral researcher: Elif Bozkurt Research programmer: Chirag Raman Project manager: Laura Alford

2 MultiComp Lab Algorithms to analyze, recognize and predict human subtle communicative behaviors in social context. 2

3 Human Communication Dynamics 3

4 Human Communicative Behaviors Verbal Lexicon Words Syntax Part-of-speech Dependencies Pragmatics Discourse acts Vocal Prosody Intonation Voice quality Vocal expressions Laughter, moans Visual Gestures Head gestures Eye gestures Arm gestures Body language Body posture Proxemics Eye contact Head gaze Eye gaze Facial expressions FACS action units Smile, frowning 4

5 Automatic Behavior Sensing MultiSense [InMind 2016] A Central Challenge: Modeling Human Communication Dynamics Audio Visual Verbal Behavioral Multimodal Interpersonal Societal 50 shades of yeah 5

6 Online Education Medical Broad Applicability Depression and PTSD with MIT, BBN and Cogito Suicide prevention with Cincinnati Hospital Autistic children with Yale University Disorders Depression Distress Autism Group learning analytics with Stanford and UCSD Virtual Learning Peer with CMU Public speaking training with USC Social Leadership Empathy Engagement Opinion mining with Univ. of Michigan Social influence with EPFL Negotiation outcomes with Microsoft Research Emotion Sentiment Persuasion Frustration 6

7 Human Communication Dynamics Lecture Outline Behavioral Multimodal Audio Visual Verbal Interpersonal Mental Health Behavior Indicators Health behavior infomatics Technologies for Healthcare Decision Support Behavioral Indicators of depression and PTSD Suicidal ideation behavior indicators Multimodal Machine Learning Core challenges in Multimodal ML Learning multimodal representations Multimodal prediction of psychological distress SimSensei: virtual interviewer 7

8 Technologies for Health Behavior Informatics MultiSense OR SimSensei Patient Clinician Report 8

9 Behavioral Indicators of Psychological Distress Distress Assessment Interview Corpus 0.8 Smile Patient Not-distressed Not-Depressed Reference Distressed Depressed DAIC 0.6 Tense Voice 0.4 Open Posture 0.2 Emotional Expressiveness Today 1 week 2 weeks Distress Not-distress 9

10 Psychological Distress: Study protocol Semi-structured interviews (~20 minutes) Three interview Phases Rapport building Intimate/clinical questions (positive and negative valence) Cool-down phase Self-Assessment Clinical Scales Depression (PHQ-9) and Post-traumatic stress disorder (PCL-C) Highly correlated with accepted clinical diagnosis (sensitivity & specificity >.80) Kroenke K, Spitzer RL, Williams JB, The PHQ-9: Validity of a brief depression severity measure, Journal of General Internal Medicine. 2001, 16(9) Weathers, F. W., Litz, B. T., Herman, D. S., Huska, J. A. & Keane, T. M. (1993). The PTSD Checklist (PCL): Reliablity, validity, and diagnostic utility, Paper presented at the 9thAnnual Conference of the ISTSS, San Antonio. 10

11 Demographics DAIC Age Mean Median Gender Male Female 65.64% 34.36% Education HS/GED Some College 2 yr college 4 yr college Post Graduate Degree 13.54% 37.91% 13.54% 26.71% 8.30% Race African American Asian White/Caucasian Hispanic Native American Military 44.32% 36.38% 4.47% 43.77% 13.62% 1.75% Branch Army Navy Marine Corps Air Force Coast Guard (those who are military) 44.31% 23.57% 19.11% 11.79% 1.22% 11

12 Co-morbidity Especially high correlations between clinical severity (0.80) Prevalence of Depression PTSD Comorbidity: Implications for Clinical Practice Guidelines and Primary Care-based Interventions, D. G. Campbell, et al., J Gen Intern Med., 22(6): ,

13 Psychological Distress Indicators Behavioral [IEEE FG best paper award] Joy Facial expr. Sad Facial expr. Vertical eye gaze Smile intensity Distress No-distress Hand self-adaptor Distress No-distress Distress No-distress Distress No-distress Legs fidgeting Voice energy std. Voice quality (NAQ) DAIC Distress No-distress Distress No-distress Distress No-distress Distress No-distress 13

14 Indicators with different trends on both genders [ACII 2013] AU4 (frown) Intensity Disgust Intensity Cross over interaction! G = G = G = G =

15 Indicators with similar trend on both genders [ACII 2013] Head Rotation Variation * Emotional Variation G = G = G = G = Gender effect! 15

16 Source: CDC Behavior Indicators of Suicidal Ideation [2010] Experiment Nonverbal indicators of suicidal ideations 60 subjects 30 non-suicidal adolescents 30 suicidal adolescents 17 non-repeater 13 repeater 16

17 Behavior Indicators of Suicidal Ideation [2010] 17

18 Verbal Behavior Indicators [TAFFC 2016] 18

19 Vocal Behavior Indicator: Voice Quality [TAFFC 2016] 19

20 Vocal Behavior Indicator: Vowel Space [ICASSP 2015] 20

21 Behavior Indicators with Schizophrenic Patients 18 semi-structured interviews (5-8 minutes per session) Clinical scales: Positive And Negative Syndrome Scale of Schizophrenia (PANSS) Brief Psychiatric Rating Scale (BPRS) The Montgomery-Asberg Depression Rating Scale (MADRS) 21

22 Behavior Indicators with Schizophrenic Patients [CHI-W 2016] Facial expressivity Schizophrenia Unusual thought symptom With doctor Gestures when alone 22

23 Vocal Visual Verbal Multimodal Machine Learning We saw the yellow dog Disorders Depression Distress Autism Multimodal Probabilistic Learning Social Leadership Empathy Engagement Emotion Sentiment Persuasion Frustration 23

24 Core Challenges in Multimodal Machine Learning Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency 24

25 1 2 Prediction Big dog on the beach t 1 t 2 t 3 t 4 t 5 t 2 t 3 t n t 6 t n Input Modalities Language Acoustic Visual

26 Multimodal Representations Multimodal Audio Visual Verbal Representation Verbal We saw the yellow dog Visual Acoustic Common Representation (Multimodal Space) 26

27 Multimodal Representations Wow! I like it! Joyful tone Common Representation (Multimodal Space) Tensed voice 27

28 Deep Multimodal Representations Audio-visual speech recognition Bimodal Deep Belief Network Image captioning [Ngiam et al., ICML 2011] [Srivastava and Salahutdinov, NIPS 2012] Multimodal Deep Boltzmann Machine Audio-visual emotion recognition Deep Boltzmann Machine [Kim et al., ICASSP 2013] Depth Video Depth Multimodal Multimodal Representation Visual Verbal Depth Verbal 28

29 Multimodal Vector Space Arithmetic [Kiros et al., Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models, 2014] 29

30 Multimodal Sentiment Analysis Sentiment Intensity [-3,+3] softmax h m h m = f W h x, h y, h z h x Text X h y Image Y Audio Z h z 30

31 Trimodal Bimodal Unimodal Speaker s behaviors Sentiment Intensity This movie is sick? Ambiguous! This movie is fair Smile Loud voice? Unimodal cues Ambiguous! This movie is sick Smile This movie is sick Frown This movie is sick Loud voice? Resolves ambiguity (bimodal interaction) Still Ambiguous! This movie is sick Smile Loud voice This movie is fair Smile Loud voice Different trimodal interactions! 31

32 Multimodal Tensor Fusion Network (TFN) h x Explicitly models unimodal, bimodal and trimodal interactions! h x h z h x h y h y h z h y h z h x h y h z [Zadeh, Jones and Morency, EMNLP 2017] h x Text X h y Image Y Audio Z h z 32

33 Experimental Results MOSI Dataset Improvement over State-Of-The-Art 33

34 Taxonomy of Multimodal Research Representation Joint o Neural networks o Graphical models o Sequential Coordinated o Similarity o Structured Alignment Explicit o Unsupervised o Supervised Implicit o Graphical models o Fusion Neural networks Model agnostic o Early fusion o Late fusion o Hybrid fusion Model-based o Kernel-based o Graphical models o Neural networks Translation Example-based o Retrieval o Combination Model-based o o o Grammar-based Encoder-decoder Online prediction Co-learning Parallel data o o Co-training Transfer learning Non-parallel data Zero-shot learning Concept grounding Transfer learning Hybrid data Bridging Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency, Multimodal Machine Learning: A Survey and Taxonomy,

35 Vocal Visual Verbal Multimodal Machine Learning We saw the yellow dog Disorders Depression Distress Autism Multimodal Probabilistic Learning Social Leadership Empathy Engagement Emotion Sentiment Persuasion Frustration 35

36 Automatic Distress Level Prediction 36

37 MultiSense Live Demonstration

38 OpenFace: An Open-source Facial Behavior Analysis Toolkit [WACV 2016] State-of-the-art performance in all tasks Real time performance (without a GPU) Open Source (both training and testing) Cross-platform (Linux and Windows) Tadas Baltrusaitis tbaltrus@cs.cmu.edu 38

39 MultiSense + SimSensei: Video Demonstration 39

40 Why are we creating an AI agent (SimSensei)? [AAMAS 2014] Compare responses when participants believe the avatar is controlled by a human or by an AI Computer-framed (N=77) vs. Human-framed (N=77) interactions SimSensei 40

41 Why are we creating an AI agent (SimSensei)? [AAMAS 2014] Fear of evaluation Impression Management Intensity of sadness displayed * 62 * 0.14 * Computer Human Computer Human Computer Human 41

42 Technologies for Health Behavior Informatics MultiSense OR SimSensei Patient Clinician Report 42

43 Human Communication Dynamics Technologies for Health Behavior Informatics Behavioral SimSensei Cicero Suicide prevention with Cincinnati Hospital Multimodal Audio Visual Verbal TeleCoach Virtual rapport OVAT Negotiation outcomes Interpersonal Mel Look-to-talk Social influence Group learning analytics with Stanford and UCSD 43

44 MERCI!