Debsubhra Chakraborty Institute for Media Innovation, Interdisciplinary Graduate School Nanyang Technological University, Singapore

Transcription

1 Debsubhra Chakraborty Institute for Media Innovation, Interdisciplinary Graduate School Nanyang Technological University, Singapore 20 th November 2018

2 Introduction Design of Experiment System Overview Results Discussion & Conclusion 2

3 One in four people suffer from mental illness at some point in their lives 1 Nearly two-thirds of them never seek treatment 1 Current clinical assessments and treatments require highly trained clinicians Methods are subjective and time-consuming Can we move towards automated objective assessments? WHO Press Release on Mental Illness 3

4 A chronic, mental disorder Three types of symptoms: Positive, Negative and Cognitive 1 Few clinical treatments for negative symptoms 2 Negative symptoms: Reduction of nonverbal behavior, such as speech, gestures, and facial expressions Negative symptoms contribute to poor function and quality of life for patients 3 Aim: Identify objective biomarkers: disorganized speech is a key symptom 1. Demily, C., & Franck, N. (2008). Cognitive remediation: a promising tool for the treatment of schizophrenia. Expert Review of Neurotherapeutics, 8(7), Murphy, Brendan P., et al. "Pharmacological treatment of primary negative symptoms in schizophrenia: a systematic review." Schizophrenia research 88.1 (2006): Kirkpatrick, Brian, et al. "The NIMH-MATRICS consensus statement on negative symptoms." Schizophrenia bulletin 32.2 (2006):

5 Problems of syntactic complexity and semantic coherence common in schizophrenia Indicators of early onset; insight into symptoms and thought disorders, and trajectory of recovery 1 Linguistic analysis a popular tool for investigation of schizophrenia speech 2, 3 However they utilize manual transcriptions, which is tedious Comparatively, analysis of schizophrenia speech with nonverbal features is lesser 4 Combining verbal and non-verbal features together for holistic understanding of schizophrenia, akin to clinicians 1. Andreasen, N. C., & Grove, W. M. (1986). Thought, language, and communication in schizophrenia: diagnosis and prognosis. Schizophrenia Bulletin, 12(3), Hong, K., Nenkova, A., March, M. E., Parker, A. P., Verma, R., & Kohler, C. G. (2015). Lexical use in emotional autobiographical narratives of persons with schizophrenia and healthy controls. Psychiatry research, 225(1-2), Minor, K. S., Bonfils, K. A., Luther, L., Firmin, R. L., Kukla, M., MacLain, V. R.,... & Salyers, M. P. (2015). Lexical analysis in schizophrenia: how emotion and social word use informs our understanding of clinical presentation. Journal of psychiatric research, 64, Parola, A., Simonsen, A., Bliksted, V., & Fusaroli, R. (2018). T138. Acoustic Patterns In Schizophrenia: A Systematic Review And Meta-analysis. Schizophrenia Bulletin, 44(suppl_1), S169-S169. 5

6 Introduction & Literature Review Design of Experiment System Overview Results Discussion & Conclusion 6

7 Study in collaboration with Institute of Mental Health in Singapore Patients with Schizophrenia (Subjects) (50) Patients (50) Age Gender Ethnicity Education Mean (years) 30.3 Range (years) Male 25 Female 25 Chinese 42 Malay 5 Indian 3 University 7 Diploma/ Vocational 27 High School 16 7

8 Each participant undergoes cognitive tasks, a semi-structured interview, and functional tasks No pre-determined time-limit for the interview No role-playing during interview On average, interview lasts for 30 minutes Each interview (audio and video) analysed in its entirety 8

9 The psychometric scale used is Negative Symptoms Assessment-16 (NSA-16) One of few rating instruments specifically designed for negative symptoms 16 item scale with scores for speech behaviour, emotional behaviour, affect, movement and daily activity etc. Ratings are on a scale of 1-6 Behaviour Fully normal behaviour 1 Behaviour minimally reduced 2 Behaviour mildly reduced, but noticeable by trained rater Behaviour moderately reduced, noticeable even by untrained rater Behaviour is markedly reduced, hampering social function Behaviour severely reduced or absent 6 Score

10 Introduction & Literature Review Design of Experiment System Overview Results Discussion & Conclusion 10

11 Lapel microphones records the audio of the participant and psychologists Audio captured in 2-channel audio file Patient and Psychologist seated 2m apart, crosstalk is minimal Speech-to-text using Kaldi 1 toolkit 1. Povey, Daniel, et al. "The Kaldi speech recognition toolkit." IEEE 2011 workshop on automatic speech recognition and understanding. No. EPFL-CONF IEEE Signal Processing Society,

12 Linguistic Inquiry and Word Count 1 (LIWC2015) Extract number of linguistic words related to thoughts, feelings, personality, and motivations Captures people s social and psychological state 78 output categories in LIWC2015 Category name Abbrev Examples Words in category Common Adverbs adverb very, really, basically 140 Netspeak netspeak coz, thx, yeah 209 Female references female Mom, sister, her, she 124 Feel feel Feel, touch, warm 128 Each category we normalized by the duration of the interview

13 Prosodic Prosodic cues based on opensmile s emobase set opensmile toolkit - a modular and flexible audio feature extractor for signal-processing applications 1 19 different statistics for the 26 signals and their delta values Total 988 features Intensity Loudness MFCC Pitch Probability of Voicing F0 envelope LSF 1. Florian Eyben, Martin Wollmer, and Bjorn Schuller, Opensmile: the munich versatile and fast open-source audio feature extractor, in Proceedings of the 18th ACM international conference on Multimedia. ACM, 2010, pp ZCR 13

14 Conversational cues capture the dynamics of conversation Account for who is talking, when and by how much 14 features related to Natural Turns, Interruption, Interjection, Speaking Percentage, Mutual Silence, Response Time etc. 14

15 Introduction & Literature Review Design of Experiment System Overview Results Discussion & Conclusion 15

16 NSA-16 ratings are from 1-6, but not all ratings are equally frequent Ratings regrouped into 2 classes - Ratings 1 & 2 class 0 ( Low symptoms: unobservable) - Ratings 3, 4, 5 & 6 class 1 ( High symptoms: observable) Classification Features: Linguistic LIWC features + opensmile Prosody features, Conversational features Labels: NSA-16 ratings as target Classifier: Linear SVM with Stochastic Gradient Descent Optimization Feature Selection: Kruskal-Wallis test based feature selection on validation set, nested inside leave-one-out cross-validation NSA ratings (High/Low) 16

17 Several NSA-16 items related to speech and emotion can be predicted with reasonably high accuracy NSA item Confusion Matrix Predicted class High Low NSA 1: Prolonged True High 8 8 Time to Respond class Low 1 33 NSA 2: Restricted Speech Quantity NSA 3: Impoverished Speech Content NSA 6: Affect Reduced Modulation of Intensity NSA 15: Reduced Expressive Gestures True class True class True class True class High 14 7 Low 4 25 High 15 9 Low 9 17 High 17 8 Low 6 19 High 10 8 Low 6 26 Accuracy Baseline Accuracy 82.00% 68.00% 78.00% 58.00% 64.00% 52.00% 72.00% 50.00% 72.00% 64.00% 17

18 Introduction & Literature Review Design of Experiment System Overview Results Discussion & Conclusion 18

19 Items related to even emotion and gestures are accurately predicted Persons who speak less, also make less gestures Symptoms are highly inter-related disruption of cognitive processes in patients Mentally ill individuals often have (subtle) differences in social behavior This paves the way for holistic objective assessment of negative symptoms, and can be translated to practical tools for (apps, social robots) Long-term assessments of mentally ill people Evaluations of treatments. Plan: We have just started a similar 3-year study related to depression (funded by Rehabilitation Research Institute Singapore--RRIS) Objective cues Mental Illness Schizophrenia Depression Symptoms Negative symptoms Neurocognitive impairments Social cognitive deficits Speech Motor Facial expressions 19

20 Thank You 20

21 Objective: Automated linguistic analysis of the interviews with schizophrenia patients Record audio during psychol-participant interviews Speaker diarization and speech recognition Natural language processing and feature extraction Classification and prediction Speech recognition (Speech to Text) Kaldi is an open source speech recognition toolkit which could enable the recognition and translation of spoken language into text by computers. (offline) Text features 21

22 Linguistic Inquiry and Word Count 1 (LIWC2015) to extract number of linguistic words related to thoughts, feelings, personality, and motivations. 78 output features (e.g., assent, work, feeling, religion) capture people s social and psychological states Normalized by the duration of interviews Diction : 42 features 31 sub-features (e.g., Praise, Satisfaction, Denial) 5 core features (Activity, Optimism, Certainty, Realism and Commonality) 4 calculated features (e.g., Insistence=[Number of Eligible Words x Sum of their Occurrences] 10) 2 common features (avg word size, num of unique words) 1. Pennebaker J W, Boyd R L, Jordan K, et al. The development and psychometric properties of LIWC2015[R] Loughran, Tim, and Bill McDonald. The use of word lists in textual analysis. Journal of Behavioral Finance 16.1 (2015):