Physiological Detection of Emotional States in Children with Autism Spectrum Disorder (ASD)

Transcription

1 Physiological Detection of Emotional States in Children with Autism Spectrum Disorder (ASD) by Sarah Sarabadani A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Department of Biomaterials and Biomedical Engineering University of Toronto Copyright by Sarah Sarabadani 2016 i

2 Physiological Detection of Emotional States in Children with Autism Spectrum Disorder (ASD) Sarah Sarabadani Master of Applied Science Department of Biomaterials and Biomedical Engineering University of Toronto 2016 Abstract Autism spectrum disorder (ASD) is associated with difficulties in emotion processing including attributing emotional states to others and processing of one s own emotional experiences. These difficulties are linked to core social impairments and increased severity of psychiatric comorbidities such as depression. The nature of these difficulties has remained largely unknown. This is partially due to limitations in obtaining reliable self report of emotional experiences in this population. Emotion detection using physiological signals is a promising direction in addressing this limitation. Physiological signals can provide a language free method for understanding emotional states in ASD. The use of this approach has not been studied in ASD. To this end we develop a physiological approach to detection of emotion in children with ASD. We showed that emotional states can be classified with accuracies>80% in a sample of children with ASD which affirms the feasibility of discriminating affective states in this population. i ii

3 Acknowledgments Foremost, I would like to express my sincere gratitude to my thesis advisor Dr. Azadeh Kushki for her patience, motivation, enthusiasm, and immense knowledge. Her guidance helped me in all the time of research and writing of this thesis. She always steered me in the right direction whenever she thought I needed it. Besides my advisor, I would like to thank the rest of my thesis committee: Dr. Jose Zariffa, Dr. Evdokia Anagnostou, and Dr. Azadeh Yadollahi, for their encouragement and insightful comments. My sincere thank also goes to Ali Samadani who always was there to answer my endless questions. I am gratefully indebted to his very valuable helps with this thesis. I would like to thank members of Autism Research Centre (ARC), especially Stephanie Chow who has always supported me and helped me putting pieces together. Finally, I must express my very profound gratitude to my parents for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this thesis. This accomplishment would not have been possible without them. Thank you. iii

4 Table of Contents Abstract... ii Acknowledgments... iii Table of Contents... iv List of Tables... vi List of Figures... vii Introduction Motivation Research Question and Objective... 2 Background Brain Activity in Emotion Processing Emotion Recognition in ASD Automatic Emotion Recognition Emotional model Choice of the emotion model Emotion elicitation Physiological Signals for Emotion Classification Electrocardiogram (ECG) Skin Conductance (SC) Respiration (RSP) Skin temperature (SKT) Existing Systems for Physiological Emotion Recognition Typically developing ASD Research Methods Participants Instrumentation Stimuli iv

5 3.4 Experimental protocol Analysis Pre-processing Feature Extraction Feature selection Classification Performance Evaluation Results Participants Demographics SAM Results Classification Results Ensemble of Classifiers Classification over arousal axis Modality Specific Results Selected Features Association of Classification Accuracy and SAM Ratings with Demographics Effect of Window Size on Accuracy Discussion and Conclusion SAM assessment Feature selection Classification Results Effect of demographic variables/behavioural measures on accuracy Conclusion References v

6 List of Tables Table 2.1: Summary of studies on typical individuals Table 3.1: Average rating of final selection of pictures Table 4.1: summary of features, SC: Skin Conductance, ECG: Electrocardiogram Table 5.1: Demographic information Table 5.2: Overall SAM results for each participant Table 5.3: Emotion specific SAM results for each participant Table 5.4: Emotion specific SAM results for each parent Table 5.5: Confusion matrix for child SAM ratings Table 5.6: Confusion matrix for parent SAM ratings Table 5.7: Top ten selected features Table 5.8: Accuracy of comparing HP vs. HN for each classifier Table 5.9: Accuracy of comparing LP vs. LN Table 5.10: Classification results of ensemble of methods Table 5.11: Comparing un-weighted and weighted ensemble of classifiers Table 5.12: Classification results using shuffled labels Table 5.13: Confusion matrices of ensemble of methods vi

7 List of Figures Figure 2.1: Discrete and dimensional model [29]... 7 Figure 2.2: SAM [30]... 8 Figure 2.3: Example of a QRS waveform in an ECG signal [54] Figure 2.4: SC signal [56] Figure 2.5: respiration sensor [61] Figure 2.6: Skin temperature sensor [62] Figure 3.1: Attachment of sensors, Procomp Infiniti hardware manual [65] Figure 3.2: experimental setup Figure 3.3: Procedure of picture selection Figure 3.4: Experimental protocol Figure 4.1: Analysis procedure Figure 4.2: Segmentation of each task Figure 5.1: Selected features, I will replace figures with larger and clear ones later Figure 5.2: Selected feature for each participant.(a) low arousal, (b) high arousal, (c) subtracting a from b Figure 5.3: Bar plot of results of ensemble of classifiers vii

8 Chapter 1 Introduction 1.1 Motivation Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by social communication difficulties and the presence of repetitive and restricted behaviors and interests [1]. ASD is also associated with difficulties in emotion processing, which may underlie some of the core social impairments in this population [2]. A large body of literature has examined emotion processing in ASD, suggesting difficulties in attributing mental and emotional states to others [3, 4, 5]. A few studies have also reported self-awareness and atypical processing of self emotional experiences in this population [6]. For example, one in two individuals with ASD is suggested to be affected by alexithymia (difficulties in distinguishing and describing internal body states) [7]. This is significantly higher than the one in ten prevalence of alexithymia in the general population [8], [9]. In addition to being closely linked to the core social impairments in ASD [10], difficulties in interpreting and processing emotions in ASD are known to be associated with increased severity of psychiatric disorders such as depression [2]. Hence, assisting individuals with ASD with perceiving and processing their internal body state is essential. Emotion detection using physiological signals is a promising direction in addressing this gap. In this context, a physiological approach to detection of emotions can provide a language-free, noninvasive, and low-cost way for characterization of emotional states in children with ASD. This work can ultimately contribute to improving self-awareness of emotions by providing users with information regarding their actual body state. In addition it will enhance our understanding of ASD-related emotion processing difficulties. 1

9 Physiological signals reflect the activity of the Autonomic Nervous System (ANS) [19]. ANS is responsible for involuntary control of organs and regulating processes such as heart rate and respiration [20]. Emotional stimuli have been shown to influence the activity of the ANS in a measureable way [21]. For example, heart rate and blood pressure tend to increase in response to anger [22]. There is extensive body of literature on characterizing physiological signals to discern emotional states in typically developed individuals [16, 31, 32, 34, 36]. However, it is unclear that these methods can be used in this population as ASD is associated with atypical ANS function [45]. 1.2 Research Question and Objective This research aims to answer this question: Is it possible to physiologically differentiate affective states in children with ASD? The goal of this study is to develop tools for physiological detection of emotions in children with ASD. The specific objective is to develop classification techniques to differentiate patterns of physiological response to four emotional states (high arousal/positive valence, low arousal/positive valence, high arousal/negative valence, and low arousal/negative valence). 2

10 Chapter 2 Background In this chapter we review previous research in the field of emotion recognition. First the influence of affective state on various physiological signals is discussed. Afterwards, emotion processing in ASD is investigated from a neurophysiologic perspective. Then, various modalities in automatic emotion recognition including facial expressions, voice, and physiological signals are explained and the choice of latter one is justified. Finally, three principal matters in developing automatic emotion recognition comprising emotional model, emotion elicitation method, and specific physiological indices of ANS activity are discussed in the following three sections. 2.1 Brain Activity in Emotion Processing ANS is one of the divisions of peripheral nervous system (PNS) that controls the involuntary function of organs. It consists of two branches; the sympathetic nervous system which is known as fight or flight system, activated in fast changes and arousal while inhibits digestion. The other branch of ANS named parasympathetic nervous system which is known as rest and digest system. It is associated with calming and regular function of the nerves and promoting digestion. The function of these two parts is opposite and as one of them enhances the physiological response, the other inhibits it [60]. There are various positions on autonomic response organization in emotion. It has been shown that valence specific patterns are more consistent with ANS activity than discrete emotion patterns [61]. James [62] has defined emotion as feeling of the body changes as they occur. He has argued that emotional states are associated with specific physiological response with 3

11 variations in symptoms between individuals. Stemmler [65] has stated that autonomic activity occurs prior to any behavioral changes due to emotional states. This argument is supported by studies on paralyzed animals where there is lack of external behavior while autonomic activation has been detected [66]. It contradicts the notion of ANS activity as a result of motoric response [67]. Stemmler also has suggested that for body protection and behavioral adaptation distinct autonomic response is required to exist. A large body of literature has examined the relation between affective states and patterns in autonomic response. They could successfully show such patterns exist in various emotional states. As one of the pioneers in this area, Ekman [23] has investigated emotion specific activities in ANS. He has considered six emotions (surprise, disgust, sadness, anger, fear, and happiness) and recorded signals of heart activity, skin temperature, skin resistance, and muscle tension. The changes were observed not only between positive and negative emotions, but also in various negative affective states. Picard [16] also could differentiate eight discrete emotions (neutral, anger, hate, grief, platonic love, romantic love, joy, and reverence), looking at ANS activity through four physiological signals (i.e. muscle tension, heart activity, skin conductance, and respiration). In another study Kim J [39] has shown there is discernible pattern between positive/high arousal, negative/high arousal, positive/low arousal, and negative/low arousal. He has employed four physiological signals to measure heart activity, skin conductance, respiration rate, and muscle tension. Kim K.H [44] has also indicated pattern between sadness, anger, stress, and surprise using signals representing heart activity, skin conductance, and temperature. These major findings along with tons of similar works suggest that detecting pattern in physiological signals in various emotional states is feasible. 2.2 Emotion Recognition in ASD ASD is associated with difficulties in identify and describing one's own emotions (alexithymia [6]). These atypicalities are suggested to be closely linked to the capacity to empathize, a key area of difficulty in ASD. 4

12 Lambie and Marcel [11] conceptualized the emotion experience using a two-level model. In this model, first-order experience of emotion is attributed to neurophysiological arousal associated with emotional states. Self-awareness of this arousal is second-order experience of emotion (interoception). In ASD, atypicalies have been reported both levels of emotional experience. While emotion-related physiological arousal is suggested to be present in ASD [12], its pattern may be atypical [13]. Interestingly, in a study by Silani et al. [6] reduced emotional awareness was not found to be associated with reduced response in the brain regions mapped to first order experience (amygdala and inferior orbitofrontal cortex), suggesting that this circuitry may not underlie alexithymia symptoms in ASD [6]. Several studies have also reported that the secondorder emotional experience (i.e., aware of bodily states) is atypical in ASD [9, 12, 14, 15]. For example, in a functional MRI study of individuals with ASD, Silani et al. [6] found a significant negative correlation between the severity of alexithymia tendencies and activity in brain regions associated with interoception (e.g., the insula). The authors then concluded that the lack of awareness of bodily states, or the decoupling between physiological arousal and conscious representation of emotions, may underlie alexithymia tendencies in ASD. 2.3 Automatic Emotion Recognition Inferring emotional states automatically by means of computer algorithms has been studied extensively, mainly in the context of enhancing human-computer interaction [16]. These algorithms employ changes in internal or external states of users for emotion recognition. The states commonly considered include facial expressions, voice, and physiological signals. Emotion detection based on facial expressions has been mainly used for enhancing humancomputer interaction as this approach requires the user to be directly in the field of view of a camera [17]. ASD has been associated with atypical facial expressions, which may affect the accuracy of these methods. Emotion detection based on voice also presents several challenges in the context of this proposal. First, such a method requires the continuous presence of verbal expression, which may not be possible for some individuals with ASD. Second, ASD is associated with atypical speech 5

13 features such as prosody [18], which may affect the recognition accuracy of these methods. Finally, voice-based emotion detection will not be appropriate for settings, such as classrooms, where noise is present in the environment or users do not continuously produce speech. Emotion detection based on physiological signals is especially appropriate for use in this project for three reasons: 1) this measurement modality can be employed across different environments and especially in naturalistic settings; 2) physiological signals are relatively independent of individuals ability profiles and cultural norms; 3) these signals can be measured non-invasively and with low-cost with relatively low burden on the user. For these reasons, the physiological approach to emotion detection is chosen for the proposed project. Three key issues need to be considered in developing automatic emotion recognition namely, choosing the emotional model, emotion elicitation method, and specific physiological indices of ANS activity. Details are provided in sections that follow. 2.4 Emotional model Two approaches are commonly used for quantitative modeling of emotions. The first, proposed by Ekman [23], assumes the existence of a discrete set of emotions. Building on this assumption, six classes of emotions (happiness, sadness, surprise, anger, disgust, and fear [24] have been commonly used in the literature. The second model of emotion challenges the discrete nature of emotion and proposes a continuous space in two dimensions of arousal and valence [25]. Valence represents the pleasantness of an emotion and ranges from negative to positive. Arousal reflects intensity of an emotion and ranges from low to high. For example, happiness has positive valence and high arousal, while sadness has negative valence and low arousal. Six core emotions are shown in the valence-arousal coordinates in Figure

14 Figure 2.1: Discrete and dimensional model, modified version of [29] Choice of the emotion model Dimensional model of emotion is consistent with results of neuroimaging studies. It has been shown that different affective states activate two brain areas (i.e., orbitofrontal cortex and amygdala), which are known to be associated with arousal and valence respectively. This suggests that the dimensional model is most consistent with representation of emotions in the brain [28]. The discrete model of emotion requires the consideration of several emotions in each state of arousal and valence. For example, anger, stress, and fear all represent states of high arousal and negative valence. The large number of discrete emotion states that need to be considered with this approach, together with practical limitations on the number of training samples that can be collected for each states challenge the development of automatic classification techniques using this model. 7

15 2.5 Emotion elicitation The most commonly stimuli in emotion detection studies in typically developing population is using International Affective Picture System (IAPS) [31, 32, 33, 34] which is a large set of color pictures validated to be effective for inducing different levels of arousal and valence [30]. It also has been used in ASD [12]. For rating the pictures a visual scale, called the self assessment manikin (SAM), has been suggested by the publishers of IAPS (Figure 2.2). The first and second rows of SAM correspond to level of valence and arousal which were previously defined. The third row represents dominance of emotion denoting the level of control vs. the level of being controlled. For instance, fear and anger both are negative emotions. Regarding dominance dimension the former is more submissive while the latter is more dominant. Geneva affective picture database (GAPED) is a relatively new system consisting of 730 pictures similar to IAPS, in three main categories: positive, neutral, and negative. Figure 2.2: SAM [30] 8

16 The majority of the studies in ASD have focused on inducing anxiety. Kushki [45] used the Stroop color-word interference task and public speaking to elicit anxiety and high arousal. Stroop is a computer task in which names of colors are printed on the screen in different colors and the participants are asked to name the color while ignoring the word. Kootz and cohen [46] used tasks associated with social communication for inducing anxiety. Jansen [47] also used public speaking for this purpose. Groden and Goodwin [48] chose various tasks from the stress survey schedule [49]. Liu used computer games to elicit three discrete emotions (i.e. liking, anxiety, and engagement) [17]. The first task was a computer game named Pong which had been previously utilized to induce anxiety [50]. In this game the player controls a paddle to strike a free moving ball. Different affective states were elicited by changing the level of difficulty. The second task was Anagram. In this task the participant names the correct word that is given with disordered letters. This game was previously suggested by Pecchinenda and Smith [51] to investigate the relation between physiology and anxiety. 2.6 Physiological Signals for Emotion Classification Several channels of physiological signals can be used to quantify the function of the ANS. Four selected signals in this study are reviewed in the sections that follow. This choice was made in consideration for participant comfort and ease of sensor attachment Electrocardiogram (ECG) ECG measures the contractile activity of the heart by capturing the action potential related to heart contraction. Depolarization of the heart ventricles produces the waveform known as QRS complex [54] (Figure 2.3). Inter beat interval (IBI) is the time interval between two R peaks in the waveform and generally ranges between 300 ms to 1500 ms [17]. Heart rate (HR) is the number of heart beats per minute (bpm) and is approximately bpm at rest. The variation in 9

17 time interval between two consecutive heart beats is called heart rate variability (HRV) [55]. Mean and standard deviation are two features in time domain that can be extracted from the IBI series. Figure 2.3: Example of a QRS waveform in an ECG signal [54] Skin Conductance (SC) SC measures the skin's ability to conduct electricity. The change in skin conductivity is associated with the activity of eccrine sweat glands, which receive input from the sympathetic nervous system [52]. The SC signal has two components: 1) a slow moving component which reflects the general activity of sweat glands and shows the ongoing level of skin conductance, and 2) faster changes influenced by environmental events and appears with an instantaneous increase in the signal. For instance, anxiogenic stimuli are shown to cause a sudden rise in the signal [39]. An example of an SC signal is shown in Figure 2.4 [56]. 10

18 Figure 2.4: SC signal [56] Respiration (RSP) In general, emotional excitement and physical activity are associated with faster and deeper breathing, while relaxation and calmness lead to slower and shallower respiration [56]. Respiration sensor is used for measuring depth and rate of breathing (Figure 2.5) [43]. Respiration signal is analyzed in the time domain to extract descriptive statistics or in the frequency domain by performing power spectral analysis. Figure 2.5: respiration sensor [61] 11

19 2.6.4 Skin temperature (SKT) Variations in skin temperature are mainly associated with changes in cutaneous blood flow. Blood flow is determined by vascular resistance which is caused by contraction and relaxation of smooth muscles [57, 58] Mean, slope, and standard deviation of temperature are three key features of SKT sensor [17] (Figure 2.6). Figure 2.6: Skin temperature sensor [62] 2.7 Existing Systems for Physiological Emotion Recognition Typically developing Based on a review work by Jerritta et al. [29], different physiological based emotion recognition studies are summarized in Table 2.1. Table 2.1: Summary of studies on typical individuals, EMG: Electromyogram, ECG: Electrocardiogram, Temp: Temperature, Resp: Respiration, SC: Skin Conductance, BVP: Blood Volume Pulse Feature emotions # of Induction selection Classification signals participants method and method Accuracy (%) reduction 44 Sad, anger, 125 Multimodal ECG used all the Support 78.4 (3 emotions) 12

20 stress, Temp features Vector 61.8 (4 emotions) surprise SC Machine sequential forward 39 Joy, Anger, Sad, Pleasure 3 Music EMG ECG SC Resp selection (SFS) sequential backward Linear Discriminant Analysis 95 (personal) 70 (group) selection (SBS) 71 Valance, Arousal 36 Robot Actions SC ECG used all the features Hidden Markov Model 83 Arousal,80 Valance (personal ) 66 Arousal, 66 Valance (group) Neutral, 16 Anger, Hate, Grief, Platonic love Romantic love, Joy, 1 Personalized Imagery EMG BVP SC Resp Fisher projection Hybrid Linear Discriminant Analysis 81(personal) Reverence 31 Happiness, Disgust, Fear 9 IAPS pictures EMG ECG SC Resp Simba algorithm Principal Component Analysis K Nearest Neighbor Random Forest (group) 62.41(group) EMG 32 Valance, Arousal IAPS pictures ECG SC Temp BVP used all the features Neural Network Classifier Valance 89.7 (personal) Arousal (personal) Resp Sad, Anger, K NN 36 Surprise, Fear, Frustration, 14 Movies SC ECG used all the features DFA Marquardt Back 71(personal):KNN 74(personal):DFA 83(personal):MBP Amusement Propagation 72 Joy, Anger, Sad, Pleasure 1 Music ECG EMG SC Resp used all the features Support Vector Machine 76 (Fission,personal) and 62 (Fusion, personal) 13

21 41 Joy, Anger, Sad, Pleasure 1 Music EMG used all the features Neural Network (personal) 73 Joy, Anger, Sad, Pleasure 1 Music ECG EMG SC Resp used all the features Linear Discriminant Analysis 83.4 (personal) 33 Amusement, Contentment, Disgust, Fear, Sad, Neutral 10 IAPS pictures BVP EMG Temp SC Resp used all the features Support Vector Machine, Fisher Linear Discriminant Analysis 90(personal) and 92(personal) 34 Anger, Interest, Contempt, Disgust, Distress, Fear, Joy, Shame, Surprise 28 IAPS pictures ECG BVP SC EMG Resp Sequential Floating Forward Selection (SFFS), Fischer Projection K Nearest Neighbour, 50(group) 90.7(personal) ASD Only a few studies have examined physiological emotion detection in ASD. The works of Groden [48] and Ben Shalom [12] investigated trends in physiological signals in response to emotional stimuli, but no automatic classification techniques were proposed. In particular, Groden considered stress and used ECG to examine the trend of heart rate in four different situations that were designed to elicit stress [49]. Shalom investigated physiological responses to pleasant, unpleasant, and neutral stimuli in children with ASD using the IAPS pictures. He used skin conductance and performed analysis of variance (ANOVA) to analyze the signals. Liu [17] showed that three emotional states (anxiety, engagement, and liking) can be classified in children with ASD using a wide range of physiological signals (i.e. ECG, EDA, EMG, BVP, temperature, bioimpedance, and heart sound). He obtained accuracies 85.0% for liking, 79.5% for anxiety, and 84.3%. Kushki [13] used heart rate to detect arousal related to anxiety in ASD, 14

22 obtaining an accuracy of 95%. To our knowledge none of the studies considered both arousal and valence axes. They considered emotions either discretely or only in arousal axis. Chapter 3 Research Methods In this chapter we discuss the experimental setup for collecting the data used to develop and test the algorithms in this thesis. Specifically, recruitment criteria, stimuli selection, instrumentation, and the experimental protocol are discussed. 3.1 Participants Fifteen participants with ASD were recruited for this study. Participants had a clinical diagnosis of ASD using DSM-IV criteria supported by the Autism Diagnostic Observation Schedule [64], and the Autism Diagnostic Interview - revised (ADI-R). Also they completed the Weschler Abbreviated Scale of Intelligence, the Social Communication Questionnaire, and the Child Behaviour Checklist (CBCL) to characterize intelligence, ASD symptomatology, and related comorbidities. All these measures were provided by Province of Ontario Neurodevelopmental Disorders (POND). The inclusion criteria for participants were age between 12 and 18 years, full-scale IQ scores above 70, and no sensory impairments such as deafness or blindness. Participants parents/caregivers also participated in the study. The inclusion criterion for parents was being able to understand instructions and respond to questions in English. The Bloorview Research Institute and University of Toronto research ethics boards approved the study. 15

23 3.2 Instrumentation Physiological signals were measured using Procomp Infiniti (Thought Technology Ltd). The specific sensors used included ECG, SC, respiration, and skin temperature. Heart activity was measured by a three lead ECG attached to the body using pre-gelled electrodes. Respiration signal was recorded by a belt sensor using a latex rubber band which wrapped around abdomen. SC was measured using a pair of 10 mm diameter dry Ag-AgCl electrodes secured to the palmar surface of the proximal phalanges of the second and third digits of the non-dominant hand. Skin temperature was measured using a thermistor fastened to the palmar surface of the distal phalanx of the fourth digit of the hand. ECG sensor captured signals at the rate of 2048 Hz and all the other sensors at the rate of 256 Hz. The attachment of the sensors to the body is shown in Figure 3.1. Figure 3.1: Attachment of sensors, Procomp Infiniti hardware manual [65] Three connected computers were used in the experiment room; one in front of participant for viewing and rating the stimuli, the other for the parent for rating the child s emotional reactions to the stimuli, and the third one to control the data collection procedures. The position of 16

24 computers was such that the parent could see the child thoroughly. The signals were recorded on the experimenter computer. The experimental setup is shown in Figure 3.2. Figure 3.2: experimental setup 3.3 Stimuli We selected visual stimuli to elicit four combinations of arousal and valence: high arousal/positive valence, low arousal/positive valence, high arousal/negative valence, and low arousal/negative valence. The stimuli were selected from IAPS and GAPED picture systems which include 956 and 730 pictures respectively. The pictures are from a variety of topics and not culturally specific. The database provided ratings for each picture in three dimensions of arousal, valence, and dominance. For IAPS, a number in 9-point scale was attributed to each picture (1: lowest arousal/pleasure/dominance, 9 the highest arousal/pleasure/dominance). These are normative ratings obtain from a sample of approximately 100 adults. For GAPED a group of 60 adults participated in a study to evaluate pictures in terms of arousal and valence. Each picture is rated in arousal and valence scales with a number between 0 and 100; (100 is the highest arousal, most positive) [66]. 17

25 IAPS has been used in several studies on ASD. Shalom used IAPS to elicit different levels of valence in high functioning children with ASD [12]. Silani [6] used IAPS to study neural correlates of emotion recognition in ASD. Bolte [52] also employed IAPS and the adapted SAM to investigate physiological responses to emotional stimuli in ASD. GAPED has also been used in studies on typical individuals [84, 85]; however, it has not been employed in studies on ASD yet. Collectively, the two databases contain over 1386 pictures. A subset of 214 pictures from these databases was selected excluding pictures of faces, erotic photos, and those depicting brutality and mutilation themes, which were considered inappropriate for the age range of the study. This selection was then refined by consulting with clinicians. During the clinician refinement process, 60 pictures (15 in each theme) were not rated with confidence. To assess the suitability of this subset, an online survey was designed and completed by 13 parents of children with ASD. Parents provided their rating as well as comments for every picture determining whether the picture elicits positive or negative emotion and whether it is weak or strong. Parents choices combined with preliminary selection altogether constituted the final set of 24 pictures in each class (96 pictures in total).the stimuli selection procedure is represented in Figure

26 1386 pictures from IAPS and GAPED preliminary selection of 214 pictures Refining selection after consulting with clinicians Parents of children with ASD provided rating for 60 pictures with concern of appropriateness through online survey Final set of 96 pictures (24 in each class) Figure 3.3: Procedure of picture selection Table 3.1: Average rating of final selection of pictures Valence Arousal HP 6.91± ±0.98 HN 3.67± ±0.67 LP 8.1± ±1.38 LN 3.41± ±0.98 Figure 3.4 shows the average ratings of pictures selected for each class on valence axis. The right and left ends correspond to most positive and most negative valence. The negative intended stimuli are closer to the center which denotes neutral valence compared to positive pictures. Figure 3.4: Location of average ratings of the stimuli for each class on valence axis 19

27 3.4 Experimental protocol For all participants written consent was obtained from parents and children who had the capacity for consent. In cases the child did not have the capacity to consent, he/she signed the assent and the parent consented on behalf of child. At the beginning of the experimental session, sensors were attached to the participant and the task was explained to both the parent and child. Participants were asked to practice the task to ensure understanding of the protocol. Participants were told to request breaks as needed. The protocol began and ended with a 15-minute baseline involving movie watching to allow for acclimation to the lab environment. Participants then viewed two blocks emotional pictures, each separated by a five minute baseline. Each block consisted of eight, 2-minute stimuli presentation blocks, which were presented in random sequence. Pictures in each block elicited one of the four affective states considered herein (2 blocks/emotion type). This resulted in four minutes of physiological data per affective state. After completing each block, both child and parent completed the SAM to assess the child s affective state during stimulus viewing. The overview of experimental protocol is shown in Figure

28 Figure 3.5: Experimental protocol 21

29 Chapter 4 Analysis In this chapter, we describe the methods and algorithms used for data analysis. The analysis pipeline is shown in Figure 4.1. First, raw data from each sensor was preprocessed to remove noise and extract baseline and stimuli segments. Next, data segments were used to extract features for classification. Given the short duration of data for each stimulus block, we focused on extracting temporal features and did not use frequency-based features. Best features for each classification problem and participant were selected using an automatic feature selection algorithm. Classification was performed per participant using two linear and five nonlinear classifiers. To improve accuracy, classification results from multiple classifiers were combined to provide final classification decisions. 22

30 Physiologica l measures Pre-processing Segmentation and baseline subtraction Feature extraction Feature selection Classification Predicted Labels Figure 4.1: Analysis procedure 4.1 Pre-processing ECG: the signal was captured at the rate of 2048 Hz. All the other signals were recorded with the sampling frequency of 256 Hz. The algorithm described in Pan and Tompkins [69] and implemented in Matlab by BioSig software [86] was used to extract the inter-beat-interval series from the ECG signal. In order to attenuate noise due to physical movement, power line noise, and baseline wander band pass filter was applied with low and high cut off frequencies of 5 Hz and 15 Hz. The identified peaks were visually reviewed for one task randomly for each participant to examine the authenticity of detection algorithm. After performing QRS peak detection algorithm, a median filter with order 9 (considering 9 points at a time) was applied on the recognized QRS complexes. Heart rate was obtained as the inverse of RR intervals. 23

31 Respiration: The signal was band passed filtered with low and high cut off frequencies of 0.1 Hz and 0.5 Hz as the shortest average breathing interval in adults is 3 seconds [81]. Peaks of signal were identified semi automatically by the criterion of being located at least 1 second apart to accommodate fast breathing due to arousal. The validity of detected intervals less than 3 seconds was visually confirmed. Skin conductance: The signal was detrended over the entire session to minimize the effects of thermoregulation and changes in sensor adhesion resulting from perspiration. The signal was then low-pass filtered using a 10th order Butterworth filter with cut-off frequency of 1 Hz. The criteria for considering a peak as SC response were rise time of 1-3 seconds, amplitude between 0.1 and 1 µs, and minimum height of 0.05 µs. To analyze SC signal the Matlab implementation of Ledalab software [87] was used. Temperature: The signal was detrended and low pass filtered with cut off frequency of 0.1 Hz. 4.2 Feature Extraction Analyses were performed offline using Matlab version 2016a. Since in classification the test data should be completely unseen by the training data, before extracting features the entire data is segmented into two parts regarded as train and test. Then, inside each set, various sub-windows are defined (Figure 4.2). Each window was used to generate one training point. Figure 4.2: Segmentation of each task To mitigate carry-over effect, the average of last two minutes of signal in previous baseline is subtracted from each task. 24

32 ECG: Since the duration of data recording was short, extraction of frequency-domain features was deemed unfeasible and only features in time domain were acquired. These included statistical temporal features, namely, mean, maximum, minimum, standard deviation, slope, and median of top and bottom quartiles of signal were derived from heart rate and RR intervals. Respiration: Analogous to ECG, the same statistical features from respiration rate and respiration interval were obtained. Skin Conductance: Number of SC responses, mean, and slope were extracted from SC signal. Temperature: Statistical features including mean, standard deviation, minimum, maximum, and slope of the signal were obtained. Table 4.1 summarizes features of each sensor. Table 4.1: summary of features, SC: Skin Conductance, ECG: Electrocardiogram Number Feature Number Feature 1 Mean RR interval 17 Median of top quartiles of respiration intervals 2 Minimum RR interval 18 Median of bottom quartiles of respiration intervals 3 Maximum RR interval 19 Mean respiration rate 4 Standard deviation of RR intervals 20 Minimum respiration rate 5 Median of top quartiles of RR intervals 21 Maximum respiration rate 6 Median of bottom quartiles of RR intervals 22 Median of top quartiles of respiration rate 7 Mean heart rate 23 Median of bottom quartiles of respiration rate 8 Minimum heart rate 24 Slope of respiration signal 9 Maximum heart rate 25 Mean temperature 10 Standard deviation of heart rates 26 Standard deviation of temperatures 11 Median of top quartiles of heart rates 27 Minimum temperature 25

33 12 Median of bottom quartiles of heart rates 28 Maximum temperature 13 Slope of ECG signal 29 Slope of temperature signal 14 Mean respiration interval 30 Mean SC 15 Minimum respiration interval 31 Slope of SC signal 16 Maximum respiration interval 32 Number of SC responses 4.3 Feature selection Given that the number of features is larger than training sample size, using all features in classification may lead to overfitting and curse of dimensionality which will give rise to reduction in prediction power of the classifier. Therefore, the most useful features should be determined. To this end, sequential forward selection and backward elimination, a methods commonly used in previous works was used. Forward selection algorithm starts with an empty set and in each run adds subsets of features not yet selected that best predict the labels until there is no improvement in prediction. Backward elimination, on the other hand, starts with a full set of features (here with the features already selected by forward algorithm) and sequentially removes features until eliminating more features does not boost the prediction. At each run of cross-validation the subset of features is divided to test and train sections. The latter one is used to train a model (here linear discriminant), and then label values for test data are calculated using that model. In the cross-validation calculation for a given candidate feature set, the number of misclassified observations was considered as loss measure to evaluate each subset. The output of this stage which was the input for classification problem was a matrix whose rows corresponded to data points obtained in each subwindow, and the columns corresponded to selected features. 26

34 4.4 Classification In this thesis, we addressed two classification problems: 1) differentiating between high arousal/negative valence and high arousal/positive valence, and 2) differentiating between low arousal/negative valence and low arousal/positive valence. We tested three classes of classification techniques, namely, K-Nearest Neighbour (KNN), linear discriminant analysis (LDA), and support vector machines (linear and kernel). These classifiers were chosen as representatives of linear and nonlinear algorithms which have been previously used in automatic classification of emotions. To further improve classification accuracy, we combined the output of the individual classifiers. This model allows for a consensus-based decision making process and has been shown to improve accuracy in various classification problems [78, 79, 80]. While different methods are available for classifier combination, we have selected the weighted majority vote scheme for this application. In this case, the final classification decision for n th data point x n, with y i as predicted label for i th classifier is defined as: Where the weight of each classifier decision w i is derived based on its training error e i [78]: denotes training error of the i th classifier defined as: Where N incorrect and N test show the number of misclassified test points and the total number of test points respectively. 27

35 4.5 Performance Evaluation Our primary outcome measure was classification accuracy defined as: Where N correct indicates the number of correctly classified test points. Classification performance was evaluated through cross-validation by randomly segmenting the data into train and test sets for 100 times, training the classifier on the training set and averaging over acquired accuracies on the test set. Accuracy of different classifiers was compared using the rank-sum test, with the null hypothesis that the classifiers perform similarly. 28

36 Chapter 5 Results 5.1 Participants Demographics A total of 15 participants with ASD participated in the study. All participants successfully completed the study. However, due to technical issues the data from one participant were excluded from analysis. The demographic information of the other participants is shown in Table 5.1. The list of medications that participants use is as follows: Biphentin, Abilify, Cetera, Ventolin,, Risperidone, Cepralex, Valproic Acid. Table 5.1: Demographic information Age (years) 14.9±1.8 Sex (Male:Female) 12:3 SCQ Score 19.9±5.8 Full-Scale IQ 99.6±19.4 Medication (Yes:No) 7:8 CBCL (Internalizing Problems) 61.4±7.2 CBCL (Externalizing Problems) 56.9±9.9 CBCL (Total Problems) 63.4±8.8 29

37 5.2 SAM Results The results of child and parent assessments are summarized in Tables 5.2, 5.3, and 5.4. These results were obtained after dichotomizing the valence into positive and negative as well as the arousal into high and low. As seen in general there is poor agreement between ratings. The only exception is agreement between child s ratings and actual label for positive stimuli. Table 5.2: Overall SAM results for each participant Agreement (%) Participant Child & Actual Parent & Actual Child & Parent Mean 25± ± ±14.9 Table 5.3: Emotion specific SAM results for each participant 30

38 Agreement between child & actual labels (%) participant Low High Positive Negative Mean 46.4± ± ± ±32.8 Table 5.4: Emotion specific SAM results for each parent Agreement between parent and actual labels (%) Participant Low High Positive Negative

39 Mean 46.4± ± ± ±24.9 Tables 5.5 and 5.6 demonstrate confusion matrix for children and parents assessing high and low arousal as well as positive and negative valence. Errors were due to negative stimuli being rated as positive and positive ones being assessed as negative. As it can be seen, for both parent and child the number of mislabeled pictures for high stimuli is slightly more than low stimuli. Regarding valence, for parent the error is comparable between two cases; however, children assessed positive pictures more correctly than negative ones. Table 5.5: Confusion matrix for child SAM ratings Actual labels Child choice n=112 High Low Neutral High Low Actual labels Child choice n=112 Positive Negative Neutral Positive Negative

40 Table 5.6: Confusion matrix for parent SAM ratings Actual labels Parent choice n=112 High Low Neutral High Low Actual labels Parent choice n=112 Positive Negative Neutral Positive Negative Figure 5.1 shows child s ratings on high and low arousal. The blue and red bars denote high and low arousal respectively. The horizontal black bar indicates the desirable case in which both arousal and valence are selected equally 4 times as it was intended. It can be found that for none of the participants ratings are balanced. In 5 cases (participants 1, 2, 8, 13, and 14) we only have one bar instead of two which means that the child only has selected one type of arousal (i.e. either high or low). This restricts us from using their labels for classification as two different labels are required for two classes. 33

41 Child's rating Participant Figure 5.1: child s rating on high and low arousal Figure 5.2 shows child s ratings regarding positive and negative valence with blue and red bars respectively. Again, here the last bar shows the ideal rating which is 4 selections for each case. Here participants 1, 9, and 13 only selected one type of valence which limits us to use their labels for classification. 34

42 Child's rating Participants Figure 5.2: child s rating on positive and negative valence 5.3 Classification Results Tables 5.7 and 5.8 represent classification accuracy for each classifier in comparison of high arousal/positive valence vs. high arousal/negative valence, and low arousal/positive valence vs. low arousal/negative valence respectively. The results are not significantly different for various classifiers as examined by rank sum test. Table 5.7: Accuracy of comparing HP vs. HN for each classifier High/positive vs. high/negative (%) Participant KNN3 KNN5 KNN7 LDA SVML SVM poly SVM rbf

43 Mean 70.6± ± ± ± ± ± ±11.8 Table 5.8: Accuracy of comparing LP vs. LN Low/positive vs. low/negative (%) Participant KNN3 KNN5 KNN7 LDA SVML SVM poly SVM rbf

44 Mean 77.3± ± ± ± ± ± ± Ensemble of Classifiers To improve outcome the classification outputs were combined using weighted majority vote as described earlier. Table 5.9 summarizes the classification results of ensemble of methods for two classification problems as well as maximum accuracy obtained from individual classifiers. Although the average result is improved by combining classifiers, rank sum test showed that it is not significantly different from the maximum distinct result. Table 5.9: Classification results of ensemble of methods Participant Accuracy (%) HP vs. HN Max accuracy of HP vs. HN for individual classifiers(%) Accuracy (%) LP vs. LN Max accuracy of LP vs. LN for individual classifiers(%)

45 Mean 81.5± ± ± ±11.8 Table 5.10 compares the results of ensemble of classifiers with and without weight. As it can be seen applying weight causes the average result improves. However, rank sum test specified that it is not significantly different from un-weighted combination. Table 5.10: Comparing un-weighted and weighted ensemble of classifiers Participant Weighted accuracy (%) HP vs. HN Un-weighted accuracy (%) HP vs. HN Weighted accuracy (%) LP vs. LN Un-weighted accuracy (%) LP vs. LN Mean 81.5± ± ± ±11.1 Table 5.11 shows the results of classification using mixed labels for which chance accuracy was acquired. It affirms the authenticity of results of ensemble of methods for which rank sum test represented they are significantly different. 38

46 Table 5.11: Classification results using shuffled labels Chance accuracy (%) Participant HP vs. HN LP vs. LN Mean 49.5± ±3.7 Table 5.12 contains confusion matrices associated with accuracies resulting from ensemble of methods. For both classification problems the number of misclassified cases is comparable between two classes. However, the first classification considering high arousal has slightly more misclassified points than comparing low arousal in second problem. 39

47 Table 5.12: Confusion matrices of ensemble of methods Actual labels Predicted labels n=8400 High/Negative High/Positive High/Negative High/Positive Actual labels Predicted labels n=8400 Low/Negative Low/Positive Low/Negative Low/Positive The results of two classification approaches are shown in Figure 5.3 to easier interpret. For all participants the accuracies are above 70%. As it can be understood from the bars, there are variations between individual results. 40

48 Figure 5.3: Bar plot of results of ensemble of classifiers 5.5 Classification over arousal axis The goal of this thesis was to find patterns along valence axis as it has already been shown that the difference between high and low arousal is detectable [13]. To examine the veracity of this supposition we also have discriminated data in high/positive vs. low/positive as well as high/negative vs. low/negative classes. The results are demonstrated in Table It can be inferred that the accuracies are considerably better than chance which confirms the possibility of distinguishing pattern in arousal level. Table 5.13: Discriminating high vs. low arousal Participant HP vs. LP (%) HN vs. LN (%)

49 Mean 85.0± ± Modality Specific Results To investigate the effect of physiological modality on the final results classification was performed using features of each signal separately. Table 5.14 sums up the sensor specific results of ensemble of classifiers. Rank sum test has indicated that there is no significant difference between the results of each sensor. Table 5.14: Signal specific results, SC: Skin Conductance, Resp: Respiration, Temp: Temperature High/positive vs. high/negative Low/positive vs. low/negative Par ECG SC Resp Temp All All ECG SC Resp Temp sensors sensors

50 Mean 73.4± ± ± ± ± ± ± ± ± ±7.8 We ran the classification for each modality without feature selection. As can be seen in Figure 5.4 the accuracies for all the participants, except one result for participant 4, are above 70%. Figure 5.4: Accuracy results with full feature set for each signal 5.7 Selected Features A histogram of selected features for two classification methods is demonstrated in Figure 5.5. As the figure shows, average SC signal is the most frequent selected signal followed by average temperature in both cases. There is no feature that is not selected at any time. 43

51 Figure 5.5: Selected features, RR-int: RR intervals of ECG, HR: heart rate, RI: respiration intervals, RR: respiration rate, Resp: respiration, Temp: temperature, SCR: skin conductance response Top ten features for each classification problem are listed in Table Features from all four sensors are among the most selected features. Table 5.15: Top ten selected features Feature ranking HP vs. HN LP vs. LN 44

52 1 Mean SC Mean SC 2 Mean temperature Mean temperature 3 Minimum temperature Minimum respiration rate 4 Minimum respiration rate Mean RR interval 5 Standard deviation of temperature Mean temperature 6 Slope of SC signal Standard deviation respiration interval 7 Mean respiration interval Minimum respiration interval 8 Slope of temperature signal Mean respiration rate 9 Mean respiration rate Maximum RR interval 10 Minimum respiration interval Slope of temperature Figure 5.6 includes heat maps indicating selected features for each participant in low arousal, high arousal, and subtraction of low from high arousal respectively. The numbers on each plot signify the number of times the feature was selected in 100 runs of classification. 45

53 (a) Frequency of selecting each feature in classifying low/positive vs. low/negative 46

54 (b) Frequency of selecting each feature in classifying high/positive vs. high/negative 47

55 (c) Difference of frequency of feature selection in two classification problems Figure 5.6: Frequency of feature selection for each participant for: (a) low arousal, (b) high arousal, (c) subtracting a from b. The numbers on the plot show the number of times each feature is selected out of 100 runs of classification 48

56 5.9 Association of Classification Accuracy and SAM Ratings with Demographics We conducted linear regression analysis to examine the effect of demographic information on classification accuracy. The results are summarized in Tables 5.16 (a) and (b). As it is indicated age, gender, IQ, and CBCL scores do not have significant effect on classification accuracy. However, SCQ score has significant effect on accuracy in comparison high/positive vs. high/negative states. We may not detect effects for other demographics due to inadequate power caused by limited sample size. Figure 5.7 demonstrates the scatter plot of accuracy results of classifying high/positive vs. high/negative against SCQ score. It indicates that the higher SCQ score, the lower accuracy. Table 5.16: Effect of demographic information on classification accuracy Regression slope Standard Error t-stat P-value Age Gender IQ SCQ * CBCL-Internalizing problems CBCL-Externalizing problems (a) Effect of demographic information on classification results of high/positive vs. high/negative Regression slope Standard Error t-stat P-value Age Gender IQ SCQ CBCL-Internalizing problems CBCL-Externalizing problems

57 (b) Effect of demographic information on classification results of low/positive vs. low/negative Figure 5.7: Accuracy of classifying HP vs. HN against SCQ scores Tables 5.17 (a) to (d) show the effect of demographics on the results of consistency of children s rating with actual labels. As can be seen, none of the parameters has significant effect on the results. Agaian, we may not be able to detect the effects due to small power because of limited sample size. Table 5.17: Effect of demographic information on SAM results Regression slope Standard Error t-stat P-value Age

58 Gender IQ SCQ CBCL-Internalizing problems CBCL-Externalizing problems (a) Effect of demographic information on results of consistency of child s ratings and actual labels for stimuli with low arousal Regression slope Standard Error t-stat P-value Age Gender IQ SCQ CBCL-Internalizing problems CBCL-Externalizing problems (b) Effect of demographic information on results of consistency of child s ratings and actual labels for stimuli with high arousal Regression slope Standard Error t-stat P-value Age Gender IQ SCQ CBCL-Internalizing problems CBCL-Externalizing problems (c) Effect of demographic information on results of consistency of child s ratings and actual labels for stimuli with positive valence Regression slope Standard Error t-stat P-value Age Gender IQ

59 SCQ CBCL-Internalizing problems CBCL-Externalizing problems (d) Effect of demographic information on results of consistency of child s ratings and actual labels for stimuli with negative valence 5.10 Effect of Window Size on Accuracy Figures 5.8 and 5.9 indicate the effect of window size in which features are extracted to be used in classification as data points on the performance of classifiers. Each bar denotes accuracy of ensemble of methods for one specific segmentation. Different window lengths are listed in Table As examined by rank sum test there is no significant difference between various cases. Figure 5.8: Results of various segmentations in classifying LP vs. LN 52