The neural correlates of narrative empathy a functional magnetic resonance imaging study.

Transcription

1 The neural correlates of narrative empathy a functional magnetic resonance imaging study. Thesis submitted in partial fulfillment of the requirements for the degree of DEGREE (Doctor of Philosophy) in Cognitive Science by Kavita Vemuri kvemuri@iiit.ac.in International Institute of Information Technology, Hyderabad (Deemed to be University) Hyderabad , INDIA July

2 Copyright Kavita Vemuri, 2017 All Rights Reserved 1

3

4 Acknowledgement Research in cognitive neuroscience is not possible without the participants/subjects who volunteer their time and brain for the studies. This thesis dedicates and acknowledges the participants young and old alike for taking part in surveys, eye tracking experiments (students/staff of International Institute of Information Technology, Hyderabad) and in functional magnetic resonance imaging studies (the participants at the National Brain Research Institute, Manesar, India). The interest and enthusiasm of the participants have made this work possible. Most importantly their participation proves that empathy exists! With thanks, I acknowledge my thesis advisor, Professor Bapiraju Surampudi, for the expertise in the domain and focus he patiently provided to the study. Also extend a big thanks to Dr. Nandini C Singh of NBRC for help in conducting the experiments at NBRC. Thank you. True empathy is to have no perceptions, but the ability to perceive. then the whole being listens. 3

5 Contents Abstract 6-8 Overview of the thesis 9-12 Chapter 1: Defining Empathy Introduction 1.1 Empathy Components and Models 1.2 Neuroscience of Empathy Pain-Empathy Empathy from Emotional expressions Relation between Altruism/Moral Judgment and Empathy Lesion studies Clinical Studies of Empathy 1.3 Summary 1.4 Some gaps in empathy research Appendix Chapter 2: Narrative Empathy Introduction Defining a Narrative 2.1 Empathy/emotion to Cinematic Narratives 2.2 Neuroscience of narrative empathy 2.3 Summary 2.4 Motivation for the Proposed Experiments 2.5 The challenges of movie as stimuli to study empathy response 2.6 The hypothesis of this study 2.7 Data-analysis, the challenge from naturalistic stimuli Chapter 3: Functional MRI and Data analysis Introduction - Functional Magnetic Resonance Imaging (fmri) 3.1 Experimental Designs 3.2 Pre-Processing of fmri data 3.3 Task-related Data analysis General Linear Model Data-driven method Independent Component Analysis sica of FMRI data Group ICA Pre-processing for ICA Sorting of the ICs 3.4 Functional Network Connectivity 4

6 3.5 Data analysis for the studies reported in this thesis Appendix Chapter 4: Evidence of Empathy response brain networks Introduction - Evidence of Empathy response brain networks 4.1 Material and Methods Subjects Stimuli Rating the movies 4.2 fmri imaging and preprocessing Data analysis 4.3 Results Animation movie Hollywood movie Indian Hindi Movie Comparative Analysis of Uncorrected versus Corrected Results 4.4 Discussion Animation Movie Hollywood movie Hindi Movie 4.5 Summary 4.6 Conclusions 4.7 Limitations and future directions Appendix Chapter 5: Dynamic functional network connectivity of empathy and default mode networks Introduction - Dynamic functional network connectivity of empathy and default mode networks 5.1 Methodology Subjects fmri image acquisition Preprocessing and data analysis Selection of ICs Dynamic functional network connectivity analysis 5.2 Results Animation Movie (S1) Indian Hindi Movie (S2) Hollywood movie (S3) 5.3 Discussion Functional Connectivity S1 (animation movie) 5

7 5.3.2 Functional connectivity for S2 (Indian Hindi movie) Functional Connectivity for S3 (Hollywood English movie) 5.4 Limitations 5.5 Conclusion Appendix Chapter 6: Empathy response to long and short narratives Introduction - Empathy response to long and short narratives 6.1 Materials and Methods Participants Stimuli and Survey fmri Experimental Design Image acquisition Data Analysis 6.2 Results Survey Neutral Narrative Emotional Narrative 6.3 Discussion Neutral scenes Emotional scenes 6.4 Conclusion and Limitations Appendix Chapter 7: Conclusion Summary 7.2 Proposed model of empathy based on the findings Appendix: Selecting task-relevant Independent Components References

8 Abstract Empathy entails the ability to understand (perspective-taking) and share the affective experiences of others, both processes considered the bedrock of all social interactions and critical for survival. Most empathy related neuroscience studies employ simple paradigms like pain-infliction or static images with emotional context and analysis focuses on either perspective-taking processing or affective experience overlooking the complex interaction among the processes. The initial studies informed how one understands pieces of the whole system with an assumption that when information from the pieces is fit the whole picture can be realized but this methodology has led to constrained theories on empathy process. To address the short-comings in understanding empathy, the focus of affective neuroscience research has shifted to studying this phenomenon as part of naturalistic social cognition. Hence with the goal to mimic real-life situations that evoke empathy, naturalistic or ecologically-valid paradigms are recent inclusions building upon the foundation provided by the earlier simple models. This study focuses specifically on narratives and empathy response. The aim of this study is to explore the neural correlates of narrative empathy using multi-modal visual movie narrative. Towards this, two functional magnetic resonance imaging (fmri) studies were conducted using a total of 5 commercial full-length short film and movie clips. The first experiment had 3 movie clips (5-8minute long) from a diverse genre (animation, Hollywood and Indian Hindi movie) and the goal of this exploratory investigation was to identify the networks of empathy cognitive (mentalizing), emotional and motor (affective experiences) and analyzing the dynamic nature of the neural activations as a function of the self-reported level of empathy experienced. The data-driven independent component analysis (ICA) method 7

9 applied to isolate the underlying brain networks revealed: a) the ability of the fictional narratives induces emotional contagion and the potential to elicit empathy response b) activation of significant, distinct but overlapping empathy networks and c) the dynamic cross-correlation analysis between the task default-mode network (DMN) and individual empathy networks revealed that emotional empathy response displayed lower correlation while cognitive empathy higher correlation with DMN, suggesting the possible role of self-reflection or attention to an internal narrative triggered by external stimuli (Andrews-Hanna et al., 2012; Gusnard et al., 2001). This is a potentially significant finding for understanding the difference in cognitive and emotional empathy in clinical conditions like autism and psychopathy. Developing on the findings from the first study, a second fmri study was designed to understand specifically the role of context in empathy response. Toward this, two diverse full-length short movies (8 minutes each) with real-life actors of ethnicity dissimilar to the participants were selected as stimuli. Functional imaging data was collected from two sets of extremely short clips (13-20seconds) with neutral and emotional scenes extracted from the full-length movie followed by the whole movie. Using a general linear model approach for data analysis, the contrasts estimated from each condition (activation for the short clips of neutral and emotional scenes versus the same clips in the full-length movie) were compared and the analysis showed that empathy sub-processes (emotional, cognitive and motor) were statistically significant when context is presented. The results also reveal the complex differential empathy processes in real-life social interactions, where responses are a function of context. Overall, the neural correlates of narrative empathy presented in this thesis, the first of the kind, suggest that a complex construct such as empathy should be best studied in with ecologically valid stimuli. The impact of the results, in studying empathy deficiency and interventions using narratives to enhance empathy, needs to be explored in the future.

10 General overview of the thesis An observer, who shares another s affective state, while being fully aware that the source of the self-feelings is the other, is said to display an empathy response. The response requires the observer to keenly perceive or imagine the other s state to trigger a similar feeling in self. Empathy connects disparate human beings based on one s ability to interpret reason and assign motive for other s state. Paucity of information compels one to rely on subjective inferences resulting in multifarious responses. Hence, a narrative enables a cognitive appraisal of the state of the 'other' which in turn allows for informed empathy response in contrast to images with vicarious emotional gestures but sparse contextual information requiring inference. The conjecture of the study rests on the hypothesis that fleeting emotional expressions with no specific contextual information can rarely evoke empathy unless one has the experience or the imagination to make knowledgeable inferences. That is, in addition to cognitive inference of what the affective state of the other is, empathy response is aided by knowledge of 'why' the other s state and this defines the understanding required for sharing of the other's feelings. A narrative is a method of conveying the reason for the state of the other by threaded events presented in a temporal sequence either in text, verbal or audio/visual format and is an integral part of social communication. Readers or viewers experience immersion in a narrative by dissecting the events presented and taking the perspective of the agent(s). Narrative is defined as a sequence of temporal events with a promise of an end (Keen 2007, 2010) to ensure attention and engagement. Empathy response to narrative requires that the viewer follows the beliefs and thoughts of the characters and allows the experience to influence emotions comparable to that depicted by character or as appropriate to the context. The brief relationship the viewer builds with the other can be a state where he/she understands the feelings expressed by the other by comprehending the context 9

11 (Gallagher, 2012) and extends to reciprocating the feelings, both evaluations considered to be the basis for empathy (Decety and Jackson, 2004). The neural networks underlying empathy responses as reported by studies using simplistic controlled stimuli have provided understandings into the crucial areas for motor mimicry, emotion recognition and perspective taking. Building on these findings, the study undertaken for this thesis applies a diverse set of movie narratives including a computer-generated animation movie to analyze the empathy response. Secondly, the study reported in this thesis, also examines the role of context (longer narrative versus short non-contextual clip) on empathy response with two full-length short-movies. Investigating the neural correlates of empathy for a long narrative, as in movies, where the viewer is informed in detail the reasons for the affective state of the actor(s) gives: a) the dynamic changes in the empathy networks as the viewer takes the perspective of the actor or processes the events unfolding in a movie, b) identifies the dynamic correlations of the three empathy networks cognitive, emotional and motor and c) highlights the difference in empathy networks for short movie clips with no context and that of the same scenes in the longer movie clips the findings are important for basic understanding of the how/why of empathy response. Importantly, empathy studies using ecologically-valid stimulus has been receiving a lot of attention, mostly because the data provides insights into the dynamic neural circuits of empathy responses in real-life. The purpose of this study is an attempt to answer some of the rudimentary questions on empathy response to narratives. Chapter 1 sets the context of the study by defining empathy, the processes underlying it, the neuroscience of empathy derived from pain-empathy studies, the findings from clinical or lesion cases and the various models proposed to date. The chapter concludes with the identification of research gaps. In Chapter 2, 10

12 narrative empathy is introduced followed by a literature review of the underlying neural correlates when contextual processes are involved. The chapter includes a broad set of hypotheses motivating the functional magnetic resonance imaging experiments carried out using visual narratives as part of this thesis. The challenges in analysis of fmri data collected from free-viewing designs long continuous stimuli with no task other than viewing and the inferences that can be drawn from the results is also covered. Chapter 3 introduces the current methods being applied and the criterion for the methods selected for analyzing the fmri data collected as part of this study. In the first study (Chapter 4), the aim was to understand the neural substrates of empathy response for three very diverse movie clips, rated for emotional valance, in a free-viewing no-task design. To isolate probable empathy networks the blind data-driven independent component analysis (ICA) method was applied on the data collected from 15 normal-healthy subjects. Group ICA reveals distinct networks reported for cognitive, emotional and motor empathy. The changing events in the narrative result in empathy attributed brain activations showing timevarying correlations which are presented in Chapter 5. The dynamic correlations between an independent component with activations associated with the taskposterior default-mode network and that of isolated empathy network independent components were examined to understand differences in functional connectivity of the cognitive, emotional and motor empathy related intrinsic networks. In the second fmri study (Chapter 6), having established by empirical evidence empathy networks for movie narratives and the dynamic functional connectivity correlated to the emotional valence of the movie clips, the focus was on comparing the effect of context or longer narrative on empathy response. Short clips from the longer movie with emotional and neutral scenes were shown to 16 normal-healthy participants and activations from this sequence were compared to activation when the short clips are part of the longer narrative. The goal was to show that complex 11

13 construct like empathy is a slow mediated process better elicited in a narrative which allows the viewer/reader to fully understand the context for the affective state of the other and then respond appropriately. In terms of processes, the findings highlight the causal relation of cognitive empathy to emotional/motor empathy response. A summary of the findings, conclusions, the limitations and the future goals of empathy neuroscience forms the concluding chapter (Chapter 7). The appendix at the end of the thesis are the selected task-relevant independent components isolated from the ICA method and used for the data analysis. 12

14 Chapter 1 Defining Empathy 1.0 Introduction The survival of life on the planet has its basis on our ability to co-exist with nature by an implicit understanding between humans and all other living beings by strong emotional bonds (like the one between the mother and child, to a stranger, to a tree etc.). Tomasello (2000), believes the success of the human species rests on the interpersonal skills, cooperation and ability to understand others. Co-existence is also possible with well-defined and monitored set of rules, but such a system fails to enhance quality of life and stems evolutionary behavior changes. Hence by the innate capability of human perception, knowledge and connecting to another being s consciousness one can realize the inner thoughts of the other to initiate an appropriate response. Animal species (including the Human) which have understood how the other s mind works or animals with developed consciousness bond with altruism and pro-social behaviors to survive as communities. Before discussing the origin of the term empathy and its myriad definitions it is vital to introduce important conceptual distinctions of the fundamental humanresponses that have been included or excluded over the century to arrive at an exact definition of empathy. The main components are sympathy, personal distress or emotional concern, emotional contagion and motor mimicry. Sympathy/emotional concern: Sympathy is a feeling of sorrow or concern for a needy person, with a goal to alleviate the suffering of the other (Eisenberg, 2000). Personal Distress/emotional concern: Personal distress is distinctive as a reactive emotion in response to other s negative emotion or situation. That is, while it is other-caused phenomenon like sympathy it is mostly self-experienced. 13

15 Emotional Contagion: A condition when people start feeling similar emotions just by association, for example: feeling happy when around with others who are joyful or vice versa, wherein one experiences the emotions as one s own rather than as a translation from others. In some sense, it is also motor mimicry which by selfanalysis leads to emotional convergence (Hartfield et al., 1992). Motor mimicry: An overt action by the observer appropriate to the situation of the other rather than one s own. Adam Smith (1959), described this as reflexive action and as the most primitive form of sympathy. In early usage, motor mimicry was empathy, while sympathy was widely used to label emotional responses. In the latter half of the twentieth century, empathy became an all-encompassing mix of intense vicarious emotion/action (observed behavior), understanding of others while making the self-other distinction (inference from behavior) and role-taking (combining behavior with inferences). In a compelling hypothesis, Bavelas et al (1987) suggest that motor mimicry or primitive empathy is a parallel communicative process. Over the years, many allied empirical evidences have shaped motor mimicry, an important one being the identification of the mirror neuron account, where neurons predominantly in the motor cortex were found to fire even when observing another performing an action (Gallese 2001;2003; Rizzolatti and Sinigaglia 2008; Gallese, Keysers, and Rizzolatti 2004). With this, empathy research started to include motor empathy, that is, the ability to reciprocate by similar actions/expressions/gestures. Personal distress, evident in social interactions is self-centered and a control mechanism to help reduce the impact of the other s distress on self. Additionally, social psychological research has identified different emotions for other versus selfimagination in a situation, with former resulting in sympathy and latter to selfdistress such as anxiety or discomfort (Batson, Early, & Salvarani, 1997). The basic distinction of sympathy being another-oriented response, personal distress being 14

16 self-oriented and mimicry being an action initiated in the self in response to other s actions point to possibly different underlying processes. However, it has not always been very clear whether the distinction arises due to context specificity of these three responses. Also, it has been difficult to isolate them in empirical research as they can occur in parallel. The need to understand self-other merger, self-other awareness and self-other distinction in empathy responses is critical to evolve appropriate experimental designs and the inferences that can be drawn from the results. Lerner (1980) and many others advocate that self-other merger enhances empathy while Hoffman (1974) and de Waal (1996) argued for a self-other distinctiveness. Eisenberg and Sulik (2012) in a critical review of Preston and Hofelich s (2012) work argues for a distinction between sympathy/compassion and suggest that true empathy refutes the latter s definition of self-other overlap as largely a function of self-experience. They cite examples of where one can feel empathy even if a particular state was not experienced personally. A very relevant example is the acute emotions that the observer feels for a probable loss for the other even though the event has not occurred yet. An inclusive approach to empathy was proposed by Davis (review: 1996), on four-scales: perspective taking, fantasy, empathic concern, and personal distress. The self-report survey based on these scales called the Interpersonal Reactivity Index is used to estimate individual differences on empathy response. Literature on empathy in the 80 sand 90 s concentrates on cognitive aspects like how one attributes metal states to another (Baron-Cohen 1997), with little or no reference to shared emotional experience. In the simulation theorists and theory theorists arguments, a representation of the target s mental state is generated in the process of empathizing with the target. A direct-perceptual access model contrary to the above says that one has direct access to one s own mind but also of the target via the perceptions formed of them (Zahavi 2011; Gallagher 2008). 15

17 However, the process of acquiring the perception model without emotional experience is not clear. In summary while many definitions have been proposed, debated and refuted there has been no consensus derived from empirical evidence to show the distinction between clearly other-oriented phenomena such as sympathy, empathic concern, compassion to that of empathy which is affective sharing involving feeling with/as the other person. In the next section, we discuss various models proposed by both theorists and experimentalists based on which brain networks are analyzed. 1.1 Empathy Components and Models Decety and Svetlova (2012) have included all the concepts like altruism, emotional contagion, emotional regulation, mentalizing (also called Theory of Mind (ToM)), personal distress and prosocial behavior, to define empathy. Each of the subsystem behaviors has neurobiological and evolutionary underpinnings and has been explored extensively under the larger system of empathy; a few of such studies are discussed in the chapter. While there is no strict consensus amongst researchers on which of the behaviors constitutes empathy, based on behavioral responses a widely accepted classification has been formulated by Walter (2012) from the initial structure analyzed by Singer and Lamm (2009). The first of which is affective-empathy, characterized by the following features: an affective state elicited by the perceived, imagined, or inferred state of another, which can also be similar (isomorphic) to the other s state and includes perspective taking for self-other distinction (Singer, 2006; review: Decety and Jackson, 2004; Keysers and Gazzola, 2007). Cognitive empathy refers to the ability to understand the feelings of the others, without necessarily taking on the affective state of the other. In many ways, this is like ToM or mentalizing the ability to take other s perspective (on beliefs, desires, intentions and emotions) and indulge in reasoning unlike the self (Premack & Woodruff, 1978; Frith & Frith, 2003; Decety & Lamm, 2006; Schnell, et al., 2011). Combining ToM and empathy leads to the following: 16

18 a) cognitive theory of mind = mentalizing about cognitive states, (b) affective theory of mind = mentalizing about affective states, which is also termed cognitive empathy and c) affective-empathy or emotional empathy (Walter, 2012). The different facets have been summarized by Zaki and Ochsner (2012) (Figure 1.1). An important distinction is that affective empathy is different from emotional contagion and mimicry- both of which are roughly automatic adoption of another s emotions but not necessarily make the self-other distinction, which is the root of empathy. Prosocial concern Empathic motivation Sympathy Empathic Concern Mentalizing Cognitive empathy Perspective taking Theory of mind Experience sharing Affective empathy Shared self-other representations Emotional contagion Figure 1.1: The main facets of empathy [reproduced from Zaki and Ochsner (2012)]. Lipps' theory (1903, summary in English at: ) was the forerunner of the perception-action model (PAM) in motor behavior and he explicitly implicated it in empathic processes. According to the perception-action hypothesis, perception of a behavior of other automatically activates one's own representations for the behavior, and output from this shared representation necessarily proceeds to motor areas of the brain where responses are prepared and executed. The distinction between PAM and empathy is that though it requires a subject to develop a similar emotional state as the target, it does not require that the subject feels the exact same subjective emotion as the target. In the PAM model proposed by de Waal, (2008; Figure 1.2), the self-other distinction is 17

19 discussed as a function of imitation and empathy. Motor mimicry and emotional contagion, being the lowest distinction (self-other) form the basis of PAM concept. Figure 1.2: The Russian doll model of empathy and imitation. The empathy induces the same emotional state in subject and target, with perception-action model (PAM) being the core. PAM comprises motor mimicry and true imitation as the basic and immediate response. The arrow on the left shows the self-other distinction as perspective taking increases.[ Reproduced from de Waal (2008)] In the Decety and Meyer (2008) model (Figure 1.3), the bottom-up and the topdown information processing are explicitly differentiated. The process of mentalizing and experience sharing both require one to understand the internal state of the other, the former is usually evoked by explicit inference about target s state. As the perception of the internal state of the other changes either due to changes in the presented stimuli or regulation by top-down processing dynamic shifts can happen among the three processes. 18

20 Figure 1.3: A schematic that shows the relation between the perception-action process and topdown processing involved in empathy. The continuity or dynamic nature of the empathy response is evident from the push-pull configuration of top-down and bottom-up processes. The meta-cognitive feedback plays a crucial role evaluating constantly one s own mental condition to react or not to the affective state of the other. [Reproduced from Decety and Meyer (2008)] Engen and Singer (2013) proposed a model (Figure 1.4) with many more factors that lead to empathy experience comprising a complex combination of regulation, generation and modulation of empathy. Empathic experiences are effected by internal representation of the state of the other by direct perception or from the internal knowledge and by the process of inference ascribe an affective state to the other. Though, lesion research (Shamay-Tsoory et al., 2009) indicates the two processes to be dissociable for experience of empathy, they would have to work in tandem for the process of generation of empathy. The regulation effect and modulator factors are entangled and hence render findings from behavioral or empirical studies subject to differing interpretations. 19

21 Figure 1.4: The factors and relationships involved in empathy as discussed by Engen and Singer (2013) [Figure reproduced from Engen and Singer (2013)]. Given the complexity of the phenomenon, investigating it with behavioral or neural data has been a challenge. Hence in summary, research has only progressed to the extent of converging on three main mechanisms of empathy, proposed initially by Blair (2005); cognitive, the ability to take other s perspective to understand their thoughts, emotional empathy, which is the ability to experience the emotions of the other and motor empathy, which is acting to alleviate the suffering of the other. This terminology will be followed in the research reported in this thesis. 1.2 Neuroscience of Empathy In this section, neuroscience studies of empathy are reviewed and the brain areas/networks that are attributed to empathy response are listed. Two main processes have been extensively studied mentalizing and sharing in pro-social contexts. Studies related to empathy induced by pain and emotional expressions are reviewed briefly. A small section is also devoted to empathy investigations on clinical conditions like psychopathy, autism spectrum disorder, schizophrenia and frontotemporal dementia. 20

22 1.2.1 Pain-Empathy Most of the initial empathy investigation focused on pain empathy- the neural correlates for observed and self-inflicted. For example, watching another pricked by a needle or a photograph by which one can infer that a human hand has been hit by a hammer or when the agent inflicts the same on self. The major contributions for pain empathy have been by the following authors - Morrison et al., (2004), Singer et al., (2004), Singer and Lamm, (2009), Apkarian et al., (2005) and pain combined with disgust by Jabbi et al., (2007). Empathy from just observation of pain was studied by Singer et al. (2004;2006). Feeling and observation of disgust was studied by Wickers et al. (2003) and the process of imagination of pain by Jackson et al. (2006a). Judgment of another in pain was investigated by Moriguchi et al. (2007) and perception and judgment of pain in others by Jackson et al. (2005). These extensive studies led to the proposal for a pain-matrix (Derbyshire SWG, 2000; Apkarian et al., 2005) of areas in the brain that were activated especially when combined with self-experience of pain. An interesting study (Preis et al., 2015) looked at habituation whether prior pain experience influenced empathy response to pain. These controlled experiments have facilitated in differentiating experiences into motor mimicry, emotional contagion and personal distress. Few selected studies are discussed in detailed in the paragraphs below. To investigate whether the shared activations are also possible in affective state, neural responses in female partners was measured by the functional Magnetic Resonance Imaging (fmri) technique when pain was inflicted on her or the male partner (Singer et al., 2004). Interesting aspect of this experiment was that abstract symbolic cues (color codes) instead of emotional expressions were used which means the empathic responses were independent of mimicry and emotional 21

23 contagion. The experiment was designed to compare pain-related brain activity in the context of self and other. They report pain-related activation for self in the anterior cingulate cortex and anterior Insula peak early (2-4s) and again at 8-12 seconds along with sensorimotor and secondary somatosensory cortex. The areas that showed activation for self/other pain condition included anterior cingulate cortex, bilateral middle insula, anterior insula and lateral cerebellum. They also found that pain-related activation in contralateral somatosensory, secondary somatosensory cortex/posterior insula and caudal anterior cingulate cortex are present for self-experienced pain and not for perceived pain in others. Botvinick et al (2005) showed participants (patients with shoulder-pain) depicting emotional facial expressions of pain and no-pain. Bilateral insula and anterior cingulate cortex showed higher activation for painful stimulation than non-pain stimulation. Saarela et al., (2007) studied the correlation between the viewer s estimated intensity of pain in faces of chronic pain patients and found that the strength of the activation in the left anterior insula and left inferior frontal gyrus during the observation of the intensified pain correlated with their self-rated empathy. With a similar goal, Jackson et al. (2005) showed participants 21 photographs of just hands and feet in possible painful situations, for example, a photograph of a hand between a door and its hinge and another of person cutting a cucumber with a knife etc. The participants were asked to gauge the pain experienced by the person (no face was shown) and from the fmri data found activation in the pain-matrix plus in the thalamus. When correlated with participant s ratings of the other s pain, higher activity in the anterior cingulate was seen, suggesting that this brain area is modulated by the subject s response to other s pain. By inflicting a moderately painful pinprick to the fingertips of the participants and witnessing another s (stranger) hand also being pinpricked showed common activity in the right dorsal anterior cingulate cortex (role in coding motivational-affective dimension of pain) 22

24 while the primary somatosensory cortex did not show any significant activity for just visual stimuli but showed response to tactile sensory (Morrision et al., 2004). In an interesting study to confirm overlapping neural activations in the brain areas attributed to pain, Rütgen et al.(2015a; 2015b) induced placebo analgesia, to modulate the first-hand experience of pain. The placebo effect was to test whether this reduces empathy for pain. A self-report and event-related potentials (ERPs) collected when participants were subjected to painful electrical stimulation or witnessed another going through the same, showed reduced amplitudes of the pain-related P2 (an ERP component indexing neural computations for pain). While there was no placebo analgesia effect on conditions unrelated to pain. The inference from this study was that a common neural process was engaged for firsthand emotion experience. The response to observation of pain inflicted or threat of infliction is considered empathy, though Singer et al., (2004) and Danziger et al., (2009) note the possibility that watching painful scenes might not induce isomorphic vicarious experiences, a point that has been contented in studies where the other is someone dear to the subject. Lamm, Decety and Singer (2011) conducted meta-analysis of 32 fmri studies using images for detection of pain empathy. They found that the stimuli used in the experimental paradigm determines the core network, for example the inferior parietal/ventral premotor cortices were recruited when the subjects viewed pictures of body parts in painful situations, and abstract visual information about the other s affective state requiring inferring the self-other distinctiveness engaged the precuneus, temporo-parietal junction, ventral medial prefrontal cortex and superior temporal cortex. The somatosensory areas were activated only for picture based paradigms. The idea that pain empathic experience is not a set of disparate areas but can be best understood by functional connections between these areas 23

25 has been recently presented by Betti & Aglioti (2016). By suggesting a networkbased approach, they propose that pain is not only affective state but that the sensorimotor can be empathically modulated. By this proposition they state that the brain first generates a unique mental representation of other s pain and then forms the appropriate motor response, basically bringing in a cognitive processing element. In a critical analysis of the much-reported overlap between empathic and nociceptive pain, Iannetti et al. (2013) using the more sensitive multivariate voxel pattern analysis(mvpa) show that the so-called pain matrix was not specific to pain at all. The discord in pain empathy findings was discussed in recent review by Zaki et al. (2016), where they address the controversy of the relationship between nociceptive pain (biological/physical in origin) and empathic pain (observing others in pain). The debate has been on whether empathic experience of pain can be classified as pain or just an experience and the authors (Zaki et al., 2016) suggest that research on pain empathy take a more holistic approach to seek answers to basic but intriguing questions like what role does context and individual traits play for the two pain types? or given the biomarkers for different sensory processing, can one differentiate the areas specific to each pain type?. Hence, as in most empathy related research, the first concept of using pain to understand empathy related brain areas is still a research in progress. The brain activations from these studies lead to labeling a pain-matrix with the following brain areas: the bilateral anterior insula (AI), rostral anterior cingulate cortex (ACC), brainstem, and cerebellum valid even when someone dear experienced pain. Interestingly the activation in this network was also observed when the stimuli was painful facial expressions (Lamm et al., 2007). A metaanalysis of the pain related empathy distinct and common areas (Lamm, Decety 24

26 and Singer, 2011) and another similar analysis (Fan et al., 2011) also report the presence of the above areas for pain empathy response. In summary, empathy for pain includes multiple processes like judgment of other s pain, perception, the self-oriented personal distress and sympathy (other-oriented) affective process. These empathy components are driven by the bottom-up sensory inputs like facial emotions, pain cues, context presented etc., and the topdown process of prior experience of pain, control and contextual understanding (Gourbert et al., 2005; model presented by de Waal, 2008 described above) Empathy from Emotional expressions In addition to studies to identify activations for empathic pain with stimuli comprising of either image depicting pain or facial expressions of pain, the task of identifying the brain areas that respond to expressions indicating sadness, happy, disgust, anger etc., by imitation have also been the focus to understand empathic resonance. The findings from such studies help in isolating the neural correlates for comprehension of emotional faces and imitation. Using a series of short-video clips with individuals telling sad and neutral stories, a Positron emission tomography (PET) technique was used to investigate the neural correlates of sympathy (Decety and Chamanide, 2003). The actors telling the stories depicted congruent or incongruent motor expression of emotion and at the end the subjects were asked to rate the mood of the narrator. As expected, sad stories invoked increased activity in the emotion and shared representation - processing areas comprising the bilateral inferior frontal cortex, right dorsal premotor cortex, bilateral superior frontal gyrus, and bilateral amygdala. For the sad/happy vs. neutral motor expression of emotion, the left inferior frontal gyrus was common to both emotions while the left dorsal premotor was only identified for the sad emotion expressions. Interpreting these results from the rating (on 25

27 liking ) provided by each participant for the narrator, it was shown that the cognitive content of the story and the motor expressions of the narrator recruit motor and emotion networks. In contrast, for the incongruent condition, brain areas associated with both emotional and cognitive conflicts were activated. To arrive at a functional architecture for empathy, Carr et al., (2003) looked at how action or imitation representation modulates emotional activity. They used a set of face photographs depicting emotions and collected data by the fmri technique, when normal healthy subjects were either imitating or observing the emotional facial expressions. The main findings were the significantly increased activity in the right amygdala, premotor face area, inferior frontal gyrus, the insula and superior temporal gyrus during imitation than for mere observation which signifies the modulation of the action representation circuit onto limbic activity. A similar fmri study by Leslie et al.(2004) using movies with facial expressions (smiling or frowning) and hand movements report that passive viewing of faces led to significant activation in the right ventral premotor area, whereas imitation showed activation in bilateral premotor area activity. They support their findings of right hemispherical dominance for emotion processing on prior work (Adolphs et al., 1996). With an aim to identify the overlapping and distinct neural mechanisms for self and other related emotional response, Schulte-Ru ther et al., (2007) showed participants photographs with emotional faces with direct or averted gaze. The participants were asked to focus on own emotional response or evaluate the emotional state as represented by the face image. The left lateral orbito-frontal and medial prefrontal cortices, bilateral inferior frontal cortex, temporal poles and cerebellum were the common network for both conditions. The other-oriented condition which is the empathy-related processing involved the mirror neuron and ToM network. For the self-oriented condition, which requires the subject to evaluate own emotional states, the medial prefrontal cortex, precuneus and the temporo-parietal regions were significant. 26

28 In children, as young as 8 months, studies have looked at comforting behavior directed to another in distress (Davidov et al., 2013). Reciprocation by facial expressions and vocalizations, reflecting empathic concern were found to be present at a very young age. By three years of age, contextual appraisal plays a role as reduced empathic concern is demonstrated based on the type of distress shown by the other. As children grow and take on a position in the society, they learn to regulate their empathy towards deserving people (Li et al., 2013) Relation between Altruism/Moral Judgment and Empathy Does empathy lead to altruism or is altruism critical for empathy has been a point of debate, with Sober and Wilson (1998) taking the view that empathy makes one to behave altruistically to help others in distress. In older neurotypical adult s empathy is core to social interactions as it regulates behaviors like social-grouping of people with similar emotional states (Cikara & Van Bavel, 2014) and bias in interactions with fellow humans or even animals. Even perspective taking, that is, imagining the self as the other showed higher altruistic behavior (Myers et al., 2012). Empathic concern also depends on moral judgment (Gleichgerrcht & Young, 2013), which dictates nearly all social interactions. In an experiment to investigate perception of pain modulated by moral judgment (Cui et al., 2016), it was found that when the target was labeled moral/neutral the EEG signal amplitude was higher in N2 (localized in the ventral medial prefrontal cortex and rostral anterior cingulate cortex) for pain depicting pictures than for non-pain pictures while for immoral target this difference was insignificant. From the results, they infer that an immoral person in pain elicits decreased affective arousal Lesion studies In a pioneering study, Shamay-Tsoory et al., (2009) aimed to understand whether the emotional contagion and the cognitive processing aspects arise out of a single 27

29 empathy system or distinct ones. To explore this, they selected 32 patients with lesions in the ventromedial prefrontal or inferior frontal gyrus and a control group of healthy subjects. They first collected their empathy index using the IRI (Davis,1983) and then showed each participant 52 photographs of eyes reflecting 13 emotions happy, sad, afraid, confident etc., and the participants had to select the correct emotion as perceived. To test the ToM network, a false-belief task was conducted. From their analysis, it was proposed that patients with lesions in the ventromedial area showed deficit in ToM/Cognitive empathy while patients with lesion in the inferior frontal gyrus displayed impaired emotional empathy. With this study, they highlighted the critical role these areas play in cognitive/emotional empathy Clinical Studies of Empathy Psychopathy among other symptoms show a lack of empathy, that is, the inability to experience the emotional states of others whether the emotions are inferred or perceived from explicit cues. Decety,Chen, Herenski and Kiehl (2013a) and Decety, Skelly and Kiehl (2013b) used a stimuli depicting pain and looked at the brain activity of incarcerated psychopaths and found that relative rf5to the nonpsychopaths, they showed less activation in the ventromedial prefrontal and orbitofrontal cortices. But interestingly they found increased activation in the insula (Lamm, Decety & Singer, 2011), and the interpretation given is that this area is implicated in cognitive processing than affective processing (Decety, Chen et al., 2013a; Decety, Skelly et al., 2013b). That is, as psychopaths do not have normal limbic input to process emotions like pain, they could rely on cognitive processing areas to analyze empathy stimuli. The conclusion drawn from these and other studies (Decety & Moriguchi, 2007, Blair, 2001, Hare & Hart, 1993) on humans with this particular clinical condition suggests that their ToM circuit is functional while the affective or emotional empathy process is limited. 28

30 Autism spectrum disorder has been shown as deficit in cognitive empathy with an intact emotional empathy, demonstrated in comparison studies with psychopaths by behavioral analysis (Jones et al., 2010; Schwenck et al., 2012), by impaired emotion recognition (Fairchild et al., 2009), fmri responses recorded for emotional stimuli ( Decety, Skelly, Yoder & Kiehl, 2014) and false-belief paradigms (Baron-Cohen, Leslie & Frith,1985). But a few studies have also showed a decreased emotional empathy scores from questionnaires (Grove et al., 2014) and in arousal levels for emotional face stimuli (Mathersul, McDonal & Rushby, 2013) A third clinical condition focused on was schizophrenia, and it has been shown that compared to normal controls, people with this condition perform poorly in emotion perception, cognitive processing and theory-of-mind (meta-analysis: Savla et al., 2013). A recent study (Horan et al., 2016) using pain paradigm to evoke empathy response found that schizophrenic participants demonstrated similar neural activity as normal controls while observing others in pain, the differences were recorded when they imagine themselves vs. others experiencing pain. Studies have also shown that loss of empathy is an early symptom and hence a diagnostic for frontotemporal dementia (Piguet et al., 2011; Rascovsky et al., 2011). It has been found that patients with this condition display diminished response to other s feelings and personal warmth (Mendez, 2006). Baez et al., (2014) used 25 animated scenarios in addition to questionnaires to explore whether loss of empathy is deficit or due to decreased executive function in patients with frontotemporal dementia. They report that these patients had deficits in affective, cognitive and moral aspects of empathy, all which depend on executive function and theory-of-mind. While empathic concern is not a factor for executive function it was effected in this clinical condition. 29

31 1.3 Summary At this stage where efforts are still on to understand the complex construct called empathy, the question of how/why/when it works is still in its infancy. So, while attempts are being made to correlate human empathy index to empathy response actions, substantial amount of work is required to understand all the sub-processes of empathy using a wide-range of stimuli. The focus of all research efforts is to find evidence if the key concepts of empathy motor mimicry, emotional response, cognitive processing are in concert and connected to each other in a sequential/parallel and probabilistic manner. This would allow for a neural model which can be used as reference for understanding the nature of deficiencies in clinical conditions like schizophrenia, psychopathy, in criminals like rapists/perverts and in developmental conditions as in autism spectrum disorder. Hence, for this study and with respect to the immersive stimulus used, we define empathy as a complex combination of cognitive understanding (cognitive empathy), emotional response (emotional empathy) to other s(the actor in the narrative) state, being aware of the self-other distinction in the physical reference plane while allowing for transient transportation. The self-other merger by transportation happens when we experience other s emotion without realizing it that it is not ours (emotional contagion) like reflexive crying when watching movies. Our definition is based on the immersive experience accorded by audio-visual fictional narratives like movies or narratives of real-life naturalistic events. Hence a viewer/reader experiences the following: understand the feelings of the other, reciprocate/mimic same or (in)congruent feelings, experience immense emotions, sympathize, pass moral judgment, and suffer personal distress. Some of these have been considered as empathy while sympathy, personal distress and concern are congruent and mostly other-directed emotions, but it is difficult to differentiate the 30

32 responses accurately for any stimuli type. Hence for the stimuli type considered, the empathy related neural responses could be a permutation or combination of some or all the above for any event in addition to large inter-subject differences. 1.4 Some gaps in empathy research With controlled stimuli like images with pain infliction and emotional faces the initial foundation for a functional architecture of empathy mechanism has been demonstrated. But, the very definition of empathy as an abstract mechanism compared to more primitive processes like sympathy, personal distress or compassion, means paradigms with complex stimulus are to be tested to make the distinction between these and empathy. A further extension to look at (in)dependencies between sympathy/compassion and empathy is required to isolate empathy as a distinct response. Additionally, as most social interactions in real-life are based on contextual information derived from bottom-up process through sensory input, a naturalistic paradigm will complement the existing knowledge on the neural correlates of empathy. As in real-life, a naturalistic paradigm of a multi-modal fictional or real narrative can tease out the neural correlates engaged for experience sharing and mentalizing. The challenge for any empathy study is in understanding whether these two processes co-activate or sequentially engage and the dependency on the stimulus features. Hence data from naturalistic paradigms can move towards a when/how model, where it is theorized that observers or perceivers deploy integrated processes as per their understanding of the cues provided and the goals of the self and other (Zaki & Ochsner, 2012). From the data collected using naturalistic models, it is possible to examine the complex empathic sub-process thus leading to a more concise method of measuring empathy. 31

33 Hence, a major gap identified in empathy research is the sparse number of studies using ecologically valid naturalistic paradigms with multi-modal stimuli. Additionally, the opportunity to take a network-based approach to empathy neural correlates rather than a segregated involvement of brain areas like that explored in early studies using pain-paradigms (Morrision et al., 2004; Singer eta al., 2004), will extend the understanding of the functional and effective brain networks. At the abstract level, this approach might help us understand how the brain generates complex altering mental representations of other s state and presents a unique response based on subjective contextual understanding. By focusing on dynamic nature of functional connectively analysis which has been to-date applied only for resting-state condition, we would be filling the gap in research of large-scale dynamic interactions that occur in the brain which have been previously deduced from pre-determined responses to controlled events. And most importantly the study will demonstrate that the neural correlates for empathy suggests a broader definition than that arrived from previous studies. In the next chapter, a background on narrative empathy will be presented along with a motivation for the proposed experiments in this thesis. 32

34 Appendix Historical background of Empathy The first recorded report on defining empathy is found in treatises of philosophers attempting to explain human feelings towards works of art or culture. While authorship for coining empathy is debated, it is widely accepted that Einfühlung (translated to Empathy in the English language) originated from the German philosopher Johann Gottfried Herder, who thought that humans look for similarities in nature and ascribe human feelings to them. He also states that empathy is required to interpret texts, culture and history (Treatise on the Origin of Language (1772)). David Hume ( ) bases the transmission of emotion between two humans; the formation of moral response and (like Herder) covers aesthetic responses to define the concept of sympathy, which he describes as a principle of communication and a natural process. Taking off from this theory, Adam Smith in the Theory of Moral of Moral Sentiments (1759), introduces imaginative perspective-taking. The distinctive point put forth by Smith, is that one needs to know the context for the target s emotional state to sympathize. In the eighteenth century, Friedrich Theodor Vischer ( ) a German philosopher extended the concept of empathy by expressing the belief that art and nature manifest as emotional beings and can be experienced with empathy, suitably supported by the artistic imagination of the viewer. Theodore Lipps ( ), an important philosopher instrumental for an all-encompassing definition of Einfühlung, claimed that during the act of perception of an object the human (viewer: as the object in question is also work of art) attributes emotional states to the object (covered in detailed in his essays Empathy and Aesthetic Pleasure and Asthetik ( )). He further classifies empathy objects into psychological life of humans, the psyche of animals, nature and works of art/culture, and in each the perceiver is said to form a relationship possible due to human tendency to imitate, draw parallels between observed facial expressions and human response, 33

35 the ability to ascribe life and feeling to inanimate objects in nature and elicit the expressions that the author/artist chooses to convey respectively. Lipps also suggests that projection of emotional states onto another human or object depends on access to similar states he/she has been through, a theory that has been debated in later studies. The strongest refutation is the nonexistence of understanding, as the Einfühlung theory fuses the perceiver and object feelings. The English term Empathy was coined by Edward B Titchener and described as process of humanizing objects, of reading or feeling ourselves into them (Titchener, 1921, p. 417). The cognitive aspect to empathy was first argued by Kohler (1929), who opined that empathy was more of understanding than sharing of feelings. This view was also supported by Herbert Mead (1934) and Jean Piaget (1965; subjects: Children), as both emphasized the need to take on or imagine the role of the other to understand the other s perspective of the world. Post-world War II, many psychologists who studied human behavior in conflict, bought research focus back to empathy. One of the major contributors was Carl Rogers (born: 1957), whose theory was phenomenological philosophy, and says that our perception of the world rather than factual conditions determines our actions or empathy response. Interestingly he later (Rogers, 1975) classifies empathy as a process than a state, that is, the ability to ascertain the dynamic feelings of the other at every moment and seek assurance at frequent intervals for accurate depiction, and to set aside one s views and values to take on that of the other s world without prejudice. A relevant definition of empathy that is applicable for empathic responses generated from narratives. The study of person perception (Bruner & Tagiuri, 1954) led to defining empathy as a measure of predictive accuracy in deciphering other s thoughts and feeling (Truax, 1967). The perspective from the 1980 s aligns to two components of empathy: the experience of feelings what the other is feeling (Batson, 1987) 34

36 referring to mimicry or simulation of motor actions and second, the cognitive processing of the feelings (Hoffman, 2000). The discussion shifts to differentiating and defining the self-other response and controversy arises from the concept of affective responses, that is, can empathy be an affective response to other s/target s situation than self s?. Affective response/empathy when defined narrowly, is the vicarious sharing of an affect but a more encompassing definition says that the empathizer & the person (target for the empathy) need to be in the same state (Coplan 2011; de Vignemont and Singer 2006; Jacob 2011), while for Hoffman (2000) it is an emotional response requiring the involvement of psychological processes that makes a person have feelings that are more congruent with another's situation than with his own situation. This definition conveys that the two emotions (target s and subject s) are not necessarily the same. Hence, in contrast to emotional contagion, empathy is based on the ability to differentiate the self and other. An important factor is introduction of perspective taking, that is, the observer s emotional response based on his/her own perspective of the world. Barnet et al., (1987) state that empathy is emotional response congruent but analogous to the one experienced by the other. In agreement with this proposition, Batson et al., (1991), who studied motives for people s prosocial /altruistic behavior, suggests that this is applicable only to certain emotions like compassionate concern, liking and similar other feelings. Hence, at the root of empathy, emotional contagion or sympathy or personal distress is emotion. To isolate emotion, the term perspective-taking was posited (Shantz, 1975), with three types a) ability to understand what other s perceive usually based on visual cues, b) ability to understand what other feels affective role taking and c) ability to comprehend what another is thinking cognitive role-taking. A distinction often cited is, the experience of feeling the affective state of the other just by association is emotional contagion, and this does not necessarily require perspective taking. Hoffmann (2000), whose prime focus was on studying the relationship between 35

37 altruism and empathy, suggested that affective response is tilted to other s situation than self s. Empathy in Indian philosophy In eastern religions and philosophies like Buddhism, the self-other empathic relationship as a virtue that requires to be inculcated for social and personal wellbeing forms the basis of harmony in society. Compassion or Daya or Karuna is described as that which makes the heart of the good move at the pain of others. It crushes and destroys the pain of others; thus, it is called compassion. It is called compassion because it shelters and embraces the distressed. - The Buddha. The teachings emphasize the cultivation of compassion to surpass social barriers and elevate misery in fellow human beings. Karuna and Samvedhana (Sanskrit language: sympathy and taking on other's distress respectively) is the closest to the definition of empathy is the response elicited when we understand (Samvedhana) and share other s feelings. The distinction between empathy and the philosophical term sympathy or the generic response compassion (merged in eastern philosophies), is that the latter responses do not require affective isomorphism or affinity due to real or perceived similarity. Sympathy and compassion are both characterized by prosocial motivation (the impulse to help others in distress) and though empathy is also associated with prosocial behavior, there can be affective empathy without prosocial response. Interesting is the role of karma theory in Hinduisms which states that each human is what he/she is supposed to be destined by past-deeds, so in this context, compassion extends to concern for the pain/suffering which leads to altruism/prosocial motivation without actually taking on to self the state of the other or even understanding the other s feelings. This cultural differentiation in the Indian context requires a thorough study using different paradigms with the very first step being a definition for empathy in the backdrop of the philosophy of Karma'. 36

38 Chapter 2 Narrative Empathy 2.0 Introduction Defining a Narrative Though story and narrative have been used interchangeably, a narrative is a structure of events while story is sequence of events. In Indian language the word Katha (Hindi and Sanskrit language) which translates to story is used synonymously for narrative too. A narrative format of presenting an event is not restricted to fictional works but to news article reporting real events to make it more engaging, though the reader/viewer takes cognizance of the factual information in the later. Fictional narratives are about characters derived from the imagination of the writer, with or without resemblance to real-world people and the reader s/viewer s interaction with the world created by the writer. Fictional narratives usually have an ending, while nonfiction narratives like news articles including economic, historical or political events do not necessarily have a closure. Narratives play a role in every culture as an important form of communication of values, religious messages or purely as entertainment. Cultural narratives like folklore or religious texts influence reasoning as explicit and implicit messages (moral values, conflict reasoning, social interactions or philosophy of life) educed by the reader/listener seep into long term memories especially when exposed early in life. By far the largest poetic narrative or katha in Indian culture has been the Mahabharata, which is a compilation of many stories with a common thread. This longest poem in the world is an epic narrative that has influenced the Hindu way of social and political life. Each katha in the narrative sets a time-space reference frame introducing characters, conditions and the causal relations between them. The complexity of powerful feelings and the subjectivisms as evoked by this epic story is an interesting research study. 37

39 Interestingly, the idea of emotions from literature comes from Indian poets, Anandavardhana (8 th century) who wrote about dhvani or the concept of being in resonance or same wavelength with the poet to fully understand it. Another great poet and scholar of the 9 th century who wrote on rasa was Abhinavagupta. Both distinguished the everyday emotions called bhavas and literary emotions called rasas, the latter said to be experienced with memories of past lives while it is not possible to decipher bhavas as it could be clouded by ego. Detailed insights into the works of these two poets are provided by Ingalls et al., (1990). In short, narratives are persuasive immersive experiences enabling role-taking, emotional transportation into the plot, sympathizing with the characters, bringing in external knowledge to analyze the events and importantly draw out emotions. Summarizing the theories on narratives and emotion published in the 21 st century, Mar et al., (2011) suggests that selection of reading literature depends on the current emotional state of the reader and hence could also influence the emotional experience during and post reading. Five possible types of emotions reader experiences have been proposed (Oatley, 1994;2009; Mar et al., 2011), which are 1) emotions of sympathy, 2) emotions of identification, 3) emotions of empathy, 4) relived emotions (from past experiences) and 5) remembered emotions. Other studies have looked at emotion and action in narrative imagery (Sabatinelli et al., 2006), reading (Yarkoni et al., 2008) and stories in children (Brink et al., 2011). Bal and Veltkamp (2013), found the long-term effects on empathy from fiction reading on subjects with higher transportation (immersion ability into another world) experience. People who read fiction had higher scores in an empathy task compared to expressions of loneliness and negativity in non-fiction readers (Mar et al.,2009). 38

40 Keen (2007; 2013) defines narrative empathy as a spontaneous vicarious sharing of feeling and perspective-taking evoked by external sensory inputs or by purely imagining another s situation. Green and Brock (2000;2004a,b) proposed the Transportation theory aimed for entertainment-education, in which the following processes are possible - an immersion that makes the readers/viewers forget the real-world concerns hence reduces their ability to formulate counter-arguments and second, the immersion simulates direct experiences. The more complex empathy response to narrative requires that the viewer follows the beliefs and thoughts of the characters and allows the experience to influence emotions comparable to that depicted by character or as appropriate to the context. 2.1 Empathy/emotion to Cinematic Narratives Like the emotional experiences evoked from literature reading as described in the preceding section, cinematic narrative response is triggered by the bottom-up process from the multi-modal sensory inputs (visual, auditory) and the top-down process of individual viewer s perceptions. While reading requires imagination to construct mental models of the described situations including that of facial /body expressions as described by the writer, in visual narratives explicit presentation of the events and characters translate to lower load on the imagination process. That is, in cinematic narrative the viewer is expected to just follows the beliefs, thoughts and actions of the characters as created by the movie creator resulting in a bonding between the viewer and the fictional character. The short duration relationship the viewer builds with the character(s) can be a state where he/she understands the feelings expressed by the actor and can extend to reciprocating the feelings, both evaluations considered to be basis for empathy (Decety and Jackson, 2004). Though the movie is a time-bound task, watching a movie can be an extremely emotional experience with the potential to evoke long-lasting influence. The movie 39

41 plot takes one through a gamut of emotions from happiness, anger and anxiousness not just from the events depicted but also from correlation to real-life experiences. The influence of movies on viewers is acknowledged by movieindustry and hence scripts are designed carefully with high empathy quotient. The emotional appeal is created by context, depiction and narrative, which is the building block for movies with the emotional contagion drawn from the sociocultural environment in which the story is set. For example, the actors in an Italian movie might show more overt facial expressions than actors in a Japanese movie. Similarly, Indian movies are rich in exploiting the bhava or sentiment factor with an intention to create rasa or emotion in the viewer (rasa is usually identified as the emotional response to the depicted or conveyed bhava). Further rasa does not exactly identify as empathy but is only said to be a one of the many types of empathic emotion (Hogan, 2008). The segmentation arises as viewer can distinguish those actions on the screen or stage does not require him/her to respond (self-other distinction) or have real-life consequences. Hence the term empathy in the cognitive film theory refers to a range of phenomena - as a conscious endeavor for perspective-taking or in other words putting oneself in another s shoes to affective mimicry and emotional contagion where emotions are captured and mimicked involuntary (Smith, 2011). Smith (2011) elaborates that the process of imagining how the other thinks or feels does mean an empathic resonance and calls this other-focused personal imagining, which he says allows the viewer/reader to comprehend the emotional frame of mind of the other. It has been debated whether emotional contagion which is selfdirected ( experience the same feelings ) and typically considered non-cognitive can be empathy, but considering the broader definition of empathy as covered in Chapter 1, the affective process is critical to empathy in the holistic approach to this complex construct when applied to movie narratives given the said influence this medium has on social behavior. 40

42 Considering the complexity of relation with a fictional event or actor, the debate is on whether movies have the power to influence human behavior or whether movies just present an exaggerated version of human behavior and hence this medium is purely of entertainment value. This leads to a premise that viewer s empathy responses are restricted to events shown on a screen and hence cannot be equated to real life social interactions and perception. While this could turn out to be true, it is not a stretch to suggest that movies have considerable influence on social perceptions. For example, a movie story inspired from a historical event, can by way of clever screen play and narrative skill teach us to empathize with the events that happened long ago while simultaneously effecting our outlook on current situations. A point of contention is reports of viewers losing selfawareness and fusing their egos with that of the character (D Aloia, 2015; Raz and Hendler, 2014) and whether this could also be empathy. A parallel proposal (Hanich, 2010) from the phenomenological ( a philosophical method of describing and reflecting on the experience of phenomena as they present themselves to consciousness ), theory of cinematic empathy includes sensation (replication involuntary of similar sensation as that experienced by the character on the screen), motor (muscular changes as per the action on the screen) and affective mimicry. Keen (2013) extends the experience of cinematic narratives by suggesting that empathy is not only for characters and their actions but for inanimate objects in the film. Supporting this proposition, the phenomenological film theory adds that by manipulating the aesthetic style and the method of narration, empathy can be evoked not just for the characters but also non-living objects. Thus, from the definitions and the probable responses, it can be safely inferred that cinematic empathy can trigger both cognitive and affective empathy response mechanisms, even if the viewer is aware of the fictional nature of the events and characters. Hence, cinematic narratives are interesting stimuli to study, in terms of its 41

43 multimodality and the wide perceptive differences in viewers. Psychology or behavioral experimental research has exposed the complexity of such data and cognitive neuroscience is being applied to decipher the basis for the complexity. 2.2 Neuroscience of narrative empathy From the definition for narrative and cinematic empathy, it is realized that empathy response is complex top-down and bottom-up processes including imagination, perspective taking (including past experiences), sub process of empathy (sympathy, personal distress, emotional contagion) and motor mimicry to name a few. Hence a cognitive empathy process that includes perspective-taking/ ToM, an emotional empathy process comprising of the sub-process of empathy like sympathy, emotional concern, personal distress and a motor empathy process from mimicry/ mirroring of the affective state of the target should be examined as correlated responses. Considering this complexity, the neuroscience of narrative empathy is still in its nascent stages with more questions than answers. To form a more comprehensive picture of the brain responses to individual processes or the probable brain networks, findings from a few relevant studies are presented. Initial studies using dynamic complex stimulus like movies have helped identify brain networks that process individual features embedded in a scene or of an entire event. Hasson, et al. (2004) report significant inter-subject correlation (ISC) in brain activity when viewing clips from the movie Good, Bad & Ugly and found correlations in the temporal and fusiform area. Bartels, et al, (2004a,b) had participants view a 22 minute clip from a James Bond movie Tomorrow Never Dies and identified functionally specialized areas that process faces, language and color. Han et al. (2005) studied differences in brain activation to movie clips with cartoon human-like/non-human-like characters and real-actor movie clips. They report differences in motion perception for real-actor and human-like cartoon characters specific to the medial prefrontal cortex (mpfc) and cerebellum. A 42

44 study by Mar et al. (2007) comparing the ability to perceive intentions from movements as performed by a similar cartoon versus real human characters report higher responses in the areas associated with mentalizing the mpfc, the superior temporal sulcus and the temporo-parietal junction to be greater for the real actors compared to the computer-generated actor. Other studies which have used text or static cartoon narratives (Sabatinelli et al., 2006; Chow et al., 2015; Brink et al., 2011; Altman et al., 2012; Schnell et al., 2010) report activations in the emotional empathy networks with anterior insula and cognitive empathy networks of mpfc. To understand the influence of past experience and knowledge about the world on understanding of a story, Chow et al, (2015), used 18 stories (written by the authors of the paper), three paragraphs long each confirming to the typical story format. Manipulations to include description of scenes to set the perception condition and bodily actions for action condition, emotional charged events (emotion condition) and factual description (control condition) were introduced in the paragraphs. Functional connectivity analysis was conducted to understand the interaction between the lower and higher-level visual and motor areas while comprehending a story rich with perceptual and motor details. The left anterior parietal area and left dorsal premotor area was significantly modulated by the participants experience with the narrated situation. The researchers of the study conclude that interactions between higher-lower level visual and motor processing systems are strongly modulated by personal experience and this in turn influences narrative comprehension. Narrative imagery, that is, the ability to imagine themselves in situations is fundamental to any narrative engagement. To investigate this, Sabatinelli et al., 2006, provided subjects brief narrative scripts over headphones, and asked them to imagine themselves engaged in the described events. The scripts consisted of 12 exemplars of pleasant scene contents, 6 of neutral scene contents, and 12 of 43

45 unpleasant scene contents. The audio narratives of 12 seconds each was followed by 12 second duration where the subjects were instructed to imagine. The brain areas of interest for each condition were identified and comparative analysis conducted. During the listening phase, the auditory cortex, retrosplenium and the left medial frontal gyrus were significant. For the imagery phase the supplementary motor area, left inferior frontal gyrus and right lateral cerebellum were identified. They conclude that scripts with higher emotional contagion (negative or positive) show enhanced signal change relative to neutral scripts. Using empathy rated cartoons and verbally presented stories, fmri data was collected to examine developmental changes of preschool and school children by Brink et al (2011). They found that affective and cognitive empathy is associated with medial and bilateral orbitofrontal cortex activation. Older children showed higher affective empathy by increased activation in the medial orbito-frontal cortex, left inferior frontal gyrus and left dorso-lateral prefrontal cortex. The brain activations in these areas were also found to be greater for the non-verbal cartoon stimuli from which they conclude that it has greater empathy response. Using a set of 80 short emotional and 40 neutral text narratives, Altman et al., (2012) investigated the change from cognitive to affective process in reading of the short emotional narratives. They were specifically interested in identifying the neural substrates active when participants liked negatively valanced narratives. Using a block design, as the stimuli duration was very short, they compared the brain activation for neutral versus unpleasant stories and the data revealed a stronger engagement of affective ToM-related brain areas with increasingly negative story valence. Unpleasant stories engaged the medial prefrontal cortex(mpfc), which the authors suggest might reflect the moral exploration of the story content. As mpfc becomes more engaged for the negatively valanced 44

46 stories, co-activation in brain areas related to affective ToM and empathy also increased. Mentalizing and/or ToM is a critical mechanism involves the process of cognitive inferencing that one engages to decipher another person s affective state (Frith and Frith, 2006). In an interesting experiment using 32 false-belief cartoons of 3 pictures, Schnell et al. (2010) conducted an fmri study to explore the neural foundation of cognitive empathy and contrast with cognitive inference on nonaffective (that is, no direct visual cues depicting explicitly the affective state) visuospatial representation of another person. The participants were asked to judge the affective or visuospatial changes with respect to their own perspective or that of the protagonists. Applying the General Linear Method (GLM), the contrast of the two perspectives were estimated. The evidence presented is the existence of a neural correlate of cognitive empathy disassociated from mentalizing of visuospatial content by the higher and simultaneous activation in the anterior mentalizing network comprising of the dorsomedial prefrontal cortex, anterior superior temporal sulcus, temporal pole and ventromedial prefrontal cortex and the limbic regions of amygdala and hippocampus. The important inference made by them is that cognitive empathy also involves references to internal affective states. Further their study showed that the higher mentalizing network for 1 st person judgments about affective states compared to visuospatial content indicates that this system activates for social perception without taking the 3 rd person perspective. Pehrs et al. (2015), used 60 empathy evoking close-up shots of actors depicting sad or neutral expressions supported by information on the state via text and music. Their special focus was to investigate the role of temporal pole as semantic hub of complex social cues and from the results state that this area acts as an integrator of multi-sensory information to facilitate meaningful interpretations. In a recently 45

47 published study, Nguyen et al., (2016) investigated the brain areas relevant for interoception by having participants listen to an emotionally salient audio narrative. The fmri data and the heart rate measured were found to be synchronized across the participants. The connectivity analysis revealed that anterior insula active for emotionally salient moments served as integration hub to the posterior insula where the interoceptive states were represented. The neural base of amusement and sadness response using nine 2-minute movie clips or TV serials clips set in a block design was experimented by Goldin et al.(2005). Using contrast analysis of fmri data from the sad/amusing presentations followed by subject-specific regression analysis with continuous rating for the clips, they found that for sad films activations in medial prefrontal cortex, inferior frontal gyrus, superior temporal gyrus, precuneus, lingual gyrus, amygdala, and thalamus were observed. Whereas for amusing films, the subject specific regression analysis demonstrated significant activations in medial-inferior frontal gyrus, dorso-lateral prefrontal gyrus, posterior cingulate, temporal lobes, hippocampus, thalamus and caudate. Studies which have specifically examined empathy using naturalistic stimuli like movies (Nummenmaa et al., 2012) required the viewer to make inferences. Two research groups ( Nummenmaa et al., 2012 and Raz et al., 2012) focused on empathy specifically, Nummenmaa et al., (2012) conducted experiments using very short movie clips with emotional scenes with minimal narrative and do not explore explicitly the role of narrative or context on empathy response. Raz et al., (2012) used video excerpts of longer duration (~10 minutes) from commercial movies and explored the dynamics between the affective empathy and the top-down cognitive/tom empathy. To look at the dynamic changes in the emotion specific brain areas as a function of the events in two long emotional movie clips (10 minutes from Sophie s Choice and 8:27minute sequence from Stepmon, both 46

48 commercial Hollywood movies), Raz et al., (2012; 2014) analyzed EEG and fmri data collected independently. Marking regions of interest, network based functional connectivity analysis was committed and regressed with the continuous rating collected from the participants. The data analysed from both the techniques revealed that the dynamics of the limbic network was associated with rated sadness intensity level. For the movies, significantly higher correlation between limbicmedial prefrontal cortex in the connectivity indicates that sadness involves regulated processes of mentalization and introspection. The correlation of the rating data with limbic lobe showed high cohesion for Stepmom, from which they suggest that different dynamics of emotional regulation could have been applied by the viewers. The concept of direct correlation between the behavioral data (rating) to neural activations is very challenging and only limited inferences can be drawn from the analysis. Nummenmaa et al., (2012) explored the brain networks when explicitly sharing other s emotional state to facilitate better understanding of intentions and actions. They used 13 segments of seconds clips from Hollywood films (When Harry Met Sally and The Godfather), depicting the actors experiencing strong positive or negative emotions or when in neutral state. Major findings were, negative valence was associated with increased Inter-subject correlation in the emotion-processing network comprising the thalamus, ventral striatum, insula and also the defaultmode network areas of precuneus, temporo-parietal junction, medial prefrontal cortex, posterior superior temporal sulcus. Seed-voxel correlation analysis confirmed that these sets of regions constitute dissociable functional networks. To summarize, the neural correlates for empathy from the various studies and the meta-analysis can be classified as follows: Emotional or affective empathy (incl motor empathy) : anterior/posterior insula, bilateral dorso-medial prefrontal cortex, supplementary motor area (SMA), 47

49 premotor areas (the mirror-neuron network noted by Rizzolatti et al., 2001, Gazzola et al., 2007), rostral anterior cingulated cortex (racc), anterior midcingulate cortex, posterior cingulated cortex,, anterior insula (AI), inferior frontal gyrus, midbrain, and temporo-parietal junction, as well as the left anterior thalamus further, the middle temporal gyrus, posterior superior temporal sulcus, posterior thalamus, hippocampus, and pallidum on the right. (meta-analysis: Fan et al. 2011, Bzdok et al., 2013). Cognitive/ ToM: Mid-cingulate cortex, bilateral ventro-medial prefrontal cortex, dorso-medial prefrontal cortex, precuneus, temporo-parietal junction, middle temporal gyrus, posterior superior temporal sulcus, inferior frontal gyrus, as well as the right middle temporal/v5.(meta-analysis: Schurz et al., 2014, Fan et al., 2011, Bzdok et al.,2013 ). 2.3 Summary Empathy related studies using pain as stimulus (reviewed in Chapter 1) and from studies using texts, cartoons, short video and one study with longer movie clips suggest the existence of three distinct empathy networks (cognitive and emotional), but overlapping activations as a function of the stimuli and task. The networks are: a) cognitive empathy: which includes processes like mentalizing or the more abstract theory of mind, with sub-processes like perspective taking, appraisal and forming mental models from past experience, b) emotional or affective empathy: where a number of sub process like sympathy, emotional contagion, concern, personal distress, compassion are dissociable from the emotional resonance of the other s affective state and c) motor empathy is the motor mimicry of the actions or expressions as perceived from the actions of the other. At the neurophysiological level, the various studies have identified distinct networks and areas for each empathy mode, while also suggesting that the stimuli/context can trigger more than one of the networks simultaneously. 48

50 2.4 Motivation for the Proposed Experiments Though cinema narratives are perceived to be unreal by viewers, empathy response using cinema narratives could help understand other-oriented processes such as moral judgment and position taking leading to altruistic behavior in reallife, as the medium of cinema can project different scenarios and human response in each can be analyzed. In other words, understanding our brain in the cinema helps in figuring how we empathize with other s emotions, actions and psychological states by observation. Notwithstanding the paradox of fiction, ( Colin Radford,1975) cinema can evoke cognitive perspective taking based on the visual/auditory sensory input and adequately supported by imagination while affective state is induced by the theory of transportation. Hence, the theoretical and the neurological studies of empathy using movies as stimuli allows for formulating and testing dynamic models to examine social interactions in diverse conditions. In the experimental work reported in this thesis, we aim to answer the fundamental but complex questions: a) How does empathy neural networks evolve as a function of the narrative? b) Do empathy studies have higher ecological validity if the stimulus allows the viewer/reader to get all the contextual information required to understand the reason for the affective state of the other? That is, can a stimulus that explicitly presents the reason for the affective state of the other provide a deeper insight into the complex neural correlates, compared to activations for stimuli with little/no context. There has been no study conducted to date to address the second question. The findings from such a study will have large implications in the way empathy paradigms have been designed and on studies which have looked at empathy deficiency or efficacy of narratives/stories based intervention for empathy development. 49

51 2.5 The challenges of movie as stimuli to study empathy response As can be inferred from the analysis of empathy research and the proposed models of the various schools of thought, there are many definitions for empathy and it is a challenge to ascertain which one defines the complex feelings one experiences in response to the affective state of the other. An account of empathy as defined by studies looking at pain infliction, moral judgment or altruism is limited when applied to the myriad combinations of experiences when one is viewing a movie or reading a story. For example, in a movie the sequence of events that lead to a young boy being dropped by parents in a boarding school, could evoke a sense of empathy and sympathy in a young viewer and anger plus empathy at the parents (even on own parents by experience of similar state as the protagonist). Adults and especially parents might feel negative emotion momentarily but analyze the cause and consequences of the action by the screen parents. Grandparents would feel empathic anger at the parents and society at large for putting a young child through a possible trauma. A complex combination of responses arises as they are influenced by the value system held by the self and as perceived from the state of the other (even if characters are fictional). Hence narrative empathy is much more complex as it also includes the paradox of fiction (Colin Radford, a reply to critics article in 1975), as emotional response to fictional events is irrational though familiar. The almost irrational state for fictional experiences, was further explained by Radford with two considerations: if the viewer/reader is convinced of the unreality, then the emotions that one frequently experience is not possible and second, the fact that we do not actually act to help the actor/character in the film/book, implies our awareness of the fictional status even if moved by the state of the actor. The third premise is that fictional characters can move us emotionally with the same intensity as living beings in real-life. The paradox proposed by Radford, of the irrational response has been debated by other film theorist and philosophers, though the reason for both the human response and the source for such a response still alludes researchers. A few theories proposed to explain this 50

52 paradox are: a) pretend theory: propounded by Kendall Walton (1978) explains the emotions as quasi-emotions, evoked due to the make-believedly and b) thought theory: where rather than believing the fictionality all one needs to do is "mentally represent" (Peter Lamarque), "entertain in thought" (Noel Carroll), or "imaginatively propose" (Murray Smith) it to ourselves and c) Illusion theory where existence beliefs are generated in the course of engagement with works of fiction. The theories presented are nowhere close to explaining the effect of fiction. The work presented in this thesis is an attempt to address some of the questions by investigating the empathy response to fictional content. Without getting into the debate of why a viewer/reader is able to feel a gamut of emotions even being aware of the fictionality of the presented stimuli, I list the following possible conditions of engagements, which might weigh the intensity but assures an empathy response: I don t relate to the actors but can to the context. I relate to the actor but not to the context. I relate both to the actors and the context I relate to neither but willing to form a short-term relation. In light of the above, cognitive empathy is the ability of the reader/viewer to understand the evolving state of the actor(s), place it in perspective with the events, make the self-other distinction constantly, place self in the other s role even if for short periods, engage in moral judgment or a combination which in game design language is switching between a god s perspective and a first-person perspective, all of which is possible if one understands the other s feeling and internalizes it. Emotional empathy is when the viewer/reader takes on the emotional states of the other. 51

53 Hence, an operational definition for narrative empathy considered for this study is: empathy for a fictional cinema narrative is the sharing or feeling or perspective-taking induced by viewing and comprehending the contextual factors for state of the other by the power of transportation afforded by the cinematic narrative, while being aware of the self-other differentiation. By this broad definition, an effort to address the individual variations in the empathic immersiveness that one wishes to allow/inhibit as a function of the cinematographic differences in conveyance of the message is attempted. 2.6 The hypothesis of this study A narrative is a method of conveying the reason for the state of the other by threaded events presented in a temporal sequence either in text, verbal or visual format and is an integral part of social communication. Readers or viewers experience a narrative by dissecting the events presented and taking the perspective of the agent(s). The short relationship the viewer builds with the other can be a state where he/she just understands the feelings expressed by the other by comprehending the context (Gallagher, 2012) and can also extend to actually reciprocating the feelings, both evaluations considered to be basis for empathy (Decety and Jackson, 2004). Fleeting emotional expressions with no specific contextual information can rarely evoke empathy unless one has the experience or the imagination to make knowledgeable inferences. A photograph showing facial expressions depicting pain or that of someone being inflicted pain could evoke disgust or personal distress which is self-oriented than other-oriented, the latter being a fundamental element for empathy. Empathy requires the imagination or information for perspectivetaking. That is, in addition to cognitive inference of what the effective state of the other is, empathy response is aided by knowledge of 'why' the other is in a particular state and this defines the understanding required for sharing of the other's feelings. 52

54 In our study, the goal was to identify the empathy networks using a diverse set of movies, to look at the fidelity of the identified networks as a function of the narrative, viewer s emotional and cognitive processes to the events. Our premise for using long narratives is: the context ( informed condition) for the affective state of the other can stimulate basic subprocess like empathic concern (otheroriented response), personal distress (self-oriented) and thus empathy cognitive, emotional, motor empathy as compared to responses for emotional images with little or no context to inform the reason for the state. From an image or a very short video clip, the viewer could make inferences by imagination or by knowledge constructs, but these are top-down and very subjective. Hence the goal of the second fmri experiment was to identify and compare specifically empathy networks for 'informed' appraisal to that of inferred state of the other by comparing fmri data collected from short movie clips to those from full-length movie. There has been no study which has examined neural correlates of empathy responses to visual narratives like fictional movies. 2.7 Data-analysis, the challenge from naturalistic stimuli Most fmri experimental designs use block-models, that is, a short (1s 30 s) stimulus is presented at some frequency interspersed with no-stimuli baseline condition and comparison of activation during stimulus and baseline time points are undertaken. The short stimuli presentation model is considered robust as noise from subject movement in the scanner and/or magnetic field drift is minimal. To study real-life responses, an ecologically valid stimulus like naturalistic scenes presented for longer duration is required. Advances in fmri equipment (field strengths of 3-7 Tesla), radio-frequency pulse sequence techniques and analysis methods have made it possible to project multimodal stimuli for longer duration with minimal signal degradation and acquisition of high-resolution functional images. This has opened experimentation with naturalistic stimuli to examine 53

55 functional connectivity networks as simultaneous activations of more brain areas are evident than for conventional block stimulus settings (Bartels and Zeki, 2004). The fmri data from a free-viewing naturalistic paradigm brings in analysis and interpretation challenges. In the Chapter 3, the basics of magnetic resonance imaging technique and the interpretation of the BOLD change signals will be covered in brief. The major focus relevant to the paradigm selected for the study will be on analyzing the various data analysis methods. 54

56 Chapter 3 Functional Magnetic Resonance Imaging and Data analysis 3.0 Introduction - Functional Magnetic Resonance Imaging (fmri) For understanding how the brain functions, an understanding of the physiological and functional architecture is required. The former provides details at the neuron and cell level while the latter is critical to understand the connections between the neuron firings and the corresponding networks that are formed. In vivo imaging techniques like magnetic resonance imaging (MRI) and Positron Emission Tomography (PET) are the most common techniques used in the field of neurosciences to explore the connections between stimulus parameters, behavior and neural activations. In this section of the chapter, a brief introduction to the functional MRI (fmri) process is presented, while the Appendix covers salient parts of the physics of MRI and the imaging pulse sequences. Imaging of the brain is based on the connection between the cerebral blood flow, the energy demand during task processes and the neural activity. Thus, genesis of fmri is from the discovery that magnetic properties of hemoglobin are different for oxygenated and deoxygenated states (Pauling & Coryell, 1936; Ogawa & Lee 1990). The underlying principle is: oxygenated hemoglobin (iron, Fe, containing protein) of the blood is diamagnetic while deoxygenated hemoglobin is paramagnetic and introduces an inhomogeneity into the nearby magnetic field while the magnetic field effect is minimal in the deoxygenated state. The unpaired electrons make it paramagnetic thus experiencing higher magnetic field. This translates to the mechanism by which the magnetic field in and around the blood vessels can be altered by varying the oxygenation levels. The increase in level of deoxyhaemoglobin is expected to decrease the signal, but due to the increase in cerebral blood flow oxyhaemoglobin increases and leads to signal enhancement. 55

57 The deoxyhaemoglobin manifests as susceptibility difference between the blood vessel and surrounding tissue and this leads to dephasing of the proton signal (Thulborn et al., 1982) and reduction in the T2 * (spin-spin relaxation time) value. The discovery lead to blood oxygenation level (BOLD) changes being correlated with neural activity (Bandettini et al.,1992; Ogawa et al., 1992; Friston et al., 1994). The neural firing due to neural activity from a task is shown to increase oxygen consumption in the specific location (Gjedde et al., 2002; Fox & Raichle, 1986). Hence the BOLD response is an indirect link to neural activity (Ogawa et al., 1990a, 1990b) or oxygen demand by the neurons (Buxton and Frank, 1997). By measuring electrical activity and BOLD signal simultaneously Logothetis et al. (2001) experiments on monkeys showed a strong correlation of the BOLD signal with local field potentials of synaptic activity. Logothetis et al. (2001; 2003) have thus presented evidence that the BOLD response is local field potential than actual spiking of the neurons. The study reveals that BOLD signal is reliable though an indirect measure of the neural activation in response to a stimulus and has been used in many studies. A model is required to connect the BOLD response and the neural activation, and a linear time-variant system model is the most commonly applied method. Modeled as a black box, where the input is the neural activation and the output is the BOLD response. There is a delay from the onset of the neural activity to the BOLD signal changes and is termed hemodynamic response to the stimulus, with a dip before (Menon et al., 1995) and after the increase in signal amplitude. The oxygen overcompensation attributed to the second dip in the hemodynamic response is attributed to synaptic activity from glucose metabolism. The BOLD response to a stimulus is slower than the neural activity, and shows a peak 5-6 seconds after the stimulus onset and goes to zero after seconds (Friston et al., 1998a). The BOLD impulse response or the hemodynamic response function (hrf) as it is called, to a 56

58 single stimulus event can be modeled as sum of two gamma functions (Friston et al., 1998a). The signal measured in this technique is a function of the proton density, the relaxation of the protons due to energy loss (T1), termed spin-lattice relaxation which is a measure of the time it takes for the tilted protons along the RF field (physics of NMR in the appendix) to return to the constant magnetic field axis, and relaxation due to phase incoherence (T2). The spin-spin relaxation time (T2) is the time it takes the protons to come out of the phase in the alternating field plane. Most experiments have stimuli presentation designed on the hrf and the signal differences in a region or inter-regional differences are claimed as evidence of the role of the particular area to the presented task. The variability in the hemodynamic response is a function of the region in the brain from which it originates and varies across subjects (Handwerker, 2004; Aguirre, 1998). This simplistic model is applicable if the events presented in the stimulus are well-separated (Vazquez and Noll, 1998). For closely spaced events in the stimulus, the time-course shape of the BOLD response retains the shape but with lower amplitude as would be the case for a continuous stimulus design. Pedregosa et al. (2015) in a review has shown that accurate hrf s can be statistically estimated from data for block and event related designs, for continuous stimulus the convolution is not so direct. The extraction of the hrf from fmri data is by selective averaging with a long interstimulus interval based on the assumption of non-overlapping responses (Bandettini & Cox, 2000). Trials with overlapping responses are averaged and shown to introduce errors (Boynton et al., 1996). The General Linear Model (description in the sections below) expands the hrf into a set of basis functions (Friston et al., 1998) for extraction of the contrast between the task-related signal and the non-task period signal. 57

59 The process of collecting and creating an image from the BOLD signals is complicated (A brief overview presented in the Appendix of this chapter), and is accomplished by: a) subjecting an object (brain) to a strong magnetic field, b) apply an external alternating radio-frequency field to excite the protons in the brain, c) receive the signals emitted by the relaxing protons after the alternating RF field is switched off and d) using inverse fast Fourier transform to construct the image from the collected signals. At the imaging level, a balance is required between the spatial precision required, the BOLD sensitivity and the high signal-to-noise ratio. Signal strength variations in the brain image is from the magnetic susceptibility differences at the boundaries between tissue-air and tissue-bone resulting in small magnetic gradients and this causes image distortion by adding to the external magnetic field gradients. This is particularly an issue in the echo planar imaging (EPI) technique (appendix) where gradients are switched and add to the localized gradients. While it is not possible to isolate the localized effects, by measuring field distortions from larger scale magnetic field distortions that affect the gradient fields an estimate of the shifts in the k-space (Appendix) values can be derived. Thus, the outcome of an MR imaging experiment is a function of: a) the scanning sequence used - because of fast acquisition rate the EPI method is used in most research studies (Appendix), b) design of the stimulus block, event or free-running. c) task-related data analysis mostly dependent on the condition being tested and the stimulus paradigm used. 3.1 Experimental Designs In functional magnetic resonance imaging (fmri) of the brain, images are acquired while cognitive or motor actions are performed (Menon et al.,1992; Kwong et al., 1992). The differences in the neural activity during the task can be detected but the challenge is in differentiating the task related signals to that from physiological artifacts like breathing or pulse rate or due to movement of the subject inside the scanner and requires diligence in designs. A common design is a block form, where 58

60 stimulus text/visual/short video is shown or a task like motor action is performed for a short-duration followed by a rest period of similar or varying duration. For example, finger tapping experiments to investigate the activity in the motor cortex, requires the participant to tap fingers for a few seconds (20-30 seconds) and then repeat the same at regular intervals. The neural activity for the finger movement duration to the rest is compared to extract the signals specific to the motor cortex area. Block designs are optimal to determine which voxels activate for the task being tested. But, as the signal is a summation of the hrf over time, the time course of the response cannot be determined accurately. Another design is event-related, where the stimulus is shown very briefly and not at any fixed intervals, for example, colors or light patches for visual cortex or face recognition tasks. The design assumes that neural activity is short and occurs in discrete intervals. The signal amplitude or the timing between the events or conditions in the stimulus being investigated is extracted and compared. As in the block design the power is a function of the events over which the averaging is committed. In terms of data analysis, the event-related design is more complex than the block-design. Several methods defining fitness measures for estimation efficiency and detection powers while allowing for randomness have been proposed for event-related designs (Liu and Frank, 2004; Wager and Nichols, 2003). Mixed designs, a combination of block and event, have also shown to be highly effective for detecting transient and sustained neural processes (Chawla et al., 1999; Visscher et al., 2003; Madden et al., 2010), by examining the temporal profile of the activity between the two processes. In this design, within a block multiple events are embedded and this is shown to be effective when investigating interaction between the neural processes active at different time-scales. By this design, it is possible to independently dissociate the regions based on the activity 59

61 profile. Some of the consideration for selecting this design is the minimum number of subjects (Peterson and Dubes, 2011) required requires that the block transitions and trails be spaced efficiently (Friston et al., 1999; Dale and Buckner, 1997). For understanding higher-order cognitive processes using complex stimulus, the block/event designs fell short due to inability to identify the appropriate control as even rest-state to task-state comparison in activation was found to be inaccurate due to the highly active nature of rest ( Stark and Squire, 2001; Raichle et al., 2001; Mazoyer et al., 2001; Biswal et al., 1995). Thus, to study cognitive processes like recall/memory, imagination or executive functions such as decision making or learning or even how one perceives the continuous multi-modal inputs as in real-life, longer non-static stimulus are required. In the free-design paradigm the duration of the stimulus is longer and no-task or self-report is acquired during the scanning. A resting-state or no-task/ free-viewing study requires the participant to just lie still and focus on a blank screen or watch a continuous visual stimulus like movie or hear a long music piece. Recording neural activity in a free-viewing naturalistic stimulus mirroring real-life processing has gained considerable interest in the recent times (Zacks et al., 2001; Bartels and Zeki, 2004; Hasson et al., 2004). Other studies include investigation into how the brain navigates a spatial layout (Maguire et al., 1999; Spiers and Maguire, 2007) or while driving (Calhoun and Pearlson, 2012; Spiers and Maguire, 2007b), all of which emphasize the need for free-viewing or naturalistic experience conducive designs in fmri. To correlate the BOLD response to stimulus events, the methods adapted were: a) rating or classification of the events in the stimulus post scanning, b) a minimally intrusive mechanism to collect participant behavior or response during the scanning, c) using methods like independent component analysis (ICA) or by multi-voxel pattern analysis derive the hidden features from the fmri data. Detailed descriptions of the statistical methods are provided in the sections below. 60

62 In summary, the experimental design which can balance the constraints of the data acquisition technique, the data analysis method, stimulus presentation design and importantly the underlying hypothesis or premise being studied, is a critical for a fmri study. 3.2 Pre-Processing of fmri data In a typical MRI scanner, the participants are made to lie on their back with the cylindrical cavity around the head. The participants are instructed to remain still with no head-movement and continue normal breathing while visual stimulus is projected on a mirror above their head provided with headphones for auditory input. For motor tasks, a keypad is provided and care taken to minimize acute physical movement as this introduces motion artifacts in the collected images. The challenge for fmri data analysis is in determining neural activity of interest and separate confounding effects like head movement and/or breathing. This is accomplished by identifying differences in signal patterns and the spatial location of the signal source. Subject movement inside the scanner causes the position of the brain and hence the voxel position within the functional image to change over time. That is, the voxel s time series does not refer to the same point in the brain and hence motion correction by way of orienting all images within a session is crucial. All the images are registered to one image (realignment) in the session and co-registration is accomplished by rigid-body six-parameter least-squares spatial transformations three rotations and three translations using intra-modal voxel similarity functions. As the slices are acquired one at a time over the repetition time (TR, detailed definition in the appendix), the resulting timing distortions among different slices in a volume are adjusted using slice time correction procedure. In slice timing correction, 61

63 the voxel time series is adjusted towards a common reference by interpolation methods like sinc interpolation. Next, to enhance the signal-to-noise ratio (SNR) and have spatial smoothness, spatial filtering or blurring is applied with a Gaussian filter. This step also enhances the signal-to-noise ratio. The width of the filter, usually between 3mm to 10mm full-width-half-maximum, determines the extent of blurring. The next stage is the intensity normalization to rescale the mean intensity of the signal to compensate for variations in the global signal within and between sessions. In the former, the compensation is on the changes in the intensity over time, and in fmri the global change is smaller and slower than the acquisition rate which manifests as slow drift. The slow drift can be removed by intensity normalization. The mean or the median intensity is used to scale all the values within the volume to a common value. For between sessions signal changes, a similar intensity normalization as in withinsession is carried out except for that instead of volume-by-volume, session-wise corrections are conducted. In spatial normalization, the structural image is normalized to a standard brain template (in our case it was the MNI template - ICBM152), and the obtained transformation (warping) parameters are applied on the co-registered fmri data. This allows for an extraction of the MNI coordinates. In standard packages like SPM (Statistical Parametric Mapping, the above described preprocessing is a stepby-step process of a) Realignment, corrects for motion. b) Co-registration: the individual functional and anatomical images are aligned, c) Spatial normalization: the images are transformed to a Montreal Neurological Institute s (MNI) standard brain space generated from 152 individual brains. A 12-parameter affine transformation is applied that first matches the whole head followed by registration processes by weighting the voxels, d) Smoothing to enhance SNR is by applying a Gaussian spatial smoothing filter as explained above. 62

64 3.3 Task-related Data analysis The fmri signal is a mixture of the brain activation, cardiac and respiratory functions which can overlap spatially and/or temporally with head movement fluctuations and machine noise. This renders the task-related signal changes estimation a challenge. To characterize the variance of this low signal across time and space, univariate or multivariate techniques have been applied to fmri data. In the univariate technique, each voxel is examined and is considered as taskrelated if the voxel values pass a predefined significance under null hypothesis and grouped as per spatial location. In the multivariate approach, the relationship between voxels is considered by estimating the correlation of the time series signals of a pair of voxels. Spatially distributed voxels showing co-activation are thus identified. In the sections below one method each from these two techniques are presented. After the preprocessing step, to detect changes in the neural activity two main methods employed are univariate correlation-technique as implemented in the SPM package (Friston et al., 1991; Friston et al., 1995) and a multivariate data-driven technique. A popular method to identify task specific brain regions is the correlation technique. In this method, the duration of a neural activation in a brain area that process a task or event is predicted and next the neural activation is modeled as a boxcar function. In this function, the value is 1 for the duration the event or stimulus lasts and 0 when turned off. Contrast of the signals from these two conditions will give the BOLD response difference. Next, to predict the BOLD response the neural boxcar function is convolved with the BOLD response denoted by the impulse function or hrf (explained in the section above). Correlation estimation between the predicted and observed BOLD values in each voxel determines task-related activity. The correlation method as 63

65 applied in the General Linear Model (GLM) is explained in detail below. A variant of the correlation-technique is the model-based (O Doherty et al., 2007) approach applied to specifically separate the brain regions associated with two distinct processes. In this approach, before the correlations are estimated, an independent computational model evolved from the behavioral data collected from each participant is fit to the BOLD signal. A model of the neural activation for each subject is constructed from the parameters estimates of the fit. Next, the GLM is used to generate the statistical parametric map, but unlike the correlation-technique the predicted neural activation is first convolved with the hrf to generate the BOLD signal. In the data-driven method, consistent signal changes across the whole brain or sets of voxels are detected and the consistency in the signals by multivariate analysis is compared. Multivariate techniques separate the data into a set of spatial patterns, enabling analysis of co-activation in spatially segregated locations within a given map. One multivariate method is the Independent Component Analysis (ICA), where the source signals are recovered from mixtures with unknown coefficients, with the assumption that the source signals are statistically independent General Linear Model In the GLM, the variation of the time courses of the output y is a linear combination of the signal x and an error term. Hence for one variable the GLM is written as: yi = xiβ + ε where β is the slope and ε the error term. For many variables, the matrix form is given by 64

66 Y = Xβ + ε In fmri, the columns of X contain the vectors corresponding to the predictor or stimulus on/off conditions and by estimation of the magnitude of the parameter in β corresponding to these vectors, the presence or absence of the activation can be detected. β is solved by: X T Y = (X T X) β The error term can be determined from this equation and statistical inferences arrived at on whether β corresponding to the activation response model is significantly different from the null hypothesis. In statistical parametric mapping (SPM), each voxel in the image is assigned a value with the assumption that the null hypothesis is false. For comparison of brain activity, only those SPMs which have statistical significance estimated by simple T- tests and by the application of thresholds are identified. The GLM is a commonly used efficient model-driven method for identifying stimulus-related brain activity for controlled and simple stimuli where the onset of an event or action is timed and hence used as regressor. The univariate techniques like SPM (Friston, 1995) fall short in extracting the intrinsic structure of the data especially when the exact response to events are not known apriori as is the case when performing higher order cognitive tasks (McKeown et al., 1998). Secondly, correlational techniques require averaging across many trails of a task presented in blocks, which does not allow for detecting task-related transient changes in the signal in tasks like learning. Phillips et al., (1984) proposed that brain s functional organization is based on localization that each function is performed by a set of brain areas and by localization, which is based on the principle that many areas spatially separated functionally connect to perform a task. Based on which, McKeown et al., (1998), 65

67 proposed that areas activated for a task are not necessarily related to the signals from brain areas affected by artifacts. To examine brain activity for non-block long duration stimulus presentation, a data-driven approach like Independent Component Analysis (ICA) where whole brain dynamics can be extracted was considered to be more effective (Calhoun et al., 2002; Malinen et al., 2007; Raz et al., 2014). A mixed method of a model-free data driven approach with ICA applied first on a single-subject level and group activation maps generated in a second-level GLM analysis was proposed by Schöpf et al. (2011) Data-driven method With new fmri data collection techniques by means of manipulating the r.f pulsesequences, increase in the strength of the magnetic field applied and the ability to present stimulus in naturalistic way inside the scanner, research interests is now focused on investigating brain processes in situations as encountered in a multimodal complex environment. Data-driven method like the independent component analysis (ICA) (Hyvärinen & Oja, 1997; McKeown & Sejnowski, 1998; Hyvärinen, 1999; Hyvärinen & Oja,2000; Calhoun and Adali, 2006) is an approach that is widely used to analyze the complex data from naturalistic viewing conditions. This method is a statistical approach wherein multidimensional data is transformed into spatial or time independent components. The principal component analysis (PCA), is also a multivariate technique, which measures the covariance between pairs of voxels and provides the spatial maps with greatest variance in the data and is used for data decomposition and dimension reduction (Friston et al., 1993). The basic goal of the PCA method is to de-correlate the signal, and hence often used as a pre-processing step for ICA, by 66

68 projecting the signal onto orthogonal axes and ranked in order. ICA then finds the statistically independent components from the mixed signal not necessarily ranked. The clustering or data segmentation is a multivariate technique where clustering of closely related data points into subsets is conducted. To capture the structure of the clustered data, methods like the Gaussian mixture model are applied by assuming that points within a subset are closely related than with those from another cluster. Cordes et al. (2000) used a hierarchical clustering (Goutte et al., 1999) on resting state data and reported high correlations at low frequencies in clusters of neighboring voxels suggesting functional connectivity (Biswal et al., 1995). The other multivariate approaches being Multi Voxel Pattern Analysis (MVPA), (Norman et al., 2006; Haynes and Rees, 2006) and the reverse-correlation method (Hasson et al., 2004). The MVPA is a machine-learning method based on the idea that if a particular brain area displays distinct response to two different events then a classification scheme can be applied to analyze the responses from the voxels of that region to identify the actual event responsible for the response. The inter-subject correlation (ISC) first applied by Hasson et al., (2004) on fmri data collected in a free-viewing experiment demonstrated voxel-wise time series correlation between subjects in the auditory and visual cortices. In this method, the hemodynamic activity of one subject is used to quantify that of another by calculating the correlation coefficients of the time series of the two subjects, while the spatial location of the activation is inferred from the across-subject similarity. Though some questions on the interpretation of the between subject correlations has been raised, Hanson et al., (2009) and Pajula et al. (2012) have shown that the inferences are reasonable. The goal of the data-driven methods is to analyze continuous neural activity collected from naturalistic stimuli presentation, without a regressor and identify a network of active spatially distributed brain areas at a given time period. Some of 67

69 the recent studies (Nummenmaa et al., 2012; Raz et al., 2014; Malenin et al., 2007 many others have been discussed in detail in Chapter 2 and in Chapters 4 and 5) have shown interesting results and challenges in data analysis. Lahnakoski et al., (2012a, b) annotated several auditory, visual features and scored motion categories in an edited clip from the feature film Match factory girl ( dir. Aki Kaurisma ki, 1990) and collected fmri data from a free-viewing design paradigm. They compared the results from applying the model-free ICA approach to those obtained by the GLM analysis. Some of the main points from the comparative analysis are: a) for a naturalistic experiment, when results are estimated for the whole dataset, neither of the methods could extract the minute spatial activity patterns in shorter time windows of the movie, hence making it difficult to make strong correlation between specific events in the movie to the activity recorded; b) though in a fairly long feature-film two events do not co-occur, but if they do then ICA groups the areas into a single intrinsic network; c) ICA as a method is highly advantageous in finding inter-subject similarity in functional connectivity. To study the time-locked inter-dependence between brain data and the content presented in a movie, Kauttonen et al., (2015) selected a non-narrative silent film ( At Land ) which depicts a young lady wandering in her surroundings showing no expressions on her face or a sense of purpose for her movements. The task from the data was to link the cinematic features to the brain activation of the viewers. For their analysis, they combined ICA, ISC and the linear regression methods and in addition to address over fitting due to large annotation set, applied the elastic-net (Zou and Hastie, 2005) method and compared to the partial leastsquares (Wold et al., 2001) method. They conclude that the elastic-net method to be more sensitive to the features of the movie but also mention that no-single method has still proved to be effective in connecting the brain activation to the dynamic changes present in naturalistic movie stimulus. 68

70 Hence challenges in data-driven methods is mostly the inferencing constraints imposed due to being purely exploratory, which implies that a priori hypothesis about any of the components cannot be tested easily and the computation time and resources required to run a ICA especially for large data sets also is a constraint. Importantly, an accurate method to identify task-only related components is still elusive considering the large number of components isolated making identification of that signifying stimulus response difficult. In the section below, the ICA method for fmri data is elaborated followed by a graph-theoretical method relevant to the research presented in this report Independent Component Analysis The standard example for the blind-source separation is the cock-tail party problem, referenced from Hyvärinen & Oja, 2000, is explained below. In the setup, the microphones were placed at different locations in a room to collect audio from persons speaking simultaneously and the challenge was to separate the signals as a function of the spatial location of the source of each voice and its time course. Let us consider 2 speakers and the signal amplitude as a function of time t from each microphone as x1(t) and x2(t), and each of the recorded signals as a weighted sum of the original source signal (s1(t) and s2(t)) then the linear equation is: x1(t) = a11 s1(t) + a12 s2(t) x2(t) = a21 s1(t) + a22 s2(t) where the value of the aij depends on the distances of the microphones from the speakers and if these values are known the above equation can be solved by classical methods. The optimal solution would be to estimate the two original signals from the recorded signals directly. But as the values of a are not known, and to solve the problem, a safe assumption is statistical independence of s1(t) and s2(t) at each time instant t. That is, the information from s1 does not give any 69

71 information on s2 and vice versa. In vector-matrix notation the above equations can be written as: X = As, By the ICA model, where only random vector X is observed, A and s are estimated. This is accomplished by first estimating the matrix A called mixing matrix as it includes the coefficients of the linear mixing and then the independent source signals can be obtained by: s = A -1 x The ICA method is used to estimate the A based on the assumed independence of the two original source signals and hence separates the signals from the mixed collected signals x. The ICA method for blind source separation assumes statistical independence of the source signals, that is, in the case of the cocktail situation the voices are assumed to be independent of each other. Similarly, the observed signals in fmri are assumed to be a linear mixture of task and non-task related processes, as long as there is no overlap in the activations in either time or space (Calhoun et al., 2001). That is, to apply ICA, five assumptions must be met, 1) statistical independence between the source signals and the source vector, ( s ), 2) the number of linearly independent mixtures ( x full rank matrix) must equal the number of sources of signal, 3) the model is assumed to be free of external noise, 4) the source signals should not have Gaussian probability distribution function except for one signal source and 5) the data is centered (zeromean) or whitened. To identify the mixing matrix and the source signals, two general classes of algorithms Infomax (Nadal & Parga, 1994; Bell & Sejnowski, 1994) and FastICA (Hyvarinen,1999) which apply intra-correlation have been 70

72 mostly applied. The Infomax algorithm is based on minimization of mutual information, while FASTICA is a fixed-point fixed-point algorithm for maximizing the statistical independence of the components by maximizing their non- Gaussianity. A reasonable assumption to make is the set s to not have Gaussian distribution, and hence the weighted sum of all components will be more Gaussian. This algorithm quantifies the non-gaussianity by maximizing the negentropy of the estimated components (Hyv arinen, 1999). Negentropy or negative entropy describes the quantum of information that can be gained from observation. That is, observation of variables with lower entropy will enable identification of a narrow range of values the variable is likely to take. Higher entropy translates to higher degree of randomness and less predictive power. The Infomax algorithm is a stochastic gradient ascent algorithm (Nadal & Parga, 1994; Bell & Sejnowski, 1994), based on the mutual information to maximize independence of the components. By maximizing the joint entropy, minimizing the mutual information is committed between two variables to reduce redundancy. Hence, ICA technique is efficient in identifying separate statistically independent source signals from a mixed set of observations. Though similar to the principal component analysis (PCA), a technique that identifies data sets that are orthogonal and uncorrelated. The advantage of the ICA method is that it does not restrict components only on orthogonality and decorrelation, so, the bases can be similar if the independence requirement is met. ICA can be applied to fmri to either spatially localize the sources of the BOLD activation (McKeown e al., 1998) or characterize the activation temporally (Biswal and Ullmer, 1999). In spatial ICA (sica), the signal sources are independent in their spatial locations and not necessarily in time, and sica isolates non-overlapping temporally coherent brain regions without limiting the temporal design (Calhoun et al., 2003). Also, temporal points are sparse data corresponding to each functional image acquisition while the 71

73 millions of voxels give dense spatial location information. The sica is considered for the analysis of the fmri data presented in this thesis Spatial ICA of FMRI data. In fmri modeling, ICA is applied to extract the spatio-temporal structure of the signal. In spatial ICA (McKeown et al., 1998), the basic assumption is that the spatial distribution of the brain areas is time invariant and statistically independent. In a matrix format, the individual voxel time courses form the columns of a matrix Y and each row in this matrix represents a spatial image at any fixed time point. That is, Y = number of time points (T) x number of voxels (V), and the number of time points would be less than the number of voxels. The preprocessing steps on the matrix area: a. Average of all the rows of the matrix Y to remove the mean spatial map. b. Each column of Y is set to unit variance to normalize the variance of each individual time course and c. Average of all the columns of Y to remove the mean time course. Post these processes the general ICA as detailed in the section on ICA above is represented as: Y = AS, where A is the T x T time courses and S is the T x V matrix of the spatial maps which are pair-wise statistically independent. The task is to find the matrix A, by choosing a function that is an estimate of the statistical dependence between rows of the spatial maps (S), by functions such as Infomax that examines the mutual information and Negentropy (Hyvärinen & Oja, 1997). 72

74 Group ICA In the ICA method, the independent components of each subject have different time courses and hence group level inferences are not a straight forward estimate. Group ICA has been attempted by approaches such as spatial correlation of the components or self-organized clustering (Calhoun et al., 2001a; Esposito et al., 2005). While these two methods allow for unique spatial and temporal features the data is noisy as the data is not unmixed uniformly for each subject. Other methods are temporal concatenation wherein each subject has a unique time course with a common aggregate spatial map. A second method is the spatial concatenation where unique spatial maps are generated but with common time course. The most efficient technique has been the temporal concatenation (Calhoun et al., 2001b; Guo & Pagnoni, 2008; Schmithorst & Holland, 2004), which when followed by aggregate ICA allows for back-reconstruction step to compute the time-courses for each subject. In this approach, a common group map is assumed even though the time courses for each subject will vary. In the software package, GIFT ( (Calhoun et al., 2001b; review: Calhoun et al.,2009) to conduct the ICA analysis the temporal concatenation has been implemented Pre-processing for ICA In the GIFT software package implementation of the ICA, the preprocessing steps include centering, where the mean is subtracted to create a zero-mean signal, a whitening process of the collected fmri signal and dimension reduction by PCA. A scaling routine is applied where each image and time course are divided by its standard deviation and the percent signal change is estimated by scaling the signal to the mean to reflect the percent signal change. 73

75 Sorting of the ICs After estimating the independent components, the sorting of the ICs is by temporal, spatial or multivariate options. The important task after the ICs are separated is to differentiate between those that signify neural signals related to the stimuli and that from artifact or system noise. Physiologically artifacts from respiration, cardiac and movement by subject effect the signal strength and hence the relaxation times measured. Scanner noise and magnet drift also adds to the artifacts and noise to the signal, making it difficult to detect the real BOLD signal change (Weisskoff, 1996). This is particularly a challenge in experiments using naturalistic and varying stimulation, wherein the hemodynamic response of the brain is not easy to model. The challenge arises because of the high dimensionality and high noise levels of fmri data, applying ICA on this huge data will lower the estimates from the ICA (Sarela and Vigario, 2003). Some of the methods to address this issue have been canonical-correlation for stimulus timing (McKeown et al., 1998; Calhoun et al., 2002) which considers features of the stimulus to group ICs (Ylipaavalniemi et al., 2009) and the ranking based on power spectra of the IC (Moritz et al., 2003). Brainatlas images to examine spatial consistency have been used for component sorting (Calhoun et al., 2008). Component reproducibility across several estimation runs is also used to rank the ICs (Himberg et al., 2004; Yang et al., 2008). Kurtosis or onelag autocorrelation (Formisano et al., 2002) is a popular method for classification. Noise and artifacts signals are high-frequency modulations, and Moritz et al., (2003) in a complex motor paradigm study ranked the components according to frequency content. The ICA technique has been used to identify spatial patterns of activation and noise in fmri data (Biswal et al., 1999; Hansen et al., 1999). Thomas et al. (2002) applied unsupervised algorithms on Principal Component Analysis (PCA) and the 74

76 ICA method to isolate the random noise (white noise) and structured noise (physiological). The PCA is a commonly used to reduce the dimensionality of the data without loss of information by removing bases with little or no significant contribution to the overall variance. To evaluate how random noise effects the time series, Thomas et al. (2002) evaluated the power spectra of each component with a white noise criterion or a cutoff. The components higher in rank which passed the white noise criterion were automatically zeroed for random noise. For structured noise, components below the cutoff were considered relevant for analysis. The frequency bands of the noise from respiration were between Hz and cardiac frequencies range between Hz. Another challenge is in estimation of the optimal number of independent components and relating this to the results (Abou-Elseoud et al., 2010; Li et al., 2007). Too few components can lead to loss in information (Green & Cordes, 2002) and excess will decrease the stability and reliability (Li et al., 2007). In principle, the number of components should be equal to the number of possible sources of signal in the data, but this makes the data to be analysed large, resulting in over fitting and spatial sparseness of the brain areas in each component. An information theoretic approach, like the minimum description-length criterion (MDL) (Rissanen, 1983) has been implemented in the GIFT software package to estimate the number of components. Comparisons of the efficiency of other information theoretic approaches estimate the optimal number of ICs has been conducted by Majeed and Avison (2014). Li et al., (2007) point to the shortcomings of the Information-theoretic criteria for component order selection as these methods are based on the likelihood of independent and identically distributed data samples. They propose a modification by a sub-sampling scheme to obtain a set of effectively independent and identically distributed samples from dependent data samples followed by application of the information-theoretic 75

77 criteria on this sample set. The time-courses of the ICs estimated by the MDL method and the areas of activation as labeled using MNI coordinate system are presented in the appendix at the end of the thesis. The ICs with areas covering mid-brain and cerebral spinal fluid are clearly isolated into separate ICs, while 4-5 ICs with activations in the prefrontal, temporal, precuneus and insula were also identified. In summary, the ICA method is a promising data analysis technique for fmri data (Bell and Sejnowski, 1995; McKeown et al., 1998b) especially for experiments where the stimulus is presented for longer duration in naturalistic viewing design. 3.4 Functional Network Connectivity Functional connectivity of spatially segregated areas as presented by the ICA method characterizes possible between region interactions for tasks or resting state. Other methods include the seed-based correlation (Cordes et al., 2000), where correlation between a selected seed region to other regions is estimated. In addition to the between regions correlations, it is of interest to analyze the relationship between spatial ICs, as even though the components are maximally independent, the time courses of ICs do exhibit temporal dependencies (Calhoun et al., 2003). The functional network (each IC is an intrinsic network) connectivity is estimated from the time series by lagged correlation (Jafri et al., 2007) averaged across the whole-time points. For understanding how the functional connectivity varies over the time duration of a stimulus, the time courses of the selected ICs were pair-wise correlated to arrive at the dynamic functional network connectivity (dfnc). The approach (Allen et al., 2012) involves subtraction of remnant noise by linear de-trending, replacing any outliers with best estimate from a third-order spline fit from the smoother part of the time course. Next step is filtering using a high- 76

78 frequency cutoff at 0.15Hz as previous work (Sun et al., 2004) has shown that functional association between the low-frequency component of the fmri data is task-related while high frequency (>0.2Hz) is usually attributed to noise from artifacts and hence show no functional associations. A covariance matrix is generated by normalizing the variance of the time courses. The dynamic covariance is calculated by a Gaussian sliding window procedure, with a window width of 30TRs or 60seconds, as 64s suggested to be optimal by Sakoğlu et al (2010). The Gaussian window alpha value was 3TRs (6 seconds) and a sliding step size of 2 TRs or 4 seconds was applied to the analysis of the data collected as part of the thesis work. For this study, we consider FNC time points only for the pairs of ICs of interest which are the empathy and default-mode networks. The correlation coefficients are Fisher z-transformed and average at each time point across all the subjects is calculated for the subject-group functional network connectivity analysis. The dfnc tool is provided in the GIFT toolbox and the methodology is reported by Allen et al. (2012). 3.5 Data analysis for the studies reported in this thesis Understanding how the human brain processes rich natural environment is the goal and challenge in neuroscience. Most experiments have studied complex constructs like empathy by designing controlled experiments with very short video clips, images or texts. Empathy is a complex construct involving intricate processes and evolves as information presented in the stimulus is assimilated. Hence looking at either simultaneous time-varying brain activity changes at regions of interest or whole brain dynamics of neurocognitive components emotional, motivational and cognitive - provides a comprehensive approach to study the processes of interaction. The studies reported in this thesis are aimed at identifying narrative empathy networks evoked for ecologically valid naturalistic stimuli. 77

79 The data-analysis methods applied include the data-driven ICA method and crosscorrelations to study dynamic functional connectivity between intrinsic networks/ics, plus the general-linear model based statistical parametric mapping to look at response differences for short events in the long stimulus. In Study 1 (Chapter 4), the ICA method was applied on fmri data collected while participants watched three diverse genre of movie clips to explore differential empathy networks as a function of the empathy rating and emotional quotient. In study 2 (Chapter 5), to investigate large scale dynamic functional network connectivity between selected ICs with empathy attributed brain areas and default-mode network was conducted to study differences in dynamic correlations of the three modes of empathy with DMN. In Study 3 (Chapter 6), to highlight the significant difference in empathy response to a short emotional or neutral scene of few seconds in a narrative (where the context is provided in full) compared to the same scene without the context, two full length short movies were used. The GLM/SPM was applied to compare the empathy related neural activity for the two conditions. 78

80 Appendix Background In this supplementary material, the quantum to classical description of nuclear magnetic resonance will be explained, the physical basis upon which Magnetic Resonance Imaging rests. The standard texts by Abragam (1961) and Callaghan (1991) covers the topic in detail. Magnetic resonance imaging process is also briefly discussed in this section. Magnetic resonance imaging Spin and magnetic fields In the quantum mechanical description of atomic nuclei, the spin angular momentum was first introduced by Dirac (1930) and the first experimental observation was by Stern and Gerlach (1924). The value of the spin angular momentum is an intrinsic property of the nucleus (for example: 1 H = ½; 2 H = 1), and magnetic resonance is observed only in non-zero values. For most medical applications, the hydrogen nuclei with unpaired electron ( 1 H) is of most interest as water and tissue in living organisms have an abundance of hydrogen atoms. The nuclei with non-zero spin behave like magnetic dipoles, précising with an angular frequency ω and in a stronger external magnetic field orient either along the field or to it. The magnetic dipole µ, is proportional to angular momentum and linked by the magnetogyric ratio. The magnetic diploes in the external field Bo have energy E from which the Hamiltonian and subsequently the energy for the eigenstates (spin-up or spindown) are derived. A transition between the two states of unequal energy is triggered by emission or absorption of a photon of frequency ν or in angular terms, the Larmor frequency ω0 =γ Bo. The strength of external magnetic field determines the ratio of spins aligned parallel or anti-parallel, with higher strength 79

81 resulting in more protons in the higher-energy state and hence higher signal strength. The spin-up and spin-down magnetic moments of an ensemble of spins leads to bulk magnetization M and explained by classical physics. The bulk magnetization vector in an external magnetic field will experience a torque and the solutions from the differential equation of motion solved for the three axes (x,y,z) give the precession of the magnetization vector about the magnetic field (shown in Figure A3.1), with an angular frequency identical to the Larmor frequency derived in the quantum mechanical description. Figure A3. 1: (a) The precession of the magnetization vector with angular frequency ω in the static magnetic field B0. (b) The magnetization under the influence of a static field B0 and a transverse field B1. In order to get a detectable signal, radio frequency pulses are applied at the resonance frequency as governed by the Larmor equation of the tracking nuclei. The RF field (B1) of 90 0 pulse tips the dipoles into a plane perpendicular to the B0 direction, and they precesses at frequency ω1. The applied RF field makes the nuclei phase coherent and when the pulse duration ends, the nuclei will relax back to lowenergy state, as free induction decay signal. If ω1= ω0, then a perfect resonance is 80

82 achieved and the transverse magnetization will be motionless and relax back to equilibrium state, as exponential decay, but if not equal then the signal oscillates at a frequency. The Fourier transformation of the free induction signal gives the value of Δ. The return to equilibrium state involves an exchange of energy to its surrounding and amongst the spins. The former is called spin-lattice relaxation, T1, and the later due to loss in phase coherences between the nuclei is spin-spin relaxation, T2, which is faster and derived from the width (full-width-half-height) of the peak of the frequency Δ. T1 happens in the Z direction while T2 in the X-Y plane. The decay of the signal is also due to magnetic field inhomogeneity in addition to the spin-spin interactions T2, that is, T2 * = 1/T2 + γπδb0., where ΔB0 is the field variation across the sample and this weighted relaxation time is smaller than T2. The relaxations induce a measurable current in a coil positioned at rightangle to the static magnetic field to collect the signal. The free-induction decay is the signal collected from the coil and is transformed to the rotating frame of the spins after phase sensitive detection and analyzed. Before the physics of resonance imaging is discussed, a summary of the NMR method is useful. In a sample, biological or chemical, with a free-proton have unaligned spins and when placed in a strong static magnetic field the spins precessing with Larmor frequency align along the direction of the applied field. To measure the Larmor precession, an oscillating magnetic field at the resonant frequency is applied orthogonal to the static field as a short pulse, which tips all the spins into the transverse place. When the pulse stops, the spins relax back to the equilibrium state at different relaxation times, and these offsets relaxation values are useful for magnetic resonance. R.F Pulse sequences and gradients In imaging, these two relaxation process determines the contrast, from which different tissue matter can be identified. For example, the T2 for grey matter in the 81

83 brain is 200msec and T1 is 900msec, for white matter T2 = 90msec, T1 = 780msec, indicating that size of the structures and interactions between the spins influence the relaxation times. The contrast imaging is derived from the relaxation times, proton density variations and blood flow, which is explained in detail in the magnetic resonance imaging section below. To measure the relaxation times from different tissue types in the brain and at different time points, the relaxation rate should be long. This is possible by manipulating the external r.f pulse applied to tilt the magnetization. The two fundamental techniques are spin echo and the gradient echo. An echo signal in imaging is similar to the rebound signal after the decay of the original signal. The main concept for imaging rests in understanding how relaxation time signals are converted into images. The mathematical concept to convert the frequencies generated from the r.f excitation in a static magnetic field into an image was proposed by Kumar, Welti and Ernst (1975) and this set the basis for MRI techniques. The first full body imaging was by Damadian (1977), who used this technique to detect tumors. In addition to the free-induction decay, the other signals post r.f pulse excitations are the Radio frequency echoes and gradient echoes. An echo signal is basically the sum of the refocusing phase of the transverse magnetization and the dephasing part and can be generated by multiple r.f. pulses applied with time-interval between the pulses and by a magnetic field gradient reversal. The spin echo also called the Hahn echo was first described by Erwin Hahn (1950) and is a method in which two or more r.f pulses irradiate the sample and the signal collected post this sequence. For example (Figure A3.2a), a 90 0 excitation pulse is followed by an pulse after a short time interval, shorter than T1, and the echo signal (the FID reconstructed), is measured post the second pulse after an equal time interval. The second pulse does not alter the transverse relaxation, T2 or the spin-spin relaxation. 82

84 Immediately after the first 90 0 pulse the spins in the transverse plane start dephasing leading to signal at echo time TE, with faster and slower spins moving away. The signal strength collected by the receiver coil will be weak due to the dephasing, and the pulse sent at t = TE/2 will tilt the spins and now rephasing or echo occurs with the faster and slower spins converging, hence a stronger signal is detected as the spin density is higher. The time between the successive r.f pulse excitation is called the repetition time (TR). A pulse sequence of three consecutive 90 0 pulse, called the Stimulated Echo, checks the accumulation of phase but the signal strength is lower. A most widely utilized sequence is the gradient Echo (GRE), where the echo signal is generated by application of a rephasing gradient pulse to cancel a preceding dephasing gradient pulse along the same axis. The transverse magnetization is back into phase by the cancelation effect due to the reversal of the gradients leading to a strong magnetic resonance signal. The time between the r.f. pulse and the peak of the echo is the echo time, TE, similar to spin-echo (Figure A3.2b). In the gradient echo, the magnetic field variations are not cancelled out as is the case with a spin echo signal, but only the dephasing is reversed. Hence the peak signal decays exponentially with a time T2 * (weighted), which includes the effects due to the magnetic field inhomogeneity. For MR imaging, the pulse sequence, the r.f pulse angle, the time intervals TE and TR are selected dependent on the type of signal being investigated. For example, a T1 weighted echo signal is detected if TE is short and TR is shorter, while long TE and TR gives a signal that is either T2 weighted (spin echo) or T2 * weighted (gradient echo). Tissues from different part of the human body have distinct relaxation times, brain tissue scanned at a magnetic field of 3Tesla is known to have a T2 * of 30msec. The white matter in the brain has a T1 value of msec and between msec for gray matter, T2 ranges from 70-90msec. 83

85 Summarizing, after the r.f excitation pulse the relaxation of the spins from the transverse toward B0 is T1-weighting. Simultaneous, decay of the spins in the transverse plane due to phase disruption is T2-weighting. The influence of other factors like magnetic field inhomogeneity increases the relaxation time and is the measured time T2 *. A spin-echo sequence which refocuses the phase reduces the field inhomogeneity effect on the T2 relaxation times. The amount of weighting in the spin echo or the gradient echo sequence is determined by the echo time TE, and gradient echo sequence can only give T2 * while spin echo peak signals are also T2-weighted. Most MR imaging schemes like the echo planar (explained in the next section) or the inversion recovery sequence are derived from these two main pulse sequence methods. For smaller volume of samples, FID signals suffice for T1& T2, while for larger samples a complex process where spatial, phase and frequency encoding is applied to arrive at spatial location specific signals from which an image can be generated. The gradient encoded signal for spatial information does not affect the spin-spin or spin-lattice interactions and hence images with anatomical and physiological information can be generated. To arrive at spatial encoding, linear magnetic field gradients (Figure A3.2b) which modify the static magnetic field are applied. A linear magnetic gradient field parallel to the static magnetic field direction divides the sample into slices. A slice is of a particular thickness and the orientations usually applied for the brain are called coronal, sagittal and axial plane. The precession frequency of the spins in a gradient field in the X-direction is ω = ω0 + γgxx, hence in the gradient field the frequency and phase of the signal depends on the location in the sample being scanned. The shape of the r.f pulse amplitude determines the frequency range at each slice. The signals from one slice at a time can be acquired by ensuring that the r.f pulse excitation affects only the spins in this slice and along one direction. To collect signals from the sample slice, the receiver coil with a resonant frequency of the 84

86 gradient field at the slice is applied. Due to the gradient applied for a time-interval the transverse component of the magnetization changes phase. The gradient makes the protons spin at different speeds and leading to phase incoherence. When the gradient is switched off, the protons at identical frequencies are now phase encoded. In the third step, to get the signal in the X-plane, an additional frequency gradient is applied along this direction by which the protons resonant to varying frequencies. By these three processes, an extremely small volume- voxel- is created, where the Gz gradient selects the axial slice, the Y-direction gradient Gy are rows of spins with different phases and X-direction gradient the columns with spins of different frequencies and hence a signal from this volume element, voxel, is a combined signal. Figure A3.2: (a) A pulse sequence and the readout from the rephasing/dephasing process. (b) waveforms of a spin-echo sequence.(reproduced from Erwin Hahn (1950) 85

87 In summary, spatial information in one direction of the selected slice is obtained from the signal collected at the instant the magnetic field gradient is applied. The information along the orthogonal phase-encoding direction in the slice is acquired by switching the gradient off after a fixed time on time. The 2D space from the frequency-phase encoding is called k-space, a Fourier transform of the image. k-space By applying a gradient in the Y-direction (ky) for a time T, switched off and then applying a second gradient in the X-direction (kx)results in a signal for a range of times t. This can be represented as signal and phase information encoded data in a 2D k-space (Figure A3.3) for a single slice. An inverse Fourier transform will give the image. For a transverse slice, the frequency encoding direction is set along the horizontal axis and phase encoding direction along the vertical direction, this is k- space raw data in matrix format. The outer rows of the raw data matrix contain the high spatial frequencies provide the information of the spatial resolution. The inner part or core part of the matrix with low spatial frequencies contains the image contrast information [Mezrich, 1995; Twieg, 1983]. The spacing between the data points and size (pixel) decides the field of view of the image, while the highest magnitude of the k-space coordinates along the X-Y-direction determines the size of the pixel. At the center of the k-space the data contains the weighting from the relaxation times or proton density and from the outer points the spatial information can be obtained. If a very thick slice is selected, then spatial encoding is applied along the three axial directions. In this case two phase-encoding gradients are applied in the Y and Z-direction with frequency encoding along the X- direction. The voxel in MR is determined by the pixel dimensions and the slice thickness and the signal quality is a function of the number of hydrogen nuclei in this volume and the points in the k-space. 86

88 ky. kx Figure A3.3: The 2D representation with each point a representing the signal and phase in the k-space, k xand k y. A limitation to the pulse sequences discussed till now is the time for filling the k- space one-data at a time. Faster methods are required as one cannot be in the scanner for long periods. Shorter repetition time, results in loss in signal intensity and hence the signal to noise ratio, while reducing the sampling data at fewer ky values compromises the image resolution as the field of view is decreased. By filling with zeros, the upto 50% of the data permissible due to the symmetry in the k- space, has shown to be not affect the field-of-view or image resolution at the expense of SNR. Sampling data at multiple ky values for each TR is possible if more than one echo is produced after each r.f pulse, by applying refocusing pulses repeatedly. This method is called the fast spin echo or turbo spin echo. Another commonly used method being the Echo Planar Imaging (Mansfield, 1977; Mansfield & Pykett, 1978) where in the gradient echo sequence, the frequencyencoding gradient along the X-direction is revered continuously (Figure 3.2b). Between the echoes thus generated, the phase-encoding gradient along the Y- direction is switched to change the ky value. The echo-planar imaging is special method of spatial encoding which affects the acquisition time and image quality but not the contrast, while EPI with spin echo or with gradient echo determines 87

89 the contrasts of the features in the image. But the longer acquisition times in the EPI method translates to spatial distortions due to differences in the tissue types and magnetic properties of the tissue. The raw data obtained from the demodulated RF signal detected by the receiver coil is k-space data, a spatial frequency transformation of the original image data. An inverse transformation of the k-space data to get the image-space is required. The EPI data contains an artifact known as Nyquist ghost, which is a lower intensity replication of the main image and by using additional phase information corrections are applied to reduce the intensity of the ghost images much lower than the main image (Schmitt et al., 1998). 88

90 Chapter 4 Evidence of Empathy response brain networks* 4.0 Introduction Understanding the neural correlates of the complex processes attributed to empathy response has been an area of increasing interest in recent years as reviewed in Chapters 1 and 2. The limitations highlighted by the studies also led to a strong interest in characterizing the neural correlates of empathy as it occurs in natural circumstances. That is, a suggestion to explore empathy process by expanding beyond the current choice of using short sentences, static images or short videos having limited contextual factors. An important short-coming of the studies is the inferences that can be drawn while using controlled designs using restrictive stimuli. This point was also emphasized in a recent review of neuroscientific research covering empathy studies by Zaki and Ochsner (2012). Towards this, findings from a functional magnetic resonance imaging (fmri) study employing long visual fictional narratives (movie clips) presented in a free-viewing design paradigm are presented in this chapter. * Vemuri, K., & Surampudi, B. R. (2015). Evidence of stimulus correlated empathy modes Group ICA of fmri data. Brain and cognition, 94,

91 The principal point in using free-viewing design with cinema as stimuli is that it offers a window into the human empathy process without interference in narrative immersion. Emotion and empathy studies using fmri have thus far considered perception of static faces and at the most extended to short clips depicting emotional scenes (Adolphs, 2002, Schulte-Rüther et al., 2007, Fusar-Poli et al., 2009). Studies related to pain (Lamm et al., 2007a, b, Singer et al., 2004) looked at empathy responses to pain inflicted on a stranger, on the participant and on friends. A meta-analysis (Sabatinelli et al., 2011) of research on empathy and emotion revealed the role of set of areas comprising the medial prefrontal cortex (mpfc), bilateral inferior and middle frontal gyrus, superior frontal gyrus (SFG), bilateral amygdala, parahippocampus and fusiform gyrus (FG) for processing emotional faces compared to responses for neutral faces. The study, while comparing responses to emotional scenes versus neutral scenes, indicated additional activation in anterior cingulate, medial dorsal nucleus (MDN), pulvinar of the thalamus and the right superior temporal gyrus (STG). A similar meta-analysis on empathy related research (Fan et al., 2011) found that most of the research papers report bilateral insula, inferior frontal gyrus, parts of the cingulate cortex (BA 32) and bilateral premotor cortex (BA 6) for empathy. Distinguishing between states termed as affective-perceptual empathy and cognitive-evaluative empathy, metaanalysis conducted by Fan et al. (2011) also reported bilateral insula responses, inferior frontal gyrus (BA 47) and right supplementary motor area (BA 6) for the former pathway and left insula and left anterior cingulate cortex (BA 32) for the latter. De Greck et al., (2012) looked at intentional empathy not restricted to emotional expressions and report a network of areas covering the anterior cingulate cortex, bilateral inferior frontal cortex and bilateral anterior insula. A study on accuracy of empathic interpersonal judgments (Zaki et al., 2009) identified the role of medial prefrontal cortex (BA 10) and the dorsomedial prefrontal cortex (BA 9) which was 90

92 also reported to be active for experienced empathy (Rameson et al, 2012). Studies on observation of action have shown to trigger empathic action and said to form the mirror neuron network. This network covers the premotor, somatosensory, inferior frontal, inferior parietal, STG and insula (Carr et al., 2003, Lawrence et al., 2006, Bastiaansen et al., 2009). A lesion study (Shamay-Tsoory et al., 2009) showed that Brodmann areas 10 and 11 may be necessary for cognitive empathy processes while BA 44 was critical for emotional empathy processes. Frith (2001) reported the significance of the paracingulate cortex (BA 32) for cognitive empathy from autism research. A study on the role of race on emotional response (Lee et al., 2008) showed activations in the limbic lobe (amygdala and hippocampus) when perceiving own race versus other race faces whereas the contrast of other race emotional face versus own race revealed activations in the frontal, occipital and parietal areas. These studies highlight that a number of areas are engaged in empathic response even for static images or simplistic short scenes. The use of naturalistic approach to look at large scale brain activation as in watching movies has been successful in providing deep insights into time-varying large scale functional connectivity networks (Hasson et al., 2004; Bartels and Zaki, 2004) correlated to the context being processed. Hasson et al. (2004) report significant inter-subject correlation (ISC) in brain activity when viewing clippings from the movie Good, Bad & Ugly and found correlations in the temporal and fusiform area. A comparison of the responses for sad and neutral films (Levesque et al., 2003) showed higher bilateral activations in the midbrain, anterior temporal pole (BA 38, 21) and right ventrolateral prefrontal cortex (BA 47). Han et al. (2005) studied differences in responses to movie clippings with human-like cartoon, nonhuman cartoon characters and real actor movie clippings and found variations in motion perception for real-actor and human cartoon characters in the medial prefrontal cortex (MPFC) and cerebellum. A study by Mar et al., (2007) comparing the ability to perceive intentions from movements by real actors and computer- 91

93 generated cartoons (sharing similar features) found higher responses in the areas associated with mentalizing the medial prefrontal cortex, the superior temporal sulcus and temporo-parietal junction for scenes with real actors compared to the depictions by the computer-generated animation actors. Studies specific to emotion responses using movies as stimuli include, Goldin et al. (2005) where 2-minute clippings with sad, amusing and neutral scenes were shown to 13 women to identify the neural correlates associated with emotion processing. For the clippings with sad scenes, responses in the medial prefrontal cortex, inferior frontal gyrus, superior temporal gyrus, precuneus, lingual gyrus, amygdala, and thalamus were reported while the amusing clipping evoked responses in the medial, inferior frontal gyrus, dorsolateral prefrontal cortex, posterior cingulate, temporal lobes, hippocampus, thalamus, and caudate. Nummenmaa et al.(2012) used 13 movie clippings with pleasant, unpleasant and emotional events and found that emotional sensations showed higher inter-subject synchronization. An interesting finding from their study is the higher ISC for negative emotions observed in the thalamus, ventral striatum and insula and the default-mode network areas such as the precuneus, medial prefrontal cortex, posterior superior temporal sulcus and the temporoparietal junction, compared to positive emotions. Raz et al.(2012;2014) collected fmri data in a free-viewing design using 10-minute movie clips from two commercial movies, where the participants rated the emotional valence on a continuous scale retrospectively. A new method called network cohesion analysis was applied to the data to explore the functional connectivity within and between empathy specific networks to correlate the acquired temporal patterns from this method to the ratings. They found support to claim the involvement of the cingulate cortex and the insula for the sharing of the bodily expressions while the medial prefrontal- temporo-parietal regions for ToM. These studies re-emphasize that a complex construct like empathy response activates multiple brain areas. But, further studies to examine the networks and the 92

94 functional connectivity of the activations are required as evidence that empathic process unfolds as a function of the perceptual cues and translate to corresponding fluctuations in the network connectivity between the brain areas. Towards this, the main goal of the experiment reported in this chapter was to explore neural correlates of empathy responses to three very diverse commercial movie clips pre-rated for emotional valance. The depiction and the narrative in the three diverse movie clips had emotional contagion to evoke empathic feelings in the viewers as indicated survey filled by the participants. We start off with the following premises supported by independent ratings on the selected movies: a) the movie clips evoke cognitive and emotional processing and b) the activations recorded are in response to the empathetic relation the subject shares with the actor(s) and the context, both assumptions supported by the ratings provided by the fmri subjects and by an independent set of participants. With this assumption, we compare the observed activation to the areas identified in the literature for different empathy modes cognitive, emotional and motor. Considering the freedesign used, the data was analyzed by the independent component analysis (ICA) method (covered in Chapter 3), to isolate spatio-temporal patterns of brain activation. The group ICA (across all the participants) allows for comments on not only the spatial distribution of the activity but also on the common temporal coherence that is essential to label them as a network subservient to various empathy modes. The ICA method has found acceptance to isolate spatially distributed brain regions showing temporally coherent responses from fmri data. This method has been applied to study resting-state and task-based experiments (Kim et al., 2009; Esposito et al., 2006; Malinen et al., 2007; movies: Raz et al., 2012; 2014). 93

95 4.1 Material and Methods Subjects Fifteen healthy multilingual college going adults (age range of years) with written informed consent took part in this study. The gender breakup was as follows: 5 females and 10 male participants. Participation filtering was exclusion due to known mental or physical health conditions, current medication usage, oneweek before the onset of monthly periods for female participants and anxiety or claustrophobia inside a scanner. The ethics committee of the International Institute of Information Technology, Hyderabad had approved the study Stimuli Participants viewed three 5-8 minute clips with no audio from the Hollywood movie The Green mile, a popular Indian Hindi movie Taare Zameen par, and the animation movie Up (the description of the movies is included in appendix at the end of this chapter). The design of the experiment is present in Figure 4.1. The sequences of events in each clip had actors conveying emotional states and a narrative that has the potential to evoke empathy. The actors in the Hollywood movie are mostly Caucasian with an African-American actor as the main character. The actors in the Indian Hindi movie were all of Indian ethnicity and the animation movie had human-like cartoons with Caucasian features. Although the order of presentation of the movie clips was not counter-balanced across subjects, any systematic effects like anxiety and fatigue can be ruled out as confounds in this study as the participants are reasonably experienced with fmri procedures. The Hindi movie which was shown second in order was familiar to the participants while the other two movies were not. Hence, following a sequence where two unfamiliar movies were shown first and last ensures sustained interest without boredom. The movies were so diverse in presentation form, cinematography, visuals etc., that any carry-over effects from one clip to the other are expected to be minimal. Resting-state data was recorded for 4 minutes 46 seconds before the 94

96 presentation of the movies, however analysis of the resting-state data is not included in here. Figure 4.1: The stimuli presentation design paradigm Rating the movies An independent set of forty participants were asked to rate the three clips on two main criteria: facial or body emotional expressions as depicted by the actors and second, the emotional context/empathy response. The order of the presentation of the clips was randomized during this rating study. The rating was on a scale of 1-5, in ascending order of emotional valance. The total score, out of a maximum score of 200, rated for expressions and context for the animation movie was 70 and 68, respectively; 153 and 142 for the Indian Hindi movie and 176 and 178 for the Hollywood movie clip. The Hollywood clip scored higher for both the parameters followed by the Indian Hindi movie with the animation coming in third. The detailed scores from the fmri and the independent participants for a questionnaire is appended at the end of the chapter (appendix). 4.2 fmri imaging and preprocessing A 3T Philips Achieva scanner set to the following configuration: Gradient echo, echo-planar images TR=2 s, TE=35 ms, flip angle=90, acquisition matrix=64 64, slice thickness=5 mm, gap=1 mm, 30 transverse slices, REC voxel MPS: 1.8 X 1.8 X 5 mm, and acquisition voxel MPS: 3.5 x 3.5 x 5.0mm, was used to record functional images. A three-dimensional T1 weighted structural image using a fast field echo (FFE) technique and a Turbo Field Echo sequence was recorded for a 95

97 duration of 4 minutes 46 seconds with a TR=8.39ms, TE=3.7 ms, 150 slices, flip angle of 8, Field of view (FOV) = 250 x 230 mm and acquired voxel volume: 0.99 x 1.12 x The number of images collected for each stimulus is as follows: Animation movie clip: 130 scans, Hollywood movie clip: 250 scans and Indian Hindi movie 130 scans Data analysis The data was preprocessed using SPM8 (Wellcome Department of Cognitive Neurology, UK (1)). The functional images were realigned to the first scan to correct for the head movement between scans. The anatomical image was coregistered with the mean functional image produced during the process of realignment. All images were normalized to a 2 x 2 x 2 mm 3 Montreal Neurological Institute (MNI) template. Functional images were spatially smoothed using a Gaussian filter with a full-width at half maximum (FWHM) parameter set to 6 mm. The ICA method follows a purely data-driven approach and hence does not require temporal signals to convolve with the hemodynamic response or the need to specify regions of interest. The ICA method has found importance in event-free or task-free experimental designs to identify functional networks. Kim et al., (2009) gives an overview of the group ICA method for fmri data highlighting how this method can be applied to make group inferences. In our study, the smoothed images from SPM8 tool were analyzed using the group ICA tool, GIFT (Mind Research Institute (2)). The minimum description length (MDL) criterion (Li et al, 2007) was opted to reduce the images to be processed. Thirty ICs were extracted by this method and to check for the consistency of the estimates from the iterative algorithm FastICA (Hyvärinen & Oja 1999) the ICASSO visualization toolbox provided in the GIFT software was used. This process is applied to evaluate the similarities for multiple runs of the FastICA algorithm (Correa, et al., 2007, Himberg & Hyvärinen, 2003, Himberg, et al., 2004). A one-sample T-test for each IC for the group of subjects was calculated and activations with a statistical 96

98 threshold of p< with voxel extent of 10 were considered. A check on correction for multiple comparisons was also done using the Family Wise Error (FWE) feature in SPM8, with p threshold set at 0.05 and voxel extent of 10. The severe thresholding of FWE filters out areas which could be relevant to the stimuli, however activation of the main areas is comparable to those obtained from the analysis using uncorrected p-value. Talairach Daemon (3) (Lancaster et al., 1997, 2000) was used to label the Brodmann areas and regions, after conversion of the MNI coordinates from SPM tool into Talairach coordinates using the approximation proposed icbm2tal(4) based on the work by Lancaster et al.(2007). The tables reported in this work show the MNI coordinates and Brodmann areas estimated from the converted coordinates. 4.3 Results Assuming complex processing while viewing natural scenes as in a movie, we expect simultaneous activation in more than one region of the brain. For selecting ICs, three methods were used, one by visual inspection of the time course signals of each component and a second rigorous method of inspecting active regions for all the 30 components as labeled by the Talairach Daemon. The former method filtered out ICs with high frequency signals which indicate physiological artifacts like breathing. In the latter method, the independent components with large voxel clusters labeled as activity predominantly in the white matter areas were filtered out. A third criterion applied was to consider ICs with temporally coherent and spatially separated areas rather than ICs with activations in a single region. The filtering steps narrowed down the final number of ICs to 8-12 with temporally coherent responses in the prefrontal cortex, thalamus and certain areas of the parietal and temporal region. The selected independent components for each stimulus are presented individually and the probable empathy networks observed in the data are presented in the discussion section. 97

99 4.3.1 Animation movie Of the 30 ICs, eleven components with activations in one or more areas consistent with empathy modes reported in the literature were selected (Table 4.1). Sixteen representative axial slices overlaid with responses from the 5 ICs (ICs 1-5 in Table 4.1) are shown in Figure 4.2. Significant responses are present in the prefrontal, temporo-parietal junction and cingulate cortex along with smaller clusters in the thalamus region. The IC depicted in color red (Table 4.1, IC1) represents areas in the prefrontal region (BA 9) and inferior parietal lobule (BA 40), areas pertaining to cognitive empathy network, while the other two areas of activation in this IC, the prefrontal areas (BA 45) and superior temporal gyrus (BA 22) correspond to motor empathy network. The activity in the superior frontal gyrus (BA 8) of the frontal eye field region possibly from eye movements is shown in blue (IC 2) along with activity in angular gyrus (BA 39). Superior frontal gyrus (BA 10) and Precuneus (BA 7) are observed in IC 3 (green color). The purple colored cluster (IC 4) indicates a network of the dorsal pathway connecting visual cortex (BA 18) to the prefrontal region (BA 10) through the inferior parietal lobule (BA 40), primary somataosensory cortex (BA 2) and the pulvinar of the thalamus. Similar pathways were reported in a study examining the where/how pathway of the visual attention network (Itti & Koch, 2001). Notable activations forming IC 5 (color- mustard) include the angular gyrus (BA 39), precuneus (BA 31), precentral gyrus (BA 6) and superior frontal gyrus (BA 8). The Brodmann area 10 was reported by Shamay-Tsoory et al (2009) to be crucial for cognitive empathy, while BA 39 and BA 40 have been reported for both cognitive and motor empathy networks. The precentral gyrus (BA 6) identified in IC 5 has been studied by researchers of the mirror-neuron system and has been included as crucial area for motor empathy. Additionally, we found in IC 6, a large cluster of activity in the parahippocampal gyrus (BA 36), an area that has been implicated in scene categorization and recognition (Epstein et al., 1998, Walther et al., 2009) and for episodic encoding (Hasson et.al., 2008b) of natural scenes. 98

100 The areas separated in ICs 7 and 8 include bilateral activity in the precentral gyrus (BA 6), precuneus (BA 7, 31), superior frontal gyrus (BA 9, 46), inferior parietal lobule (BA 40) and the middle temporal gyrus (BA 37). The superior frontal gyrus (9, 10, 46) areas are considered to be part of the cognitive empathy. Areas like the superior frontal gyrus, inferior frontal gyrus and inferior temporal lobe areas are said to form the action-perception network (Decety and Grèzes, 1999). Right middle temporal gyrus (BA 37,BA 39) and left postcentral gyrus (BA 5) form IC 9. The role of the middle temporal gyrus has been not well defined but this area found to be involved in intentional empathy (de Greck et al., 2012). Posterior cingulate (BA 23,24,30) forms part of the posterior default mode network and these areas are observed in IC 10. The fusiform face area located in BA 37, and attributed to face recognition (Sergent et al., 1992; Ghuman et al., 2014) was found in IC

101 Table 4.1: Independent Component Analysis results for the Animation movie clip: The MNI x,y,z coordinates, T values, brain regions and corresponding Brodmann areas, after conversion of the MNI coordinates into Talairach coordinates of the selected independent components for the Animation movie clip showing activations in areas that form part of the empathy networks. IC s Regions Latera lity cluster T value coordinates (x,y,z) 1 Inferior Frontal Gyrus R Middle Frontal Gyrus L Superior Parietal Lobule R Inferior Parietal Lobule R Brodmann area Thalamus L VLN Middle Frontal Gyrus R Fusiform Gyrus R Middle Temporal Gyrus R Middle Temporal Gyrus R Supramarginal Gyrus R Superior Parietal Lobule L Inferior Parietal Lobule L Superior Frontal Gyrus R Angular Gyrus L Cingulate Gyrus R Superior Frontal Gyrus R Precuneus R Postcentral Gyrus L Inferior Parietal Lobule R Thalamus R Pulvinar Lingual Gyrus R Precentral Gyrus R Precentral Gyrus R Middle Frontal Gyrus L Angular Gyrus L Precentral Gyrus L Precentral Gyrus L Precuneus L Posterior Cingulate L Superior Temporal Gyrus R Superior Frontal Gyrus L Postcentral Gyrus L Parahippocampal Gyrus L Paracentral Lobule R Precentral Gyrus L Precuneus L Superior Frontal Gyrus L Precuneus R

102 Precentral Gyrus R Inferior Parietal Lobule L Precuneus L Inferior Frontal Gyrus L Middle Temporal Gyrus R Middle Temporal Gyrus R Postcentral Gyrus L Posterior Cingulate L Posterior Cingulate L Cingulate Gyrus L Precuneus R Fusiform Gyrus L Fusiform Gyrus L Middle Occipital Gyrus L Figure 4.2: Activation map when subjects viewed Animation movie clip: five independent components (IC 1, 2, 3, 4 and 5 in Table 4.1) with activations in the prefrontal, parietal, temporal and premotor indicative of the probable activation of areas related to empathy mode network are shown. Sixteen axial slices are shown. The areas and other details are listed in Table 4.1. The color code is as follows: IC1: red, IC2: blue, IC3: green, IC4: pinkish purple and IC5: mustard. 101

103 4.3.2 Hollywood movie Nine independent components indicating activity in predominantly the grey matter regions are selected (Table 4.2). Sample axial slices of five components (IC 1-5, table 4.2) with activations covering spatially separated temporally coherent areas, probably indicative of empathy response are shown in Figure 4.3(a). The major activations separated in IC 1 (color red, Figure 4.3a) are right hemispheric middle frontal gyrus (BA 46), the superior parietal lobule (BA 7) along with smaller cluster in the medial frontal gyrus (BA 8) extending to BA 6. Bilateral activity in parietal (BA 40), frontal gyrus (BA 8, 9), visual cortex (BA 18), precentral gyrus (BA 6), dorsal and ventral cingulate cortex (BA 23, 24, 31) are the areas isolated in IC 2 (color blue). The areas covered in green (IC 3) are bilateral activations in the cingulate gyrus (BA 24), left inferior frontal gyrus areas (BA 45, 9) and right precentral gyrus (BA 6) extending to inferior prefrontal gyrus (BA 9). Activations in the Brodmann areas (2, 6, 9, 40, 45) indicate possible motor empathy pathway. The component, IC 4 in purple includes bilateral activity in inferior parietal lobule (BA 40), left middle frontal gyrus (BA 46, 9), inferior temporal gyrus (BA 37), small cluster of activation in precentral gyrus (BA 44) and the pulvinar of the thalamus region. Activation in BA 9, 46, and 40 (shown in IC 4) suggest possible cognitive empathy network while BA 44 is reported for emotional empathy (Shamay-Tsoory et. al., 2009) and faces and/or bodies activate the fusiform gyrus (BA 37) area (Bartels and Zeki, 2004). The areas in mustard color (IC 5) show activation areas similar to IC 2 with bilateral activity in the superior and middle temporal gyrus (BA 39), cingulate gyrus (BA 31, 23), left middle and inferior frontal gyri (BA 9, 45) and precentral gyrus (BA 6). Four ICs (6, 7,8 and 9 shown in Table 4.2) with activations in the thalamus region and insula along with other areas are shown in Figure 4.2(b). The areas covered in color red include the medial dorsal nucleus (MDN) and a smaller cluster over the pulvinar of the thalamus region and the areas in blue (IC 7) show activity in the putamen and pulvinar areas. These areas may have a role in processing emotional scenes (Sabatinelli, 2011). The activations 102

104 isolated in IC 8 (green) include high activation for the group in the superior temporal gyrus ( BA 22), an area found to have strong inter-subject correlation in a study using movie as a stimulus (Hasson et.al., 2004). In addition to superior temporal gyrus the IC also shows responses in the postcentral gyrus (BA 40) and smaller clusters covering the middle frontal gyrus (BA 46) and cingulate gyrus (BA 24), a network that was also reported to participate in top-down attention control (Hopfinger et al., 2000). The IC in purple (IC 9) includes activation in the insula reported to be active for pain sensation and perception (Singer, 2006) and the middle prefrontal cortex (BA 9) reported for cognitive empathy. Table 4.2: Independent Component Analysis results for the Hollywood movie clip: Nine of the total 30 ICs separated for the Hollywood movie clip possibly implicated in different empathy modes. The MNI (x,y,z) coordinates, T values, brain regions and corresponding Brodmann areas from MNI to Talairach coordinates of the selected independent components. ICs Regions Laterality cluster T value coordinates (x,y,z) mm Brodmann area 1 Middle Frontal Gyrus R Superior Parietal Lobule R Posterior Cingulate R Inferior Parietal Lobule L Medial Frontal Gyrus R Medial Frontal Gyrus R Inferior Parietal Lobule R Middle Frontal Gyrus R Middle Frontal Gyrus R Middle Frontal Gyrus L Inferior Parietal Lobule L Cuneus R Cuneus L Cingulate Gyrus L Cingulate Gyrus R Cingulate Gyrus L Precentral Gyrus R Medial Frontal Gyrus L Anterior Cingulate R Cingulate Gyrus L Cingulate Gyrus R Inferior Frontal Gyrus L Middle Frontal Gyrus L Inferior Parietal Lobule L Precentral Gyrus R

105 Inferior Frontal Gyrus R Inferior Frontal Gyrus L Postcentral Gyrus R Postcentral Gyrus R Inferior Parietal Lobule L Middle Frontal Gyrus L Inferior Frontal Gyrus L Thalamus R Pulvinar Inferior Parietal Lobule R Superior Parietal Lobule R Inferior Temporal Gyrus L Cingulate Gyrus L Paracentral Lobule L Cuneus R Precentral Gyrus R Cingulate Gyrus L Posterior Cingulate R Superior Temporal Gyrus L Middle Frontal Gyrus L Inferior Frontal Gyrus L Middle Temporal Gyrus R Precentral Gyrus L Lingual Gyrus R Precentral Gyrus R Thalamus R MDN Posterior Cingulate R Thalamus R Pulvinar 7 Postcentral Gyrus R Lentiform Nucleus L Putamen Thalamus R Pulvinar Precuneus R Superior Parietal Lobule R Cingulate Gyrus L Middle Occipital Gyrus R Inferior Frontal Gyrus R Insula L Superior Parietal Lobule L Superior Temporal Gyrus L Postcentral Gyrus R Cingulate Gyrus L Middle Frontal Gyrus R Precuneus L MDN: Medial Dorsal Nucleus. T-Value estimated at p< with voxel cluster size threshold of

106 Figure 4.3: Activation map when subjects viewed Hollywood movie clip. (a) five independent components (IC1, 2, 3, 4 and 5 in Table 4.2) with activations in the prefrontal, parietal, temporal and premotor possibly implicated in empathy mode networks are shown. Sixteen axial slices are shown. The areas and other details are listed in Table 4.2. The color code is as follows: IC1: red, IC2: blue, IC3: green, IC4: pinkish purple and IC5: mustard. (b) Activation map of areas in the limbic lobe when subjects viewed Hollywood movie clip: isolated in four ICs (IC6,7,8,9 Table 4.2) with activity in the thalamus region and insula specifically shown on 12 axial slices. Color code IC6: red, IC7: blue, IC8 : green and IC9: purple Indian Hindi Movie Nine independent components indicating activity in grey matter regions are selected (Table 4.3). Axial slices of the first five components listed in Table 4.3 (IC 1-5) are shown in Figure 4.4(a). The first IC (IC1) shown in color red in the figure indicates grey matter activity in the superior parietal lobule (BA 7) and middle frontal gyrus (BA 8) while other areas in red are white matter. The areas covered in blue (IC 2) represent activity in the right hemisphere frontal gyrus region 105

107 (BA 6, 9, 8), left superior frontal gyrus (BA 10), bilateral inferior parietal lobule (BA 40) and bilateral cingulate gyrus (BA 31, 23, 32). Significant activity in the right and left cingulate gyrus (BA 24) with smaller clusters over inferior parietal lobule (BA 40) and left inferior frontal gyrus (BA 44, 45) are marked in green. The independent component shown in pinkish-purple (IC 4) represent sizeable cluster in the left inferior parietal lobule (BA 40), right precuneus (BA 7) extending to inferior and superior parietal lobule (BA 40, 7) and smaller clusters in the medial frontal gyrus (BA 9, 6). The areas in mustard (IC 5) indicate responses mostly in the left hemisphere region in precentral gyrus (BA 9), medial frontal gyrus (BA 8) and temporal gyrus (BA 22, 21). The five ICs indicating activations in the prefrontal region suggest cognitive (BA 9, 10) and motor (BA 44, 45, 6) empathy modes as per the existing literature. The next four ICs (6,7,8 and 9 shown in Figure 4.4(b)) indicate significant activations in the thalamus along with responses in the fusiform gyrus (IC 6: BA 37) and precentral gyrus (IC 7: BA 4,6). Activation in insula (BA 13) extending from the middle/superior temporal lobe was seen in two ICs (IC 8 and 9) combined, postcentral gyrus (BA 3, 6), cuneus (BA 19) and smaller clusters in the frontal gyrus (BA 8). Insula has been reported to be active for pain sensation and perception (Singer, 2006), and in sensorimotor integration and emotional processing (Cauda, et al., 2011). The somatosensory cortex (BA 3) and premotor (BA 6) observed in ICs 7,8,9 are areas associated with the mirror neuron system and the motor empathy network. 106

108 Table 4.3: Independent Component Analysis results for the Indian Hindi movie clip: 11 ICs out of the 31 from the analysis of the Indian Hindi movie clip showing possible activation of the empathy networks. The MNI (x,y,z) coordinates in mm, T values, brain regions and corresponding Brodmann areas by converting MNI to Talairach coordinates of the selected independent components. ICs Regions Laterality cluster T value coordinates (x,y,z) mm Brodmann area 1 Superior Parietal Lobule R Middle Frontal Gyrus R Precuneus R Superior Frontal Gyrus R Middle Frontal Gyrus R Superior Frontal Gyrus R Cingulate Gyrus R Cingulate Gyrus L Superior Frontal Gyrus L Inferior Parietal Lobule R Inferior Parietal Lobule L Cingulate Gyrus L Cingulate Gyrus R,L Inferior Parietal Lobule R Inferior Frontal Gyrus L ,45 Precentral Gyrus L Inferior Parietal Lobule L Precuneus R Inferior Parietal Lobule R Superior Parietal Lobule R Medial Frontal Gyrus L Middle Frontal Gyrus L Precentral Gyrus L Superior Temporal Gyrus L Inferior Parietal Lobule L Middle Temporal Gyrus L Medial Frontal Gyrus L Postcentral Gyrus R Thalamus - VLN R Fusiform Gyrus R Cuneus L Precentral Gyrus L Precentral Gyrus L Middle Frontal Gyrus R Middle Temporal Gyrus R Insula R Precuneus R Postcentral Gyrus R Precentral Gyrus R

109 Precuneus R Precuneus L Anterior Cingulate R Superior Temporal Gyrus L Cuneus R Middle Frontal Gyrus R Middle Frontal Gyrus R Inferior Parietal Lobule R VLN: ventral lateral nucleus. T-Value estimated at p< with voxel cluster size threshold of 10. Figure 4.4: Activation maps when subjects viewed Hindi movie clip. (a) five independent components (IC1, 2, 3, 4 and 5 in Table 4.3) with activations in the prefrontal, parietal, temporal and premotor indicative of probable empathy mode network areas are shown. Sixteen axial slices are shown. The areas and other details are listed in Table 4.3. The color code is as follows: IC1- red, IC2: blue, IC3: green, IC4: pinkish purple and IC5: mustard. (b): Activation maps of areas specific to the limbic lobe when subjects viewed Hindi movie clip: four ICs (IC 6,7, 8,and 9 in Table 4.3) highlighting the activity in the thalamus and insula, premotor and superior temporal gyrus region are presented... Color code IC6: red, IC7: blue, IC8: green, IC9: purple Comparative Analysis of Uncorrected versus Corrected Results: To correct for multiple comparisons, family-wise error (FWE) correction was applied to the activations and to the T values of the selected ICs. In this chapter, the results we presented so far are of analysis using an uncorrected threshold of p<.0001, with a voxel extent threshold of 10. These thresholds are quite 108

110 conservative and potentially reduce type I errors (false alarms). However, in the results reported in this section we compare these results with those obtained using FWE that corrects for errors due to multiple comparisons. It is to be noted that FWE can lead to type II errors (missing true effects). The results depicted in Figure 4.5 bear out our intuition in using uncorrected but conservative threshold over FWE-I of correction. While the cluster sizes became reduced when family-wise error threshold (p<0.05) was applied, the overall profile of activities is comparable to that obtained for uncorrected threshold of p< Figure 4.5 illustrates activity estimated from both these methods for a single IC taken for each type of stimulus material. FWE correction was implemented using SPM8 on IC1 in Table 4.1 (Animation movie), IC1 in Table 4.2 (Hollywood Movie) and IC 2 in Table 4.3 (Indian Hindi movie) of the results sections. Figure 4.5a: The FWE and the p uncorrected activations shown on glass brain masks for one IC (IC 1 in Table 4.1) of the animation movie data. We have labeled only the larger voxel clusters by their Brodmann areas. As can be inferred from the images, the areas of activations are almost similar, but the cluster sizes are smaller for the FWE analysis. Figure 4.5b: For the Hindi movie data (IC2 from Table 4.3), the FWE correction reveals no activity in the parietal areas (BA 40), whereas the uncorrected mask shows large activity in BA 40. Another area is BA 10 in the left hemisphere which is a larger cluster for the p uncorrected compared to FWE analysis. Figure 4.5c: For the Hollywood movie data (IC1 from Table 4.2), bilateral prefrontal lobe activity covering BA 9, 46 appears to be comparable for both statistical thresholds, while the parietal lobe activity is much more conservatively estimated for FWE though BA 7 and 40 were identified at this threshold as against only BA 7 being active for uncorrected analysis. 109

111 4.4 Discussion The factors that contribute to empathy response from a viewer to the state of the actor or to the event that is being depicted in a movie are many-fold and can vary widely among viewers. To qualify each of the parameters of divergence, focused experiments have been conducted by researchers and the results integrated to deduce the combined effects by using short (20-30 seconds) stimuli or static images with emotional contagion (Adolphs., 2002, Schulte-Rüther et al., 2007, Fusar-Poli et al., 2009). A few studies have reported findings from a single movie shown for long duration where the focus was on studying activation changes in a region or an area of interest (Hasson et al., 2004, 2008). The study reported in this article extends the previous studies using complex stimuli, especially movies, by combining long duration stimuli and a diverse set of movies, to identify areas attributed to empathy response. Considering the diversity of the movie clips selected in this study, the goal was to acquire a macro-level understanding of responses to narrative empathy, a condition that involves the sharing of feeling and perspective-taking brought on by activities like reading, viewing, hearing, or imagining narratives of another s situation and condition (Keen, 2006). The activation obtained is categorized as belonging to various empathy modes based on the assumption that the responses from viewer could be a mix of sharing the actor s emotional state (motor and emotional empathy), taking perspective of the other person(s) and the events while simultaneously being able to differentiate the self-versus-other experiences. The comparative ratings collected from participants on macro features such as expressions and context of the clips were also used as reference (Appendix). Group ICA method was applied as the experiment was a passive viewing non-event based design and because the primary interest was to identify functional maps that evolve over time. The analysis focuses on the presence of activity evolving over a longer event rather than sudden response to a specific event. In addition to activity corresponding to a selected set of 8-12 independent components (ICs), the method also isolated spatial maps or ICs with 110

112 activations in one region of the brain, like the visual cortex or mid-brain. Though the primary focus was on highlighting areas for empathy modes, distinct pathways like the action-perception network or individual areas like the fusiform gyrus are also mentioned. To check for false alarms (type I errors), a conservative statistical threshold (pvalue) of with a voxel extent of 10 was applied. To correct for multiple comparisons, the results using family wise error (FWE) method provided in SPM8 on selected ICs were cross-checked. As shown in the Figure 4.5 when a conservative threshold enforced by FWE was applied, though the extent of activation reduced in all conditions, the areas of activity did not differ from that observed with p< The result for each stimulus is discussed individually in the sub-sections below Animation Movie For the animation movie (Figure 4.2 and Table 4.1), we did not detect considerable activity in the thalamus region, except in one IC where pulvinar was isolated in a network connecting precentral, frontal and the parietal cortex while activation in the insula area was not identified in any of the ICs. Both these areas are crucial for emotion processing and in pain-empathy studies (Fan et al., 2011, Singer et al., 2004, 2006). Another area, the ventrolateral prefrontal area (BA 44) also considered to be critical for emotion processing (Shamay-Tsoory et al., 2009) was also not detected. The findings at the level of neural correlates are in accord with the lower ratings on the emotional expressions and context of this clip. The areas associated with cognitive empathy such as BA 38, 32, and 46 were not identified too, while activation in BA 9,10 was significant. Comparative studies on cartoon versus live action (Han et al., 2005, Mar et al., 2007) for perception of action report higher activation in the prefrontal gyrus (BA 9, 10) for live-action, but we noticed comparable activation in these areas for the animation clip. The data suggests that for animation movie, cognitive and motor empathy networks are 111

113 more evident than the emotional empathy network. The findings point to the possibility that viewer, while being able to take perspective of the situation of the actor(s), probably falls short of sharing the experiences as he/she distinguishes the actors as computer-generated agents. Considering that the narrative is about events similar to real life situation the absence of activation in some of the crucial emotion processing areas could only be attributed to the viewers perception to animated agents Hollywood movie The Hollywood movie scored high for both emotional expressions as depicted by the actors and of the narrative which seems to be reflected in the areas of activation of relevance to the empathy modes (as depicted in Figure 4.3 and Table 4.2). Considerable activity was observed in the medial dorsal nucleus (MDN) of the thalamus reported to be central for processing highly emotional scenes (Sabatinelli et al., 2011). Additionally, activation in the pulvinar, insula, superior temporal gyrus, and BA 44 indicate emotional empathy response. The somatosensory (BA 3) and thalamus network was also reported as part of an emotional response network (Nummenmaa et. al., 2008). We also notice evidence for functional networks associated with cognitive (BA 9, 10, 46, 39, 40) and motor (BA 3, 4, 6, 22, 45) empathy modes. The activity in the emotional empathy network point to a likelihood that movies is not influenced by similarity bias either from ethnicity or cultural identity, though this needs to be further explored with multiple ethnic groups and movies Hindi Movie The five ICs (IC 1-5 in Table 4.3 and Figure 4.4) indicate temporally coherent networks comprising areas from cognitive and motor empathy modes, covering the prefrontal, parietal and the superior temporal gyrus of temporal cortex. The 112

114 middle frontal gyrus (BA 46) activity was not identified as in the animation movie. Distinct areas (ventral lateral nucleus, ventral posterior nucleus, Pulvinar) in the thalamus and a larger cluster of activity in the insula (IC 9) compared to the Hollywood movie are noticeable. These active areas along with response in BA 44 and the superior temporal gyrus (BA 22) suggest a possible emotional empathy network. The observed areas indicate the presence of all the three empathy networks, as was the case for the Hollywood movie. Considering that many scenes with emotional expressions by the actors are depicted and the narrative is high on emotion, the lower rating given in the survey as compared to the Hollywood movie could be due to the fact that the viewers were familiar with the story and hence were cognizant of the happy ending (which was not shown ). 4.5 Summary In concurrence with the ratings provided by participants for emotional context and expressions, absence of activity in critical areas like the insula and BA 44 in the animation movie suggests a possible lower emotional empathy response to computer-generated animation characters, even though the narrative was of an common real-life event. Comparing the response in these two areas for the Hollywood and Indian movies, the activity in the insula is higher in the latter converse to the ratings given by participants, while activity in the medial dorsal nucleus (MDN) reported to be crucial for emotion scene processing was present only for the Hollywood movie stimulus. The activity of MDN for Hollywood movie is supported by the rating for emotional context. The cognitive empathy areas, especially the prefrontal region (BA 9, 10) was comparable for all the three stimuli. The dorsolateral prefrontal area (BA 46), a region attributed to emotion regulation, cognitive empathy and also proposed to be involved in higher order cognitive processing as studied by Hasson et al.(2008c) from inter-subject correlations using a movie clip in a free-viewing experiment, was not observed in the Indian Hindi movie. A very small voxel cluster in the BA 46 area for the 113

115 animation movie is noticed, while the response was significant for the Hollywood movie. Activation in the ventrolateral prefrontal gyrus (BA 47) was not observed for all the three stimuli. These two prefrontal areas are associated with both cognitive and emotional empathy networks and reported to be active also in a compassion eliciting experiment (Mercadillo et al., 2011). Though our experimental design constrains any definitive conclusions, the higher activation in BA 46 for the Hollywood movie could be in response to the scene where an explicit electric-chair execution is shown, which could have triggered compassion and/or emotional empathy from the viewers. The activity in areas, BA 39, 40, 22, aligned to cognitive and motor empathy was common to all the three clips though voxel activation cluster size of BA 22 was lesser for the animation movie. These areas are also mentioned for social perception resulting from processing of events, human interactions and narrative (Hasson et al., 2004, 2008a, b). The smaller voxel cluster size of BA 22 suggests a possibility of a differential social perception for computer generated character scenes compared to those with human actors. Motor empathy activation which includes the mirror neuron network (BA 3, 4, 6) was comparable for all the stimuli indicating an absence of bias in action perception and corresponding response to computer generated actors. This requires focused experiments on the motor areas as action-perception studies (Mar et al., 2007) indicated differences in the prefrontal, including the precentral gyrus and temporal areas for computer generated versus live-action sequences. In addition, identifying functionally relevant networks the ICA method allows for making inferences on the temporal coherence of the areas isolated in individual spatial maps. For example, BA 22, which is attributed to empathy process, was isolated in more than one IC for the same stimulus. An inference that can be drawn could be the multiplicity of the role of BA 22 it assumes is based on the 114

116 connections it forms in a particular network of areas active at that instant. Similarly, BA 6 (premotor cortex) is isolated in many ICs sharing a time course of activity with different set of areas, suggestive of an overlap in functional role of this area in all the three empathy modes. Our work adds to the growing application of group ICA method for non-resting state paradigms like multi-modal complex stimuli. The group averaged time series signals associated with each spatial map or independent component can be analyzed to make assumptions on the nature of the stimulus. For this study, we restricted analysis to characterize responses at the population-level than investigating between-subject variations or individual subject responses to events. Characterizing inter-subject variations requires controlled stimuli and genesis of the subject-to-subject variations. 4.6 Conclusions The exploratory study on a diverse set of movies, pre-rated for emotional context, yielded evidence of empathy networks using the ICA method. The classification of the areas into empathy modes or networks was based on previous literature to identify areas or networks associated for each of the empathy modes. The results of the Hollywood and Indian Hindi movie give us interesting starting points to study the role of narrative and cinema as a medium to influence perception of racial and cultural bias in empathy response. Narratives exert a powerful influence over our understanding of ourselves and other living beings. They change our values and passions thus effecting the tacit social contracts we maintain with the world around us. A narrative with a positive message on emotional trauma of fellowhuman being in Africa can induce empathy in another human of Indian race, conditional to the context. Though in studies on pain empathy it was found that we recognize emotions of people with whom we share a culture or ethnicity or relationship. Studies have also shown that perspective taking or understanding of 115

117 thoughts of another is moderated by culture values, Cheon et al, (2011) found that Asians exhibited higher perspective, while Cheng et al (2010) report that it is higher than for a loved one. Zuo and Han (2013) report that racial bias reduces by reallife experiences with other-race individuals and thus lessens bias for empathy response. It has also been shown in a pain-empathy study by Contreras-Huerta et al. (2014) that automatic bias towards own-race and a later top-down cognitive evaluation does not differentiate between races. With forced attention to other s feelings and inclusion of the other in social-group, a decrease racial-bias in empathy response (Sheng & Han, 2012) can be observed. But of specific relevance is the compelling argument that Harrison (2011) makes by proposing that act of reading can overcome the bias observed in relationship between people and that narrative empathy has the potential to prevent future cases of bias. The absence of crucial emotion processing areas for the animation clip hints at a possible difference when perceiving computer generated actor(s), which opens avenues for studying genre of computer-generated animations, avatars and role of narrative in animation movies. Also reflected by the lower rating on empathy scale provided by fmri and independent participants for the animation movie. The conclusions on empathy responses in correlation to the stimulus are tentative at best and require controlled experiments to make reliable inferences. Further research is planned to understand in depth the role of narrative in empathy response for different formats of representation (2D, 3D), especially in different genre of animation movies. 4.7 Limitations and future directions The experiment design imposes a few constraints for strong inferences. One of the constraints being the use of diverse set of movie clips of varying levels of depiction 116

118 of emotion. Movies with fairly equal doses of emotional depictions could have facilitated stronger generalizations while comparing the emotional responses to the real-actor movies versus computer generated movies, in particular. The presentation of the movies clips were not-counterbalanced, thereby limiting the power of statistical inference. In analysis, correlations based on time courses of a set of components with areas forming empathy modes would help understand sequential responses between networks, a factor not considered in depth in this study. 117

119 Tools: (1) SPM8 ( (2) Group ICA ( (3) Talairach Daemon Appendix A1: Nature of Stimulus Material: Animation Movie : The movie Up (2009) is produced by the Pixar Animation Studio. The clip used as stimulus shows the introductory part where the elderly widower, Carl Fredricksen gets married to Ellie who also idolizes renowned explorer Charles F. Muntz. The visual narrative conveys the couple s inability to bear children which they dream and plan for. They move on with life and start saving for a dream trip to paradise falls. The passing of time is shown with a sequence of events wherein the savings are spent on fixing ordinary expenses like flat tires of the car to repairing the roof. They grow old and Carl suddenly remembers Ellie s desire for the trip to paradise falls. He pools his savings and buys the tickets. He takes Ellie to a picnic spot up a hill to surprise her with the tickets, but Ellie collapses on the climb up and is shown hospitalized. She urges Carl to go on with his dream and passes away. The events with emotional contagion are when the couple realizes that they cannot have children, subtly depicted when each saving is spent on unexpected expense and as they are shown growing old and the final scene when Ellie dies. Indian Hindi Movie: Released in 2007, the movie Taare Zameen par (literal meaning: stars on the earth) centers around a 10-year old dyslexic child, Ishaan, who is unable to cope with regular school. The clip shown starts with a scene where the father of Ishaan is angry with him for missing school and loitering alone on the streets of Mumbai. Due to poor school performance, the school calls the parents 118

120 for counseling, after which the father decides to send Ishaan to a boarding school. Ishaan is heartbroken and pleads with his parents. His mother is caught in between deciding what is good for Ishaan in the long run and sorrow at the thought of letting him go away and expresses helplessness. The next sequence shows Ishaan confused about his state and dreams of being torn away from his mother in a railway station. The clip ends with Ishaan being dropped off at a boarding school and the parents heartbroken but helpless. In this movie, there are scenes which are very emotional, while not projecting any of the actors, the father especially, as a villain. Hollywood Movie: The Green Mile, released in 1999 is about Duncan, a giant black man convicted of raping and killing two young white girls condemned to be executed on the electric chair. The scene starts off with Duncan being taken from his jail-cell by guards, who feel sorry for him. He is made to sit on the electric chair and the electrodes are placed on him as the guard s cry from helplessness and sorrow at seeing a good innocent man and a friend being put to death by the system. Duncan shakes hand with the chief (played by Tom Hanks) and the electric shock switch is turned on. The last scene shows Tom Hanks placing a chain that Duncan gives him on his dead body. The events are interesting as one understands that the guards are sad and helpless, while Duncan though resigned to his fate is afraid of the dark rather than death itself. 119

121 Rating Post fmri scanning participant survey. The questionnaire is a self-report on the negative emotion depicted by the actors, the events in the narrative and self-experienced (in response to the state of the other ) On a semantic scale of 1-5, with 1 least emotional to 5 being the maximum. 1. Overall Emotional valance of the movie clip Green Mile : Taare Zameen pe : Up : Emotional Scene intensity Green Mile : Taare Zameen pe : Up : Depiction of emotions by the actors/characters in the movie Green Mile : Taare Zameen pe : Up : Self-experienced response Green Mile : Taare Zameen pe : Up : In the independent survey conducted with 40 participants, questions 2,3,4 were marked after a 90 second clip of the movie stimuli. Four sets of rating for each movie were analyzed. 120

122 Figure 4A: The top plot is the independent rating from 40 participants for 4 scenes. The below histogram plot is the average rating from the fmri participants post-scanning. 121

123 Chapter 5 Dynamic functional network connectivity of empathy and default mode networks* 5.0 Introduction In the data-driven method of analysis of the fmri data collected from three movie clips, presented in Chapter 4, the intrinsic empathy networks were identified. The empathy networks thus identified provided the insights into the dynamics of the spatially segregated but temporally coherent networks. Considering that the stimuli were presented continuously for 5-8 minutes, and the events evolved, the next aim was to examine the time-varying activations in the networks. From the time courses of the selected independent components (segregated by Independent component analysis method) specific to empathy areas and seed network as the posterior default-mode network (DMN), cross-correlation was estimated to examine the competitive relationships between DMN and the empathy networks. The findings are significant for understanding stimulus relevant time varying functional network connectivity in the identified empathy networks and the relation to task derived DMN. Temporal coherence between widely distributed active regions of the brain provides insights into anatomical and functional connectivity (Friston, 1994). * Vemuri, K., & Surampudi, B. R. (2015). An exploratory investigation of functional network connectivity of empathy and default mode networks in a free-viewing task. Brain connectivity, 5(6),

124 Estimation of the cross-correlations for windowed time series signals of activations collected using moving images like movies allows for investigation of the dynamic functional connectivity between the isolated brain areas or networks. Narratives and empathy have been covered in detail in chapters 1,2 and hence a very short note in included to set the context of the reported study. A movie is an interesting experience as it integrates the complexity of natural scenes of a sequence of events wrapped into a narrative in a compressed time, permitting the viewer to evolve multiple thought processes. Narratives in movies are designed to manipulate the viewer s feelings even if temporarily, to take others (in this report the other/actor implies the character or role played by the actor) perspective. The sequence of events depicting temporal developments that engage the viewers but also promise an end is defined as narrative empathy (Keen, 2007). The viewer s reactions to the events in a movie as in real life situations are subjective and a function of the ability to infer others emotional state from the context or from facial expressions. The relationship that the viewer builds with the actor(s) could transcend from emotional ( I feel what you feel ) to cognitive ( I understand your feelings ) modes of empathy (Decety & Jackson, 2004, review: Walter, 2012). Investigation into the neural activation of a single region or multiple regions like the visual attention network using a movie paradigm have been extensively studied (Hasson et al., 2004; Bartels and Zeki, 2005; Levesque et al., 2003; Mantini et.al 2013; Betti et al., 2013). Responses to emotional events as presented in very short-clips from movies has also been studied and networks specific to a multitude of emotions identified (Goldin et al. 2005; Nummenmaa et al., 2012). The empathic feelings and engagement can vacillate in a narrative and is a slowly evolving empathic response and hence dynamic functional connectivity analysis will give insights into probable synergies between empathy modes. 123

125 The mechanisms involved in empathy processes are reported to be top-down from perspective or cognitive processing and bottom-up inputs from sensory systems, translating to distinct emotional and cognitive processes (Adams, 2001; Preston & de Waal, 2002; Decety & Jackson, 2004). Hence, it can be safely assumed that an emotional narrative, either a real-life or a fictional event, by its complex content can trigger empathy comprising emotional responses, selfreflections, theory-of-mind (ToM) and cognitive reasoning (Premack & Woodruff, 1978; Frith & Frith, 2003; Decety & Lamm, 2006; Schnell, et al., 2011). The default mode network (DMN) covering the middle temporal lobe, posterior/anterior cingulate cortex, tempo-parietal junction and medial prefrontal cortex was originally reported in resting-state experiments and was shown to become deactivated during a goal oriented task (Vincent et al., 2008; Buckner et al.,2008; Mazoyer et al., 2001; Raichle et al., 2001). Later studies suggest coactivation of the DMN for cognitive tasks like projecting into the minds of others or ToM, self-reflections, for empathy response, inferring belief and intentions, autobiographical and prospective memory or while engaged in mind-wandering in resting-state condition (Sestieri et al., 2011; Buckner et al., 2008; Buckner and Carroll, 2007; Laird et al., 2011). The presence of the DMN for less challenging cognitive tasks (Greicius & Menon, 2004) and dynamic suppression or deactivation in these regions for focused performance in high-demand tasks (Mazoyer et al. 2001; Kelly et al., 2008; Weissman et al. 2006; Mayer et al., 2010; Tomasi et al., 2006) provides the basis for this study where we investigate the dynamic functional network connectivity (dfnc) of an identified DMN to that of the empathy networks. Different analytical methods have been applied to study the intrinsic functional architecture that supports the interaction between two or more active regions or between networks. Rubinov & Sporns (2010) applied graph theory methods to 124

126 characterize functional networks where the fmri time series of the functionally relevant area is a node and the edge is the time-series correlation between two such nodes. As considering only pre-selected areas might fail to reveal larger networks that are engaged in most cognitive tasks, the Bayes net model rectifies this by estimating on all the nodes simultaneously. A review of the various methods and advantages of each is presented by Smith et al., (2011). The ICA method is a blind source separation technique which provides statistically independent component (IC) networks from fmri data (Biswal and Ulmer, 1999; Calhoun, 2001), and has been particularly applied to experiments that are non-epoch-based experimental designs (explained in Chapter 3). This method separates the mixed source signal from each voxel into spatially independent source signals and the regions with temporally coherent source signals constitute independent components or intrinsic networks(mckeown et al., 1998; Calhoun et al., 2012). Estimate of the pair-wise cross-correlation on the time course data of the ICs provides functional network connectivity (FNC) (Jafri et al., 2008). The question is: If the ICA method un-mixes components on the principle of temporalindependence, what would correlation analysis between time courses of two components suggest? There is a possibility that there exists a weak temporal dependency between components and hence analysis of the correlation between component time courses be used to evaluate group differences (Jafri et al., 2008). Second, there is a functional overlap of the areas in more than one component sharing temporal coherence with other active regions and it is of interest to understand the role of these areas in different networks. A third aspect is the identification of networks that modulate the connectivity between two networks (Buchel and Friston, 2000). Hence, dynamic functional connectivity can identify communication between networks and examining the fluctuations in correlations gives insights into the networks engaged in complex processing required for dynamic multi-modal stimuli, such as movies. 125

127 Dynamic connectivity by calculating the cross-correlations, has been applied to study spontaneous fluctuations during resting-state (Allen et al., 2014; Arbabshirani et al., 2013) by using a sliding-window method for comparing healthy controls and schizophrenic patients (Jafri et al., 2008; Yuan et al., 2012) and extended to cognitive task (Assaf et al., 2009; Sakoğlu et al., 2010; Fox et al., 2006; Coste et al., 2011). Working with patients diagnosed with somatoform pain disorder and healthy controls, Otti et al. (2013) looked at resting-state functional network connectivity in pain-related networks like the default mode, cingular-insular and the sensorimotor networks and reported significant FNC s between these networks but no group differences. The FNC from the time courses of DMN and the ventral and dorsal attention networks (Corbetta et al., 2002) isolated by the ICA method studied during resting-state was found to show modulation during taskperformance (Smith et al., 2009; Calhoun et al., 2008). Correlation comparison between active regions with known anatomical connections and unconnected active areas showed higher values for the anatomical networks while viewing natural scenes (Bartels and Zeki, 2005) demonstrating the relevance of free-viewing long duration stimuli to understand relationship of the functional and anatomical networks. These studies offer evidence that a) complex stimuli like movies with emotional content can reveal many networks of spatially segregated active areas forming the DMN and stimulus related responses like empathy networks and b) exploring the cross-correlations from the time courses of the identified ICs provides insights into time varying functional connectivity of response to stimulus. In this study, we identify putative empathy networks from fmri data analyzed by group ICA method of three diverse movies, rated for emotional contagion and empathy eliciting context a popular animation movie (S1: Up ) clipping, an Indian Hindi movie (S2: Taare Zameen par ) and Hollywood movie (S3: Green Mile ). We use dynamic functional network connectivity analysis methodology on 126

128 time courses of the independent components estimated by the group independent component analysis (ICA) method. The analysis is expected to provide insights into the time-varying functional connectivity between the networks in response to the narrative. The hypothesis of the study is that empathy modes in response to a narrative would have differential modulation with the default mode network (DMN). 5.1 Methodology Subjects Functional magnetic resonance imaging (fmri) data was collected from fifteen healthy multi-lingual college-going adults (age range of years, females: 5 and males: 10). The data from one of the subjects was rejected due to excessive motion artifacts. Three movie clippings of approximately of 4 minutes each were used as stimuli. The story-line of the events shown in the clips is presented in Chapter 4 (appendix). These clippings were additionally rated independently by 40 participants for emotion depiction, context and self-emotion response at four timeintervals (appendix, Chapter 4). In a post-interview and survey the fmri subjects rated S3 highest for all parameters, followed by S2 and then S1. It should be noted that the fmri subjects and majority of the independent survey participants were familiar with the Indian movie (S2). The order of the presentation was not counterbalanced across subjects but the familiar Hindi movie was second in the order, preceded and followed by unfamiliar movies. Resting-state data was also recorded for 4 minutes 46 seconds before the presentation of the stimulus, but is not considered for analysis in this study. The ethics committee of the International Institute of Information Technology, Hyderabad had approved the study and all the required informed consents were taken from the participants of the study. 127

129 5.1.2 fmri image acquisition fmri data was collected from a 3T Philips Achieva scanner, using gradient echo, echo-planar images, with a TR of 2 s, TE=35 ms, flip angle=90, acquisition matrix=64 64, slice thickness of 5 mm at a gap=1 mm resulting in 30 transverse slices at an acquisition voxel density of 3.59 x 3.59 x 5 mm, reconstructed to 1.8 x 1.8 x 5mm. Three-dimensional T1-weighted structural image was acquired using a fast field echo (FFE) technique and a Turbo Field Echo sequence, run for a duration of 4 minutes 46 seconds with a TR of 8.39ms, TE=3.7ms giving 150 slices at a flip angle of 8, Field of view (FOV) = 250 x 230 mm and acquisition voxel size of 0.99 x 1.12 x 2.0mm. Due to variation in the length of the movie clippings, the number of images collected for each stimulus was different. While 130 scans each were recorded for S1 and S2, 250 scans were acquired for S Preprocessing and data analysis The preprocessing steps of realignment, co-registration, normalization and smoothing were done using SPM8 ( The functional images (T2*) were realigned to the first scan to correct for any head movement during the scan duration. The anatomical image (T1) was co-registered with the mean functional image, output from the realignment step. The images were normalized to a 2 x 2 x 2 mm 3 Montreal Neurological Institute (MNI) template and the spatial smoothing was done with a Gaussian filter (with a full-width at half maximum parameter set to 6 mm). Group independent component analysis using the GIFT tool ( was performed over the entire group of subjects, wherein the preprocessed data from all the subjects is concatenated and the aggregated data set is reduced to temporally coherent independent components (Calhoun et al., 2001). The interactive FastICA (Hyvärinen and Oja, 1997) and ICASSO provided in the GIFT toolbox were applied to check for consistency of the estimates. Individual subject time courses 128

130 and spatial maps are back- reconstructed (Calhoun et al., 2001). To determine the number of components the minimum description length criteria (Li et al, 2007) provided in the tool was applied. The number of scans recorded for the Hollywood clipping was reduced to 130 from 230, to enable group dynamic FNC analysis and uniform sorting of the ICs. A careful deletion of scans from the Hollywood movie (Stimulus S3) was done to ensure that data of the brain response to scenes of high emotional valence was not eliminated. Twenty-eight ICs for the group were extracted using this method. Group statistics on each IC across all the subjects to test for significance from zero was done by a one-sample T test (p<0.001). The regions of activation and the Brodmann area labels was output from the Talariach Daemon tool ( after conversion of the MNI coordinates into Talairach space following the method proposed by Lancaster et al. (2007). Of primary interest are ICs indicating the default mode network (DMN) and ICs with areas reported as comprising empathy networks Selection of ICs The ICs were selected using three parallel methods, the first being visual inspection for artifacts, second process looks at correlation with white matter, grey matter, cerebral spinal fluid (CSF) and DMN template available in the GIFT templates. The third method was to acquire labels of all the areas of activation for each IC using the Talariach Daemon (3) with a 5mm cube region of interest. The ICs with voxel clusters showing maximum white or CSF region activation were rejected. The ICs were spatially correlated with the 4 defined templates and selected based on highest correlation (grey matter and DMN) and lowest correlation criterion (White matter and CSF) an approach which has been reported to be consistent (Garrity et al., 2007; Harrison et al., 2008). The templates used in this study are provided in the GIFT toolbox. In addition, the ICs were subjected to multiple regressions analysis with respect to the self-reported rating on empathy collected at 4 time-intervals for each movie clipping, with the rating as the 129

131 regressor. For two ICs (IC13, IC15) of SI, there was significance at p-value of <0.05, while for all other ICs in the three stimuli the regression coefficients were not significant. Consequently, one IC indicating posterior DMN and four ICs with areas reported for empathy modes with conclusive identification from the significant activation in areas attributed to the three networks by examining all the 28 ICs isolated by the ICA method Dynamic functional network connectivity analysis The time courses of the selected ICs were pair-wise cross- correlated to arrive at the dynamic functional network connectivity (dfnc). The approach (Allen et al., 2014) involves subtraction of remnant noise by linear de-trending, replacing any outliers with best estimate from a third-order spline fit from the smoother part of the time course. Next step is filtering using a high-frequency cutoff at 0.15Hz as previous work (Sun et al., 2004) has shown that functional association between the low-frequency component of the fmri data is task-related while high frequency (>0.2Hz) is usually attributed to noise from artifacts and hence show no functional associations. A covariance matrix is generated by normalizing the variance of the time courses. The dynamic covariance was calculated by a Gaussian sliding window procedure, with a window width of 30TRs or 60seconds, as 64s was suggested being optimal by Sakoğlu et al., (2010). The Gaussian window alpha value was 3TRs (6 seconds) and a sliding step size of 2 TRs or 4 seconds was applied. The number of windows is the window size subtracted from the number of time points (TRs), which translates to 100 windows in our case. For this study, we consider FNC time points only for the pairs of ICs of interest to test the hypothesis. The dfnc tool is provided in the GIFT toolbox and the methodology is reported by Allen et al. (2014). The correlation coefficients were Fisher z- transformed and average at each time point across all the subjects was calculated. Figure 5.1 is a schematic of the complete process followed in this study. 130

132 Figure 5.1: A block diagram representation of the analysis steps. GM: grey matter, WM: white matter, CSF: cerebral spinal fluid, TCs: time courses. 5.2 Results We present dynamic functional network connectivity analysis applied on the time courses of the IC corresponding to DMN and putative empathy networks and examined the time-varying pair-wise cross-correlation. Empathy response to specific events depicted in the movies collected from the fmri subjects post scanning and through an independent survey rating on emotional self-response at four intervals of each clipping supports, a) the ratings show that the movie clippings evoked emotional response and b) the response varied across the narrative. The variation in the ratings suggests that attributing the activation seen in related areas in the spatial maps as empathy modes is reasonable. The correlation with DMN is of particular interest as it is found to be deactivated and shows very low signal changes for attentional and cognitive tasks (Mazoyer et al., 2001). For S1 (Animation movie) one IC with areas covering the posterior DMN and four ICs with networks attributable to cognitive empathy were identified. Of the similar number of ICs selected for S2 (Indian Hindi movie) and S3 (Hollywood movie) it was possible to classify an IC with activation in the bilateral posterior cingulate 131

133 implying a posterior DMN, an IC as emotional empathy network, one IC with areas attributed to nearly all the empathy modes and 2 ICs with cognitive empathy network. Additional observation was the components selected for S3 had significantly higher number of active areas, covering prefrontal, premotor, temporal, thalamus and visual cortex regions Animation Movie (S1) Table 5.1 lists the ICs selected for analysis from the animation movie data and Figure 5.2a presents the spatial maps along with the Fisher z-transformed average correlation (Figure 5.2b) over the stimulus presentation duration, while figure 5.2c is the survey rating plot. The fmri participants comparative rating on emotional response to S1 was low, while the scene-wise independent survey shows a sudden increase in emotional feelings to the last scenes (Figure 5.2c). Inspection of the ICs does not reveal activation in the crucial emotion processing areas like insula and temporal lobe (BA 22). The independent components (IC13, IC15, IC19, IC25) with significant activation in the frontal cortex (BA 9,10,46), parietal (BA 40) and cingulate gyrus (BA 32,24) suggests a cognitive empathy network and presence of pre-post central gyrus in IC13 suggests a motor empathy. The IC26 with posterior and anterior cingulate showed highest correlation to the defined reference DMN template (provided with the ICA toolbox) and hence can be considered as a posterior DM network. The average cross-correlation values for the DMN and IC25 are higher and positive with a periodic oscillation, while plots of the other three ICs (IC13,15,19) reveal lower variance and exhibit positive and negative correlations over the total scan period. The confidence interval at 90% estimated for each of the pairs (figure in Annexure) show that for IC13 and IC15 the correlation being equally spaced between the lower and upper interval bounds while for IC19 and IC25, the variability seems much larger. 132

134 Table 5.1: The ICs indicating DMN and empathy networks for the Animation movie (S1) (T value at p<0.001, FDR corrected < 0.05 and voxel threshold: 10). The MNI coordinates were transformed into Talairach space and Brodmann area labels extracted from the Talariach Daemon. The last column is the suggested putative empathy mode for each IC. IC Cluster Size T- Value MNI Coordinates (x,y,z in mm) Lobe Laterality/Region Brodmann Area Putative Empathy network Limbic R/CG 32 CE/ME Limbic L/CG Frontal R/MFG Frontal R/SFG Occipital R/Lingual Gyrus Frontal R/Precentral Gyrus 6 L/Supramarginal Parietal Gyrus Frontal L/Precentral Gyrus Parietal R/IPL 40 R/Postcentral Parietal Gyrus Frontal L/MFG Frontal L/CG Sublobar R/Lentiform Nucleus (Putamen) CE Frontal L/MFG Occipital L/Precuneus Occipital L/Cuneus Limbic L/CG Frontal R/IFG 9 CE Frontal R/MFG 46 R/Supramarginal Parietal Gyrus Frontal L/IFG Frontal R/MFG Parietal R/IPL 40 CE Limbic Parietal R/SPL 7 L/Parahippocampal Gyrus Hippocampus Frontal R/SFG Frontal R/MFG

135 Limbic L/PC 23 DMN Occipital L/Cuneus Occipital R/Cuneus Occipital L/Cuneus Limbic L/AC Limbic L/CG Parietal R/IPL 40 CG: cingulate gyrus, MFG: medial frontal gyrus, SFG: superior frontal gyrus, IPL: inferior parietal lobe, IFG: inferior frontal gyrus, SPL: superior parietal lobe, PC: posterior cingulate, AC: anterior cingulate. Figure 5.2: Independent components (ICs) showing putative empathy networks and the default mode network for the animation movie (S1). (a) The spatial maps of the independent components with DMN (IC 26) and empathy modes (IC 13, IC15, IC19, IC25. (b) The Fisher z- transformed cross-correlation between DMN and IC13, IC15, IC19, IC25 for all subjects and (c) The normalized average rating of participants (fmri and independent) on selfemotional response at four equal intervals Indian Hindi Movie (S2) The IC identified as DMN (IC26) included medial frontal gyrus (BA 9,46) in addition to the ventral posterior and ventral anterior cingulate gyrus (BA 23,24) and dorsal anterior cingulate gyrus (BA 32). Figure 5.3 presents the spatial maps of 134

136 the selected ICs (figure 5.3a), the corresponding FNC between DMN and empathy networks (figure 5.3b), and the rating plot from the survey (figure 5.3c). IC13, with activation extending to the frontal gyrus, precentral gyrus, insula, cingulate gyrus and parietal lobe, suggests functional overlap of areas attributed to all the three empathy modes. The presence of significant activation in the insula (BA13) in IC15 along with temporal lobe activity indicates an emotional empathy network and existence of mostly frontal lobe activation in IC19 and IC25 suggests cognitive empathy. The Hindi movie was rated second for emotional response by the fmri subjects and in the scene-wise rating by the independent participants the change is marginal with two scenes scoring slightly higher (Figure 5.3c). The rating scores indicate that all the four scene windows were equally emotional, with minor decreases for the first and last window. The FNC plots (Figure 5.3b) show periodic fluctuations for three ICs (IC13, IC19 and IC25), with both positive and negative correlations value for IC19. The emotional empathy network (IC15) shows an initial negative and then an almost constant positive correlation or low FNC variability during most of the stimulus time duration with a sudden increase towards the end. The FNC dynamics for IC13, with activation in the frontal cortex with a smaller cluster of activation in the insula that is reported for emotional empathy, also shows lower fluctuations. In the annexure (S2 in figure A1) the confidence interval at 90% for the averaged cross-correlation for each IC pair is presented. As can be seen, the confidence limits are tighter for IC13 and IC15 as compared to IC19 and IC

137 Table 5.2: The five ICs representing the DMN and putative empathy networks for the Hindi movie (S2) (T value at p<0.001 and voxel threshold: 10). The final column is the suggested putative empathy mode labeled for each IC. T- Valu e MNI coordinate s (x,y,z in mm) Lobe Laterality/Region I C Cluster size Frontal R/SFG 6 Brodmann Area Putative Empath y network ME/CE /EE Limbic R/CG Frontal R/MFG Frontal L/MFG Sublobar L/Insula Frontal L/Precentral Gyrus Parietal R/IPL Parietal L/IPL Tempor al R/TTG 41 EE Sublobar, temporal L/Insula, sub-gyral 13, 21 Tempor al L/STG 41 Tempor al L/STG Limbic L/PC Limbic R/PC Parietal L/Precuneus Limbic R/AC Parietal R/AG 39 CE Frontal R/MFG Frontal R/IFG Frontal L/IFG Tempor al R/FG Parietal L/Precuneus Parietal L/Precuneus Limbic L/Parahippocampal 27 CE Tempor al L/FG Tempor al L/Sub-Gyral Hippocamp us Frontal R/SFG Frontal R/SFG Frontal R/MFG Frontal L/MFG Parietal L/IPL

138 Parietal R/IPL Limbic L/CG Frontal L/MFG Frontal R/ M/IFG 10, Parietal L/Precuneus Tempor al L/MTG Limbic L/CG 23 DMN Limbic R/CG Limbic R/CG Occipita l R/Precuneus Occipita l L/Cuneus Occipita l L/Precuneus Frontal R/MFG Frontal R/MFG Parietal R/IPL Limbic L/CG 32 CG: cingulate gyrus, MFG: medial frontal gyrus, SFG: superior frontal gyrus, IPL: inferior parietal lobe, TTG: transverse temporal gyrus, IFG: inferior frontal gyrus, SPL: superior parietal lobe, PC: posterior cingulate, AC: anterior cingulated,fg: Fusiform gyrus,ag: angular gyrus,stg: superior temporal gyrus, MTG; middle temporal gyrus.. Figure 5.3: ICs and correlation coefficients for the Indian Hindi movie (S2). (a) The z-axis slices of the selected ICs (IC26, IC13, IC15, IC19,IC25) are shown, (b) The Fisher z- transformed average pair-wise cross-correlation plots for all subjects for the selected ICs, and (c) The rating on self-emotional response at equal intervals in the movie averaged across all the survey participants. 137

139 5.2.3 Hollywood movie (S3) The ICs for the Hollywood movie stimulus data reveal activation in higher number of functionally relevant areas. Figure 5.4a presents selected 5 ICs with four of the ICs showing activation in areas reported for empathy networks (Table 5.3) in addition to an IC with posterior DMN. The FNC dynamics is calculated by taking the average of the Fisher z-transformed cross-correlation value for all subjects for the posterior DMN and the four ICs and shown in Figure 5.4b. The independent components (IC13,19,25) with significant activation in the frontal cortex areas (BA9,10,46) along with bilateral inferior parietal lobule (BA40 in IC13,1C25) and medial temporal lobe (BA21) suggesting cognitive empathy. Insula (BA13) in IC15 suggest an emotional empathy network while IC26 with activation in the posterior cingulate gyrus and precuneus indicates posterior DMN confirmed by conducting multiple regression analysis using a reference DMN template (provided in the ICA toolbox). The FNC dynamics plot (Figure 5.4b) shows nearly uniform in-phase periodic fluctuations for all the 4 ICs with DMN, while IC13 and IC15 show negative correlations too. The variation in the correlation with DMN for IC19 is lower compared to IC13 and IC25, though the three ICs have frontal cortex activation and are identified as cognitive empathy networks. The scene-wise emotion rating indicates a linear increase (Figure 5.4c) in conjunction with the narrative and the scenes. The confidence interval at 90% is high and uniform for all the four IC pairs (S3 in Figure 5A1). 138

140 Table 5.3: The five ICs selected for the Hollywood movie (S3) with posterior DMN and putative empathy networks. (T value at p<0.001, with FDR corrected at <0.05 and voxel threshold: 10). IC Clu ster size T- Value MNI coordinat es (x,y,z in mm) Lobe laterality/region Brodman n Area Putative Empathy network Parietal R/IPL 40 CE Limbic R/AC Frontal R/SFG Frontal R/MFG Parietal L/IPL Frontal R/IFG Sublobar L/Insula 13 EE/ME Frontal R/MFG 9 CE Frontal R/MFG Frontal L/IFG Frontal R/MFG Tempora l R/MTG Frontal L/MFG Parietal R/IPL 40 CE Parietal L/IPL Frontal R/MFG Frontal R/SFG Limbic L/Parahippocampal Parietal L/Postcentral Gyrus 3 Sublobar R/Putamen Parietal R/Postcentral Gyrus Limbic L/CG Tempora l L/FG Occipital R/Precuneus Occipital L/Cuneus Parietal L/Sub-Gyral Sublobar L/lentiform nucleus (putamen) Frontal R/MFG Frontal L/SFG Tempora l L/FG Occipital L/FG Frontal L/SFG Sublobar L/Thalamus (MDN) Parietal L/Precuneus 7 139

141 Frontal L/Paracentral Lobule Limbic L/PC 23 DMN Occipital R/Precuneus Occipital L/Cuneus Parietal L/AG Parietal L/IPL Limbic R/CG Limbic L/CG Parietal R/IPL Parietal R/SPL 7 CG: cingulate gyrus, MFG: medial frontal gyrus, SFG: superior frontal gyrus, IPL: inferior parietal lobe, TTG: transverse temporal gyrus, IFG: inferior frontal gyrus, SPL: superior parietal lobe, PC: posterior cingulate, AC: anterior cingulated,fg: Fusiform gyrus,ag: angular gyrus,stg: superior temporal gyrus, MTG; middle temporal gyrus.. Figure 5.4: ICs and correlation coefficients for the Hollywood movie (S3). (a) The spatial maps (ICs) indicative of DMN (IC26) and empathy networks (IC13, IC15, IC19, IC25). (b) The pairwise Fisher z-transformed cross-correlation plots averaged across all the subjects and (c) The rating on self-emotion response (black) as perceived by the participants of the survey. 140

142 5.3 Discussion The fmri data was collected from 15 young adults (20-25 years of age) while they watched clippings selected from a diverse set of movies. The movie clippings were pre-rated by an independent set of participants and by the fmri participants postscanning for emotional context. The rating scores provide reasonable ground to assume varying empathy response to events as processing of the movie narrative is mediated by various processes such as cognitive appraisal of the actor s situation and the emotional state induced in the viewer. The data-driven group ICA method was applied as the experiment was a free-viewing paradigm aimed at identifying functional networks that are active over the entire stimulus period rather than as a response for a specific event or at a certain time point. Three parallel processes were applied to identify stimuli dependent ICs, with maximum weightage accorded to the filtering on CSF,WM and GM and the areas of activation in each IC. The ICs from the group ICA analysis were examined and identified as cognitive, motor and emotional empathy networks in addition to default mode network, based on activation in the areas attributed to the these networks (meta-analysis: Sabatinelli et al., 2011; Fan et al., 2011; lesion study: Shamay-Tsoory et al., 2009; Hynes et al.,2006; Schulte-Ruther et al.,, 2007; Rizzolatti et al., 2001; Rizzolatti, 2005; Gazzola et al., 2007; Blair, 2005; Carr et al., 2003; Engen and Singer 2012). The exploratory study identifies empathy networks based on those reported in published research. As a result the current study does not allow us to make major interpretations but supported by the ratings scores it allows for tentative inferences forming the basis for further research to consolidate the current findings. The basic premise of our study is that multimodal complex stimuli like movies can evoke a multitude of time-varying event specific responses which translate to simultaneous (de)activation in spatially separated regions of the brain and also across networks. The presence of simultaneous activation in many areas which cannot be categorised into definite functional networks and conjecturing the role 141

143 each area plays in different network configurations makes interpretation of data from complex stimuli challenging. The dynamic correlation analysis between the default mode network (DMN) separated from the free-viewing experiment and emotional responses to the stimuli is aimed at inferring the role of the DMN in non-resting state condition. Understanding the change from equilibrium state (Raichle et al., 2001) of this network for empathy response will help in analysis of clinical conditions like autism and schizophrenia, conditions which show deficits in emotional response. Considering that the ICA method was applied and motion or noise signals were not regressed out as usually done when using general linear method (Desjardins et al., 2001), the effect of global signal regression leading to anti-correlations (Murphy et al., 2009) can be ruled out Functional Connectivity S1 (animation movie) The dfnc of the animation movie is interesting and contrasts with that of the liveaction movie data as the ratings indicate constant emotional response for 3/4 th the movie clipping with an increase for the last scene and the fmri data does not reveal any activation in the areas attributed to emotional empathy, while cognitive and motor networks were comparable (see Figure 5.2). The dfnc plots indicate lower correlation for IC13, IC15 and IC19 with the DMN, where IC13 suggests a motor and cognitive empathy network and the latter two indicating cognitive empathy. The low correlation values suggest possible disengagement with DMN especially for the network (IC13) with motor empathy areas. The higher correlation and fluctuations for IC25 also identified as cognitive empathy could be due to the presence of parahippocampal gyrus which is reported as a primary hub of the DMN (Ward et al., 2014) and this area also did not show deactivation in a working memory task in contrast to other DMN areas (Koshino et al, 2014). The IC25 correlation with DMN (IC26) should also be interpreted cautiously from the 90% confidence interval estimates which show instability at the upper interval (S1 in Annexure Figure 5A1) 142

144 5.3.2 Functional connectivity for S2 (Indian Hindi movie) The FNC with IC15, an IC with substantial activation in the insula (BA 13) hints at emotional empathy network being relatively stable and has very low correlation with an increase in correlation towards the end of the stimulus period. The rating on emotional response from the survey indicates smaller changes between scenes and a decrease in rating at the end. The lower and relatively stable FNC of IC13 noticed for IC15 with areas that are identified to be from the all the three empathy modes indicates a lower engagement between the networks possibly due to greater focus for emotion processes and inhibiting mind-wandering for which the DMN network is associated. The components (IC19, IC25) representing cognitive empathy show recurrent FNC fluctuations with DMN, with IC25 having higher correlation values possibly due to the presence of activation in the parahippocampal gyrus. The confidence interval at 90% indicates a relatively uniform lower confidence for the upper interval for all the four ICs Functional Connectivity for S3 (Hollywood English movie) The fmri recording was for 460 seconds and for group analysis the time points of all stimuli have to be equal (260 seconds), hence time points of scenes with no relevant action for example in S3, there is a 90 second scene showing Duncan being led by the guards from his cell to the execution room. The cut-scenes were also checked for emotional content while also taking care of minimal disruption in the narrative. The FNC showed uniform periodic fluctuations in all the 4 selected ICs with IC25 and IC19 with probable cognitive empathy network having significant activation in the dorsal medial prefrontal cortex showing higher correlation values, which could mean functional overlap with DMN as this network has also been implied for self-referential processing. The IC15 with activation in insula shows negative and lower correlation values for a short period of time which is consistent with the findings for S2 and decidedly indicating that 143

145 emotional empathy response could lead to disengagement of DMN. The correlation plot of IC13 with areas covering cognitive empathy shows a similar trend which could be due to the presence of significant activation in the anterior cingulate (BA24) reported for emotional empathy (Engen and Singer, 2012). The plots for S3 show non-random distinct periodic oscillation for all the ICs, except IC19 which shows lower modulations. The rating for emotional response increases over the movie narrative in line with the events supporting the presence of significant activations in the areas attributed for emotional empathy network. The confidence interval estimate at 90% shows stability of correlations observed for all the four ICs, which is in contrast to the findings for the other two stimuli. For all stimuli, the positive correlation of FNC with probable cognitive empathy networks is consistent with reported findings of DM networks involved in theoryof-mind (Review: Buckner et al., 2008; Spreng et al., 2010) process, a precursor for cognitive empathy (Mars et al., 2012) and for social cognition (Buckner et al., 2008; Buckner and Carroll, 2007; Laird et al., 2011; Mars et al., 2012). The negative and lower correlations noticed for ICs with emotional and motor empathy network indicate possible suppression of DMN as emotion processing requires focused attention observed in prior studies on attention and DMN (review: Buckner et al., 2008; Fox et al., 2005& 2006; Fransson, 2006). The stimulus-dependent response relations between the DMN and motor empathy networks suggest a more complex affiliation than the posited suggestion of predominantly negative correlation for goal-directed tasks (Rizzolatti, 2001, 2005; Weissman, et al., 2006) or during selfrelated cognition (Uddin et al.,2006) especially when extended to natural scene stimuli. The relationship between task-negative networks or DMN and the taskpositive activation in a task paradigm comprising the dorsal parietal and lateral prefrontal cortex ( Raichle et al.,2001; Shulman et al., 1997; Spreng et al., 2010; Uddin et al., 2009) is either one of tension between the two or a dual-process model (Evans, 2003;Jack et al.,2012), with independent and simultaneous 144

146 activation in response to an external stimulus. In a study that compares working memory network to DMN it was also reported that during the task preparation state both the networks were activated while execution stage in a verbal working memory task saw a deactivation (Koshino et al., 2014). This observation when applied to empathy can hint at positive correlation during the cognitive processing of the events in the stimulus followed by negative correlation for scenes with higher emotional contagion. The positive correlation values observed between DMN and the ICs with cognitive empathy and the lower correlation with non-periodic fluctuations could possibly be due to emotional and motor empathy being more demanding. This inference needs to be verified in future by more focussed experiments. Though intra-stimulus comparison was the focus of the current study considering the diversity of the movies used, it was interesting to examine the dfnc of each selected IC in a combined plot. Figure 5.5 depicts grouped data, combined according to the empathy network identified. The ICs (S1 & S2: IC13, S3: IC15) with activation in the precentral gyrus region (BA 3, BA 6) with either cognitive or motor empathy network show lower fluctuations for all the stimuli (figure 5.5a). The dfnc plot for the emotional empathy network (figure 5.5b) shows lower fluctuation for S2 (IC13 & IC15), whereas for S3 we see an almost periodic fluctuation. The empathy response ratings for S3 was higher and the correlation plot for empathy network (figure 5.5b) shows negative or inhibition for a timeduration. The plots for cognitive empathy (figure 5.5c) network of IC25 with significant activation in the parietal lobe (BA 40) and hippocampus areas disclose comparable high periodic variation while IC19 with common activations in the medial frontal cortex, extending to BA6 for S2 and S3, showed synchronous fluctuations. The correlation plots and the neural correlates identified for the ICs, tentatively suggest that emotional empathy and motor cortex activation result in 145

147 inhibition or even anti-correlation values, while cognitive empathy is mostly positive and higher. Figure 5.5: A combined plot of the cross-correlations for the three stimuli, a) the IC (IC13), indicative of a motor empathy network, b) the FNC of the IC identified as emotional empathy network and c) the 2 ICs (IC 19, IC25) with areas attributed to cognitive empathy network. From the combined plot of Figure 5.5 we can infer a possible dual-process, where distinct cognitive modes are applied for empathy response in social information processing like inferring the state of mind of others and of objects. The motor and emotional empathy network correlation plots suggest a possible tension between the two networks, leading to an inhibition of the DMN. 146

148 Taken together, these results suggest a possibility of attention being more focused for emotional events reducing mind-wandering or self-reflections, which effect cognitive empathy. As causality cannot be established by the current analysis, anticorrelation may also imply a control of emotional empathy by indulgence in perspective taking by distinguishing the self (viewer) and other (actor) or vice versa which implies the role of DMN in perspective taking. FNC analysis is an insightful tool to understand how regions or networks evolve and engage in real-life natural scene processing and may be helpful in investigating empathy deficiency in clinical conditions like schizophrenia and autism 5.4 Limitations In terms of limitations, we would like to mention that the stimuli were rated for emotional context of the narrative and the depiction of emotions by actors but the participants did not tag specific events during the scanning process as it would intervene in the immersion in the narrative. This prohibits us from making conclusions on causes for change in state and the nature of engagement at different time points during viewing of the movie clipping. However, rating provided by independent participants show the possible trends. Second, the regression analysis with the independent ratings taken at 4 intervals exhibited significant p-value for some ICs, however regression analysis by itself did not allow us to identify all the task-relevant ICs. A focused experimental design is required to make a direct inference between the empathy networks and the self-reported empathy response. Third, by not using a model-based functional connectivity analysis like dynamic causal modeling (DCM), our results do not allow for inferences on causality between the networks. 5.5 Conclusion This exploratory study was aimed at understanding the dynamic connectivity changes between DMN and putative empathy networks isolated from a free- 147

149 viewing experiment to decode correlation between the networks during the processing of dynamic and complex stimuli. The identification of empathy networks is based on the areas reported to be active in many empathy response studies and the confirmation that our stimulus does evoke empathy is also supported by the empathy rating scores provided by participants. The study using long natural viewing paradigm does not allow strong interpretations on the causal role of the areas and networks to the stimulus but we can safely assume that moving images with narrative are engaging. Second, activation in certain areas reported for empathy response for a particular experimental paradigm like observation of someone being in pain might not necessarily be applicable for all emotion evoking stimuli and this is the effort of all ongoing research on multiple roles assumed by a single area or a network by experimenting with different stimulus types. Our exploratory study is a step in this direction as natural viewing with no intermediate tasks permits us to look at complex constructs like empathy as it is not a sudden reflex response but that which requires context assimilation and strategizing an appropriate response. Considering that the movies were diverse, the scrutiny of the data was within movie than inter-movie though group ICA allows for comparison of global trends. Further experiments comparing short and long empathic narratives are being planned that will highlight differences in empathy networks and the dynamic correlations between time series of specific areas and networks, respectively. 148

150 Appendix The dynamic functional network connectivity plot with 90% confidence interval estimation. Figure 5A1 : The dfnc plots of the four ICs (Black: IC13,Red: IC15, Green: IC19, Blue: IC25) cross-correlated with the DMN (IC26), with 90% Confidence interval calculated for each of the stimuli. 149

151 Chapter 6 Empathy response to long and short narratives* 6.0 Introduction To empathize, one needs to have the ability to understand what another is experiencing to reciprocate appropriately (Decety & Jackson, 2004). Many definitions or conditions of human/animal response behavior are posited to define empathy most succinct being: the response elicited when we understand, respond and share other s feelings while all along aware of the distinction between the state of the other and self (Decety 2005; Blair, 2005). A further distinction put forth is one s ability to realize that the other s state is the reason for self s affective state (de Vignemont & Singer, 2006). Engen and Singer (2013) provide a model with all the factors that lead to empathy experience comprising a complex combination of regulation, generation and modulation of empathy. A correlated emotional reflexive reaction and the cognitive aspect (Singer, 2006; review: Decety and Jackson, 2004; Keysers and Gazzola, 2007; Walter, 2012) is also reported as empathy. The ability to take other s perspective, a crucial requisite for empathy and indulge in reasoning unlike the Theory-of-mind (ToM) (Premack & Woodruff, 1978; Frith & Frith, 2003; Decety & Lamm, 2006; Schnell, et al., 2011) is vital for human coexistence. * Vemuri K. Mohit Goel. Surampudi BR The long and short of narrative empathy. under review 150

152 Studies have looked at empathy to pain inflicted on other (Lamm et al., 2007a,b, 2008; Singer et al., 2004 ), emotional faces and short emotional scenes (Adolphs, 2002; Schulte-Rüther et al., 2007; Fusar-Poli et al., 2009) and extended to experienced empathy (Rameson et al., 2012). Fleeting emotional expressions with no specific contextual information can rarely evoke empathy unless one has experience or the imagination to make knowledgeable inferences. That is, in addition to cognitive inference of what the affective state of the other is, empathy response is aided by knowledge of 'why' the other is in a particular state and this defines the understanding required for sharing of the other's feelings. A narrative is a method of conveying the reason for the state of the other by threaded events presented in a temporal sequence either in text, verbal or visual format and is an integral part of social communication. Readers or viewers experience a narrative by dissecting the events presented and taking the perspective of the agent(s). The short relationship the viewer builds with the other can be a state where he/she just understands the feelings expressed by the other by comprehending the context (Gallagher, 2012) and can extend to actually reciprocating the feelings both evaluations considered to be basis for empathy (Decety and Jackson, 2004). Narrative empathy is defined as a sequence of temporal events with a promise of an end (Keen 2007, 2010) to ensure attention and engagement. Empathy response to narrative requires that the viewer follows the beliefs and thoughts of the characters and allows the experience to influence emotions comparable to that depicted by character or as appropriate to the context. Studies have looked at emotion and action in narrative imagery (Sabatinelli et al., 2006), reading (review: Oatley, 1994) and story reading in children (Brink et al., 2011). Neural correlates of comprehension of narrative either presented in a speech format or as written text (Giraud, 2000; Xu et al., 2005; Yarkoni et al., 2008; Wilson et al., 2008; Mason and Just, 2006, 2009; review: Mar, 2004) revealed activation in the temporal lobes and dorsomedial prefrontal cortex (Stowe et al., 2005; Hasson et al., 2008a; Xu et 151

153 al., 2005; Mason and Just 2006) the later area is linked to Theory-of-mind (Frestl et al., 2008). Another relevant study by Altmann et al. (2012) looked at passive reading of short stories with emotional contagion and report higher engagement of the affective ToM-related brain areas as the negative valance of the story increases. It is also possible that movies or books evoke emotional contagion (as defined by Bernhadt & Singer, 2012) and reflexive mimicry to facial expressions and rarely direct empathic concern (again as defined by Bernhadt & Singer, 2012) as the viewer is constantly aware that the other is an actor and the events are not real. Multimodal complex stimuli like movie narratives are plurmedia system where a story is presented sequentially by a series of events, the comprehension of which is derived from expressions, gestures, actions, music and language is considered to be highly immersive. In most studies using movies the clippings were short with minimal narrative (Goldin et al., 2005; Nummenmaa et al., 2012) and do not explore explicitly the role of narrative or context on empathy response. Presenting a complete explanation for the state of the other as in a long rich narrative and studying the underlying neural responses is particularly interesting as it provides insights into variations in natural scene comprehension by viewers, applied in segmentation of the events by the brain (Zacks et al., 2010). Stories, neutral and emotional, narrated with congruent or incongruent facial emotional expressions showed that sad stories increased activity in the amygdala and parieto-frontal areas and not-so for incongruent facial expressions (Decety & Chaminade, 2003). Temporary affiliation between the viewer and actors has shown to involve the empathy modes (Preston & de Waal, 2002; Decety & Jackson, 2004), which was investigated in detailed in a previous study (Chapter 4 & Vemuri & Surampudi, 2015) using three commercial movie clips from different genre and cultural settings. Evidence of empathy networks consistent to the empathy response self-reports by participants was provided in Chapter

154 Neural correlates of empathy response were initially studied by using video clips of patients in pain which revealed responses in insula, amygdala and anterior medial cingulate cortex in subjects instructed to imagine the feelings of pain (Lamm et al., 2007a). Extending the pain-empathy findings Lamm et al.(2007b) studied the self and other-oriented empathy response by cognitive appraisal when the outcome of a painful medical procedure was mentioned to the participants and compared the activations when they just viewed the procedure. Signal changes were reported in the anterior cingulate cortex, right lateral middle frontal gyrus and the ventromedial orbitofrontal cortex indicating empathic concern for a conspecfic in pain is modulated by cognitive perspective taking when context is provided. Hence, empathy is an evolving complex response involving a number of processes and studying neural correlates with stimuli having minimal information of the reason underlying other s state constraints the understanding of the brain activations. The goal of the present study is to identify and compare empathy networks for 'informed' (wherein context is presented in detail) appraisal to that of inferred (wherein minimal or no context is provided) state of the other using movie narrative. Our premise for the study is: a narrative which presents the reason for the emotional state of the other can stimulate stronger empathy responses with the self-other distinction though by the transportation theory a transient merger of the self-other is also probable. At the neural level, activations in the areas attributed to emotional and cognitive empathy network, like the insula /frontal lobe would be considerably higher for informed emotional scenes compared to inferred, is the expectation based on the premise. There has been no study which has compared the neural correlates for short emotional scenes and when the same are part of a longer narrative. To minimize familiarity bias, we selected foreignlanguage (non-english too) movies with actors of not the same ethnicity as the participants and with subtitles in English language. 153

155 6.1 Materials and Methods Participants Sixteen healthy university (age: mean (M) = 25.56, stddev = 3.32, range= years) post-graduate students (9 male and 7 female) took part in this study. All the subjects are avid western movie fans, as elicited from an informal interview before the scanning, and have above average English language comprehension skills. That is, have English as primary language medium of learning throughout their academic years. The human ethics committee of the International Institute of Information Technology, Hyderabad cleared the study and all subjects gave written informed consent and were paid for their participation. In addition, 40 participants (age: mean (M) = 21.73, stdev = 2.32, range = years) with similar education and English language proficiency as the fmri participants took part in an independent survey to rate the movies and the individual scenes Stimuli and Survey Two short feature films: An Egyptian film titled These Times (M1) and a polish movie Most or The Bridge (M2) were selected for this study (storyline and description of the scenes selected in Annexure at the end of the chapter). Both the movies were edited to run for 8m 45sec approximately and the short clippings were seconds long. The short clippings were sequential scenes from the longer version and following the observation by Plantinga (1999) close-ups of the emotional faces were shown for at least seconds as required to evoke faceemotional response. The rationale for considering these two movies was a) the movies had actors who were not Indians, to reduce confounding influences due to intra-cultural biases arising from economic, caste, language and physical similarity bias and b) The direction was concise and had very poignant instances of change in narrative. The movies were rated for overall empathy context and scene-wise rated for self and other-oriented response categorization. 154

156 Forty participants of the same age and education levels as the fmri subjects viewed the movies and rated the neutral and emotional scenes on a 1-5 semantic differential scale (Osgood, 1952). Nine participant survey data was discarded due to double marking on the sheet. The subjects viewed the 2 short clippings (neutral and emotional) with a pause in between to allow for scoring on empathy followed by the longer clipping which was again paused after the same scenes as shown in the short clippings. The order of presentation of movies M1 and M2 were counterbalanced with 2 groups of 20 each. The sixteen fmri subjects were instructed to watch the movie clips and complete a post-scanning survey rating for neutrality and emotional content of the selected scenes (detailed survey results and questionnaire in the appendix). The participants also answered scene specific questions designed to evaluate their self-other oriented response. The detailed questionnaire and scores from the post-scan rating provided by the fmri participants and those of the independent participants is provided in the appendix at the end of the chapter fmri Experimental Design Two short clips of seconds of scenes rated as neutral and emotional in context were followed by the longer version (8 minutes 45 seconds) as shown in the experiment design diagram in Figure 6.1. The order of presentation of the 2 movies was counterbalanced across subjects. Resting state data for 5 minutes was collected once at the beginning for each subject before the stimulus was presented. For the passive viewing experiment, the participants were instructed to watch the mirror-screen on which the stimulus was projected. 155

157 Figure 6.1: The experimental design for the fmri data collection. The resting state data was collected for 5 minutes. "The Time" (M1): Neutral: 32 seconds, Emotional: 22 seconds, Complete clip: 8 minutes 45 seconds. M2: Most or The bridge. Neutral : 13 secs. Emotional : 9 secs, Complete clip: 8 minutes 35 seconds. The order was counterbalanced by showing M2 first to half the participants Image acquisition Magnetic resonance Imaging data were acquired on a 3 Tesla Philips Achieva scanner located at the National Brain Research Center, Gurgoan, India. Structural images ((TE=3.7 ms; TR=8.4 ms; flip angle=8 ;matrix= ; voxel size=1 1 1 mm, ) were acquired with a T1- weighted sequence using fast field echo technique and Turbo Field Echo sequence. Functional images were collected using a gradient echo planar T2* sequence with TR = 2seconds, TE = 35 ms, flip angle of 90 degrees, FOV = 230mm and acquisition matrix = 64x64. Each functional image comprised 30 transverse slices with a thickness of 5 mm, a gap of 1 mm, acquired voxel density of 3.59 x 3.59 x 5mm reconstructed voxel density of 1.8 x 1.8 x 5mm. A total of 262 functional scans for the longer movies and 30 scans for M1 and 36 scans for M2 for the shorter clippings were acquired Data Analysis After conversion of the DICOM format to NIFTI format the scans were realigned, registered, normalized and spatially smoothed with a Gaussian kernel set at 6 mm FWHM (full-width at half maximum). The individual scans were realigned to the first scan and normalized to the ICBM s 152 MNI template provided in the SPM8 template library written into of 2x2x2mm voxel dimension. Statistical analysis was carried out using the general linear method (1) and by conjunction of contrasts 156

158 between the emotional and neutral conditions. The sub-set of scans (M1: Neutral = 13 scans, Emotional = 14 scans; M2: Neutral = 20 scans, Emotional=14 scans) from the longer movie corresponding to the neutral and emotional scenes as shown in the shorter-clips was included in the block design independently and the contrast estimated for the 4 conditions for M1 and M2. The statistical parametric maps corresponding to the contrasts between the emotional/neutral sequences in the longer movie to that shown as a short clipping were acquired for each subject with a global height threshold of p < A group level random effects analysis was then conducted using a 2 sample T-Test on the contrast maps for each condition with a threshold of p<0.01, family-wise error corrected, and a voxel extent of 10. The anatomical regions corresponding to the MNI coordinates were interpreted with Anatomy Toolbox version 1.7 (Eickhoff et al., 2005) and the 3D fmri statistical images overlaid on an inflated brain mask using the visualization toolbox bspmview (2). 6.2 Results To assess differential response to neutral and emotional scenes embedded in the longer narrative two time-windows with the same scenes as the shorter clippings were considered in an event-design and the following contrast conditions were analyzed: C1: neutral_long > neutral_short, rc1: neutral-short> neutral_long, C2: emotional_long> emotional_short, rc2: emotional_short > emotional_long ( where the > sign implies more ) were extracted using 2 -Sample T-Test from the contrast maps estimated for each subject. Two different movies were analyzed to confirm that the empathy networks are a function of the response to the stimuli, but are analyzed independently and significant activation for each condition and stimuli presented. 157

159 6.2.1 Survey Independent survey and post-scanning self-reports by fmri subjects The two movies used as stimulus in the study vary in story, cultural setting and cinematography, hence were rated separately for self and other empathy response. Post-scanning, the fmri participants answered questions designed to understand whether they could detect the state of the actor(s) and differentiate it to selffeelings and secondly asked to rate specific scenes for emotional level. The average rating on a semantic scale of 1-5 with the highest number indicating negative emotion and lowest for neutral, for M1 s neutral scenes was (stdev = 0.75) and for emotional scenes (stdev = ). For M2, the neutral scenes average value was (stdev = ) and for emotional scenes the average was 4.25 (stdev = ). Scene specific questions were designed for participants to express the actual emotion they experienced sad, distress, happy etc., and judgmental, that is, if they (dis)agreed to the decision taken by the actor and what they would have done if in the same situation. Four of 16 participants report feeling angry for an action taken by the character in M2 while the rest reflected the emotions of the character ( sadness ). In a sequence of scenes showing acute sadness depicted by one of the actors in M1, three of the 16 subjects expressed anger (judgmental) at the person who had placed the character in that particular state. The 40 participants of independent survey were shown short clippings first and were asked to rate emotional context and their self-emotional response for the short neutral and emotional scenes. The longer movie was presented next and the same scenes as in the shorter version were again rated for the same parameters. The semantic scale rating with lowest score indicting least in negative emotion context was lower for the short clips than for the same in the longer narrative for emotional scenes (mean, M1:short= 2, long = 2.68; M2: short=1.16, long=4.26). A two sample T-Test with a significance level at p<0.05 was conducted on a semantic differential scale rating of emotion level on the neutral and emotional 158

160 short clips and the same clips when embedded in the movie (Table 6.1). The two sample T-test result was not significant for only one condition (Neutral short and long clip comparison) for M2. From the significance values, it can be inferred that the emotional scenes viewed within the longer movie evoked higher emotional response and that secondly the selected clips were appropriately neutral or emotional. Table 6.1: The 2-sample T-test analysis of the ratings from the independent survey. Participant size: 40, significance level: p<0.05. The underlined P-value for M2 indicates that the 'p' was not significant for this condition. Condition T value (M1,M2) P-value (M1,M2) Neutral_short, Neutral long , , Emotional_short, Emotional_long 2.843, , To investigate the brain response differences with and without context elucidating the reason for the state or action depicted by the character (s) we compare the activations of the conditions for neutral and emotional scenes. The fmri participants rated the overall emotional quotient and answered questions modeled along the IRI empathy index (Davis, 1983), to check for self-other merge or distinction relative to the scenes. The same was also collected from the 40 independent participants. In addition, a continuous rating (1 data-point/s) as the short and long movie clips were being viewed was collected from 18 independent participants of the similar age and demographics as the fmri participants (Figure 6.2). The rating values at the selected movie clips for neutral and emotional scenes are confirmed. 159

161 Figure 6.2: The continuous rating on a scale of 0-1, with neutral at 0.5. (a) The circles in red/green at 2 time points are the neutral and emotional scenes shown as short-clips. (b) the rating for the short clips. M_Neu_short : denotes the neutral scenes and M_emo_short : are emotional scenes Neutral Narrative (C1: neutral_long > neutral_short, rc1: neutral-short> neutral_long, C2: emotional_long> emotional_short, rc2: emotional_short > emotional_long) The contrast activations for the neutral scenes for M1 and M2 are listed in Table 6.2 and the brain mask with activations shown in Figure 6.3. The data corresponding to the neutral scenes watched in the full-length movie (C1) for M1 revealed activation in cerebellum extending to the parahippocampal gyrus, fusiform gyrus, putamen and the middle frontal gyrus (yellow/red colors in Figure 6.3a), while superior temporal, right inferior frontal gyrus (IFG) (Brodmann area 44) and putamen was seen for condition rc1 (blue color in Figure 6.3a). The parahippocampal gyrus is reported for self-referential processing during recall (Muscatell et al., 2010), in processing of emotional facial expressions (Fusar-Poli et 160

162 al., 2009), in scene categorization and recognition (Epstein et al. 1998, Walther et al., 2009) and for episodic encoding (Hasson et.al. 2008b) of natural scenes while the fusiform gyrus is reported in studies on face recognition tasks (Hasson et al., 2003). Similar contrast comparisons for M2 revealed activation in the medial cingulate cortex, superior and bilateral inferior frontal gyrus in addition to middle frontal gyrus for condition C1 (Table 6.2, yellow/red colors in Figure 6.3b). For rci, significant activation was seen in the precunneus extending to the inferior occipital gyrus, bilateral inferior frontal gyrus, precentral gyrus and hippocampus (blue colors, Figure 6.3b). Table 6.2: The coordinates, cluster size, region and T-Values (p< 0.01 (corr) and cluster size threshold: 10 voxels) of activations for the conditions (long>short and vice-versa) pertaining to neutral scenes. Neutral_Condition Laterality/Region Voxels Cluster Size T- value MNI coordinates (x,y,z) M1: C1 L Cerebelum (Crus 1) R ParaHippocampal Gyrus R Middle Frontal Gyrus L Fusiform Gyrus R Cuneus R Putamen R Insula Lobe M1: rc1 R Caudate Nucleus L Superior Temporal Gyrus R Mid Orbital Gyrus R IFG (p. Opercularis) L IFG (p. Orbitalis) R Putamen

163 M2: C1 L Middle Frontal Gyrus R MCC L Superior Frontal Gyrus L IFG (p. Triangularis) R Middle Frontal Gyrus R Superior Frontal Gyrus L Inferior Parietal Lobule R IFG (p. Triangularis) M2: rc1 R Precuneus R Lingual Gyrus L Inferior Occipital Gyrus R Caudate Nucleus L IFG (p. Triangularis) R IFG (p. Triangularis) L Precentral Gyrus L Hippocampus L posterior-medial frontal IFG: inferior frontal gyrus, MCC: Medial Cingulate Cortex. Figure 6.3: The T-statistical values from the activations for the two conditions comparing short/long clips of neutral scenes. A threshold of p< 0.01 (FDR corr) and voxel extent of 10 was applied and the activations are overlaid on coronal inflated brain mask. a) M1: C1: neutral_long > neutral_short (in yellow/red), rc1: neutral_short > neutral_long (blue). b) M2: C1: Neutral_long > Neutral_short (yellow/red), M2: rc1: Neutral_short > Neutral_long (blue). The red and dark blue colored cluster indicated higher T-value. 162

164 6.2.3 Emotional Narrative In the comparative analysis of the emotional scenes the regions with activation (Table 6.3) that survived the threshold (P<0.01, corrected) for M2 (Figure 6.3b) was greater than for M1 (Figure 6.4a). For condition C2 (emotional_long > emotional_short) bilateral insula activation was observed for M2 with right IFG (p.orbitalis) extending to right insula region and for M1the right insula activation was isolated. Other areas include middle frontal gyrus, superior frontal and temporal gyrus for M1 and fusiform, pre-post central, middle temporal and frontal gyrus for M2. For the reverse contrast ec2 (emotional-short >emotional_long), the occipital, precunneus, middle occipital gyrus (BA 47/11) and IFG (BA 46) was active for both stimuli. Table 6.3: The regions, T-values, MNI coordinates of the 2 conditions in movies M1 and M2 for emotional scenes. The significance was set to p<0.01 (corr) and cluster size threshold to 10 voxels. Conditions Laterality/Region Voxels Cluste r Size T- valu e MNI coordinates (x,y,z) M1: Emotional_long> Emotional_short R Rolandic Operculum R Insula Lobe L Middle Orbital Gyrus L Superior Medial Gyrus L Putamen L Superior Temporal Gyrus M1: Emotional_short>Emot ional_long R Linual Gyrus

165 164 L Precuneus L Middle Orbital Gyrus M2: Emotional_long> Emotional_short L Postcentral Gyrus L Precentral Gyrus R Fusiform Gyrus R Cerebelum (IV-V) L Superior Parietal Lobule L SupraMarginal Gyrus R Putamen R IFG (p. Orbitalis) L Hershls Gyrus/Insula Lobe L IFG (p. Opercularis) R Superior Frontal Gyrus L Middle Temporal Gyrus L Middle Frontal Gyrus

166 L Cerebelum (IV-V) L posterior-medial frontal R SupraMarginal Gyrus R Angular Gyrus M2: Emotional_short>Emot ional_long L Inferior Occipital Gyrus L MCC R Precuneus L SupraMarginal Gyrus R IFG (p. Triangularis) IFG: inferior frontal gyrus, MCC: Medial Cingulate Cortex Figure 6.4: The activations from the emotional scenes contrasts overlaid on an inflated brain mask for M1/M2. A threshold of p< 0.01 (FDR corr) and voxel extent of 10 was applied and the activations are overlaid on a standard T1 template provided in SPM 8. a) M1:C2: Emotional_long > Emotional_short (yellow/red), M1:rC2: Emotional_short > Emotional_long (blue). b) M2: C2: Emotional_long > Emotional_short (yellow/red); M2: rc2: Emotional_short > Emotional_long (blue). The red and dark blue colored cluster indicated higher T-value. 165

167 6.3 Discussion The current study was to investigate the brain networks active when emotional and neutral scenes are viewed with a context, compared to the same scenes watched as short clips. The clips were rated for emotional or neutral narrative context and in addition to emotional face or gesture expressions. The areas of activation attributed to the empathy modes cognitive, motor and emotional were identified for each condition. Context provides the perspective to help the viewer comprehend the reason for the particular state of the other and hence can moderate an appropriate empathy response. Inferring with minimal information depends on individual creative imagination or deduction from experiences which requires higher cognitive appraisal. The act of engagement in an audio-visual narrative as in a movie involves many cognitive processes like prediction of events, judging the actions, confirming predictions, recall and recognition in addition to self-emotional responses similar to real-life event processing. Underscoring this, the self-report collected from the fmri participants reveal a complex engagement with the narrative covering a range of feelings of being judgmental of the actions of the characters enacted, feeling extremely sad to the turn of events, understanding the emotions depicted and strategizing actions self would have taken if in the same state as the other. Empathy in real-life depends on understanding and responding accordingly to another's feelings (Decety & Jackson, 2004) and hence our premise is that informed empathy response is higher than inferred response Neutral scenes The neutral scenes in the longer narrative for both the movies (M1,M2) revealed higher activation in the middle and inferior frontal gyrus (BA10,9,11) areas reported to be critical for cognitive empathy network (Shamay-Tsoory et al., 2009). The neural correlates attributed to Theory of Mind in a narrative reading task compared to non-related sentences point out the role of the middle prefrontal cortex (Fletcher et al., 1995; Xu et al., 2005; Mason and Just 2007) in comprehension. 166

168 For M1, significant fusiform gyrus activation was also observed for neutral scenes in the longer movie as the scenes had close-ups focused on the face. In the shorter neutral clip comparison (neutral: rc1) for M1 the IFG (p. Opercularis,BA 44;) an area reported to be crucial (Shamay-Tsoory et al., 2009) for emotional empathy for processing of neutral and emotional scenes (Levesque et al., 2003), motor (Carr et al., 2003) and in conjunction with p.orbitalis of IFG ( BA 47) in self-other focused study on empathy response (Schulte-Ruther et al., 2007) was interesting as the rating on emotion context for the short-clip was low. A possible explanation is the dual role of the IFG areas (BA 45,46,47) in language processing and comprehension (review: Price, 2000). As the sub-titles in English language were displayed at the bottom of the screen (foreign languages polish/arabic spoken) the activation in the language areas can be attributed to reading and comprehension. Observations of motor actions in the other results in activation in the motor cortex region (Rizzolatti et al., 2001,2005, Gazzola et al., 2007) and is significant for M2 (pre-post central gyrus) in both neutral and emotional scenes as the stimuli had actors depicting physical movements and hence the engagement of the motor empathy network. The post-scan rating by the fmri participants on the specific scenes considered for analysis is in conjunction with the neural correlates wherein the neutral scenes of M1 were rated slightly higher (average: 1.815) on emotion compared to the neutral scenes of M2 (average: ), which could possibly explain the activation in the insula in M1 (neutral:c1). The p-value for the neutral scenes in the independent survey were significant for M1 and not so for M2. This conveys that the neutral scenes rating for M1 in short and long narratives were affected by the context while for M2 the rating was nearly equal suggesting that context did not add to the comprehension of the sequence of events. 167

169 6.3.2 Emotional scenes The post scanning rating for emotional scenes was higher (mean: 4.25) for M2 than for M1 (mean: 3 875), and in the independent survey the mean was higher for M2 compared to M1, while the p-value from the T-test was statistical significant for the long and short comparison. The insula recognized to be central for emotional empathy response was common and higher for the emotional scenes viewed within the longer narrative confirming that context is crucial for stronger self-other oriented emotional empathy response. Additionally, areas associated with cognitive empathy (like the middle prefrontal gyrus) were co-active with emotional empathy network for the longer clips signifying that context evokes perspective analysis driven empathy response. The networks identified from the data also confirm to the anatomical data which suggests a network connecting the insula, posterior parietal, superior temporal and inferior frontal network (Augustine,1996) with insula as the conduit to limbic areas for processing of emotional scenes ( Carr et al.,2003). Overt depiction by physical actions of distress and grief (M2) triggers activation in the motor empathy network while subtle facial expressions or a gesture (M1) does not evoke similar activation in this region. The presence of an integrated empathy response comprising areas attributed to cognitive and motor networks implies that perspective and appraisal are critical for emotional empathy, which is in conformity to Feshbach s (1987) observation that empathy is the result of cognitive and effective empathy, also proposed by Blair (2005). The role of information in manipulating cognitive appraisal and subsequently empathy response is similarly reported in a study to evaluate distress to pain inflicted on the other (Lamm et al., 2007b). Using behavioral and the fmri data from two diverse movies analyzed independently we demonstrated the role of narrative in activation of distinct empathy networks for neutral and emotional scenes. Further we establish that emotional empathy in a narrative context is a cognitive-driven cogent response and 168

170 subconscious than immediate reaction to a facial expression or non-contextual emotional scenes, though it is possible that human brain can fill or imagine a narrative either from previous experience or by being creative in a highly subjective manner. Our findings are significant for research in understanding complex functional networks as we show that a narrative which adds a semantic context to the viewer engages and disengages many areas of the brain simultaneously and also evolves rapidly. Functional and effective connectivity analyses could potentially unravel these and we plan to undertake this analysis in future. Application would include testing for empathy when presented with a narrative s in conditions like autism and schizophrenia. Considering the potential influence of culture on narratives, it would be interesting also to study how narratives introduced in early childhood can help in better social interaction. 6.4 Conclusion and Limitations In conclusion, we demonstrate for the first time a strong neural evidence of empathy network activation variation as a function of visual narrative. The neural correlates of empathy using stimuli like movies is complex considering that the viewer is cognizant that events are fictional and could be applying a control mechanism to reiterate this to self but, an immersive narrative could empathy. Considering that processing of events in a narrative entails evaluation of each scene with respect to the events that precede it and by prediction of the reaction by the actor, the activation data from neutral or emotional scenes for the longer movie included a more complex functional empathy network. One limitation of the study is the diversity of the movies we used and this does not allow for a direct comparison of the two movies. Secondly, ecological validity is always at the cost of experimental control and considering that the participants were not asked any intermediate responses like questions or explicit instructions to experience empathy, we relied on the power of the narrative immersiveness, the post-scanning report and independent survey. A third constraint is the use of only 169

171 one short clip of neutral or emotional scene, but changing the design by showing more short scenes could have a potential detrimental effect on the viewer's narrative experience. 170

172 Appendix fmri and Independent participant questionnaire A6.1: Overall The questionnaire is a self-report on the negative emotion depicted by the actors, the events in the narrative and self-experienced (in response to the state of the other ). On a semantic scale of 1-5, with 1 least emotional to 5 being the maximum. 1. emotional expressions 2. emotional context 3. Self-experienced emotional response General rating on the movie/narrative. Figure A6.1: The overall rating from the fmri and the independent participants. 171

173 A6.2: Scene Specific To support the selection of the scenes/clip from the longer movie as having lower emotional context or Neutral and higher emotional context, the fmri participants were asked to score on a semantic scale of 1:5 with 1 being the least and 5 being the maximum. Table A6.1: The rating from the fmri participants, post-scanning for the specific scenes shown in the short clips, shows that the scenes were rated as Neutral or Emotional. (average rating by 16 participants) Classification Emotional Neutral M1:Actor driving the car Neutral Scene 1 5 M1: Mother being left behind bythe son, and she looking at him Emotional scene 4 1 M2: Father playing with the son on the tracks Neutral Scene 1 2 M2: Father crying on the train platform Emotional scene 4 1 The independent participants (non-fmri) of 40, were presented the clips in the same order as shown to the fmri participants, that is, Short Neutral clip Short emotional clip full movie. After each clip, the rating scale as above was presented. The independent participants rated the same scene (neutral or emotional) shown in the longer movie. The Table A6.2, shows the average values from the 40 participants. 172

174 Table A6.2: The average ratings on the neutral and emotional clips of movies M1, M2 by 40 independent participants. Movie: M1 Neutral Scene Short Clip: Actor driving the car 1 Full Movie: Actor driving the car 2 Emotional scene Short Clip: Mother being left behind bythe son, and she looking at him 2 Full Movie: Mother being left behind bythe son, and she looking at him 3 Movie: M2 Neutral Scene Short Clip: Father playing with the son on the tracks 1 Full Movie: Father playing with the son on the tracks 2 Emotional scene Short Clip: Father crying on the train platform 1 Full Movie: Father crying on the train platform 4 In addition to a rating on a fixed scale, subjective experience as also collected from the fmri participants. Description of specific scenes was given and the participants were asked to express their feelings. 173

175 Movie M1, Scenes described are: 1. The scenes/events that show the mother waiting on the bench, evoked the following feelings, the subjects noted the following responses: Anger, Disturbed, Sadness, or a mix of all these emotions. 2. When the young man walks over to talk to the mother, in the evening, you felt: Nice, Glad, Intrigued, Surprised. 3. The last scene in which the young man reads the note given to the mother by her son, evoked: Anger at the son, Extreme sadness or a mix of both. Movie M2: 1. The scenes which show the father playing with his son near the lake, was: Touching, Reminded of own childhood, both 2. When the father, as bridge operator, decides to turn the lever to close the bridge even it means loss of his son, your feelings on his actions: Sad, but right thing to do; Anger at the father; Should have thought of saving both; a mix of all these feelings/responses. 3. The scenes after the train goes by, the subsequent search and his grief, was: Extremely disturbing; Sad. A6.3 Movie Story Movie 1( M1) Film name : These Times ( The film is directed by Ramy El Gabry, starring Khaled Megald, Abdallah Alnahas and Awatef Helmy and is set in Cairo, Egypt. The short 11 minute movie (8:52 minutes of the movie was considered for the study, the credits shown at the end of the movie were cut-off), opens with a scene of a young man dressed for office 174

176 standing in his balcony and sees an old woman being led by a man to a seat facing the waterbody in front of his house. The woman is instructed by her son to wait for him saying he has to go for some work. He gives her a piece of paper telling her that it has his contact details. The woman is shown sitting on the seat, watching people go by and also talking to a young child who halts at the seat. The man who sees her in the morning, is driving home (20secs, short neutral scene) from office in the evening and see her still sitting at the same place. He crosses the road and walks over to her. He starts a conversation and she mentions that she is waiting for her son to return. She also narrates how she will be seeing her grandson for the first time and shows the man the chain she bought for her grandson and a photograph. Being thirsty she accepts water from the man. He then offers to call her son, to find out the reason for the delay. Initially she hesitates as she does not want to trouble the son and the man too. But then relents and hands over the slip of paper. The man opens the paper and sees that the message written by the son is whoever finds this woman, please move her to an old age home. The last scene zooms in as the man s expressions is understood by the woman and her face reflects acute sorrow (14 seconds, short emotional clip). Movie 2 (M2) File Name: Most (re-titled The Bridge ), 4AA3vtN_nQjREGLsJIGO&index=3 ( The full length is 32 minutes, for the experiment 8:32 seconds was edited. The parallel narrative of a young drug-addict is not shown). Directed by Bobby Garabedian, written and produced by William Zabka had won the 2003 Palm springs best of festival-award and other nominations too. The film is about a father employed as the railroad drawbridge operator, who takes his eight-year old son to work one-day. They both walk along a railway track talking to the 175

177 engine room (20 s, short neutral clip), and the father tells his son to stay at the edge of a nearby lake and try fishing. A ship comes by and the bridge is drawn up. The bridge has a train track and has to be locked down for the train. On that day, the train happens to arrive early and the son notices the smoke from the steam engine and shouts a warning to his father. But the father does not hear it over the noise in the engine room. The son, runs towards the manual lever at the junction where the bridge rises up. The father looks out of the window and sees that his son is not near the lake, he then notices the train and realizes that his son is trying to close the lever. Meanwhile the boy, when attempting to move the lever, slips and falls on the gears. The father has to make the cruel decision whether to pull the lever down and save the lives of hundreds in the train or his son who will be crushed to death if the lever is pulled down. With great agony, he pulls the lever down and the train chugs by he rushes down the engine room towards the gear box and breaks-down. A women passenger looks out of the window of the train to see the father crying in anguish (14s, short emotional scene), oblivious of the sacrifice of the father has made. The movie ends, with the father in a new city and seeing the same woman cross the road with a little boy and realizes that life should go on. Tools: 1. SPM8 ( 2. Visualization tool: 176

178 Chapter 7 Conclusions 7.1 Summary Considering the challenge in defining exactly what constitutes stimuli or event independent Empathy, for this study we define empathy as a complex combination of cognitive understanding (cognitive empathy),emotional response (emotional empathy), sympathy, compassion and emotional contagion while being aware of the self-other distinction. The self-other merger happens when we experience other s emotion without realizing it that it is not due to our own personal state (emotional contagion) like reflexive crying triggered by the state of the protagonist in a movie. For narrative empathy, the operational definition considered for this study is : empathy for a fictional cinema narrative is the sharing, feeling and perspective-taking induced by comprehending the contextual factors for state of the other by the power of transportation afforded by the cinematic narration, where the self-other distinction is transient. By these broad definitions for empathy and narrative empathy, an effort to include the individual variations in the immersiveness or engagement and cinematographic differences is attempted. The work presented in this thesis was aimed at exploring the neural bases of empathy for naturalistically presented narratives. Towards this, a set of two functional magnetic resonance imaging (fmri) experiments using long movie clips (5-10 minutes) in a free-running design were conducted. Data-driven analytical method was applied to isolate the empathy specific networks. As visual narratives present events of constantly evolving and varying emotional valence the timevarying empathy response was derived by dynamic functional connectivity analysis. 177

179 Empirical studies of empathy have mostly applied simplistic static stimuli presented via a controlled experimental design, requiring the subject to make inferences on the state of the target ( other ) with minimal contextual information. While these studies are effective in identifying the core brain areas attributable to empathy response to simple events, they fall short in explaining the complex process underlying dynamic empathy response, which could comprise of some or all the following: mentalizing, self-other responses, affective-empathy, sympathy, compassion etc., as in real-life social interactions. Though, Keen (2006) differentiates narrative empathy to sympathy, as the later refers to an emotion that is related but does not mirror the target s feelings and the former enables the viewer/reader to reciprocate the emotions/sensations of the target, the responses are intertwined, that is, sympathy is a constituent of empathy. From the point of view of narrative perceivers, sympathy is further sub-divided (Giovannelli, 2009) into a) anticipatory when viewer sympathizes with the protagonists though he/she does not yet display emotions, b) conditional wherein the viewer having knowledge of something the protagonist does not possess, experiences sympathy and c) by proxy when the viewer sympathizes based on self-view of a situation which is not reciprocated by the protagonist. Similarly, the viewer can also empathize with future experiences or possible stages/states of the other. Giovannelli (2009), further adds that intensity of sympathetic responses proves the significance of empathy to sympathy. Hence, to study empathy response in all its complexity narratives provide the perceivers an engaging experience as in real life. Thus, the broad aims of the study were: a) to identify complex brain networks attributable to empathy response using multi-modal narratives, atypical of real-life situations. b) Use data-driven methods to extract the spatial networks and by using functional connectivity analysis investigate the time-varying empathy response in relation to the events in the stimuli. c) compare the empathy response to stimuli with minimal context to complete narrative. The goals set were important 178

180 contribution as most empathy studies to date have applied simplistic and noncontextual stimuli to differentiate empathy modes of affective-perspective taking and mentalizing. The first study (Chapter 4) provides evidence of distinct but overlapping empathy networks. For the animation movie, activation in the thalamus region, insula and the ventrolateral prefrontal area crucial for emotional empathy were not observed. The neural findings are in concurrence with the lower ratings on the emotional score for this clip from the survey. Activation in the middle prefrontal cortex region and precentral gyrus associated with cognitive and motor empathy was significant. The data suggests that for animation movie, activation in the cognitive and motor empathy networks are statistically more significant than the emotional empathy network. The findings point to the possibility that the viewer, while being able to take perspective of the situation of the actor(s), probably falls short of sharing the experiences as he/she distinguishes the actors as computer-generated agents. The Hollywood/English movie data shows considerable activity in the thalamus, insula, superior temporal gyrus and ventrolateral prefrontal cortex, areas attributed to emotional empathy. Motor and cognitive empathy networks covering the middle prefrontal cortex, post-central gyrus, temporal pole, ventrolateral prefrontal gyrus is also evident. This is in accordance to the ratings on emotional scenes by the subjects. The Hindi movie data indicate temporally coherent networks comprising areas from cognitive and motor empathy modes, covering the prefrontal, parietal and the superior temporal gyrus of temporal cortex. Distinct areas in the thalamus region and a larger cluster of activity in the insula comparable to the Hollywood movie were also identified. These active areas along with response in pars opercularis (part of the inferior prefrontal gyrus) and the superior temporal gyrus suggest activation of the emotional empathy network. The observed areas indicate 179

181 the presence of all the three empathy networks, as was the case for the Hollywood movie. The results from the data-driven analysis method show that activation in areas attributed to empathy networks is correlated to the features of the stimuli and importantly confirms theoretical understanding that fictional narratives does evoke empathy response in viewers. Though further studies with focused stimuli set is required to make a strong conclusion, the findings from this initial study indicate a difference in emotional empathy response to animated/computer generated actors to human actors even though the context is emotional. In study 2 (Chapter 5), the dynamic functional connectivity results were conducted to examine the stimulus driven time-varying engagement of the empathy networks. The reference network was an independent component with activation in the areas of a posterior default-mode network (DMN). The DMN has been shown to play a role in moral judgment (Reniers et al.,2012; Harrison et al., 2008), deactivation in any attention demanding tasks (Raichle and Snyder, 2007), social understanding (Schilbach et al., 2008; Laird et al., 2011) and self-referential mental activity (Gusnard et al., 2001). By considering the DMN as the reference/seed to conduct cross-correlation with networks of the cognitive, emotional and motor empathy areas, the aim was to explore the role of self-referral processing and attention attributed to DMN for each empathy mode. The DMN regions have been reported to exhibit deactivation for resource demanding tasks (Vincent et al.,2008; McKiernan et al., 2003) though in some cognitive activities higher activation is also reported (Andrews-Hanna, 2010; Spreng & Grady, The functional connectivity with identified emotional empathy networks shows lower and negative cross-correlation indicative of possible increase in focus (attention) for emotional events and hence reduction in self-reflections or self-referential analysis. Across all the three stimuli, a positive and higher correlation was evidenced for the 180

182 cognitive empathy networks with the default-mode network (DMN), supporting previous studies which has shown that the DMN is crucial for social understanding of others (review by: Li, Mai & Liu, 2014) by self-other differentiation. Additionally, the medial prefrontal cortex and posterior cingulate cortex part of the DMN are also attributed to cognitive empathy network. The functional connectivity analysis with varying emotional contagion, provided the first conclusive evidence of emotional empathy possibly inhibiting DMN. Though causality cannot be established with the data analytic method applied, an inference that emotional empathy inhibits mind-wondering (attributed to DMN) can be made. The findings suggest that dynamic functional connectivity analysis is an insightful tool to understand how regions or networks evolve and engage in real-life natural scene processing and may be helpful in investigating empathy deficiency in clinical conditions like schizophrenia, autism and in frontotemporal dementia. The main contribution of the work reported in this thesis to the research on the neuroscience of empathy, is the evidence of the neural networks for context/narrative processing to empathize with another. By using naturalistic experimental designs, compelling evidence was derived to show that empathy being a complex process involving many brain areas, a non-contextual stimulus does not sufficiently elicit empathy response. By comparing the networks activated for neutral and emotional scenes, presented as standalone short clips or embedded into the longer narrative, we found that cognitive and emotional empathy areas had higher activation for the neutral/emotional scenes in the longer narrative. Interestingly for emotional scenes within the longer narrative, the cognitive empathy attributed region of the middle frontal gyrus shared temporal coherence with emotional empathy areas, signifying a possible over-lapping of the affectiveperspective empathy network. The areas identified also conform to the anatomical 181

183 data which suggests a network connecting the insula, posterior parietal, superior temporal and inferior frontal network (Augustine,1996) with insula as the conduit to limbic areas for processing of emotional scenes ( Carr et al.,2003). Overt depiction by physical actions of distress and grief (Movie 2: M2) triggers activation in the motor cortex areas of empathy network while subtle facial expressions or body gestures (Movie 1: M1) does not evoke similar activation in this region, a factor which needs to be further examined as this infers that motor cortex firing for empathy, including reflexive imitation, is subject to actual biological motion of the other. The presence of an integrated empathy response comprising areas attributed to cognitive and motor networks implies that perspective and appraisal are critical for emotional empathy, which is in conformity to Feshbach s (1987) observation that empathy is the result of cognitive and affective empathy, also proposed by Blair (2005). From the experiment conducted to analyze the role of full-narrative in empathy response, the major findings are: the behavioral and the fmri data from two diverse movies (M1, M2) analyzed independently demonstrate that context enhances activation of distinct empathy networks for neutral and emotional scenes when watched in the longer movies as compared to the same scenes watched independently. Further, emotional empathy in a narrative context is a cognitive-driven cogent response and not limited to a response to a facial expression or non-contextual emotional scenes, though it is possible that human brain can fill or imagine a narrative either from previous experience or by being creative. 7.2 Proposed model of empathy based on the findings: Many definitions for empathy have been proposed (Chapter 1,2), debated and refuted but there has been no consensus derived from empirical evidence to show the distinction between other-oriented responses like sympathy, empathic concern, 182

184 compassion to that of empathy which is affective sharing as feeling with/as the other person. A modified model for narrative empathy is proposed (Figure 7.1). More specifically, I propose that sympathy, compassion and emotional contagion need to be understood as a mechanism of one s empathic engagement with the other. That is, empathy contributes to sympathy response manifested by altruism to alleviate the state of the other. The schematic is indicative of the complex processes attributed to narrative comprehension and the subsequent empathy response. For example, experiencing incongruent or congruent feelings to those depicted by the other can be a strong indication of emotional empathy. While perspective-taking, a cognitive process of understanding the other s feelings can drive or inhibit altruism and subsequently sympathetic response. Figure 7.1: The complex processes that comprise empathy are depicted, with the direction of the arrows showing probable inter-dependencies. The findings of the neural networks attributed to empathy in the studies presented in the thesis, confirms that empathy is a complex process involving both cognitive 183

185 processing and emotional response. The data also confirms that viewer readily takes the perspective of the other/actor in the fictional narrative and imaginatively experience similar feelings, supporting the transportation theory. But, the self-reports by the fmri participants also point to a clear sense of the viewer holding on the self-perspective even as he/she was deeply engaged in the narrative. That is, the participant/viewer while expressing anger at the action of one of the actors, simultaneously empathizes with the other (actor) as self-analytic processing distinguishes the two responses. The results are significant to show that a narrative which adds a semantic context to the viewer engages and disengages a complex network of areas of the brain while concurrently evolving, thus emphasizing the complexity of narrative empathy. In addition, the findings also show that neural representations are shared and motor actions rely on physical actions depicted by the other (also suggested by Preston and de Waal (2002)) as the viewer is aware that the events are fictional and hence constantly evokes the executive function to inhibit actual motor action. This is corroborated by evidence from the studies reported in this thesis, showing that motor cortex activation as being significantly higher in stimuli which had explicit physical movement or overt facial expressions compared to the movie clips with slow movement. Motor empathy, which includes the mirror-neuron network of the pre-post central gyrus was shown to dependent on the physical movement depicted by the actors, suggesting a possible stimulus specific activation rather than a core empathy network. By this model, the distinction of empathy responses when there exist a shared experiences (by similarity biases, personal experience of similar state) with the other and in the absence of it is elaborated by the empirical evidence for narrative (fictional) stimuli. A similarity (for example: real-actors versus computer generated or a self-experienced event being experienced by the other) leads to shared representations and hence higher emotional empathy response with possible action response or altruism. In non-similarity conditions, 184

186 cognitive empathy dominates though emotional empathy co-activates. An objective perspective leads to activation and/or inhibition of the executive control based on a feedback mechanism that regulates the emotional response in later case. A point for counter argument can be the cinematography or subjective appeal of the narrative. This can be tested by studying similar narratives produced in animation and with real-life actors. Considering the complexity of empathy responses, a modified model of empathy specific to narrative empathy is proposed (Figure 7.2). In this model I argue, based on the neural findings, though viewers evoked empathy response independent of familiarity or similarity bias, higher affective empathy for narrative empathy could be influenced by familiarity/similarity biases the viewer/reader and/or the selfother over-lap the viewer allows for emotional events in the narrative. When the other s experience is evaluated with respect to self-feelings (congruent or not) increased activation in the emotional empathy and motor areas supporting mimicking emotions or expressions as expressed by the other is observed. Figure 7.2: A model for narrative empathy, where subjective experiences are differentiated. Effective isomorphism (similarity) with the state of the other (actor) or even events in the narrative could lead higher response in the emotional empathy. The process of understanding the state of the other but not sharing the feelings could be due to lower isomorphism or regulation. The motor empathy though is 185

187 subject to the overt gestures/expressions in the stimuli to certain extent or higher emotional empathy that could lead to activation of motor empathy. In conclusion, we demonstrate for the first time a strong neural evidence of empathy network activation as a function of visual narratives confirming that fictional narratives have the potential to evoke empathy response like real-life situations. The neural correlates of empathy using stimuli like movies is complex considering that the viewer is cognizant that events are fictional and could be applying a control mechanism to reiterate this to self but an immersive narrative evokes empathy though a self-other distinction is constantly being reinforced as established from the distinct but co-activated empathy networks the research study was able to identify. Future scope and studies The findings from this study provide the foundation to the use of narrative to induce empathy changes, that is, as a tool for positive social perceptions. Text or audio-visual narratives of suffering by humans with whom we share no ethnic or cultural relation evoke strong empathic and associated responses in us. This extends to narratives of non-human animal species too. Systematic research is required to understand the effect narratives have in development or formative years in humans. The findings might result in a more empathic world. A team of us have started looking at social perceptions influence on rape victim empathy, and initiated survey and fmri studies using narratives of rape. We hope that these findings, first of its kind in the country, will give insights into the empathy response differences light on the gender driven empathy (female rape victim as perceived by male participant) and the profession (bar dancer, doctor, teacher etc.,) of the victim. A second study is designed using faces labelled as rape victims from different social status and varying skin color tones and examining the comparative empathy networks. 186

188 Appendix Selecting task-relevant Independent Components by Visual Inspection, areas of activation and time course signal. Artifacts related to motion and physiological sources in fmri data can negatively impact the signal-to-noise ratio and importantly can give false-positives. The preprocessing applied is not effective in removal of noise that is convolved with the signal. The ICA method has shown the ability to separate neural signal sources of interest to that of structured or random noise into separate components. The decomposition of the fmri data into cognitive or task related and physiological components is the goal from a ICA method. The ICs or intrinsic networks are separate active areas coactivated with noise from subtle movements and/or cardiac respiratory pulsations. The selection of the number of the components in ICA packages is by the Minimum Description length (MDL) and Bayesian Inference criterion (BIC)- GIFT or Laplacian method in probabilistic ICA (Melodic, part of FMRIB's FSL package. The ICA method isolates artifact noise into components with time courses showing slowly varying fluctuations (McKeown et al., 1998), as reproduced in figure A1. For the ICA methods applied on the free-design paradigm fmri data collected using long continuous stimulus, the MDL method was used to initially estimate the number of components. From the ICs just estimates, task-relevant areas were identified by visual inspection (based on the reference provided by McKeown et al., (1998)) and by the areas labeled from the MNI coordinates for each component after applying the 1 Sample T-test at p<0.001 and minimum cluster size of 10 voxels. The 22 ICs thus derived for movie M2 (Chapter 6) along with the time courses and areas of activation is shown in the Figures A2:A7. 187

189 Figure A1: Different classes of components detected by ICA decomposition of Stroop task fmri data.. Negative z values mean those voxels are activated opposite to the plotted time course. (Reproduced from McKeown et al., 1998). Group ICA on the fmri data collected for movie M2 (Chapter 6). Figure A2: Unsorted components IC1 to IC4, the time courses of the four ICs and the areas of activation in each of the ICs 188

190 . Figure A3: ICs 5 to IC8. Figure A4: ICs 9 to IC

191 Figure A5: ICs 13 to IC 16. Figure A6: ICs 17 to ICs

192 Figure A7: IC 21 and IC 20. The areas of activations from each of the ICs provide conclusive data for taskrelevant activation. The time courses do not show the clear fluctuation difference as reported by McKeown et al., 1998, for the Stroop test task, as the stimulus was presented in free-viewing format continuously and hence task-related activation ICs also include noise signals from activation in white matter and also areas like the Declive, Culmen and Cerebral Spinal Fluid. From the ICs shown above, ICs21,22 (Insula,Frontal cortex, Temporal Gyrus) and ICs 15,16 (Mostly the Frontal gyrus) would be of interest for empathy networks. The ICs with activation in visual cortex, parahippocampal gyrus and fusiform gyrus are important to confirm visual and cognitive processes like face recognition by the viewers. 191