An investigation of basic facial expression recognition in autism spectrum disorders

Transcription

1 Cognition and Emotion ISSN: (Print) (Online) Journal homepage: An investigation of basic facial expression recognition in autism spectrum disorders Simon Wallace, Michael Coleman & Anthony Bailey To cite this article: Simon Wallace, Michael Coleman & Anthony Bailey (2008) An investigation of basic facial expression recognition in autism spectrum disorders, Cognition and Emotion, 22:7, , DOI: / To link to this article: Published online: 15 Oct Submit your article to this journal Article views: 354 View related articles Citing articles: 32 View citing articles Full Terms & Conditions of access and use can be found at Download by: [The University of British Columbia] Date: 23 September 2015, At: 08:44

2 COGNITION AND EMOTION 2008, 22 (7), An investigation of basic facial expression recognition in autism spectrum disorders Simon Wallace Department of Psychiatry, University of Oxford, Oxford, UK Michael Coleman University College London, London, UK Anthony Bailey Department of Psychiatry, University of Oxford, Oxford, UK This study was designed to test three competing hypotheses (impaired configural processing; impaired Theory of Mind; atypical amygdala functioning) to explain the basic facial expression recognition profile of adults with autism spectrum disorders (ASD). In Experiment 1 the Ekman and Friesen (1976) series were presented upright and inverted. Individuals with ASD were significantly less accurate than controls at recognising upright facial expressions of fear, sadness and disgust and their pattern of errors suggested some configural processing difficulties. Impaired recognition of inverted facial expressions suggested some additional difficulties processing the facial features. Unexpectedly, the clinical group misidentified fear as anger. In Experiment 2 feature processing of facial expressions was investigated by presenting stimuli in a piecemeal fashion, starting with either just the eyes or the mouth. Individuals with ASD were impaired at recognising fear from the eyes and disgust from the mouth; they also confused fearful eyes as being angry. The findings are discussed in terms of the three competing hypotheses tested. Deficits in recognising and producing facial affect partly characterise the social impairments in autism (ICD-10; World Health Organization, 1992) and a substantial body of research has attempted to identify what causes these deficits. The conclusions of studies that tested recognition of basic facial expressions (anger, disgust, fear, surprise, sad, happy, neutral; Ekman Correspondence should be addressed to: Simon Wallace, Department of Psychiatry, University of Oxford, Warneford Hospital, Headington, Oxford OX3 7JX, UK. simon.wallace@psych.ox.ac.uk This study was funded by an MRC UK studentship for Simon Wallace. We would like to thank all those who gave their time to participate in this study. We would also like to thank Professor Paul Ekman for giving us permission to use his stimuli. # 2008 Psychology Press, an imprint of the Taylor & Francis Group, an Informa business DOI: /

3 1354 WALLACE, COLEMAN, BAILEY & Friesen, 1976) in individuals with autism spectrum disorders (ASD) have ranged from the suggestion that deficits are a primary cause of impaired interpersonal relations (Hobson, 1991) to claims that there are no significant difficulties in this domain (Adolphs, Sears, & Piven, 2001). The findings of previous research (reviewed below) could be best summarised as evidence of impaired perceptual processing of facial expressions (the configural processing hypothesis), difficulties recognising complex mental states (Theory of Mind hypothesis) or specific impairments in fear recognition (atypical amygdala functioning hypothesis). The aims of this study were to clarify the nature of the putative impairment in facial affect recognition and to explain the pattern of findings in the context of existing hypotheses. Configural processing hypothesis Neuropsychological models of face processing in typical development have proposed that the structural encoding of facial identity and facial expressions use a shared specialised cognitive strategy (Bruce & Young, 1986; Calder & Young, 2005; Haxby, Hoffman, & Gobbini, 2000). There is no consensus about the nature of this cognitive strategy (Carey & Diamond, 1977; Diamond & Carey, 1986; Farah, Wilson, Drain, & Tanaka, 1998; Maurer, Le Grand, & Mondloch, 2002; Rakover, 2002; Sergent, 1984; Tanaka & Farah, 1993; Yovel & Kanwisher, 2004) but Maurer et al. (2002) outlined three types of configural processing that are integral to the development of typical expertise in face processing: first-order configural processing (the prototypical facial form of eyes, above the nose, above the mouth); second-order configural processing (the spatial arrangement of these features); holistic processing (processing the whole stimulus without part decomposition or as a perceptual snapshot). When the orientation of face stimuli is inverted the ability to process configural properties is disrupted and the observer becomes reliant on a feature or piecemeal processing strategy (Farah et al., 1998; Valentine, 1988; Yin, 1969). The importance of configural processing to the successful recognition of facial expressions has been highlighted in a number of previous studies (Calder, Young, Keane, & Dean, 2000; Calder & Jansen, 2005; Ekman & Friesen, 1976; McKelvie, 1995). Typically developing individuals show consistent patterns of errors when misidentifying pairs of facial expressions that share similar physical properties (e.g., anger and disgust; fear and surprise; sad and neutral; Russell, 1980; Russell & Bullock, 1985; Schlosberg, 1941, 1952; Young et al., 1997) and it has been argued that a configural processing strategy is necessary to discriminate these pairs of expressions apart and identify them accurately (McKelvie, 1995). When faces are inverted and a feature processing strategy is relied upon, there is an increase in the frequency of these misidentification errors being made (McKelvie, 1995).

4 FACIAL AFFECT RECOGNITION IN ASD 1355 In individuals with ASD, impaired configural processing and an overreliance on feature processing has been suggested to underlie the difficulties in processing facial affect (Celani, Battachi, & Arcidiacono, 1999; Davies, Bishop, Manstead, & Tantum, 1994; Teunisse & de Gelder, 2001). In support of this hypothesis, children with autism are reported to be either more accurate than controls at processing inverted facial expressions (Hobson, Ouston, & Lee, 1988) or fail to show the typical drop in affect recognition performance associated with face inversion (Tantam, Monaghan, Nicholson, & Stirling, 1989), although significant inversion effects have recently been reported in tests of face identity (Teunisse & de Gelder, 2003; Lahaie et al., 2006). In studies that examined patterns of errors when misidentifying facial expressions, Castelli (2005) reported no differences between children with and without ASD, whereas Pelphrey et al. (2002) found that adults with ASD confused angry expressions as being fearful more often than typically developing controls. Gross (2004) argued that some individuals with ASD may show atypical patterns of response errors to facial expressions because they look at the lower regions of the face. This suggestion is consistent with cognitive (Joseph & Tanaka, 2003; Langdell, 1978) and visual scan path (Klin, Jones, Schultz, Volkmar, & Cohen, 2002) studies, which reported that individuals with ASD atypically pay more attention to the lower rather than the upper part of the face. Theory of Mind hypothesis Studies of facial affect recognition in clinical and typically developing samples have traditionally used the basic facial expression (anger, disgust, fear, surprise, sad, happy, neutral) stimuli developed by Ekman and Friesen (1976) and Matsumoto and Ekman (1988). Along with an improved understanding of the processes underlying the development of Theory of Mind (the ability to attribute mental states to oneself and to other people; Brothers, 1990; Premack & Woodruff, 1978), there has been a broadening of the facial expressions tested to include complex mental states such as distrust or regret (Baron-Cohen et al., 1996; Baron-Cohen, Wheelwright, & Jolliffe, 1997). Baron-Cohen (1995) set out his hypothesis of impaired Theory of Mind (ToM) in autism and in support of this account he developed a number of cognitive assessments of empathising skill that involved aspects of face processing (Baron-Cohen, Spitz, & Cross, 1993; Baron-Cohen, Campbell, Karmiloff-Smith, Grant, & Walker, 1995; Baron-Cohen et al., 1997; Baron-Cohen, Wheelwright, Hill, Raste, & Plumb, 2001). Probably the best known is the Reading the Mind in the Eyes test (Baron-Cohen et al., 1997, 2001), on which both individuals with ASD (Baron-Cohen et al., 1997, 2001) and their parents (Baron-Cohen & Hammer, 1997) show impaired recognition of mental states expressed in the eyes. These findings have been

5 1356 WALLACE, COLEMAN, BAILEY interpreted as evidence that individuals with ASD are impaired at recognising complex mental states because of the load on ToM abilities, whereas recognition of basic facial expressions does not have such a ToM load and so performance is intact (Adolphs et al., 2001; Baron-Cohen et al., 1997; Golan, Baron-Cohen, & Hill, 2006). It is difficult to reconcile the claims of the ToM hypothesis with reports of impaired recognition and perception of basic facial expressions by individuals with ASD (Celani et al., 1999; Hobson, 1986a,b, 1991; Hobson et al., 1988; Howard et al., 2000; Pelphrey et al., 2002; Tantam et al., 1989; Teunisse & de Gelder, 2001; Weeks & Hobson, 1987). As suggested by Castelli (2005) the use of heterogeneous experimental designs, stimulus sets and participant groups has led to much inconsistency in the pattern of findings reported. Moreover, there remains no clear demarcation between what constitutes a complex and a basic affective state. For example, Baron- Cohen et al. (1993) reported impaired recognition of surprise (as a complex cognitive emotion) but intact recognition of happy and sad (basic emotions) in children with ASD, whereas Baron-Cohen et al. (1997) later reported intact recognition of surprise as a basic emotion in adults with ASD. The atypical amygdala functioning hypothesis One failing of early studies of how individuals with ASD process and recognise facial expression was that not all six basic emotions were tested (e.g., Baron-Cohen et al., 1993; Davies et al., 1994; Hobson et al., 1988). The previous lack of emphasis on characterising the full extent of facial affect processing deficits is an important omission, as individuals with ASD may be impaired at recognising some expressions and not others: a pattern occasionally seen in individuals with localised neurological damage (Adolphs, Tranel, Damasio, & Damasio, 1995; Sprengelmeyer et al., 1996). Indeed, the findings from functional imaging studies of typically developing individuals and studies of patients with lesions suggest that some regions of the brain may mediate general processing of facial affect (Adolphs, Damasio, Tranel, & Damasio, 1996; Adolphs, Damasio, Tranel, Cooper, & Damasio, 2000), whereas others may underlie the processing of specific emotions (Phillips et al., 1997; Whalen et al., 1998). A particular focus has been the role of the amygdala in typical recognition of fearful facial expressions (Adolphs et al., 1995; Morris, Öhman, & Dolan, 1999; Whalen et al., 1998); although there is also limited evidence to suggest that the amygdala may have a role in recognising other emotions (Adolphs et al., 1995; Young et al., 1995) or making complex social judgements about faces (Adolphs, Tranel, & Damasio, 1998).

6 FACIAL AFFECT RECOGNITION IN ASD 1357 Current theories of atypical amygdala functioning in individuals with ASD (Bachevalier, 2000; Baron-Cohen et al., 2000) would predict specific impairments in fear recognition, a pattern of findings that has been reported in some (Howard et al., 2000; Pelphrey et al., 2002) but not all studies (Adolphs et al., 2001; Baron-Cohen et al., 1997; Castelli, 2005). The general role that the amygdala has in responding to threatening stimuli (Adolphs, Russell, & Tranel, 1999) may explain why children with ASD confuse angry faces as being fearful (Pelphrey et al., 2002). Functional imaging studies have reported atypical activation of the amygdala as individuals with ASD implicitly process facial expressions (Critchley et al., 2000) and while they complete the Reading the Mind in the Eyes Task (Baron-Cohen et al., 1999). Howard et al. (2000) associated atypical structural development of the amygdala with poor performance at recognising fearful faces and judging eye gaze direction; although how atypical development of the amygdala affects these two aspects of face processing remains unclear. Current aims One of the major aims of this study was to examine the ability of a large group of adults with ASD to recognise basic facial expressions. Previous studies of basic facial expression recognition in ASD have often used small samples (e.g., N8 in Adolphs et al., 2001) without enough statistical power to be confident about making group comparisons. EXPERIMENT 1 In Experiment 1 predictions of group performance were based on the three competing hypotheses described in the introduction: 1. Configural processing hypothesis: a. Individuals with ASD will not show significant inversion effects in recognising facial expressions. b. Individuals with ASD will misidentify pairs of facial expressions that share similar physical properties (anger and disgust, fear and surprise, sad and neutral) significantly more frequently than controls. 2. Theory of Mind hypothesis: a. Individuals with ASD will not be impaired relative to controls at recognising basic facial expressions. 3. Atypical amygdala functioning hypothesis:

7 1358 WALLACE, COLEMAN, BAILEY a. Individuals with ASD will be significantly impaired compared to controls at recognising faces expressing fear. Methods Participants The aim was to generate statistical power.8 and our sample size of 56 participants (28 clinical/28 control) generated power of.84, using ANOVA and with alpha set to.05. The clinical group (individuals with ASD) comprised twenty-eight (26 male) older adolescents and adults (age range: 1651). Two of the clinical participants were not included in the analyses because of concerns about the reliability of their diagnosis. The majority of the clinical participants had been referred to the Maudsley Hospital in London and had received their diagnosis from a psychiatrist; the remainder had received their diagnoses from psychiatrists and clinical psychologists at other hospitals in the UK. There were 15 individuals with a diagnosis of autism and 11 with a diagnosis of Asperger syndrome, according to ICD-10 criteria (World Health Organization, 1992), but the samples were not large enough nor matched on IQ to enable the making of useful comparisons of face processing performance between autism and Asperger syndrome. The diagnoses of 22 of the 26 individuals (23 male) included in the final analysis were confirmed using the Autism Diagnostic InterviewRevised (ADI-R; Lord, Rutter, & LeCouteur, 1994). All these individuals scored above algorithm cut-off on the social (10), communication (8) and repetitive (3) domains from the ADI-R. Four ADI-Rs could not be completed because either the parents were too ill or had died. Twenty-eight (25 male) typically developing control participants (age range: 1654) were recruited via advertising in the south east area of London. The data from 2 control participants were not analysed because their performance IQ was below 75, leaving 26 (23 male) participants included in the final analyses. Informed written consent was obtained from all participants. Twenty-six individuals with ASD were matched with 26 non-clinical controls using Raven s Progressive Matricesstandard edition (RPM; Raven et al., 1992) and the British Picture Vocabulary Scale (BPVS; Dunn et al., 1982), as performance and verbal measures respectively. The groups were also matched for chronological age and handedness (from self-report of hand preference). Independent t-tests showed that there were no significant group differences on RPM IQ or chronological age; because of ceiling effects a MannWhitney U-test was used to show no group differences in BPVS score (Table 1).

8 FACIAL AFFECT RECOGNITION IN ASD 1359 TABLE 1 Participant details for Experiments 1 and 2. Independent samples t-test analyses showed that there were no significant group differences in age (years) or measures of IQ. MannWhitney test showed no group differences on BPVS raw score Group Mean (SD) Range N Experiment 1 Age in years Control 31 (9) Clinical 32 (9) BPVS a raw score Control 153 (9) Clinical 148 (13) RPM b IQ Control 98 (12) Clinical 101 (18) Experiment 2 Age in years Control 30 (7) Clinical 30 (9) Verbal IQ c Control 115 (14) Clinical 118 (14) Performance IQ c Control 116 (13) Clinical 122 (7) Full IQ c Control 117 (13) Clinical 122 (10) Notes: a British Picture Vocabulary Scale (Dunn, Dunn, & Whetton, 1982). b Ravens Progressive Matrices (Raven, Court, & Raven, 1992). c Verbal, performance and full IQ derived from Wechsler Abbreviated Scale of Intelligence (WASI; Wechsler, 1999). Stimuli One hundred twelve stimuli were selected from the Ekman and Friesen (1976) and Matsumoto and Ekman (1988) pictures of seven facial expressions: happy, sad, anger, disgust, fear, surprise and neutral (see Figure 1). The experimental stimuli were assigned to one of two stimulus sets: A or B. Each set contained fifty-six pictures (eight poses for each of the seven affective states). An oval crop was applied to each stimulus to mask the hair Figure 1. Examples of the stimuli used in Experiment 1. Stimuli were taken from the Ekman and Friesen (1976) and Matsumoto and Ekman (1988) pictures of facial affect.

9 1360 WALLACE, COLEMAN, BAILEY and ears, so that the non-affective facial features would not distract the clinical participants (Weeks & Hobson, 1987). Flipping the face about its vertical axis produced an inverted face. The flipped version was saved into a directory of inverted faces. There were four sets: set A; set A inverted; set B and set B inverted. There were 56 pictures in each of the four sets, each measuring pixels. Procedure Participants completed a number of face and visual processing tasks but only the data from the facial expression recognition experiment are presented here. Participants sat at a viewing distance of approximately 55 cm from the computer monitor. On each trial a red fixation cross was shown for 350 ms before a picture was presented for 2000 ms. The exposure duration of the test stimulus was based upon pilot data that replicated the level of performance in typically developing individuals previously reported (McKelvie, 1995). To limit after-processing of the test image, the face picture Figure 2. Procedure of Experiment 1. A red fixation cross appeared for 350 ms, followed immediately by a target face for 2000 ms. The face was replaced by a mask for 350 ms and then a response form appeared until the participant made a decision by using the mouse to click on the text box. There was a 1500 ms interval from when the participant made a response to when the next trial began.

10 FACIAL AFFECT RECOGNITION IN ASD 1361 was immediately replaced by a pattern mask (Costen, Shepherd, Ellis, & Craw, 1994) displayed for 350 ms. When the mask disappeared, the names of the seven basic facial expressions were presented in text boxes on the screen. Once the participant had responded by using the mouse to click on their chosen text box, the next trial began after an interval of 1500 ms. An example of the trial sequence is shown in Figure 2. Experimental stimuli were presented by programs using the bit block transfer functions of the Windows Application Programmer s Interface (API). The experimental program was run using a PC and the stimuli were presented on a Vision Master pro 400 monitor. Participants completed 9 practice trials and 112 experimental trials (56 upright; 56 inverted); a short break was taken between presentations of the two stimuli sets. Participants were told that they would see pictures of faces on the computer screen and that their task was to identify the emotion expressed by each face. The examiner named the seven target facial expressions and all participants were able to give an approximate definition of each emotional state or a synonym. Participants were told that when the target face disappeared from the computer screen it would be replaced by the names of the seven expressions. The task was to use the computer mouse to click on the name most closely describing the emotional state they had seen. Participants were told that they would perform the task twice, once with the faces presented upright and once with them presented upside down. Participants in each group were counterbalanced on whether they saw set A or set B first and whether the set was presented upright or inverted. Results The 2 dependent variables were mean percentage accuracy at correctly recognising each facial expression and mean error rates when facial expressions were misidentified. Accuracy scores on a trial were not included when a participant s reaction time was more than four times the length of their mean reaction time. Previous studies of face processing have shown slower response times in individuals with ASD compared to typically developing controls (Teunisse & de Gelder, 2003; Wallace, Coleman, Pascalis, & Bailey, 2006) and so between-group comparisons using reaction times were not made. Parametric tests were used as the group means and were normally distributed. Planned group comparisons were made to test the competing experimental hypotheses outlined above. Regression analyses of recognition accuracy with measures of cognitive ability Linear regression analyses revealed no relationship in the control group between either BPVS raw score or RPM IQ and mean percentage

11 1362 WALLACE, COLEMAN, BAILEY TABLE 2 Regression analyses (R scores): between cognitive ability (BPVS raw score & RPM IQ) and mean percentage accuracy for recognition of the 7 facial expressions (upright and inverted) Group Cognitive measure Expression recognition upright Expression recognition inverted Control BPVS RPM Clinical BPVS.467**.418* RPM Notes: *pb.05; **pb.025. recognition accuracy for identifying facial expressions presented either upright or inverted. A significant relationship was found in the clinical group, however, between language ability and mean percentage recognition accuracy for identifying upright, R 2.218, F(1, 25)6.695, pb.05, and inverted, R , F(1, 25)5.067, pb.05, expressions, but there was no relationship with performance IQ (see Table 2). Consequently BPVS raw score was included as a covariate in all group comparisons. Group performance at recognising facial expressions A repeated measures ANCOVA of percentage accuracy recognising facial expressions as a function of Orientation (upright/inverted) and Facial Expression (anger, disgust, fear, happy, surprise, sad and neutral), which included BPVS as a covariate, revealed no main effects of Orientation or Facial Expression. A non-significant trend of Group by Orientation was observed, F(1,49)3.455, p.069, as inverting stimuli reduced the accuracy of the control group more than it did the clinical group. Exploration of inversion effects in the two groups using paired samples t-tests revealed a significant drop in recognition accuracy in both the control, t(25)15.019, pb.0001, and clinical groups, t(25)10.409, pb.0001, when faces were inverted. Summed mean percentage recognition accuracy across all seven facial expressions was calculated and the control group was significantly more accurate than the clinical group at recognising expressions presented upright, F(1, 49)19.452, pb.0001, and inverted, F(1, 49)4.895, pb.05. There was a 3-way interaction of Group by Orientation by Facial Expression, F(6, 45)3.231, p.01. Analysis of individual plots (GroupOrientation for each facial expression) revealed a Group Orientation interaction for recognition of sadness, F(1, 49)19.028, p.0001, as the control group was significantly more accurate than the clinical group at recognising sadness in the upright condition, F(1, 49)12.240, pb.001,

12 FACIAL AFFECT RECOGNITION IN ASD 1363 whereas the clinical group was more accurate at recognising sadness in the inverted condition, F(1, 49)2.292, ns. MANCOVAs were used to explore whether individuals with ASD were significantly impaired at recognising any of the individual facial expressions. The control group was significantly more accurate than the clinical group at identifying expressions of fear, F(1, 49)12.782, pb.001, disgust, F(1, 49) 9.187, pb.001, and sadness, F(1, 49)12.240, pb.001, in the upright condition. In the inverted condition the control group was significantly more accurate at recognising neutral expressions, F(1, 49)11.827, pb.001 (see Table 3). Analyses of error patterns Table 4 shows the mean correct and incorrect responses of both groups when identifying upright facial expressions. For example, the top row of Table 4 shows that the control group recognised anger correctly in 73% of trials but on 10% of the trials they mistook anger as disgust (the incorrect response most frequently made for each expression is highlighted in bold in Table 4). A 77 repeated measures ANCOVA on the mean percentage response was conducted as a function of the emotion to be recognised (Target Emotion) by the actual response that was made (Response Emotion). There was no significant Group interaction with Response Emotion, F(6, 44) There was a significant Group interaction of Target Emotion with Response Emotion, F(36, 14)4.388, pb.05. Analysis of the group means using an ANCOVA revealed that the clinical group misidentified fearful expressions as being angry significantly more frequently than the control group, F(1, 49)13.169, p.001. The clinical group did not consistently mistake angry faces as fearful (see Table 4). The ANCOVA showed that contrary to the second prediction of the configural processing hypothesis, the clinical group did not mistake fear for surprise, or anger for disgust, significantly more often than controls. The clinical group did, however, confuse disgust for anger, F(1, 49)4.534, pb.05, and sadness for neutral, F(1, 49)4.745, pb.05, significantly more often than controls. Discussion The main aim of Experiment 1 was to test three competing hypotheses of how individuals with ASD recognise facial expressions. Contrary to the first prediction of the configural processing hypothesis, individuals with ASD showed a significant inversion effect in recognising facial expressions. Notably, although a larger inversion effect was found in the control group, the scale of the inversion effect shown by the clinical group strongly suggests a different strategy for processing upright compared to inverted faces. Partly in line with the second prediction of the configural processing hypothesis, the

13 TABLE 3 Group mean recognition accuracy for correctly identifying each facial expression, when presented either upright or inverted (standard deviations are in parentheses). When BPVS score was included as a covariate, the control group was significantly more accurate than the clinical group at recognising disgust, fear and sadness in the upright condition and neutral in the inverted condition Anger Disgust Fear Happy Surprise Sadness Neutral Upright Inverted Upright Inverted Upright Inverted Upright Inverted Upright Inverted Upright Inverted Upright Inverted Control 73 (22) 47 (24) 88 (12) 52 (29) 85 (17) 47 (30) 99 (7) 99 (3) 84 (22) 70 (27) 86 (11) 28 (18) 89 (15) 82 (15) Clinical 62 (17) 43 (25) 72 (23)* 42 (24) 62 (24)* 33 (23) 93 (13) 91 (17) 68 (22) 56 (28) 68 (22)* 35 (21) 77 (26) 58 (27)* Notes:*pB.001. Asterisks indicate that the clinical group was significantly impaired in mean recognition accuracy performance compared to the control group (the value in the above cell) WALLACE, COLEMAN, BAILEY

14 TABLE 4 The mean percentage correct and incorrect responses made for each target emotion (standard deviations in parentheses). It can be seen from the top row that the control group recognised angry faces correctly on 73% of the trials and incorrectly responded that angry faces were disgusted on 10% of the trials. Highlighted in bold is the incorrect response that was made most frequently by each group, for facial expressions presented upright Anger Disgust Fear Happy Surprise Sadness Neutral Target emotion: ANGER upright Control 73 (17) 10 (12) 6 (8) 0 3 (5) 5 (9) 3 (3) Clinical 62 (22) 20 (19) 5 (9) 0 2 (4) 7 (10) 4 (4) Target emotion: DISGUST upright Control 11 (12) 88 (12) (2) 0 0 Clinical 18 (15) 72 (22) 2 (8) 1 (2) 1 (3) 4 (10) 2 (5) Target emotion: FEAR upright Control 1 (4) 1 (5) 85 (17) 0 10 (13) 2 (4) 1 (5) Clinical 17 (19) 6 (10) 62 (24) 1 (3) 10 (14) 2 (4) 2 (5) Target emotion: HAPPY upright Control (7) 1 (5) 0 0 Clinical 0 1 (2) 0 93 (13) 4 (10) 0 2 (5) Target emotion: SURPRISE upright Control (24) 0 84 (23) 1 (3) 2 (6) Clinical 2 (7) 1 (3) 22 (17) 3 (8) 68 (22) 1 (3) 3 (7) Target emotion: SADNESS upright Control 1 (3) 0 3 (8) 0 5 (19) 86 (20) 5 (8) Clinical 7 (12) 5 (9) 6 (9) 0 2 (6) 68 (21) 12 (14) Target emotion: NEUTRAL upright Control 2 (5) (6) 0 6 (10) 89 (15) Clinical 7 (14) 1 (4) 3 (6) 1 (4) 3 (8) 8 (11) 77 (25) FACIAL AFFECT RECOGNITION IN ASD 1365

15 1366 WALLACE, COLEMAN, BAILEY clinical group made significantly more errors confusing disgust for anger and sadness for neutral compared to the control group, a pattern of errors that McKelvie (1995) proposed was evidence of impaired configural processing. Quite why significant group differences were not observed in the frequency of confusing anger for disgust or neutral for sadness is unclear, although possibly it is because significant group differences were not found in recognition accuracy of anger and neutral, reducing the probability of identifying significant group differences in error patterns. The pattern of overall recognition accuracy (mean performance across all seven expressions) showed that the control group was significantly more accurate than the clinical group at recognising upright facial expressions, which could be accounted for by either absent or impaired configural processing. The fact that the clinical group showed a significant inversion effect and a generally similar pattern of errors compared to the control group would suggest impaired rather than absent use of configural processing. An unpredicted finding from Experiment 1 was that the clinical group was less accurate than the control group at recognising inverted facial expressions. Despite research into facial expression recognition in typical development being traditionally focused on the importance of configural processing (Calder et al., 2000; Calder & Jansen, 2005; Ekman & Friesen, 1976; White, 1999), there is good evidence to suggest that some expressions can be recognised almost as well using a feature processing strategy (Ellison & Massaro, 1997); in particular, happiness from the mouth (Meulders, De Boeck, Van Mechelen, & Gelman, 2005), sad from the eyes and cheek (Kohler et al., 2004) fear from the eyes (Adolphs et al., 2005; Emery, 2000; Kohler et al., 2004; Mandal, Pandey, & Madan, 1992; Smith, Cottrell, Gosselin, & Schyns, 2005; Whalen et al., 2004) and disgust from the region around the nose and mouth (Hu et al., 1999; Smith et al., 2005). The finding in Experiment 1 that individuals with ASD have difficulties recognising inverted facial expressions suggests that they have feature processing deficits in addition to the more commonly reported configural processing ones. There was conflicting evidence as to whether individuals with ASD use different strategies to recognise upright and inverted facial expressions. Although individuals with ASD did show a significant inversion effect, suggesting the use of different strategies depending on the orientation of the face, a linear relationship was found in the clinical group between language ability and recognition of facial expressions, independent of stimulus orientation, suggesting some commonality in the processing strategy adopted in recognising upright and inverted expressions. It has previously been suggested that individuals with ASD use a strategy when looking at facial expressions that is partially verbally mediated (Grossman, Klin, Carter, & Volkmar, 2000); alternatively, it may be that verbal IQ may correlate with the severity of the underlying neurobiological impairment.

16 FACIAL AFFECT RECOGNITION IN ASD 1367 What is clear, however, is that the impaired recognition of facial expressions in Experiment 1 was not attributable to diminished verbal ability, which was included in the analyses as a covariate. The Theory of Mind hypothesis predicted that individuals with ASD would not show impairments in recognising any of the seven basic emotional states tested here. Contrary to this prediction, the clinical group showed both a general impairment in recognising facial expressions (mean percentage accuracy summed across all seven expressions) and also significant impairments in recognising fear, disgust and sadness. The current finding of impaired sadness recognition is in direct contrast to Baron-Cohen et al. s (1993) report of impairments in cognitive emotional states such as surprise but intact recognition of simple emotions such as sadness. The finding that adults with ASD have difficulty in recognising sadness (a cognitively simple emotion) suggests that deficits in facial affect recognition are not confined to processing complex cognitive states. The discrepancies between our findings and previous reports of intact recognition of the seven basic emotional states might be explained by the inadequate statistical power of some previous studies and the failure to restrict exposure duration of stimuli, thereby allowing participants to use alternative processing strategies. The third hypothesis tested was the atypical amygdala functioning hypothesis, which predicted specific impairments in recognising fearful facial expressions. The findings in Experiment 1 of impaired recognition of sadness and disgust are not in line with this hypothesis, but there was evidence that impaired fear recognition was qualitatively different from impaired sadness and disgust recognition. Whereas the control group showed the usual error pattern of misidentifying fear as surprise (McKelvie, 1995; Russell, 1980; Russell & Bullock, 1985; Schlosberg, 1941, 1952; Young et al., 1997), the clinical group atypically misidentified fear as anger. This atypical misidentification made by individuals with ASD could be related to clinical observations and experimental studies that report greater attention to the lower parts of the face (Klin et al., 2002; Langdell, 1978) and reduced looking toward the eyes (Hobson & Lee, 1998; Klin et al., 2002). Behavioural studies in humans (Adolphs et al., 2005; Mandal et al., 1992; Smith et al., 2005; Whalen et al., 2004) and non-human primates (Emery, 2000) suggest that fear is best recognised from the eye region. Therefore, if individuals with ASD attend more to the mouth they may mistake fearful faces as being angry. Alternatively, along with a previous report of adults with ASD mistaking angry faces as fear (Pelphrey et al., 2002), the finding from Experiment 1 of fear being confused as angry may suggest atypical processing of threat-related stimuli. This explanation would be more in line with the atypical amygdala functioning hypothesis as a number of functional imaging studies of typical populations and reports of individuals with neurological damage have suggested that the amygdala has a specific role in

17 1368 WALLACE, COLEMAN, BAILEY the perception and response to threat-related stimuli, including anger and fear (Adams, Gordon, Baird, Ambady, & Kleck, 2003; Calder, Lawrence, & Young, 2001; Phillips, Drevets, Rauch, & Lane, 2003; Suslow et al., 2006). In summary, the findings from Experiment 1 suggest that the difficulties individuals with ASD have in recognising some facial expressions may be the consequence of impaired use of configural and feature processing strategies. Deficits in fear recognition could not be fully explained by impaired use of a configural processing strategy. Therefore, Experiment 2 was designed to explore the extent of feature processing impairments and to test whether the confusion of fear as anger was from the mouth region. EXPERIMENT 2 In Experiment 2 participants were required to identify facial expressions as the number of facial features was gradually increased, starting with either the eyes or the mouth alone (1 facial feature), then with the nose added (2 features) and finally with all 3 internal features present (eyes, nose and mouth). The recognition of facial expressions of fear, surprise, disgust and anger were tested. Some of the neurobiological substrates underlying typical processing of fear and disgust have been previously identified (Adolphs et al., 1995; Phillips et al., 1997; Sprengelmeyer et al., 1996) and testing the ability of individuals with ASD to recognise these expressions may provide some insight into possible impaired neurobiological functions. Surprise and anger were tested as they are typically confused for fear and disgust respectively (Russell, 1980; Russell & Bullock, 1985; Schlosberg, 1941, 1952). Experiment 2 used a novel experimental design, only included four facial expressions and tested a reduced number of participants because of time restrictions; therefore, the findings should be interpreted with respect to these three methodological constraints. Our hypothesis was that individuals with ASD would confuse fearful mouths (presented on their own) as being angry significantly more often than control participants. Methods Participants Twenty of the original 26 participants from each group were contacted and asked to participate in further cognitive tests, some 69 months after they had participated in Experiment 1. Fifteen individuals with a diagnosis of autism or Asperger s syndrome (12 male) and 15 typically developing controls (13 male) consented to participate. The aim was to generate statistical power.8 but with a total sample size of 30 participants (15

18 FACIAL AFFECT RECOGNITION IN ASD 1369 clinical/15 control) statistical power was.57, assuming group comparisons utilised ANOVA, and.7 if comparisons were made using t-tests. The reduction in statistical power should be considered in particular when interpreting the ANOVA findings from Experiment 2. The Wechsler Abbreviated Scale of Intelligence (WASI; Wechsler, 1999) was administered to explore whether the finding of a relationship between verbal ability and emotion recognition performance would stand up against a measure that was not limited to vocabulary. The clinical group (individuals with ASD) and the typically developing control group were well matched using the WASI. There were no significant group differences in chronological age or in performance, verbal or full-scale IQ measures (Table 1). Stimuli Pictures of 14 individuals each expressing disgust, fear, anger and surprise were selected from the Ekman and Friesen (1976) and Matsumoto and Ekman (1988) pictures of facial affect. Each face image was cropped so that just the eyes, the eyes and nose; just the mouth, the mouth and nose; and the eyes, nose and mouth were present. Examples of the stimuli used in Experiment 2 are shown in Figure 3. There were 168 test pictures in total. Participants had seen each of the faces once before in Experiment 1 but the Figure 3. Examples of stimuli used in Experiment 2. On each trial the stimuli appeared over three incremental steps, starting with either just the eyes or the mouth alone, then with the nose conjoined and finally all three features. Participants made a response after each step.

19 1370 WALLACE, COLEMAN, BAILEY duration between Experiments 1 and 2 meant that it was unlikely that there were any significant practice effects. Procedure Each trial was made up of three stimulus presentations. (1) A red fixation cross was shown for 350 ms before the first stimulus in the sequence (either a picture of the mouth alone or a pair of eyes alone) was presented for 2000 ms. A pattern mask was then displayed for 350 ms. When the mask disappeared, the names of the 4 facial expressions (disgust, fear, anger and surprise) were presented in boxes on the screen. There was an interval of 1000 ms before (2) a fixation cross again appeared for 350 ms, then a stimulus of the original face feature with the nose conjoined (either eyes nose or mouthnose) was presented for 2000 ms and was replaced by the mask (350 ms). The names of the four facial expressions then reappeared. There was an interval of 1000 ms before (3) the fixation cross appeared once again for 350 ms, followed by the final stimulus of the trial containing the eyes, nose and mouth (2000 ms). This picture was replaced by the mask (350 ms) and then the facial expression names reappeared. The participants identified the facial expression three times during each trial, after stimulus presentation of 1, 2 or 3 facial features, using the mouse to click on one of the four labelled boxes appearing on the computer screen. There was an interval of 1500 ms between the end of one trial and the beginning of the next. There were 56 trials in total and 168 stimulus responses were made by each participant. A break was taken after 19 and 38 trials for the participants to rest. Six practice trials were completed so the participant was accustomed to the procedure. Results The dependent variables were mean percentage accuracy for correctly recognising facial expressions and mean error rates when facial expressions were misidentified, at each of the three stages within a trial. Accuracy scores were not included when a participant s reaction time was more than four times the length of their mean reaction time. Parametric tests were used as the group means were normally distributed. Planned group comparisons tested whether individuals with ASD would confuse fearful mouths as being angry more frequently than controls. Linear regression analyses revealed no relationships in the clinical group between WASI verbal IQ, R 2.115, F(1, 25) 1.691, p.216, and accuracy scores judging facial affect. A trend was found, R 2.197, F(1, 25)3.181, pb.01, when BPVS raw score was used to explore the

20 FACIAL AFFECT RECOGNITION IN ASD 1371 relationship between verbal ability and facial expression recognition ability. Neither verbal IQ nor BPVS raw score was included as a covariate in the subsequent analyses. Group performance at recognising each facial expression A repeated measures ANOVA of mean percentage accuracy for correctly identifying a facial expression was conducted as a function of which Facial Feature Began a trial (eyes or mouth), the number of Facial Features Presented during the trial (one, two or three features) and the Target Facial Expression (fear, surprise, anger or disgust). First, there was a main effect of the number of Facial Features Present in a trial: 1, 2 or 3 features, F(2, 27) , pb Second, there was a significant interaction of the Facial Feature that Began a trial (eyes or mouth) with Target Facial Expression, F(3, 26)8.241, pb.001. Third, there was a significant group interaction of the facial feature that began a trial with target facial expression, F(3, 26) 3.092, pb.05. The main effect and interactions were explored using pairwise t-tests. Mean percentage accuracy improved to a larger degree when three rather than two features were presented, t(29)3.571, pb.001, compared to the level of improvement when two features rather than one feature was presented. Analyses of the first interaction showed that expressions of disgust were recognised more accurately if the trial began with the mouth rather than the eyes, t(29)3.122, pb.01. Similar analyses were used to explore the group interaction. The control group was significantly more accurate at identifying fearful expressions if the trial started with the eyes than the mouth, t(14)3.109, pb.01, and disgust when the trial started with the mouth than the eyes, t(14)3.506, pb.01. In the clinical group, whether a trial began with the mouth or the eyes did not significantly affect the recognition performance of any facial expression. Moreover, pairwise t- tests showed that fear was recognised more accurately by the control group from the eyes alone compared to the mouth alone, t(14)2.544, pb.01, and disgust was recognised more accurately from the mouth alone compared to the eyes alone, t(14)3.371, pb.01. The clinical group showed no advantage for recognising fear from the eyes compared to the mouth or disgust from the mouth compared to the eyes (see Table 5). Analyses of error patterns As in Experiment 1 the error patterns were calculated for when facial expressions were misidentified. A similar analysis of the frequency of errors when misidentifying a facial expression was conducted in Experiment 2 to explore the nature of impaired fear recognition (e.g., if fearful eyes were recognised by the control group at 74% accuracy then what was the distribution of errors across anger, surprise and disgust). Repeated measures

21 1372 WALLACE, COLEMAN, BAILEY TABLE 5 Mean percentage accuracy (standard deviations in parentheses) for correctly identifying target facial expressions (anger, disgust, fear or surprise). Participants were required to recognise facial expressions in incremental steps: (1) when the eyes or the mouth were shown alone; (2) when that feature was conjoined with the nose; and (3) when all three internal facial features were present Anger Disgust Fear Surprise Eyes alone Control 70 (23.5) 47 (25.6) 74 (15.5) 67 (22.7) Clinical 61 (27.3) 45 (23.5) 49 (28.5)** 55 (21.5) Eyes and nose Control 72 (16.6) 63 (24.6) 83 (16.4) 64 (24.0) Clinical 58 (30.8) 60 (23.7) 52 (22.0)*** 57 (22.1) Eyes, nose, mouth Control 85 (15.7) 84 (16.1) 98 (5.0) 71 (21.2) Clinical 66 (25.8) 80 (16.0) 71 (20.2)*** 65 (24.1) Mouth alone Control 51 (24.7) 79 (22.8) 54 (22.4) 72 (24.7) Clinical 52 (32.3) 62 (25.7) 44 (28.9) 59 (20.1) Mouth and nose Control 60 (28.1) 90 (19.8) 67 (13.9) 75 (28.3) Clinical 55 (29.0) 68 (29.8)* 47 (36.9) 64 (24.2) Mouth, nose, eyes Control 82 (26.7) 91 (12.9) 92 (10.6) 75 (25.0) Clinical 70 (21.5) 72 (29.8)* 76 (22.0)* 65 (27.0) Notes: *pb.05; **pb.01; ***pb.001. Asterisks indicate that the 2 values within a cell differ significantly. ANOVAs of error frequencies were conducted as a function of whether the trial began with the eyes or the mouth, the number of feature present (1, 2 or 3); and which emotion the target facial expression was misidentified as (e.g., either anger, disgust or surprise for fearful expressions). There were no significant group interactions with any of the factors in the frequency of errors made when misidentifying facial expressions. A direct test of the experimental hypothesis of Experiment 2 showed that there were no significant group differences in confusing fearful mouths as being angry, t(28) However, the clinical group misidentified fearful eyes as angry in 23% of trials when the eyes were presented alone, whereas the control group made this error in 5% of trials, t(28)3.3, pb.01. Discussion In line with previous studies of typically developing populations (Adolphs et al., 2005; Emery, 2000; Kohler et al., 2004; Mandal et al., 1992; Smith et al., 2005; Whalen et al., 2004), the control group in Experiment 2 demonstrated

22 FACIAL AFFECT RECOGNITION IN ASD 1373 the importance of processing the eyes to successfully recognise fear. They were more accurate at recognising fearful expressions on a trial sequence that began with the eyes alone compared to the mouth alone. Performance within a trial, at the stage when just one feature was presented (eyes alone or mouth alone), showed that typically developing individuals were significantly more accurate at recognising fear from the eyes than the mouth. Conversely, the clinical group showed almost no advantage for recognising fear from the eyes. Our hypothesis for Experiment 2 was not supported as fear was not confused as anger when the mouth was presented alone; conversely, this error only occurred when the eyes were presented alone. In fact much of the observed impairment in fear recognition can be explained by the misidentification of fearful eyes as being angry (the clinical group misidentified fearful eyes as being angry at a mean frequency of 23%, whereas the control group made this error at a mean frequency of 5%). It is hard to explain this pattern of misidentification as fearful and angry eyes do not share similar physical properties: eyes expressing fear involve raising the brow and increasing the presence of the sclera (Kohler et al., 2004; Whalen et al., 2004), whereas angry eyes involve a lowering of the brow (Kohler et al., 2004). Therefore, the misidentification of fearful eyes as angry could be associated with the impairment in higher-level cognitive interpretation of eye stimuli reported by Baron-Cohen et al. (1997, 2001). In that case fear from the eyes may not be a basic emotional state but a complex cognitive state. Equally, independent of whether an emotional state is basic or cognitively complex, individuals with ASD may have difficulty in recognising affective states that are disproportionately reliant on processing emotional information from the eyes (e.g., difficulty recognising sad from the eyes may underlie part of the impaired recognition of this emotion observed in Experiment 1). A final explanation for these findings, which is partly based on the atypical amygdala functioning hypothesis, is that individuals with ASD find fearful eyes threatening. It has been suggested previously that eyes may cause a negative physiological response in individuals with ASD (Volkmar, Sparrow, Rende, & Cohen, 1989) and more recently Dalton et al. (2005) reported in a functional imaging study positive correlations between duration of eye gaze fixation and amygdala activation. Dalton et al. (2005) proposed that eye gaze is avoided by individuals with ASD to reduce negative emotional over-arousal associated with threat. There is a large amount of sclera visible in fearful eyes, the gaze is directed straight at the observer and they are emotionally evoking stimuli. Therefore, atypical activation of the amygdala in response to fearful eye stimuli may lead to these stimuli being misidentified as threatening or angry. The impairments in Experiment 2 were not restricted to how individuals with ASD recognise emotions from the eyes. The clinical group did not show the typical pattern of increased accuracy in recognising disgust if the trial began with the mouth alone rather than the eyes alone; neither were they