Revisiting the right-ear advantage for speech: Implications for speech displays

INTERSPEECH 2014 Revisiting the right-ear advantage for speech: Implications for speech displays Nandini Iyer 1, Eric Thompson 2, Brian Simpson 1, Griffin Romigh 1 1 Air Force Research Laboratory; 2 Ball Aerospace; Wright Patterson Air Force Base, OH, 45433 [nandini.iyer.2, eric.thompson.22.ctr, brian.simpson.4, griffin.romigh]@us.af.mil Abstract Cherry (1953) reported that when listeners were presented with dichotic signal over headphones, they could reliably report words presented to the attended ear, while only being aware of the gross properties of the talker in the unattended ear. More recently, Gallun et al. (2007) showed that there were large differences in performance on dichotic tasks depending on ear of presentation, with significantly larger errors occurring when the target was presented to the left, rather than right ear (i.e., a right-ear advantage). In the current experiment, we explored two factors, type of signal in the non-target ear and uncertainty about the target ear, and their effects on right-ear advantage. The results indicated that the right-ear advantage was modulated by two factors: 1) nature of the speech stimuli presented in the unattended ear, and 2) target ear uncertainty. Substantial differences were observed between listeners in the tasks, leading to varying amounts of right-ear advantage across listeners for the listening conditions tested. These results and their implications for the design of multichannel speech communication displays are discussed, and the use of these methods is recommended as a useful screening tool for selection of personnel who listen to and use multichannel speech displays. Index Terms: dichotic speech perception, right-ear advantage 1. Introduction Early neurophysiological and behavioral work has demonstrated the presence of asymmetries in auditory perception. Much of the experimental work has employed a dichotic listening task, where two auditory stimuli are simultaneously presented, one to each ear, and the listener is asked to report the stimuli presented to one or both ears. [1, 2] was one of the first to demonstrate a right-ear advantage for linguistic material, and argued that the propensity for selecting the stimulus presented to the right ear was due to anatomical properties of the auditory system, particularly due to the fact that the right ear is connected to the language dominant left hemisphere of the brain by preponderantly contralateral connections. And while there has been continuing evidence for the structural theory [3], results from [4] suggests that other mechanisms, such as attention, might also be at play. For example, [5] has suggested that the right-ear advantage can be reduced, or eliminated when a listeners is directed to attend to the left ear. Despite the fairly substantial body of literature on auditory, and more specifically speech asymmetries, few researchers have addressed the potential implications that these results might have on the design of spatial auditory displays. Spatial interfaces endeavor to enhance communications by taking advantage of listeners ability to effectively segregate speech streams when they are spatially separated, thereby improving speech intelligibility and decreasing workload [6]. More recently, a speech display capable of accommodating up to seven simultaneous talkers was proposed [7]; this display utilizes the differences in auditory spatial acuity, so that talkers are more closely spaced in the front and more widely spaced in the periphery. Laboratory studies based on such a spatial speech display resulted in listeners performing about 30-40% better on intelligibility tasks compared to standard diotic communications. Note that the optimal display proposed makes no provisions for the aforementioned asymmetries, nor does it provide any guidelines for adapting such displays for specific users based on measured or known asymmetries. Recently, large differences were reported in dichotic tasks between the two ears [8]; specifically, there were large increases in identifying a target presented to the left, rather than right ear, suggesting a right-ear advantage. In their experiment, they reported an increase in the right-ear advantage, even in conditions where the target ear was clearly indicated in advance. This finding is especially surprising because it contradicts the results from studies that report that the right-ear advantage can be reduced or switched by directing attention to the left ear. This result is also not consistent with early research [9], that suggested that listeners can only process stimuli arriving at one of the ears in a selective attention task. The findings suggest that the right-ear advantages observed in tasks of selective attention might be related to failures in retrieval and processing of the speech stimuli involving more central processes, or simply a propensity for listeners to adopt less optimal strategies in the task. It is important to distinguish between these two possibilities, because they have different implications for the design of speech-based displays. If listeners show a right-ear advantage due to adoption of non-optimal strategies, then it would warrant the development of training modules in order to move them to adopt optimal task-related strategies. On the other hand, if right-ear advantages persist due to failure of retrieval or storage at more central locations, then it would warrant the development of selection or screening tests for listeners who are better able to use spatial speech displays. At the very least, a spatial speech-based display has to incorporate a design guide for listeners who show persistent right-ear advantages even when they are directed to attend to a target ear. The aim of the current study was to explore the parameter space for dichotic speech tasks in order to gain an understanding of conditions under which right-ear advantages might be expected. Speech intelligibility for target signals presented to the left and right ear was measured under three listening conditions with increasing uncertainty about the target ear. In the first condition, with least uncertainty, the target ear was pre-cued at the beginning of every trial, and the target ear remained fixed throughout a block of trials. In the second condition, the uncertainty was increased by randomly varying the target ear within a block of trials, but the target ear was always pre-cued at the onset of every trial. In the last condition, where uncertainty was largest, the target could vary from trial-to-trial and this information was not provided to the listener until after the stimulus presentation (i.e., a post-cued Copyright 2014 ISCA 457 14-18 September 2014, Singapore

trial). We predicted that listeners adopting optimal strategy should exhibit little to no right-ear advantage in conditions with low uncertainty, and only minimal advantage in high uncertainty conditions mostly due to failures of selections of the target ear, which would manifest itself in the pattern of errors. On the other hand, listeners with non-optimal task strategies would consistently show a right-ear advantage in all listening conditions. Moreover, we also measured the right-ear advantage with three different types of maskers (noise, irrelevant speech and relevant speech maskers), because these three types of maskers are known to mediate performance on dichotic speech intelligibility tasks differently [10]. Adding a contralateral noise masker has no effect on speech intelligibility, but speech maskers can interfere with target identification significantly based on the nature of the speech. For instance, a contralateral relevant speech masker can interfere with target identification to a larger extent than irrelevant speech maskers, mostly because of its similarity to a target signal [11]. [12] has proposed that a speech masker results in competition of central processing resources, so that inputs from the two ears are processed simultaneously to some extent, but only one input can be analyzed at a time. Presumably, any cues that the listeners can use to distinguish which input should be processed will result in better performance. We argue that disambiguating a target phrase from similar interfering stimuli requires more resources than disambiguating a target from irrelevant stimuli, and should result in large right-ear advantages either due to poor selection strategies or greater interferences with central storage and retrieval processes. For the purposes of this study, two types of speech maskers were included to study the extent to which right-ear advantage may be influenced by the nature of a speech masker. 2.1. Listeners 2. Method Ten listeners (19-32 years; five male, five female) participated in the experiment. All had normal audiometric thresholds (<20 db HL at octave frequencies 250 Hz-8 khz) in both ears. All listeners had experience with the stimuli and the tasks used in the experiment, and were paid for their participation. Of the ten listeners, two were left-handed and the remaining were right-handed. 2.2. Stimuli In the experiment, target sentences were drawn from the Coordinate Response Measure (CRM) corpus [13], and consisted of the form Ready [call sign], go to [color] [number] now. Eight possible call-signs (Arrow, Baron, Charlie, Eagle, Hopper, Laker, Ringo, Tiger), four possible colors (red, blue, green, white) and eight possible numbers (1 to 8) spoken by eight talkers (four males, four females) yielded a total of 2048 phrases in the corpus. In this experiment, only phrases containing the two call signs Baron and Charlie were used (a total of 512 phrases). The target phrase was always presented at 65 db SPL along with an ipsilateral (i.e., sameear) noise masker, which was varied according to the signalto-noise ratio (SNR). To maximize predictability, when the target ear was the right ear, a CRM phrase starting with the call sign Baron was selected, whereas when the target ear was the left ear, a CRM phrase with the call sign Charlie was selected. One of three types of maskers was also presented to the contralateral ear (i.e., the ear opposite the target ear): relevant speech masker, irrelevant speech masker and noise. In the relevant speech masker condition, a CRM phrase of the same-sex talker as that presented in the target ear, containing the call sign Baron (when the target ear was the left ear) or Charlie (when the target ear was the right ear), with mutually exclusive color-number keywords was presented to the contralateral ear. The irrelevant speech maskers were excerpts from recordings (two males, two females) of The Wealth of Nations by Adam Smith and were randomly selected for every trial so that the targets and masker talkers were spoken by a same-sex talker. The last type of masker was a noise masker, which was an independently generated token compared to the target ear noise token. When a contralateral speech masker was present, it was accompanied by a noise masker at the same SNR as that in the target ear, so that the stimuli in both ears were degraded to the same extent. Additionally, when a relevant speech masker was present, a control condition was run in which no noise maskers were present in either ear (represented at +). All stimuli were sampled at 44.1 khz. 2.3. Procedures The stimuli were generated on a PC running Matlab and were presented via a sound card (RME Hammerfall DSP Multiface II) to the listeners over headphones (Sennheiser HD280 Pro) while they were seated in sound-treated booths. At the onset of every 25 trial block, the listeners were provided detailed instructions about the listening conditions in the block. Their task was to respond with the color-number combination of a target phrase, presented either to the LEFT ear (always a phrase containing the call sign Charlie ) or the RIGHT ear (always a phrase containing the call sign Baron ). The target phrase was presented with a noise masker at varying SNRs (-6 to -18 db in 3 db steps); the SNR was randomly selected on each trial. In some blocks, a contralateral masker (noise, irrelevant or relevant speech) was presented; when noise was present contralaterally, the level of the noise was the same as that in the target ear. When speech maskers were presented, they were presented with a noise masker at the same SNR as that of the noise in the target ear. Control conditions included blocks with no contralateral masker. There was a total of 150 blocks; each data points represents 60 trials per condition. Within a block of trials, the target ear remained constant (FIXED) or varied from trial-to-trial (RANDOM). When the target ear was randomly selected on every trial, listeners were informed about which ear contained the target phrase via an arrow displayed on a computer monitor before the onset of a trial (PRECUE) or after the trial (POSTCUE). After stimulus presentation, the listeners responded with the color-number pair that they heard on a custom keyboard with 32 response keys (numbers 1-8 on colored backgrounds). 3. Results Figure 1 depicts average proportion correct color-number responses as a function of SNR when the target ear was the right (red diamonds) or left (blue circles). The dark and light colored symbols represent performance in FIXED and RANDOM blocks respectively. Panel a represents performance in the control, no contralateral masker condition and panel d shows performance with a noise only contralateral masker. Panels b and e represent performance in listening conditions with a pre-cued and post-cued contralateral irrelevant speech masker and noise. Similarly, panels c and f depict performance with a contralateral, pre- 458

and post-cued relevant speech masker and noise. In each of the panels, the green line represents a logistic fit to the data obtained in the no masker and contralateral noise masker conditions (panels a and d). masker in the post-cue condition (~5.2 db). The trend for improved performance when a target is presented to the right, rather than left ear, is also apparent in the control condition, when listeners just heard two CRM sentences, one in each ear (+ condition in panels c and f). Figure 1: Proportion correct color-number responses for left-ear (blue circles) and right-ear (red diamonds) targets as a function of SNR. The dark colored symbols are from FIXED blocks and the light-colored symbols are from RANDOM blocks. Each panel shows the data from a different contralateral masker and cuing condition: a) No contralateral masker, b) Irrelevant speech + Noise, pre-cue, c) CRM speech + Noise, pre-cue, d) Noise, e) Irrelevant speech + Noise, post-cue, f) CRM speech + Noise, postcue. The green dashed line in each panel is a best-fit logistic function to the data from the None and Noise conditions, shown to ease comparisons across the conditions. For each listening condition, an operational definition of right-ear advantage was the offset measure: the offset was measured as the midpoint between the psychometric functions of each of the average performance curves, and compared to the offset of the logistic fit. This allowed for a comparison across the listening conditions compared to a control condition (no contralateral masker) and a condition where not much right-ear advantage was expected. Comparing the proportion of correct color-number responses when the target phrase was presented with no contralateral masker (panel a) or with a noise contralateral masker (panel d), a right-ear advantage was not apparent. Further, there was no difference in performance between the random and fixed trial blocks; i.e., uncertainty about the target ear did not lead to any difference in performance when the contralateral masker was absent or was noise. When an irrelevant speech masker was presented along with the noise (panel b), and the listener was informed about the target ear prior to the trial, there was a shift in performance of the average offset from that of the logistic fit by approximately 1.1 db. However, there was no significant difference in performance between left-ear and right-ear presentation of the target, and nor was there any difference between the FIXED and VARIABLE conditions. However, when the target ear was cued after the trial (panel e), there was a small (~0.5 db) difference in performance when the target was presented to the right compared to the left ear. The minimal right-ear advantage observed with an irrelevant speech masker (only in the post-cue condition) increased when the contralateral masker was relevant speech. From panel c, the right-ear advantage is about 1 db even in listening conditions when the target ear was fixed throughout the block. When the target ear was randomly chosen in the block, the right-ear benefit was larger (~2.2 db). The greatest right-ear advantage was obtained with a contralateral relevant Figure 2: Difference in offset in the logistic function fit to the individual listener data for the left and right ears with a contralateral relevant speech masker in a) the Pre-cue Fixed condition (yellow stars), b) Pre-cue Random conditions (purple triangles) and 3) Postcue Random condition (green squares). The means across subjects are also shown with 95% confidence interval error bars. In order to compute the right-ear advantage for each of the ten listeners in the study, a threshold difference (Left-Right ear) was computed for each subject and is plotted in Figure 2. For each listener, the threshold difference was calculated computing the offset in db (midpoint of the resulting psychometric function for each listener in each condition) for the right and left ear presentations of the target and then subtracting the two offsets. These differences are believed to reflect the magnitude of the right-ear advantage for each individual listener. Only conditions with a contralateral relevant speech masker are plotted (since the greatest advantages were observed in these conditions), and are ordered in terms of increasing difference in the post-cue, random target ear condition (green squares in figure 2). As is apparent in the figure, there appears to be substantial differences between listeners, so that some of them show very little difference in performance between the ears in all conditions tested, whereas others show large differences even when they were asked to attend to the same ear throughout a whole block of trials (precued, fixed blocks, depicted by yellow stars in the figure). 4. Discussion and Conclusions In this study, there was no right-ear advantage when the interference in the contralateral ear was non-speech. When an irrelevant speech stimulus was presented to the contralateral ear, there was a small advantage when a target signal was presented in the right, rather than left ear, and was seen only in conditions when the target ear was post-cued. When relevant speech was presented contralaterally, the right-ear advantage was substantial, especially in conditions when the target ear varied randomly from trial-to-trial and when the target ear was post-cued; but the right-ear advantage was also evident in conditions with the least amount of uncertainty. There were substantial differences among listeners across conditions. Only one of the listeners showed a left-ear advantage (listener 1), and one showed near-optimal performance in all conditions (i.e., no ear advantage), demonstrating the ability to switch their attention as directed in the listening conditions. Perhaps surprisingly, some listeners had difficulty separating the stimulus in the target ear from that at the non-target ear, even when the target was clearly indicated in advance (listeners 6 and 8 in the pre-cued, fixed conditions; see Figure 2). These results are consistent with the results of [9] and [14], and suggest that listeners can 459

only process the semantic content of a stimulus arriving at one ear. Many listeners (4, 5, 9 and 10) could switch their attention as directed before the trial, but showed a strong tendency to report only the right-ear stimulus if there was any uncertainty regarding the ear of presentation of the target (5 and 9 in Figure 2). The findings from this study are consistent with other studies where there was little or no interference with contralateral noise or irrelevant speech maskers [11]. When a right-ear advantage was observed with an irrelevant speech masker, it was only seen in conditions where there was some uncertainty about the target ear (random, post-cue condition). Note that in this condition, there was no ambiguity about the target phrase in this condition, because listeners only heard one CRM phrase; so, listeners could select the target message without much competition. The right-ear advantage in this condition appears to be related to the inability to suppress an irrelevant speech interferer when it is presented to the dominant (right) ear. We speculate that listeners, by default, generally adopt the strategy of listening to the right ear, so that when an irrelevant speech interferer is presented to the right ear, they have to switch and attend to the left ear, which could result in some loss of information (~0.5 db measured in the experiment). Based on results from this experiment, it appears that for right-ear advantages to occur there has to be speech in both ears. Figure 3: Cumulative proportion of responses as a function of SNR for the contralateral CRM conditions. White areas represent the correct color and number responses. Cyan areas show the proportion of masker color and number intrusions. The yellow area shows the hybrid target-masker responses with either the target color and masker number or vice versa. The maroon area shows the proportion of responses in which either the color or number was from neither the target nor the masker. The right-ear advantage is primarily seen when the interferer is relevant speech. Even in conditions with the least amount of uncertainty, the presence of a relevant speech masker resulted in a right-ear advantage. This is consistent with the notion that there is preferential processing of rightear stimuli due to hemispheric asymmetries. The size of this effect is small (~1 db), but increases to 2 db when uncertainty about the target ear is present (RANDOM vs. FIXED trials). The increase in right-ear advantage when the target ear is randomly selected on every trial suggests that some listeners are adopting a non-optimal listening strategy of disregarding the cued target ear and attending to the right ear on every trial before switching and attending to the opposite ear. The fact that the cost of switching is slightly greater with a relevant masker (~0.5 db) compared to that with irrelevant maskers suggests that some of the cost could also be due to interference from the relevant speech masker in the storage/retrieval process, or effort expended to suppress the contralateral relevant masker. One way to distinguish between mechanisms that listeners might have used in the task is to study the pattern of errors that were generated. If the cost was related to interference in storage or retrieval, we would predict that there would be more semantically-similar (intrusions) errors when the target ear was the left ear due to preferential processing of the right ear stimulus. On the other hand, if the cost was related to increased effort to suppress the contralateral masker due to similarity, we would expect more random errors, i.e., errors that belong neither to the target nor the masker. Figure 3 plots the distribution of listeners responses among correct target keywords in white (color and number of the target word; denoted by C for color, and N for number with the subscript T to denote target ; C T &N T ), intrusions from non-target ear in cyan, i.e., the masking phrase (both color and number belonged to the CRM phrase in the non-target ear; C M &N M ), intrusions where either the color and number belonged to the target or masker in yellow (C T &N M or C M &N T ), and responses where the color or number keywords contained randomly selected choices in maroon (random and belonged to neither the target not the masker (C R or N R ). The middle panel in the figure depicts performance in the condition when the target ear varied on every trial but it was pre-cued. It is apparent from the data that a large proportion of errors in the left ear were due to a tendency for listeners to report color number keywords from the right ear, suggesting that most of the right-ear advantage in this condition occurred due to interference from right-ear stimulus. The intrusion errors from the right ear increase in conditions with maximum uncertainty; i.e., the post-cue condition (right-most panel), and it also results in the maximum right-ear advantage observed in the study (~6 db). While the results can be partially explained based on an interference mechanism (large right-ear intrusion when the target ear was left), response competition might not account for all the errors. The alternative is that listeners selected the wrong phrase to process. If this was true, and they were occasionally aware of this error, then we would predict that there would be significantly larger random errors. Examining figure 3, this does not appear to be the case. Listeners appear to make a non-optimal choice of listening exclusively to the right-ear, to the extent that they report mostly keywords heard in the right ear, even when they are asked about the keywords in the left ear. This finding is in agreement with [9] finding that listeners can only process information from one channel at a time. Whether the right-ear advantage is related to preferential treatment of right ear signals or due to listeners adopting a non-optimal right-ear bias is not clear. Further studies are needed to determine if training procedures can alleviate some of the right-ear advantages seen in these results. With regards to using spatial speech-based displays, the right-ear advantages appear to be sufficiently large to be of interest to those who design such speech displays. One such design principle could be based on the dynamic adaptation of such displays based on prioritization of the incoming communication messages, so that the most important messages are presented in the right hemi-field, and lower priority messages are displayed in the left hemi-field. What is clear from the current study is that the adaptive displays should only be used when multiple simultaneous messages are likely to be contextually-similar. Further studies are warranted to determine whether stimulus factors, such as level of signals can be used to offset these asymmetries noted in the study, or whether other factors such as training and experience are sufficient to overcome these asymmetries. 460

5. References [1] D. Kimura, Some effects of temporal-lobe damage on auditory perception. Canadian Journal of Psychology, 15, 156 165, 1961. [2] D. Kimura, Functional asymmetry of the brain in dichotic listening. Cortex, 3, 163 178, 1967. [3] R. J. Davidson & K. Hughahl, Baseline asymmetries in brain electrical activity predict dichotic listening performance. Neuropsychology, 10, 241-246, 1990. [4] M. Hiscock, R. Inch, and M. Kinsbourne, Allocation of attention in dichotic listening: Differential effects on the detection and localization of signals. Neuropsychology, 13(3), 404-414, 1999. [5] A. E. Asbjørnsen and K. Hugdahl, Attentional effects in dichotic listening. Brain and Language, 49, 189-201, 1995. [6] R. S. Bolia, Effects of spatial intercoms and active noise reduction headsets on speech intelligibility in an AWACS environment. Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting, 100-103, 2003. [7] D. S. Brungart, and B. D. Simpson, Optimizing the spatial configuration of a seven-talker speech display. Proceedings of the 9th Meeting of the International Community for Auditory Display, 188-191, 2003 [8] F. J. Gallun, C. R. Mason, G. Kidd, Jr., Task-dependent costs in processing two simultaneous auditory stimuli Perception and Psychophysics. 69, 757-771, 2007. [9] E. C. Cherry. Some experiments on the recognition of speech, with one and with two ears, Journal of the Acoustical Society of America, 25, 975 979, 1953. [10] T. L. Arbogast, C. R. Mason, and G. Kidd, Jr., The effect of spatial separation on informational and energetic masking of speech. Journal of the Acoustical Society of America, 112, 2086-2098, 2002. [11] D. S. Brungart, and B. D. Simpson, Within-ear and across-ear interference in a cocktail-party listening task. Journal of the Acoustical Society of America, 112, 2985-2995, 2002. [12] A. M. Treisman, Strategies and models of selective attention. Psychological Review, 76, 282-299, 1969 [13] R. S. Bolia, et al., A speech corpus for multitalker communications research. Journal of the Acoustical Society of America, 107, 1065-1066, 2000. [14] N. Wood, & N. Cowan, The cocktail party phenomenon revisited: Attention and memory in the classic selective listening procedure of Cherry (1953), Journal of Experimental Psychology: General, 124, 243-262, 1995. 461