VERBAL OVERSHADOWING IN SPEAKER IDENTIFICATION: FACE AND VOICE

Transcription

1 VERBAL OVERSHADOWING IN SPEAKER IDENTIFICATION: FACE AND VOICE Anne S. Rosen, Denis Burnham, & Caroline Jones MARCS Auditory Laboratories, University of Western Sydney ABSTRACT: The phenomenon of verbal overshadowing, whereby verbally describing a stimulus impairs ability to recognise that stimulus, is now well established in visual identification tasks. We conducted the first test of whether verbal overshadowing also occurs in speaker identification, that is, whether verbally describing a voice impairs subsequent identification of that voice. Participants, 48 in an auditory, and 49 in a visual identification group, were assigned to one of two subgroups: Description, in which participants wrote a description of a voice they had heard 20 minutes previously, or No-Description, who did not write a description. Participants were then asked to identify the voice (or face, depending on the identification group) from an eight-voice (or -face) line-up. Accuracy results indicate that verbal overshadowing occurs both in face and speaker identification. Confidence ratings indicate that manipulations of voice quality influenced listeners voice identification judgements more than manipulations of fundamental frequency. INTRODUCTION Verbal overshadowing refers to the phenomenon whereby verbally describing a stimulus impairs the ability to recognise that stimulus. In the first investigation of this effect, Schooler and Engstler- Schooler (1990) found participants who viewed a face, then verbally described it, were less accurate in later identifying the face than participants who had not been instructed to produce a verbal description. Thus they established that verbal overshadowing occurs for visual recognition. The existence of this effect in face recognition is of practical concern in the area of eyewitness identification (Schooler & Engstler-Schooler, 1990). However, verbal overshadowing may in fact be of wider concern, to the extent that it can affect the accuracy of identification via other sensory modalities. Indeed verbal overshadowing of taste memory has been found (Melcher & Schooler, 1996). If verbal overshadowing also occurs in the auditory domain, there would be important implications for the reliability of earwitness testimony and identification. The aim of this study is to determine whether verbal overshadowing occurs in the auditory domain. The particular case we investigate here is whether verbally describing a voice impairs recognition of that voice in a subsequent speaker identification task. There are two reasons for suspecting that speaker identification may be affected by verbal overshadowing. Intuitively, speaker identification would appear to rely on the memory for a kind of perceptual experience that is particularly difficult to put usefully into words. Secondly, speaker identification is not a trivial task: people s ability to recognise even familiar voices is not good (Ladefoged & Ladefoged, 1980; Rose & Duncan, 1995), voices appear to be more difficult to learn and identify than faces (Legge, Grosmann, & Pieper, 1984), and the descriptions which people are typically able to generate about voices are not very specific or extensive (Yarmey, 2001). Speaker identification appears to be sufficiently unreliable that experienced researchers in the area of earwitness identification typically advocate caution in prosecuting defendants solely or mainly on the basis of earwitness testimony. For example, Deffenbacher, Cross, Handkins, Chance, Goldstein, Hammersley, and Read (1989) argue that earwitnessing is so error-prone as to suggest that no case should be prosecuted solely on identification evidence involving an unfamiliar voice. In light of the difficulty of identifying unfamiliar voices, then, it is particularly important to determine whether producing a verbal description of a voice might increase or decrease accuracy in later identification of a speaker. In this study, the primary aim was to investigate whether verbalising has an effect on how accurately adults perform in a subsequent speaker identification task in a suspect-present line-up. Distractor voices were generated via acoustic manipulations of the target voice and another voice. These Accepted after full review page 309

2 manipulations allowed us to address a secondary aim; the extent to which listeners are sensitive to different in voice cues in an auditory line-up situation, in this case voice quality versus fundamental frequency (F 0 ). METHOD Subjects The participants were 97 adults (74 females, 23 males), first year psychology students at the University of Western Sydney, with a mean age of 23 years. Participants received course credit for their participation. All had self-reported normal hearing and vision. All were fluent speakers of English. The majority (76) spoke English as their first language, and of the other 21, 10 were born in Australia and 11 were not. The distribution of the latter 11 across conditions was approximately equal. Stimuli All participants first viewed a 30-second video (staged bank robbery, from the video used by Schooler and Engstler-Schooler (1990), and then participated in a visual (face) or auditory (voice) identification task. In the visual condition, a line-up of eight colour photo faces (target plus seven distractors, displayed sequentially or simultaneously) was presented in Powerpoint on a Compaq Presario 1200 laptop. Presentation mode (sequential vs. simultaneous) was manipulated in the visual condition, in order to facilitate comparison with the auditory condition (where only a sequential mode is possible). For the auditory condition, a line-up of eight voices (target plus seven distractors) was constructed, all pronouncing the audio component of the video script: Just follow the instructions; don t press the alarm and you won t get hurt.. The line-up of eight voices, shown in Table 1, consisted of the original unaltered voice (Voice 1); three distractors (Voices 2, 3 and 4), which were F 0 -altered versions of the original from the video (digitised at 44.1 khz, with 16 bit resolution), with F 0 was manipulated using STRAIGHT (V30kr16) (Kawahara et al. 1999); and four other distractors (Voices 5-8), which were F 0 - altered versions of a second speaker. This speaker was a male native speaker of American English, who closely mimicked the target speaker s articulation and prosody. In duration, the target was s, and the second speaker s was s. In pitch, the values are similar for target (mean=99 Hz, range= Hz) and F 0 -matched second speaker (Voice 7;mean=102 Hz, range= Hz). Background noise from the original video was mixed into Voices 5-8, to ensure that background noise cues did not differ between speakers and therefore affect identification judgements. Table 1. F 0 characteristics of line-up voices Procedure Voice Status Speaker Mean F 0 (Hz) 1 Target Distractor Distractor Distractor Distractor Distractor Distractor Distractor Participants were randomly allocated to one of four groups: Voice/Description (n=24), Voice/No- Description (n=24), Face/Description (n=25: simultaneous line-up of faces (n=13), sequential line-up of faces (n=12)) or Face/No-Description (n=24: simultaneous (n=12), sequential (n=12)). All participants first viewed the video of the bank robbery on a colour monitor with inbuilt speakers (TEAC Televideo MV-1440) at a distance of 2m from the screen. Following this, all participants completed 20 minutes of pen and paper exercises (crosswords and spot the difference puzzles). Then participants in the Description groups were instructed to spend 5 minutes writing a description of the face or voice (depending on the condition) of the robber from the video. The No-Description groups did not write a description. Finally all participants were given an identification task. In the Voice conditions participants heard a single presentation of the line-up of eight voices (in one of two pseudorandomized orders) at a Accepted after full review page 310

3 comfortable listening level, played from CD on the laptop, via external speakers (Yamaha Creative SoundBlaster SBS300). In the Face conditions participants viewed a single presentation of the videoonly (no voices) line-up of eight faces presented in either the simultaneous ordering (in which all faces were displayed for 30 s), or the sequential ordering (in which each face was presented for 3 s each). Participants in each group indicated for each face or voice their confidence that the face/voice was that of the robber on a scale ranging from 1 ( strongly disagree ) to 9 ( strongly agree ). Participants were told that it was possible that the robber was not in the line-up. After each face had been rated, participants were asked to make a single identification of which face/voice (if any) was that of the robber, and to rate their confidence in their decision, again on the 1 to 9 rating scale. RESULTS Two separate sets of analyses were conducted, one for the accuracy of identification data, and another for the confidence rating data in the Voice conditions. Accuracy data The accuracy of participants responses was analysed for verbal overshadowing (Description vs. No- Description conditions) and test modality (Face vs. Voice). In addition, a separate analysis of the Sequential vs. Simultaneous factor was conducted for the Face groups. Figure 1 shows the Face Recognition and Voice Recognition results (% correct identifications with standard error bars; the dashed line indicates chance level performance, 12.5%). In all conditions there were many more incorrect than correct identifications; these visual and auditory tasks were quite difficult. Regarding the Face Recognition data, fewer participants in the Description group identified the face accurately than did No-Description participants. A preliminary analysis of the presentation mode in the Face identification test (simultaneous vs. sequential) showed no significant difference between the number of correct identifications in the simultaneous (36% = 9/25) and the sequential (20.8% = 5/24) groups, X 2 (1) = 1.380, p=0.240, so presentation mode did not affect participants accuracy. A test for verbal overshadowing in Face identification revealed that accuracy in the Face Description condition (12% = 3/25) was significantly lower than in the Face No-Description condition (45.8% = 11/24), X 2 (1) =6.868, p<0.01. Thus there was a verbal overshadowing effect in the Face conditions. Description No-Description 50 % correct 25 0 Face Condition Voice Figure 1. Identification accuracy. Most importantly, it was also found that there was a verbal overshadowing effect in voice recognition. Voice Description participants were significantly less accurate (12.5% = 3/24) than Voice No- Description participants (37.5% = 9/24), X 2 (1) = 4.000, p<0.05. Thus both Face and Voice verbal overshadowing effects were found. F 0 versus Voice Quality: Confidence ratings in the Voice conditions Accepted after full review page 311

4 For the Voice groups, an analysis was run on participants self-rated level of confidence for each voice, to explore how participants confidence might differ as a function of differences between the suspect s and the actual robber s voice in (a) fundamental frequency or (b) voice quality. Figure 2 shows the mean confidence levels (with standard error bars) for the eight voices given by the participants in the Description and No-Description conditions. In general, the confidence levels are quite low. Many participants selected values around 5 ( neutral ), particularly for the target voice and its F 0 -modified versions (Voices 2-4), ranging towards moderate disagreement that a voice was that of the robber. This indicates that the participants were often ambivalent that a voice was that of the robber, even when listening to the target voice. Mean confidence level Description No-Description target Line-up Voice Figure 2. Mean confidence levels in Voice conditions. Planned contrasts within a 2 x (8) analysis of variance revealed no significant difference overall between confidence levels in the Description vs. No-Description conditions. However, the participants were more confident that a suspect s voice was the robber s when the voice was the target or an F 0 - modified version (Voices 1-4) than a version of the second speaker s voice (Voices 5-8), F (1, 46) = , p< With regard to the F 0 manipulations, confidence levels within the four versions of the robber s voice (Voices 1-4) differed significantly, F (3, 141) = 2.842, p<0.05, and posthoc tests (with Bonferroni adjustment) reveal that this was due to significantly higher confidence levels for the target voice (Voice 1, F 0 = 99 Hz) than for Voice 3 (F 0 = 118 Hz), t (47) = 2.853, p<0.008 (see Figure 2). On the other hand, confidence levels for F 0 -manipulated versions of the second speaker s voice (Voices 5-8) were not significantly different from one another, F (2.698, ) = 2.754, p>0.05. Together these results indicate that overall, F 0 did not play a large role in affecting judgements. As a check on this, a post-hoc test of confidence ratings for the target (Voice 1) and the closest F 0 match for the alternative voice (Voice 7) revealed greater confidence for the former, F (1, 46) = , p<0.01. This suggests that pitch matching alone is not sufficient to affect the participants confidence levels. Thus voice quality appears to be the most salient aspect of the auditory stimuli for voice identification, with pitch modifications playing little or no part. DISCUSSION AND CONCLUSIONS The results of this experiment confirm and extend those of previous studies: we confirm that verbal overshadowing occurs in face identification (Schooler & Engstler-Schooler, 1990), and show that this effect extends to speaker identification. The accuracy rates in both face and speaker identification were statistically above chance for the No-Description conditions, and around chance levels in the Description group, and percent correct in both the Face and Voice No-Description groups was significantly higher than in their Description group counterparts. Our primary aim in this experiment was to conduct a first test of verbal overshadowing in speaker identification. The pattern of results in this study is consistent with Schooler and Engstler-Schooler s (1990) proposal that the verbalisation negatively affects the recall (but not coding) of perceptual memories which are hard to express usefully in words. Further experiments will be necessary to test Accepted after full review page 312

5 different hypotheses to determine why this negative effect occurs, but it can be noted that our participants did comment that they found it particularly difficult to write a description of the voice. Participants were typically able to describe the voice as male, adult, American accent, not loud and low pitch. However, they found it difficult to provide many other descriptors, and had trouble trying to verbalise for the full five minutes. There seemed to be nothing particularly idiosyncratic about the voice in the video that participants noticed and thought worth describing. We suspect that the lack of idiosyncrasy contributed to the voice being difficult to describe, and identify from the line-up. Other researchers in the area of speaker identification (e.g. Kunzel, 1994) have suggested that for a typical untrained listener, describing and identifying unfamiliar voices is difficult in the absence of any striking characteristics, such as extreme hoarseness, a distinctive regional accent, or a speech impediment. In connection with the finding of verbal overshadowing for speaker identification in this study, it is of interest to note that verbalising did not appear to affect differentially listeners ability to recognise the target voice on the basis of F 0 or voice quality. That is, participants who verbalised were not more likely to make a misidentification based on an F 0 recognition error, for instance, than participants in the control condition; and the same is true of the voice quality manipulation. Participants in both the experimental and control conditions were more confident that a voice was that of the robber when it matched the target in voice quality (but not F 0 ) than when it matched the target in F 0 alone. Perhaps this reflects the fact that the participants verbalisations, as just mentioned, were fairly vague in their descriptions of both voice quality and F 0, as might be expected for a low-pitched voice with a nonextreme F 0 and voice quality with no particular idiosyncrasies. The results of this study provide the first evidence that verbal overshadowing can occur in speaker identification. In addition, the results allow us to comment upon other factors in speaker identification: familiarity, and the degree of exposure. In this study, participants heard a completely unfamiliar voice on videotape for just a few seconds. They were then required to decide whether the voice was present in a suspect-present line-up of tape-recorded voices, after a 25-minute delay. Very few of the participants were able to identify the target voice correctly. This is not unexpected, as the voice was unfamiliar, and was presented in a very short recording, which the participants heard only once. Deffenbacher et al. (1989) found that listeners who heard a target voice on only one occasion were less accurate in identification than listeners who had multiple exposures to a target voice (but the same overall length of exposure, in terms of length of total voice sample). Yarmey, Yarmey, Yarmey and Parliament (2001) found that speaker identification is more successful with a highly-familiar speaker (immediate family member or best friend) than with a moderately-familiar speaker (coworker, team-mate, club-mate, or general friend) than with a low-familiarity speaker (casual acquaintance). Interestingly, Yarmey et al. (2001) also found that as familiarity decreased, the false alarm rate increased. In our study, the false alarm rate in speaker identification was very high, at 67% (32/48). The proportion of false rejections, participants who said that the target voice was not present, was very low, at 8% (4/48). The effect of verbalising on the listeners judgement that the voice is or is not present is yet to be investigated. Voice familiarity and degree of exposure to the voice may interact with verbalisation in voice identification. These factors could be investigated in future studies by separately manipulating verbal expertise, by testing a comparison group of expert witnesses, e.g., phoneticians or others trained in speech and vocal quality; and familiarity, by systematically varying the degree to which the target voice is known to the (expert or non-expert) listener; and degree of exposure, by varying the number of times the target voice is presented in the study. A major factor to be borne in mind in all such future studies is that from a practical legal point of view, it is crucial that we assess the effect of verbalisation on accuracy of identification (by different kinds of listeners, under different circumstances) as well as on the false alarm rate. ACKNOWLEDGEMENTS This paper is based on research undertaken by the first author in a Psychology BA Honours thesis (2001), at the University of Western Sydney (Bankstown), supervised by the second author. The research was approved by the UWS Human Research Ethics Panel, under approval number HREC 01:22. REFERENCES Accepted after full review page 313

6 Deffenbacher, K., Cross, J., Handkins, R., Chance, J., Goldstein, A., Hammersley, R., & Read, J.D. (1989). Relevance of voice identification research to criteria for evaluating reliability of an identification. Journal of Psychology 123, Dodson, C.S., Johnson, M.K., & Schooler, J.W. (1997). The verbal overshadowing effect: Why descriptions impair face recognition. Memory and Cognition 25, Kawahara, H., Katayose, H., de Cheveigne, A., & Patterson, R.D. (1999). Fixed point analysis of frequency to instantaneous frequency mapping for accurate estimation of F0 and periodicity. Proceedings of EUROSPEECH 99, Kunzel, H. (1994). On the problem of speaker identification by victims and witnesses. Forensic Linguistics 1, Ladefoged, P. & Ladefoged, J. (1980). The ability of listeners to identify voices. UCLA Working Papers in Phonetics 49, Legge, G.E., Grosmann, C., Pieper, C.M. (1984). Learning unfamiliar voices. Journal of Experimental Psychology: Learning, Memory, and Cognition 10, Melcher, J.M. & Schooler, J.W. (1996). The misremembrance of wines past: Verbal and perceptual expertise differentially mediate verbal overshadowing of taste memory. Journal of Memory and Language 35, Rose, P. & Duncan, S. (1995). Naive auditory identification and discrimination of similar voices by familiar listeners. Forensic Linguistics 2, Schooler, J.W. & Engstler-Schooler, T.Y. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive Psychology 22, Yarmey, A.D. (2001). Earwitness descriptions and speaker identification. Forensic Linguistics 8, Yarmey, A.D., Yarmey, A.L., Yarmey, M.J., & Parliament, L. (2001). Common sense beliefs and the identification of familiar voices. Applied Cognitive Psychology 15, Accepted after full review page 314