Audio-visual synchrony perception

Transcription

1 Audio-visual synchrony perception van Eijk, R.L.J. DOI: /IR Published: 01/01/2008 Document Version Publisher s PDF, also known as Version of Record (includes final page, issue and volume numbers) Please check the document version of this publication: A submitted manuscript is the author s version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher s website. The final author version and the galley proof are versions of the publication after peer review. The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication Citation for published version (APA): Eijk, van, R. L. J. (2008). Audio-visual synchrony perception Eindhoven: Technische Universiteit Eindhoven DOI: /IR General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. Users may download and print one copy of any publication from the public portal for the purpose of private study or research. You may not further distribute the material or use it for any profit-making activity or commercial gain You may freely distribute the URL identifying the publication in the public portal? Take down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. Download date: 08. Nov. 2016

2 Audio-visual synchrony perception Rob L.J. van Eijk

3 The work described in this thesis was carried out under the auspices of the J.F. Schouten School for User-System Interaction Research. An electronic copy of this thesis in PDF format is available from the website of the library of the Technische Universiteit Eindhoven ( c 2008, Rob L.J. van Eijk, The Netherlands CIP-DATA LIBRARY TECHNISCHE UNIVERSITEIT EINDHOVEN Eijk, Rob L.J. van Audio-visual synchrony perception / by Rob Lambertus Jacobus van Eijk. - Eindhoven: Technische Universiteit Eindhoven, Proefschrift. - ISBN NUR 778 Keywords: Synchrony perception / Temporal interval discrimination / Psychophysics Cover design: Paul Verspaget Printing: Universiteitsdrukkerij Technische Universiteit Eindhoven

4 Audio-visual synchrony perception PROEFSCHRIFT ter verkrijging van de graad van doctor aan de Technische Universiteit Eindhoven, op gezag van de Rector Magnificus, prof.dr.ir. C.J. van Duijn, voor een commissie aangewezen door het College voor Promoties in het openbaar te verdedigen op donderdag 29 mei 2008 om uur door Rob Lambertus Jacobus van Eijk geboren te Asten

5 Dit proefschrift is goedgekeurd door de promotoren: prof.dr. A.G. Kohlrausch en prof.dr. J.F. Juola Copromotor: dr.ir. S.L.J.D.E. van de Par

6 Contents 1 Introduction Intersensory timing Subjective simultaneity Sensitivity to temporal interval differences Perceptual and cognitive aspects of intersensory timing Stimulus complexity Overview of this thesis Audio-visual synchrony and temporal order judgments: Effects of experimental method and stimulus type Introduction Method Results General discussion Temporal interval discrimination thresholds depend on perceived synchrony for audio-visual stimulus pairs Introduction Experiment 1: Thresholds at 0 ms, PSS, and 2 PSS Experiment 2: Additional thresholds within and beyond the synchrony range General discussion Effects of visual predictive information on audio-visual temporal order and simultaneity judgments Introduction Method Results General discussion

7 5 Visual predictive information in audio-visual stimuli influences asynchrony detection thresholds Introduction Method Results General discussion General discussion Main findings Future research A Notes on the literature overview of PSS estimates Bibliography Summary Samenvatting Acknowledgments Curriculum Vitae

8 1 Introduction 1.1 Intersensory timing Most events in our natural environment, such as when we listen to someone speaking in front of us or observe a book falling to the floor, provide us with information via the different sensory modalities that is generally integrated into a single multisensory representation (King, 2005; Spence, 2007; Stein and Meredith, 1994). These examples illustrate the importance of auditory and visual modalities when passively perceiving events. When engaging in interaction with the environment, also the importance of the tactile modality becomes apparent. When writing a text using a personal computer, for example, one can feel the keyboard beneath one s fingers, hear the sound that is produced by pressing a key, and see the corresponding character appear on the screen. In this example both visual and auditory information are provided by transmission through the external world, whereas the tactile stimulation is co-located with the physical object (the keyboard). As such, tactile stimulation is generally limited to nearby events, whereas auditory and visual stimulation can be provided by events occurring within a relatively large range of distances (Gepshtein et al., 2005; Hillis et al., 2002; Miyazaki et al., 2006). Given the relatively low speed of sound, the auditory component of a perceived event will always reach an observer later than the visual component, and this difference increases with physical distance. These examples from daily life reveal that audio-visual integration does not require a physically synchronous presentation of auditory and visual components of a multisensory event, but must to some extent be tolerant of temporal disparities. In the physical world, the timing relationship between auditory and visual signals arising from our natural environment is determined by the properties of some physical process, e.g., the physical moment of impact of the objects involved and the properties of the media carrying the signals to the respective sensory receptors. Auditory and visual information are not always directly stemming from our natural environment, however, but may also be provided through the reproduction of prerecorded audio-visual material on television or in the cinema, or may be generated in real time in a virtual environment. In artificial environments such as television, teleconferencing systems, and games, the reproduction of timing relations between sensory modalities is completely dependent on technology. Temporal disparities between auditory and visual components may have a

9 2 1 Introduction detrimental effect on perceived quality of audio-visual presentations (Rihs, 1995), and may, for example, also hamper the feeling of presence in a virtual environment (see Kohlrausch and van de Par, 2005, for an overview of audio-visual research in the context of multimedia applications). Therefore, it is important to control the temporal relationship between signals of different modalities in a way that is perceptually optimal. In realistic computer games and interactive virtual reality environments the auditory signal is not readily available, but has to be rendered in real time from a physical model of the environment (Moeck et al., 2007; Murphy and Rumsey, 2001). As such, the complexity of the sound rendering system may, for example, be scaled (Murphy and Rumsey, 2001) such that the sound quality is optimal, given the available processing resources, but that the delay with which the auditory component is presented is still well within perceptually acceptable limits. The aforementioned examples illustrate the importance of psychophysical studies that explore the limits of audio-visual synchrony perception. The remainder of this introductory chapter is organized as follows: The concept of subjective simultaneity is treated in detail in section 1.2, along with the most common experimental methods used to obtain estimates of the point of subjective simultaneity. Furthermore, an overview is provided of the estimates of subjective simultaneity that have been reported in the literature. In section 1.3 sensitivity to audio-visual temporal intervals is discussed, along with different experimental methods used to estimate sensitivity. Section 1.4 provides an overview of the perceptual and cognitive aspects that may influence subjective simultaneity and sensitivity to audio-visual temporal intervals. Section 1.5 deals with the influence of stimulus complexity on measures of audio-visual synchrony perception. This introduction is concluded by section 1.6, which gives an overview of this thesis and derives the central research topics to be addressed. 1.2 Subjective simultaneity Definition and methodology Subjective simultaneity in the context of audio-visual perception is generally expressed by the point of subjective simultaneity (henceforth PSS), that indicates the relative auditory delay (ms) between components of a bimodal stimulus for which the perception of synchrony occurs. By convention (Arrighi et al., 2006; Aschersleben and Müsseler, 1999; Enoki et al., 2006; Vatakis and Spence, 2006a,b; Zampini et al., 2003a,b, but see

10 1.2 Subjective simultaneity 3 Lewald and Guski, 2004; Spence et al., 2003), positive auditory delays indicate that the auditory component trails the visual component (with 0 ms being physical synchrony of the auditory and visual stimulus components). Negative values are used for the far less-frequent occurrence of an auditory component of some event occurring before its visual counterpart (see also section 1.2.2). In the literature, the PSS is defined in different ways, depending on the experimental method that is used. The two methods that are most commonly used are the synchrony judgment (SJ) task, and the temporal order judgment (TOJ) task. In the SJ task, observers are asked to judge whether the auditory and visual components of a stimulus are synchronous or not. Stimuli are presented with variable onset asynchronies and the SJ task thus yields a relatively direct measure of perceived synchrony (e.g., Fujisaki et al., 2004; Stone et al., 2001; Zampini et al., 2005b). Figure 1.1 shows a schematic representation of the response pattern in such a task. The black curve indicates the observed proportions of synchronous responses as a function of the relative audio delay between the auditory and visual components in the stimulus. The grey curve indicates the non-synchronous response proportions. In an SJ task the PSS is defined as the midpoint of the range of delays that are predominantly judged to be synchronous (termed synchrony range in this thesis). Two variations can be discerned in the synchrony judgment task: (1) the SJ2 task, in which there are only two judgment categories: simply synchronous, and non-synchronous, and (2) the SJ3 task, in which there are three response categories: audio first, synchronous, and video first. In the TOJ task (e.g. Aschersleben and Müsseler, 1999; Spence et al., 2003; Sternberg and Knoll, 1973; Zampini et al., 2003a,b) observers are asked to indicate which of two modalities was stimulated first by responding with either audio first, or video first. The response pattern in such a task is shown schematically in Figure 1.2. The proportion of video first responses (black curve) increases monotonically with increasing audio delay, while the proportion of audio first responses (grey curve) decreases correspondingly. The PSS is estimated as the point at which the proportion of audio first judgments equals the proportion of video first judgments, the TOJ 50% point.

11 4 1 Introduction Synchronous Asynchronous 0.8 Response proportion Audio delay (ms) Figure 1.1: Schematic synchronous (black) and asynchronous (grey) response curves as a function of the audio delay (ms). The intersection points between synchronous and asynchronous response curves, termed synchrony boundaries, are indicated using vertical dashed lines. The synchrony range corresponds to the range of delays between the synchrony boundaries. The PSS is defined as the midpoint of the synchrony range Video first Audio first 0.8 Response proportion Audio delay (ms) Figure 1.2: Schematic video first (black) and audio first (grey) response curves as a function of the audio delay (ms). The intersection point between audio first and video first response curves, the TOJ 50% point, is used as PSS estimate. The temporal window of integration is defined as the range of 25% to 75% response proportions, indicated using vertical dashed lines. The just-noticeable difference (JND) is defined by subtracting the delay at the 50% point from the audio-visual delay at the 75% point. As such, the width of the temporal window of integration equals 2 JND.

12 1.2 Subjective simultaneity Physical and physiological aspects In our natural environment the relative timing relation between an auditory and a visual signal is influenced by their propagation speeds, and inside our body by sensory transduction and neural conduction times. As was mentioned in section 1.1, auditory and visual signals are subject to different conduction times of their respective media that cause the auditory component of an audio-visual event to always reach the sensory receptors of an observer later than the visual component (Spence and Squire, 2003). That is, whereas the arrival of a visual signal is almost instantaneous due to a propagation speed of approximately m/s, the arrival of an auditory signal may be substantially delayed due to a propagation speed of approximately 340 m/s (i.e., in air it takes approximately 3 ms for an auditory signal to travel a distance of 1 m). When auditory and visual signals reach the human ears and eyes, the associated physical (light and sound) energy has to be converted into neural activity by the process of transduction occurring in the relevant sensory receptors. That is, light stimulation of the retina is transmitted to the optic nerve through a phototransduction process taking place in the rods and cones, followed by a chain of neurochemical stages, lasting around 50 ms (Arrighi et al., 2006; Schiffman, 2001). Sound energy is transduced by transforming sound waves into mechanical motion from the ear drum, via the middle ear to the cochlea where it is transformed into nerve impulses (Schiffman, 2001). As the acoustic transduction process only takes about 1 ms and travel times in the inner ear are below 10 ms, sound transduction in the human ear is about ms faster than light transduction in the human eye (Arrighi et al., 2006; King, 2005). This temporal disparity is even further increased due to longer neural transmission times in the visual system (King, 2005). As a result, only when an audio-visual event occurs at a distance of approximately 15 m, do the corresponding auditory and visual signals arrive simultaneously at the sensory cortices. For audio-visual events that occur within this so-called horizon of simultaneity (Pöppel, 1988), the auditory component will arrive in our brain first, whereas for events that occur beyond the horizon of simultaneity the visual component will arrive first. That is, an auditory stimulus presented at a relative delay of less than 45 ms will arrive in the brain before the corresponding visual component, whereas an auditory stimulus presented at a relative delay larger than 45 ms will arrive later.

13 6 1 Introduction Literature overview of PSS estimates The physical and physiological aspects treated in section lead to the expectation that the PSS should occur near the point of physical synchrony or at some positive audio delay, i.e., when the auditory component lags behind the visual component. More specifically, due to the faster transduction of sound, an auditory lag is required at the level of the peripheral sensory receptors for auditory and visual signals to arrive simultaneously in the brain. Furthermore, adaptation to positive audio delays perceived in every-day life may very well have shifted the PSS in the direction of larger, more positive delays (due to temporal recalibration; see section 1.4.2). Due to temporal ventriloquism (see section 1.4.3) the moment of occurrence of a visual stimulus may be biased in the direction of a trailing auditory stimulus. As temporal ventriloquism critically depends on the auditory signal that follows the visual signal (although see Aschersleben and Bertelson, 2003), it may result in a larger tolerance for positive, audio trailing delays, but not for negative, audio-leading delays. As such, temporal ventriloquism may result in a shift of the PSS in the direction of larger, more positive delays with the auditory component trailing the visual component. Research in the area of perceived audio-visual synchrony has made use of a wide range of stimulus types (see, e.g., Arrighi et al., 2006; Enoki et al., 2006; Keetels and Vroomen, 2005; Vatakis and Spence, 2006a) and experimental methods (see, e.g., Dixon and Spitz, 1980; Exner, 1875; Vatakis et al., 2008; Vroomen et al., 2004). Stimuli varied from simple (e.g., a flash of light accompanied by an audible click; see, e.g., Aschersleben and Müsseler, 1999; Hamlin, 1895; Jaśkowski et al., 1990) to complex (e.g., a video of a person speaking, or playing a musical instrument; see, e.g., Dixon and Spitz, 1980; Hollier and Rimell, 1998; Vatakis and Spence, 2006a). Comparing PSS values derived from different experimental methods shows that TOJ and SJ tasks often yield different results. A non-exhaustive overview of PSS values reported by or estimated from various studies is shown in Table 1.1. Given the context of this thesis the overview in Table 1.1 is restricted to publications about perceived temporal relations between auditory and visual modalities (i.e., other modalities and unimodal studies are excluded) that aim to uncover the perception of audio-visual synchrony under normal conditions (i.e., studies that attempt to manipulate synchrony perception by, for example, manipulating attention, or by exposing participants to asynchronous audio-visual adaptation stimuli are excluded).

14 1.2 Subjective simultaneity 7 Table 1.1: PSS values (ms) for audio-visual stimuli reported by or estimated from studies using different methods and stimulus types. Negative values indicate that the auditory component of the stimulus led the visual component, whereas positive values indicate that the visual component led, at the point at which both judgments were at the 50% point (TOJ task), or at the midpoint of the synchronous judgment range (SJ2 and SJ3 tasks). The range of reported PSS values is indicated by the minimum and maximum (separated by... ), or by the standard deviation if individual PSS values were not reported. N is the number of participants in the cited study, n/a stands for not applicable, and n/r for not reported (or impossible to derive). See Appendix A for additional information. Study Note on stimulus TOJ task SJ task 1 PSS (range; ms) N PSS (range; ms) N Flash-click (stationary) stimulus Aschersleben and Müsseler (1999) -13 (n/r) 16 Bald et al. (1942) Hidden sound source -9 (n/r) 32 Visible sound source -1 (n/r) 30 Bloch (1887; in Hamlin, 1895) -4 (n/r) n/r Dinnerstein and Zlotogura (1968) +71 (±61) 23 Enoki et al. (2006) Sudden appearance +36 ( ) 11 Exner (1875) +50 (n/a) 1 Fujisaki et al. (2004) No adaptation +4 (±31) 7 Adaptation to 0 ms -10 (±35) 7 Hamlin (1895) -19 ( ) 2 Hirsh and Fraisse (1964) +29 (n/r) (n/r) 8 Hirsh and Sherrick (1961) +5 (n/r) 5 Jaśkowski et al. (1990) +48 ( ) 3 1 All SJ studies used an SJ2 task, with the exception of Exner (1875), who used an SJ3 task.

15 8 1 Introduction Table 1.1: (continued) Study Note on stimulus TOJ task SJ task PSS (range; ms) N PSS (range; ms) N Keetels and Vroomen (2005) +8 (n/r) 15 Rutschmann and Link (1964) -43 ( ) 2 Smith (1933) -8 (n/r) 40-2 (n/r) 40 Spence et al. (2003) +20 (±24) 8 Stone et al. (2001) +51 ( ) 17 Teatini et al. (1976) Visual stimulus left -7 ( ) 5 Visual stimulus right +5 ( ) 5 Tracy (in Hamlin, 1895) -10 ( ) 6 Vatakis et al. (2008) Adaptation to 0 ms +1 (n/r) ( ) 13 Vroomen et al. (2004) Adaptation to 0 ms -6 (±24) (±12) 10 Whipple (1899) Single presentation -13 ( ) 5 Repeated presentation -4 ( ) 6 Zampini et al. (2005b) Same stimulus location +22 ( ) 40 Different stimulus location +33 ( ) 40 Zampini et al. (2003a) Same stimulus location +60 (±17) 9 Different stimulus location +75 (±19) 9 Simple (motion) stimulus Arrighi et al. (2006) Biological motion +60 (n/r) 3 Non-biological motion +35 (n/r) 3 Random motion +20 (n/r) 3 Aschersleben and Müsseler (1999) -17 (n/r) 16

16 1.2 Subjective simultaneity 9 Table 1.1: (continued) Study Note on stimulus TOJ task SJ task PSS (range; ms) N PSS (range; ms) N Dixon and Spitz (1980) +56 (n/r) 18 Enoki et al. (2006) Free fall +74 ( ) 11 Hollier and Rimell (1998) Short visual cue (pen) +28 (n/r) 12 Long visual cue (axe) +40 (n/r) 12 Lewkowicz (1996) +24 ( ) 10 Vatakis and Spence (2006a) Object action +63 (±90) 28 Complex stimulus Dixon and Spitz (1980) +64 (n/r) 18 Hollier and Rimell (1998) +38 (n/r) 12 McGrath and Summerfield (1985) +30 ( ) 12 Rihs (1995) +40 (n/r) 18 Smeele (1994) -105 (±30) (±41) 6 van Wassenhove et al. (2007) +26 (n/r) 20 Vatakis and Spence (2006a) Speech -36 (±143) 28 Guitar music +65 (±169) 28 Piano music -84 (±259) 28

17 10 1 Introduction The most striking results shown in Table 1.1 are the negative PSS values, which represent the situation in which the auditory stimulus had to lead the visual stimulus for the pair to be interpreted as synchronous. Since the PSS values reported in Table 1.1 are generally measured at the level of the peripheral sensory receptors of the observers (with the exception of the study by Bald et al., 1942, that reports delays measured at the stimulus source), negative external delays indeed are highly unnatural. It can be seen from Table 1.1 that negative overall PSS values are reported mainly for the TOJ task. In their review of the audio-visual TOJ literature, Neumann and Niepel (2004) found that the majority of studies yielded a negative PSS and conclude that [o]n the whole, these studies clearly suggest a negative PSS as the rule [p. 254]. Later on they qualify this conclusion by referring to TOJ studies that yielded a positive PSS and they state that this sheds some doubt on the generality of the finding that the PSS is situated at a negative SOA [p. 254]. Thus, from the review of the literature presented in Table 1.1 it may be concluded that the two tasks (TOJ and SJ) might be measuring different things. That is, the SJ task emphasizes the judgment of synchrony vs. successiveness, whereas the TOJ task emphasizes the judgment of order, which requires the perception of successiveness for correct perception (Allan, 1975; Hirsh and Sherrick, 1961). Indeed, Shore et al. (2005, p. 1260) report that their... present findings corroborate the claim (Allan, 1975; Hirsh and Sherrick, 1961) that judgments of temporal order and judgments of simultaneity (versus successiveness) are fundamentally different. In the context of their unimodal experiments on tactile temporal processing, Shore et al. (2005, p. 1252) state that... it has been argued that TOJs require more information about the stimuli before a correct response can be made and that... this increased processing requirement might reveal more subtle effects than the simpler simultaneity judgments used in previous studies. Furthermore, Zampini et al. (2003a, p. 208) note that TOJ and SJ tasks... may reflect very different processes/mechanisms (i.e. one related to multisensory binding, and the other related to temporal discrimination instead... ). Such differences could call into question whether estimates of parameters, such as the PSS, are independent of the experimental method. Although explanations for the differences in PSS values shown in Table 1.1 can be based on differences in experimental methods, this hypothesis has not been fully addressed before within a single study. It has been suggested that differences between PSS estimates derived from SJ and TOJ tasks should be experimentally investigated (Shore et al., 2002; Zampini et al., 2005b). Indeed, Fujisaki et al. (2004, see

18 1.3 Sensitivity to temporal interval differences 11 section for details) measured recalibration of audiovisual simultaneity using both synchrony judgment and temporal order judgment tasks in a within-subject design and obtained similar adaptation effects for both tasks, although the effect was less stable for the TOJ task. Smeele (1994) found a significant difference between PSS values obtained from SJ and TOJ data for 9 out of the 10 speech stimuli she used when comparing results for the six participants common to both experiments. Interestingly, she found a very high correlation between TOJ and SJ PSS values, and a constant shift between TOJ and SJ PSS values of 94 ms (with TOJ PSS values being more negative). A between-subjects design comparing SJ2 to TOJ was employed by Smith (1933) who reported consistent results between the two tasks in that both produced negative overall PSS values, although he did report that individual differences were somewhat greater in the SJ2 task. Vatakis et al. (2008) found a significant PSS shift when exposing participants to an audiovisual speech video with the auditory speech lagging behind the visual stream. The PSS shift, however, was only observed in the SJ task, but not in the TOJ task. Vroomen et al. (2004) also report a between-subjects study in which TOJ and SJ2 are compared using the same stimuli. They found similar shifts in the PSS using the two methods after adaptation to a series of stimulus pairs with specific offsets of their audio-visual components (although the absence of a significant difference between the two judgment tasks may have been due to a lack of statistical power; see also section 1.4.2). In summary, the literature comparing the effect of experimental method on audio-visual synchrony perception is not only limited, it has also produced different results for TOJ and SJ procedures. 1.3 Sensitivity to temporal interval differences A concept that is related to the point of subjective simultaneity, is the sensitivity with which people can discriminate between different audio-visual temporal intervals. A clear example is a thunderstorm that is approaching a group of hikers or receding from them. By estimating the temporal interval between the (visual) lightning flash and the (auditory) thunder, one can determine that a thunderstorm is approaching if following temporal intervals are progressively shorter in duration. Determining whether the temporal interval is shorter, however, does require that the difference in the length of the temporal interval is large enough to be detected.

19 12 1 Introduction At least three different methods for measuring sensitivity to audio-visual asynchrony have been reported in the literature. In one method, various delays between an auditory and a visual stimulus are introduced in a test sequence using the method of constant stimuli. The just-noticeable difference (JND) can then be determined from the slope of the response curves in a temporal order judgment (TOJ) paradigm (e.g., Hirsh and Sherrick, 1961; Vatakis and Spence, 2006a). Related are the sensitivity measures derived from a synchrony judgment (SJ) task, which can be derived from the slopes at the synchrony boundaries, or from the width of the range of synchronous responses (e.g., Arrighi et al., 2006; Vatakis et al., 2008; Zampini et al., 2005b,c). In a third method, discrimination thresholds can be determined directly by using an adaptive procedure with more than one observation interval (e.g., McGrath and Summerfield, 1985) Just-noticeable difference (JND) In TOJ experiments, sensitivity is generally characterized by the JND, which can be determined by fitting the video first data to a cumulative Gaussian distribution (e.g., by means of a Probit Analysis (Finney, 1952) in which the proportions are converted to standard z-scores and fitted with a straight line across audio-visual onset asynchrony; see, e.g., Hirsh and Sherrick, 1961; Rutschmann and Link, 1964). The JND is defined by subtracting the audio-visual delay at the 50% point from the audio-visual delay at the 75% point, and thus is inversely related to the slope (see also Figure 1.2). 2 The JND then defines an interval around the PSS (TOJ 50% point), called the temporal window of integration (Navarra et al., 2005; Spence and Squire, 2003), within which participants are unable to accurately determine the temporal order of an auditory and a visual stimulus. The temporal window of integration thus is defined as the range of 25% to 75% response proportions in a TOJ task, and its width equals 2 JND Sensitivity derived from a synchrony judgment task In a synchrony judgment (SJ) experiment, several methods for estimating sensitivity can be used. The synchrony boundaries define the range of delays that are predominantly judged to be synchronous, and thus can be seen as asynchrony detection 2 A different way to determine the JND is by dividing the distance of the 75% point to the 25% point by two, which results in the average distance of the 75% point and the 25% point to the 50% point. Due to the symmetrical shape of the (fitted) cumulative Gaussian distribution, both approaches yield identical JNDs.

20 1.3 Sensitivity to temporal interval differences 13 thresholds (see also Figure 1.1). The audio first synchrony boundary, which is always located at a negative, audio-leading delay, is generally closer to physical synchrony than the video first synchrony boundary (Arrighi et al., 2006; Enoki et al., 2006; Lewkowicz, 1996). This suggests that observers are more sensitive to negative, audio-leading delays than to positive, video-leading delays (i.e., sensitivity to audio-visual asynchrony is asymmetrical). When synchronous responses are fitted using a Gaussian distribution, the standard deviation is commonly used as a measure of sensitivity (Arrighi et al., 2006; Vatakis et al., 2008; Zampini et al., 2005b,c). A similar measure of sensitivity is provided by the width of the synchrony range. Finally, the slopes at the synchrony boundaries of the synchronous response curve indicate sensitivity at the transition from perceived synchrony to perceived asynchrony Discrimination threshold In contrast to the TOJ and SJ methodology, which use judgments of individual audio-visual pairs presented by using the method of constant stimuli, audio-visual discrimination thresholds are determined by using two or three successive audio-visual pairs that are to be discriminated in tasks using the method of limits. In such experiments subjects have to discriminate between a standard reference stimulus with a given audio-visual delay and a stimulus with a smaller or larger audio-visual delay. One of the few studies of audio-visual discrimination thresholds was performed by McGrath and Summerfield (1985), who used a three-interval, two-alternative forced-choice procedure with synthetic audio-visual approximations to bilabial consonant-vowel syllables. In their procedure, the first interval always contained the physically synchronous reference (or standard) stimulus with which stimuli in the following two intervals had to be compared. Subjects were to indicate which of the two latter intervals contained the target stimulus, with an audio-visual delay different from the reference (the other stimulus always matched the standard). Depending on the subject s response, the amount of audio-visual asynchrony in the target stimulus was changed adaptively. McGrath and Summerfield (1985) reported an average negative threshold of 79 ms and an average positive threshold of 138 ms. Grant et al. (2004) used a two-interval, two-alternative forced-choice procedure with correct-answer feedback. Participants had to judge which of two films of a female talker appeared to be out of sync. One of the films always was presented

21 14 1 Introduction in (physical) synchrony, whereas the other film contained an adaptively controlled amount of audio-visual asynchrony. Grant et al. (2004) found a negative threshold of approximately 50 ms, and a positive threshold of approximately 200 ms. The same procedure was used by van de Par and Kohlrausch (2000), who used an animation of a white disc that accelerated downward until it hit a bar after which it returned. This visual animation was accompanied by a short acoustic impact sound. Van de Par and Kohlrausch (2000) reported a negative threshold of 29 ms, and a positive threshold of 85 ms. In all the adaptive procedures, thresholds were defined as the audio-visual delay that led to a specific percentage of correct responses, e.g., 70.7% when using the 1-up, 2-down procedure (Levitt, 1971). Although the reported thresholds vary, all studies reported that positive thresholds are larger than negative thresholds. This suggests greater sensitivity to an auditory advance than to a visual advance relative to the point of objective simultaneity (POS; although see Sinex, 1978, for some exceptions) Comparison of sensitivity measures Whereas TOJ data only yield a single measure of sensitivity, both synchrony judgment and temporal interval discrimination tasks yield two measures of sensitivity. Similar to the asymmetric sensitivity to audio-leading and video-leading delays in the synchrony judgment literature (section 1.3.2), differences in sensitivity to negative and positive audio-visual asynchronies were also demonstrated using threshold measurements (section 1.3.3). Explanations for this asymmetry in thresholds (Dixon and Spitz, 1980; Grant et al., 2004; McGrath and Summerfield, 1985; van de Par and Kohlrausch, 2000; see Alais and Carlile, 2005, for a recent review) refer to the natural temporal relations between an auditory and visual event in the real world, due to the relatively low speed of sound (see section 1.2.2). Grant et al. (2004) and van de Par and Kohlrausch (2000) suggested that, as a result of these natural temporal relations, the human perceptual system might have adapted its processing to tolerate and even bind sensory events over a range of relative delays into a common event interpretation that extends more liberally into the visual leading range than it does for stimulus pairs in which the auditory component leads. It is not known if the larger positive thresholds observed at a 0-ms reference delay persist over a large range of reference delays, or if they are localized mainly

22 1.4 Perceptual and cognitive aspects of intersensory timing 15 near the point of objective simultaneity (POS). It could also be expected that delay discrimination thresholds should increase proportionally in both directions from the POS, in accordance with Weber s Law (as demonstrated in the slope estimates from a TOJ study by Alais and Carlile (2005), in which they manipulated the apparent distance of a sound source to vary the PSS). Another possible prediction for the relative size of discrimination thresholds can be derived from the perceived synchrony of the reference delay. That is, if the perception of synchrony is categorical in nature, then discrimination thresholds should be large within the range of perceived synchrony, but decrease as test delays approach either side of the synchrony category boundary, where perceived synchrony yields to the clear perception of audio first or video first. The two aforementioned possibilities are discussed in more detail in Chapter Perceptual and cognitive aspects of intersensory timing It has been suggested that the tolerance of small asynchronies between auditory and visual components of a single event may be explained by (1) the human ability to adapt to every-day life exposure to auditory delays (Fujisaki et al., 2004; Navarra et al., 2005; Vroomen et al., 2004; Vatakis et al., 2008), (2) the auditory capture of vision, (Aschersleben and Bertelson, 2003; Bertelson and Aschersleben, 2003; Morein-Zamir et al., 2003; Vroomen and Keetels, 2006), or (3) the compensation for the distance of an audio-visual event (see, e.g., Alais and Carlile, 2005; Arnold et al., 2005; Lewald and Guski, 2004; Sugita and Suzuki, 2003). The aforementioned explanations are discussed below in more detail together with the associated perceptual phenomena. Furthermore, the effect of spatial disparity on PSS and JND values is also treated here Spatial disparity In nature, auditory and visual components of a common event originate from the same location. In laboratory settings and when using technical systems, however, visual stimuli are often produced using computer or television screens. Auditory stimuli are produced by headphones or speakers. As a consequence, auditory and visual stimuli need no longer originate from a common spatial position. Spence et al. (2003) presented auditory and visual stimuli from two possible positions. Two target LEDs were positioned 26 cm on either side of a fixation LED (62 cm in front of the observer). The two loudspeakers were placed directly behind

23 16 1 Introduction each target LED. Flash-click stimuli were presented from either the same or a different spatial position with relative delays of ±10, ±30, ±55, ±90, and ±200 ms. Participants judged the temporal order of auditory and visual stimuli. No significant effect of spatial disparity on the PSS was found (+20 ms, on average). The JND values for the same position (53 ms), however, were significantly larger than JND values for the different position (42 ms). From these results Spence et al. concluded that spatial disparity improves the accuracy with which people can perform a temporal order judgment task. Zampini et al. (2003a) used an experimental set-up that was almost identical to the one used by Spence et al. (2003). The relative delays, however, were ±20, ±30, ±55, ±90, and ±200 ms. In Experiment 1, participants performed a modality temporal order judgment (i.e., they indicated which modality came first). The JND values for stimuli presented from the same position (32 ms) were significantly larger than those for stimuli presented from different positions (22 ms), essentially confirming the results of Spence et al. (2003). Furthermore, Zampini et al. also found a significant shift in the PSS, which was 60 ms for the same condition, and 75 ms for the different condition. In yet another very similar experimental set-up (Zampini et al., 2005b) participants performed an SJ2 task, but now for relative delays of 0, ±20, ±30, ±70, ±200 ms. Standard deviations of the synchrony curve (a fitted Gaussian) were significantly larger for the same condition (114 ms) than for the different condition (91 ms). Furthermore, PSS values were smaller for the same condition (19 ms) than for the different condition (32 ms). In line with results from TOJ tasks, participants were more likely to respond with simultaneous when stimuli were presented from the same position, than when they were presented from different positions. Zampini et al. (2003b) manipulated the spatial location of the auditory stimulus by presenting it either by headphones or from a centrally positioned loudspeaker. Fixation light and target lights were located directly in front of the loudspeaker. Participants judged the temporal order of auditory and visual stimuli presented at relative delays of ±20, ±30, ±55, ±90, and ±200 ms. No effect of the spatial position of the auditory stimulus was found on JND values (86 ms), nor on PSS values (29 ms) in Experiment 1. In subsequent experiments, Zampini et al. stated that lower JNDs previously reported for stimuli originating from different spatial locations, may critically depend on presentation of the stimuli to different sides of the body midline; i.e., lower JNDs are found when stimuli are presented to different cerebral hemispheres, but not when stimuli are presented from different spatial locations within the same hemifield.

24 1.4 Perceptual and cognitive aspects of intersensory timing 17 The influence of hemispheric redundancy was also studied by Keetels and Vroomen (2005). Flash-click stimuli were presented from loudspeakers placed at 10, 30, and 50 to the left and right of fixation, and LEDs placed directly in front of the two loudspeakers at 10. Participants had to judge the temporal order of audio-visual stimuli that were presented at various relative delays (0, ±30, ±60, ±90, ±120, ±240 ms) and various spatial disparities (same location, or at 20 or 40 spatial separation in same or different hemifields). No significant effect of spatial disparity on the PSS (+8 ms) was found. The JNDs, however, were smaller (1) when stimuli were presented from different spatial positions, (2) when stimuli were presented from different hemifields, and (3) for stimuli presented at 40 than for stimuli presented at 20 spatial disparity. Keetels and Vroomen (2005) concluded that audio-visual JNDs depend on the relative spatial and hemispheric disparity at which stimuli are presented. In summary, spatial disparity may produce a shift in the point of subjective simultaneity, and improve accuracy in temporal order judgments. Such a performance difference was not demonstrated, however, when the auditory stimulus was presented over headphones instead of over a loudspeaker that was effectively in the same position as the visual stimulus Temporal recalibration As a result of the different propagation speeds of light and sound, humans are accustomed to perceiving every-day life events with auditory signals arriving later at the sensory receptors than the corresponding visual signals. As such it is expected that observers are more tolerant of lagging audio than of lagging video in integrating the sensed components of an event. If our perceptual system has indeed adapted to the auditory delays present in our natural environment, it is possible that temporal recalibration occurs when participants are exposed for some time to delays that are different from those experienced in every-day life. Fujisaki et al. (2004) tested the expectation of temporal recalibration by exposing participants to audio-visual tone-flash stimuli with different, constant delays for several minutes. Participants performed an SJ2 task after exposure to relative delays of -235 ms, 0 ms, and +235 ms (in separate sessions). Exposure to a delay of 0 ms resulted in a PSS value of -10 ms. After exposure to a delay of -235 ms an average PSS value of -32 ms was obtained, which constitutes a shift in the direction of more negative, audio leading delays compared to the 0-ms adaptation condition. A PSS value of +27 ms was

25 18 1 Introduction found after exposure to a +235 ms delay. Interestingly, there was a small but significant difference in the PSS between the no-adaptation condition (+4 ms) and the condition in which participants were exposed to physically synchronous stimuli (-10 ms). Fujisaki et al. (2004, p. 774) offered a possible explanation by stating that the no-adaptation condition might have been more affected by pre-adaptation to the natural environment, in which audio signals tend to be delayed relative to visual signals. Besides a PSS shift, Fujisaki et al. also found a widening of the synchrony range in the direction of the adapted lag. Recalibration of subjective simultaneity was also studied by Vroomen et al. (2004). Similar to Fujisaki et al. (2004), they exposed participants for 3 min to flash-click stimuli with a constant relative delay of 0, ±100, or ±200 ms (in separate sessions). After exposure, half of the participants performed a temporal order judgment (TOJ) task, and the other half performed a simultaneity judgment (SJ2) task using the same stimuli as in the exposure phase. Results were not significantly different for the TOJ and the SJ2 task, although this may have been due to a lack of power in the between-subjects analysis. The PSS values were approximately -10 ms following exposure to negative, audio leading delays, and approximately +10 ms after exposure to positive, video leading delays. Vroomen et al. found that the effect of exposure leveled off for adaptation delays around ±100 ms. Furthermore, no PSS shifts were found in a control experiment using exposure lags of ±350 ms. Vroomen et al. suggest that temporal recalibration is limited for audio-visual exposure lags in the range where a sound can capture the perceived onset of a light (see the section on temporal ventriloquism further ahead). Somewhat different results were reported by Navarra et al. (2005). They exposed participants to videotapes of audio-visual speech, or of a hand playing the piano that was either presented in physical synchrony, or with an auditory delay of 300 ms. Participants performed a temporal order judgment (TOJ) task using a (flash-click) stimulus that was different from the exposure stimulus. Furthermore, the TOJ task was performed while participants monitored the exposure stimulus (although some practice trials preceded the actual measurement phase). Different from the two studies above, no PSS shifts were reported. Similar to Fujisaki et al. (2004), however, Navarra et al. did find a widening of the temporal window of integration following exposure to asynchronous speech or music videos. That is, an average JND of 118 ms was reported after exposure to a synchronous stimulus, but an average JND of 135 ms was found after exposure to an asynchronous stimulus. No temporal recalibration effects were observed

26 1.4 Perceptual and cognitive aspects of intersensory timing 19 after exposing participants to a delay of 1000 ms. Similar to Vroomen et al. (2004), Navarra et al. suggested that the audio-visual lag used in the asynchronous exposure stimulus must remain within the temporal window of integration. Vatakis et al. (2008) used a procedure similar to that used by Navarra et al. (2005). They exposed participants to videotapes of audio-visual speech, presented in physical synchrony, or with an auditory delay of 300 ms. While participants monitored the exposure stimulus, they performed a TOJ or an SJ2 task (in separate sessions), using a flash-click stimulus. Similar to Navarra et al. (2005) they reported larger JNDs after exposure to asynchronous stimuli for TOJ data (139 ms vs. 117 ms), but no effect on PSS estimates. The SJ task, however, resulted in a significant PSS shift in the direction of the video leading exposure lag (+16 ms vs. +1 ms). Similar to Navarra et al. (2005) and Fujisaki et al. (2004), a wider synchrony range was found after asynchronous exposure for the SJ task (i.e., a larger standard deviation of the fitted Gaussian distribution; 157 ms vs. 139 ms). The effect of spatial disparity on temporal recalibration was investigated by Keetels and Vroomen (2007). Participants were exposed for 3 min to flash-click stimuli with a delay of ±100 ms (in different sessions). The location of the auditory stimulus during exposure was manipulated as a within-subjects variable (but between blocks), such that it either (1) corresponded to the location of the light and the fixation point, or (2) was laterally displaced by a distance of 70 cm. After exposure, participants performed a TOJ task. The location of the auditory stimulus during the TOJ task was manipulated as a between-subjects variable, such that it was always co-located with the light source (for half the subjects), or that it was always laterally displaced (for the other half of the subjects). In line with studies discussed above (Fujisaki et al., 2004; Vatakis et al., 2008; Vroomen et al., 2004), a PSS shift was reported when comparing exposure to -100 ms (-8 ms) with exposure to +100 ms (+5 ms). The location of the sound during exposure and measurement did not produce any main or interaction effects. Keetels and Vroomen (2007) thus concluded that spatial disparity does not affect temporal adaption in the audio-visual domain. Since spatial disparity was shown to affect the accuracy with which participants can make temporal order judgments, this may be a surprising result. Keetels and Vroomen offered an explanation for this finding by pointing out that the role of space in hearing is to steer vision, and that spatial co-localization thus need not be a requirement for intersensory pairing to occur. Finally, Miyazaki et al. (2006) performed an audio-visual temporal order judgment