Quantifying temporal ventriloquism in audio-visual synchrony perception

Transcription

1 Quantifying temporal ventriloquism in audio-visual synchrony perception Irene A. Kuling 1, James F. Juola 2,3 & Armin Kohlrausch 2,4 1 VU University Amsterdam 2 Eindhoven University of Technology 3 University of Kansas 4 Philips Research Europe Eindhoven 1

2 TEMPORAL VENTRILOQUISM What is temporal ventriloquism? Ø (Spatial) Ventriloquism Ø Strong influence of visual information Ø Temporal ventriloquism Ø Strong influence of auditory information? 2

3 TEMPORAL VENTRILOQUISM How is temporal ventriloquism measured? Ø Change in perceived temporal position of light flash caused by auditory click. Ø Evidence found in changed slope of psychometric function for visual temporal order Morein-Zamir et al.,

4 TEMPORAL VENTRILOQUISM How is temporal ventriloquism measured? Ø First measurement: Temporal order discrimination between two visual flashes Ø Second measurement: Place two sounds, one before, one after, the two flashes, and measure again visual temporal order Morein-Zamir et al.,

5 TEMPORAL VENTRILOQUISM How is temporal ventriloquism measured? Conclusion: Presenting a sound before the first light and after the second light improved performance relative to baseline, as if the sounds pulled the perception of lights further apart in time, i.e., closer to the two acoustic clicks, revealing the attraction by sound. 5

6 TEMPORAL VENTRILOQUISM What do we know about it? Ø Effect has not been quantified in terms of induced shift as a function of SOA. Ø It is possible to quantify by calculating the temporal shift at 25 and 75% points of psychometric function. Ø It has been shown that effects seems to be stronger for trailing stimuli, it seems to exist for temporal distances up to 225 ms, but the number of delays tested was always small (3, 4 values) Ø Certainly no such effect as in temporal adaptation where shift size seems to be proportional to SOA in adapter stimuli 6

7 TEMPORAL VENTRILOQUISM Questions? Ø In which range does it appear? Ø Does it work only on the closest neighbor? Ø How large is the temporal shi9? A A V V 7

8 EXPERIMENTAL CHALLENGE q Can we measure and quantify temporal ventriloquism in a (more) direct way? q Can we systematically describe the temporal shift as a function of the temporal delay between the auditory and visual signal? 8

9 EXPERIMENTAL CHALLENGE q Can we measure and quantify temporal ventriloquism in a (more) direct way? 9

10 EXPERIMENTAL CHALLENGE q Can we measure and quantify temporal ventriloquism in a (more) direct way? q A measure for the =me of percep=on of a short auditory or visual signal 10

11 METHOD Method known from psychoacoustics and speech perception research Schimmel & Kohlrausch,

12 12 Auditory marker auditory target (1)

13 13 Auditory marker auditory target (2)

14 14 Auditory marker auditory target (3)

15 15 Auditory marker visual target

16 EXPERIMENTAL DESIGN Auditory markers with auditory target A-A A A A A A 700 ms Unimodal A-V V-A Auditory markers with visual target A A V A A Visual markers with auditory target V V A V V V-V Visual markers with visual target V V V V V 16

17 EXPERIMENTAL DESIGN Ø 12 participants Ø 20 trials per conditions Ø Blocked design 17

18 RESULTS 50 Mean adjusted values (ms) A-A -50 Unimodal 18

19 RESULTS Mean adjusted values (ms) A-A V-V -50 Unimodal 19

20 RESULTS 50 Mean adjusted values (ms) A-A A-V V-A V-V Unimodal 20

21 EXPERIMENTAL CHALLENGE q Can we measure and quantify temporal ventriloquism in a (more) direct way? q A measure for the time of perception of a short auditory or visual signal 21

22 EXPERIMENTAL CHALLENGE q Can we measure and quantify temporal ventriloquism in a (more) direct way? ü A measure for the time of perception of a short auditory or visual signal 22

23 TEMPORAL VENTRILOQUISM Audio-visual markers + visual target 23

24 EXPERIMENTAL DESIGN Auditory markers with auditory target A-A A A A A A 700 ms Unimodal A-V V-A Auditory markers with visual target A A V A A Visual markers with auditory target V V A V V V-V Visual markers with visual target V V V V V 24

25 EXPERIMENTAL DESIGN Auditory markers with auditory target A-A A A A A A 700 ms Unimodal A-V V-A Auditory markers with visual target A A V A A Visual markers with auditory target V V A V V V-V Visual markers with visual target V V V V V Multimodal Audio-Visual markers with visual target A A A A AV-V V V V V V 25

26 EXPERIMENTAL DESIGN Multimodal Audio-Visual markers with visual target A A A A AV-V V V V V V 26

27 TEMPORAL VENTRILOQUISM Audio-visual markers + visual target A A A A What if SOA = 0 V V V V V Ø Intuitive: A+V components are in synchrony è no effect (V-V) Ø Auditory dominance: -30 ms (A-V) Ø Based on unimodal results: strong auditory + weak visual è in between, but closer to -30 ms than to 0 ms 27

28 RESULT SOA = 0 50 Mean adjusted values (ms) A-A A-V V-A V-V AV(0)-V -50 Unimodal Multimodal 28

29 RESULT SOA = 0 50 Mean adjusted values (ms) A-A A-V V-A V-V AV(0)-V -50 Unimodal Multimodal 29

30 MULTIPLE SOAS Ø 13 SOAs (initially 5) Ø SOA range from -300 ms (audio first) till 300 ms (visual first) Ø 18 participants (5 with 5 SOAs, 6 with 8 SOAs, 7 with all) 30

31 RESULTS Mean adjusted target position (ms) Temporal shift of adjusted visual target SOA (ms) Audio first Video first 31

32 EXPERIMENTAL CHALLENGE q Can we measure and quantify temporal ventriloquism in a (more) direct way? q A measure for the time of perception of a short auditory or visual signal q Can we systematically describe the temporal shift as a function of the temporal delay between the auditory and visual signal? 32

33 EXPERIMENTAL CHALLENGE ü Can we measure and quantify temporal ventriloquism in a (more) direct way? ü A measure for the time of perception of a short auditory or visual signal q Can we systematically describe the temporal shift as a function of the temporal delay between the auditory and visual signal? 33

34 DEVELOPING A MODEL Linear fit? Mean adjusted target position (ms) Temporal shift of adjusted visual target SOA (ms) Audio first Video first 34

35 DEVELOPING A MODEL Cubic function? SOA (ms) Cubic fit 2 R =

36 DEVELOPING A MODEL Cubic function Relation with synchrony curve Synchronous response proportion SOA (ms) Cubic fit R 2 =

37 DEVELOPING A MODEL Cubic function Relation with synchrony curve Synchronous response proportion 27% SOA (ms) Cubic fit 2 R =

38 DEVELOPING A MODEL Combined curve (based on Maij et al., 2009) Ø Linear component (effect of audio on visual) Ø Gaussian component (synchrony curve) 38

39 DEVELOPING A MODEL Combined curve y = (ax + d) e x b 2c 2 Ø 4 parameter fit Ø 2 parameter fit (b and c based on synchrony curve) 39

40 MODEL Perceived temporal position of the target flash (ms) SOA (ms) (A) R 2= parameter fit (B) SOA (ms) (C) SOA (ms) 40

41 MODEL Perceived temporal position of the target flash (ms) (A) SOA (ms) 2 R = R 2= parameter fit - 2 parameter fit (B) SOA (ms) (C) SOA (ms) 41

42 DISCUSSION 50 Accuracy of adjustments for the two withinmodality conditions: V-V has about 50 % larger SE than A-A Mean adjusted values (ms) A-A V-V Auditory accuracy (mean sd for A-A): 38.4 ms Unimodal Visual accuracy (sd for V-V): 55.1 ms Ø This can be seen as basis for models that are based on accuracy in timing perception for the two modalities (Bayesian models). Ø It fits nicely to the results of Hartcher-O Brien & Alais (2011) 42

43 DISCUSSION Shifts obtained in across-modality conditions: New way of establishing perceptual delays Mean adjusted values (ms) A-A A-V V-A V-V Unimodal 43

44 DISCUSSION Results for multimodal markers: Temporal attraction of visual stimulus by audio seems to be dependent on temporal distance Perceived temporal position of the target flash (ms) (A) SOA (ms) R 2= R 2= Ø Relation complicated by processing/transduction differences 44

45 CONCLUSIONS ü We developed a method to measure and quantify temporal ventriloquism in a (more) direct way ü The temporal shift can be modelled as a function of the temporal delay between the auditory and visual signal ü The model is a combination of a linear component (representing the auditory dominance in the temporal domain) and a Gaussian component (synchrony curve) 45

46 QUESTIONS? Thank you for your attention! 46