Auditory gist perception and attention

Transcription

1 Auditory gist perception and attention Sue Harding Speech and Hearing Research Group University of Sheffield POP Perception On Purpose

2 Since the Sheffield POP meeting: Paper: Auditory gist perception: an alternative to attentional selection of auditory streams? Sue Harding, Martin Cooke and Peter König accepted for WAPCV (Workshop on Attention and Performance in Cognitive Systems), Hyderabad, India, January 07 Visit to Osnabrück (November 06)

3 WAPCV paper highlights brief review of idea of gist perception in vision with specific reference to Reverse Hierarchy Theory (Hochstein and Ahissar) Facets of RHT: only the gist of the scene is processed initially processing of the gist is rapid the focus of attention is deployed according to prior knowledge and the perception of the gist detailed analysis is performed on the part of the scene within the focus of attention unattended parts of the scene are not well differentiated main part of paper: supporting evidence from auditory experiments for ideas in RHT suggestions for the nature of gist representations and the use of saliency maps

4 Gist perception in vision a) lack of detail c) jumbled images (Levin et al 01) General features of the image can be identified despite the lack of fine detail b) global before local Global (large F, C) is perceived before local (small L) Jumbled images can still be categorised Detail is not important for an overview or gist

5 Gist perception in hearing (1) Rapid processing of the gist of an auditory scene seems to occur: Intuitively: people are aware of multiple sources in the general environment they hear without listening music from an orchestra or pop/rock band is initially heard as a whole a room full of talkers is heard as babble rather than individual conversations speech is heard as a whole rather than as individual formants and noise bursts

6 Gist perception in hearing (2) Experimental evidence for gist perception in hearing (concentrating on speech perception): time-reversed speech (Saberi & Perrott 1999) Speech waveform (top) is broken into segments of length e.g. or ms; each segment is reversed (A,B) (similar to jumbled images) Normal speech -reversed speech ( ms segments)

7 Gist perception in hearing (3) Sine-wave speech (Bailey et al. 1977, Remez et al 1981) Formant tracks are mimicked by time-varying sinusoids Normal speech Limited information is sufficient to give a speech percept Sine-wave speech Prior knowledge helps! So does training

8 Focus of current work Investigating gist perception and possible gist representations in hearing What type of representation is sufficient to provide the gist of an auditory source? coarse/broad or fine/sparse? In audition, may need fine information such as frequency harmonics at an early stage (for segregation according to pitch)

9 Auditory gist representations (1) Some possibilities for auditory gist representations: low or high pass filtered sounds; High pass: channels 33 to 64 High pass: channels 45 to 64 High pass channels Low pass: channels 1 to Low pass: channels 1 to Low pass channels

10 Auditory gist representations (2) Some possibilities for auditory gist representations: low or high pass filtered sounds; selected frequency channels; Selected channels (1 of 4) Selected channels (1 of 8)

11 Auditory gist representations (3) Some possibilities for auditory gist representations: low or high pass filtered sounds; selected frequency channels; high energy glimpses; High energy glimpses (threshold 0.5) High energy glimpses (threshold 0.6)

12 Auditory gist representations (4) Some possibilities for auditory gist representations: low or high pass filtered sounds; selected frequency channels; high energy glimpses; selected time frames Selected time frames (1 of 4) Selected time frames (1 of 8)

13 Focus of current work (2) Investigating gist perception and possible gist representations in hearing What type of representation is sufficient to provide the gist of an auditory source? manipulate signal, test whether listeners can identify source possible manipulations include: low/high pass filtering all or selected broad or narrow frequency channels high energy glimpses brief temporal segments possible types of sound include: speech (voiced, whispered) animal sounds music (instrument or melody identification) pitch other environmental sounds, e.g. traffic, building work, wind, sea different scales of representation may be required for different types of sound (may depend on listener s prior knowledge)

14 Other research possibilities analogies with hybrid images ( hybrid sounds ) low/high pass filtering; glimpsing; natural hybrids of pitch and timbre; repeated sounds; priming by temporal contrast (e.g. separate formant) or intensity contrast (e.g. louder formant) auditory categorisation (analogy with Oliva & Torralba s spatial envelope) determine perceptual dimensions (e.g. roughness, tunefulness, sharpness) apply principal components analysis to signals to determine patterns of features corresponding to perceptual dimensions

15 Hybrid images (1) (Oliva & Schyns 1997, Oliva 05) Two ways of looking at the same (hybrid) image Percept depends on scale used, which depends on duration of presentation distance of presentation priming of observer

16 Hybrid images (2) (Oliva et al. 06) Two ways of looking at the same (hybrid) image Percept depends on scale used, which depends on duration of presentation distance of presentation

17 Hybrid sounds Observers can be primed to see the fine detail in a hybrid image first - suggests that attention is directed to the scale suitable for the task (Oliva & Schyns 1997) What might auditory analogies with hybrid images show? Can listeners be primed to hear fine or coarse scale sounds preferentially? What do fine and coarse scales mean in hearing? Possibilities for creating hybrid sounds include: low/high pass filtering; glimpsing; natural hybrids of pitch and timbre; repeated sounds; priming by temporal contrast (e.g. separate formant) or intensity contrast (e.g. louder formant)

18 Spatial envelope in vision (1) Oliva & Torralba (06) Oliva & Torralba (06) propose a model that includes both local and global processing at different spatial scales Local receptive fields are combined to form global receptive fields, without segmentation or grouping of low-level features

19 Spatial envelope in vision (2) Oliva & Torralba (06) Principal component analysis (for a particular feature such as orientation) is used to produce global feature templates Different templates are created at different scales

20 Spatial envelope in vision (3) Oliva & Torralba (06) Oliva & Torralba (01) propose that scenes can be defined in terms of properties such as: openness, perspective, naturalness, expansion, depth, complexity, ruggedness, symmetry (to give the spatial envelope of a scene) rather than by the type of object the scene contains Scenes with similar properties should have similar spatial envelopes Properties are determined by combining global feature templates

21 Summary Ideas in Reverse Hierarchy Theory seem to be applicable to hearing as well as vision Auditory experimental evidence backs up the idea of initial gist perception followed by detailed scrutiny within the focus of attention Ideas for gist representations and saliency maps suggested for vision may also be applicable in hearing Questions applicable to both vision and hearing Should gist representations be coarse/broad or fine/sparse? Could there be multiple gist representations at different scales? Do hybrid images show two scales of gist representation in use? Do Oliva & Torralba s spatial envelopes provide a useful way of identifying the gist of a visual scene? How can a saliency map be combined with a gist representation?