This will be accomplished using maximum likelihood estimation based on interaural level

Transcription

1 Chapter 1 Problem background 1.1 Overview of the proposed work The proposed research consists of the construction and demonstration of a computational model of human spatial hearing, including long term \precedence-related" eects. Using acoustic signals from the eardrums of a mannequin head as input, the proposed model will estimate the elevation and azimuth, relative to the head, of a sound source in space. This will be accomplished using maximum likelihood estimation based on interaural level and time dierences (ILDs and ITDs). ILDs and ITDs will be weighted at \onsets" (as described in [24]) and will be calculated as functions of time for a number of \critical band" frequency regions. An old-plus-new heuristic will be employed to replicate the Franssen and other precedence-related eects. The tools required to complete the research include signal processing and pattern recognition techniques, as well as an understanding of the cues humans use to localize sound events. The signal processing required for this project involves lter design, signal analysis and synthesis, and probabilistic estimation. The pattern recognition technique to be used is maximum likelihood estimation, but extensions may be made to incorporate other techniques. The maximum likelihood estimator \templates" will be derived from head-related transfer function data measured from a mannequin head. The psychoacoustics behind the model are drawn from many sources. Good overviews of the subject include [2] and [22]. Much inspiration is also drawn from Bregman's Auditory Scene Analysis ([3]), especially in conjunction with the old-plus-new heuristic. Part of the processing is intended to model the \precedence eect" (or \law of the rst wavefront") that has been demonstrated in human listeners, and is based on Zurek's model ([23], [24]). The completed thesis will detail all stages of the model and will compare the performance of the model with that of human listeners. In particular, the \precedence eect" experiments reported in [23] will be performed. It is expected that the model's performance will bear strong relation to human performance. Also, experiments will be performed to test the Franssen and other precedence-related eects. The form of the long-term model should suggest human psychoacoustic experiments which may be performed to test the model's generalization. 1

2 1.2 Introduction to spatial hearing The ability ofhuman listeners to determine the location of sound sources around them is not fully understood. It is widely accepted that several cues are used to estimate the location of sound sources in space. These cues include ILDs, ITDs, and the ratio of direct to reverberant energy, aswell as additional cues provided by headmovements. This research concentrates on ILDs and ITDs as the primary cues for localization. In short, ILD is the dierence (in db) of the signal levels measured at the two eardrums in response to a sound at a particular point in space. Similarly, ITD is the dierence in arrival time between the signals at the two eardrums. In this research, ILD and ITD are not single numbers for a given spatial location. Rather, they vary as a function of location and frequency. This distinction is important because the eects of head-shadowing and pinna ltering are frequency dependent. Henceforth, the frequency dependent ILD and ITD will be denoted the interaural spectrum. It is reasonable to assume that frequency analysis is performed at each ear individually prior to interaural spectrum estimation, since the human auditory periphery functions in this manner Head shadowing and the spherical head model To a rst approximation, ILDs and ITDs can be explained by a spherical head model. This model assumes that the head is a rigid sphere, isolated in space, with ideal pinpoint pressure sensors at the ends of a diameter representing the ears. By solving the acoustic wave equation for such a conguration, ITD and ILD can be determined as functions of frequency. This solution has been derived by many researchers. One such analysis is presented in [13]. When ITD and ILD are modelled in this way, \cones of confusion" arise. For a given interaural spectrum, there exists a locus of points for which ITDs and ILDs are identical. A cone (whose axis is a line drawn between the ears and whose origin is the center of the head) is a fair approximation of this surface. With this model, there are no cues available to resolve positional ambiguity on such a cone Head-related transfer functions It is widely accepted that the pinnae (outer ears) and the listener's body provide additional cues that enable position on a cone of confusion to be resolved to some degree. These additional cues are due to acoustic reections from parts of the pinnae or body, and result in systematic \distortions" of the interaural spectrum. A reasonable representation of the eect of body and pinnae is the so-called head-related transfer functions (HRTFs). A head-related transfer function is a measure of the acoustic transfer function between a point in space and a point in the ear canal of the listener. From these HRTFs, the interaural spectrum may be calculated by forming the complex ratio of the transfer functions for the two ears. HRTFs vary considerably from person to person, but the types of distortions imparted by the pinnae follow some general patterns, so meaningful comparisons may be made Issues not addressed In this thesis (and in most of the literature), at least two important localization cues used by human listeners are largely ignored. Because a slight shift in the position of the head (by rotation or \tilt") can alter the interaural spectrum in a predictable manner, human listeners 2

3 may use head movements to resolve positional ambiguities on \cones of confusion". This fact has been largely ignored in psychoacoustics literature for several reasons. First, it is intuitively obvious how head movements can resolve positional ambiguities. Second, human listeners are capable of auditory localization without head movements (which presents a more tractable research problem). Third, it is very dicult to construct a computational model including head movements and demonstrate that it works in any non-trivial situations. Also, there is little information on the forms of head movement employed by human listeners. A second set of cues that is being ignored is the monaural spectral information embodied in the HRTFs. The location and shape of \ridges" and \notches" in the monaural spectra may be useful cues if assumptions are made about the spectrum of the sound source (such as local \atness"). In the current research, no assumptions are made about the source spectrum, so monaural localization cues can not be employed. Another important cue that is being ignored is the ratio of direct to reverberant energy. This ratio can be used to estimate the relative distance of sources in an acoustic space. Because this research addresses only elevation and azimuth, this cue is not relevant. A complete model of localization would address all of these issues. 1.3 Previous work Lateralization models (\1D" localization) Some models of spatial hearing have previously been developed. Notably, most of these fall into the \one-dimensional localization" category. These models typically estimate only a single parameter, the subjective \lateralization" of a stimulus. In this context, lateralization means \left to right position" inside the head. Some models that fall into this category include [9], [16] and [18]. Lateralization models typically have several failings. Most of the experimental data that the models are designed to reproduce result from \unnatural" sounds. Lateralization models are most often used in conjunction with stimuli presented on headphones, with either uniform interaural spectra or laboratory-generated distortions of the interaural spectrum. These stimuli never occur in natural listening contexts, so the application of the experiments (and thus the models) to true \localization" is questionable. Also, few lateralization models attempt to model precedence-related eects. In [6], it is suggested that laboratory generated distortions of ILDs and ITDs that do not arise in the \real world" may result in sound events that appear to occur inside the head. This view is also supported in [9] Azimuth and elevation (\2D" localization) Few eorts have been made to model localization in more than one dimension (which may be largely due to the general unavailability ofhrtf data to the research community). However, at least two such models have been constructed. In [21], the authors summarize a model where ILD spectrum templates were constructed from HRTF data. Gaussian noise was added to these templates and the results fed to a pattern recognition algorithm in an attempt to determine if sucient information was available to distinguish position based on the ILD spectrum alone. The authors report that discrimination was possible, and that as the available bandwidth of the signal was reduced, localization ability was reduced in much the same manner as in humans. 3

4 One model that attempts to estimate both azimuth and elevation from binaural signals is described by Dudain[5]. Duda's model uses only interaural level dierences, which isavalid choice because his HRTF data (SLV data from [19] and 1991 KEMAR data) contains little useful low-frequency information (an artifact of the measurement technique). As predicted in [2], the most useful ITD information is expected to be present at frequencies below approximately 1.4 khz. Duda's model succeeds, with a maximum-likelihood estimator similar to the one proposed in this thesis, in the estimation of both azimuth and elevation with accuracy Duda likens to that of humans. There are two notable shortcomings in Duda's model. First, the failure to use ITD information makes it dicult to realistically compare the model performance with human performance. Second, Duda uses head-related impulse responses (HRIRs) as both training and test data, rather than attempting to use \natural" signals. Thus, he shows that the HRIRs contain information that can resolve positional ambiguity, but he fails to show how this information can be exploited in a natural listening environment. Neither model makes any attempt to model precedence-related eects Precedence-related eects In a reverberant environment, \direct" sound from a sound source arrives at a given point in space slightly before energy reected from various surfaces in the acoustic space (this is simply due to the geometric fact that the shortest distance between two points is a straight line). For evolutionary reasons, it may have been important for localization to be based on the direct sound, which generally reveals the true location of the sound source ([24]). Regardless of the origin of the precedence eect, it is true that for purposes of localization, the interaural spectrum is weighted more heavily by the auditory periphery at the onsets of sounds. Quantitative measures of this eect are presented by Zurek in [23]. The experimental measurements in [23] and the model described in [24] are the basis of the precedence eect model employed in this thesis. The precedence eect, as it has just been described, is a relatively short term eect. The suppression mechanism described in [24] operates on a time scale of a few milliseconds. However, localization in a room is fairly robust in the presence of reverberation, which operates on a time scale of seconds. To deal with such long term eects, the localization mechanism must be extended. Several long term precedence-related eects have been noted. For example, when only the sharp onset of a tone is played through one loudspeaker, while the remainder of the tone is presented through a second loudspeaker, the tone is localized at (or near) the rst loudspeaker. This is an example of the Franssen eect, and it is robust in the presence of head movements and for long duration tones. 1.4 Motivation and goals The primary goal of this thesis work is to present a localization model that analyzes \natural" binaural signals, estimating both azimuth and elevation. As in the human auditory periphery, onsets are strongly weighted in the determination of the interaural spectrum. The weighting of onset information is intended to model the precedence eect. This weighting, combined with an old-plus-new heuristic, and may allow the model to distinguish the actual location of a sound source in the presence of acoustic reections and reverberation and may 4

5 oer an explanation of the Franssen and other long-term precedence-related eects. Another important aspect of the proposed model is that it divides up the frequency spectrum, which allows the estimation of an interaural spectrum. The motivation behind this frequency analysis is physiologically driven. The proposed model is intended to be able to resolve \cone of confusion" ambiguities for a sound source in a reverberant environment. It should be able to \localize" many types of sound sources robustly. Some of these sources might include human voice (sharp attacks), musical instruments (relatively soft attacks), and \noise" signals. It will be interesting and instructive to see how the model resolves \paradoxical" signals with conicting or unrealistic binaural cues. As the model develops, its performance will be gauged against that of human listeners. The specic comparisons planned will be discussed in a later section. 5

6 Chapter 2 Procedure 2.1 Pieces of the model Overview of the model The model described in this research consists of several independent pieces that may be separately considered and implemented. Figure 2-1 is a block diagram of the proposed model. "Precedence Effect" Model Filter Bank Intensity Envelope Onset Detection From Right Ear Left Ear From Right Ear Interaural Spectrum Estimation Onset related "Suppression" Position Estimation Figure 2-1: Block diagram of the proposed model. All signal lines (except the initial ear input) consist of multiple channels Filter bank The \front end" of the model is an analysis lter bank. Currently, a\gamma-tone" lter bank ([17]) modelled after the basilar membrane is being employed. Other lter banks, possibly including generalized constant-q, and the short-time Fourier transform are also being considered. The author believes that the exact form of the lter bank is not of paramount importance. Rather, the important feature of all of the lter banks mentioned is that they divide up the frequency spectrum into portions that can be individually analyzed. 6

7 2.1.3 Intensity envelope The second stage of the proposed model is the calculation of an intensity envelope for each lter bank channel. The resulting envelope is used by the onset detector to determine the appropriate \suppression" for the signal as a function of time ([24]) and is employed in the long term precedence eect model as part of the old-plus-new heuristic. The intensity envelope may also be used to estimate interaural timing information, which some authors have suggested may be an important localization cue at frequencies above approximately 1.4 khz ([2]). The exact details of the intensity envelope processing have not been determined at this time Interaural spectrum The interaural spectrum is formed from the outputs of the lter bank and the intensity envelope processor. As specied earlier, ILD and ITD will be estimated as functions of frequency. The details of this portion of the model have notyet been determined and are likely to change as the project progresses Precedence eect model Long term eects such as the Franssen eect might be explained by an old-plus-new heuristic with some memory decay related to the time of the last \onset". It is expected that a simple mechanism may be constructed that will duplicate many dierent eects, including localization in the presence of reverberation and the Franssen eect. Additionally, it is expected that the same mechanism will exhibit performance similar to humans in traditional \precedence eect" experiments (see [23]). The development and implementation of the precedence eect model is expected to form a signicant portion of the total work. The form of the model is expected to be similar to the suppression mechanism described in [24], coupled with an old-plus-new heuristic for long term suppression Maximum likelihood position estimation The maximum likelihood position estimator uses the output of the interaural spectrum estimator, weighted by onset-related suppression, as its feature vector (Note that the feature vector varies with time, and thus the outputs of the estimator will also be time-varying.). Using a central limit theorem argument, one can argue that the estimated interaural spectrum will have a vector Gaussian distribution, centered on means that can be estimated directly from the HRTF data. If the variance of the measurements can be adequately estimated, the maximum likelihood estimator can use that information to compare the feature vectors in a Mahalanobis distance sense. In eect, this implements automatic \optimum" weighting of the \reliable" parts of a signal. The likely result is that ILD information is weighted more heavily in higher frequency ranges, and ITD information is weighted more heavily in lower frequency ranges. It will be interesting and informative to compare the features that the model favors with those that human listeners seem to favor. One of the principal advantages of using the Gaussian distribution assumption with a maximum likelihood estimator is that the estimation can be performed with simple matrix 7

8 multiplications. In a small temporal window, if there are only onsets in a small subset of the frequency bands, a meaningful estimate can be derived from only the available information (through a smaller matrix multiplication). Another result of this is that multiple source locations can be estimated from the same source signal by dividing up the spectrum appropriately. There are still several issues that will require serious consideration in this portion of the model. First and foremost, it must be determined whether or not the Gaussian model of the interaural spectrum is a reasonable assumption. A method of extracting the mean feature vectors from the HRTF data is needed. Also, a reasonable method for estimating the variance of the various parameters must be constructed. Finally, in order for the model to \graduate" from discrete to continuous estimation, a method of interpolating between mean vectors must be determined. This might possibly be done with radial basis functions, but that is beyond the scope of the problem at this point. 2.2 Test signals The inputs to the proposed localization model are the acoustic signals measured at the eardrums of a \dummy-head microphone". It is quite dicult to obtain directly test signals that cover a broad range of locations around the head, so another approach has been taken. Using head-related impulse responses measured for a dummy-head microphone, it is possible to synthesize binaural signals for all measured locations around the head. The principal data requirement for this thesis is a set of head-related transfer function measurements. To this end, a set of HRTFs (actually HRIRs) of a KEMAR mannequin has been measured (see [10]). The data set contains the impulse responses at each eardrum for 710 sound source positions on a 1.4 meter radius sphere around the mannequin head (at elevations between -40 and 90 degrees). These impulse responses can be used both for synthesis of test signals and for generating mean vectors and variance estimates of the interaural spectrum for use in the maximum-likelihood estimator. The synthesis process at its simplest consists only of convolution of a monophonic source signal with the appropriate HRIRs. The resulting signal is akin to a sound source presented in an anechoic environment, and thus does not sound very \natural". To overcome this problem, it is a simple matter to introduce reections and reverberation to the signal. This helps to \externalize" the sound and makes the signal sound more natural (see [6] and [9]). Additionally, variation of the direct to reverberant level ratio results in variation of perceived sound source distance ([6]). While the proposed model does not attempt to determine sound source distance, this might be a useful extension to the current research. An application called \space" has been developed to synthesize test signals using the KEMAR HRIRs. space allows the user to \spatialize" monophonic sound les by convolution with HRIRs and the addition of reverberation. It is designed to make it a simple matter to mix together several \spatialized" sounds and ouput a single sound le with the resulting signal. The resulting stereophonic signal is sampled at 44.1 khz and is intended to be perceptually similar to the signal that would arrive at the eardrums of KEMAR from appropriately positioned sources. Test signals will be synthesized from speech, music, and noise signals. Possibly, \paradoxical" signals will also be tested. 8

9 2.3 Comparison with published data Once the model is complete, its performance will be gauged against human performance. The precedence eect data obtained in [23] may be directly compared with data obtained from this localization model under \identical" experimental circumstances. The localization performance of the model will be examined for various reverberation levels and the relationship of localization blur to reverberation level will be quantied. Signals that elicit the Franssen eect in human listeners will be presented to the model and the presence of the eect in the model's \perception" will be conrmed. Other long term precedence-related eects will also be tested, in hopes that further precedence eect experiments for human listeners will be suggested by the model's performance. 2.4 Time schedule The proposed time schedule for this project is shown in Table 1. It is, of course, subject to large variations as parts of the model are developed, enhanced, and t together. Description KEMAR HRTF measurements Development of signal synthesis software Synthesis of test signals Filter bank implementation Intensity envelope implementation Precedence eect development and implementation interaural spectrum estimation implementation Maximum likelihood estimator development and implementation Replication of data in [23] Other comparisons with human data Total Estimated time requirement 60 hours 40 hours 10 hours 10 hours 20 hours 80+ hours 40 hours 40 hours 40 hours 20+ hours 360+ Hours Table 1: Time schedule for thesis project 9

10 Chapter 3 Equipment and facilities The computing resources required to complete this thesis include a UNIX workstation equipped with the MATLAB software package and C development tools, which will be provided by the Machine Listening group of the MIT Media Laboratory under the direction of Professor Barry Vercoe. 10

11 Bibliography [1] Jens Blauert. \Some Consideration of Binaural Cross Correlation Analysis". Acustica, 39(2):96{104, [2] Jens Blauert. Spatial Hearing. MIT Press, Cambridge, MA, [3] Albert S. Bregman. Auditory Scene Analysis. MIT Press, Cambridge, MA, [4] H. Steven Colburn and Nathaniel I. Durlach. \Models of Binaural Interaction". In Handbook of Perception, volume IV, chapter 10, pages 467{518. Academic Press Inc, [5] Richard O. Duda. \Estimating azimuth and elevation from the interaural intensity difference". Unpublished draft of a paper to be submitted to J. Acoust. Soc. Am., September [6] N. I. Durlach, A. Rigopulos, X. D. Pang, W. S. Woods, A. Kulkarni, H. S. Colburn, and E. M. Wenzel. \On the Externalization of Auditory Images". Presence, 1(2):251{257, Spring [7] Nathaniel I. Durlach and H. Steven Colburn. \Binaural Phenomena". In Handbook of Perception, volume IV, chapter 10, pages 365{466. Academic Press Inc, [8] Daniel P. W. Ellis and Barry L. Vercoe. \A perceptual Representation of Sound for Auditory Signal Separation". In presented at the 123rd meeting of the Acoustical Society of America, Salt Lake City, May [9] Werner Gaik. \Combined evaluation of interaural time and intensity dierences: Psychoacoustic results and computer modeling". J. Acoust. Soc. Am., 94(1):98{110, [10] Bill Gardner and Keith Martin. \HRTF Measurements of a KEMAR Dummy-Head Microphone". Perceptual Computing Technical Report #280, [11] William Morris Hartmann. Localization of a source of sound in a room. In AES 8th International Conference, [12] William Morris Hartmann and Brad Rakerd. \Localization of sound in rooms IV: The Franssen eect". J. Acoust. Soc. Am., 86(4):1366{1373, [13] George F. Kuhn. \Physical Acoustics and Measurements Pertaining to Directional Hearing". In W. A. Yost and G. Gourevitch, editors, Directional Hearing, chapter 1, pages 3{25. Springer-Verlag, New York,

12 [14] A. V. Oppenheim and R. W. Schafer. Discrete-Time Signal Processing. Prentice-Hall, Englewood Clis, NJ, [15] Athanasios Papoulis. Probability, Random Variables, and Stochastic Processes. McGraw-Hill, New York, NY, third edition, [16] Trevor M. Shackleton, Ray Meddis, and Michael J. Hewitt. \Across frequency integration in a model of lateralization". J. Acoust. Soc. Am., 91(4):2276{2279, April [17] Malcolm Slaney. Auditory Toolbox. Apple Technical Report #45, Apple Computer, Inc., [18] R. M. Stern, A. S. Zeiberg, and C. Trahiotis. \Lateralization of complex binaural stimuli: A weighted-image model". J. Acoust. Soc. Am., 84(1):156{165, April [19] F. L. Wightman and D. J. Kistler. \Headphone simulation of free-eld listening. I: Stimulus Synthesis". J. Acoust. Soc. Am., 85(2):858{867, [20] Frederic L. Wightman and Doris J. Kistler. Hearing in three dimensions: Sound localization. In AES 8th International Conference, [21] Frederic L. Wightman, Doris J. Kistler, and Mark E. Perkins. \A New Approach to the Study of Human Sound Localization". In W. A. Yost and G. Gourevitch, editors, Directional Hearing, chapter 2, pages 26{48. Springer-Verlag, New York, [22] W. A. Yost and G. Gourevitch, editors. Directional Hearing. Springer-Verlag, New York, [23] P. M. Zurek. \The precedence eect and its possible role in the avoidance of interaural ambiguities". J. Acoust. Soc. Am., 67(3):952{964, March [24] P. M. Zurek. \The Precedence Eect". In W. A. Yost and G. Gourevitch, editors, Directional Hearing, chapter 4, pages 85{105. Springer-Verlag, New York, [25] P. M. Zurek. \A note on onset eects in binaural hearing". J. Acoust. Soc. Am., 93(2):1200{1201,