Sensory Cue Integration

Transcription

1 Sensory Cue Integration Summary by Byoung-Hee Kim Computer Science and Engineering (CSE)

2 Presentation Guideline Quiz on the gist of the chapter (5 min) Presenters: prepare one main question Students: read the material before the class Presentation (30 min) Include all equations and figures Limit of slides: maximum 20 pages + appendix (unlimited) Discussion (30 min) Understanding the contents Pros and cons / benefits and pitfalls Implications of the results Extensions or applications 2

3 Quiz (5 min) Q. (question on the gist of the chapter) List and explain briefly ideal observer models of cue integration 3

4 Contents Motivations and arguments Problems and experiments Ideal-observer models Linear models for maximum reliability Bayesian estimation and decision making Nonlinear models: generative models and hidden variables Issues and concerns Appendix 4

5 Estimation from Various Information Environment location 3D orientation Vision cues size depth Sensory information auditory cues Texture / shading Linear perspective binocular disparity, stereopsis Cue integration haptic cues Estimation and decision/action Motion planning Motor planning 5

6 Uncertain relationship btw cues and environmental properties Is this optimal? - Variability in the mapping btw the cue and a property - Errors in the nervous system s measurement of the cue - Measured cue values vary unpredictably across viewing conditions and scenes - Estimates may be based on assumptions about the scene and will be flawed if those assumptions are invalid 6

7 Motivations Studying perceptual computations Modeling cue combination General introduction to the fiend of cue combination from the perspective of optimal cue integration 7

8 Arguments The organism can make more accurate estimates of environmental properties or more beneficial decisions by integrating multiple sources of information Observers should be more likely to approach optimal behavior in tasks that are important for survival ideal-observer analysis is a critical step in the iterative scientific process of studying perceptual computations 8

9 Problems and experiments Estimation Target Surface orientation Distance to a drop-off Cues Visual / Haptic Visual / auditory Experimental Task Walk blindfolded toward the drop-off / Movement planning Size Visual / Haptic Checking JND, PSE Depth Visual (texture, shading) Seeing ridges as real objects or as computergraphic image 9

10 Ideal-observer models Cue combinations from the perspective of optimal cue integration Building ideal observers helps formulate the scientific questions that need to be answered before we can understand how the brain solves these problems Models Linear models for maximum reliability Bayesian estimation and decision making Nonlinear models: generative models and hidden variables 10

11 Linear models for maximum reliability Assumptions An observer has access to unbiased estimates of a particular world property from each cue The cues are Gaussian distributed (Gaussian noise) and conditionally independent (n cues è n independent, Gaussian random variables) The minimum-variance unbiased estimator is a weighted average of the individual estimates from each cue (eq. 1.1) r i : cue s reliability (inverse variance ) 11

12 BAYESIAN ESTIMATION AND DECISION MAKING 12

13 Bayesian decision theory as a more general framework Pitfalls of the linear model Providing important insights into human perceptual and sensorimotor processing Only provides a local approximation to the ideal observer Bayes Rule s: scene properties d: data likelihood prior posterior Normalizing term 13

14 Bayesian decision theory as a more general framework Bayesian decision maker Compute the posterior distribution Choose an estimate, a course of optimal action, based on the loss function An optimal choice of action is one that maximizes expected gain P(s): A model of the environment. Prior distribution on the scenes P(d s): Noisy sensory data d conditioned on a particular state of the world a(d): optimal action t: outcome of the decision or action plan. For estimation, g(t,s): negative of loss, or gain Special cases - ML estimation - MAP estimation - Mean of the posterior 14

15 Bayesian decision theory and cue integration Cue integration Assumption: sensory data associated with each cue are conditionally independent Likelihood and posterior Special cases For Gaussian, the MAP (maximum a posteriori) estimate and the mean of the posterior both yield a linear estimation procedure Flat prior yield the posterior as the product of cue likelihoods Conditional independence does not hold è weights should cover the covariance structure of the data 15

16 Bayesian integration of sensory cues Examples of two simple cases A: Two cues to object size, visual and haptic, each have Gaussian likelihoods B: Two visual cues to surface orientation are provided: skew symmetry (a figural cue) and stereo disparity 16

17 NONLINEAR MODELS: GENERATIVE MODELS AND HIDDEN VARIABLES 17

18 Problems and models in nonlinear cases Conditions under which optimal cue integration is not linear (cues interact) Cue disambiguation Raw sensory data from different cues are often incommensurate Mixture priors (Ch. 9) The true prior is a mixture of distributions Causal inference (Chs. 2, 3, 4, 13) Cues may derive from different sources The observer should infer the structure of the scene before estimation 18

19 Cue Disambiguation Viewing distance: hidden variable Estimation target Relative depth Cues Disparity Velocity * Promotion: preliminary conversion of cue values into common units 19

20 Use case of a mixture prior: Bayesian model of slant from texture Discrepant cue: cues may suggest very different values for some scene property Estimation target slant Cues Disparity Texture Long-tail A: compression cue. mixture of likelihood has long tail B: small cue conflicts. Disparities (red) suggest a slant which is slightly differ from the compression cue (blue) C: large cue conflicts. Model selection / model switching 20

21 Causal inference Cues may be derived from different sources The observer need to infer the structure of the scene, not just to estimate Location estimation from auditory and visual cues When two stimuli are presented in nearby locations, subjects estimates of the auditory stimulus are pulled toward the visual stimulus (the ventriloquist effect) When they are presented far apart, they appear to be separate sources and do not affect one another Model: Bayesian inference of structural models Probabilistic description of a generative model of the scene (two step process in Fig. 1.4) An observer has to invert the generative model and infer the locations of the visual and auditory sources location 21

22 Take home messages Bayesian decision theory provides a completely general normative framework for cue integration The representational framework used to model specific problems depends critically on the structure of the information available and the observer s task 22

23 THEORY MEETS DATA 23

24 Methodology A variety of experimental techniques has been used to test theories of cue integration Example: combination of visual and haptic cues to size Four kinds of stimuli: visual-only; haptic-only; two-cue, consistent stimuli; two-cue inconsistent stimuli Threshold value (just-noticeable difference, JND) is used to estimate the underlying single-cue noise To find the point of subjective equality (PSE) 24

25 Overview of results Experimental supports optimality of human perception Optimal linear cue integration Cue promotion is an issue for many cueintegration problems Evidence for robustness in intrasensory cue combination Human performance appears to be consistent with the predictions of mixture-prior model 25

26 ISSUES AND CONCERNS 26

27 Issues and Concerns Realism and unmodeled cues The lack of realism and the dearth of sensory cues in the laboratory may place the perceiver in situations for which the nervous systems is ill suited and therefore may perform suboptimally Considering unmodeled cues seems to be important (Buckley and Frisby, 1993) Estimation of uncertainty Measurement of the reliability of individual cues For intramodal cue integration, difficulties arise in isolating a cue Single-cue discrimination experiments are used to estimate the uncertainty associated with individual cues 27

28 Issues and Concerns Estimator bias Sensory calibration: sensory estimators maintain internal consistency and external accuracy Variable cue weights How human observers estimate and represent cue reliability? One suggestion: neural population code (Ch. 21) Simulation of the observer Where the prior comes from and how to estimate it è three different approaches in recent years (one in Ch. 11) 28

29 APPENDIX 29

30 OPEN QUESTIONS How is cue reliability estimated and represented in the nervous system? How optimal is cue integration w.r.t. the information that is available in the environment? When human cue integration is demonstrably suboptimal, what design considerations does the suboptimality reflect? 30

31 Marr's Tri-Level Hypothesis Computational level what does the system do (e.g.: what problems does it solve or overcome) and, equally importantly, why does it do these things Algorithmic/representational level how does the system do what it does, specifically, what representations does it use and what processes does it employ to build and manipulate the representations Implementational level how is the system physically realized (in the case of biological vision, what neural structures and neuronal activities implement the visual system) 31

32 32

33 33

34 The real-ridge experiment The real-ridge experiment The computer-display experiment 34

35 35