Active sensing. Ehud Ahissar 1

Active sensing Ehud Ahissar 1

Active sensing Passive vs active touch Comparison across senses Basic coding principles -------- Perceptual loops Sensation-targeted motor control Proprioception Controlled variables Active vibrissal touch: encoding and recoding 2

Eye movements during fixation 3

sensory encoding: What receptors tell the brain Sensory organs consist of receptor arrays: somatosensation audition vision ~200 m Finger pad 10 m cochlea 10 m retina Spatial organization => Spatial coding ( which receptors are activated ) Movements => Temporal coding ( when are receptors activated ) 4

Temporal coding in action 5

Coding space by time 1. Spatial frequency 2. Spatial phase 6

Touch: Temporal encoding of spatial features Darian-Smith & Oke, J Physiol, 1980 anesth. monkey, MR fibers 7

RA fiber Vel - constant f = SF * V dt = dx / V 8

SF Vel SA fiber 9

SF Vel RA fiber V1 V2 V3 G1 G2 G3 10

SF Vel PC fiber 11

Coding ranges 12

Coding space by time 1. Spatial frequency 2. Spatial phase 13

Vision: Temporal encoding due to eye movement space Veye RF(1) RF(2) space retinal outputs 1 2 time 14

Vision: Temporal encoding due to eye movement x space Veye RF(1) RF(2) space retinal outputs 1 2 t time 15

Vision: Temporal encoding due to eye movement x space Veye RF(1) RF(2) space retinal outputs 1 2 t time 16

Vision: Temporal encoding due to eye movement x space Veye RF(1) RF(2) space retinal outputs 1 2 t time 17

Vision: Temporal encoding due to eye movement x space Veye RF(1) RF(2) space retinal outputs 1 2 t time 18

Spatial vs temporal coding Spatial Temporal faster better resolution scanning allows sensing in between receptors 19

Passive vs Active sensing of stationary objects Passive Active threshold low high accuracy low high Systems involved sensory Sensory + motor coding spatial Spatial + temporal Processing speed fast slow Used in detection Exploration Localization Identification 20

Central processing of touch where touch begins? at the receptors? with sensor movement? in high brain centers? 21

Sensory-motor loops of the vibrissal system Cortex Primary Sensory Cortex Secondary Primary Motor Cortex Zona Incerta VPM-dm VPM-vl Thalamus POm - VL Thalamic Nuclei Sensory extralemniscal Cerebellar/Olivary Red Nucleus Pontine Reticular Nucleus Superior Colliculus Brainstem Reticular Formation Motor Trigeminal Nuclei Brainstem Loop + Facial Nucleus Trigeminal Ganglion Vibrissae 22

Sensory-motor loops of the vibrissal system Cortex E D C Identification ( what ) Localization ( where ) B Whisking A m a The old view Thalamus WT T W Brainstem WT T W 23

Cortex E D C Identification ( what ) Localization ( where ) B Whisking A m a Thalamus WT T W Brainstem WT T W 25

Sensory-motor loops of the vibrissal system E D C B A m a WT T W WT T W 26

Central processing of touch where touch begins? at the receptors? with sensor movement? in high brain centers? Active touch does not begin at the receptors 27

Motor control Closed loops Proprioceptive feedback Reflexes tool for probing loop function Controlled variables motor vs sensory 28

Motor control Closed loops Proprioceptive feedback Reflexes tool for probing loop function Controlled variables motor vs sensory 29

Excitation Contraction Coupling Phase 1: Firing of Motor Neuron Input to motorneuron Phase 2: Release of Neurotransmitter 30

Excitation Contraction Coupling Phase 1: Firing of Motor Neuron Input to motorneuron Phase 2: Release of Neurotransmitter Phase 3: Muscle contraction 31

Open-loop system Information flows in one direction (from neurons to muscles Input to motorneuron 32

Open-loop system Information flows in one direction (from neurons to muscles Closed-loop system Input to motorneuron Information flows in a closed loop: from neurons to muscles and from muscles to neurons What kind of information? 33

Closed-loop system The direct feedback from muscles and joints is mediated by proprioceptive signals Proprioceptive receptor types Name: Muscle spindle receptors Golgi tendon organs Joint receptors Sensitive to: muscle length muscle tension Flexion, extension 34

Proprioceptive receptor types Name: Muscle spindle receptors Golgi tendon organs Joint receptors Sensitive to: muscle length muscle tension Flexion, extension Location: Fleshy part of the muscle Between muscle and tendon Joint capsule Parallel to muscle fibers Serial to muscle fibers 35 Between bones

Proprioceptive receptor types Name: Muscle spindle receptors Golgi tendon organs Joint receptors Sensitive to: muscle length muscle tension Flexion, extension 36

Motor control Closed loops Proprioceptive feedback Reflexes tool for probing loop function Controlled variables motor vs sensory 37

What proprioceptors encode? 38

Proprioceptive receptor types Name: Muscle spindle receptors Golgi tendon organs Joint receptors Sensitive to: muscle length muscle tension Flexion, extension From Arthur Prochazka, University of Alberta 39

Proprioceptive receptor types Name: Muscle spindle receptors Golgi tendon organs Joint receptors Sensitive to: muscle length muscle tension Flexion, extension Encode: force f = k 1 F 40

Proprioceptive receptor types Name: Muscle spindle receptors Golgi tendon organs Joint receptors Sensitive to: muscle length muscle tension Flexion, extension Encode: Length + velocity f = k 1 L + k 2 V 0.6 force f = k 1 F angle f = k 1 41

PID control Proportional (to the controlled variable) Integral (of the controlled variable) Derivative (of the controlled variable) Present Past Future 43

Negative feedback loop Characteristic: The effect of a perturbation is in opposite direction Requirement: The cumulative sign along the loop is negative Function: Can keep stable fixed points 44

Positive feedback loop Characteristic: The effect of a perturbation is in the same direction Requirement: The cumulative sign along the loop is positive Function: amplifies perturbations + 45

Motor control Closed loops Proprioceptive feedback Reflexes tool for probing loop function Controlled variables motor vs sensory 46

The stretch reflex probes the control function of muscle spindles 47

Is the loop positive or negative? The stroke stretches the spindle As a result the muscle contracts The result opposes the perturbation => negative FB loop 48

the anatomical loop Muscle spindle excites the motor neuron Motor neuron excites muscle fibers Muscle contraction suppresses spindle response 49

Proprioceptive receptor types Name: Muscle spindle receptors Golgi tendon organs Joint receptors Sensitive to: muscle length muscle tension Flexion, extension Encode: force f = k 1 F Why spindles fire at rest? 50

What about the flexor muscles? Positive or negative loop? What is the underlying circuit? 51

Pain reflex Positive or negative? What is the underlying 52 circuit?

Motor control Closed loops Proprioceptive feedback Reflexes tool for probing loop function Controlled variables motor vs sensory 53

Basic principles of closed-loop control 55

Set point - + Vd f Vs Vm=f(-Vs) Vs=g(Vm) Vs Vm V s0 V m0 Vm V 56

Set point - + Vd f Vs Vm=f(Vd-Vs) Vs=g(Vm) Vs Vm V s0 Vsd V md V m0 Vm V 57

Direct control without direct connection - + Vd f Vs Vm=f(Vd-Vs) Vs=g(Vm) Vs Vm V s0 Vsd V md V m0 Vm V 58

Nested loops + - f 2 - + Vd f Vs Vm=f(Vd-Vs) Vs=g(Vm) Vs Vm V s0 Vsd V md V m0 Vm V 59

Parallel loops + - f 2 Vs Vm2=f(Vd-Vs)Vs=g(Vm2) - + f Vm2 V s0 Vsd V md V m0 Vm2 Vs Vm1 Vs V s0 Vsd Vm1=f(Vd-Vs)Vs=g(Vm1) V V md V m0 Vm1 60

Parallel loops + - f 2 Xs Xm=f(Xd-Xs) Xs=g(Xm) - + f Xm X s0 Xsd X md X m0 Xm Xs Vs Vm Vs Vm=f(Vd-Vs) Vs=g(Vm) V s0 Vsd V V md V m0 Vm1 61

Closed loops in active sensing The controlled variables can be - - + f 2 + f Motor (via Xs) (velocity, amplitude, duration, direction, ) Sensory (Xs) (Intensity, phase, ) Object (via Xm Xs relationships) (location, SF, identity, ) Xs Vs Vm Xm V 62

Servo control Vm = (Vd-Vs)dt - + Vdesired Vs Vm= Vs= Vmotor Vd = V s0 Vsensed V m0 Vm V 63

Active sensing in the vibrissal system 68

Sensory signal conduction The vibrissal system 69

whisker Sensory signal conduction The vibrissal system Meisner Merkel Ruffini Lanceolate free endings 70

Motor control of whiskers Intrinsic muscles 73 Dorfl J, 1982, J Anat 135:147-154

Follicle as a motor-sensory junction Motor signals move the follicle and whisker Follicle receptors report back details of self motion = proprioception Plus perturbations of this motion caused by the external world Dorfl J, 1985, J Anat 142:173-184 74

Reception of neuronal signals in the brain Exteroception reception of the external world via the six senses: sight, taste, smell, touch, hearing, and balance Interoception: reception of the internal organs of the body Proprioception (from "one's own" and reception) reception of the relationships between the body and the world. Afferent signals that relate to the external world contain: Reafferent (self-generated) sensory signals Exafferent (externally generated) sensory signals 75

Proprioceptive receptor types Name: Muscle spindle receptors Golgi tendon organs Joint receptors Sensitive to: muscle length muscle tension Flexion, extension 76

Motor control of whiskers Intrinsic muscles 77 Dorfl J, 1982, J Anat 135:147-154

Vibrissal proprioception Each follicle contains ~2000 receptors About 20% of them convey pure proprioceptive information 78

Vibrissal system Skeletal system Proprioceptive loop Proprioceptive loop 79

Whiskers come with different muscle sizes Intrinsic muscles 0.5 mm 80 Dorfl J, 1982, J Anat 135:147-154

Whisking behavior reflections of control loops 81

Closed loops in active sensing The controlled variables can be - - + f 2 + f Motor (via Xs) (velocity, amplitude, duration) Sensory (Xs) (Intensity, phase, ) Object (via Xm Xs relationships) (location, SF, identity, ) Xs Vs Vm V Xm Tradeoffs Controlled versus modulated variables How to figure it out? perturbations 82

Vibrissal system Proprioceptive loop Skeletal system Proprioceptive loop Contact loop Contact loop? 83

Loop sign effect of context Negative contact loop Positive(?) contact loop Mitchinson, Martin, Grant, Prescott (2007) Proc R Soc Biol Sci 84

Perception of external objects Object localization What signals must the brain process in order to infer a location of an external object in space? Reafferent + exafferent signals 85

Mimicry of active touch Figure 22-8 The firing patterns of mechanoreceptors in the superficial layers of the skin encode the texture of objects rubbed across the skin. A. 1. The nerve responses to textures are measured with the hand immobilized. The receptive field of a single receptor on a monkey's finger is stimulated with an embossed array of raised dots on a rotating drum. The pattern moves horizontally over the receptive field as the drum rotates. The experimenter thus controls the speed of movement and the location of the dot pattern in the receptive field. The pattern is moved laterally on successive rotations to allow the dots to cross the medial, central, and lateral portions of the receptive field on successive rotations. The composite response of an individual nerve fiber to successive views of the raised dots simulates the distribution of active and inactive nerve fibers in the population. 2. Sequential action potentials discharged by individual receptors during each revolution of the drum are represented in spatial event plots in which each action potential is a small dot, and each horizontal row of dots represents a scan with the pattern shifted laterally on the finger. B. Spatial event plots of three types of mechanoreceptors to dot patterns with different spacing. Slowly adapting Merkel disk receptors and rapidly adapting Meissner's corpuscles differentiate between dots and blank space when the spacing of the dots exceeds the receptive field diameter. A receptor fires bursts of action potentials for each dot, spaced by silent intervals. As the dots are brought closer together, the resolution of individual dots blurs. Pacinian corpuscles do not distinguish texture patterns because their receptive fields are larger than the dot spacing. (Reproduced from Connor et al. 1990.) 86

What the whiskers tell the rat brain Reafference: Their own movement ( Whisking ) Exafference: Touch 87

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 88

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 89

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 90

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 91

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 92

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 93

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 94

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 95

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 96

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 97

What the whiskers tell the rat brain Whisking space Whisker position vs. time time 98

What the whiskers tell the rat brain Reafference: Their own movement ( Whisking ) Exafference: Touch 99

What the whiskers tell the rat brain Touch space Whisker position vs. time time 100

What the whiskers tell the rat brain Touch space Whisker position vs. time time 101

What the whiskers tell the rat brain Touch space Whisker position vs. time time 102

What the whiskers tell the rat brain Touch space Whisker position vs. time time 103

What the whiskers tell the rat brain Touch space Whisker position vs. time time 104

What the whiskers tell the rat brain Touch space Whisker position vs. time time 105

What the whiskers tell the rat brain Touch space Whisker position vs. time time 106

What the whiskers tell the rat brain Touch space Whisker position vs. time time 107

What the whiskers tell the rat brain Touch space Whisker position vs. time time 108

What the whiskers tell the rat brain Touch space Whisker position vs. time time 109

What the whiskers tell the rat brain Touch space Whisker position vs. time time 110

Whisking: What the whiskers tell the rat brain How can the brain use this information? space Whisker position vs. time time Touch: contact with object space Whisker position vs. time time 111

Whisking: What the whiskers tell the rat brain How can the brain use this information? space? Whisker position vs. time time Touch: contact with object space? Whisker position vs. time time 112

Whisking: How can the brain extract the location of the object space Whisker position vs. time time Touch: contact with object 113

Whisking: How can the brain extract the location of the object space Whisker position vs. time time Touch: contact with object 114

Orthogonal coding of object location Vertical object position is encoded by space Horizontal object position is encoded by time Radialobject position is encoded by rate 116

Active sensing The End 117