Introduction to Computational Neuroscience Lecture 10: Brain-Computer Interfaces Ilya Kuzovkin
So Far
Stimulus So Far
So Far Stimulus What are the neuroimaging techniques you know about?
Stimulus So Far
Stimulus So Far
So Far Stimulus What can we do with this data?
So Far Stimulus Try to understand the neural code Diagnose diseases Psychological studies Guess stimulus from the data
So Far Stimulus Try to understand the neural code Diagnose diseases Psychological studies Guess stimulus from the data
How this can be useful?
How this can be useful?
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost
Electrocorticography (ECoG) Higher temporal (3 ms) and spatial (1 mm) resolutions Higher amplitudes Lower vulnerability to artifacts (eye blinks, etc.)
Electrocorticography (ECoG) Higher temporal (3 ms) and spatial (1 mm) resolutions Higher amplitudes Lower vulnerability to artifacts (eye blinks, etc.) http://wiki.neurotycho.org/ecog_for_primates http://www.sciencedirect.com/science/article/pii/s0168010203000075
Electrocorticography (ECoG) Higher temporal (3 ms) and spatial (1 mm) resolutions Higher amplitudes Lower vulnerability to artifacts (eye blinks, etc.) http://wiki.neurotycho.org/ecog_for_primates http://www.sciencedirect.com/science/article/pii/s0168010203000075
Electrocorticography (ECoG) Higher temporal (3 ms) and spatial (1 mm) resolutions Higher amplitudes Lower vulnerability to artifacts (eye blinks, etc.) http://wiki.neurotycho.org/ecog_for_primates http://www.sciencedirect.com/science/article/pii/s0168010203000075
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP
Machine Learning
Machine Learning
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP
P300 Test subject is instructed to wait for specific stimulus All targets flash in random order ~300ms after the the expected stimulus is presented, test subject generates positive peak in central and parietal cortex
P300 Test subject is instructed to wait for specific stimulus All targets flash in random order ~300ms after the the expected stimulus is presented, test subject generates positive peak in central and parietal cortex
P300 Test subject is instructed to wait for specific stimulus All targets flash in random order ~300ms after the the expected stimulus is presented, test subject generates positive peak in central and parietal cortex Explain the algorithm which will allow user to type letters
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP
Visually Evoked Potential (VEP) 1. Each target (square) flashes in a unique way 2. Brain react on each flashing pattern in a unique* way 3. User stares at the target he is interested in (letter N ) 4. We see how his brain reacts 5. From that we know what was the target he was looking at
Visually Evoked Potential (VEP) t-vep (time modulated) Only one target is ON at a time http://sccn.ucsd.edu/~yijun/pdfs/ieeecim09.pdf
Visually Evoked Potential (VEP) t-vep (time modulated) Only one target is ON at a time f-vep (frequency modulated) Each target is flashing on it s own frequency http://sccn.ucsd.edu/~yijun/pdfs/ieeecim09.pdf
Visually Evoked Potential (VEP) t-vep (time modulated) Only one target is ON at a time f-vep (frequency modulated) Each target is flashing on it s own frequency c-vep (code modulated) Bit sequence is generated (m-seq) EEG response pattern is recorded Each target is represented by the same sequence with a shift Target is identified by calculating the correlation between signal x and all of the patterns http://sccn.ucsd.edu/~yijun/pdfs/ieeecim09.pdf
Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP
Summary of Neuroimaging / BCI Techniques Technology Electrical Magnetic Optical Name EEG ECoG Intracortical MEG fmri fnirs Invasive Portable!! Cost From $100 to $30,000+ $1000 grid $2000 per array $1 mln $2-3 mln $200,000 Temporal resolution 50 ms 3 ms 3 ms 50ms 1-2 s 1 s Spatial resolution 1+ cm 1 mm 0.5 mm - 0.05 mm 5 mm 1 mm voxels 5 mm Pattern! classification VEP ERD/ ERS P300 Performance 2 class 90% 3 class 80% 4 class? Large number of targets 2 cls 90% Large number of targets 8 cls 90% High* ~ same as EEG based 4 cls 90% 2 cls 90%
Questions?
One example of BCI system Leigh R. Hochberg et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm Nature, 17 May 2012 If time allows
Neural Coding
Neural Coding of Hand Kinematics
Neural Coding of Hand Kinematics
Neural Coding of Hand Kinematics
Neural Coding of Hand Kinematics Experiment 1: 23/25 neurons are correctly described by equations (4) and (5)! Experiment 2: 39/42 neurons correctly described by (4) and (5)
Neural Coding of Hand Kinematics Experiment 1: 23/25 neurons are correctly described by equations (4) and (5)! The relationship between the kinematics of the arm and the behavior of the neurons is strong Experiment 2: 39/42 neurons correctly described by (4) and (5)
uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference
uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons.
uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons.
uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Posterior probability Likelihood Prior probability Hypothesis (hand motion) Evidence (sequence of observed firing rates) Marginal likelihood (can be ignored since it is the same for all hypothesis)
uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons.
uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Likelihood term models the probability of firing rates given a particular hand motion
uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Likelihood term models the probability of firing rates given a particular hand motion linear Gaussian model could be used to approximate this likelihood and could be readily learned from a small amount of training data
uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Likelihood term models the probability of firing rates given a particular hand motion linear Gaussian model could be used to approximate this likelihood and could be readily learned from a small amount The prior term defines a of training data probabilistic model of hand kinematics and was also taken to be a linear Gaussian model.
Learning the model
Definitions
Definitions
Parameters of the model
Parameters of the model H is the relation between the firing rates of each of the neurons and states of the arm Q is covariance matrix of the noise
Parameters of the model H is the relation between the firing rates of each of the neurons and states of the arm Q is covariance matrix of the noise A is the relation between the state at time k+1 and the state at time k W is covariance matrix of the noise
Parameters of the model H is the relation between the firing rates of each of the neurons and states of the arm Q is covariance matrix of the noise A is the relation between the state at time k+1 and the state at time k W is covariance matrix of the noise Matrices A, H, Q, W is what we want to learn from the training data
The Learning
Decoding
Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference Note that now x and z and everything else refer to the test data
Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference
Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference The probability that the hand can move in the way it did
Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference The probability that the hand can move in the way it did The probability that hand can end up in the state where it was in time k-1
Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference the Kalman filter operates recursively on streams of noisy input data to produce a statistically optimal estimate of the underlying system state. (Wikipedia)
Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference
Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference
Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference
Results