Introduction to Computational Neuroscience

Transcription

1 Introduction to Computational Neuroscience Lecture 10: Brain-Computer Interfaces Ilya Kuzovkin

2

3 So Far

4 Stimulus So Far

5 So Far Stimulus What are the neuroimaging techniques you know about?

6 Stimulus So Far

7 Stimulus So Far

8 So Far Stimulus What can we do with this data?

9 So Far Stimulus Try to understand the neural code Diagnose diseases Psychological studies Guess stimulus from the data

10 So Far Stimulus Try to understand the neural code Diagnose diseases Psychological studies Guess stimulus from the data

11 How this can be useful?

12 How this can be useful?

13 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP

14 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost

15 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost

16 Electrocorticography (ECoG) Higher temporal (3 ms) and spatial (1 mm) resolutions Higher amplitudes Lower vulnerability to artifacts (eye blinks, etc.)

17 Electrocorticography (ECoG) Higher temporal (3 ms) and spatial (1 mm) resolutions Higher amplitudes Lower vulnerability to artifacts (eye blinks, etc.)

18 Electrocorticography (ECoG) Higher temporal (3 ms) and spatial (1 mm) resolutions Higher amplitudes Lower vulnerability to artifacts (eye blinks, etc.)

19 Electrocorticography (ECoG) Higher temporal (3 ms) and spatial (1 mm) resolutions Higher amplitudes Lower vulnerability to artifacts (eye blinks, etc.)

20 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP

21 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost

22 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost

23 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost

24 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP How it works? Temporal resolution Spatial resolution Advantages Disadvantages Portable? Cost

25 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP

26 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP

27 Machine Learning

28 Machine Learning

29 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP

30 P300 Test subject is instructed to wait for specific stimulus All targets flash in random order ~300ms after the the expected stimulus is presented, test subject generates positive peak in central and parietal cortex

31 P300 Test subject is instructed to wait for specific stimulus All targets flash in random order ~300ms after the the expected stimulus is presented, test subject generates positive peak in central and parietal cortex

32 P300 Test subject is instructed to wait for specific stimulus All targets flash in random order ~300ms after the the expected stimulus is presented, test subject generates positive peak in central and parietal cortex Explain the algorithm which will allow user to type letters

33 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP

34 Visually Evoked Potential (VEP) 1. Each target (square) flashes in a unique way 2. Brain react on each flashing pattern in a unique* way 3. User stares at the target he is interested in (letter N ) 4. We see how his brain reacts 5. From that we know what was the target he was looking at

35 Visually Evoked Potential (VEP) t-vep (time modulated) Only one target is ON at a time

36 Visually Evoked Potential (VEP) t-vep (time modulated) Only one target is ON at a time f-vep (frequency modulated) Each target is flashing on it s own frequency

37 Visually Evoked Potential (VEP) t-vep (time modulated) Only one target is ON at a time f-vep (frequency modulated) Each target is flashing on it s own frequency c-vep (code modulated) Bit sequence is generated (m-seq) EEG response pattern is recorded Each target is represented by the same sequence with a shift Target is identified by calculating the correlation between signal x and all of the patterns

38 Invasive Non-invasive EEG Microelectrodes ECoG fnirs fmri MEG Method Machine Learning P300 VEP

39 Summary of Neuroimaging / BCI Techniques Technology Electrical Magnetic Optical Name EEG ECoG Intracortical MEG fmri fnirs Invasive Portable!! Cost From $100 to $30,000+ $1000 grid $2000 per array $1 mln $2-3 mln $200,000 Temporal resolution 50 ms 3 ms 3 ms 50ms 1-2 s 1 s Spatial resolution 1+ cm 1 mm 0.5 mm mm 5 mm 1 mm voxels 5 mm Pattern! classification VEP ERD/ ERS P300 Performance 2 class 90% 3 class 80% 4 class? Large number of targets 2 cls 90% Large number of targets 8 cls 90% High* ~ same as EEG based 4 cls 90% 2 cls 90%

40 Questions?

41 One example of BCI system Leigh R. Hochberg et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm Nature, 17 May 2012 If time allows

42

43

44

45

46 Neural Coding

47 Neural Coding of Hand Kinematics

48 Neural Coding of Hand Kinematics

49 Neural Coding of Hand Kinematics

50 Neural Coding of Hand Kinematics Experiment 1: 23/25 neurons are correctly described by equations (4) and (5)! Experiment 2: 39/42 neurons correctly described by (4) and (5)

51 Neural Coding of Hand Kinematics Experiment 1: 23/25 neurons are correctly described by equations (4) and (5)! The relationship between the kinematics of the arm and the behavior of the neurons is strong Experiment 2: 39/42 neurons correctly described by (4) and (5)

52 uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference

53 uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons.

54 uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons.

55 uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Posterior probability Likelihood Prior probability Hypothesis (hand motion) Evidence (sequence of observed firing rates) Marginal likelihood (can be ignored since it is the same for all hypothesis)

56 uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons.

57 uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Likelihood term models the probability of firing rates given a particular hand motion

58 uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Likelihood term models the probability of firing rates given a particular hand motion linear Gaussian model could be used to approximate this likelihood and could be readily learned from a small amount of training data

59 uses Bayesian inference techniques to estimate hand motion from the firing rates of multiple neurons. Likelihood term models the probability of firing rates given a particular hand motion linear Gaussian model could be used to approximate this likelihood and could be readily learned from a small amount The prior term defines a of training data probabilistic model of hand kinematics and was also taken to be a linear Gaussian model.

60 Learning the model

61 Definitions

62 Definitions

63 Parameters of the model

64 Parameters of the model H is the relation between the firing rates of each of the neurons and states of the arm Q is covariance matrix of the noise

65 Parameters of the model H is the relation between the firing rates of each of the neurons and states of the arm Q is covariance matrix of the noise A is the relation between the state at time k+1 and the state at time k W is covariance matrix of the noise

66 Parameters of the model H is the relation between the firing rates of each of the neurons and states of the arm Q is covariance matrix of the noise A is the relation between the state at time k+1 and the state at time k W is covariance matrix of the noise Matrices A, H, Q, W is what we want to learn from the training data

67 The Learning

68 Decoding

69 Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference Note that now x and z and everything else refer to the test data

70 Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference

71 Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference The probability that the hand can move in the way it did

72 Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference The probability that the hand can move in the way it did The probability that hand can end up in the state where it was in time k-1

73 Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference the Kalman filter operates recursively on streams of noisy input data to produce a statistically optimal estimate of the underlying system state. (Wikipedia)

74 Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference

75 Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference

76 Decoding was performed using a Kalman filter which gives an efficient recursive method for Bayesian inference

77 Results

78