Hybrid Brain-Computer Interfaces

Size: px

Start display at page:

Download "Hybrid Brain-Computer Interfaces"

Juliet Morrison
6 years ago
Views:

1 Hybrid Brain-Computer Interfaces Lecture by at MPI Leipzig, 16th July 2014

2 Outline BCI Aim Technology: EEG Method: feature extraction, machine learning Results NIRS Hemodynamics Technology NIRS-EEG-BCI Experiment Results Discussion 2

3 Aim

4 Aim

5 Aim

6 Brain-Computer Interface (BCI) BCI: Translation of human intentions into a technical control signal without using activity of muscles or peripheral nerves

7 Overview of BCI Technology

8 TECHNOLOGY: EEG 8

Recap: EEG Measures the electrical activity of the brain by capturing electrical potential differences on the scalp surface Non-invasive method High

9 Recap: EEG Measures the electrical activity of the brain by capturing electrical potential differences on the scalp surface Non-invasive method High temporal resolution, but low spatial resolution Signal sensitive to: facial muscle contractions electrostatic artifacts head conductivity variation,...

Recap: Analysis of EEG Data Different neural responses Event-related potentials (ERPs) Change in activity of neuronal populations small voltage shifts Event-related (de)synchronization (ERD /

10 Recap: Analysis of EEG Data Different neural responses Event-related potentials (ERPs) Change in activity of neuronal populations small voltage shifts Event-related (de)synchronization (ERD / ERS) Change in synchrony of neuronal populations change of power in frequency bands - Exogeneous EEG components: induced by a stimulus/task - Endogenous EEG components: ongoing brain activity

11 Areas of the Brain Brain lobes: Frontal, Parietal, Temporal, Occipital

12 Topographic Mapping

13 Spectrum of Brain Activity Idealized spectrum of macroscopic brain activity

14 Modulation of Brain Rhythms Most rhythms are idle rhythms, i.e., they are attenuated during activation. - Alpha-rhythm (around 10~Hz) in visual cortex: - Mu-rhythm (around 10~Hz) in motor & sensory cortex:

15 Sensori-Motor-Rhythm

16 Sensori-Motor-Rhythm

17 ERD/S Curves of Motor Imagery ERD/S for selected frequency band (ERD: decrease of power, ERS: increase of power) (8-13 Hz) (absolute value) ERS

18 METHOD: FEATURE EXTRACTION, MACHINE LEARNING 18

19 SMR-based BCI Systems

20 ML Approach machine learning techniques and adaptive signal processing needed due to considerable variability

21 Linear Discriminant Analysis d 1 d 2 Question: does the new measurement x belong to class 1 or class 2? The distance between the class means class 1 LDA constructs a hyper plane (separation line) using the variance of each class and the overall variance x belongs to class 2 Non-linear machine learning methods are also available! 21

Experimental Design: Calibration 1 st step: recording data for calibration: Motor imagery task: left hand, right hand, right foot Instruction for

22 Experimental Design: Calibration 1 st step: recording data for calibration: Motor imagery task: left hand, right hand, right foot Instruction for participants by visual cues (arrows: left, right, down) Study here: large scale study with 80 participants. 75 trials per motor imagery condition and participant

23 Modulation of SMRs Motor imagery, data from one ideal participant with prototypical patterns:

24 Experimental Design: Feedback 2 nd step: Feedback: Instruction to participant (motor imagery, same as before), Cross starts moving according to classifier output Cross stops after 4s: final position is evaluated

25 Training CSP-based Classification (1) Determine most discriminative frequency band, (2) band-pass filter EEG in that band, (3) extract single trials using the time interval in which ERD/ERS is expected, (4) calculate and select CSP filters, (5) apply them to EEG single trials, (6) calculate the log variance within trials. Result: low dimensional feature vector for each trial (dimensionality = number of selected CSP filters) (7) Train a linear classifier on the features, such as LDA Result: classifier

26 CSP Processing Pipeline

27 Extraction of Band-Power Features (1) raw signal spectral analysis select discriminating frequency band (in grey) with heuristic: ERD/S: power modulation in selected frequency band

28 Extraction of Band-Power Features (2)

29 Analysis of Motor Imagery Conditions: Spectra Step (1): determine a frequency band with good discrimination between selected conditions (marked in grey here) select frequency band, e.g. using Butterworth/Chebychev filters (see EEG lecture)

30 ERD/S Curves of Motor Imagery

31 ERD Curves of Motor Imagery: Topography

32 CSP Analysis Goal: find spatial filters that optimally capture modulations of brain rhythms Observation: power of brain rhythms ~ variance of bandpass-filtered signal

33 CSP Analysis

34 RESULTS 34

35 Results 35

36 Conclusion a BCI system can be controlled based on the sensorimotor rhythm (SMR) BBCI approach: let the machines learn Experimental Design 1 st step: calibration recording data and let the classifier learn the specific neural response of a participant: > Feature extraction: determine frequency band, calculate ERD/S, determine CSP filters that optimally capture the spatial distribution -> LDA 2 nd step: feedback using this classifier, let the participant control a BCI in real-time BUT: 20-30% of the subjects are not able to perform a BCI

37 Outline BCI Aim Technology: EEG Method: feature extraction, machine learning Results NIRS Hemodynamics Technology NIRS-EEG-BCI Experiment Results Discussion 37

Concentration Changes of Oxy- and Deoxygenated Hemoglobin Blood exchanges molecules with the glia cells in the capillary bed Glia cells supply the neurons with energy Oxygen which is

38 Concentration Changes of Oxy- and Deoxygenated Hemoglobin Blood exchanges molecules with the glia cells in the capillary bed Glia cells supply the neurons with energy Oxygen which is needed for the energy metabolism Oxygen is supplied by hemoglobin Neuronal activity leads to a delayed blood flow increase Oxygenated hemoglobin, deoxygenated hemoglobin due to washout 38

39 From Stimulus to Hemodynamic Response magnetic properties fmri-response 39

40 Time courses 3 oxy-hb deoxy-hb 2 Average 2 c [ M] c [ M] time [s] time [s] c [ M] After every block of stimulation (shaded blue boxes) concentration of oxygenated hemoglobin Cyt-Ox (oxy-hb) increases while deoxygenated hemoglobin (deoxy-hb) decreases 0 An 36 average over stimuli provides the hemodynamic response time [s] Height of amplitude as sign of activity in the measured cortical area 40

41 Applications Neurovascular coupling Newborns and children (e.g. development of language) Brain-Computer Interfaces Bed-side monitoring of patients Freely behaving subjects -> natural environments 41

42 TECHNOLOGY 42

43 Photon Transport in the Human Brain Tissue Near-Infrared light can penetrate the brain banana-shaped measurement volume for non-invasive NIRS 43

44 Why Near-Infrared Light? In the NIR range of light absorption of water and hemoglobin is low Wavelengths have to be chosen between 700 and 950 nm Penetration depth of light is enhanced Cortex can be reached This spectral range is called optical window Outside this optical window light is absorbed by the first millimeters of the skin 44

45 Changes in local absorption are measured with two wavelengths. Simultaneous measurements with light at two distinct wavelengths (760 nm, 830 nm): 760 nm is more strongly absorbed by Hb 830 nm is more strongly absorbed by HbO 2 Independent measurement of deoxy-hb and oxy-hb Lambert-Beer-Law Molar extinction coeff. [cm -1 M -1 ] Hb 10 3 HbO Wavelength [nm] 45

46 Non-Invasive Measurements at Several Locations on the Scalp Placing light sources and detectors at several places at the head allows coverage of several brain areas Penetration depth depend on separation of source and detector Penetration depth is limited because number of photons coming back is logarithmically depended on source-detector separation (>4 cm not higher than systemic noise) 46

47 Whole-Head Coverage Enhancement in number of sensors 32 detectors and 20 sources allow 80 measurement channel covering the whole head Long-range networks feasible for research 47

48 Probes Sensors can be attached to EEGcaps Reference to standard system/space Top: NIRx, bottom: Hitachi system 48

49 Combination with Electrophysiological Recordings 49

50 Measurement Systems NIRX: Dynot ISS: Imagent Hitachi: ETG Techen: CW4/5 50

51 Wearable System Recent developments in miniaturization Measurements in natural environments become feasible First experiments by riding a bicycle look promising wireless data acquisition is also possible 51

52 Mobility Piper, Mehnert, et al., 2013, Neuroimage 52

53 Mobility Brain data can be acquired even during riding a bycicle NIRS reveals meaningful results on ongoing brain activity despite movement artifacts Piper, Mehnert, et al., 2013, Neuroimage 53

54 Temporal Resolution NIRS measures relative slow oscillations (fastest frequency of HRF is ~0.2 Hz) Heartbeat (1-1.5 Hz) has to be adequately sampled for filtering Nyquest-theorem suggest double of the fastest frequency of interest, i.e. min. 3 Hz -> NIRS instruments measure with a sampling frequency of 3-10 Hz

Spatial Resolution In depth: limited to 2-3 cm into the head, deeper is not possible because too few photons will come back -> cortex is only touched Spatial resolution is also 2-3 cm depending on

55 Spatial Resolution In depth: limited to 2-3 cm into the head, deeper is not possible because too few photons will come back -> cortex is only touched Spatial resolution is also 2-3 cm depending on the distance of source and detector Spatial resolution can be enhanced by overlapping banana-shaped volumes and source localization approximation (similar to EEG, will be subject in later lectures) to sub-centimeter scale

56 Spatial Resolution Habermehl, Mehnert, et al., 2012, Neuroimage Koch, Mehnert, et al., 2010, Frontiers Neuroenergetics NIRS can have a spatial resolution in the sub-centimeter range Thumb and pinky are distinguishable in the somatosensory areas 56

Simultaneous Measurements with fmri as Golden Standard NIRS fmri Right hand gripping task -> activity in left motorrelated cortex General Linear

57 Simultaneous Measurements with fmri as Golden Standard NIRS fmri Right hand gripping task -> activity in left motorrelated cortex General Linear Model analysis (described in the next section) separately calculated for each instrument, reveals similar activity patterns for both instruments 57

58 Correlation of Time Courses seconds NIRS fmri NIRS and FMRI can correlate up to 0.8 during tasks 58

59 PROS + CONS 59

60 Pro s and Con s Pros Portability Determination of oxy- and deoxyhemoglobin Low price ( t Euro) Study the basis of functional imaging Newborns and children Freely behaving adults Bedside imaging Good spatial resolution Cons Penetration depth around 1.5cm -> Only top region of cortex is accessible Poor spatial resolution (1-3cm) Sensitive to extracerebral absorption changes (blood pressure) No/poor quantification Temporal resolution about 3-10 Hz 60

61 Conclusion Hemodynamic is caused by neuronal activity (increase blood flow) HRF is slow and temporal shifted fmri + NIRS depend on local concentration changes of hemoglobin fmri provides 3D and high resolution images of cortical activity NIRS can penetrate only the surface of the brain NIRS can be used for people and studies ill-suited for MRI (strong magnetic field, small space), like children or patients treated with deep brain stimulation, NIRS is suitable for bed-side monitoring and freely behaving subjects 61

62 Influence of Stimulus length Length of stimulus influences length of hemodynamic response In the domain of seconds this relationship can be assumed linear Longer stimuli increase amplitude Block design is preferable for higher signal-to-noise ratios 62

63 Overlapping Responses hemodynamic response stimulus Problem: hemodynamic response function (HRF) need ~30 s to come back to baseline Event-related stimulus designs lead to overlapping HRFs HRFs sum up linearly in the domain of seconds Requires deconvolution (e.g. Regression, General Linear Model approach, ) time [s] 63

64 Outline BCI Aim Technology: EEG Method: feature extraction, machine learning Results NIRS Hemodynamics Technology NIRS-EEG-BCI Experiment Results Discussion 64

65 Results 65

66 Aim Simultaneous EEG + NIRs might enhance the performance of a BCI Combining good spatial (NIRS) and temporal (EEG) resolution Still mobile 66

67 Machine Learning for Brain-Computer Interfaces NIRS/ EEG Aim: neuro-feedback Problem: data analysis in real-time Solution: Training of a machine learning algorithm (classifier) in a calibration phase, application in the feedback phase 67

68 Brain Computer Interfaces Imagery movements lateralized neuronal activation Real-time classification controlling the cross 68

69 Brain Computer Interfaces Calibration: feature extraction calculation of a classifier which maximizes the difference between classes/conditions using all features (data) Feedback: application of classifier performance Physiological interpretation of machine learning methods? 69

70 Linear Discriminant Analysis d 1 d 2 Question: does the new measurement x belong to class 1 or class 2? The distance between the class means class 1 LDA constructs a hyper plane (separation line) using the variance of each class and the overall variance x belongs to class 2 Non-linear machine learning methods are also available! 70

71 Setup 14 subjects, executed and imagery movements during simultaneous NIRS-EEG-BCI. Imagery movements were online classified with the EEG signals.

72 NIRS classification accuracy over time

73 NIRS Scalp Evolution Lecture by at MPI Leipzig, 16th July 2014

74 Data Analysis

75 Improvements by NIRS

76 SUMMARY 76

77 Summary EEG based BCIs work well for most of the subjects but 20-30% are illiterates Feature extraction: frequency, temporal window, CSP LDA NIRS can provide additional information and thereby enhance the performance Hemodynamics Problems Slow response How to combine the instruments? 77

78 References P. L. Nunez, R. Srinivasan, A. F. Westdorp, R. S. Wijesinghe, D. M. Tucker, R. B. Silberstein, and P. J. Cadusch, EEG coherency I: statistics, reference electrode, volume conduction, Laplacians, cortical imaging, and interpretation at multiple scales, Electroencephalogr Clin Neurophysiol, 103(5): , G. Dornhege, J. del R. Millan, T. Hinterberger, D. McFarland, and K.-R. Müller, Eds., Toward Brain-Computer Interfacing. Cambridge,MA: MIT Press, B. Blankertz, M. Tangermann, C. Vidaurre, S. Fazli, C. Sannelli, S. Haufe, C. Maeder, L. E. Ramsey, I. Sturm, G. Curio, and K.-R. Müller, The Berlin Brain-Computer Interface: Non-Medical Uses of BCI Technology, Front Neuroscience, 4: 198, 2010, URL tract, open Access. B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K.-R. Müller, Optimizing Spatial Filters for Robust EEG Single-Trial Analysis, IEEE Signal Process Mag, 25(1): 4156, 2008, URL 78

79 Further Reading: Hemodynamics Hellmuth Obrig and Arno Villringer (2003):Beyond the Visible Imaging the Human Brain With Light. Journal of Cerebral Blood Flow & Metabolism 23:1 18 Jöbsis, F. F. (1977). Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science, 198: Delpy, D. T., Cope, M., van der Zee, P., Arridge, S., Wray, S., and Wyatt,J. (1988). Estimation of optical pathlength through tissue from direct time of flight measurement. Phys Med Biol, 33(12): Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K. R., and Blankertz, B. (2012). Enhanced performance by a hybrid NIRS-EEG brain computer interface. Neuroimage, 59(1): White, B.R. et al. (2009) Resting-state functional connectivity in the human brain revealed with diffuse optical tomography. NeuroImage Boynton, G. M., Engel, S. A., Glover, G. H., and Heeger, D. J. (1996). Linear systems analysis of functional magnetic resonance imaging in human v1. Journal of Neuroscience, 16(13):

80 Thank you!

81 Chronologische Kreuzvalidierung (n=4) Daten (Features) training Modell z.b.: LDA test

82 Chronologische Kreuzvalidierung (n=4) Daten Modell z.b.: LDA

83 Chronologische Kreuzvalidierung (n=4) Daten Modell z.b.: LDA

84 Chronologische Kreuzvalidierung (n=4) Daten Modell z.b.: LDA

85 Results Asterisks mark significant results comparing combined NIRS-EEG vs EEG only Lecture by at MPI Leipzig, 16th July 2014

Clinically Available Optical Topography System

Clinically Available Optical Topography System Clinically Available Optical Topography System 18 Fumio Kawaguchi Noriyoshi Ichikawa Noriyuki Fujiwara Yûichi Yamashita Shingo Kawasaki OVERVIEW: Progress