Hybrid Brain-Computer Interfaces

Hybrid Brain-Computer Interfaces Lecture by at MPI Leipzig, 16th July 2014

Outline BCI Aim Technology: EEG Method: feature extraction, machine learning Results NIRS Hemodynamics Technology NIRS-EEG-BCI Experiment Results Discussion 2

Aim

Brain-Computer Interface (BCI) BCI: Translation of human intentions into a technical control signal without using activity of muscles or peripheral nerves

Overview of BCI Technology

TECHNOLOGY: EEG 8

Recap: EEG Measures the electrical activity of the brain by capturing electrical potential differences on the scalp surface Non-invasive method High temporal resolution, but low spatial resolution Signal sensitive to: facial muscle contractions electrostatic artifacts head conductivity variation,...

Recap: Analysis of EEG Data Different neural responses Event-related potentials (ERPs) Change in activity of neuronal populations small voltage shifts Event-related (de)synchronization (ERD / ERS) Change in synchrony of neuronal populations change of power in frequency bands - Exogeneous EEG components: induced by a stimulus/task - Endogenous EEG components: ongoing brain activity

Areas of the Brain Brain lobes: Frontal, Parietal, Temporal, Occipital

Topographic Mapping

Spectrum of Brain Activity Idealized spectrum of macroscopic brain activity

Modulation of Brain Rhythms Most rhythms are idle rhythms, i.e., they are attenuated during activation. - Alpha-rhythm (around 10~Hz) in visual cortex: - Mu-rhythm (around 10~Hz) in motor & sensory cortex:

Sensori-Motor-Rhythm

ERD/S Curves of Motor Imagery ERD/S for selected frequency band (ERD: decrease of power, ERS: increase of power) (8-13 Hz) (absolute value) ERS

METHOD: FEATURE EXTRACTION, MACHINE LEARNING 18

SMR-based BCI Systems

ML Approach machine learning techniques and adaptive signal processing needed due to considerable variability

Linear Discriminant Analysis d 1 d 2 Question: does the new measurement x belong to class 1 or class 2? The distance between the class means class 1 LDA constructs a hyper plane (separation line) using the variance of each class and the overall variance x belongs to class 2 Non-linear machine learning methods are also available! 21

Experimental Design: Calibration 1 st step: recording data for calibration: Motor imagery task: left hand, right hand, right foot Instruction for participants by visual cues (arrows: left, right, down) Study here: large scale study with 80 participants. 75 trials per motor imagery condition and participant

Modulation of SMRs Motor imagery, data from one ideal participant with prototypical patterns:

Experimental Design: Feedback 2 nd step: Feedback: Instruction to participant (motor imagery, same as before), Cross starts moving according to classifier output Cross stops after 4s: final position is evaluated

Training CSP-based Classification (1) Determine most discriminative frequency band, (2) band-pass filter EEG in that band, (3) extract single trials using the time interval in which ERD/ERS is expected, (4) calculate and select CSP filters, (5) apply them to EEG single trials, (6) calculate the log variance within trials. Result: low dimensional feature vector for each trial (dimensionality = number of selected CSP filters) (7) Train a linear classifier on the features, such as LDA Result: classifier

CSP Processing Pipeline

Extraction of Band-Power Features (1) raw signal spectral analysis select discriminating frequency band (in grey) with heuristic: ERD/S: power modulation in selected frequency band

Extraction of Band-Power Features (2)

Analysis of Motor Imagery Conditions: Spectra Step (1): determine a frequency band with good discrimination between selected conditions (marked in grey here) select frequency band, e.g. using Butterworth/Chebychev filters (see EEG lecture)

ERD/S Curves of Motor Imagery

ERD Curves of Motor Imagery: Topography

CSP Analysis Goal: find spatial filters that optimally capture modulations of brain rhythms Observation: power of brain rhythms ~ variance of bandpass-filtered signal

CSP Analysis

RESULTS 34

Results 35

Conclusion a BCI system can be controlled based on the sensorimotor rhythm (SMR) BBCI approach: let the machines learn Experimental Design 1 st step: calibration recording data and let the classifier learn the specific neural response of a participant: > Feature extraction: determine frequency band, calculate ERD/S, determine CSP filters that optimally capture the spatial distribution -> LDA 2 nd step: feedback using this classifier, let the participant control a BCI in real-time BUT: 20-30% of the subjects are not able to perform a BCI

Outline BCI Aim Technology: EEG Method: feature extraction, machine learning Results NIRS Hemodynamics Technology NIRS-EEG-BCI Experiment Results Discussion 37

Concentration Changes of Oxy- and Deoxygenated Hemoglobin Blood exchanges molecules with the glia cells in the capillary bed Glia cells supply the neurons with energy Oxygen which is needed for the energy metabolism Oxygen is supplied by hemoglobin Neuronal activity leads to a delayed blood flow increase Oxygenated hemoglobin, deoxygenated hemoglobin due to washout 38

From Stimulus to Hemodynamic Response magnetic properties fmri-response 39

Time courses 3 oxy-hb deoxy-hb 2 Average 2 c [ M] 1 0-1 c [ M] 1 0-2 -3 0 36 72 108 144 180 216 252 288 324 360 396 time [s] -1 0 10 20 30 time [s] c [ M] 0.2 0.1 0.0-0.1-0.2 After every block of stimulation (shaded blue boxes) concentration of oxygenated hemoglobin Cyt-Ox (oxy-hb) increases while deoxygenated hemoglobin (deoxy-hb) decreases 0 An 36 average 72 108 144over 180 216 stimuli 252 288provides 324 360 396 the hemodynamic response time [s] Height of amplitude as sign of activity in the measured cortical area 40

Applications Neurovascular coupling Newborns and children (e.g. development of language) Brain-Computer Interfaces Bed-side monitoring of patients Freely behaving subjects -> natural environments 41

TECHNOLOGY 42

Photon Transport in the Human Brain Tissue Near-Infrared light can penetrate the brain banana-shaped measurement volume for non-invasive NIRS 43

Why Near-Infrared Light? In the NIR range of light absorption of water and hemoglobin is low Wavelengths have to be chosen between 700 and 950 nm Penetration depth of light is enhanced Cortex can be reached This spectral range is called optical window Outside this optical window light is absorbed by the first millimeters of the skin 44

Changes in local absorption are measured with two wavelengths. Simultaneous measurements with light at two distinct wavelengths (760 nm, 830 nm): 760 nm is more strongly absorbed by Hb 830 nm is more strongly absorbed by HbO 2 Independent measurement of deoxy-hb and oxy-hb Lambert-Beer-Law Molar extinction coeff. [cm -1 M -1 ] 10 6 10 5 10 4 Hb 10 3 HbO 2 10 2 400 500 600 700 800 900 1000 Wavelength [nm] 45

Non-Invasive Measurements at Several Locations on the Scalp Placing light sources and detectors at several places at the head allows coverage of several brain areas Penetration depth depend on separation of source and detector Penetration depth is limited because number of photons coming back is logarithmically depended on source-detector separation (>4 cm not higher than systemic noise) 46

Whole-Head Coverage Enhancement in number of sensors 32 detectors and 20 sources allow 80 measurement channel covering the whole head Long-range networks feasible for research 47

Probes Sensors can be attached to EEGcaps Reference to standard 10-20 system/space Top: NIRx, bottom: Hitachi system 48

Combination with Electrophysiological Recordings 49

Measurement Systems NIRX: Dynot ISS: Imagent Hitachi: ETG Techen: CW4/5 50

Wearable System Recent developments in miniaturization Measurements in natural environments become feasible First experiments by riding a bicycle look promising wireless data acquisition is also possible 51

Mobility Piper, Mehnert, et al., 2013, Neuroimage 52

Mobility Brain data can be acquired even during riding a bycicle NIRS reveals meaningful results on ongoing brain activity despite movement artifacts Piper, Mehnert, et al., 2013, Neuroimage 53

Temporal Resolution NIRS measures relative slow oscillations (fastest frequency of HRF is ~0.2 Hz) Heartbeat (1-1.5 Hz) has to be adequately sampled for filtering Nyquest-theorem suggest double of the fastest frequency of interest, i.e. min. 3 Hz -> NIRS instruments measure with a sampling frequency of 3-10 Hz

Spatial Resolution In depth: limited to 2-3 cm into the head, deeper is not possible because too few photons will come back -> cortex is only touched Spatial resolution is also 2-3 cm depending on the distance of source and detector Spatial resolution can be enhanced by overlapping banana-shaped volumes and source localization approximation (similar to EEG, will be subject in later lectures) to sub-centimeter scale

Spatial Resolution Habermehl, Mehnert, et al., 2012, Neuroimage Koch, Mehnert, et al., 2010, Frontiers Neuroenergetics NIRS can have a spatial resolution in the sub-centimeter range Thumb and pinky are distinguishable in the somatosensory areas 56

Simultaneous Measurements with fmri as Golden Standard NIRS fmri Right hand gripping task -> activity in left motorrelated cortex General Linear Model analysis (described in the next section) separately calculated for each instrument, reveals similar activity patterns for both instruments 57

Correlation of Time Courses seconds NIRS fmri NIRS and FMRI can correlate up to 0.8 during tasks 58

PROS + CONS 59

Pro s and Con s Pros Portability Determination of oxy- and deoxyhemoglobin Low price (100-250t Euro) Study the basis of functional imaging Newborns and children Freely behaving adults Bedside imaging Good spatial resolution Cons Penetration depth around 1.5cm -> Only top region of cortex is accessible Poor spatial resolution (1-3cm) Sensitive to extracerebral absorption changes (blood pressure) No/poor quantification Temporal resolution about 3-10 Hz 60

Conclusion Hemodynamic is caused by neuronal activity (increase blood flow) HRF is slow and temporal shifted fmri + NIRS depend on local concentration changes of hemoglobin fmri provides 3D and high resolution images of cortical activity NIRS can penetrate only the surface of the brain NIRS can be used for people and studies ill-suited for MRI (strong magnetic field, small space), like children or patients treated with deep brain stimulation, NIRS is suitable for bed-side monitoring and freely behaving subjects 61

Influence of Stimulus length Length of stimulus influences length of hemodynamic response In the domain of seconds this relationship can be assumed linear Longer stimuli increase amplitude Block design is preferable for higher signal-to-noise ratios 62

Overlapping Responses hemodynamic response stimulus Problem: hemodynamic response function (HRF) need ~30 s to come back to baseline Event-related stimulus designs lead to overlapping HRFs HRFs sum up linearly in the domain of seconds Requires deconvolution (e.g. Regression, General Linear Model approach, ) 0 10 20 30 40 50 60 time [s] 63

Outline BCI Aim Technology: EEG Method: feature extraction, machine learning Results NIRS Hemodynamics Technology NIRS-EEG-BCI Experiment Results Discussion 64

Results 65

Aim Simultaneous EEG + NIRs might enhance the performance of a BCI Combining good spatial (NIRS) and temporal (EEG) resolution Still mobile 66

Machine Learning for Brain-Computer Interfaces NIRS/ EEG Aim: neuro-feedback Problem: data analysis in real-time Solution: Training of a machine learning algorithm (classifier) in a calibration phase, application in the feedback phase 67

Brain Computer Interfaces Imagery movements lateralized neuronal activation Real-time classification controlling the cross 68

Brain Computer Interfaces Calibration: feature extraction calculation of a classifier which maximizes the difference between classes/conditions using all features (data) Feedback: application of classifier performance Physiological interpretation of machine learning methods? 69

Setup 14 subjects, executed and imagery movements during simultaneous NIRS-EEG-BCI. Imagery movements were online classified with the EEG signals.

NIRS classification accuracy over time

NIRS Scalp Evolution Lecture by at MPI Leipzig, 16th July 2014

Data Analysis

Improvements by NIRS

SUMMARY 76

Summary EEG based BCIs work well for most of the subjects but 20-30% are illiterates Feature extraction: frequency, temporal window, CSP LDA NIRS can provide additional information and thereby enhance the performance Hemodynamics Problems Slow response How to combine the instruments? 77

References P. L. Nunez, R. Srinivasan, A. F. Westdorp, R. S. Wijesinghe, D. M. Tucker, R. B. Silberstein, and P. J. Cadusch, EEG coherency I: statistics, reference electrode, volume conduction, Laplacians, cortical imaging, and interpretation at multiple scales, Electroencephalogr Clin Neurophysiol, 103(5): 499515, 1997. G. Dornhege, J. del R. Millan, T. Hinterberger, D. McFarland, and K.-R. Müller, Eds., Toward Brain-Computer Interfacing. Cambridge,MA: MIT Press, 2007. B. Blankertz, M. Tangermann, C. Vidaurre, S. Fazli, C. Sannelli, S. Haufe, C. Maeder, L. E. Ramsey, I. Sturm, G. Curio, and K.-R. Müller, The Berlin Brain-Computer Interface: Non-Medical Uses of BCI Technology, Front Neuroscience, 4: 198, 2010, URL http://www.frontiersin.org/neuroprosthetics/10.3389/fnins.2010.00198/abs tract, open Access. B. Blankertz, R. Tomioka, S. Lemm, M. Kawanabe, and K.-R. Müller, Optimizing Spatial Filters for Robust EEG Single-Trial Analysis, IEEE Signal Process Mag, 25(1): 4156, 2008, URL http://dx.doi.org/10.1109/msp.2008.4408441 78

Further Reading: Hemodynamics Hellmuth Obrig and Arno Villringer (2003):Beyond the Visible Imaging the Human Brain With Light. Journal of Cerebral Blood Flow & Metabolism 23:1 18 Jöbsis, F. F. (1977). Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science, 198:1264 1267 Delpy, D. T., Cope, M., van der Zee, P., Arridge, S., Wray, S., and Wyatt,J. (1988). Estimation of optical pathlength through tissue from direct time of flight measurement. Phys Med Biol, 33(12):1433 1442. Fazli, S., Mehnert, J., Steinbrink, J., Curio, G., Villringer, A., Müller, K. R., and Blankertz, B. (2012). Enhanced performance by a hybrid NIRS-EEG brain computer interface. Neuroimage, 59(1):519 529. White, B.R. et al. (2009) Resting-state functional connectivity in the human brain revealed with diffuse optical tomography. NeuroImage 47148 156. Boynton, G. M., Engel, S. A., Glover, G. H., and Heeger, D. J. (1996). Linear systems analysis of functional magnetic resonance imaging in human v1. Journal of Neuroscience, 16(13):4207 4221. 79

Thank you!

Chronologische Kreuzvalidierung (n=4) Daten (Features) training Modell z.b.: LDA test

Chronologische Kreuzvalidierung (n=4) Daten Modell z.b.: LDA

Results Asterisks mark significant results comparing combined NIRS-EEG vs EEG only Lecture by at MPI Leipzig, 16th July 2014