Ttle A hybrd bran-computer nterface combnng the EEG and NIRS Author(s) Ma, L; Zhang, L; Wang, L; Xu, M; Q, H; Wan, B; Mng, D; Hu, Y Ctaton The 2012 IEEE Internatonal Conference on Vrtual Envronments, Human-Computer Interfaces, and Measurement Systems (VECIMS 2012), Tanjn, Chna, 2-4 July 2012. In IEEE VECIMS Proceedngs, 2012, p. 159-162 Issued Date 2012 URL http://hdl.handle.net/10722/181798 Rghts IEEE Internatonal Symposum on Vrtual Envronments, Human- Computer Interfaces and Measurement Systems Proceedngs. Copyrght IEEE.
A Hybrd Bran-Computer Interface Combnng the EEG and NIRS Lan Ma, Lxn Zhang, Lu Wang, Mnpeng Xu, Hongzh Q, Bakun Wan, Dong Mng* Department of Bomedcal Engneerng Tanjn Unversty Tanjn 300072, Chna * (rchardmng@tju.edu.cn) Yong Hu Department of Orthopaedcs and Traumatology Ka Shng Faculty of Medcne The Unversty of Hong Kong Hong Kong, Chna Abstract Compared to the conventonal bran-computer nterface (BCI) system, the hybrd BCI provdes a more effcent way for the communcaton between the bran and the external devce. The Electroencephalography (EEG) sgnal and the change of oxygenaton n the bran are two prevalng approaches used n the BCI. However, sngle physologcal sgnal couldn t provde enough nformaton for a satsfed BCI. Ths paper proposes a hybrd BCI system based on the combnaton of the EEG sgnal and the cerebral blood oxygen changes measured by the nearnfrared spectroscopy system (NIRS) to detect the state of motor magery (MI). The result shows that the average recognton rate can acheve above 75.04% and the hghest rate 91.11%, whch are hgher than when only usng EEG or NIRS. It suggests that the proposed hybrd BCI system has a good performance n the combnaton of these two dfferent sgnals. Further nvestgaton may help develop better BCIs wth hgh accuracy and sgnfcant effcency. Keywords- Hybrd BCI; EEG; NIRS;Motor magery I. INTRODUCTION A bran computer nterface (BCI) s a drect communcaton pathway between the bran and an external devce [1]. BCIs are often drected at assstng, augmentng, or reparng human cogntve or sensory-motor functons and have focused prmarly on neuroprosthetcs applcatons that am at restorng damaged hearng, sght and movement. Electroencephalography (EEG) measures scalp voltage fluctuatons resultng from onc current flows wthn the neurons of the bran [2,3]. EEG based BCI has been most studed, due to ts fne temporal resoluton, easy recordng and low set-up cost. Many patents wth amyotrophc lateral scleross beneft from the successful use of such BCI systems [4,5,6]. Ivan Volosyak presents a EEG-based Bremen BCI system, whch allowed one of the subjects n an onlne experment to reach a peak nformaton transfer rate (ITR) of 124 bt/mn [7]. EEG-bases BCIs have many dfferent modaltes, such as P300, steady-state vsual evoked potental (SSVEP), and motor magery (MI) [8]. Ths study focuses on motor magery by whch an ndvdual rehearses or smulates a gven acton [9].It has been proved that motor magery retans many of the propertes, n terms of temporal regulartes, programmng rules and bomechancal constrants, whch are observed n the correspondng real acton when t comes to executon [10]. It s now wdely used as a technque to enhance motor learnng and to mprove neurologcal rehabltaton n patents after stroke [11,12]. When people thnk, the oxygen content of blood wll change n the bran whch can be measured by the near-nfrared spectroscopy (NIRS). BCI systems based on the NIRS have been developed rapdly n recent. R. Staram et al develop a motor magery based BCI by usng NIRS measured oxyhemoglobn sgnals. NIRS systems have also been successfully appled to the research of workng memory and emoton changes [13,14]. In recent years, BCI paradgms combnng two dfferent mental control sgnals, called hybrd BCIs, have been studed ncreasngly as a potent approach to mprove BCI system [15,16]. A hybrd BCI s composed of two BCIs, or at least one conventonal BCI and another system. Hybrd BCIs could nvolve a second type of nput operatng sequentally and/or smultaneously. The second nput mght be another BCI, whch mght requre the user to perform addtonal mental tasks. The second nput mght use on other physologcal sgnals [17]. A hybrd BCI controls faster and has a hgher accuracy rate. Ths paper proposes a hybrd BCI system based on the combnaton of the EEG sgnal and the cerebral blood oxygen changes measured by the near-nfrared spectroscopy system (NIRS) to detect the state of motor magery. II. MATERIALS AND METHODS A. Subjects Sx partcpants (four males and two females), aged from 22~26, rght-handed, wthout any antecedent of neurologc or orthopedc mparment affectng upper lmbs take part n the experment. All partcpants were nformed of the procedure and the natural consequence of the study and sgned the consent. All partcpants have passed the movement magery questonnare and they have a tranng tme before experment. B. Experment Ths study took motor magery as the mental task. Durng the experment the partcpant sat n front of a computer montor and vewed a drecton cartoon. One complete mental 978-1-4577-1759-8/12/$26.00 2012 IEEE
task followed the chronologcal order shown as Fg. 1. Frst, t started wth a 5-second restng stage (RS) n whch subjects were asked to keep calm and relax themselves. Then, after a cue lastng one second, subjects were asked to follow the tps on the screen to mage ther rght or left hand movement for 7 seconds wth no actual movement durng the magery stage (IS). After a daly tranng for a week, each subject performed 450 complete tasks whch were equally dvded nto 15 sets. Between every two sets, there were 5 mnutes for subjects to have a rest. Fgure 1. the chronologcal order of a completed mental task C. Date recordng and perporcessng receved the reflected lght emtted by the near-nfrared lght through the cerebral cortex and sent to the DAQ card. All these sgnals are stored n the computer after dgtal converson by the DAQ card at a sample frequency of 1000Hz. In the pre-processng, the EEG sgnal was bandpass fltered at 0.1 to 30 Hz whle the cerebral blood oxygen sgnal was bandpass fltered at 0.06 to 0.09 Hz. And then the EEG sgnal was down sampled at 250Hz whle the cerebral blood oxygen sgnal at 10 Hz. D. Support vector machne The prncpal dea of the support vector machne (SVM) [18] s to fnd the separatng hyperplane between two classes, so that the dstance between the hyperplane and the closest ponts from both classes s maxmal. In other words, t needs to maxmze the margn between the two classes. Snce the EEG sgnal s nonlnear and s not the case that two states are lnearly separable, we used a support vector machne wth the Gaussan radal-bass functon K( f, f j 2 ) exp( f f ) (1) j as a kernel. f and f j are the support vectors n the feature space. The outcome of the SVM for a new sample s the value for Fgure 2. The structure schematc drawng of the system There are two dfferent types of channels to detect the bran actvtes. The structure schematc drawng of the system s gven n Fg. 2. Ths system ncludes two parts: the EEG acquston sub-system whch nvolves electrodes, EEG sgnal amplfer (type: 9216SM, made by SYMTOP Company), and the data acquston card (DAQ card, type: USB-6251, made by Natonal Instrument Company) and the Blood oxygen acquston system whch nvolves near-nfrared lghts (760nm), constant current drver, photo detectors, and the DAQ card. Two EEG electrodes, two near-nfrared lghts and sx photo detectors were fxed on a shadng materal set on the forehead. The EEG electrodes were located FP1 and FP2 followng the 10-20 system. Each referenced to an electrode put on the rght ear. The near-nfrared lghts were located n lne wth the EEG electrodes and between them. Sx photo detectors (Z1, Z2, Z3, Z4, Z5 and Z6) were equally set nto two parallel lnes shown as Fg. 2. The channels from left to the rght at the top row are: Z1, Z2, Z3. And the channels from left to the rght at the underneath row are: Z4, Z5, Z6. Each photo detectors had a dstance of 2.89cm from the nearest near-nfrared lght. And the dstance between the near-nfrared lght was 3.75cm. When subjects executed the mental task, the EEG sgnal and the cerebral blood oxygen were changng accordngly. The electrodes set on the forehead transported the changed EEG sgnal to the amplfer whose outputs lnk to the DAQ card. Wth the same tme, the photo detector located on the forehead n y( f, w, b) w y K( f, f ) b (2) 1 Where f are the support vectors chosen from the tranng set wth known class labels and w are lagrange multplers. III. RESULTS TABLE 1. The average PSD of dfferent mental states Average Power Spectrum Densty (PSD)(dB) Channel FP1 ChannelFP2 Left-hand Rght-hand Left-hand Rght-hand Frequency Imagery Imagery Imagery Imagery (Hz) RS IS RS IS RS IS RS IS 12 3.25-3.26 3.32-3.39 2.21-2.26 2.20-1.88 13 3.91-5.72 3.81-4.94 2.71-3.10 2.68-2.88 14 2.79-3.06 2.69-2.44 3.11-2.89 2.59-1.90 Average 3.32-4.01 3.27-3.59 2.68-2.75 2.49-2.22 Event related desynchrony (ERD) s a common phenomenon when people executes motor magery. The phenomenon s often found on the sensormotor area by means of power spectrum densty analyss (PSD). In ths study, PSD were used to analyze the EEG sgnals. Table 1 shows the average PSD durng the restng stage (RS) and the magery stage (IS) at the frequency range from 12 to 14 Hz. For each condton, the largest absolute value s n dark background. Most of them are at 13 Hz. As can be seen from the table, t s easy to dstngush the motor magery from the restng state as the average PSD of the motor magery s
negatve whle the average PSD of the restng stage s postve whether at channel FP1 or channel FP2. However, the electrodes located n the opposte sdes to the hand doesn t have a steady larger absolute PSD than that located n the same sdes when subjects perform rght or left motor magery. For example, the absolute PSD of the rghthand motor magery at the channel FP1 (-5.72) s larger than that at the channel FP2 (-3.10), whch s n accordng wth the left-hand motor magery (the PSD s -4.94 at channel FP1 and - 2.88 at channel FP2). In addton, the absolute PSD at the channel FP1 s almost larger than that at the channel FP2 n the same condton. For example, channel FP1 has a larger value than channel FP2 at 13 Hz when subjects were n restng stage. In most cases, the absolute PSD of the left-hand magery s larger than that of the rght-hand magery at the correspondng electrodes. But a pared t-test shows there s no sgnfcant dfference between them. Thus the state of left-hand magery and the rght-hand magery were easy to mx-up n terms of the PSD features at channel FP1 and FP2. TABLE 2. Average ampltude of the cerebral blood oxygen sgnal at 0.06-0.09Hz Channel Left-hand Imagery Rght-hand Imagery RS IS RS IS Z1 3.11-3.42 4.78-4.67 Z2 3.59-1.98 3.54-3.53 Z3 4.66-3.55 5.45-4.14 Z4 4.80-5.66 7.35-7.32 Z5 1.52-2.17 3.73-7.06 Z6 1.08-5.85 3.11-4.00 Ths study used 760nm LED as a lght source to provde near-nfrared lght, whose prmary absorber at the bran tssue s deoxyhemoglobn. It supposes that the concentraton of deoxyhemoglobn only changed when subjects perform mental task. So the ntensty of reflected lght detected by the photo detector changed wth a perod lastng 18 seconds. In other words, there s a 0.077Hz-frequency wave whch carres vtal nformaton about cerebral blood oxygen changes nduced by the motor magery. Table 2 shows the average ampltude of the cerebral blood oxygen sgnal at 0.06-0.09Hz. Just as the EEG sgnal, the cerebral blood oxygen changes show smlar trends. For each mental state, the largest absolute value s n dark background. Most of them are at channel Z4. On the bass of the table, t s easy to dstngush the motor magery from the restng state as the average ampltudes of the motor magery are negatve whle the restng stage s postve at all channels. As can be seen from the table, the absolute value of the left motor magery at the channel Z3 (-3.55) s larger than that at the channel Z1 (-3.42) whle the rght-hand motor magery (the absolute value s -4.67 at channel Z1 and -4.14 at channel Z3). In a smlar way, the absolute value of the left motor magery at the channel Z6 (-5.85) s larger than that at the channel Z4 (- 5.66) whle the rght-hand motor magery (the absolute value s -7.32 at channel Z4 and -4.00 at channel Z6). Mostly, the absolute value of the rght-hand magery s larger than that of the left-hand magery at the correspondng photo detectors. But there also a pared t-test shows that no sgnfcant dfference appears between them. Therefore t s hard to tell the state of left-hand magery from the rght-hand magery by the cerebral blood oxygen changes at all channels. Fgure 4. The classfcaton accuracy of all subjects when usng dfferent features As the dfferent knds of mental tasks are hardly to be separated by the EEG sgnal or the NIRS sgnal n terms of the PSD analyss, we try to dstngush the magery state from rest state of the bran. In ths study, there are three dfferent knds of features: the EEG sgnal feature, the cerebral blood oxygen feature and the combnaton feature of them. SVM was used for mental state recognton. 5-fold cross valdaton was adopted to make the classfcaton accuracy robust. The results show that the averaged accuracy of combned features can acheve 82.79%, whch are hgher than the others (Fg. 4). For all subjects, the combnaton features acheve a hgher accuracy than other sngle features. The sngle EEG feature has a hgher accuracy than that of the sngle NIRS feature for the most of the subjects. The maxmum accuracy of 91.11% can be found n the combnaton features for subject 2. Whle the EEG feature only acheves 87.78% to the max and the NIRS feature acheves 79.45%. IV. DISCUSSION Dfferent from conventonal motor magery sgnal recordng methods and postons, ths study records the EEG and NIRS smultaneously on the forehead area. The PSD features of the EEG sgnal and ampltude of the NIRS sgnal are taken n consderaton for analyzng the bran actvtes. It s clear that the magery stage can be easly separated from the restng stage no matter whch knd bran sgnal s used, but t s dffcult to make a dstncton between left and rght hand movement. From the vew of the classfcaton results, the combnaton of both features s better than the sngle EEG or NIRS feature to separate the magery state from the rest state. It mples the good performance of the proposed hybrd BCI system and needs further researches to develop a greater effcency system. ACKNOWLEDGMENT Ths research was supported by Natonal Natural Scence Foundaton of Chna (No. 30970875, 90920015, 61172008, 81171423), Natonal Key Technology R&D Program of the
Mnstry of Scence and Technology of Chna (No. 2012BAI34B02) and Program for New Century Excellent Talents n Unversty of the Mnstry of Educaton of Chna. REFERENCES [1] Zander TO, Kothe C, Towards passve bran-computer nterfaces: applyng bran-computer nterface technology to human-machne systems n general, J Neural Eng. Vol. 8, pp 025005, Aprl 2011. [2] Slvon S, Ramos-Murgualday A, and Cavnato M, Bran-computer nterface n stroke: a revew of progress, Cln EEG Neurosc. Vol. 42, pp 245-52, October 2011. [3] Saa JF, Cetn M, A latent dscrmnatve model-based approach for classfcaton of magnary motor tasks from EEG data, J Neural Eng. Vol. 9, pp 026020, March 2012. [4] Njboer F, Sellers EW, and Mellnger J, A P300-based brancomputer nterface for people wth amyotrophc lateral scleross, Cln Neurophysol. Vol. 119, pp 1909 1916, August 2008. [5] Sellers EW, Vaughan TM, and Wolpaw JR, A bran-computer nterface for long-term ndependent home use, Amyotroph Lateral Scler. Vol. 11, pp 449-455, October 2011. [6] Vaughan TM, McFarland DJ, and Schalk G, The Wadsworth BCI Research and Development Program: at home wth BCI, IEEE Trans Neural Syst Rehabl Eng. Vol. 14, pp 229-233, October 2006. [7] Volosyak I, SSVEP-based Bremen-BCI nterface--boostng nformaton transfer rates, J Neural Eng. Vol. 8, pp 036020, June 2011. [8] Jeannerod M, Neural smulaton of acton: a unfyng mechansm for motor cognton, Neuromage. Vol. 14, pp 103-109, July 2001. [9] Mulder T, de Vres S, and Zjlstra S, Observaton, magnaton and executon of an effortful movement: more evdence for a central explanaton of motor magery, Exp Bran Res. Vol. 163, pp 344-351, June 2005. [10] Jackson PL, Lafleur MF, and Maloun F, Potental role of mental practce usng motor magery n neurologc rehabltaton, Arch Phys Med Rehabl. Vol. 82, pp 1133-1141, August 2001. [11] Sjoerd de Vres and Theo Mulder, Motor magery and stroke rehabltaton: a crtcal dscusson, J Rehabl Med. Vol. 39, pp 5-13,January 2007. [12] Pchorr F, De Vco Fallan F, and Cncott F, Sensormotor rhythmbased bran computer nterface tranng: the mpact on motor cortcal responsveness, J Neural Eng. Vol. 8, pp 025020, Aprl 2011. [13] Power SD, Kushk A,and Chau T, Automatc sngle-tral dscrmnaton of mental arthmetc, mental sngng and the no-control state from prefrontal actvty: toward a three-state NIRS-BCI, BMC Res Notes. Vol. 5, pp 141, March 2012. [14] Fazl S, Mehnert J, and Stenbrnk J, Enhanced performance by a hybrd NIRS-EEG bran computer nterface, Neuromage. Vol.59, pp 519-529, January 2012. [15] Müller-Putz GR, Bretweser C, and Cncott F, Tools for Bran- Computer Interacton: A General Concept for a Hybrd BCI, Front Neuronform. Vol. 5, pp 30, November 2011. [16] Pfurtscheller G, Allson BZ, and Brunner C, The hybrd BCI, Front Neurosc. Vol.4, pp 30, Aprl 2010. [17] Brunner C, Allson BZ, and Altstätter C, A comparson of three bran-computer nterfaces based on event-related desynchronzaton, steady state vsual evoked potentals, or a hybrd approach usng both sgnals, J Neural Eng. Vol. 8, pp 025010, March 2011. [18] V. Vapnk, The Nature of Statstcal Learnng Theory, Sprnger, New York, NY, USA, 1995.