EEG-Based Brain Computer Interface System for Cursor Control Velocity Regression with Recurrent Neural Network

Transcription

1 EEG-Based Brain Computer Interface System for Cursor Control Velocity Regression with Recurrent Neural Network Haoqi WANG the Hong Kong University of Science and Technology Abstract This paper presents the decoding of EEG signal using recurrent neural network to predict the movement velocity of computer cursor. I. INTRODUCTION A. BACKGROUND BCI Brain Computer Interface (BCI) combines the techniques of brain activity recording, feature extraction and processing, and computer commands interpretation [1]. BCI is now being explored in many applications, including but not limited to human augmentation, lie detection, security and authentication. [2] There are two general categories of BCI, depending on the brain signal acquisition methods: invasive and non-invasive methods [3]. Invasive recording approaches implant electrodes under the scalp. The neural activity of the brain is measured either on the cortical surface or from within the motor cortex intracortically. The most significant advantage is that they provide high temporal and spatial resolution, increasing the quality of the obtained signal and its signal to noise ratio [4]. However, these techniques suffer from plenty of issues. Aside from usability issues rising from the involvement of surgical procedure, problems related to the systems output have occurred. The small size of the monitored brain regions by those implants is considered one of them. Once implanted, they cannot be shifted to measure brain activity in another area. Besides, the body adaptation to the new object, which may fail, can cause medical complications. Problems regarding the stability of implants and protection from infection can arise as well. Thus the usage of invasive recording in real world has been usually restricted to the BCI based medical applications for a few disabled users [5]. Noninvasive methods do not require external object implantation into the brain of the subject. Thus it avoids the surgical procedures or permanent device attachment needed by invasive acquisition. Various assessment methods for different types of measured signals exist such as functional magnetic resonance imaging (fmri), functional nearinfrared spectroscopy (fnirs), magnetoencephalography (MEG), and electroencephalogram (EEG) [4]. EEG As a non-invasive BCI measurement technique, EEG provides the brain electrical activity recording from the surface of the scalp, where electrodes are placed on the scalp to pickup the electrical current generated by the brain [6]. EEG data contains rhythmic activity, which reflects neural oscillations, which are described by

2 frequency, power and phase. Such oscillations occur at certain frequencies. These include delta, alpha, gamma, theta and meta. Research has found correlations between these rhythms and different brain states [7]. For instance, alpha frequency of the brain activity is often captured by commercial EEG headsets for measuring the degree of meditation. B. EXPERIMENT SETTINGS AND DATA In this experiment, the subjects were asked to track the moving 2-D cursor on the screen by thinking that they were using their dominant hand to control the cursor moving the same way as it was on the screen. In the mean time, the EEG signal is recorded wirelessly with Emotiv EPOC headset of 14 channels. The headset with hydrated electrodes was put on the scalp of the subject. TestBench, a Emotiv software was used to ensure the signal quality during the recording process. Meanwhile, the signal was processed by a high pass filter at 0.16Hz and a low pass filter at 30 Hz. 14 channel EEG data and cursor movement were recorded simultaneously at a sample rate of 128Hz. The data are stored by BCI2000 software. There are ten trails in total for each subject. The cursor moves along the horizontal axis for five trails and along the vertical axis for another five trails. Each trail lasts for 60 seconds. The cursor moves forward and backward repeatedly, but it is not strictly periodic. C. RELATED STUDY Research has been conducted to study the control of a computer cursor using signals from imagination of the movement of body parts [8] in 1-D [9], 2-D [10] [11] and 3-D [12]. Different command signals in the cursor control task are mapped by corresponding sensorimotor rhythm [13] changes from different mental states. Besides the sensorimotor approach to control a cursor, some researches studied the hybrid EEG paradigm for the cursor control. A target practice brain computer interface system based on mental states was used to control 1-D cursor in 2006 [14]. Later, Steady State Visual Evoked Potential method was implemented on 2-D cursor control problem [15]. In 2010, the P300 potential and mental states were used in the 2-D cursor control problem [16]. One of the drawbacks of the noninvasive brain computer interface systems in 2-D or 3-D cursor control mentioned above is that it requires weeks to months for training to gain a satisfied performance. Subjects need such lengthy time to learn to modulate certain frequency bands during a neural activity in the process of moving the cursor to a specific target. Moreover, it should be noted that the fatigue phenomenon reported by some researchers and subjects influences the cursor control experiments. Also, the discrete control of cursor directions caused by switching imaginary body parts is also an issue of these paradigms. A new noninvasive EEG-based method of 2-D cursor control was introduced using a way similar to invasive devices that can minimize the training time. Positive performance can be attained just after approximately forty minutes of training. The subjects with electrodes implanted in their brains can reach high success rates in target acquisition with continuous imagined kinematics of one body part. [17] These studies show that natural imagined body kinematics paradigm can greatly reduce the training time and can give insight to generic model that requires no training. In the previous study, Multiple Linear Regression decoder model was used to predict the velocity of the computer cursor. In this project, Recurrent Neural Network models is implemented for the cursor velocity regression. II. METHOD In this project, there are three major steps for EEG signal decoding. Firstly, to 2

3 acquire the useful band frequency data, the raw EEG signals pass through a low pass Butterworth filter. Secondly, the filtered data are fed into two kinds of recurrent neural network. One consists of long short term memory (LSTM) cells. The other is made of gated recurrent unit (GRU) cells. Thirdly, the prediction results are compared to the real cursor movement data to evaluate the effectiveness of the model. There are two types of score for the effectiveness calculation, R squared value and correlation goodness of fit. Three things will be discussed in this paper, the performance between LSTM model and GRU models, the validity of the RNN models, and the effectiveness of the recurrent neural network model to all the subjects. A. FILTERING The Butterworth filter is a type of signal processing filter designed to have a frequency response as flat as possible in the passband.[18] A 4th order low pass Butterworth filter of 2Hz cut-off frequency was used. The main purpose of applying a low pass filter is to reduce the high frequency fluctuations for more smooth input signal of the neural network. According to some literature [19], the EEG signal under 2Hz has a close relation with the cursor control. Fig. 1. Fig. 2. Fig. 3. Butterworth filter result. Butterworth filter result. Butterworth filter result. B. RECURRENT NEURAL NETWORKS A recurrent neural network (RNN) is a class of artificial neural network where connections between nodes form a directed graph along a sequence. This allows it to exhibit dynamic temporal behavior for a time sequence. Unlike feedforward neural networks, RNNs can use their internal state (memory) to process sequences of inputs [20]. This makes them applicable to tasks in this experiment since our input data is a sequence of time. Two models were implemented separately for this process. The Long Short Term Memory (LSTM) model is composed of LSTM units which have cells, Fig. 4. Butterworth filter result. input gates, output gates and forget gates. The Gated Recurrent Unit (GRU) consists of update gates and reset gates. The data for each trail lasts for 60 seconds, with a sample frequency of 128Hz, which is 7680 samples in total. The corresponding cursor movement data is also 128Hz for 60 seconds. The result comparison and effectiveness of the two models are discussed below. 3

4 C. CROSS VALIDATION Leave One Out cross validation method is used to prevent the model over fitting problem. Since there are five trails for each regression model, one trail was left out for validation while the other four trails were for training. Therefore, five models are generated according to five different training and testing combinations. D. EVALUATION The models were evaluated using two types of score called Goodness-of-Fit (GoF). This scoring technique separated the trial into segments of 5 seconds. The first type is to average the Pearson correlation scores between the predicted and actual cursor velocities. Then, the averaged value of the Pearson correlation scores over each trial was defined as the GoF. The second type is to average the R squared value scores between the predicted and actual cursor velocities. These methods can provide a better representation of fit by not allowing one improperly fit window to reduce the overall models score. GoF 1 = 1 M GoF 2 = 1 M M i=1 M i=1 Corr(V i decoded,v i observed ) R 2 (V i decoded,v i observed ) where V decoded and V observed represents the decoded velocity and the observed velocity for the ith segment, respectively. The two kinds of scores are of similar trends in terms of the evaluation of the model goodness. Therefore, the correlation score will be further illustrated next as an example. III. RESULTS From the tables in the appendix, it is shown that the LSTM and the GRU model have very similar performance and accuracy. In a random seeded machine learning process, the highest correlation mark of the LSTM horizontal regression is The correlation mark of the GRU horizontal regression can reach up to The best vertical regression of LSTM score is And the vertical regression of GRU model goes up to The table also reveals that the horizontal cursor movement prediction is generally better than the vertical cursor movement prediction for the same subject. It is worth noting that the prediction accuracy varies from subject to subject. For example, subject 25 performs very well in both horizontal and vertical LSTM trails with an average score of for horizontal trails and an average score of for vertical trails. Subject 21 performs well in the horizontal tests but poorly in the vertical tests. Subject 14 can perform equally well results on both types. However, there are subjects who have no significant patterns such as subject 16. The pictures next page are some prediction result using LSTM neural network. The red line is the actual cursor movement after normalization. The blue line is the prediction result. IV. CONCLUSIONS In this project, recurrent neural network was implemented in the form of LSTM and GRU respectively, and the performance of the two models has greatly improved. Interesting conclusions of the subject based EEG decoding interpretation were drawn. It is shown that the RNN is an effective method for EEG decoding. Though many factors contribute to affecting the accuracy of the model, such as the choice of the recurrent neural network cells, the filter frequency, and the control parameters, the most determinant variable is the subject itself. 4

5 Fig. 5. Horizontal Trail. Fig. 10. Vertical Trail. Fig. 6. Horizontal Trail. Fig. 11. Vertical Trail. Fig. 7. Horizontal Trail. Fig. 12. Vertical Trail. Fig. 8. Horizontal Trail. Fig. 13. Vertical Trail. Fig. 9. Horizontal Trail. Fig. 14. Vertical Trail. 5

6 APPENDIX A. LSTM horizontal correlation score Run1 Run2 Run3 Run4 Run5 subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject

7 B. LSTM vertical correlation score Run1 Run2 Run3 Run4 Run5 subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject

8 C. GRU horizontal correlation score Run1 Run2 Run3 Run4 Run5 subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject

9 D. GRU vertical correlation score Run1 Run2 Run3 Run4 Run5 subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject subject

10 ACKNOWLEDGMENT This project is sponsored by the National Science Foundation through Research Experience for Undergraduates (REU) award, with additional support from the Joint Institute of Computational Sciences at University of Tennessee Knoxville. This project used allocations from the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by the National Science Foundation. In addition, the computing work was also performed on technical workstations donated by the BP High Performance Computing Team. Special thanks to the mentors: Dr. Xiaopeng Zhao, Dr. Kwai Wong and Soheil Borhani. REFERENCES [1] Van Gerven, M., Farquhar, J., Schaefer, R., Vlek, R., Geuze, J., Nijholt, A., Ramsey, N., Haselager, P., Vuurpijl, L., Gielen, S. And Desain, P The brain-computer interface cycle. Journal of Neural Engineering. 6, 1-10 [2] J. van Erp, F. Lotte, M. Tangermann Braincomputer interfaces: beyond medical applications Computer, 45 (4) (2012), pp [3] K.-R. MULLER, A. Kubler Toward brain computer interfacing Massachusetts Institute of Technology (2007), pp [4] Abdulkader, S., Atia, A. and Mostafa, M. (2018). Brain computer interfacing: Applications and challenges. [5] D.S. Tan, A. Nijholt, Brain-computer interfaces: applying our minds to human-computer interaction [6] Niedermeyer E.; da Silva F.L. (2004). Electroencephalography: Basic Principles, Clinical Applications, and Related Fields. Lippincott Williams & Wilkins. [7] Intro to Brain Computer Interface. NeurotechEDU, learn.neurotechedu.com/introtobci/ [8] Morash, V., et al., Classifying EEG signals preceding right hand, left hand, tongue, and right foot movements and motor imageries. Clinical neurophysiology, (11): p [9] Wolpaw, J.R., et al., An EEG-based brain-computer interface for cursor control. Electroencephalography and Clinical Neurophysiology, (3): p [10] Wolpaw, J.R. and D.J. McFarland, Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proceedings of the National Academy of Sciences of the United States of America, (51): p [11] Xia, B., et al., A combi- nation strategy based brain computer interface for two-dimensional movement control. Journal of neural engineering, (4): p [12] McFarland, D.J., W.A. Sarnacki, and J.R. Wolpaw, Electroencephalographic (EEG) control of threedimensional movement. J Neural Eng, (3): p [13] Ernst Niedermeyer, Fernando Lopes da Silva Electroencephalography. Basic principles, Clinical Applications and Related Fields. 3rd edition, Williams & Wilkins Baltimore 1993 [14] Trejo, L.J., R. Rosipal, and B. Matthews, Braincomputer interfaces for 1-D and 2-D cursor control: designs using volitional control of the EEG spectrum or steady-state visual evoked potentials. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, (2): p [15] A hybrid ERD/SSVEP BCI for continuous simultaneous two dimensional cursor control. Journal of neuroscience methods, (2): p [16] Li, Y., et al., An EEG-based BCI system for 2- D cursor control by combining Mu/Beta rhythm and P300 potential. Biomedical Engineering, IEEE Transactions on, (10): p [17] Kim, S.-P., et al., Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. Journal of neural engineering, (4): p. 455 [18] Wikipedia contributors. (2018, June 12). Butterworth filter. In Wikipedia, The Free Encyclopedia. Retrieved 17:24, July 27, 2018, from w/index.php?title=butterworth_ filter&oldid= [19] Jinhua Zhang, Jiongjian Wei, Baozeng Wang, Jun Hong, and Jing Wang, Nonlinear EEG Decoding Based on a Particle Filter Model, BioMed Research International, vol. 2014, Article ID , 13 pages, [20] Wikipedia contributors. (2018, July 25). Recurrent neural network. In Wikipedia, The Free Encyclopedia. Retrieved 14:34, July 30, 2018, from org/w/index.php?title=recurrent_ neural_network&oldid=