Faculty of Electrical Engineering Universiti Teknologi Malaysia

Transcription

1 CONTROL OF ONE-DIMENSIONAL CURSOR MOVEMENT BY NONINVASIVE BRAIN-COMPUTER INTERFACE IN HUMANS SITI ZURAIMI SALLEH A thesis submitted in fulfilment of the requirements for the award of the degree of Master of Engineering (Electrical) Faculty of Electrical Engineering Universiti Teknologi Malaysia APRIL 2011

2 iii To my mother and father, husband, and sensei, for their precious moments

3 iv ACKNOWLEDGEMENT In the name of Allah, the Almighty and the Merciful. Alhamdulillah, praise to Allah S.W.T for the guidance, strength and bless which He gave upon me to complete this research work. First and foremost, I would like to express my sincere gratitude and appreciation to my beloved supervisor, Dr. Norlaili bt Mat Safri for her advice, knowledge, supports, kindness and patience throughout this research work. I also would like to express my thanks to Dr. Fauzan, Mr. Afzan, Mrs. Husnaini and Ms. Ashikin for their useful discussion, assistance and supports. I am also indebted to Ms. Wan and Ms. Fiza from Medical Electronic Lab, Universiti Teknologi Malaysia for spending their time of my overtime works. Finally, special appreciation is dedicated to my husband, family and friends for their constant support and prayers during my research work.

4 v ABSTRACT Noninvasive brain-computer interface (BCI), using electroencephalogram (EEG) that records from scalp, could provide a new non-muscular channel for sending messages and command to the external world. The objectives of this study are to determine parameters that will drive cursor in BCI using noninvasive EEG signal and to control one dimensional cursor movement using the extracted parameters. This experimental-based study involved six normal subjects aged from 20 to 26 years old. Subjects were asked to perform tasks in two condition i.e. control condition and task condition. In the control condition, subjects were required to relax (resting) and fix their eyes on the centre of the screen without image displayed on the screen. In the task condition subjects were tasked to imagine a movement to move the cursor on the screen towards the target. These control and task conditions were repeated four times and each condition lasted for 10 seconds. Using Fast Fourier Transform, data in frequency domain for control and task have been obtained and analyzed in two of time interval of 1024 ms and 512 ms. Frequency is divided into six groups, i.e. delta band (0-4 Hz), theta band (4-7 Hz), alpha band (8-13 Hz), beta band (13-30 Hz), gamma band (31-50 Hz) and high gamma band (>51 Hz). Each frequency band in all frequencies of the task condition has been compared to the control condition. The present study finds optimum difference in delta frequency band between resting and active imagination at the parietal region. Furthermore, the parietal region is associated with sensory interaction and could be one of the input regions to control cursor movement. However, it is found that the delta frequency band is only applicable to a specific one-dimensional cursor movement as any imagination may produce the same results. Nevertheless, this study provides a platform for a more advance two-dimensional cursor movement study.

5 vi ABSTRAK Pengantaramuka otak-komputer (BCI) tak invasif menggunakan elektroensefalogram (EEG) yang direkod dari kulit kepala boleh menyediakan laluan baru tanpa otot untuk menghantar isyarat dan arahan ke persekitaran. Objektif kajian ini adalah untuk mengenalpasti beberapa parameter yang akan menggerakkan kursor dalam BCI dengan menggunakan isyarat EEG tak invasif dan mengawal pergerakan kursor dalam satu dimensi menggunakan parameter yang telah diekstrak. Kajian yang berunsurkan ekperimen ini melibatkan enam orang subjek normal berusia di antara 20 hingga 26 tahun. Kesemua subjek perlu melaksanakan tugas dalam dua keadaan iaitu keadaan tugasan dan keadaan kawalan. Dalam keadaan kawalan, subjek hanya perlu berehat dengan memandang ke arah tengah skrin tanpa paparan imej. Manakala dalam keadaan tugasan, subjek diminta membayangkan satu pergerakan untuk menggerakkan kursor ke arah target pada skrin. Kedua-dua keadaan ini diulang sebanyak empat kali dan setiap satunya berlangsung selama 10 saat. Menggunakan jelmaan Fourier cepat, data dalam bentuk frekuensi diperolehi dan dianalisis dalam dua sela masa iaitu 1024 ms dan 512 ms. Julat frekuensi pula dibahagikan kepada enam kumpulan iaitu delta (0-4 Hz), teta (4-7 Hz), alfa (8-13 Hz), beta (13-30 Hz), gama (31-50 Hz) dan gama tinggi (> 51 Hz). Setiap frekuensi dalam keadaan tugasan dan kawalan dibandingkan. Kajian ini mendapati bahawa perbezaan frekuensi dalam keadaan rehat dan imaginasi adalah optimum dalam julat delta di bahagian parietal. Dalam kajian ini, bahagian parietal berkaitan dengan interaksi deria dan menjadi salah satu input untuk mengawal gerakan kursor. Walau bagaimanapun, didapati bahawa jalur frekuensi delta hanya boleh diaplikasi ke atas gerakan kursor satu dimensi sahaja di mana sebarang imaginasi boleh memberikan hasil keputusan yang sama. Namun demikian, kajian ini juga memberi ruang yang menjurus kepada kajian gerakan kursor dua dimensi.

6 vii TABLE OF CONTENTS CHAPTER TITLE PAGE DECLARATION DEDICATION ACKNOWLEDGEMENT ABSTRACT ABSTRAK TABLE OF CONTENTS LIST OF TABLES LIST OF FIGURES LIST OF ABBREVIATIONS LIST OF APPENDICES ii iii iv v vi vii xi xii xvi xvii 1 INTRODUCTION Introduction Problem Statement Objectives 3

7 viii 1.4 Scope of Project Project Contribution Thesis Outline 4 2 LITERATURE REVIEW Introduction Electroencephalography EEG Waveform The International of Electrode Placement System Brain-Computer Interface (BCI) Invasive and Noninvasive BCI Imagination vs. Actual Movement Electrophysiological Sources Signal Processing Published Work on BCI Published Work on Neurofeedback Published Work on Cursor Control Published Work on Other Control Discussion on Published Work in BCI 20 3 METHODOLOGY Introduction Experimental Set Up Equipment and Supplies 23

8 ix Preparation of Subject Installing of Electrodes onto Electrode Board Adapter Clean Up Data Acquisition and Recording Process Participant Conditions Data Acquisition Data Analysis Flow Chart of Data Analysis Software Development 31 4 FREQUENCY ANALYSIS AND FINDINGS Introduction EEG Signals Feature Extraction Analysis ms Time Interval ms Time Interval Mean for All Trials Feature Translation 45 5 DISCUSSION Introduction Parameters Determination 48

9 x 6 CONCLUSION Conclusion Recommendation for Future Work 52 REFERENCES 53 Appendices A-C 57-74

10 xi LIST OF TABLES TABLE NO. TITLE PAGE 2.1 Frequency band for EEG signals Electrophysiological sources used in BCI Electrodes position Channels combination Hit rate and mean time Comparison of hit rates and movement precision Size of target, cursor and workspace 47

11 xii LIST OF FIGURES FIGURE NO. TITLLE PAGE 1.1 Scope of project EEG waveforms (a) Delta wave (b) Theta wave (c) Alpha wave (d) Beta wave (e) Gamma wave The electrode placement system Basic design and operation of a BCI system Present-day human BCI system types (a) Sensorimotor rhythm (b) Slow cortical potentials 14

12 xiii 2.4(c) P300 evoked potential (d) Cortical neuronal activity Equipment used in the experiment (a) Electro-cap (b) Electro board adapters (c) Body harness (d) Disposable sponge disks (e) Blunted needle/syringe kit (f) Ear electrodes (g) Electrode board adapter connector Sequence of data acquisition Division of data for 1024 ms and 512 ms time interval Flow chart of parameter determination process Flow chart of finalized parameter determination process (a) Syntax of workspace (b) Workspace (a) Syntax of baseline (b) Baseline (a) Syntax of cursor (b) Cursor 33

13 xiv 3.9(a) Syntax of target (b) Target Example of 1 second of raw EEG signal from subject 1 at P Z channel Example of power spectrum of EEG signal from subject 1 at P Z channel Example of 1 second data of EEG signal from subject (a) The occurrences of maximum DP for trial 1 for 1024 ms time interval (b) The occurrences of maximum DP for trial 2 for 1024 ms time interval (c) The occurrences of maximum DP for trial 3 for 1024 ms time interval (d) The occurrences of maximum DP for trial 4 for 1024 ms time interval (a) The occurrences of maximum DP for trial 1 for 512 ms time interval (b) The occurrences of maximum DP for trial 2 for 1024 ms time interval (c) The occurrences of maximum DP for trial 3 for 1024 ms time interval 42

14 4.5 (d) The occurrences of maximum DP for trial 4 for 1024 ms xv time interval Average of percentage of the maximum DP for 1024 ms time interval Average of percentage of the maximum DP for 1024 ms time interval Translation equations in C structure Location of cursor and target in workspace 47

15 xvi LIST OF ABBREVIATIONS ALS - Amyotrophic lateral sclerosis BCI - Brain-computer interface DP - Difference power ECoG - Electrocorticography/electrocorticogram EEG - Electroencephalography/electroencephalogram ERP - Event-related potential Hz - Hertz ms - Milisecond SCP - Slow cortical potential SMR - Sensorimotor rhythm

16 xvii LIST OF APPENDICES APPENDIX TITLE PAGE A Coding of cursor movement 57 B Coding to determine parameters 70 C List of publications 74

17 1 CHAPTER 1 INTRODUCTION 1.1 Introduction For normal people, communication is a need to undergo their daily activities. Communication is a process to transmit or transfer information, thought or by feeling by or to or between people or groups. It is a connection allowing access between persons by either verbal contact or action. However, there are people who suffer from locked-in syndrome which means that they are completely unable to control any muscle that prevent them from communicating with their caregivers or environment [1]. For such users, a braincomputer interface (BCI) is the only hope for even communicating with loved ones, controlling even simple devices like televisions or lamps or otherwise expressing oneself. BCI is a novel augmentative communication system that translates human intentions into a control signal for an output device such as a computer application [2] or a mobile robot [3], in which users send information using brain activity alone without conventional peripheral nerves and muscles [3].

18 2 BCI can be divided into two general categories, i.e., invasive and noninvasive [3]. Most of noninvasive BCI systems use electroencephalogram (EEG) signals; i.e., the electrical brain activity recorded from electrodes placed on the scalp. The main source of the EEG is the synchronous activity of thousand of cortical neurons. Measuring the EEG is a simple noninvasive way to monitor electrical brain activity, but it does not provide detailed information on the activity of single neurons (a few μvolts) and noisy environment (especially if recording outside shield rooms) [3]. In invasive BCI systems, the activity of single neurons (their spiking rate) is recorded from microelectrodes implanted in the brain. Such systems are being studied mainly in nonhuman primates, for example see [4]. These invasive BCIs face substantial technical difficulties and entail significant clinical risks as they require that recording electrodes be implanted in the cortex and function well for long periods, and they risk infection and other damage to the brain [5]. For human, therefore, noninvasive BCI systems are applied due to clinical risks and ethics [3]. 1.2 Problem Statement Among of the most important evaluation in one or two dimensional cursor movement-bci is hit rate. It is an onset element to design a practical application of BCI that depends in speed and accuracy. If the speed and accuracy can be substantially improved, the range of applications and the number of potential users would greatly increase [1]. In one dimensional cursor movement-electrocorticogram (ECoG) study, they achieved success rates of % [6] whereas in two-dimensional cursor movement-eeg study, hit rate was accomplished until 92% [5].

19 3 1.3 Objectives This study was conducted with the objective of: To determine parameters that will drive or control cursor in brain-computer interface (BCI) using noninvasive EEG signal. To determine parameters that can produce faster hit rate of brain-computer interface. 1.4 Scope of Project Figure 1.1 depicts the block diagram for this study s scope of work. First, EEG is recorded from the scalp and digitalized using acquisition system. Then the digitalized signals are subjected to feature extraction procedures, such as spectral analysis or spatial filtering. Afterwards, translation algorithm converts the EEG feature into command of the cursor movement. At the monitor screen, the cursor moves and hit target. Acquisition System Feature Extraction Translation Algorithm Cursor Movement Subject s head Figure 1.1 Scope of project

20 4 1.5 Project Contribution BCI uses human brain signal to control other devices. This technology has advances significantly in the world but not many researches can be found in Malaysia. Advanced research has lead to potential use in medical field as well as in computer games industry. This research has few purposes; firstly to develop human capital in BCI for Malaysia. Secondly, is to provide platform for a more advance application such as a two-dimensional cursor movement. A study by other researchers has found that slow frequency gave highest correct response rate when presented with visual presentation of target and non-target. This research also has found slow frequency as optimum parameter to control cursor movement. However, it is noted that the methodology for the experimental set-up was weak, leading to limited application to only specific one-dimensional cursor movement as any imagination may bring about the same results. However, with proper methodology and feature extraction, there is possibility of using slow frequency for BCI application that will increase the versatility of the field. The contribution of this research would be in the developed in-house software of BCI. The developed software is integrated with an existing hardware. The software main contribution is that it is an open source, has adjustable cursor and target size, and is panel window size-independent software. With the robust software, a more advance application can be developed and together with correct methodology, the objectives are obtainable. 1.6 Thesis Outline This report is divided into six chapters. This initial chapter presents introduction to the project which includes the project background, problem

21 5 statement, objectives of the project, the main objectives, scope of the project and the project contribution. This chapter is referred to conclude the findings of the project in conclusion part. Chapter 2 discussed about reviewed literatures and published works on BCI in view of applied methodologies over the years. This chapter also discussed on several types of electrophysiology sources and experimental designs including general overview of BCI study. The project methodology is covered in Chapter 3 that include discussion on the processes and steps involve in managing the raw data from EEG, the formula that being used and the information on software and hardware utilize in this study. Chapter 4 contains the result of feature extraction and feature translation in offline analysis. In feature extraction, EEG signals were analyzed using Fast Fourier Transform (FFT) in two time interval i.e ms and 512 ms. Afterward, two parameters from the transformed data in frequency domain were extracted, i.e., scalp location and frequency band. Applying a value coordinated by those two parameters, translation algorithm was performed. The performance in mean time, hit rate and movement precision were observed and compared with previous studies. Chapter 5 summarizes the results and findings from the previous chapter. The values of maximum difference in power which are pointed by the two parameters might be use to control and one-dimensional cursor movement. The overall project is rationalized and concluded in the final chapter, Chapter 6. Some suggestion and recommendation are also discussed.

22 6 CHAPTER 2 LITERATURE REVIEW 2.1 Introduction Brain-computer interface (BCI) is an important medical device for patients who have lost control of their motor faculties, and no longer outwardly express their needs and thoughts. A BCI enables these locked in patients to control a computer with their brain activity for communication, mobility and other purposes. This chapter introduces brain waves or electroencephalogram (EEG) as basic disciplines in BCI as well as basic design of BCI and also the type of BCI. Discussions about literatures and published works on BCI over the years are also summarized. 2.2 Electroencephalography The discovery by Hans Berger that the brain develops a low-level subaudiofrequency electrical activity has led to establishment of a neurophysiological specialty known as electroencephalography [7]. Electroencephalography (EEG) is a

23 7 measuring and recording method of electrical activity recorded from the scalp and the tracing result is known as electroencephalogram. The electroencephalogram or EEG is widely used by physicians and scientists to study brain function and to diagnose neurological disorders EEG Waveform An EEG waveform is a wave of electricity generated from the brain nerves and an information signal to indicate the cerebrum activity. Therefore, an EEG waveform offers important data to judge whether the brain activity is normal or not. However since an EEG waveform is taken through the skull, the absolute voltage is very small and it is difficult to indicate the activities of each brain part in detail owing to this natural insulation and also to the depth of each part. Usually, EEG signals are divided into a few bands according to their frequencies. They are summarized in Table 2.1 and the waveforms are shown in Figures 2.1 (a)-(e). Table 2.1 Frequency band for EEG signals Frequency band Delta Theta Alpha Beta Gamma Frequency range (Hz) 0 4 Hz 4 7 Hz 8 13 Hz Hz Above 31 Hz

24 8 Delta and theta waves are known collectively as slow waves. Delta waves normally are seen in deep sleep in adults as well as in infants and children while theta waves normally are seen in sleep at any age. Delta waves have the largest amplitude of all waves. Alpha waves generally are seen in all groups but are most common in adults. They tend to be present posteriorly more than anteriorly and are especially prominent with closed eyes and with relaxation. Alpha activity disappears normally with attention, for example mental arithmetic, stress and opening eyes. Beta waves are observed in all age groups. They tend to be small in amplitude and usually are symmetric and more evident anteriorly. Drugs, such as barbiturates and benzodiazepines [8], can augment beta waves. (a) Delta wave (b) Theta wave (c) Alpha wave (d) Beta wave (e) Gamma wave Figure 2.1 EEG waveforms

25 9 2.3 The International of Electrode Placement System For the purposes of EEG signals recording, there is a system known as the International of Electrode Placement System [9]. This system is the most widely used method to describe the placement of electrodes at specific intervals along the head. As shown in Figure 2.2, each electrode site has a letter to identify the lobe, along with a number or another letter to identify the hemispheric location. The letters F, T, C, P and O represent Frontal, Temporal, Central, Parietal and Occipital, respectively. Even numbers 2, 4, 6, and 8 indicate the electrodes at right hemisphere while electrodes at the left are indicated by odd numbers 1, 3, 5 and 7. Other than that, small letter z refers to an electrode placed in the midline. The 10 and 20 are referred to the 10% and 20% inter-electrode distance [9]. Figure 2.2 The of electrode placement system

26 Brain-Computer Interface A BCI is a system that allows a user to communicate with the environment only through cerebral activity, without using muscular output channels. To establish a direct link between the brain and a computer, the cerebral activity is measured and then analyzed with the help of signal processing and machine algorithms. Once a certain mental activity has been detected by the computer, a response can be displayed on a screen or a command can be sent to a device for example a robot or a remote control. The main application area for BCI is assistive technology for ablebodied as well as handicapped subjects. The basic design of BCI is shown in Figure 2.3. BCI SYSTEM SIGNAL ACQUISITION DIGITIZED SIGNAL Feature Extraction SIGNAL PROCESSING Translation Algorithm DEVICE COMMANDS Figure 2.3: Basic design and operation of any BCI system [1] In the BCI system, signals from brain are acquired by electrodes on the scalp and processed to extract specific signal features that reflect the user s intent. Then, these features are translated into commands that will operate a device [1].

27 Invasive and Non-invasive BCI BCI can be divided into two major methodologies which are invasive or noninvasive method. Invasive BCIs, that use activity recorded by brain implanted microor macroelectrodes, have higher spatial resolution, and might provide control signals with many degrees of freedom; whereas non-invasive BCIs, that use brain signals recorded with sensors outside the body boundaries [3], are convenient, safe and inexpensive, but have low spatial resolution. Another method called electrocorticography (ECoG) is an intermediate BCI methodology [10]. This moderately invasive method is using ECoG activity that recorded from the cortical surface. ECoG has higher spatial resolution than EEG, i.e., tenths of millimeter versus centimeter. It has broader bandwidth and higher amplitude, i.e., Hz and μv rather than 0-40 Hz and μv in EEG, respectively [6]. Experience of BCI research in human beings has so far primarily involved non-invasive EEG-based investigation, for example [5] and most intracortical (invasive) BCI involved animal studies [4]. Although there was a belief that only invasive BCI could provide users with real-time multidimensional control, previous study by Wolpaw et al. [5] presented that multidimensional control also could be provided using non-invasive BCI. They demonstrated that multidimensional control of a cursor movement on a computer screen can be learned in a few session of training with non-invasive BCI in humans. The subjects were able to move a cursor within ten seconds into one of eight goals appearing randomly at one of the four corners of the screen. The flexibility, speed and learning performance in generally equal to that seen when invasive multielectrode BCI systems were tested in animals [5]. At present, it seems probable that different recording methods will be useful for different applications, different users, or both [10].

28 Imagination vs. actual movement The goal of BCI research is to provide new communication option to people with severe neuromuscular disorders such as Amyotrophic Lateral Sclerosis (ALS), brainstem stroke, cerebral palsy and spinal cord injury [11]. Although people suffered from ALS or other severe motor disabilities, but 67.5% of them [12] appear that the cognitive functions are not impaired. The imagination of a movement generates the same mental processes and even physical processes as the performance of the movement [13]. Furthermore, brain signals produced by imagination movement are similar as actual movement. It is supported by a previous study on a tetraplegic patient, 31 years old male with a complete traumatic spinal cord injury (SCI) and five healthy volunteers with agematched acted as control group [14]. The patient underwent somatosensory evoked potential (SEP) and motor evoked potential (MEP) recordings to confirm the extent and location of cerebrospinal disconnection. They found that contrasts of imagery of repetitive hand and foot movement versus rest in the patient demonstrated significant activation of sensorimotor networks similar to the patterns defined by contrasts of active movement versus rest in the healthy control group. Another study [6] showed the spatial and spectral foci of task-related electrocorticogram (ECoG) activity, as revealed in the r 2 analysis, were usually similar for action and for imagination of the same action (e.g. saying the word move and imagining saying the word move ) Electrophysiological sources In BCI systems, electrophysiological sources refer to the neurological mechanisms or processes employed by a BCI user to generate control signals. Noninvasive BCI use the electrical activity of the brain recorded with single or multiple

29 13 electrodes from the scalp surface as input signal for BCI control. Usually participants are presented with stimuli or are required to perform specific mental tasks while the electrical activity of their brain is being recorded. Specific features of the EEG are either regulated by the BCI user such as slow cortical potential or sensorimotor rhythm, or elicited by sensory stimulation such as P300. A review from Bashahati et al. [15] reported that current BCIs fall into seven main categories, based on neuromechanics and recording technology they use. Five major groups categorized by Wolpaw [1] are sensorimotor activity, P300, visual evoked potential (VEP), slow cortical potentials (SCP) and activity of neural cell (ANC). The other two, added by Bashahati et al. [15] are response to mental tasks and multiple neuromechanics. All of them are summarized in Table 2.2 with simple descriptions. Table 2.2: Electrophysiological sources used in BCI [15] Electrophysiological Sources Sensorimotor activities P300 Visual evoked potential (VEP) Slow cortical potential (SCP) Activity of neural cell (ANC) Response to mental tasks Multi-neuromechanism Description Changes in brain rhythms (mu, beta, gamma) Typically evoked in the EEG over the parietal cortex a positive peak at about 300 ms after the stimulus is received. Small changes in the ongoing brain signal. They are generated in response to a visual stimulus. Slow, non-movement potential changes generated by subjects. When movements are executed in the preferred direction of neurons, the firing rates of neuron in the motor cortex are increased. Assume that different mental tasks lead to distinct, task-specific distributions of EEG frequency patterns over the scalp. Use a combination of two or more of the above-mentioned neuromechanisms.

30 14 Figures 2.3 (a)-(d) show the present-human BCI system types which are sensorimotor rhythm, slow cortical potential, P300 evoked potential and cortical neuronal activity, respectively. (a) Sensorimotor rhythm (b) Slow cortical potentials (c) P300 evoked potential (d) Cortical neuronal activity Figure 2.4 Present-day human BCI system types [1] Sensorimotor rhythm (SMR) is a majority preferable method of BCI because it is related to the motor cortex with distributions of somatosensory areas. It includes a frequency of 10 Hz (range 8-11 Hz) often mixed with a beta, around 20 Hz and a gamma component, around 40 Hz recorded over somatosensory cortices. It is most preferable over C 3 and C 4. SMR desynchronizes with movement, movement imagery and movement preparation (event-related desynchronization, ERD), and increases or

31 15 synchronizes (event-related synchronization, ERS) in the post-movement period or during relaxation [16]. Slow cortical potential (SCP) is among the lowest frequency features of the scalp-recorded EEG generated in cortex. Negative SCPs are typically associated with movement and other functions involving cortical activation, while positive SCPs are usually associated with reduces cortical activation [1]. SCP is used in [17] for thought translation device as one of the BCI application. The P300 component of event-related potential (ERP) is a positive deflection in the EEG time-locked to stimuli presentation. It is typically seen when participants are required to attend a stream of rare target stimuli [16]. P300 is prominent only in the responses elicited by the desires choice, and BCI uses this effect to determine the user s intent. A P300-based BCI has an apparent advantage in that it requires no initial user training. It is typically response to a desired choice [1] Signal Processing BCI technology is used to record and analyse brain signals to determine the output that is desired by the user such as which letter to select for spelling a word, which direction to move a cursor and so on. Therefore, signal processing is one of the important components in BCI research. This signal processing has two stages [10]. The first stage is feature extraction which is the measurement of the characteristics of the signals that encode the output. These features can be simple measures such as amplitudes of particular evoked potentials or of particular rhythm. The second stage is the translation of these signal features into device commands using a translation algorithm. Brain signal characteristics such as rhythm amplitudes are translated into command that specify outputs, such as letter selection or cursor

32 16 movement. Translation algorithm can be simple such as linear equation [11] or complex such as neural network [18] or support vector machines [19]. In this study, signal processing is used to extract features and the method used was frequency analysis. Bashashati et al. [15] reviewed that the magnitude of the Fourier Transform (FT) of the signal may be considered as a representation with perfect spectral resolution but with no temporal information. Frequency-based features have been widely used in signal processing because of their ease of application, computational speed and direct interpretation of the result. Recent and current studies seek to improve the speed and accuracy of this control by improving the selection of signal features and their translation into device commands, by incorporating additional signal features and by optimizing the adaptive interaction between the user and the system [11]. 2.5 Published Work on BCI A bibliometric study in [20] reviewed the growing interest with associated number of publications over the last two decades. Earliest paper describing a BCI system was published in 1973 by J. J. Vidal [21]. Two decades later, the number of papers grew substantially until over a thousand papers. [20] found that the most productive authors in BCI research are G. Purtscheller from Graz University of Technology (Austria), N. Birbaumer from University of Tubingen (Germany) and J. R. Wolpaw from Wadsworth Center (New York, USA). For instance, Wolpaw and his group have been published papers to move cursor in one dimensional and two dimensional. They focused on mu and beta

33 17 rhythm [5]. Their published work on two-dimensional (2D) cursor movement motivated current study for developing local brain-computer interface system. Using mu and beta rhythm component, their research tested on four disables subject produced the result that falls within invasive study using monkey. They emphasized that non-invasive of BCI also could provide multidimensional movement control without any implanted in the brain. In view of movement time, precision and accuracy, the results were comparable to these with invasive BCIs. BCI research areas include evaluation of invasive and non-invasive technologies to measure brain activity, development of new BCI applications, evaluation of control-signal (i.e. patterns of brain activity that can be used for communication), development of algorithm for translation of brain signals into computer commands, and the development and evaluation of BCI system specifically for disabled subjects Published Work on Neurofeedback Neurofeedback is a type of biofeedback that uses real time displays of electroencephalography to show brain activity, often with a goal of controlling central nervous system. It is a form of training or exercise for the brain which is assisted with a very sophisticated technology and guided or directed by knowledge gained through the advances of neuroscience. Neurofeedback in advanced is used to improve attention, anxiety, mood and behaviour. Regardless of clinical and psychological application, the currently most advanced BCI make use of the fact that humans can acquire control over EEG parameters by means of neurofeedback, i.e., real-time visual or auditory feedback of a specific EEG parameter.

34 18 For example, previous study by Hwang et al. [22] has developed a kind of neurofeedback systems to train motor imagery by presenting participants with timevarying activation maps of their brain, using a real-time cortical rhythmic activity monitoring system. By employing ten healthy subjects, five subjects in training group and another five subjects in control group, they conducted experiments consisted of two sessions which were motor imagery training session and EEG recording session. In the first session, participants were trying to increase their mu rhythm activations around the sensorimotor cortex while they were watching their cortical activation map through the real-time. In the second session, two EEG recordings were performed each before and after motor imagery training session to demonstrate the effect of their neurofeedback-based motor imagery training system. They found that there were meaningful changes of brain activity in training group around sensorimotor rhythm (10 and 20 Hz) compared to control group. With this, they confirmed the effectiveness of their neurofeedback-based motor imagery training in used of BCI application Publish Work on Cursor Control Cursor control movement is one of the application and assessment in BCI. Basically the study starts with one-dimensional cursor movement, horizontal or vertical and continues with advanced 2D or 3D cursor movement. As mentioned earlier, Wolpaw and colleagues have been published SMR-based BCI in cursor movement. Other than that, a study by Wilson et al. [23] published in Journal of Visualized Experiments (JoVE) measured the amplitude of the EEG signals in selected frequency bins into a device command, i.e., the horizontal and vertical velocity of computer cursor. They were utilizing BCI2000 application as analysis tool. Using the power in mu (8-12 Hz) and beta (18-28 Hz), the user in the study was

35 19 asked to make several different imagined movements with their hands and feet to determine the unique EEG features that change with the imagined movements. They determined which EEG features are strongly correlated with each movement by finding the large r-squared values in the plots produced from the analysis tool and found on channels 9, 10, 17, 18 and 19 according to international system electrode placement. The subject s goal was to move the cursor to the correct target using imagined movements corresponding to the desired direction of movement (e.g. right hand to move right or feet to move down). They found that a trained subject should be able to move cursor to the target within 1 or 2 seconds Published Work on Other Control The concept of BCI stems from a need for alternative, augmentative communication and control options for individuals with severe disabilities, though its potential uses extend to rehabilitation of neurological disorders, brain-state monitoring as well as gaming [16]. Previous study by Lalor et al. [24] addressed an application of steady-state visual evoked potential (SSVEP) based BCI design within a real-time gaming framework. The video game involved the movement of an animated character within a virtual environment Finke et al. [25] also presented a BCI game named the MindGame, based on the P300 event related potential. In the MindGame interface, P300 events were translated into movements of a character on a three-dimensional game board. While game and multimedia control with a BCI is inherently a challenging field, it potentially also opens the door for interesting applications in a clinical context [25].

36 20 Another study with different application was published by Bell et al. [26]. Their study was about to control a humanoid robot using non-invasive BCI. They also used P300 evoked potential as their subject s command to the robot. The P300 was used to discern which object the robot should pick up and which location the robot should bring the object to. The robot which was including a pan-tilt camera head for visual feedback to the user and two hands for grasping objects, transmitted the image of objects discovered by its cameras back to user for selection. The user was instructed to attend to the image of the object of their choice while the border around each image was flashed in a random order. Machine learning techniques were used to classify the subject s response to a particular image flash as containing a P300 or not, thereby the user s choice of the object could be inferred. A similar procedure was used to select a destination location. From the study, their results showed that a command for the robot can be selected from four possible choices in 5s with 95% accuracy [26]. In other study, a group of researcher from Singapore [27] was developing a new BCI application which could also provide novel ways of entertainment. They developed software and Graphical User Interface (GUI) on music composition using BCI. They investigated the potential of BCI in music composition encompassing the topic of gaming, entertainment and possibly rehabilitation with the objectives to investigate other communication channel than spelling words and to stimulate brainstorming on future BCI research avenues while reviewing the limitations of the current state-of-the-art [27]. 2.6 Discussion on Published Work in BCI Sensorimotor rhythms (SMRs) are the most often used of EEG component in non-invasive BCI system. Produced in sensorimotor cortex and associated areas, they are mostly directly related to movement. In addition, previous studies also suggested

37 21 that people could learn to control their SMRs amplitude. However, patients or users need to be trained several time to control their rhythms. For example in [5]. Moreover, not all users are able to obtain control of their brain waves after intensive training. P300 signal is one of the typical patterns captured from EEG at the central cortex and also mostly choose as a method in BCI system. Despite of its robustness, the such system may require permanent attention to external stimuli which may be fatiguing for some users. Considered as imagined-movement mental-task BCI system, the method in this study does not require a priori selection of subject-specific frequency and not required in-between interface like the screen of alphanumeric characters used in the VEP-based BCI. Our method assumed that different mental tasks, i.e., task and control condition, lead to distinct, task-specific distributions of EEG frequency patterns over the scalp.

38 22 CHAPTER 3 METHODOLOGY 3.1 Introduction This chapter discusses the methodology used in this study. It concerns two main issues of how the data were collected and generated as well as how they were analyzed towards the purpose of this study which is to examine and determine the parameter to drive cursor movement. This chapter includes experimental set up, the equipment, procedures and the flow chart of the parameter selection. 3.2 Experimental Set Up The experiments were conducted at Medical Electronic Lab, Faculty of Electrical Engineering, UTM. EEG signals were recorded using Neurofax EEG-9000 V data acquisition from Nihon Kohden, Japan. The experiments roughly took about one and half hour of setting up that includes preparing the subject, recording the data and cleaning the subject and equipments.

39 Equipment and Supplies There were a few equipments and supplies that need to be used together with EEG machine to complete the experiment. They were ECI TM electro-caps, electrode board adapter, body harness, disposable sponge disks, ECI Electro-Gel TM, blunted needle/syringe kit, ear electrodes, color-coded head measuring tape and electrode board adapter connector as shown in Figure 3.1(a)-(g). (a) Electro-cap (b) Electrode board adapter (c) Body harness (d) Disposable sponge disks (e) Blunted needle/syringe kit (f) Ear electrodes (g) Electrode board adapter connector Figure 3.1 Equipments used in the experiment

40 Preparation of Subject To operate EEG experiments, the subject must be relaxed with no stress and fatigue to avoid the EEG machine recorded unwanted signals. Therefore, to prepare the subject, steps taken were as follows: a. Body harness was put on the subject chest to support the cap. b. Before applying the suitable cap, the circumference of subject s head was measured. c. Ear electrodes were attached at the ears and gel was added. d. Disposable sponge disks were placed around F P1 and F P2 channels to absorb perspiration and prevent the spread of electrode gel onto the forehead. e. Then, the proper size of cap was slipped onto the subject s head and the cap straps were attached to the body harness. f. Cap connector was connected to the electrode board adapter. g. Each electrode cavity was filled with electrode-gel by moving the blunted needle/syringe rapidly back and forth. h. The electrode impedance was checked. i. Finally, the subject was moved to the chair for recording Installation of Electrodes onto Electrode Board Adapter The Electrode Board Adapter is used to connect the Electro-Cap to EEG instruments. Each wire/connector pin (socket) on the Electrode Board Adapter is color-coded for easy identification. Each wire contains of two tips which are white and red tip. Each of them represents the electrode placement as in Tables 3.1.

41 25 Table 3.1: Electrodes position White Tip Wire Color Red Tip F P1 Brown F P2 F 3 Red F 4 C 3 Orange C 4 P 3 Yellow P Green 0 2 F 7 Blue F 8 T 3 Violet T 4 T 5 Gray T 6 GND White C Z F Z Black P Z Clean Up When EEG recording finished, the cap, ear electrodes and body harness were removed from subject. ECI TM Electro-Caps must be cleaned frequently for sanitary reasons. In addition, if all the gel is not washed from a cap, the material will lose its elasticity and shortened the life of the cap.

42 Data Acquisition and Recording Process Participant Six normal subjects aged from 20 to 26 years old, gave informed consent and participated in the study Conditions Subjects sat on a chair facing a monitor screen, placed one meter in front of the subjects. In Control condition, subjects were asked to relax (resting). In Task condition, subjects were asked to imagine voluntary movement, e.g. imagine moving a cursor on a monitor screen Data Acquisition EEG signals were obtained from 19 scalp electrodes, placed on the scalp based on electrode placement system as discussed in section 2.3 with reference electrodes attached to left and right ears. Signals were recorded with pass bands of Hz and stored in a personal computer with a sampling frequency of 1 khz. A single trial lasted for 10 second and four trials were performed for each condition with intervening rest period to avoid fatigue as shown in Figure 3.2.

43 27 CC TC CC TC CC TC CC TC 10s 10s 10s 10s 10s 10s 10s 10s Trial 1 Trial 2 Trial 3 Trial 4 Notes: CC: Control Condition TC: Task Condition Figure 3.2 Sequence of data acquisition Data Analysis A 10 seconds data of Control (CC) and Task (TC) conditions (Figure 3.2) were divided into ten frames so that each frame consists of one second data. To obtain optimum speed and accuracy, the EEG data were divided at various intervals, i.e., 1024 ms and 512 ms as illustrated in Figure second of data 1 st frame 2 nd frame. 10 th frame 1024 data 1024 data 1024 ms Time Interval 1 st frame 2 nd frame. 10 th frame 512 data 512 data 512 ms Time Interval Figure 3.3 Division of data for 1024 ms and 512 ms time interval

44 28 Using Fast Fourier Transform (FFT), for each interval, data in frequency domain of each frame were obtained. For Control condition, five frames in the middle were averaged and assigned as baseline. Frequency was divided into six groups, i.e., delta band (0-4 Hz), theta band (4-7 Hz), alpha band (8-13 Hz), beta band (13 30 Hz), gamma band (31-50 Hz) and high gamma band (>51 Hz). Each frequency band in all frequencies of the Task condition was compared to the Control condition. Differences between these two condition were obtained and observed using Equation 1. difference in power (DP) (f) = power in Task (f) - power in Control (f) (1)

45 Flow Chart of Data Analysis Figure 3.4 shows the flow to analyze EEG signals using C programming in Linux. The process was repeated four times to analyze four trials of one subject. START Record EEG signals Control condition (10 s) Task condition (10 s) Calculate power spectrum using FFT Calculate power spectrum using FFT Average Control power spectrum Compute difference power between Task and Control power spectrum Find maximum difference power occurrence at which scalp location and frequency band END Figure 3.4 Flow chart of parameter determination process

46 30 After that, all maximum DP values were gathered into one folder to choose a single scalp location and frequency band. The process was summarized as in Figure 3.5. The maximum DP values from identified parameters were used to move cursor in further step. START Get maximum DP from Trial 1 Get maximum DP from Trial 2 Get maximum DP from Trial 3 Get maximum DP from Trial 4 Scalp location and frequency bands were identical? NO YES Average maximum DP for all trials Determine parameters END Figure 3.5 Flow chart of finalized parameter determination process

47 Software Development The software development has been carried out using the X Window System (commonly referred to as X or X11) on LINUX platform. This system is a networktransparent graphically windowing system based on a client/server model. Full coding commands can be referred in Appendix A (Movecursor.c) and B (Features.c). Both of them were developed using C language in LINUX. However in this section, the explanation focuses on the main graphics that associated to the purpose of the study. Figures 3.6(a), 3.7(a), 3.8(a) and 3.9(a) show some writing codes in X. Figure 3.6(a) shows the command to develop a workspace. The size of workspace was optimized according to the dimension of screen computer used, as shown in Figure 3.6(b). Figure 3.6(a) Syntax of workspace Figure 3.6(b) Workspace

48 32 In addition, the baseline inside the workspace as shown in Figure 3.7(b) was drawn using the coding command as in Figure 3.7(a). The coding drew one horizontal line and one vertical line which met at the centre. Figure 3.7(a) Syntax of baseline Figure 3.7(b) Baseline Furthermore, Figure 3.8(a) shows the syntax to build a block of rectangular cursor in blue colour. This variable cursor size was initiated at the centre of the screen. For example, the cursor size in this study was 10 x 10ixels as shown in Figure 3.8(b).

49 33 Figure 3.8(a) Syntax of cursor Figure 3.8(b) Cursor Figure 3.9(a) shows the language structure to display target. The target was a graphic located at the right side of the workspace as shown in Figure 3.9(b). It was inconstant value and designed in black colour. The size of the target was 30 x 30 pixels used in this study.

50 34 Figure 3.9(a) Syntax of target Figure 3.9(b) Target In the situation of cursor movement, it should jump from the initial state according to a value indicated by extracted parameters. If the cursor succeed hit the target within ten seconds, a reward in form of string text would appear.

51 35 CHAPTER 4 FREQUENCY ANALYSIS AND FINDINGS 4.1 Introduction This chapter presents the result and analysis of the project. The results are arranged into two parts which are feature extraction results and offline feature translation results. 4.2 EEG signals Figure 4.1 shows an example of raw EEG signals of one channel in one second, taken from subject 1 at trial one located at P Z channel. Meanwhile the example of power spectrum of EEG signal is shown in Figure 4.2. From 19 channels of EEG signals recorded, however, there were only nine channels analyzed i.e. F 3, F Z, F 4, C 3, C Z, C 4, P 3, P 4, P Z and P 3 (Figure 4.3) that represents the location commonly used in BCI research.

52 36 Notes: Task Control Figure 4.1 Example of 1 second of raw EEG signal from subject 1 at P Z channel Notes: Task Control Figure 4.2 Example of power spectrum of EEG signal from subject 1 at P Z channel

53 37 Notes: Task Control F3 FZ F4 C3 CZ C4 P3 PZ P4 1s Figure 4.3 Example of 1 second data of EEG signal from subject Feature Extraction Analysis To extract mean values from EEG signals, two time intervals were applied, i.e., 1024 ms time interval and 512 ms time interval (Figure 3.3). The findings from both time intervals were compared.

54 ms Time Interval Figures 4.4(a)-(d) show the results of percentage of maximum DP using 1024 ms time interval for all trial (6 subjects x 10 frames). For trial 1 (Figure 4.4 (a)), maximum DP was found at P 4 site with 56.7% occurrence. For trial 2, (Figure 4.4 (b)), maximum DP was observed at another site i.e. P 3 with 51.7% occurrence. For trials 3 and 4 (Figures 4.4 (c) and (d)), both of maximum DP was found at P Z site with 61.7% and 60.0% occurrence, respectively. Generally, the maximum DP occurred at delta band, which ranged from 0 to 4 Hz, for all trials. (Delta, P 4, 56.7%) (a) Trial 1