Development of an Optical Brain-Computer Interface Using Dynamic Topographical Pattern Classification

Transcription

1 Development of an Optical Brain-Computer Interface Using Dynamic Topographical Pattern Classification by Larissa Christina Schudlo A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Graduate Department of Institute of Biomaterials and Biomedical Engineering University of Toronto Copyright by Larissa Christina Schudlo 2012

2 Abstract Development of an Optical Brain-Computer Interface Using Dynamic Topographical Pattern Classification Larissa Christina Schudlo Master of Applied Science Graduate Department of Institute of Biomaterials and Biomedical Engineering University of Toronto 2012 Near-infrared spectroscopy (NIRS) in an imaging technique that has gained much attention in brain-computer interfaces (BCIs). Previous NIRS-BCI studies have primarily employed temporal features, derived from the time course of hemodynamic activity, despite potential value contained in the spatial attributes of a response. In an initial offline study, we investigated the value of using joint spatial-temporal pattern classification with dynamic NIR topograms to differentiate intentional cortical activation from rest. With the inclusion of spatiotemporal features, we demonstrated a significant increase in achievable classification accuracies from those obtained using temporal features alone (p < 10-4 ). In a second study, we evaluated the feasibility of implementing joint spatial-temporal pattern classification in an online system. We developed an online system-paced NIRS-BCI, and were able to differentiate two cortical states with high accuracy (77.4±10.5%). Collectively, these findings demonstrate the value of including spatiotemporal features in the classification of functional NIRS data for BCI applications. ii

3 Acknowledgments First and foremost, I would like to express sincere gratitude to my supervisor, Dr. Tom Chau, for his guidance and encouragement, which was fundamental in the completion of this work. His energy and enthusiasm has truly been a source of motivation over the past two years. Gratitude is also expressed to my committee members, Dr. Elaine Biddiss and Dr. Eric Bouffet, and external examiner, Dr. Tony Easty, for their helpful comments and suggestions. I would also like to thank the members of the PRISM lab for their support. In particular, I extremely grateful for the endless advice Sarah Power has given me, as well as the technical support provided by Ka Lun Tam. In addition, I would like to express my appreciation to Nicholas Schudlo for his help with the artwork in papers of this thesis. Finally, I would like to acknowledge the financial support of the Natural Sciences and Engineering Research Council of Canada, donors of the Kimel Family Graduate Student Scholarship in Paediatric Disability, Holland Bloorivew Kids Rehabilitation Hospital, and the Institute of Biomaterials and Biomedical Engineering at the University of Toronto. iii

4 Contents 1. Introduction Motivation Research Questions & Objectives Thesis Outline Background Brain-Computer Interfaces Neurovascular Coupling Near-Infrared Spectroscopy Measuring Hemoglobin Concentrations Measurements & Instrumentation NIRS-BCI Applications NIR Topographical Imaging Source-Detector Configuration Image Construction Cortical Activity and Pattern Classification Feature Extraction Mental Arithmetic & the Prefrontal Cortex Dynamic Topographical Pattern Classification of Multichannel Prefrontal NIRS Signals Abstract Introduction iv

5 3.2.1 NIRS-BCIs & Classification Review of Studies Incorporating Visualization Image Moments Motivation Methods & Materials Participants Instrumentation Experimental Protocol Signal Preprocessing Data Analysis Results & Discussion s Feature Window s Feature Window Limitations in Calculating Hemoglobin Concentrations & Generating Images Conclusion Automatic Differentiation of Prefrontal Hemodynamic Activity Due to Mental Arithmetic and Rest Using Online NIR Topographical Pattern Classification Abstract Introduction Potential for NIRS as a BCI Modality Activation Task Classifier Training Control Paradigm Objectives Materials & Methods Participants v

6 4.3.2 Instrumentation Experimental Protocol Activation Task Mental Arithmetic Offline and Online Data Collection Data Analysis Preprocessing Feature Extraction Feature Selection & Online Classification Offline Classification of First Experimental Session Results Discussion Online Classification Differentiating Mental Arithmetic and Rest Training Protocol Qualitative Results - Feedback & Mental Strategies Comparison to Existing Studies Study Limitations Conclusion Conclusions Contributions Future Work Investigation of Other Cognitive Tasks Determining Optimal Task Interval Duration Evaluating the Effects of Learning and Practice on Task Performance Broadened Testing to Include More Practical Situations Movement Towards the Target Population References vi

7 A. Pseudo Online Results of Study vii

8 List of Tables Table 3.1 Values used for specific extinction coefficient (ε) and differential path factor (χ) in modified Beer-Lambert Law Table 4.1 Values used for specific extinction coefficient (ε) and differential path factor (χ) in modified Beer-Lambert Law Table 4.2 Online classification results Table 4.3 Per session and overall average ratings of helpfulness of continuous feedback (rating ± standard deviation) Table A.1 Overall across-participant average classification accuracies for four different classifiers considered in the pseudo online classification of study Table A.2 Overall across-participant average sensitivities for four different classifiers considered in the pseudo online classification of study Table A. 3 Overall across-participant average specificities for four different classifiers considered in the pseudo online classification of study Table A.4 Individual participant mean classification accuracies in pseudo online classification of study 1 achieved using the classification algorithm implemented in study Table A.5 Individual participant mean sensitivities in pseudo online classification of study 1 achieved using classification algorithm implemented in study viii

9 Table A.6 Individual participant mean specificities in pseudo online classification of study 1 achieved using classification algorithm implemented in study ix

10 List of Abbreviations ANN Artificial Neural Network BCI EEG Brain-Computer Interface Electroencephalography fmri Functional Magnetic Resonance Imaging Hb HbO IIR Deoxygenated Hemoglobin Oxygenated Hemoglobin Infinite Impulse Response LDA Linear Discriminant Analysis NIR Near-Infrared NIRS Near-Infrared Spectroscopy PET PFC Positron Emission Tomography Prefrontal Cortex SFFS Sequential Forward Floating Search thb Total Hemoglobin x

11 List of Figures Figure 2.1 Neurovascular Coupling Figure 2.2 Path travelled by photons from source to detector Figure 2.3 Absorption spectrum of electromagnetic energy in nm range Figure 2.4 Source-detector arrangements for various measurement point densities Figure 3.1 Source-detector configuration Figure 3.2 Timing diagram of example trial Figure 3.3 Overall across-participant average accuracies for the 4 classification schemes for (a) 0-20s feature window and (b) 0-10s feature window Figure 3.4 Frequency at which each feature type was selected from pool containing both spatiotemporal and temporal features over all 5 sessions for 0-20s feature window Figure 3.5 Individual participant mean classification accuracies across all 5 sessions for the 4 classification schemes for (a) 0-20s feature window and (b) 0-10s feature window Figure 3.6 Example signals of most frequently selected (a) spatiotemporal features for 0-20s feature window (b) temporal features for 0-20s feature window (c) spatiotemporal features for 0-10s feature window and (d) temporal features for 0-10s feature window Figure 3.7 Example trials of the most frequently selected spatiotemporal feature, and the most frequently selected temporal feature and a sequence of normalized [Hb] NIR topograms for a mental arithmetic response interval at (a) the start and (b) the end of a session xi

12 Figure 4.1 Source-detector Configuration Figure 4.2 Annotated visual interface Figure 4.3 Timing diagram of example trial Figure 4.4 Classifier training and testing protocol Figure 4.5 Individual participant and mean classification accuracies per session Figure 4.6 Online classification accuracy versus days between session 1 (initial training session) and either session 2 or session 3 (online sessions) xii

13 Chapter 1 Introduction 1.1 Motivation Severe motor impairments can hinder an individual s ability to communicate and interact with their surroundings, limiting or even eliminating their independence. Establishing a means of communication without the use of muscles or peripheral nerves would provide these individuals the ability to express themselves and exert environmental control. Brain-computer interfaces (BCIs) constitute one approach to establish these unconventional channels of communication and interaction. Several different brain-monitoring modalities have been investigated for establishing a brain-device pathway and among them, near-infrared spectroscopy (NIRS) has recently gained much attention. This modality has proven to be successful in imaging cortical activity induced by a variety of tasks, including motor imagery [1 7] and higher-order cognitive tasks [8 19]. However, its application toward establishing a BCI is still in its infancy. To date, NIRS-BCI studies have focused on classifying activity using temporal features, derived from the timecourse of optical signals or hemoglobin concentrations measured from discrete locations throughout the measurement area. Alternatively, functional imaging studies have suggested that the spatial characteristics of a function response are also rich in information [3], [13], [20 25]. Yet, this information has not been considered in the automatic differentiation of cortical states via NIRS. In this thesis, the primary objective was to investigate the value of using joint spatialtemporal pattern classification with dynamic NIR topographic maps to differentiate intentional cortical activation from baseline. Furthermore, the secondary objective was to determine the feasibility of incorporating this method of analysis in an online NIRS-BCI. Considering the

14 CHAPTER 1. INTRODUCTION 2 spatial attributes of activity in conjunction with temporal changes may afford a better understanding of the dynamic response observed, and in turn, improve the classification of different cortical states. 1.2 Research Questions & Objectives To evaluate the efficacy of employing dynamic topographical pattern classification in an NIRS- BCI, the following questions were investigated: 1. How does the rate of automatically detecting intentional cortical activation change when spatiotemporal features extracted from dynamic NIR topograms are considered for classification, in addition to or in place of temporal features of hemodynamic activity? 2. Can we automatically detect intentional cortical activation at an accuracy exceeding 70% using an online pattern classification algorithm which incorporates spatiotemporal features, where 70% is the minimum accuracy necessary for effective communication with a BCI [26]? To answer these questions, the immediate objectives of this thesis were: 1. To determine the accuracy at which intentional prefrontal activation induced by mental arithmetic can be differentiated from rest using a classification algorithm that incorporates spatiotemporal features extracted from dynamic NIR topograms. 2. To compare the accuracy of a classifier deploying exclusively temporal features of NIRS signals to that of a classifier incorporating spatiotemporal features, where the goal is to differentiate between intentional prefrontal activation and rest. 3. To develop an online NIRS-BCI with visual feedback using a pattern classification algorithm that incorporates a combination of spatiotemporal and temporal features to automatically differentiate intentional prefrontal activation from rest.

15 CHAPTER 1. INTRODUCTION Thesis Outline Following this introductory chapter, Chapter 2 presents background information regarding BCI research, the mechanisms of NIRS and monitoring functional brain activity via hemodynamic response, NIRS-BCI studies to date, NIR topography, pattern analysis techniques employed in image processing, and cortical activation induced via mental arithmetic. To address its immediate objectives, two studies were completed during the course of this thesis. The first study is presented in Chapter 3, which focused on objectives 1 and 2, demonstrating the ability to accurately differentiate mental arithmetic from rest using a classification algorithm that incorporates spatiotemporal features and improving upon the results achieved using temporal features alone. The analyses for this study were conducted offline. Chapter 4 presents the second study of this thesis, which addressed the third objective, measuring the accuracy with which intentional cortical activation can be differentiated from rest using both spatiotemporal and temporal features in an online NIRS-BCI with user feedback. Finally, Chapter 5 summarizes the main contributions of this research and suggests possible areas for future work.

16 Chapter 2 Background 2.1 Brain-Computer Interfaces A BCI is a pathway that connects the brain with an external device, and aims to provide a means of expression without the need for muscular input. In such a pathway, the user produces characteristic patterns in their brain activity, typically induced by a mental task, to indicate intent. The distinct activity is then recognized and translated by the system into a command signal to be used, for example, for controlling an assistive device. Thus, this technology circumvents the body s natural output pathway and can be employed as a channel of communication or control for individuals who lack the voluntary motor control to use conventional, movement-based assistive devices due for example, to amyotrophic lateral sclerosis, brainstem stroke, or severe cerebral palsy [27]. A variety of techniques, both invasive and non-invasive, have been used to measure brain activity for BCI applications. Invasive BCIs measure brain activity through electrodes implanted within the brain, whereas non-invasive BCIs make use of sensors on the outside of the body. The most popular modality employed in non-invasive BCIs is electroencephalography (EEG), which measures electrical signals on the scalp produced by neural activity [28]. Although this technique is well-developed and has excellent spatial and temporal resolution, it is inherently noisy and can be corrupted by signals produced by muscles, motion artefact and environmental sources. Conductive paste is often required to hold electrodes in place on the skin surface, further reducing the mobility of the user, and corrosion of electrodes prevents long-term use. Additionally, this modality can require manipulation of signals that individuals normally do not

17 CHAPTER 2. BACKGROUND 5 exhibit natural control over, resulting in difficult adaptation and frustration for the user [29]. Functional imaging modalities such as functional magnetic resonance imaging (fmri) and Positron Emission Tomography (PET) have also been used in non-invasive BCIs, and rely on cerebral hemodynamics to differentiate brain states [30 32]. However, these methods of measuring hemodynamic activity are very impractical due to high cost and lack of portability. 2.2 Neurovascular Coupling Localized hemodynamic activity can be used to detect a functional response in the brain through an indirect vascular response to increased neural activity. This phenomenon is known as neurovascular coupling. To carry out metabolic reactions within biological tissues, key molecules are required. Among these substances is oxygen, which is transported by hemoglobin in the blood. Consequently, as the degree of activity within a cortical region increases, the heightened consumption of oxygen is met by an augmented supply of blood. By monitoring the intensity of hemoglobin concentration over time, increased cerebral blood flow can be detected and thus, heighted cortical activity can be deduced (Figure 2.1) [33 35]. One way this series of physiological reactions can be induced is through the performance of a cognitive task. Change in neural activity Change in oxygen demand Change in localized cerebral blood flow Change in oxy-hb & deoxy-hb Figure 2.1 Neurovascular Coupling. A change in neural activity in the brain will indirectly result in changes in oxygenated and deoxygenated hemoglobin concentrations. This indirect vascular response can used to detect functional brain activity.

18 CHAPTER 2. BACKGROUND Near-Infrared Spectroscopy Recently, NIRS has become a promising modality for measuring tissue hemodynamics and assessing functional brain activity [35], [36]. It is a non-invasive imaging technique that uses near-infrared light to measure hemoglobin concentration, and can be used to detect activity in the cerebral cortex by irradiating light through the scalp and meningeal layers of the brain. This optical imaging technique is an attractive alternative to other functional imaging modalities as it is portable, safe and lower in cost. While NIRS may not have the same spatial resolution as fmri or PET, it can be used in everyday environments, making it suitable for BCI applications [37] Measuring Hemoglobin Concentrations NIRS measures the concentration of oxygenated and deoxygenated hemoglobin within biological tissues by exploiting variations in biophotonic characteristics. Photons are emitted from the scalp into the cortical tissue by a light source, known as an optode, and travel in a predictable, curved path (Figure 2.2). Upon returning to the surface, the resultant light intensity is measured by a photodetector some distance away. For determining the concentrations of biological molecules present within the cortical area, the degree of light attenuation within each source-detector pair, henceforth referred to as a channel, is the optical property of interest. Sources of this attenuation include both scatter and absorption. Scattering is a consequence of photon interactions with different tissue boundaries. Because this does not change with the levels of hemodynamic activity, it is assumed to be constant for all measurements. Conversely, absorption is a result of photon interaction with different molecules, which varies according to the degree of activity within the tissue. Furthermore, each type of molecule absorbs a characteristic wavelength of light. Therefore, the extent of attenuation of each wavelength is indicative of the molecular composition within the tissue.

19 CHAPTER 2. BACKGROUND 7 Figure 2.2 Path travelled by photons from source to detector. Photons are emitted into the tissue and are assumed to travel in a curved path toward a detector on the surface some distance away. Image from [37]. To measure hemoglobin concentrations, NIRS utilizes electromagnetic energy with wavelengths in the range of 650 to 900 nanometers. Within this optical window, light absorption is dominated by hemoglobin molecules, and the degree of absorption due to tissue and water is small in comparison (Figure 2.3). Moreover, the extent of light absorption at each wavelength is dependent on the oxidative state of the hemoglobin molecule with which it interacts. Due to the unique absorptive properties of each chromophore, separation of wavelengths allows for the concentrations of oxygenated and deoxygenated hemoglobin to be calculated independently. Figure 2.3 Absorption spectrum of electromagnetic energy in nm range. NIRS utilizes wavelengths of light within the 'optical window' of approximately 650nm to 900nm, where hemoglobin provides maximum absorption and surrounding water and tissue absorption is minimized. Image from

20 CHAPTER 2. BACKGROUND 8 By comparing hemoglobin concentrations to baseline levels, regions of heightened neural activity can be deduced. Typically, mental activation induces a decrease in the concentration of deoxygenated hemoglobin, followed by an increase in the concentration of oxygenated hemoglobin. Because the increase in oxygenated hemoglobin generally surpasses the metabolic demand, an overall increase in total hemoglobin concentration is also observed [36]. However, deviations from this characteristic response pattern have been noted [14], [33], [38]. To evaluate the hemodynamic activity within a cortical region, the absolute oxygenated and deoxygenated hemoglobin concentrations can be calculated using the modified Beer-Lambert Law: A λ = attenuation of light intensity at wavelength λ I oλ = the intensity of light entering the tissue at wavelength λ I λ (t) = the intensity of light exiting the tissue at wavelength λ at time t ε λ C λ χ d G λ = the specific extinction coefficient of the absorber (molar concentration) at wavelength λ = concentration of the absorber at wavelength λ = the differential path factor at wavelength λ = geometric distance between the emitter and detector = a term that is influenced by the geometry of the medium and accounts for the loss of intensity due to scattering. However, assumptions regarding the travel of light through the tissue must be made in order to apply this law. Consequently, these suppositions diminish the accuracy of the estimated concentrations and the spatial resolution of NIRS. Though scattering is the most dominant photon-tissue interaction at NIRS wavelengths, as previously mentioned, it is believed to be unaffected by changes in regional cerebral blood flow. Therefore, it remains unchanged, despite fluctuations in activity [36]. Consequently, G λ is assumed constant, and can be eliminated from the modified Beer-Lambert Law if the change in concentration is calculated rather than the absolute value [14]. Additionally, the medium for photon travel is assumed to be homogeneous.

21 CHAPTER 2. BACKGROUND 9 However, since the light travels through multiple tissue layers, each with distinct optical properties, this assumption is invalid [36]. Lastly, the changes in light intensity are believed to be homogenous within the sampling medium, which further detracts from the quantitative accuracy of the concentrations calculated [36]. Nonetheless, by accepting these simplifications, changes in oxygenated, deoxygenated, and total hemoglobin concentrations can be estimated with the following equations: [ ] [ ] [ ] [ [ ] ] (2.2) [ ] [ ] [ ] (2.3) Measurements & Instrumentation To measure the hemodynamic activity within an area with a useful degree of spatial resolution, an array of source-detector pairs is often used to take NIRS measurements. The midpoint of each channel, known as the measurement point, is the position of greatest light attenuation [39], and the distance between source and detector dictates the depth at which this maximum attenuation occurs. A minimum source-detector spacing is required to ensure an adequate signal is captured, and enhanced light penetration is achieved with increased separation. However, deeper photon travel into the tissue also results in greater scattering, reducing the signal-to-noise ratio of the signal that reaches the detector. Typically, separation distances between two and three centimetres provide an optimal combination of signal quality and depth penetration [40]. At these source-detector separations, photons travel approximately one to three centimeters below the surface of the scalp, enough to reach the outermost layers of the cortex [5], [41]. Currently, three types of NIRS systems exist [1]. The simplest is a continuous-wave system, which only provides qualitative changes of oxygenated and deoxygenated hemoglobin concentrations within the tissue. Alternatively, the other two varieties, time- and frequencyresolved systems, measure the total path length travelled by the photons between each source and

22 CHAPTER 2. BACKGROUND 10 detector, allowing for more accurate concentration estimates to be made. With a time-resolved NIRS system, the path length is determined using the duration of time required for photons to travel from source to detector. In contrast, the phase shift observed between source and detector of high-frequency modulated light (in the megahertz range) is related back to the photon path length in a frequency-resolved system NIRS-BCI Applications While EEG is the primary modality deployed in non-invasive BCI research [27], NIRS has become an increasingly popular alternative. Though NIRS-BCI research is still in the developmental stages, there is a significant body of evidence demonstrating the ability to detect functional activity resulting from various mental tasks with this optical imaging technique. This capability can be exploited as a brain-device pathway, in which the user conveys the intention of triggering a BCI output through a task-induced hemodynamic response. The heightened degree of blood flow in a predetermined region of the brain can then be recognized and translated into a command signal. Among the most popular measurement regions for BCI development is the motor cortex, and a number of NIRS studies have demonstrated positive results in detecting activation induced by imagined motor movements in this cortical region [1 6]. Consequently, this type of task could be used as a basis for control of an NIRS-BCI for individuals who cannot necessarily activate the motor cortex through physical movements due to neuromuscular impairments. However, for individuals with congenital or long-term motor deficiencies, it may be difficult or impossible to produce a significant response in this region of the brain [42 44]. In addition, hair can compromise optode coupling and the strength of the optical signals measured, making the motor cortex a suboptimal location for taking NIRS measurements [5], [45]. Alternatively, the prefrontal cortex (PFC) may be a suitable location to take hemodynamic measurements, as this area of the brain is less likely to be implicated in motor disabilities. This cortical region, located underneath the forehead, is involved in many complex activities including problem solving, working memory and making emotional connections. Neural activation in this area can be induced by a variety of cognitive tasks including mental arithmetic

23 CHAPTER 2. BACKGROUND 11 [8], [9], [12], [14 16], [38], [46], music imagery [8], [9], [15], [17], verbal exercises [11 13], [46 48] and working memory tasks [10], [37], as well as other higher-order processes such as emotional responses [49] and preference [50], [51]. The ability to detect the performance of these mental processes supports the use of this cortical region in establishing a BCI. Moreover, these cognitive tasks may be particularly advantageous in BCI control, as they are more intuitive to perform than motor-based tasks, and would therefore reduce the mental demand on the BCI user. A major advantage of employing NIRS over other brain-monitoring modalities in a BCI is the reduced training requirements. NIRS harnesses a signal that typically results from natural thought processes. This helps to reduce the training necessary to gain adequate control of one s cortical activity and thus, increases the ease-of-use of the BCI. However, the primary drawback of using this modality in a BCI, particularly in an online system, is the inherent latency associated with a hemodynamic response. Induced functional activity is typically detected five to eight seconds post-stimulus, which places a considerable limit on the response time of the system [2], [40]. Yet, it has been found that significant hemodynamic activation can be classified in as short as 2.5 second following stimulus onset [49]. The provision of real-time feedback reflecting the strength of activation has also proven to enhance the intensity of the hemodynamic response [6], [30], which in turn, can improve the efficacy of detecting functional activity and optimize the achievable transfer rate of the system. However, despite the promise of establishing an online NIRS-BCI and the potential influence of user feedback on one s ability to modulate cortical activity, data analyses for NIRS-BCI studies have primarily been performed offline. 2.4 NIR Topographical Imaging NIR topography is a functional imaging technique that utilizes the composite of NIRS measurements to generate two-dimensional topographic images that represent hemodynamic activity throughout the entire irradiated area [52]. The same advantages and disadvantages of NIRS are inherent in this imaging technique. However, it has the added benefit of providing a visualization of the location, shape and spatial extent of hemodynamic activation in the region. Additionally, a series of NIR topograms can be compiled to generate a dynamic NIR topogram,

24 CHAPTER 2. BACKGROUND 12 to highlight the progression of the hemodynamic response over both time and space. These attributes of functional activity have been found to be rich in information [13], [20 24]. However, despite this potential value, automatic differentiation of spatial patterns within dynamic NIR topograms has yet to be investigated. Thus far, functional imaging studies incorporating NIR topography have primarily aimed at demonstrating the reliability of this modality to image localized blood flow and detecting functional activity through the comparison of static images at different time points [3], [45], [53 55]. Due to photon scatter and the assumptions often associated with NIRS (see section 2.3.1), images generated via NIR topography suffer from a reduced spatial resolution in comparison to other functional imaging techniques. As the gold standard of topographic imaging, fmri offers a spatial resolution of approximately 5 millimeters, and in comparison, the resolution of NIR topography is four to five times greater. However, establishing an optical BCI that incorporates NIR topograms requires the detection of distinct spatial activation patterns associated with each brain state, and not necessarily exact measurements of hemoglobin concentrations over an area. The aim is not to understand the mechanisms of functional brain activity or to use the images for diagnostic purposes, but rather to extract repeatable patterns from the estimated representation of activity. Nonetheless, selecting an appropriate source-detector configuration and image generation algorithm can improve the quality of the image, and provide a more informative representation of the hemodynamic response Source-Detector Configuration The source-detector configuration employed in measuring NIRS signals can significantly impact the spatial resolution of an NIR topogram. Similar to NIRS, the source-detector spacing for NIR topography is fixed and is typically between two and three centimeters. However, the density of measurement points (or sampling points) can be varied to improve the resolution of the images generated. Specifically, a denser array of channels with overlapping measurement regions can enhance the image quality and the ability to detect focal changes [39], [56]. Yammamoto et al. [39] tested various source-detector configurations (Figure 2.4) using an absorptive source with

25 CHAPTER 2. BACKGROUND 13 known size and a heterogeneous phantom mimicking the layers of the head and its optical properties. Not only did an increased source-detector density improve the spatial sensitivity of the NIR topograms, it reduced the variability in the estimated size and position of activation as the absorber distances were adjusted with respect to the measurements points. Interestingly, the Double-density arrangement was most effective in terms of maximizing spatial resolution and minimizing probe density, as the Quadruple-density arrangement did not offer significant improvements in resolution over the sparser arrangement. For pattern classification purposes, higher resolution and contrast within NIR topograms could potentially improve the reliability with and speed at which activation is detected and feedback can be given [30]. Yet, variations in both the size and shape of activation induced by motor tasks have been successfully detected using sparse probe densities comparable to that of the Lattice arrangement [3], [23], [45], [53], [57].

26 CHAPTER 2. BACKGROUND 14 Figure 2.4 Source-detector arrangements for various measurement point densities. Source-detector separation is 3 cm for each arrangement. The distance between measurement points is 2.1cm for the Lattice Arrangement, 1.5 cm for the Double-Density Arrangement, and 1.1 cm for the Quadruple-Density Arrangement. Image from [36] Image Construction In addition to the source-detector configuration, the resolution of an NIR topogram is also impacted by the method utilized for image reconstruction. Because the path travelled by each

27 CHAPTER 2. BACKGROUND 15 individual photon is unique, the exact spatial distribution of attenuation changes is difficult to predict. Consequently, assumptions and approximations regarding the passage of light through tissue are often made in order to estimate activation for producing an NIR topogram. Conventional methods of NIR topogram generation employ smoothing and interpolation algorithms [36]. The measured changes in light intensity are first mapped to the midpoint of each source-detector pair, as this is the area of highest light attenuation within each channel [56], followed by an estimation of intermediate values. From the collection of known quantities, the unknown values have been approximated using a variety of methods including linear [3], [53], backprojection [5], [7], [11], [23], [57], spline [13], [39], [56] and nearest-neighbour [58] interpolation. With this straight-forward approach of NIR topogram generation, it is assumed that all photons travel a uniform path length through a homogeneous medium, and any changes in light intensity are strictly a consequence of absorption in the tissue. These assumptions, though conventional, reduce the spatial resolution of the resultant image. However, no a priori knowledge of the tissue properties or layers is required, making this method computationally inexpensive. In addition, the time and frequency-domain NIRS approaches, which measure the path length travelled by photons, can help to improve the depth and spatial resolution of the images [36]. For more accurate image construction, the head can be modelled as a heterogeneous medium and photon scattering events can be accounted for using Monte Carlo simulations [56], [59], [60]. In this approach, individual photon paths are estimated, and the accumulated trajectories are used to generate a probability distribution of photon travel throughout the area. This distribution, or spatial sensitivity profile, can then be used to model the transport of light through the head and approximate the dispersion of absorption change in each channel. The transport of millions of photons must be modelled throughout the medium to obtain reliable statistics [61]. With this approach, accounting for photon scatter helps to enhance the spatial resolution of the NIR topograms generated in comparison to those created using a simple spatial interpolation technique. However, the inverse problem of the model must be solved to determine the values of absorption change for all pixels in the image. The solution requires an iterative approach (for example by using the Newton Raphson method) and must be done for each intensity measurement taken. Consequently, this technique is computationally expensive and not necessarily suitable for implementation in a real-time system [59].

28 CHAPTER 2. BACKGROUND 16 Lastly, another approach for NIR topogram generation has been to use finite difference or finite element modelling to depict the passage of light through the head with the partial diffusion equation. These models divide the measurement area into a mesh, and determine a solution to a partial differential equation for each section. Finite difference models divide the measured area into simple shapes, and therefore can only be used for straightforward geometries. Finite element modelling is often used for more complex shapes but requires the domain to be divided into irregular components. Because this approach does not take individual photon lengths into consideration, it is computationally faster than methods employing Monte Carlo simulations [62]. However, it requires an assumed partial differential equation to model light movement for the particular surface under consideration, and an approximation of surface parameters and boundary conditions. Additionally, it is often limited to a two-dimensional model of light propagation due to computational expense, and does not account for diffusion occurring in the direction perpendicular to the source-detector plane [60]. 2.5 Cortical Activity and Pattern Classification Essential in establishing a BCI is the ability to differentiate characteristic patterns associated with each cognitive state. Statistical pattern classification, specifically supervised learning, is a technique often used to differentiate cortical patterns of activation for BCI development. It involves the development of an algorithm which evolves or learns the characteristic patterns of example data, or training data, of different classes, and these patterns are then used to classify new inputs. This method of pattern recognition is particularly suitable for BCI application as physiological signals tend to have high inter-subject and inter-trial variability [63]. With subjectspecific training data, the supervised algorithm learns to classify each person s natural brain activity rather than expecting the subject to evoke a very particular response. In order to implement a statistical classifier employing supervised learning, the data must be processed in two steps. First, features are extracted from the data. Then these features serve as inputs to the classifier algorithm. The features used for classification of brain signals must capture the characteristic information unique to each cognitive state. Consequently, they must be selected carefully to optimize the achievable classification rates [64].

29 CHAPTER 2. BACKGROUND Feature Extraction Prior to classification, feature extraction is performed to summarize key attributes of the data and maximize the signal-to-noise ratio. To date, NIRS-BCI studies have primarily focused on classifying activity using temporal features, derived directly from the one-dimensional optical or concentrations signals measured and summarize the time-course of activity. This type of discriminatory feature often includes the mean amplitude, variance, rate of change (slope), skewness, and kurtosis of a signal, or simply the intensity measured from each channel at each time point [1], [5], [8], [15], [49], [63]. However, features derived from a series of time-linked signals do not necessarily account for the interdependence among multiple measurement locations, or the variation of hemodynamic activity over both time and space. Both these aspects of activation are exemplified in the topographic patterns within dynamic NIR topograms and may be of value in the classification of cortical activity. Features containing this type of information are referred to as spatiotemporal features, and image moments, which are features often used in image processing, may be suitable for extracting this type of information from dynamic spatial maps Image Moments Image moments are spatial integrals or summations evaluated over any region of interest of an image function. They serve to summarize the image in a more compact form, and are often used in statistical approaches for pattern classification within static images [65 67]. The number of moments extracted from an image is proportional to the degree of information captured from it, and often only a small number are required for classification applications [68]. Though, the requisite quantity of these pattern-sensitive features is dependent on the intended application as well as the type of moment utilized. A variety of moment types exist, each varying by the polynomial basis from which they are derived and consequently, this dictates the nature and degree of information each moment retains [65]. Moments fall under two broad categories,

30 CHAPTER 2. BACKGROUND 18 namely orthogonal and non-orthogonal, which can be further divided into continuous and discrete varieties. Equivalent to statistical moments in mathematics, non-orthogonal moments take the general form: where f(x,y) is the intensity distribution of the image, x and y are the pixel coordinates, and j+k is the moment order. These moments represent properties such as the mean, variance, total power, and centroid location of the image function. They can also be derived so they are invariant to linear transformations such as scale, translation, rotation and/or reflection. Typically, moment invariants are used to classify simple objects contained in static images, such as letter characters [68]. However, they have also been applied toward classifying activation patterns in fmri images. Ng et al. [20] used moment invariants to analyze cortical activity during a finger tapping exercise and were able to find regions of significant activation that traditional intensity-based methods overlooked. Building on their initial findings, Ng et al. [21] also analyzed continuous changes in moments over time to differentiate patterns of cortical activity from two different frequencies of squeezing a rubber ball. Lower-order statistical moments have commonly been used for interpreting visual information, particularly from binary images. However, if higher order moments are required, orthogonal moments may be more effective [66]. Derived from orthogonal polynomials, orthogonal moments project an image function onto an orthogonal basis and take the general form: where K is a variable specific to the type of moment, f(x,y) is the intensity distribution of the image, x and y are the pixel coordinates, m+n is the moment order, and P m (x) and P n (y) are polynomials that are determined from known coefficients. For example, orthogonal Legendre moments are calculated with P m (x) and P n (y) polynomials derived from known Legendre coefficients. Orthogonal moments produce a larger variation in magnitude for different shapes and patterns in comparison to the non-orthogonal variety, as well as a reduced sensitivity to image noise, making them better suited for pattern analysis and shape classification [66], [67].

31 CHAPTER 2. BACKGROUND 19 Continuous orthogonal moments utilize continuous polynomials as a projection basis. Though this moment type retains the benefits associated with an orthogonal basis, the continuous nature of the complex polynomials induces a higher computational load than statistical moments, rendering them unsuitable for real-time implementation. However, orthogonal moments of a discrete nature have recently been introduced, which take the form: Unlike continuous polynomials functions, which require an initial discretization prior to their application in the image coordinate space, discrete polynomials can be applied directly to an image, resulting in moment with superior computation speed and information capture [69], [70]. Though discrete orthogonal polynomials are not necessarily invariant to linear transformation, this can be addressed through an initial normalization of the image function. In addition, the choice of orthogonal polynomial dictates the specific attributes of the image function emphasized within the moments derived [69 71], which may be a useful trait for classifying hemodynamic activity. 2.6 Mental Arithmetic & the Prefrontal Cortex Activation in the PFC can be induced by a variety of higher-level cognitive tasks, including mental arithmetic [8], [14], [15], [19], [72], [73]. Though the role of this cortical region in number processing is not precisely understood, it is believed that mental stress and the involvement of working memory induced by an arithmetic task produces the heighted activity. Mental arithmetic tasks have been used successfully in a number of previous NIRS studies [8], [9], [14], [15], [19], and functional imaging studies suggest that this task may be particularly suitable for evaluating the spatial patterns of a hemodynamic response [72], [73]. Xie et al. [72] conducted an fmri study imaging cortical activation resulting from mental subtraction and found that in healthy subjects, both the intensity and spatial extension of brain activity increased with task difficulty in various regions, including the inferior and middle frontal gyri. It is believed that the increased complexity in calculations lead to an enhanced involvement of the

32 CHAPTER 2. BACKGROUND 20 subjects working memory, resulting in heightened activity in the areas of the brain involved, including the left prefrontal lobe. Burbaud et al. [73] also found that the area of significant activation (measured by the number of active pixels) in different cortical regions, including the left dorsolateral PFC, was higher during mental subtraction in comparison to a control task requiring subjects to simply think of different numbers. Consequently, significant changes in both the intensity and spatial attributes of cortical activity may be produced by a mental arithmetic task that requires participants to continually process numbers via arithmetic calculations, while retaining the previous results in memory for subsequent calculations.

33 Chapter 3 Dynamic Topographical Pattern Classification of Multichannel Prefrontal NIRS Signals This chapter describes a study investigating the ability to automatically differentiate prefrontal activity due to mental arithmetic from rest using spatiotemporal features and the value of including these features in the classification algorithm (Objectives 1 and 2). NIRS signals were measured from ten able-bodied participants during intervals of intentional mental activation and rest, and the activity was classified using four different classification methods employing various combinations of spatiotemporal and temporal features. From our analysis, we concluded that the inclusion of spatiotemporal features can significantly enhance the ability to accurately differentiate intentional mental activity from rest beyond rates achievable with temporal features alone. Different windows of the task interval were also considered, separately, for feature extraction, and optimal features for each task duration were identified. This study represents the first investigation of automatic classification of mental activity in which spatiotemporal features extracted from dynamic NIR topograms were incorporated into the classification algorithm. This chapter is being prepared for submission and publication. Sections 3.1 and 3.2 contain some repeated introductory and background content. Sections 3.3 to 3.6 consist of new content.

34 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS Abstract Near-infrared spectroscopy (NIRS) is an optical imaging technique that has recently been considered for brain-computer interface (BCI) applications. To date, NIRS-BCI studies have primarily made use of the temporal changes in brain activity to differentiate mental states. However, functional brain imaging studies have indicated that the spatial distribution of hemodynamic activity is also rich in information. Thus, the progression of a response over both time and space may be valuable to brain state classification. In this paper, we present a novel analytical method that exploits spatiotemporal information from dynamic NIR topograms for single-trial classification. The efficacy of this approach is demonstrated through a comparative analysis of four different classification schemes performed on multichannel NIRS data collected from the prefrontal cortex during a mental arithmetic activation task. Employing a linear discriminant classifier, data were analyzed using spatiotemporal features, temporal features, and a collective pool of spatiotemporal and temporal features. Lastly, we considered a majority vote combination of three classifiers; each established using one of the above feature sets. Additionally, two separate task durations (20 seconds and 10 seconds) were considered for feature extraction. With features from the longer task interval, the highest overall classification accuracy was achieved using the majority voting classifier (76.1 ± 8.4 %), which provided a significant increase (p < 10-4 ) above that obtained using temporal features alone (73.5 ± 8.5 %). While results from the shorter task duration were lower, the classifier employing only spatiotemporal features (with an average accuracy of 67.9 ± 9.3 %) did provide a significant improvement (p < 10-4 ) in average classification accuracy compared to the rate achieved using only temporal features (64.4 ± 8.4%). Collectively, these findings demonstrate the discriminative power of the proposed analytical method and suggest a value in including spatiotemporal features in the analysis of functional NIRS data for future BCI applications.

35 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS Introduction Recently, near-infrared spectroscopy (NIRS) has gained attention as a brain-monitoring modality in brain-computer interface (BCI) research for individuals with severe motor impairments. Such a pathway can be used to establish a channel of communication or environmental control, in which controlled hemodynamic activity is used to initiate a command. The user simply performs a mental task to produce characteristic patterns in their brain activity, and the induced hemodynamic response is then recognized and translated into an output. Though NIRS has been used extensively to detect functional activity from a variety of motor imagery [1 5] and higherlevel cognitive [8], [9], [11], [13 17], [33], [37], [38], [47 49], [74], [75] tasks, its application toward establishing a BCI is still in its infancy. Accuracies of automatically differentiating cortical states have ranged from 70% to 90% [1], [2], [5], [8], [9], [15], [19], [49], [51], and while results have been positive thus far, enhancements in classification rates would further the development of this technology as a practical communication option. Central to this objective is expanding upon the algorithm used to convert brain activity into an output command [27], [64], [76]. With the intention of improving the predictive performance of such algorithms for NIRS- BCIs, a novel feature genre for single-trial classification of hemodynamic brain activity is presented in this paper NIRS-BCIs & Classification To date, NIRS-BCI studies have primarily focused on classifying activity using temporal features derived from the amplitude of optical signals [8], [9], [15], [51] or hemoglobin concentrations [1], [2], [5], [16], [17], [19], [49] measured from discrete locations (or channels ). With this approach, each optical channel is treated as if it is independent from the others. However, brain activity can be both spatially and temporally correlated [77] and features derived from a series of time-linked, one-dimensional signals do not necessarily account for the interdependence among multiple sites. Considering the variation in activity over both time and space may afford a better understanding of the dynamic hemodynamic response [57], [58]. In

36 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 24 turn, including this information could potentially improve the automatic differentiation of cortical states Review of Studies Incorporating Visualization To visualize the spatial characteristics of a hemodynamic response, the composite of NIRS measurements can be used to produce two-dimensional images representing coritcal activity throughout the entire irradiated area. This is known as NIR topography [52]. Typically to generate the topographic images, the measured concentrations are first mapped to the midpoint of each channel (or the measurement point ), followed by an approximation of intermediate values using linear [3], [53], back projection [5], [7], [11], [23], [57], spline [13], [39], [56] or nearest-neighbour [58] interpolation. The resulting static NIR topograms can either be used to visualize hemodynamic activity at specific time points, or compiled into a dynamic map that exemplifies the progression of a response over both time and space. Thus far, functional imaging studies incorporating NIR topography have investigated the reliability of this modality to detect task-induced localized blood flow. As a result, the spatial and temporal trends within the topographic images have been analyzed descriptively and for the most part, qualitatively [3], [5]. In a single-subject study, Maki et al. used both static and dynamic NIR topograms to describe changes in the spatial and spatiotemporal characteristics of hemodynamic activity, respectively, resulting from a finger-opposition task [3]. The static topograms were compared to a magnetic resonance image, to confirm that the response detected by NIRS was indeed over the motor cortex. On the other hand, the dynamic spatial maps revealed global changes around the motor cortex in addition to expected regional variations exemplified in the static topograms. This latter finding suggests a potential value in evaluating both the spatial and temporal attributes of a hemodynamic response, particularly when done simultaneously through dynamic NIR topograms. Only a limited number of studies have quantified the spatial characteristics of activity detectable within NIR topograms. In a ten-subject study, Franceschini et al. estimated the extent of cortical activation using static topograms generated ten seconds following the onset of three different

37 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 25 stimulus tasks, namely finger-opposition, finger tactile stimulation and electrical median nerve stimulation [23]. The researchers found that the spatial extent and magnitude of activation varied across the three stimulation tasks, supporting the ability to quantitatively differentiate functional activity via spatial information. However, static images were only generated at a specific time point in the task interval, rather than throughout the interval, despite their own findings of temporal variability among the different tasks Image Moments To summarize the spatial and temporal progression of a hemodynamic response, features such as image moments can be extracted from dynamic NIR maps. Image moments are spatial summations evaluated over an image function, and are often used to represent the image in a more compact form for reconstruction or statistical pattern recognition [65 67]. In traditional classification applications, moments have been deployed for the comparison of content within static images, rather than for pattern evaluation across a sequence of images. Different types of image moments exist, each varying by the choice of polynomial basis from which they are derived. Consequently, this dictates the nature and quantity of information that is extracted from an image [65]. In general, image moments fall under two broad categories, namely non-orthogonal and orthogonal, which can further be divided into continuous and discrete varieties. Derived from orthogonal polynomials, discrete orthogonal moments take the general form: m = 0, 1,... N x n = 0, 1,... N y where f(x,y) is the intensity distribution of an N x by N y image, x and y are the pixel coordinates, m+n is the moment order, and P m (x) and P n (y) are two-dimensional orthogonal polynomials determined from known coefficients. Due to its orthogonality, this variety of moment has a

38 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 26 reduced sensitivity to image noise and minimized redundancy in the information captured in comparison to those derived from a non-orthogonal basis [66], [69], [70]. Thus, fewer features are required to extract the same degree of information from an image, making this moment genre more effective in pattern recognition [66]. Furthermore, discrete polynomials can be applied directly to the image; they do not require discretization prior to moment computation (as in continuous polynomials.) The initial discretization accompanying continuous polynomials involves a numerical approximation that affects the orthogonal nature of the moments, and consequently, diminishes their representative capabilities. Overall, discrete orthogonal moments are computationally efficient while affording features that are more discriminatory than those derived from a non-orthogonal and/or continuous base set [69], [70]. In particular, the Tchebichef variety provides a superior level of energy compaction and signal decorrelation, capturing a high degree of uncorrelated information in each individual moment [71]. For this reason, this moment type may be particularly well-suited for use in classification problems, where one aims to maximize the information contained in a minimal number of descriptive features [78] Motivation Though NIR topograms have been used to image hemodynamic activity, the potential of automatically identifying spatial and temporal patterns present within dynamic spatial maps has yet to be investigated. In this paper, we propose a novel approach for analyzing functional NIRS data that quantitatively extracts spatiotemporal information from dynamic NIR topograms and determine the value of including such information in classifying hemodynamic events. If such an approach can improve the ability to differentiate cortical states, it would provide an alternate method of analysis for NIRS-BCIs and increase the versatility of this communication pathway [26]. In the following sections, the proposed method of spatiotemporal feature extraction is outlined and evaluated with multichannel NIRS data collected from the prefrontal cortex during a mental arithmetic activation task.

39 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS Methods & Materials Participants Twelve able-bodied adults (4 male, mean age = 25.4 ± 4.38 years) were recruited from the staff and students at Holland Bloorview Kids Rehabilitation Hospital (Toronto, Canada) to participate in the study. Participants were self-selected and had: 1) No known neurological, cardiovascular, respiratory, psychiatric, drug- or alcoholrelated conditions that could potentially compromise the reliability of the data collected or the participants ability to follow the experimental protocol. 2) Normal or corrected-to-normal vision. Participants were asked to refrain from drinking caffeinated or alcoholic beverages, or smoking three hours prior to each experimental session. Ethics approval was obtained by Holland Bloorview Kids Rehabilitation Hospital and the University of Toronto, and written consent was obtained from all participants Instrumentation NIRS measurements were acquired using a multi-channel frequency-domain near-infrared spectrometer (Imagent Functional Brain Imaging System from ISS inc., Champaign, IL). Using a custom-made polyurethane headband, ten NIR sources and three photomultiplier detectors were secured to the participant s forehead, arranged in a trapezoidal configuration (Figure 1). Sources were paired together, each pair containing one 690nm light source and one 830nm light

40 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 28 source, in order to deliver two wavelengths simultaneously to each location. The arrangement was placed on the participant s forehead such that the centre of the configuration was aligned with the participant s nose and the bottom of the configuration sat just above the eyebrows. Only signals arising from source-detector separation distances of three centimetres were considered, as this is the accepted optimal source-detector separation for measuring light absorption in the cerebral cortex [39]. Thus, the source-detector configuration resulted in a total of 9 interrogation sites over the prefrontal cortex. Figure 3.1 Source-detector configuration. Each solid circle represents a detector, each open circle represents a source, and each X represents the measurement point for the neighbouring source-detector pair. Each source is comprised of one 690nm and one 830nm NIR light source. Light from the NIR sources was intensity modulated at a frequency of 110 MHz, delivered to the forehead, and returned to the photomultiplier detectors. Detector amplifiers were modulated at a frequency of MHz, resulting in the recording of data at a cross-correlation frequency of 5 khz. The sources were turned on and off cyclically such that only one source was on at a time, and a complete cycle consisted of one sequence through each of the spectrometer s 16 sources (only 10 of which were used). Each source remained on for 10 periods of the cross-correlation frequency, for a total duration of 2 milliseconds. To account for source switching time, the first 2 waveforms were discarded and the remaining 8 waveforms were averaged. An 8-point Fast Fourier Transform was performed on the resultant waveform to obtain frequency domain measurements (AC intensity, DC intensity and phase) at a sampling rate of Hz.

41 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS Experimental Protocol The experimental protocol has been previously reported in [16]. Briefly, participants completed five experimental sessions conducted on different days, each approximately 1.5 hours in duration. During each session, participants completed 32 trials in which they answered multiplechoice questions by performing a cognitive task (explained below). To commence a trial, participants clicked a Start Trial button. A question and 3 possible answers ( A, B and C ) were presented on a computer screen for 8 seconds. The choices were then highlighted sequentially for 20 second each, interspersed with 12 second rest periods during which none of the options were highlighted. The trial concluded with a 3 second rest for a total trial duration of 95 seconds. The timing sequence of an example trial is shown in Figure 3.2.

42 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 30 Figure 3.2 Timing diagram of example trial. At the start of the trial, participants were presented with a multiple choice question and 3 possible answers on screen for 8 seconds. Each of the 3 choices was then highlighted sequentially for 20 seconds, punctuated with a 12 second rest period during which none of the choices were highlighted. The trial concluded with a 3 second rest period. Mental arithmetic questions were presented below the multiple choice question.

43 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 31 Trials were performed in 4 blocks of 8 trials each, giving participants the opportunity to take a short break in between each block. Participants initiated the start of each block by clicking a Start Block button, which began with a 95 second rest period, followed by the 8 system-paced trials of multiple-choice questions. To answer the multiple-choice questions, participants were instructed to perform a mental arithmetic task to indicate the selection of a highlighted option. Specifically, they performed successive mental subtraction. During each of the 20 second response intervals, a 3-digit number and a smaller number (between 4 and 13) were presented on the screen below the multiplechoice question. Participants began by subtracting the smaller number from the larger, and continued throughout the interval by subtracting the smaller number from the result of the previous subtraction (e.g = 522, = 515, = 508, etc). A different pair of numbers was presented for each response interval. During any intervals when participants were not selecting an answer, they were told to refrain from performing the mental arithmetic task, but could otherwise allow any natural thought processes to occur. To limit any motion artifact in the data collected, participants were also asked to avoid movement during the trials. From the 95 second trials, only the three 20-second response intervals (when each multiplechoice answer was highlighted) were extracted and used for analysis. To ensure an equal number of mental arithmetic and rest samples were collected, only simple multiple choice questions with obvious answers were asked (e.g. Which of these is a triangle? ). A combination of questions with one, two, three or no correct answers were asked, such that a total of 48 samples of each class (i.e. activation and rest) were collected per session. For further details regarding the experimental protocol, see [16] Signal Preprocessing Optical signals collected from each channel of the NIRS system were first digitally filtered to remove physiological noise, stemming from cardiac activity (a subject-dependent value, between 0.8 and 1.2 Hz), respiration (approximately 0.3 Hz) and the Mayer wave (spontaneous fluctuations due to arteriole pulse at approximately 0.1 Hz) [14], [79]. The raw signals were

44 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 32 filtered using a third-order Tchebichef type II low-pass filter with the following specifications: a pass band cut-off frequency of 0.1 Hz, a stop band cut-off frequency of 0.5 Hz a pass-band loss no more than 6 db, and a minimum stop-band attenuation of 50 db. Following filtering, the optical signals were converted to deoxygenated, oxygenated and total hemoglobin concentrations changes, denoted [Hb], [HbO] and [thb] respectively, using the modified Beer Lambert Law: [ [ ] [ ] ] [ ] [ [ ] [ ] ] (3.2) [ ] [ ] [ ] (3.3) where ε λ,hb and ε λ,hbo represent the specific extinction coefficient of deoxygenated and oxygenated hemoglobin at wavelength λ, I λ (t o ) is the baseline intensity of light at an initial time t o, I λ (t) is the light intensity measured under activation conditions at any time t, χ λ denotes the differential path factor at wavelength λ, and d is the geometric distance between the emitter and detector. The values for the specific extinction coefficients for each wavelength and choromophore combination [80] as well as the differential path factor quantities for each wavelength [23], [81] were taken from literature, and are shown in Table 1. The mean optical intensity over the initial 95 second rest period of each block was used as the baseline intensity I λ (t o ). Following the calculation of concentration values, each signal was parsed into its 20 second response intervals and spatiotemporal and temporal features were extracted from these segments

45 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 33 Table 3.1 Values used for specific extinction coefficient (ε) and differential path factor (χ) in modified Beer- Lambert Law. Variable Value ε 690nm,Hb mm -1 cm -1 ε 830nm,Hb mm -1 cm -1 ε 690nm,HbO mm -1 cm -1 ε 830nm,HbO mm -1 cm -1 χ 690nm 6.51 χ 830nm Data Analysis To determine the value of including spatiotemporal information in the classification of cortical states, a comparative analysis of four different classification schemes was conducted. These included classifiers constructed using spatiotemporal features only, temporal features only, and a combination of spatiotemporal and temporal features. The fourth scheme combined these three classifiers using a majority vote [82]. Additionally, two different windows of the response interval were considered for feature extraction: the full response interval, i.e. the 0-20s window, and the first half of the response interval, i.e. the 0-10s window. Classification was performed separately for each time window. Consideration of the shorter window was intended for increasing the information transfer rate of the BCI in future applications.

46 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS Spatiotemporal Features A Proposed Feature Genre for NIR Topographical Pattern Classification From the composite of known hemoglobin concentrations, NIR topograms were generated via spatial interpolation. The measured concentration values were first mapped to the midpoint of each source-detector pair [36], [39] followed by a cubic interpolation to estimate intermediate values at 9 equally spaced intervals between proximate measurement points (see Figure 1). Because the original density of measured values was sparser than desired, interpolation was used to produce a finer image grid and allow for a smoother estimate of hemodynamic activity [58]. For each response interval, all pixel values within the dynamic NIR topograms were normalized to fall within a magnitude range between zero and one over the feature window of interest. This scaling ensured that any features extracted from the images would be within the same order of magnitude, regardless of inter-trial variability in NIRS signal strength due to differences in optode coupling and headgear positioning. To summarize the spatiotemporal characteristics of activation, discrete orthogonal image moments derived from a Tchebichef polynomial basis were extracted from each NIR topogram using the formula adopted from Zhu et al. [71]. For the classification problem of interest, the requisite quantity of these pattern sensitive features was determined through a progressive reconstruction of NIR topograms using an increasing number of moments. It was found that including moments through the fourth order yielded reconstructed images that were qualitatively similar to the originals, and thus, sufficiently summarized the attributes of activity at a given time point. Consequently, zero to fourth order moments were extracted from each image (i.e. for all possible combinations of m+n = [ ]), totalling fifteen image moments. The moments of a given order for each response interval were then concatenated to produce fifteen onedimensional signals that collectively summarized the hemodynamic response over both time and space. Previous studies have demonstrated that the performance of a mental task can evoke consistent spatial [3], [23] or temporal [2], [3], [14], [15] changes in hemodynamic activity. Thus, it was expected that features which encapsulate both of these aspects of a hemodynamic response

47 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 35 would also demonstrate consistent changes following the onset of an activation task. Therefore, the slope of the linear regression of each image moment signal was calculated over the feature window of interest to summarize the spatiotemporal information they contained. This produced a total of 45 spatiotemporal features for each time window (3 image types ([Hb], [HbO], and [thb]) x 15 image moments) Temporal Features For the purposes of comparison, temporal features were extracted from the one-dimensional [Hb], [HbO] and [thb] signals collected from each individual source-detector channel, similarly to [15]. Initially, each concentration signal was normalized with its own mean and standard deviation (calculated from the feature window of interest.) Subsequently, the slope of the linear regression was extracted from each normalized signal, resulting in 27 temporal features for each time window (3 signal types ([Hb], [HbO], [thb]) x 9 channels.) Feature Selection & Pattern Classification For the two-class problem of interest (i.e. differentiating hemodynamic activity resulting from mental arithmetic and rest), feature selection was performed using a Sequential Forward Floating Selection (SFFS) algorithm [83]. In this algorithm, the Fisher criterion was used as to assess the discriminative capabilities of each candidate feature set. Let x denote a feature vector within a given candidate feature pool. To evaluate the Fisher criterion, a weight vector, w, and boundary constant, b, of the discriminant function: that provided a maximum separation of classes was determined algebraically using a set of labelled training samples. With these values, the training samples were projected onto the onedimensional space defined by the discriminant function above. The projections of each of the

48 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 36 two classes were then used to evaluate the Fisher criterion of the feature set in question, computed via: where and s 2 are the mean and variance, respectively, of each class. The Fisher criterion was evaluated in this manner, rather than in the original feature space, so that higher dimensionalities could be assessed. From each candidate feature pool of interest, a four-dimensional feature set was selected by the SFFS algorithm and classification was performed using Linear Discriminant Analysis (LDA). Accuracies were evaluated for each of the four classification schemes via 25 runs of 6-fold crossvalidation. Note that accuracies were estimated separately for each session, from each participant, to mitigate any inter-session variability in participant concentration and fatigue, and optode placement and coupling. These factors can affect the strength and location of the hemodynamic response [11], and in turn, the optimal features for each session. The 25 runs of cross-validation yielded 150 classification rates, which were averaged to provide a mean classification accuracy for each session. Statistical comparisons between the four classification schemes were performed simultaneously using a mixed linear model analysis. 3.4 Results & Discussion Data from two participants were excluded prior to analysis. One female participant experienced considerable discomfort throughout all sessions, and one male participant did not fully adhere to the experimental instructions. We anticipated that these factors may have compromised the integrity of the data, and therefore these data were excluded from analysis. The results for the remaining ten participants are presented. One session from participant 3 was excluded from analysis due to discomfort she experienced during the session, and one block of data from the third session for participants four, six and eight was lost due to technical issues with the instrumentation (and therefore only 72 trials were collected).

49 Classification Accuracy (%) Classification Accuracy (%) CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS s Feature Window Across participants, the highest classification accuracies were achieved using features extracted over the 0-20s window for each of the four classification schemes. Mean classification accuracies (sensitivity, specificity) across all sessions and participants using spatiotemporal features, temporal features, spatiotemporal + temporal features, and majority voting were 74.4 ± 9.0% (74.0%, 75.0%), 73.5 ± 8.5% (74.8%, 72.3%), 74.4 ± 8.6% (75.2 %, 73.5 %) and 76.1 ± 8.4% (77.1 %, 75.2 %), respectively, where sensitivity = and specificity =, (TP = true positive, TN = true negative, FP = false positive, and FN = false negative.) Results are summarized in Figure 3.3(a); bars indicate standard error of the mean. Mean classification accuracies for each of the four classification schemes exceeded 70%, a threshold that has been deemed acceptable for effective communication with a BCI [26]. a) Overall Across-Participant Average 0-20s Feature Window * * * b) Overall Across-Participant Average 0-10s Feature Window * * * Spatiotemporal Spatiotemporal + Temporal Majority Voting Temporal Spatiotemporal Spatiotemporal + Temporal Majority Voting Temporal Classification Scheme Classification Scheme Figure 3.3 Overall across-participant average accuracies for the 4 classification schemes for (a) 0-20s feature window and (b) 0-10s feature window. Bars indicate standard error of the mean. * indicates significant difference between classifiers. Because average classification rates obtained with single classifiers (spatiotemporal, temporal, and spatiotemporal + temporal) were comparable (p for all pair-wise comparisons among the three classifiers), these findings alone do not suggest any value in utilizing the proposed

50 % of Features CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 38 spatiotemporal features. However, the majority voting classification scheme provided a significant increase over accuracies obtained by any of the other three classifiers, including the classifier established using only temporal features (spatiotemporal features: p = ; temporal features: p < ; spatiotemporal + temporal: p = ). This suggests that, in a multiclassifier approach, spatiotemporal features can add value to the differentiation of task-related hemodynamic activity. Figure 3.4 shows the frequency at which the spatiotemporal and temporal feature types were selected from the combined spatiotemporal + temporal feature pool. In general, both spatiotemporal and temporal features were chosen for classification, indicating that the resulting classifier was sufficiently distinct from the classifiers established exclusively with spatiotemporal or temporal features. Therefore, all three individual classifiers contributed unique information to the majority vote, such that this classification scheme yielded an enhanced mean classification accuracy. Frequency of Feature Types Selected from Feature Pool Containing Spatiotemporal + Temporal Features Spatiotemporal Features Temporal Features Participant # Figure 3.4 Frequency at which each feature type was selected from pool containing both spatiotemporal and temporal features over all 5 sessions for 0-20s feature window. Individual classification accuracies across all sessions for each participant are presented in Figure 3.5(a); bars indicate standard error of the mean. Although the majority voting classifier provided at most a 2.6% increase in mean accuracy across all participants, select participants (P1, P4 and P10) showed considerable increases in classification rates when spatiotemporal

51 Classification Accuracy (%) Classification Accuracy (%) CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 39 features were included in the analysis. From a qualitative inspection, these individuals showed variability in the location of prominent brain activation over the course of each experimental session. Because the spatiotemporal features utilized consider information from multiple channels throughout the measurement area, it is likely these features were still able to capture the hemodynamic changes, resulting in higher classification accuracies than those achieved using temporal features alone. Of the ten participants, seven achieved classification accuracies exceeding 70% using the majority voting classifier (P3, P4, and P8 did not), and three of these individuals (P1, P7, and P10) even exceeded 80%. In comparison, only one of these participants (P7) surpassed an average accuracy of 80% when only temporal features were used for classification. a) Individual Mean Classification Accuracies 0-20s Feature Window Spatiotemporal Features Spatiotemporal + Temporal Features Majority Voting Temporal Features b) 0-10s Feature Window Spatiotemporal Features Spatiotemporal + Temporal Features Majority Voting Temporal Features Participant # Figure 3.5 Individual participant mean classification accuracies across all 5 sessions for the 4 classification schemes for (a) 0-20s feature window and (b) 0-10s feature window. Bars indicate standard error the mean. Average spatiotemporal and temporal signals for the most frequently selected feature of each type for a single participant are shown in figure 3.6(a) and (b). Note that the features show a

52 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 40 clear difference in trends between mental arithmetic and rest over the full 20 second response interval. For the same participant and session, Figure 3.7 depicts the most frequently selected temporal features, the most frequently selected spatiotemporal feature, and a sequence of [Hb] NIR topograms for two different mental arithmetic response intervals. Note that the right side of the NIR topograms represents the left side of the participant s prefrontal cortex. Figure 3.7(a) shows a response interval collected at the start of the session, and Figure 3.7(b) shows a response interval collected toward the end of the session. In both trials, a prominent decrease in [Hb] is evident in the NIR topograms. In the response interval shown in Figure 3.7(a), this decrease in [Hb] is reflected in both the spatiotemporal and temporal signals and matches the trend typically observed during the mental arithmetic task intervals, seen in Figure 3.6(a) and (b). However, the location of prominent decrease in [Hb] changes slightly between the two trials. The earlier trial shows a sizable decrease in [Hb] within a medial-superior region (channels C2, C4 and B4), whereas the latter trial shows considerable changes within a more left-lateralized region (channels A2, C2 and C3). Because the spatiotemporal features extract information from the entire measurement area, rather than a single location, the typical hemodynamic response is still captured in the spatiotemporal signal for the latter task interval in Figure 3.7(b). However, the temporal signal during this interval does not depict the observable decrease in [Hb], and in fact exhibits a response different from the average trend.

53 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 41 Figure 3.6 Example signals of most frequently selected (a) spatiotemporal features for 0-20s feature window (b) temporal features for 0-20s feature window (c) spatiotemporal features for 0-10s feature window and (d) temporal features for 0-10s feature window. Each plot is the average of all 48 intervals corresponding to either mental arithmetic or rest. For the 0-20s feature window, both spatiotemporal and temporal features show distinct differences between mental arithmetic and rest. For 0-10s window, the spatiotemporal feature shows a greater distinction between mental arithmetic and rest than the temporal feature.

54 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 42

55 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 43 Figure 3.7 Example trials of the most frequently selected spatiotemporal feature, and the most frequently selected temporal feature and a sequence of normalized [Hb] NIR topograms for a mental arithmetic response interval at (a) the start and (b) the end of a session. NIR topograms represent [Hb] at time points five seconds apart throughout the task interval. Both intervals show a decrease in [Hb] in the sequence of NIR topograms, though the location of prominent activity differs. The spatiotemporal feature encapsulates the decreasing trend for both response intervals. The temporal feature does not demonstrate this trend in the latter interval s Feature Window Using features extracted from the 0-10s window, mean accuracies (sensitivity, specificity) for each classification scheme were as follows: spatiotemporal feature pool ± 9.3 % (66.9%, 69.0%), temporal feature pool ± 8.4 % (65.9%, 63.2%), spatiotemporal + temporal feature pool ± 10.0 % (67.6%, 67.0%), majority voting ± 9.7 % (68.5%, 67.8%). Results are summarized in figure 3.3(b). Due to the latency typically associated with a hemodynamic response (5-8 second post-stimulus) [2], [40], it is not surprising that accuracies obtained for the 0-10 second feature window were lower than those achieved using features from the 0-20 second window. As is evident in Figures 3.6(a) and (b), the change observed in hemodynamic activity is likely to be less prominent in this shorter duration in comparison to the full 20 second response interval. However, all three classification schemes that included spatiotemporal features offered significant improvements in average accuracy over the result achieved using temporal features alone (p < for all three classification schemes). These results suggest that information reflecting the spatial modulation of a hemodynamic response can enhance the ability to automatically detect smaller differences in hemodynamic activity evident within a shorter response interval. Consequently, the proposed spatiotemporal features may be particularly suitable to utilize in classification when speed is prioritized over accuracy by the BCI user. For this shorter time interval, the highest classification accuracy was achieved using the majority voting classification scheme. However, the average accuracy is not significantly higher than those achieved by the other two classifiers incorporating spatiotemporal features. The diminished performance of majority voting may be due to differential discriminative power of its component classifiers for this shortened feature window [84] (i.e. the classifier established using temporal features produced results that were significantly lower than the other two component classifiers, as demonstrated in Figure 3.3(b).) Of the three methods employing spatiotemporal features, the

56 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 44 classifier constructed using only spatiotemporal features would be most advantageous to implement as it has the smallest candidate feature pool. It is, therefore, computationally least expensive, yet still competitive with respect to accuracy. Individual classification accuracies across all sessions for each participant are presented in figure 3.5(b). For this shorter time window, five participants (P1, P2, P6, P9, and P10) achieved average classification accuracies greater than 70% using spatiotemporal features. In comparison, only three participants (P6, P9 and P10) achieved mean accuracies exceeding this threshold necessary for effective communication using only temporal features. Average spatiotemporal and temporal feature signals for the most frequently selected feature of each type for a single participant are shown in Figure 3.6(c) and (d). Note that the differences between the mental arithmetic and rest states are more prominent in the signals reflecting spatiotemporal characteristics of the hemodynamic response than signals reflecting temporal characteristics Limitations in Calculating Hemoglobin Concentrations & Generating Images To generate the NIR topograms, simplifying assumptions were made in calculating, situating, and interpolating between measured hemoglobin concentrations. By using the differential path factor in the modified Beer-Lambert Law, it is assumed that all photons travel a constant path length from source to detector through a homogeneous medium and light attenuation results solely from absorption within the tissues. Any deviations in photon path length or light attenuation resulting from the scattering of photons in the heterogeneous medium are disregarded. Subsequently, the hemoglobin concentrations calculated, and therefore the NIR topograms generated, are not necessarily quantitatively precise [36]. Under the simplification that all light absorption occurred at the midpoint of each source-detector pair, measured hemoglobin concentrations were mapped to this position in each channel [36], [39]. Though it is convention, this practice ignores any light attenuation due to scatter or

57 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS 45 interaction with extra cerebral tissue. Consequently, highly localized estimates of activity cannot be obtained. However, general trends in activity can still be detected [39]. This simplification also suggests that all light attenuation occurs at a constant cortical penetration and the resultant image represents activity at this particular depth. Despite this coarse depth resolution, it has been found that the distance travelled by photons is approximately constant when the source-detector separation exceeds 2.5 centimetres [81]. Therefore, this associated error was minimized by employing an appropriate source-detector configuration. Lastly, spatial interpolation is typically used to estimate the intermediate values between the measured hemoglobin concentrations in NIR topogram generation [3]. Because this method does not account for the behaviour or light within the tissue, the spatial sensitivity of the image is further limited. The images generated portray broadened features and an inexact estimation of the location of activity [39]. Methods which model the propagation of light through the tissue have been developed in attempts to enhance the accuracy of the estimated blood flow [56], [59], [60], [62]. However, these sophisticated techniques are computationally intensive, rendering them unsuitable for use in future online BCI applications. Nonetheless, general trends detected within images constructed using a straight-forward interpolation approach have been validated by comparing the observed hemodynamic response with results obtained with other measurement modalities [3], [11], [48]. Due to these simplifications, the NIR topograms generated for the analytical method presented in this paper are not necessarily quantitatively accurate in terms of concentration values, or exact in estimating the size and location of activation. However, establishing an optical BCI requires the detection of general patterns in activity and not necessarily a quantitatively precise representation of the hemodynamic response. Our aim was not to understand the mechanisms of functional brain activity, but rather to extract repeatable spatial and temporal patterns for distinguishing different cortical states.

58 CHAPTER 3. DYNAMIC TOPOGRAPHICAL PATTERN CLASSIFICATION OF NIRS SIGNALS Conclusion A novel analytical method for NIRS-BCI applications has been presented and evaluated. The proposed spatiotemporal features were extracted from dynamic NIR topograms for single-trial classification in a two-choice BCI system. For a longer task interval, spatiotemporal features can be used with temporal features in a multi-classifier approach employing majority voting to significantly increase classification accuracies from those achieved by individual classifiers established with either of these feature types. Alternatively, for shorter task intervals, spatiotemporal features alone can provide significant improvements in distinguishing cortical states beyond rates achievable with temporal features alone. Collectively, these findings demonstrate the discriminative power of the spatiotemporal characteristics of a hemodynamic response measured via NIRS. The analytical approach presented in this paper can be used to enhance classification rates in future NIRS-BCI applications, and thus contribute to the development of this technology as a veritable communication pathway.

59 Chapter 4 Automatic Differentiation of Prefrontal Hemodynamic Activity Due to Mental Arithmetic and Rest Using Online NIR Topographical Pattern Classification Building on the work completed in Chapter 3, this chapter describes a study investigating the feasibility of establishing an online system-paced NIRS-BCI utilizing a classification algorithm that incorporates the optimal combination of spatiotemporal and temporal features determined in the previous study (Objective 3). The online system was tested with 10 able-bodied individuals, who used mental arithmetic to intentionally modify BCI output and remained in an unconstrained rest state otherwise. To help gauge performance levels and facilitate learning, visual feedback was provided to participants. From our results, we concluded that establishing an effective online NIRS-BCI was indeed possible, and the online classification accuracies achieved were comparable to results obtain in our previous offline study (study 1). This study represents the first investigation in establishing an NIRS-BCI using online pattern classification of prefrontal cortical activity collected under a system-paced control paradigm. This chapter is being prepared for submission and publication. Sections 4.1 and 4.2 contain some repeated introductory and background content. Sections 4.3 to 4.7 consist of new content.

60 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION Abstract Near-infrared spectroscopy (NIRS) has recently gained attention as a non-invasive brain monitoring modality for brain-computer interfaces (BCIs), which may serve as alternative access pathways for individuals with sever motor impairments. For NIRS-BCIs to be used as part of a real communication pathway, reliable online operation must be achieved. The few online studies to date have not accommodated an unconstrained rest state in which the user could remain for an indeterminate period of time, precluding their practical clinical implication. Further, the potentially discriminative power of spatiotemporal characteristics of activation has yet to be considered in an online NIRS system. In this study, we developed and evaluated a system-paced NIRS-BCI admitting a no-control rest state and deploying online topographic pattern classification. With a dual-wavelength, frequency domain near-infrared spectrometer, measurements were acquired over nine sites of the prefrontal cortex while ten able-bodied participants selected letters from an on-screen scanning keyboard via intentionally controlled brain activity (using mental arithmetic). Participants were provided dynamic NIR topograms as continuous visual feedback of their brain activity as well as checkboxes as binary feedback of the BCI s decision (i.e. letter selected or not). To classify the rest and intentional activation states, temporal features extracted from the NIR signals and spatiotemporal features extracted from the dynamic NIR topograms were used in a majority vote combination of multiple linear classifiers. An overall online classification accuracy of 77.4 ± 10.5% was achieved across all participants. These results demonstrate that mental arithmetic, a combination of temporal and spatiotemporal features, and dynamic topograms yield a potent combination of mental task, signal feature and visual feedback for an online system-paced NIRS-BCI. The proposed system can provide a framework for future online NIRS-BCI development and testing.

61 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION Introduction For individuals with neuromuscular impairments, the ability to communicate or interact with the surrounding environment can be limited or non-existent. Yet, typical assistive technologies are often movement-based, and thus unusable by these individuals. Brain-computer interfaces (BCIs) constitute one approach to establish a channel of communication or environmental control without the need for muscular input [27]. In such a pathway, the user produces characteristic patterns in their brain activity, typically induced by a mental task, to indicate intent. The distinct activity is then recognized and translated by the system into a command signal, to be used, for example, for controlling an assistive device Potential for NIRS as a BCI Modality NIRS is a non-invasive imaging technique that uses near-infrared light to measure hemoglobin concentrations within biological tissue. It has been used to image cortical activity associated with a variety of mental tasks, including imagined motor movements [1 5] and higher level cognitive exercises [8], [9], [11], [13 15], [17], [19], [33], [48], [49]. This functional brain imaging capability can be exploited in an NIRS-BCI, in which a user s intentions are conveyed through a task-induced hemodynamic response. NIRS-BCI studies to date have largely focused on offline pattern classification [5], [8], [9], [15], [19], [49]. Although reliable online classification is necessary for eventual clinical application, only a limited number of online NIRS-BCI studies have appeared in the literature. With small numbers of participants, Coyle et al. [1] and Chan et al. [17] implemented online classification of cortical activity resulting from motor imagery and fast mental singing, respectively, and provided classifier feedback to participants during use of the BCI. Though, an unconstrained rest state that would permit the user to avoid a BCI output for an indeterminate period of time was not accommodated. Abdelmour and Huppert [7] detailed the automatic differentiation between left and right hemisphere activation of the motor cortex in real-time. However, activation was

62 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 50 elicited by overt finger tapping, a task that individuals with severe motor disabilities would have difficulty performing. The remaining studies involving real-time NIRS systems controlled via cognitive task-induced activation have focused on the provision of observable, online feedback rather than the automatic classification of hemodynamic activity [18], [46]. In light of the above, further development of online NIRS-BCI hinges on the judicious selection of the activation task, the classifier training scheme and control paradigm, so as to admit an unconstrained rest state while promoting effective user control. Each of these key considerations is introduced below Activation Task Central to BCI performance is the ability to detect mental states via differentiable brain signals. To achieve significant modulations in brain activity, the BCI user can either be trained to volitionally control their cortical activity as an acquired skill, or employ a specific cognitive task to produce the desired changes. The latter approach can reduce the degree of training needed to produce significant variations in brain activity, and is thus appealing for initial online BCI development and naïve user training [64]. Motor imagery is among the most popular tasks utilized for BCI control and has been used successfully in a previous online NIRS-BCI studies with able-bodied individuals [1]. However, individuals with congenital or long-term motor impairments may find it difficult or impossible to elicit a significant response in the motor cortex [42 44]. A suitable alternative is to take measurements from the prefrontal cortex (PFC), an area of the brain often persevered in the face of motor disabilities, and activated through higher-level cognitive tasks such as mental arithmetic [8], [9], [14], [15], [19], music imagery [8], [9], [15], [17] and verbal tasks [11], [13], [48]. These tasks can be more intuitive to perform in comparison to those which are motorbased, and thus, may minimize the mental effort required of the user while decreasing the necessary training requirements, both important considerations for an online NIRS-BCI. Indeed, online classification of prefrontal hemodynamics has been recently demonstrated using mental signing [17].

63 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION Classifier Training The protocol implemented for training the brain state classifier within a BCI can have profound bearing on an online system s usability. As a matter of principle, the number of samples collected for classifier training must adequately represent the true distributions of hemodynamic activity associated with each mental state so as to ensure generalization during online usage. To achieve this, training and online testing of the BCI can be carried out within a single session, or over the course of multiple sessions. In the former case, the amount of training data collected prior to testing is limited as long data collection sessions are impractical; user fatigue can degrade task performance and the strength of a hemodynamic response [11], [85]. While the multi-session alternative guarantees more training data and shorter data collection periods, poor inter-session generalization due to day-to-day hemodynamic variation becomes a significant concern [64]. However, visual feedback reflecting instantaneous hemodynamic activity can improve intersession consistency of task performance and regulation of cortical activity [86] Control Paradigm A BCI s control paradigm dictates when and how data can be analyzed for the detection of intentional brain activity (or an intentional control state ) and consequently, the necessary effort exerted by the user. Synchronous control paradigms are easiest to realize. Under this paradigm, the system is only vigilant (i.e. evaluating mental activity) during predefined intervals and the outputs generated are time-locked to these time periods. Furthermore, an intentional control state must be generated during each vigilant period, regardless of whether the user wishes to alter or maintain the system's state. This type of system is impractical for communication because it does not support a no-control state during which the user does not have to explicitly control his or her brain activity [87]. A more practical alternative would be a system that is vigilant at all times and fully supports a no-control state. This mode of operation, identified as asynchronous control, offers a more natural means of communication, but is much more challenging to realize than any other BCI paradigm [64].

64 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 52 Intermediate to synchronous and asynchronous control is a paradigm proposed by Mason et al. termed system-paced control [87], [88]. Similarly to synchronous control, a system-paced BCI is only vigilant periodically. However, during these times, the user is afforded the option of either entering an intentional-control state to generate a command, or remaining in an unconstrained rest state to avoid any output. Because the user must only volitionally control their cortical activity when an output is desired, the mental demand of this type of system is much lower than that of a synchronous BCI. System-paced control is also easier to realize than an asynchronous system, making it an appealing paradigm for a preliminary online NIRS-BCI Objectives Given the considerations above, we developed and evaluated an online system-paced NIRS-BCI admitting a no-control state and offering continuous visual feedback of hemodynamic activity. To our knowledge, this is the first study to attempt online classification of prefrontal NIRS data in a system-paced control paradigm. In particular, we sought to determine the level of accuracy achievable via online pattern classification of prefrontal hemodynamic activity corresponding to mental arithmetic and an unconstrained rest state. Furthermore, we were interested in whether or not the consolidation of training data over multiple days would enhance classifier accuracy to levels deemed acceptable for alternative communication technologies [26]. 4.3 Materials & Methods Participants 10 able-bodied adults (5 male, mean age = 26.0 ± 3.14 years) were recruited from the staff and students of Holland Bloorview Kids Rehabilitation Hospital (Toronto, Canada). Participants were self-selected according to the following criteria: normal or corrected-to-normal vision, and

65 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 53 no known neurological, cardiovascular, respiratory, psychiatric, drug-related or alcohol-related conditions that could potentially compromise the validity of the data collected or participants ability to abide by the experimental protocol. Participants were asked to refrain from drinking caffeinated or alcoholic beverages, or smoking three hours prior to each experimental session. Ethics approval was obtained from Holland Bloorview Kids Rehabilitation Hospital and the University of Toronto, and written consent was obtained prior to study participation Instrumentation NIRS measurements were acquired from the prefrontal cortex using a multi-channel frequencydomain near-infrared spectrometer (Imagent Functional Brain Imaging System from ISS Inc., Champaign, IL). Arranged in a trapezoidal configuration, ten NIR sources and three photomultiplier detectors were secured to the participant s forehead using a custom-made polyurethane headband (Figure 4.1). The arrangement was positioned such that it was centrallyaligned with the participant s nose and the bottom row of the configuration sat just above the eyebrows. Sources were paired together, each pair containing both a 690nm and 830nm light source, in order to deliver two wavelengths simultaneously to a single location. Within the configuration, only signals arising from source-detector pairs (hereafter referred to as channels ) separated by a distance of three centimeters were considered, as this is the optimal channel width for measuring light absorption in cerebral tissue [39]. As a result, optical signals were acquired from nine interrogation sites over the prefrontal cortex.

66 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 54 Figure 4.1 Source-detector Configuration. Each open circle represents a source, comprised of a 690nm and an 830 nm NIR light source. Each closed circle represents a photo detector. Each X represents a measurement point. Light from the NIR sources was intensity modulated at a frequency of 100 MHz, emitted into the forehead, and measured by photomultiplier detectors upon returning to the surface. The detector amplifiers were modulated at a frequency of MHz, resulting in the recording of data at a 5 khz cross-correlation frequency. Sources were turned on and off cyclically, such that only a single source was on at a time and a complete cycle consisted of a single sequence through each of Imagent s 16 NIR sources. Each source remained on for ten periods of the cross-correlation frequency (two milliseconds) and the signal was sampled at a rate of 20 khz. Of the ten waveforms sampled, the first two were discarded to account for the time needed to switch between sources and the remaining eight waveforms were then averaged. A four-point Fast Fourier Transform was performed on the resultant waveform to obtain a final DC intensity at a sampling rate of Hz. Due to limitations in online processing speed, the signal was down sampled for a final sampling frequency of Hz Experimental Protocol Participants completed three 1.5 hour experimental sessions, each on a different day. For each session, participants completed a series of trials during which they guessed letters using a mental arithmetic task (explained below) to reveal a blank word puzzle. An example of the visual interface used in the experimental sessions is shown in figure 4.2 and the timing sequence of an example trial is shown in Figure 4.3. To initiate each trial, participants clicked a Start Trial

67 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 55 button on the visual interface. This action prompted a screen to appear, on which the blank word puzzle and three letters to be presented. From these letters, participants were asked to manually preselect the letters of choice by clicking on a button below each letter. Within each trial, the number of selectable letters was restricted to ensure the collection of an equal number of intentional control and no-control response intervals. For a given trial, it was possible to select between zero and three letters. Figure 4.2 Annotated visual interface. Participants were presented with the timing cues of the experimental paradigm, mental arithmetic questions, and feedback (continuous and discrete) on a visual interface.

68 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 56 Figure 4.3 Timing diagram of example trial. Prior to the start of each trial, participants were presented with a word puzzle and 3 letters. After manually preselecting between 0 and 3 of the letters, the 3 letters were displayed on a new screen. The letters were then highlighted sequentially, punctuated with periods of rest. Mental arithmetic questions were presented below the row of letters.

69 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 57 Once manual letter selection was finalized, participants clicked an OK button. A new screen appeared where the same three letters were displayed, each with a checkbox appearing below, and letters that had been manually preselected were underlined. Following a five second rest period, each letter was highlighted in turn for 20 seconds, with 12 second rest periods imposed between letter highlights. Continuous feedback in the form of an updated NIR topogram representing oxygenated hemoglobin was presented during each 20 second response. Images were generated using the methods outlined in Chapter 3. Following each response interval, the binary classification decision was presented in the checkbox below the most recently highlighted letter at an utmost delay of 5 seconds. The checkbox was either marked with an X if mental arithmetic was detected or remained blank otherwise. Continuous and discrete varieties of feedback were provided as both have proven to be beneficial in acquiring learnt control of one s hemodynamic activity [89], [90]. Additionally, the BCI accuracy (number of response intervals correctly classified / total number of response intervals) was presented on the interface and updated after each classification. The trial concluded with an eight second rest interval, resulting in a total trial duration of 100 seconds. Upon initialization of the subsequent trial, previously selected letters were revealed in the word puzzle and the next three letters were presented. Each session consisted of 32 trials of letter selection, yielding a total of 96 response samples per session (32 trials x 3 samples per trial). Trials were performed in four blocks of eight trials each and an equal number of samples from both the intentional control and no-control state were collected within each block (i.e. 12 samples per class per block). Participants initiated each block by clicking a Start Block button, which began with an initial 60 rest period followed by eight trials of letter guessing. Participants were afforded the opportunity to take a short break between blocks. After finishing each session, participants were asked to complete a short questionnaire rating their experience with the BCI Activation Task Mental Arithmetic To indicate a letter selection, participants elicited an intentional control state by performing a mental arithmetic task, namely successive mental subtraction. For each 20 second response

70 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 58 interval during which a letter was highlighted, an arithmetic question was presented below the row of letters on the user interface. The question was always the subtraction of a small number (between 4 and 14) from a three-digit number. If the highlighted letter was to be selected, participants successively subtracted the smaller number from the previous difference (e.g = 288, = 280, = 272, etc) for the duration of the response interval. A different pair of numbers was presented for each response interval. During any interval when participants were not selecting a letter, they refrained from performing the mental arithmetic task but were otherwise permitted to rest and allow natural thoughts to occur. In addition, participants were explicitly instructed to avoid any movement, gross (e.g., adjustments in posture) or fine (e.g., furrowing of the brow) during the trials to mitigate signal contamination with motion artifacts Offline and Online Data Collection Session one data were collected offline and were exclusively used for classifier training. Prior to online classification in sessions two and three, the BCI s statistical classifier was trained with a combination of data from session one and the current test session (Figure 4.4). In session 2, data from the first block were combined with data from session one for classifier training, yielding a total of 120 training samples (96 samples from session one + 24 samples from either session two). The same held true for session three. In this way, for each session that the classifier was tested online, it was exposed to some session-specific training data. As shown in Figure 4.4, data from the remaining three blocks of the session (72 samples) were classified online. Throughout the offline portions of each experimental session, letters preselected by the participants were shown in the word puzzle. However, during the online blocks, only the letters selected using the BCI (i.e. intervals which the intentional control state was identified) were revealed.

71 CHAPTER 4. ONLINE NIR TOPOGRAPHICAL PATTERN CLASSIFICATION 59 Figure 4.4 Classifier training and testing protocol. All samples collected in the first experimental session plus the samples collected in the first block of session 2 or 3 were used as training samples. Classifier training was conducted between blocks 1 and 2 of sessions 2 and 3, and the samples collected from the remaining 3 blocks of the session were classified online. 4.4 Data Analysis Preprocessing Throughout each response interval, the effects of physiological noise, typically due to cardiac activity ( Hz), respiration ( Hz) and the Mayer Wave (approximately 0.1 Hz), were removed by digitally low-pass filtering each optical intensity in real-time using a thirdorder Tchebichef infinite impulse response (IIR) cascade filter. The filter was designed with a low frequency cutoff of 0.1 Hz, a high frequency cutoff of 0.5 Hz, and a passband ripple of 0.1 db. Following filtering, the optical intensity of each channel was converted to oxygenated, deoxygenated, and total hemoglobin concentrations changes, denoted [Hb], [HbO] and [thb] respectively, using the modified Beer Lambert Law: [ [ ] [ ] ] [ ] [ [ ] [ ] ] (4.1) [ ] [ ] [ ] (4.2)