Flashing color on the performance of SSVEP-based braincomputer. Cao, T; Wan, F; Mak, PU; Mak, PI; Vai, MI; Hu, Y

Transcription

1 itle Flashing color on the performance of SSVEP-based braincomputer Interfaces Author(s) Cao, ; Wan, F; Mak, PU; Mak, PI; Vai, MI; Hu, Y Citation he 34th Annual International Conference of the IEEE EMBS, San Diego, CA., 28 August-1 September In IEEE Engineering in Medicine and Biology Society Conference Proceedings, 2012, p Issued Date 2012 URL Rights IEEE Engineering in Medicine and Biology Society. Conference Proceedings. Copyright Institute of Electrical and Electronics Engineers.; 2012 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.; his work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

2 34th Annual International Conference of the IEEE EMBS San Diego, California USA, 28 August - 1 September, 2012 Flashing Color on the Performance of SSVEP-based Brain- Computer Interfaces eng Cao, Feng Wan, Peng Un Mak, Pui-In Mak, Mang I Vai, Yong Hu Abstract A critical problem in using steady-state visual evoked potential (SSVEP) based brain-computer interfaces (BCIs) for clinical and commercial use is the visual fatigue the user may suffer when staring at flashing stimuli. Aiming at the design of user-friendly BCIs with satisfactory performance, this work is to preliminarily investigate how different colors influence the SSVEP (i.e. frequency or phase) and system performance. he results show that white stimuli can lead to the highest performance, followed by gray, red, green and blue stimuli. I. INRODUCION Steady-state visual evoked potential (SSVEP) based on brain-computer interfaces (BCIs) have received much attentions in recent decades. SSVEP can be considered as a steady periodic response to a repetitive visual stimulus at frequencies higher then 6Hz. An SSVEP based BCI enables the users to select among several commands by gazing at different visual stimuli encoded with different properties (e.g. frequency and phase). Compared with other BCI techniques, SSVEP based BCIs have some advantages including high information transfer rate (IR), good stability and little training [1]. Although the SSVEP based BCIs have been investigated by many research groups through large numbers of laboratory demonstrations, it still has lots of challenges in clinical and commercial use. A critical problem is the visual fatigue the user may suffer when staring at flashing stimuli in SSVEP based BCIs. Visual fatigue will make the users uncomfortable when operating this system and consequently reduce the performance. In order to alleviate the fatigue and make the user comfortable, the visual stimulator with monochrome color should be improved (e.g. cartoon pictures). herefore, this work is to preliminarily explore how different colors influence the BCI performance. he perception of light for human eye is through the retina, which contains two kinds of light-sensitive cells: rod and cone cells. he rod cells have response on luminance but cannot distinguish different colors. he ability of the human eye to distinguish colors is based upon the varying sensitivity of cone cells to the light of different wavelengths. here are three kinds of cone cells in the retina: red cones, green cones and blue cones. herefore, human eyes will have different perception to different colors; as a result, different colors of the stimuli will elicit different SSVEPs. At present, green, he authors are with the Department of Electrical and Computer Engineering, Faculty of Science and echnology, University of Macau, Av.Padre omás Pereira, aipa, Macau, China. ( ct.tengcao@gmail.com) /12/$ IEEE 1819 red, gray, black and white stimuli have been used for SSVEP based BCIs [2-6]. It is reported in [7] that red, yellow and blue stimuli have different effects on the SSVEPs. In the case of 10 Hz stimulation frequency, red stimuli can elicit the strongest responses. he strength of the elicited SSVEPs by the blue stimuli is lower than red stimuli and higher than yellow stimuli. he power intensity of SSVEPs elicited by red, green and blue stimuli is investigated in [8]. It is reported that the SSVEPs elicited by blue color has the highest power intensity and red color is lower than blue color but higher than green color. In [9], the results show that the composed colors can elicit higher amplitude SSVEPs than the monochrome colors. However, in [7-9] the authors only explored how the different colors influence the amplitude and frequency of SSVEPs, but the phase of the SSVEPs and system performance have not been evaluated. Since SSVEP is also phase locked to the visual stimuli, the phase information has been encoded in the stimuli to increase the number of targets when presenting the stimuli on the LCD/CR monitors [1], [10-12]. Furthermore, use both the frequency and phase information to enhance the classification accuracy and speed as well. herefore, five colors: red, green, blue, gray and white are selected to explore how different colors influence the SSVEPs (i.e. frequency and phase) and system performance are investigated in this paper. II. MEHOD A. Feature Extraction he frequency and phase information are both used to generate visual stimuli in our experiments. he frequency feature is normally extracted by fast Fourier ransform (FF) [4], which is based on one channel SSVEP data. Since canonical correlation analysis (CCA) belongs to a multivariable statistical method, it is a promising method to extract the frequency feature for multichannel SSVEP based BCI. CCA is employed to compare the EEG signals with reference signals which have been preinstalled in terms of sine and cosine expressions to calculate the CCA coefficients. Consider the multi-channel EEG signals X, the reference signal Y and their linear combinations x X W x and y Y W y. CCA finds the weight vectors, W x andw y, which maximize the correlation between x and y, as shown in (1). More details can be found in [13]. he higher CCA coefficient means the SSVEP signal has higher correlation with the stimuli signals. Moreover, the system discrimination accuracy and discrimination speed will be enhanced by the higher CCA coefficients.

3 max p( X, Y ) Wx, Wy E[ X Y ] E[ X X ] E[ Y Y ] E[ W XY W ] x E[ W XX W ] E[ W YY W ] x x y y y (1) In addition, the phase angle of SSVEP is extracted by FF according to (2) and (3). For instance, the SSVEP signal evoked by w (rad/s) stimulus signal so that its phase angle 0 ( x) can be calculated. When extracting the phase feature from SSVEP signal, smaller phase variance will enhance its both classification accuracy and speed. With the large phase variance, the adjacent phase values will be overlapped with each other. Figure 1. he distribution of five flickers in the visual stimulator for offline experiments. 1 ( x) tan [Im{ X ( w )} / Re{ X ( w )}] 0 0 (2) X ( w) FF ( x) (3) B. Experiments Setup In our experiments, an LCD monitor was used as the visual stimulator (ViewSonic 22, refresh rate 120 Hz, pixel resolution). Five colors: white (255, 255, 255), red (255, 0, 0), green (0, 255, 0), blue (0, 0, 255) and gray (128, 128, 128) were included in the experiment and the visual stimulator was programmed in Microsoft Visual C and DirectX DirectDraw 7. he background color was kept as black color and the phase value of all flickers were predefined as zero degree. Due to the 120 Hz refresh rate of the LCD monitor, Hz, 15 Hz, Hz, 12 Hz, 10.9 Hz, 10 Hz, 9.23 Hz, 8.57 Hz, 8 Hz and 7.5 Hz were selected in the experiments to explore how different colors influenced the CCA coefficient and phase of SSVEPs. he distribution of flickers in the visual stimulator was shown in Fig. 1. Five frequencies were modulated in these five flickers in each experiment. We first collected the EEG data for one group of five frequencies with white color and then repeated the experiment for another group of five frequencies with white color. And then we repeated the experiments for other four colors and the time interval between two groups of experiments was 5 min. During the experiments, the stimulus was flickering for 6s for target identification and the following 4s interval was left for the subject to shift his gaze to next target. he subjects were asked to gaze flicker 1 to 5 in turn and repeated this procedure for another two times, so the total time for one group of experiment was 150 sec. Each subject would do this whole experiment for three times. hen four frequencies with higher CCA coefficients and lower phase variance would be selected to encode the stimuli to do the online experiment in testing the system performance. he distribution of flickers was shown in Fig. 2. In each line of the flickers, the frequencies were the same only the phases were different. he differences between adjacent phases should be chosen as large as possible which depended on the selected frequencies. Figure 2. he distribution of sixteen flickers in the visual stimulator for online experiments. Five subjects (aged from 22 to 28 years old, 3 males and 2 females) with normal or corrected-to-normal vision participated in the experiments and were selected from the subjects with BCI experience. he subjects had been tested by Ishihara test for color blindness and all of them were proved to be with normal color vision. he subjects were seated in a comfortable chair in front of the visual stimulator with a distance of about 60 cm. 6 standard EEG electrodes placed on PO Z, P 3, P 4, O Z, O 1, and O 2 were used as input channels. EEG signals were collected by an amplifier (g.usbamp, Guger echnologies, Graz, Austria) and were filtered by 0.5Hz to 60Hz band-pass filter and sampling rate was 600Hz. III. RESULS Fig. 3 shows the average CCA coefficients of all five subjects. It is obvious that the SSVEPs elicited by white color have the highest CCA coefficients in all frequency range. In the frequency range from 8 Hz to 15 Hz for white color, the standard deviation between different frequencies is 0.03 which is very small. he CCA coefficients of 7.5 Hz and Hz are the lowest two for white color, but still larger than other colors. he CCA coefficients of SSVEPs for gray color are smaller than white color in all frequency range. he standard deviation between different frequencies from 8 Hz to 15 Hz is about 0.04 and CCA coefficients of 7.5 Hz and Hz have around 0.1 differences from the average value for gray color. For the red color, it has similar distribution as gray color from 10.9 Hz to Hz. When the frequencies are lower than 10.9 Hz, the CCA coefficients for red color are smaller than those for gray color. he standard deviation between different frequencies for green color is 0.03 in all frequency range. When the frequencies are lower than 10 Hz, red, green and blue colors have the similar distribution. For the frequencies larger than 10Hz, the CCA coefficient 1820

4 Figure 3. CCA distribution due to different frequencies. Figure 4. Phase variance distribution due to different frequencies. differences between the surrounding frequencies are large for blue color. he lowest CCA coefficients for blue color are also located in this range. herefore, white color has the highest CCA coefficient for all frequencies; from 7.5 Hz to 12 Hz, the second choice is gray color and then the rest three colors; from 12 Hz to Hz, blue color has the lowest CCA coefficient and the rest three colors can be selected as the second choices to encode the visual stimuli. Fig. 4 presents the average phase variance of all five subjects. White color still has the lowest phase variance and the standard deviation between different frequencies from 8 Hz to 15 Hz is 0.69 degree. It seems the 7.5 Hz and Hz are not good than other frequencies both for CCA coefficient and phase variance. Red color has the lowest phase variance in 8 Hz and Hz, but higher in other frequencies. From 8 Hz to 12 Hz, the standard deviation between different frequencies for green color is 2.12 degree and the 15 Hz also has the low phase variance. he standard deviation among surrounding frequencies for blue color is very large, so blue color has the highest sensitivity for the used frequencies. he gray color has lower sensitivity for the used frequencies higher than 8.5 Hz. herefore, white color is still the best choice to encode the phase information into the visual stimuli. Since blue color is very sensitive to the used frequencies, it is the worst color to encode the phase information. For Hz and17.14 Hz, green color is better not to be selected. For other frequencies, green color is better than red color but worse than gray color. hen the average CCA coefficient and average phase variance of all ten frequencies are calculated due to different color and the results are demonstrated in Fig. 5 and Fig. 6. In Fig. 5, the largest average CCA coefficient is obtained by white color which is 0.56±0.06. he CCA coefficient of gray color is 0.44±0.06 which is lower than white color but higher than others. For the red, green and blue colors, blue has the lowest CCA coefficient of 0.34 and green has the lowest standard deviation of CCA coefficient of 0.03 respectively. Since white color has highest CCA coefficient, it means the 1821 white color can elicit the SSVEPs with highest correlation with the visual stimuli. herefore, the SSVEPs elicited by white color visual stimuli will be detected with higher accuracy and higher speed. Furthermore, the system performance IR in terms of accuracy, speed and number of targets will be enhanced with the same number of targets. In addition, the red color has the highest standard deviation of the CCA coefficient and green color has the lowest standard deviation of the CCA coefficient. Hence, the CCA coefficients of SSVEPs elicited by red color visual stimuli are more depend on the used frequencies and less depend on the used frequencies for the green color. In Fig. 6, the lowest phase variance is also obtained by white color which is 4.75±1.59 degree. he phase variance of gray color is higher than white color and lower than others. he phase variances of SSVEPs for red and green colors are similar which are 11.40±5.73 degree and 11.88±8.41 degree respectively. he blue color has the highest phase variance of 16.79±5.82 degree. he white color has the lowest standard deviation of 1.59 degree and green color has the highest standard deviation of 8.41 degree. herefore, white color is still the best choice to encode the phase information into the visual stimuli, because of its small phase variance and small standard deviation of phase variance. he phase variances of SSVEPs for green color are more depend on the used frequencies than other colors. Since blue color has the highest phase variance, the accuracy will be affected. When use blue color as the stimuli to encode the phase information, the adjacent phase values will conflict with each other. herefore, when encode the phase information in the visual stimuli; it is better to choose the color which elicits the SSVEPs with lowest phase variance as well as the smallest standard deviation of phase variance. From the CCA coefficient and phase variance of SSVEPs for different colors, it seems that the white color is the first choice to be used to generate the visual stimuli and the gray color is the second choice. he results of red, green and blue are not good as white and gray colors. he reason of why white and gray colors are better than red, green and blue colors may be related to principle of the perception of light of Figure 5. CCA coefficient distribution due to different colors. Figure 6. Phase variance distribution due to different colors.

5 human eyes. As mentioned, the red, green and blue colors are three basic colors which can only elicit the red, green and blue cone cells respectively. For the white and gray color, they can simultaneously elicit three types of cone cells. As a result, more intense SSVEP signals will be appeared on the occipital lobe. herefore, the SSVEPs elicited by the white and gray colors are stronger than red, green and blue colors. Compared with white color and gray color, the white color has larger contrast ratio than of the gray color under the same background color black. Hence, the white color elicits the better SSVEPs than the gray color. hen 10.9 Hz, 12 Hz, Hz and 15 Hz, with four different phases for each, are selected to encode the stimuli to test the system performance. ABLE I presents the average accuracy and average IR over all five subjects. Use white color as the visual stimuli, the average accuracy is 96.25±4.52 % and average IR is 36.61±3.88 bpm respectively. hen the gray color achieves average accuracy of 92.92±7.88 % and average IR of 34.20±6.10 bpm. he accuracy is decreased by 3.46 % and the IR is decreased by 6.58 % when use gray color instead of white color. For the red, green and blue colors, the accuracy reduced by 7.79 %, 9.09 %, % and the IR reduced by %, %, % compared with the result of white color. It is obvious that, use white color as the stimuli will lead to the best system performance, followed by gray, red, green and blue colors. ABLE I. Accuracy (%) IR (bpm) AVERAGE BCI PERFORMANCE DUE O DIFFEREN COLORS White Gray Red Green Blue 96.25± ± ± ± ± ± 7.31 IV. CONCLUSION 87.50± ± ± ± 8.92 In this paper we have investigated how different colors influence the SSVEPs and the BCI system performance. he motivation of this work is to improve the visual stimuli in SSVEP based BCIs to alleviate the visual fatigue in operating the system. hrough the experiments, the following results are obtained: Firstly, SSVEPs elicited by white color has the highest CCA coefficient. he CCA coefficients of SSVEPs elicited by red color visual stimuli are more depend on the used frequencies and the green color is less depend on the used frequencies. Secondly, white color is still the best choice to encode the phase information. he blue color elicits the SSVEPs with highest phase variance which will make the adjacent phase values overlap with each other. hirdly, the white color achieves the highest accuracy of 96.25±4.52 % and IR of 36.61±3.88 bpm, followed by gray, red, green and blue colors. Future work may include quantitative description of the user visual fatigue when using SSVEP based BCIs, that is, to use an index to indicate the visual fatigue. In addition, to make the flashing stimuli to be more user-friendly, cartoon pictures instead of monochrome color can be considered for more comfortable user experience. ACKNOWLEDGMEN his work was supported in part by the Macau Science and echnology Development Fund (Grant FDC/036/2009/A) and the University of Macau Research Fund (Grants RG059/08-09S/FW/FS, RG080/09-10S/WF/FS and MYRG139(Y1-L2)-FS11-WF). REFERENCES [1] Y. J. Wang, X. R. Gao, B. Hong, C. Jia, and S. K. Gao, Brain- Computer Interfaces based on Visual Evoked Potentials, IEEE Eng. Med. Biol. Mag., vol. 27, no. 5, pp , September [2] D. H. Zhu, J. Bieger, G. G. Molina, and R. M. Aarts," A Survey of Stimulation Methods Used in SSVEP-Based BCI system," Comput. Intell. Neurosci., pp , [3] X. R. Gao, D. F. Xu, M. Cheng, and S. K. Gao, A BCI-based Environmental Controller for the Motion-Disabled, IEEE rans. Neural Syst. Rehabil. Eng., vol. 11, no.2, pp , June [4] G. R. Müller-Putz, R. Scherer, C. Brauneis, and G. Pfurtscheller, Steady-State Visual Evoked Potential (SSVEP)-based Communication: Impact of Harmonic Frequency Components, J. Neural Eng., vol. 2, no. 4, pp , [5] R. Wahnoun, R. Saigal, Y. Gu, N. Pacquet, S. DePauw, A. Chen, S. Sami, K. D. Nielsen, A Realtime Brain-Computer Interface based on Steady-State Visual Evoked Potentials, Proc. of the 7th Annual Int. Functional Electrical Stimulation Society Conf., vol. 2, pp. 2-4, [6] Y. J. Wang, R. P. Wang, X. R. Gao, B. Hong, and S. K. Gao, A Practical VEP-based Brain-Computer Interface, IEEE rans. Neural Syst. Rehabil. Eng., vol. 14, no. 2, pp , June [7] D. Regan, An Effect of Stimulus Colour on Average Steady-State Potentials Evoked in Man, Nature, vol. 210, no. 5040, pp , [8] Z. H. Wu and D. Z. Yao, Comparison of Steady-State Visually Evoked Potential Evoked by Different Monochromatic Light (In Chinese), J. Biomed. Eng., vol. 25, no. 5, pp , October [9] M. L. Huang, P. D. Wu, L. Z. Bi and Y. Liu, On Applying Flash Visual Evoked Potential in Brain-Computer Interface (In Chinese), Computer Application and Software, vol. 26, no. 11, pp , November [10] P. L. Lee, J. J. Sie, Y. J. Liu, C. H. Wu, M. H. Lee, C. H. Shu, P. H. Li, C. W. Sun and K. K. Shyu, An SSVEP-Actuated Brain Computer Interface Using Phase-agged Flickering Sequences: A Cursor System, Ann. Biomed. Eng., vol. 38, no. 7, pp , July [11]. Kluge and M. Hartmann, Phase Coherent Detection of Steady- State Visual Evoked Potentials: Experimental Results and Application to Braincomputer Interfaces, Proc. of 3rd Int. IEEE EMBS Conf. on Neural Engineering, pp , May [12] C. M. Wong, B. Y. Wang, F. Wan, P. U. Mak, P. I. Mak and M. I. Vai, An Improved Phase-agged Stimuli Generation Method in Steady- State Visual Evoked Potential Based Brain-Computer Interface, Proc. of 3rd Int. IEEE EMBS Conf. on Biomedical Engineering and Informatics, pp , October [13] G. Y. Bin, X. R. Gao, Z. Yan, B. Hong, and S. K. Gao, An Online Multi-Channel SSVEP-based Brain-Computer Interface Using a Canonical Correlation Analysis Method, J. Neural Eng., vol. 6, no. 4, pp , June