IEEE. Proof. IN the last decade, much success has been achieved in de-

Transcription

1 TRANSACTIONS ON NEURAL NETWORKS 1 Decoding Stimulus Reward Pairing From Local Field Potentials Recorded From Monkey Visual Cortex Nikolay V. Manyakov, Rufin Vogels, and Marc M. Van Hulle, Senior Member, AQ:1 Abstract Single-trial decoding of brain recordings is a real challenge, since it pushes the signal-to-noise ratio issue to the limit. In this paper, we concentrate on the single-trial decoding of stimulus reward pairing from local field potentials (LFPs) recorded chronically in the visual cortical area V4 of monkeys during a perceptual conditioning task. We developed a set of physiologically meaningful features that can classify and monitor the monkey s training performance. One of these features is based on the recently discovered propagation of waves of LFPs in the visual cortex. Time frequency features together with spatial features (phase synchrony and wave propagation) yield, after applying a feature selection procedure, an exceptionally good single-trial classification performance, even when using a linear classifier. Index Terms Decoding, local field potentials, phase synchrony, visual cortex, wave propagation. 17 I. INTRODUCTION IN the last decade, much success has been achieved in de- 19 coding neural activity for motor control or motor planning, 2 for example for controlling a prosthetic device [1] [3]. To 21 a much lesser extent, progress has been made in decoding 22 visual information (for example, [4], [5]) and higher level 23 cognitive processes that organize behavior [6], [7]. In order 24 to achieve real-time decoding performance, for example to 25 control a prosthetic device, single-trial decoding is essential 26 and averaging over several trials is not an option. Manuscript received July 6, 29; revised June 25, 21 and September 7, 21; accepted September 8, 21. The work of N. V. Manyakov was supported in part by the European Commission under Grant IST The work of R. Vogels was supported in part by the Human Frontier Science Program under Grant RGP 18/24, the Queen Elisabeth Medical Foundation of Belgium, the Interuniversity Attraction Poles Program Belgian Science Policy under Grant IUAP P6/54, the Flemish Regional Ministry of Education (Belgium) under Grant GOA 1/19, the Belgian Fund for Scientific Research Flanders, and the Excellence Financing Program under Grant EF 25. The work of M. M. V. Hulle was supported in part by the Excellence Financing Program under Grant EF 25 and the CREA Financing Program under Grant CREA/7/27 of the Katholieke Universiteit Leuven, the Belgian Fund for Scientific Research Flanders under Grant G and Grant G.588.9, the Interuniversity Attraction Poles Program Belgian Science Policy under Grant IUAP P5/54, the Flemish Regional Ministry of Education (Belgium) under Grant GOA 1/19, and the European Commission under Grant STREP , Grant IST , and Grant IST The authors are with the Laboratorium voor Neuro- en Psychofysiologie, Katholieke Universiteit Leuven, Leuven 3, Belgium ( NikolayV.Manyakov@med.kuleuven.be; Rufin.Vogels@med.kuleuven.be; Marc.VanHulle@med.kuleuven.be). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier 1.119/TNN A second complication, specifically for chronic intracranial 27 recordings, is that action potentials (spikes) are often lost 28 even after a few of recording due to cell expiration, 29 inflammation, reactive gliosis, and scarring. As an alternative, 3 local field potentials (LFPs) are often used. They represent the 31 summation of excitatory and inhibitory dendritic potentials 32 [8]. LFPs are more robust against signal degradation and, in 33 contrast to spikes, they can be easily recorded over longer 34 periods (chronic recordings). Using LFPs, equal performance 35 as, or even a slightly better performance compared to, 36 decoders based on action potentials has been achieved in 37 brain machine interfacing [9] [13]. 38 In this paper, we concentrate on the single-trial decoding 39 of LFP activity, chronically recorded with a microelectrode 4 array implanted in the visual area V4 of two macaque mon- 41 keys. Since we consider recordings over several months, we 42 concentrate on LFPs as the input to our decoder. In particular, 43 we consider event-related LFPs recorded in response to the 44 following two oriented stimuli (Fig. 1), one for which the 45 monkeys receive a fluid reward, and another not (i.e., a 46 classical conditioning paradigm). In Fig. 2 and, two 47 traces of LFPs (single trial) are shown (blue and green curves) 48 for the rewarded (left figure) and the unrewarded (right figure) 49 stimuli, for training day 36 and electrode #8 of monkey 1, as well as the average response (red curves in left and 51 right figures), for the same electrode, and averaged over the 52 same training day. We observe that, as expected, the trial-to- 53 trial variability in the LFP responses is large [see Fig. 2(c)], 54 but we have found that the average response does not vary /$ greatly from day to day, at least after some training have 56 elapsed [compare the averaged traces of the recordings for the 57 rewarded and unrewarded stimuli for training 36 and in Fig. 2 (c), for monkey 1]. 59 As is common in the analysis of LFPs, as well as in 6 electroencephalogram (EEG) studies, several trials need to be 61 considered for averaging before the stimulus-related response 62 can be detected. When the signal of interest is large, fewer 63 trials for averaging will be needed. As was shown in [14], 64 which details the conditioning training of the experiment 65 considered in this paper, an early neural response was observed 66 that was suggested to be related to sensorial learning, and 67 a later one to attentional learning. Based on this, temporal 68 features are developed in this paper, and supplemented with 69 a number of spatial features that serve as the input to the 7 classifier whose aim is to distinguish the rewarded from 71

2 2 TRANSACTIONS ON NEURAL NETWORKS amplitude (µv) amplitude (µv) Fig. 1. Examples of rewarded and unrewarded stimuli. the unrewarded stimulus responses. We target for single-trial classification, since it pushes the signal-to-noise ratio issue to the limit (Fig. 2), and since it is a requirement for the future development of a decoder performing in real time. Below, we will first describe the details of the experimental procedure, the implantation of the microelectrode array, and the signal conditioning. Then, we will develop three types of features for the classifier, wavelet-based ones for the individual electrodes (temporal features), and two types of spatial features phase synchrony between electrode pairs and wave propagation in the array. The latter one is inspired by the recently discovered propagation of waves of LFPs in the visual cortex [15]. Next, we perform feature selection so as to optimize our set of features, and apply a linear classifier. We show the classification performance and the contribution of the different types of features therein. We consider the recordings made for two monkeys and verify that the obtained decoding performances are similar. II. EXPERIMENT Two rhesus monkeys were implanted with a Utah array into the prelunate gyrus (area V4). The array measures 4 4 mm and consists of 1 1 electrodes (4 of them are wireless) with.4 mm distance between them, 8 µm in diameter, 1 mm long, and with tips metalized with platinum. Recordings were made with a Bionic Cerebrus system. The LFP signals were obtained by filtering the recorded signals between.3 and 2 Hz. The stimuli were obliquely oriented sinusoidal gratings (2 c/deg, diameter size 4 visual angle for monkey 1 and 4 for monkey 2) with phases randomized across presentations (Fig. 1), and superimposed on a sinusoidal noise background. The gratings were partially occluded by sinusoidal noise, during conditioning (see further), the signal-to-noise ratio was 2%. During training, a different sinusoidal noise background, which filled the display, was presented every ms. At random intervals, a sinusoidal grating was presented for ms. The grating orientation could be either for monkey 1 (112.5 for monkey 2) for the rewarded stimulus or 67.5 for monkey 1 (22.5 for monkey 2) for the unrewarded stimulus. The presentation of the (112.5 ) grating was paired with a fluid reward, while no reward was given at any other time instant (classical conditioning). The reward was given 4 ms after presentation of the grating pattern and, thus, partially overlapping with the grating presentation. For convenience, we will further refer to these stimuli as the rewarded and unrewarded ones. Each monkey was trained to fix a small dot on the screen ( fixation window), amplitude (µv) amplitude (µv) (c) (d) Fig. 2. and Example of event-related LFP responses (blue and green curves), for rewarded (left panel) and unrewarded (right panel) stimuli, recorded with electrode #8 for day 36 of training for monkey 1, for 3 ms after stimulus onset. Red curves correspond to the average LFP within the same day. (c) Average rewarded (blue) and unrewarded (red) responses with error bars (standard deviations) from electrode #8 for day 37 of training for monkey 1. (d) Example single-trial recordings from all 96 electrodes of the array for the last day before the reversal for monkey 1 (day 37). Two distinct groups of recordings are clearly visible, based on the signal shape and the within group synchrony. 1 1 and the reward was given only when the monkey maintained 12 fixation during stimulus presentation (the monkeys eye 121 movements were monitored using a noninvasive infrared eye 122 tracking device: Iscan, Burlington, MA). The gratings were 123 always preceded by 1 6 noise backgrounds. The stimuli were 124 presented 7.2 eccentric in the right lower visual field for 125 monkey 1 (foveally for monkey 2), the position was based 126 on a preliminary visual field mapping. 127 For our analysis, we looked only in the 3-ms time interval 128 after stimulus onset (i.e., ms before the reward) to avoid 129 any influence from receiving the reward, as was done in [14], 13 and concentrate on the differences in the responses to the 131 rewarded and unrewarded stimuli. Also, we considered only 132 those recordings that were preceded by three background noise 133 images and were observed by the monkey without failing to 134 fix a small dot on the screen. In this way, on average, rewarded and 282 unrewarded recordings per day for monkey (147 rewarded and 168 unrewarded for monkey 2) were 137 retained for further analysis. The constructed classifier was 138 trained on the LFP responses of a given day and validated as 139 a predictor of the LFP responses of the next day. 14 After 37 of conditioning for monkey 1 and for monkey 2, the stimulus reward pairing was reversed, the 142 unrewarded stimulus became the rewarded one, and vice versa, 143 after which conditioning continued with this new setting for for monkey 1 and 52 for monkey III. TIME FREQUENCY SINGLE ELECTRODE FEATURES 146 First, we will analyze the recordings from single electrodes, 147 showing changes in LFPs. In our data, the recordings can 148 be divided in two groups on the basis of the similarity in 149

3 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX scale 1 scale Fig. 3. Difference in wavelet scalograms of the averaged waveforms of the LFP responses to the rewarded and unrewarded stimuli for electrodes #18 and #8 for monkey 1. Diamonds indicate local extrema. 1 the electrodes LPF signals [see Fig. 2(d)]. This division is 151 always consistent from trial to trial. From each group, we 152 choose a representative electrode (the relation between the 153 electrodes and the level of similarity will be described later), 154 namely electrodes #18 and #8 for monkey 1 (electrodes #1 155 and #59 for monkey 2), located in the upper-left and bottom- 156 right corners of the array. As discriminative features, we take 157 the coefficients from the continuous wavelet transformation 158 (CWT) [16] of the LFP signal s(t), definedas W(a, b) = 1 ( ) t b 159 s(t) ψ dt a a 16 where b is a time shift, a is a scale factor, and ψ is the 161 predefined mother wavelet (we use the Daubechies wavelet db7) with zero mean, ψ(t) dt =. As a first time 163 frequency feature subset from the wavelet transformation, we 164 extract those coefficients that correspond to the local extrema 165 in the time scale plots of the difference between the averaged 166 rewarded and unrewarded stimulus responses, averaged over 167 the trials within 1 day [see Fig. 3 for electrodes #18 and #8 168 of day 36 for monkey 1]. Furthermore, since the extrema are 169 not necessarily the best separating features, we also divide the 17 scalograms into 2 by 2 squares to avoid taking too many 171 redundant features. Within each square, the best separable 172 position was chosen on the basis of a t-statistic. After taking 173 the best scale and time combination from each square, we 174 ranked the found combinations according to their t-values 175 obtained in the previous step. Only the 1 best ranked 176 features were retained in this way. Thus, for each of the two 177 electrodes considered, we have those 1 best (according to 178 the previous procedure) separating features in terms of scale 179 and time combinations and the local extrema. During the 18 classification stage, we do not need to calculate the whole 181 CWT, we only need to perform a template-matching in the 182 selected time and scale combinations. As a result, this does not take much computational effort, and can be done online For an online application, additional attention should be paid to the selection of the length of the buffer as well as the size of the mother wavelet s support. The feature extraction itself needs the calculation of the whole 184 CWT, but this can be done offline. 185 IV. PHASE SYNCHRONY 186 Besides the stimulus-dependent differences between the 187 recordings in the time frequency domain, the interaction be- 188 tween pairs of electrodes can also be helpful in distinguishing 189 between stimuli. Some fundamental considerations to support 19 this were discussed in [17]. Indeed, by studying the relation 191 between channels, we can gain additional information because 192 a population of neural responses can have a temporal dimen- 193 sion not explicitly present in the dynamics of a single neuron s 194 response, and the population s synchrony can emerge even 195 without any modulation present in the firing rates of individual 196 neurons. As a proof of this concept, in [18] it was shown that 197 the monkey s neocortical neurons change their coherent firing 198 in response to different stimuli, even though the firing rates of 199 the individual neurons did not necessarily change. In addition, 2 there also exists evidence for different degrees of coupling in 21 the visual cortex during information processing [19], [2]. In 22 light of these findings, we examined the synchrony between 23 pairs of simultaneous LFP recordings in the array. In particular, 24 we examined phase synchrony. 25 The phases of two coupled nonlinear oscillators may syn- 26 chronize even if their amplitudes are uncorrelated [21]. Unlike 27 coherence, which computes the linear correlation between 28 two stationary signals as a function of frequency [22] but 29 does not separate the effects of amplitude and phase in the 21 correspondence between these signals [23], phase synchrony 211 describes exactly the similarity of their rhythmicities. Two 212 signals x and y are synchronized when the phase locking 213 condition ϕ xy (t) = ϕ x (t) ϕ y (t) constant, with ϕ x 214 and ϕ y the phases of x and y, applies for any time t. For 215 estimating the instantaneous phases, several approaches have 216 been described in the literature. More common are the methods 217 based on the wavelet transform (see [23]) and the Hilbert 218 transform. We used the latter one in our study, because for 219 the wavelet-based method we need to estimate all wavelet 22

4 4 TRANSACTIONS ON NEURAL NETWORKS level of synchronization Fig. 4. Temporal evolution of the difference in average synchrony between the rewarded and unrewarded stimuli after stimulus onset (vertical axes in millseconds) for each day of training (horizontal axis). The two curves in the panels trace the peaks in the absolute difference in average synchrony for two different time intervals. The blue curve is for the early interval, the pink curve is for the late interval. The black line indicates the reversal moment. Time course of the average level in synchrony (vertical axis) in the Utah array for the rewarded (green line) and unrewarded (blue) stimuli for all of training (horizontal axis) for one of the two LFP groups defined in Fig. 2(d). Yellow strips indicate weeks. Red line indicates the reversal moment. coefficients, for all scaling parameters, within a frequency band of interest, and this will be computationally too intensive a task, if a real-time application is envisaged. When deriving the phases from the recorded signal s(t), an analytical signal ζ(t) is constructed, such as ζ(t) = s(t) + ih(s(t)) = A s (t)e iϕ s (t), where the analytic amplitude is A s (t) = s 2 (t) + H 2 (s(t)) and the analytic phase is ϕ s (t) = arctan(h (s(t))/s(t)). Here, H (s(t)) is the Hilbert transform of signal s(t), definedas H (s(t)) = 1 π P.V. s(τ) t τ dτ where P.V. denotes that the integral is taken in the sense of the Cauchy principal value. Although the phase ϕ s (t) and amplitude A s (t) can be unambiguously estimated for arbitrary signals, they have a clear physical meaning only for narrowband signals [24]. The aforementioned difference ϕ xy (t) between two phases is rarely analyzed directly. Instead, indices of bivariate phase synchrony, such as the mean phase coherence [25] (also called phase locking value [23]), the Shannon synchronization index [26] and the stroboscopic approach [27], are often used for estimating the level of synchrony (for a review see [28]). Here, we consider the mean phase coherence as our synchrony measure. This index, estimated as γ x,y = e iϕxy(t) = cosϕ xy (t) 2 + sinϕ xy (t) 2 where denotes the average over time, measures how the relative phase difference is distributed over the unit circle. The result is in the interval [, 1], where 1 corresponds to perfect synchrony. Previously, Montemurro and co-workers [29] have shown that the low-frequency LFP phase yields additional information, compared to the neural spike counts, when recording from the primary visual cortex of anesthetized macaques observing natural movies. The additional amount of information decreases from 54% in the 1 4 Hz band of the LFP phase to become equal to the spike count information 255 for LFP phase frequencies greater than 24 Hz. Hence, for 256 our analysis, we also take the low-frequency component of 257 the LFP phase up to 3 Hz. Assume that we are looking into 258 the evolution (in ms) of the difference in the average phase 259 synchrony (averaged across pairs of electrodes and all trials 26 within the same day) between recordings for the rewarded 261 and unrewarded stimuli, after stimulus onset [3] (calculated 262 within a sliding window of length 4 ms). From this, we can 263 conclude that a difference in synchrony is present which is 264 more prominent in two intervals (at least after a number of 265 training have elapsed), between and 12 ms and and 26 ms for monkey 1 [Fig. 4] and between 8 and ms and 2 and 26 ms for monkey 2 [Fig. 5]. These 268 two intervals could correspond to two stages in learning, 269 a sensorial one and an attentive one [14]. Between the 27 monkeys, the difference in timing of these intervals can be 271 due to differences in stimulus size or eccentricity, to the 272 recorded cortical layers, or to slightly different electrical 273 properties of the Utah array. The difference is noticeable by 274 the increase in the level of synchrony between the two groups 275 of electrodes mentioned above, but we can also see the same 276 effect on a smaller scale within only one group [Fig. 4]. 277 We observe that, during training, the phase synchrony for 278 both the rewarded and unrewarded stimuli increases, and 279 from 15 onward a significant difference between the 28 two types of stimuli appears (responses to the background 281 stimuli are not displayed, but they possess the same level 282 of synchrony as the responses to the unrewarded stimuli). 283 We should also mention that, immediately after the reversal 284 of the stimulus reward association, the difference in phase 285 synchrony is related to the stimulus, not the reward (which 286 indicates the effect of learning), however, after some of 287 further training, the difference in synchrony is re-established 288 but now with respect to the stimulus that is associated with 289 the reward. All these results recommend phase synchrony as 29 a discriminative feature for the classifier. 291

5 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX day Rewarded angle Fig. 5. Temporal evolution in the difference in average synchrony between rewarded and unrewarded stimuli after stimulus onset (vertical axes in ms) for each day of training (horizontal axes) for the second monkey. The two curves in the panels trace the peaks in the absolute difference in average synchrony for two different time intervals. The blue curve is for the early interval, the pink curve is for the late interval. The black line indicates the reversal moment, the dashed lines indicate the breaks in training (when no recording was done). Changes in wave propagation directions as a function of training (vertical axes) for the rewarded stimulus and for the second monkey. The red line indicates the reversal moment, the dashed lines indicate breaks in training. V. PROPAGATING WAVES Recently, Rubino and co-workers [15] discovered the occurrence of LFP waves in the monkey s motor cortex. In our case, when plotting phase synchrony as a function of the distance between the electrodes, we found that synchrony decreases with increasing distance (result not shown). This might be a sign of propagating waves within the array. To check this, we use an approach similar to Rubino and coworkers, and compute a phase-based estimate of optical flow [31], in which the electrode array is regarded as a ten by ten pixel image. By coupling the grey level of each pixel to the LFPs of the corresponding electrode at time t, the array becomes a frame in an image sequence. To estimate the optical flow, the array s LFP recordings become a movie in the form s(x, y, t), wherex and y refer to the pixel/electrode position. The previously derived phases are now referred to as ϕ(x, y, t). The velocity of the coherent activity in the array is defined as the velocity of the contours of constant phase [31]. This velocity v = (dx/dt, dy/dt) is computed by taking the total derivative of ϕ(x, y, t) = C with respect to time dϕ = ϕ v + ϕ dt t. The velocity direction, which is perpendicular to the contours, is ϕ = ( ϕ/ x, ϕ/ y). This velocity can be estimated for each electrode in the array at every moment of time t. A propagating wave requires that the individual directions of the electrodes are aligned. Hence, as a measure of alignment, ϕ / ϕ is used, where. denotes spatial averaging across all electrodes at a particular moment in time (thus ϕ is the norm of the average velocity vector, averaged over all electrodes, ϕ is the average of the norms of the velocity vectors of all electrodes). Only for well-aligned cases, where the above-described measure is higher than.5, the direction of the wave velocity vector of the whole array is estimated as ϕ with magnitude ϕ/ t / ϕ, where ϕ/ t is the absolute value of the spatial average taken over all electrodes for ϕ/ t at a 327 particular moment in time. 328 The histogram of Fig. 6 shows the distribution of the 329 wave directions for the different trials of day 33, for mon- 33 key 1, and for each stimulus type. The bimodal shape of 331 the distribution indicates that two principal directions are 332 present. Throughout training, these two principal directions 333 change slightly in angle [Fig. 6 (d) for monkey 1]. After 334 the stimulus reversal, we detect a shift in these directions 335 [Fig. 6 and (c)]. Suppose we take from Fig. 6 the 336 section above the reversal moment and use it to replace the 337 corresponding section of Fig. 6(c), or vice versa, then we 338 obtain a smooth transition for the reversal moment, as in Fig (d) (background stimulus case). Because of this, we can use 34 wave direction as a feature for decoding which stimulus is 341 associated with the reward. Indeed, the distribution of the wave 342 propagation direction for the rewarded and the unrewarded 343 stimuli differ significantly ( p<.1 permutation test), starting 344 from day The same holds for the second monkey 345 [Fig. 5]. For this monkey, we can clearly see that one wave 346 direction prevails over the other [the same effect is present in 347 the first monkey as well, but on a smaller scale, see Fig and (d)]. On the other hand, the shift in the wave direction, 349 after stimuli reversal, is more prominent in monkey 2. 3 We also estimated the timing of the propagating wave. 351 Fig. 7 and (d) shows the timing of the two wave directions 352 (vertical axis) as a function of training (horizontal axis) 353 for monkey 1, for the rewarded and the unrewarded stimuli. 354 The wave direction shown in Fig. 7 is indicated in white and 355 that of Fig. 7(c) is in black. Comparing these figures with the 356 synchrony results in Fig. 4 reveals that the time interval 357 during which a wave with a constant direction is present 358 (around ms after stimuli onset) corresponds to the Note that a significant difference between the responses to the rewarded and background stimuli appears only after day 16, and disappears for 3 after the reversal moment. The responses to the unrewarded and background stimuli do not differ till the reversal moment.

6 6 TRANSACTIONS ON NEURAL NETWORKS relative frequency.12 day angle Unrewarded angle (c) day day Rewarded angle Background angle (d) Fig. 6. Histograms of directions of propagating waves within the array, for all trials of day 33, and for all time moments in a 3-ms time interval after stimulus onset, for the rewarded (green) and unrewarded (red) stimuli., (c), and (d) Changes in wave propagation directions as a function of training (vertical axes) for rewarded, unrewarded, and background stimuli, respectively. Red line indicates the reversal moment. early time interval of a prominent difference in average synchrony between the responses to the rewarded and unrewarded stimuli. This interval also corresponds to the one described in [14] and was attributed to sensorial learning in V4. Another source of information for distinguishing between the two stimuli is the magnitude of the speed of the wave propagation vector. If we plot the difference in the speed magnitudes for the rewarded and unrewarded stimuli for each time instant after stimulus onset and for each training day, we obtain a significant difference in the later stage of the response (i.e., 17 ms after stimuli onset) after some training have elapsed (Fig. 8 for monkey 1). This difference also follows the reward, not the stimuli, even after the reversal, but only after some of further training. Based on the above, we use the direction and magnitude of the propagating waves in the array as features for discriminating between the two stimuli. VI. FEATURE SELECTION Our feature selection procedure consists of two stages. First, we perform a filter-based feature selection. This is done for the time frequency features, the synchrony features, and the wave propagation features, separately. For this, the entire set of available recordings from the previous day was considered. This was required for our case because it allowed us to reduce the number of features for further assessment and to save computational time in the next stage. As time frequency features, we took the local extrema in the difference between the average rewarded and unrewarded responses in the wavelet scalogram (for the 3-ms interval after stimulus onset, we applied a transformation for the scale factor a, ranging from 1 to 3) for electrodes #18 and #8 and, additionally, the 1 best separating features (in the (c) Fig. 7. and (c) Wave propagation in the Utah array after rewarded stimulus onset, for the last day before the reversal (day 37), centered at 7 ms for the go direction of the wave, and centered at 16 ms for the back direction (c). Grey scales indicate delays in ms of wave propagation, the scale is shown to the right of each panel (note the differences in range). Crosses indicate nonconnected electrodes. and (d) Changes in the direction of wave propagation in the Utah array during presentation of the rewarded and the unrewarded (d) stimuli as a function of time after stimulus onset (in ms) (vertical axis), and as a function of training (horizontal axis). White and black indicate the two propagation directions (go and back). Red vertical line indicates the reversal moment. sense of the t-statistic), expressed in terms of scale and time 392 combinations for each electrode (see Section III). 393 As phase synchrony features, we took the level of synchrony 394 between all electrode pairs within nonoverlapping windows 395 of 2 ms (so for each electrode pair we had 3/2 = synchrony features). These features were sorted in terms of 397 their contribution in distinguishing between the two stimuli 398 using the t-statistic. Only the 1 best (according to previous 399 method) phase synchrony features were retained. 4 As wave propagation features, the direction and magnitude 41 of optical flow in each electrode, for each moment after the 42 stimulus onset, were calculated. Among these also, the 1 43 best separating ones were taken according to t-statistic, as 44 above. 45 The second stage of our feature selection consist of a 46 wrapper based on the group method of data handling (GMDH) 47 [32], which is a breadth-first search algorithm that minimizes 48 the hold-out error. This algorithm constructs, for each iteration 49 i, thesets i with cardinality n of the best subsets C ij (where 41 j = 1,...,n). This means that S i ={C i1, C i2,...,c in } (in 411 the first step S 1 consist of the n best single discriminative 412 features). Each of these subsets C ij consists of i features from 413 the whole feature space with dimension N. The transition 414 from one iteration (i) to another (i + 1) causes a new set 415 of n(n i) subsets (where from each subset C ij we have 416 (N i) new subsets, which is obtained by adding to C ij one 417 from all possible (N i) single features that are not yet in 418 this subset). From all n(n i) subsets obtained in this way, 419 the best n subsets are chosen by an external criterion (we use 42 k-fold cross-validation with some classifier) to generate a new 421 (d)

7 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX Fig. 8. Temporal evolution in the difference in speed magnitude between average brain responses for rewarded and unrewarded stimuli after stimulus onset (vertical axis in ms), and for each day of training (horizontal axis). Black vertical line indicates the reversal moment. set S i+1. This means that all data from the previous day are divided into k nonoverlapping intervals. And we construct a classifier based on feature subset C ij using (k 1) of these k intervals and estimate the prediction accuracy on the remaining kth interval. This can be done in k different ways (depending on which parts to take for the construction of the classifier and its validation). Thus, for each feature subset C ij we have k values for the prediction accuracies r ij as a result of k-fold cross-validation. We assign to each such feature subset C ij the mean of these accuracies r ij. After ranking these feature subsets according to their mean accuracies, we select the n top subsets that form a new set S i+1. As a stopping criterion, we verify the absence of an increase in performance in d subsequent iterations and take the best subset in the latter d iterations. Thus, we determine how many and which features we have to take. After that, we construct the classifier for this best subset of features, but now on the whole data from the previous day (i.e., all k intervals). And this classifier is then applied on the data of next day to estimate the decoding performance. GMDH-based feature selection was done in three different modes: 1) for the whole set of previously selected (based on the filter procedure) features; 2) only for single electrode time frequency features; and 3) only for the synchrony and the wave features. Feature selection was done on the basis of the recordings of the previous day. As an external criterion, a fivefold cross validation was used by partitioning the previous day into training and validation sets. Around 15 2 features were selected for each day. As a classifier, linear discriminate analysis (LDA) was chosen. The classifier was trained on data from the previous day and tested on data of the current day. In this way, we obtain only one classification accuracy result for the current day. VII. RESULTS The accuracy of the classification was determined for all pre and post reversal of monkey training. Fig. 9 shows the accuracy of the single-trial classification as a function of training for the three feature set combinations mentioned above, for monkey 1 and for monkey 2. We can see that the spatial features (waves and phase synchrony) yield a 46 significant gain in the classification performance (p <.1 for 461 the Friedman test) after several of training. This supports 462 our conjecture that they should be considered for decoding 463 microelectrode array recordings. The classification accuracy 464 also enables us to monitor the monkey s training progress. 465 Before the stimulus rewarded pairing reversal, the single-trial 466 accuracy increases from chance level toward a stable 87% for 467 monkey 1 (82% for monkey 2). 468 The question is whether the activity distribution in the 469 microelectrode array is based on a representation related to the 47 stimulus feature (orientation) or to the learned reward-related 471 valence of the stimulus feature. Note that we cannot rely 472 on the performance of the classifiers trained on the previous 473 day s activities to answer this question. Indeed, the classifier 474 adopts whatever features support the classification. In any case, 475 a drop in the classification performance will occur for the 476 day just after the reversal. To come back to the question, 477 we have applied the classifier trained for the day before 478 the reversal to all after the reversal. We observe that, 479 for several after the reversal moment, the performance 48 is restored (see magenta curve in Fig. 9, considering all 481 features). This means that the activity distribution that we had 482 before the reversal is restored. This convincingly shows that 483 we are classifying stimulus reward pairing and not individual 484 stimuli. 485 We also applied an exhaustive feature selection for each 486 of the aforementioned feature subsets (wavelet coefficients 487 from electrodes #18, #8, phase synchrony, magnitude and 488 direction of speed of the propagating waves) for monkey 489 1, using LDA as a classifier for day 36. These four classi- 49 fiers were grouped using AdaBoost [33]. The classification 491 accuracy for the recordings of day 37 (i.e., the classifiers 492 trained and boosted from the recordings of day 36) were 493 around 83.7% on average (different results were obtained 494 for different sequences of classifier inclusion in greedy Ad- 495 aboost). This performance is lower than what was obtained 496 with the previous method. On the other hand, AdaBoost 497 allows us to assess the influence of each classifier (i.e., 498 for each feature subset) in the decoding process by their 499 differential contributions in the classification result, we ob- tained on average.5893 for electrode #18,.3489 for elec- 1 trode #8,.1834 for phase synchrony, and.1728 for the 2 waves. This shows once more the necessity of the spatial 3 features. 4 VIII. DISCUSSION 5 A. Decoding Based on LFPs 6 We developed a strategy for decoding rewarded from un- 7 rewarded stimuli from LFPs recorded with a microelectrode 8 array implanted in area V4 of the rhesus monkey. The use of 9 LFPs was justified by several reports in the literature, showing 51 equal or even slightly better decoding accuracy than decoders 511 based on single (multi) unit activity [9] [13]. In [9], it was 512 shown that hand movement target and velocity can be inferred 513 from multiple LFPs in single trials nearly as efficiently as from 514 multiple single-unit activity recorded from the same electrodes 515

8 8 TRANSACTIONS ON NEURAL NETWORKS 8 8 accuracy (%) 6 4 accuracy (%) Fig. 9. Classification accuracy as a function of training (horizontal axis) for monkey 1 (left) and monkey 2 (right panel). Blue, red, and black curves show the classification accuracy for the current day for the classifier trained on recordings from the previous day. The blue curve is for the classifier based on wave and synchrony features, the red curve for only time frequency features, and the black curve for wave, synchrony, and time frequency features. The magenta curves show the classification accuracy, after the reversal moment, of the classifier built on the last day before the reversal (thus, without retraining the classifier afterwards), the classifier uses the whole set of features (for wave, synchrony, and time frequency features). The red vertical line indicates the reversal moment. Yellow strips indicate the when recordings were made. in the monkey motor cortex. Pesaran and co-workers [1] found that LFP activity in the parietal cortex can be used for discriminating between preferred and anti-preferred directions of hand movement with the same accuracy as the spike rate (LFPs in a frequency band 25 9 Hz were used), and that it predicts the time of a planned movement with better accuracy than with the spike rate (with use of LFPs in a band 2 Hz). Encouraged by these findings, we also considered the LFP signal for decoding. It can be used as the sole source for decoding, or as an additional one to spikes (if they are available). In other LFP decoding studies, recordings were considered that were done sequentially with a single electrode at different recording sites [12], tetrode [1], or with four electrodes [9], or 2 5 electrodes [13], or eight electrodes [11] temporally inserted transdurally in implanted chambers. This is all different from our experiment, where we had a chronically implanted electrode grid (Utah array) with 96 recording sites. Chronic implantations are quite different from acute ones, where the recordings are done in one session, and whereby spikes are likely to be omnipresent. Mehring and co-workers [9] tested the influence of the correlation between simultaneously recorded LFPs on the decoding of hand movements. They observed that there was essentially no difference between the decoding power of simultaneously recorded LFPs (correlation present) and LFPs recorded on different but viewed as being simultaneously acquired (correlation absent). We argue against treating the sequential recordings as simultaneous ones, since it is not appropriate for real-time single-trial decoding. On the other hand, Mehring and co-workers used classifiers that are based on single electrode features that only rely on amplitude changes. This means that, contrary to our case, the classifier does not take into account the relation between electrode recordings. In our research, we have shown that spatial information in the array yields a gain in classification accuracy. B. Features for Decoding 552 In decoding stimulus-associated reward LFPs, we are the 553 first to use wavelet coefficients. The latter can be viewed as a 554 generalization of the previously used time frequency methods 555 in the decoding of invasive recordings, for LFP decoding, the 556 peak-to-peak amplitude between the negative and successive 557 positive peak [9], the subsampled time series amplitudes [9], 558 and the amplitudes in different frequency bands [11], [12] 559 have already been used. But instead of subtracting amplitudes 56 in particular frequency bands, we considered all coefficients 561 of the CWT, which yields a much better localization both 562 in the time and frequency domains [16], and extracted par- 563 ticular locations in the scalogram as features for decoding. 564 The wavelet-based features used in this paper are similar 565 to those used in noninvasive brain computer interfaces for 566 detecting event-related potentials in EEG recordings [34]. 567 We also share the application of the Student s t-statistics to 568 the coefficients of CWT, but we differ in the way feature 569 extraction is performed, we took the best separable locations 57 in the wavelet scalogram based on the t-value in every square 571 obtained after dividing the whole scalogram. Moreover, we 572 took the difference between the scalograms of the averaged 573 responses to the two stimuli and identified extremes in the 574 resulting scalogram. In addition, we also performed feature 575 selection to eliminate redundancy and to simplify the decoding 576 process afterwards, which again distinguishes our work from 577 [34]. From the single electrode features we constructed, we 578 can assert that we have a proper design, since in [34] the 579 authors show a better performance of their t-cwt classifier 58 compared to other methods [peak picking, area computa- 581 tion, discrete wavelet transform, and principal component 582 analysis (PCA)]. 583 We have shown a significant gain in classification accuracy, 584 compared to single electrode features, when additionally using 585 spatial features. Here, we should make a distinction be- 586 tween spatial filtering techniques and spatial features. Spatial 587

9 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX filtering techniques, such as PCA, independent component analysis, common spatial pattern (CSP) (for their comparison in connection with brain signal decoding, see [35]), construct linear spatial combinations of the electrode recordings, so that from these transformed signals features can be defined (for example, variances, as in CSP) for the classifier to work on. This is different from the use of spatial features that directly characterize the relations between several electrode recordings for the classifier to work on. In this paper, we considered the latter strategy. As a first spatial feature, phase synchronization was used. In spite of several studies on neural synchronization in brain areas (see [36], [37]), to our knowledge the index of phase synchrony has not yet been applied to LFP classification. For mental task decoding from EEG data, phase locking was applied in [38], and was shown to yield an increase in performance when used simultaneously with features from the power spectral density in the alpha band. Compared to the latter, we have justified our choice for selecting phase synchrony as a feature, by showing the time intervals where a difference in phase synchrony occurs in the responses to the two stimuli. These intervals correspond, putatively, to sensorial and attentional learning, respectively, as hypothesized in [14], based on the amplitudes of averaged recordings. As a second spatial feature, we have used wave propagation in the array. To the best of our knowledge, this is the first time this feature is used for decoding brain signals. The estimation of propagating waves in the Utah array was introduced in [15], which led to the discovery of waves (in the beta range) propagating in two opposite directions in the motor cortex. Similar to that study, we also have found two principal directions of propagating waves but now in a visual area (area V4) during stimulus reward association learning. But in addition to that, we tracked the waves behavior during training. We have found, again for the first time, that the direction of the propagating waves changes during training in a such way that after several it is possible to use wave direction for discriminating between stimuli differing in associated reward. We have also found that, for the rewarded stimulus, the velocity of the propagating wave becomes larger than for the unrewarded stimulus, after about 2 ms after stimuli onset. These two findings justify wave propagation as a feature for decoding brain signals. Complementary to that, we showed that the direction of propagating waves depends on the time after stimuli onset. In the interval between and ms after stimuli onset (i.e., the time interval possibly related to sensorial learning), we have a constant direction for both stimuli, before and after the stimulus reversal in the training paradigm (see Fig. 8). It is tempting to speculate that the propagating waves especially those during the later interval are related to the propagating feedback depolarization waves that were observed by Roland et al. [39] in the ferret s visual cortex. As a classifier, we have used LDA. Based on that, we have performed feature selection as a wrapper. It is quite possible that, with a nonlinear classifier (for example, kernel support vector machine), we would obtain a better classification performance. But the training of such a classifier, together with feature selection, will take much more time. 646 Hence, we have restricted ourselves to a simple LDA, leav- 647 ing open the possibility to obtain an even better single-trial 648 performance. 649 C. Perceptual Learning 6 With our study, we also confirmed previous findings [14] 651 on the effect of stimulus reward training on visual responses 652 in area V4. But, in addition to the mentioned study, where the 653 effect of such training was discovered in the time frequency 654 domain, we have shown the effects of stimulus reward training 655 in terms of changes in velocities and directions of propagating 656 waves, and in changes in the relation between simultaneous 657 recordings from different electrodes (phase synchrony). Frankó 658 et al. [14] left open the question whether the observed changes 659 in LFPs with stimulus reward training were local to V4 or 66 instead were due to volume conduction from other brain 661 areas. Several results of the present analysis strongly argue 662 against volume conduction but, in contrast, point to changes 663 of the local V4 response with stimulus reward associative 664 learning. Both the decrease of phase synchrony with increasing 665 interelectrode distance and the LFP wave propagation across 666 cortical surface suggest a local origin of the LFP. The latter 667 would not be present when the LFP measured in V4 were due 668 to volume conduction. Frankó et al. also stressed that, during 669 the course of the response, two apparent dissociable processes 67 occur, one, which they related to sensorial learning, is apparent 671 shortly after stimulus appearance, and a second one, which 672 they related to attentional effects, occurs later. The present 673 analysis support this distinction by showing that synchrony 674 and wave propagation features also differ between these two 675 response phases. 676 IX. CONCLUSION 677 We have developed a new strategy for decoding invasive 678 recordings made with a microelectrode array. Our strategy 679 was based on three types of features applied to LFPs: wavelet- 68 based features of individual electrodes; and two types of spatial 681 features, i.e., phase synchrony between electrode pairs and 682 wave propagation on the array. We have shown that the latter 683 two new features yield a significant improvement in single- 684 trial decoding accuracy and, therefore, could be candidate 685 LFP features for other decoding applications (for example, for 686 brain machine interfacing). From a neuroscience perspective, 687 the excellent single-trial classification performance of the 688 rewarded versus unrewarded stimuli indicates that stimulus 689 reward association has a profound effect on LFPs in the 69 extrastriate area V4. How these effects are related to spiking 691 activity requires further study. 692 ACKNOWLEDGMENT 693 The authors are deeply indebted to Dr. E. Frankó of the 694 Laboratorium voor Neuro- en Psychofysiologie, Katholieke 695 Universiteit Leuven, Leuven, Belgium, for sharing her exper- 696 imental data. 697

10 1 TRANSACTIONS ON NEURAL NETWORKS REFERENCES [1] M. A. Nicolelis, Brain-machine interface to restore motor function and probe neural circuits, Nat. Rev. Neurosci., vol. 4, no. 5, pp , May 23. [2] G.Santhanam,S.I.Ryu,B.M.Yu,A.Afshar,andK.V.Shenoy, A high-performance brain-computer interface, Nature, vol. 442, no. 799, pp , Jul. 26. [3] M. Velliste, S. Perel, M. C. Spalding, A. S. Whitford, and A. B. Schwartz, Cortical control of a prosthetic arm for self-feeding, Nature, vol. 453, no. 7198, pp , Jun. 28. [4] N. V. Manyakov and M. M. Van Hulle, Decoding grating orientation from microelectrode array recordings in monkey cortical area V4, Int. J. Neural Syst., vol. 2, no. 2, pp , Apr. 21. [5] G. B. Stanley, F. L. Fei, and Y. Dan, Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus, J. Neurosci., vol. 19, no. 18, pp , Sep [6] S. Musallam, B. D. Corneil, B. Greger, H. Scherberger, and R. A. Andersen, Cognitive control signals for neural prosthetics, Science, vol. 35, no. 5681, pp , Jul. 24. [7] B. Pesaran, S. Musallam, and R. A. Andersen, Cognitive neural prosthetics, Current Biol., vol. 16, no. 3, pp. R77 R8, Feb. 26. [8] G. Buzsáki, Large-scale recording of neuronal ensembles, Nat. Neurosci., vol. 7, no. 5, pp , May 24. [9] C. Mehring, J. Rickert, E. Vaadia, S. C. de Oliveira, A. Aertsen, and S. Rotter, Inference of hand movements from local field potentials in monkey motor cortex, Nat. Neurosci., vol. 6, no. 12, pp , Dec. 23. [1] B. Pesaran, J. S. Pezaris, M. Sahani, P. P. Mitra, and R. A. Anderson, Temporal structure in neural activity during working memory in macaque parietal cortex, Nat. Neurosci., vol. 5, no. 8, pp , Aug. 22. [11] J. Rickert, S. C. de Oliveira, E. Vaadia, A. Aertsen, S. Rotter, and C. Mehring, Encoding of movement direction in different frequency ranges of motor cortical local field potentials, J. Neurosci., vol. 25, no. 39, pp , Sep. 25. [12] H. Scherberger, M. R. Jarvis, and R. A. Anderson, Cortical local field potential encodes movement intentions in the posterior parietal cortex, Neuron, vol. 46, no. 2, pp , Apr. 25. [13] R. L. Spinks, A. Kraskov, T. Brochier, M. A. Umilta, and R. N. Lemon, Selectivity for grasp in local field potential and single neuron activity recorded simultaneously from M1 and F5 in the awake macaque monkey, J. Neurosci., vol. 28, no. 43, pp , Oct. 28. [14] E. Frankó, A. R. Seiz, and R. Vogels, Dissociable neural effects of long term stimulus-reward pairing in macaque visual cortex, J. Cogn. Neurosci., vol. 22, no. 7, pp , Jul. 21. [15] D. Rubino, K. A. Robbins, and N. G. Hatsopoulos, Propagating waves mediate information transfer in the motor cortex, Nat. Neurosci., vol. 9, no. 12, pp , Nov. 26. [16] I. Daubechies, Ten Lectures on Wavelets. Philadelphia, PA: SIAM, [17] G. Buzsáki, Rhythms of the Brain. London, U.K.: Oxford Univ. press, 26. [18] E. Vaadia, I. Haalman, M. Abeles, H. Bergman, Y. Prut, H. Slovin, and A. Aertsen, Dynamics of neuronal interactions in monkey cortex in relation to behavioural events, Nature, vol. 373, no. 6514, pp , Feb [19] R. Eckhorn, A. M. Gail, A. Bruns, A. Gabriel, B. Al-Shaikhli, and M. Saam, Different types of signal coupling in the visual cortex related to neural mechanisms of associative processing and perception, Trans. Neural Netw., vol. 15, no. 5, pp , Sep. 24. [2] R. Pashaie and N. H. Farhat, Self-organization in a parametrically coupled logistic map network: A model for information processing in the visual cortex, Trans. Neural Netw., vol. 2, no. 4, pp , Apr. 29. [21] A. Pikovsky, M. G. Rosenblum, and J. Kurths, Synchronization: A Universal Concept in Nonlinear Science. Cambridge, U.K.: Cambridge Univ. Press, 21. [22] G. C. Carter, Coherence and time delay estimation, Proc., vol. 75, no. 2, pp , Feb [23] J.-P. Lachaux, E. Rogriguez, J. Martinerie, and F. J. Varela, Measuring phase synchrony in brain signals, Human Brain Mapping, vol. 8, no. 4, pp , [24] B. Boashash, Estimating and interpreting the instantaneous frequency 769 of a signal. II. Algorithms and applications, Proc., vol. 8, no. 77 4, pp , Apr [25] F. Mormann, K. Lehnertz, P. David, and C. E. Elger, Mean phase 772 coherence as a measure for phase synchronization and its application 773 to the EEG of epilepsy patients, Physica D, vol. 144, nos. 3 4, pp , Oct [26] P. Tass, M. G. Rosenblum, J. Weule, J. Kurths, A. Pikovsky, J. Volk- 776 mann, A. Schnitzler, and H. J. Freund, Detection of n:m phase locking 777 from noisy data: Application to magnetoencephalography, Phys. Rev. 778 Lett., vol. 81, no. 15, pp , Oct [27] M. G. Rosenblum and A. Pikovsky, Detecting direction of coupling 78 in interacting oscillators, Phys. Rev. E, vol. 64, no. 4, pp , Oct [28] E. Pereda, R. Q. Quiroga, and J. Bhattacharya, Nonlinear multivariate 783 analysis of neurophysiological signals, Prog. Neurobiol., vol. 77, nos , pp. 1 37, Sep. Oct [29] M. A. Montemurro, M. J. Rasch, Y. Murayama, N. K. Logothetis, and 786 S. Panzeri, Phase-of-firing coding of natural visual stimuli in primary 787 visual cortex, Current Biol., vol. 18, no. 5, pp , Mar [3] N. V. Manyakov and M. M. Van Hulle, Synchronization in monkeys 789 visual cortex analyzed with information-theoretic measure, Chaos, vol , no. 3, pp , Sep [31] D. J. Fleet and A. D. Jepson, Computation of component image velocity 792 from local phase information, Int. J. Comput. Vis., vol. 5, no. 1, pp , Aug [32] A. G. Ivakhnenko and G. A. Ivakhnenko, The review of problem 795 solvable by algorithms of the group methods of data handling (GMDH), 796 Pattern Recogn. Image Anal., vol. 5, no. 4, pp , [33] Y. Freund and R. E. Schapire, A decision-theoretic generalization of 798 on-line learning and an application to boosting, J. Comput. Syst. Sci., 799 vol. 55, no. 1, pp , Aug [34] V. Bostanov and B. Kotchoubey, The t-cwt: A new ERP detection and 81 quantification method based on the continuous wavelet transform and 82 Student s t-statistics, Clin. Neurophysiol., vol. 117, no. 12, pp , Dec [35] G. Dornhege, M. Krauledat, K.-R. Müller, and B. Blankertz, General 85 signal processing and machine learning tools for BCI analysis, in 86 Toward Brain-Computer Interfacing, G. Dornhege, J. R. Millán, T. 87 Hinterberger, D. J. McFarland, K.-R. Müller, Eds. Cambridge, MA: MIT 88 Press, 27, pp [36] P. Fries, J.-H. Schroeder, P. R. Roelfsema, W. Singer, and A. K. 81 Engel, Oscillatory neural synchronization in primary visual cortex as a 811 correlate of stimulus selection, J. Neurosci., vol. 22, no. 9, pp , May [37] L. Melloni, C. Molina, M. Pena, D. Torres, W. Singer, and E. Rodriguez, 814 Synchronization of neural activity across cortical areas correlates with 815 conscious perception, J. Neurosci., vol. 27, no. 11, pp , Mar [38] E. Gysels and P. Celka, Phase synchronization for the recognition of 818 mental tasks in a brain-computer interface, Trans. Neural Syst. 819 Rehabil. Eng., vol. 12, no. 4, pp , Dec [39] P. E. Roland, A. Hanazawa, C. Undeman, D. Eriksson, T. Tompa, 821 H. Nakamura, S. Valentiniene, and B. Ahmed, Cortical feedback 822 depolarization waves: A mechanism of top-down influence on early 823 visual areas, Proc. Natl. Acad. Sci., vol. 13, no. 33, pp , 824 Aug Nikolay V. Manyakov received the M.Sc. degree in 826 mathematics from the Belarussian State University, 827 Minsk, Belarus, in 1998, and the Ph.D. degree in the- 828 oretical informatics from the Belarussian State Uni- 829 versity of Informatics and Radioelectronics, Minsk, 83 in He is a Research Fellow at the Computational 832 Neuroscience Group, Laboratorium voor Neuro- en 833 Psychofysiologie, Katholieke Universiteit Leuven 834 Medical School, Leuven, Belgium. His current re- 835 search interests include computational neuroscience, 836 neural networks, machine learning, data mining, and signal processing. 837

11 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX Rufin Vogels received the Ph.D. degree in experimental and clinical psychology from the Katholieke Universiteit Leuven (K. U. Leuven), Leuven, Belgium, in His Ph.D. thesis was on psychophysical studies on human visual orientation discrimination. He was a Post-Doctoral Research Fellow in Peter Schiller s Laboratory at Massachusetts Institute of Technology, Boston, recording instriate and extrastriate cortexes of behaving monkeys. Currently, he is a Professor at the Division of Neurophysiology, K. U. Leuven Medical School. His current research interests include visual cortical processing in awake behaving monkeys using single-cell recording and other invasive techniques. Marc M. Van Hulle (M 97 SM ) received the 852 M.Sc. degree in electrotechnical engineering and 853 the Ph.D. degree in applied sciences from the 854 Katholieke Universiteit Leuven (K. U. Leuven), 855 Leuven, Belgium. He also received the B.Sc.Econ. 856 and M.B.A. degrees. He received the Doctor 857 Technices degree from Queen Margrethe II of 858 Denmark, in 23, and an Honorary Doctoral 859 degree from Brest State University, Brest, Belarus, 86 in He is currently a Full Professor at the K. U. 862 Leuven Medical School, where he heads the Computational Neuroscience 863 Group of the Laboratorium voor Neuro- en Psychofysiologie. In 1992, he 864 was with the Brain and Cognitive Sciences Department, Massachusetts 865 Institute of Technology, Boston, as a Post-Doctoral Scientist. He has authored 866 a monograph titled Faithful Representations and Topographic Maps: From 867 Distortion- to Information-Based Self-Organization (John Wiley, 2; also 868 translated into Japanese) and 2 technical publications. His current research 869 interests include computational neuroscience, neural networks, computer 87 vision, data mining, and signal processing. 871 Dr. Van Hulle is an Executive Member of the Signal Processing 872 Society, Associate Editor of TRANSACTIONS ON NEURAL NETWORKS, 873 Computational Intelligence, and Neuroscience and International Journal 874 of Neural Systems. He is a member of the program committees of several 875 international conferences on machine learning, informatics, information 876 processing, etc. 877

12 AUTHOR QUERY AUTHOR PLEASE ANSWER THE QUERY AQ:1 = Please provide the expansions for the CREA.

13 TRANSACTIONS ON NEURAL NETWORKS 1 Decoding Stimulus Reward Pairing From Local Field Potentials Recorded From Monkey Visual Cortex Nikolay V. Manyakov, Rufin Vogels, and Marc M. Van Hulle, Senior Member, AQ:1 Abstract Single-trial decoding of brain recordings is a real challenge, since it pushes the signal-to-noise ratio issue to the limit. In this paper, we concentrate on the single-trial decoding of stimulus reward pairing from local field potentials (LFPs) recorded chronically in the visual cortical area V4 of monkeys during a perceptual conditioning task. We developed a set of physiologically meaningful features that can classify and monitor the monkey s training performance. One of these features is based on the recently discovered propagation of waves of LFPs in the visual cortex. Time frequency features together with spatial features (phase synchrony and wave propagation) yield, after applying a feature selection procedure, an exceptionally good single-trial classification performance, even when using a linear classifier. Index Terms Decoding, local field potentials, phase synchrony, visual cortex, wave propagation. 17 I. INTRODUCTION IN the last decade, much success has been achieved in de- 19 coding neural activity for motor control or motor planning, 2 for example for controlling a prosthetic device [1] [3]. To 21 a much lesser extent, progress has been made in decoding 22 visual information (for example, [4], [5]) and higher level 23 cognitive processes that organize behavior [6], [7]. In order 24 to achieve real-time decoding performance, for example to 25 control a prosthetic device, single-trial decoding is essential 26 and averaging over several trials is not an option. Manuscript received July 6, 29; revised June 25, 21 and September 7, 21; accepted September 8, 21. The work of N. V. Manyakov was supported in part by the European Commission under Grant IST The work of R. Vogels was supported in part by the Human Frontier Science Program under Grant RGP 18/24, the Queen Elisabeth Medical Foundation of Belgium, the Interuniversity Attraction Poles Program Belgian Science Policy under Grant IUAP P6/54, the Flemish Regional Ministry of Education (Belgium) under Grant GOA 1/19, the Belgian Fund for Scientific Research Flanders, and the Excellence Financing Program under Grant EF 25. The work of M. M. V. Hulle was supported in part by the Excellence Financing Program under Grant EF 25 and the CREA Financing Program under Grant CREA/7/27 of the Katholieke Universiteit Leuven, the Belgian Fund for Scientific Research Flanders under Grant G and Grant G.588.9, the Interuniversity Attraction Poles Program Belgian Science Policy under Grant IUAP P5/54, the Flemish Regional Ministry of Education (Belgium) under Grant GOA 1/19, and the European Commission under Grant STREP , Grant IST , and Grant IST The authors are with the Laboratorium voor Neuro- en Psychofysiologie, Katholieke Universiteit Leuven, Leuven 3, Belgium ( NikolayV.Manyakov@med.kuleuven.be; Rufin.Vogels@med.kuleuven.be; Marc.VanHulle@med.kuleuven.be). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier 1.119/TNN A second complication, specifically for chronic intracranial 27 recordings, is that action potentials (spikes) are often lost 28 even after a few of recording due to cell expiration, 29 inflammation, reactive gliosis, and scarring. As an alternative, 3 local field potentials (LFPs) are often used. They represent the 31 summation of excitatory and inhibitory dendritic potentials 32 [8]. LFPs are more robust against signal degradation and, in 33 contrast to spikes, they can be easily recorded over longer 34 periods (chronic recordings). Using LFPs, equal performance 35 as, or even a slightly better performance compared to, 36 decoders based on action potentials has been achieved in 37 brain machine interfacing [9] [13]. 38 In this paper, we concentrate on the single-trial decoding 39 of LFP activity, chronically recorded with a microelectrode 4 array implanted in the visual area V4 of two macaque mon- 41 keys. Since we consider recordings over several months, we 42 concentrate on LFPs as the input to our decoder. In particular, 43 we consider event-related LFPs recorded in response to the 44 following two oriented stimuli (Fig. 1), one for which the 45 monkeys receive a fluid reward, and another not (i.e., a 46 classical conditioning paradigm). In Fig. 2 and, two 47 traces of LFPs (single trial) are shown (blue and green curves) 48 for the rewarded (left figure) and the unrewarded (right figure) 49 stimuli, for training day 36 and electrode #8 of monkey 1, as well as the average response (red curves in left and 51 right figures), for the same electrode, and averaged over the 52 same training day. We observe that, as expected, the trial-to- 53 trial variability in the LFP responses is large [see Fig. 2(c)], 54 but we have found that the average response does not vary /$ greatly from day to day, at least after some training have 56 elapsed [compare the averaged traces of the recordings for the 57 rewarded and unrewarded stimuli for training 36 and in Fig. 2 (c), for monkey 1]. 59 As is common in the analysis of LFPs, as well as in 6 electroencephalogram (EEG) studies, several trials need to be 61 considered for averaging before the stimulus-related response 62 can be detected. When the signal of interest is large, fewer 63 trials for averaging will be needed. As was shown in [14], 64 which details the conditioning training of the experiment 65 considered in this paper, an early neural response was observed 66 that was suggested to be related to sensorial learning, and 67 a later one to attentional learning. Based on this, temporal 68 features are developed in this paper, and supplemented with 69 a number of spatial features that serve as the input to the 7 classifier whose aim is to distinguish the rewarded from 71

14 2 TRANSACTIONS ON NEURAL NETWORKS amplitude (µv) amplitude (µv) Fig. 1. Examples of rewarded and unrewarded stimuli. the unrewarded stimulus responses. We target for single-trial classification, since it pushes the signal-to-noise ratio issue to the limit (Fig. 2), and since it is a requirement for the future development of a decoder performing in real time. Below, we will first describe the details of the experimental procedure, the implantation of the microelectrode array, and the signal conditioning. Then, we will develop three types of features for the classifier, wavelet-based ones for the individual electrodes (temporal features), and two types of spatial features phase synchrony between electrode pairs and wave propagation in the array. The latter one is inspired by the recently discovered propagation of waves of LFPs in the visual cortex [15]. Next, we perform feature selection so as to optimize our set of features, and apply a linear classifier. We show the classification performance and the contribution of the different types of features therein. We consider the recordings made for two monkeys and verify that the obtained decoding performances are similar. II. EXPERIMENT Two rhesus monkeys were implanted with a Utah array into the prelunate gyrus (area V4). The array measures 4 4 mm and consists of 1 1 electrodes (4 of them are wireless) with.4 mm distance between them, 8 µm in diameter, 1 mm long, and with tips metalized with platinum. Recordings were made with a Bionic Cerebrus system. The LFP signals were obtained by filtering the recorded signals between.3 and 2 Hz. The stimuli were obliquely oriented sinusoidal gratings (2 c/deg, diameter size 4 visual angle for monkey 1 and 4 for monkey 2) with phases randomized across presentations (Fig. 1), and superimposed on a sinusoidal noise background. The gratings were partially occluded by sinusoidal noise, during conditioning (see further), the signal-to-noise ratio was 2%. During training, a different sinusoidal noise background, which filled the display, was presented every ms. At random intervals, a sinusoidal grating was presented for ms. The grating orientation could be either for monkey 1 (112.5 for monkey 2) for the rewarded stimulus or 67.5 for monkey 1 (22.5 for monkey 2) for the unrewarded stimulus. The presentation of the (112.5 ) grating was paired with a fluid reward, while no reward was given at any other time instant (classical conditioning). The reward was given 4 ms after presentation of the grating pattern and, thus, partially overlapping with the grating presentation. For convenience, we will further refer to these stimuli as the rewarded and unrewarded ones. Each monkey was trained to fix a small dot on the screen ( fixation window), amplitude (µv) amplitude (µv) (c) (d) Fig. 2. and Example of event-related LFP responses (blue and green curves), for rewarded (left panel) and unrewarded (right panel) stimuli, recorded with electrode #8 for day 36 of training for monkey 1, for 3 ms after stimulus onset. Red curves correspond to the average LFP within the same day. (c) Average rewarded (blue) and unrewarded (red) responses with error bars (standard deviations) from electrode #8 for day 37 of training for monkey 1. (d) Example single-trial recordings from all 96 electrodes of the array for the last day before the reversal for monkey 1 (day 37). Two distinct groups of recordings are clearly visible, based on the signal shape and the within group synchrony. 1 1 and the reward was given only when the monkey maintained 12 fixation during stimulus presentation (the monkeys eye 121 movements were monitored using a noninvasive infrared eye 122 tracking device: Iscan, Burlington, MA). The gratings were 123 always preceded by 1 6 noise backgrounds. The stimuli were 124 presented 7.2 eccentric in the right lower visual field for 125 monkey 1 (foveally for monkey 2), the position was based 126 on a preliminary visual field mapping. 127 For our analysis, we looked only in the 3-ms time interval 128 after stimulus onset (i.e., ms before the reward) to avoid 129 any influence from receiving the reward, as was done in [14], 13 and concentrate on the differences in the responses to the 131 rewarded and unrewarded stimuli. Also, we considered only 132 those recordings that were preceded by three background noise 133 images and were observed by the monkey without failing to 134 fix a small dot on the screen. In this way, on average, rewarded and 282 unrewarded recordings per day for monkey (147 rewarded and 168 unrewarded for monkey 2) were 137 retained for further analysis. The constructed classifier was 138 trained on the LFP responses of a given day and validated as 139 a predictor of the LFP responses of the next day. 14 After 37 of conditioning for monkey 1 and for monkey 2, the stimulus reward pairing was reversed, the 142 unrewarded stimulus became the rewarded one, and vice versa, 143 after which conditioning continued with this new setting for for monkey 1 and 52 for monkey III. TIME FREQUENCY SINGLE ELECTRODE FEATURES 146 First, we will analyze the recordings from single electrodes, 147 showing changes in LFPs. In our data, the recordings can 148 be divided in two groups on the basis of the similarity in 149

15 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX scale 1 scale Fig. 3. Difference in wavelet scalograms of the averaged waveforms of the LFP responses to the rewarded and unrewarded stimuli for electrodes #18 and #8 for monkey 1. Diamonds indicate local extrema. 1 the electrodes LPF signals [see Fig. 2(d)]. This division is 151 always consistent from trial to trial. From each group, we 152 choose a representative electrode (the relation between the 153 electrodes and the level of similarity will be described later), 154 namely electrodes #18 and #8 for monkey 1 (electrodes #1 155 and #59 for monkey 2), located in the upper-left and bottom- 156 right corners of the array. As discriminative features, we take 157 the coefficients from the continuous wavelet transformation 158 (CWT) [16] of the LFP signal s(t), definedas W(a, b) = 1 ( ) t b 159 s(t) ψ dt a a 16 where b is a time shift, a is a scale factor, and ψ is the 161 predefined mother wavelet (we use the Daubechies wavelet db7) with zero mean, ψ(t) dt =. As a first time 163 frequency feature subset from the wavelet transformation, we 164 extract those coefficients that correspond to the local extrema 165 in the time scale plots of the difference between the averaged 166 rewarded and unrewarded stimulus responses, averaged over 167 the trials within 1 day [see Fig. 3 for electrodes #18 and #8 168 of day 36 for monkey 1]. Furthermore, since the extrema are 169 not necessarily the best separating features, we also divide the 17 scalograms into 2 by 2 squares to avoid taking too many 171 redundant features. Within each square, the best separable 172 position was chosen on the basis of a t-statistic. After taking 173 the best scale and time combination from each square, we 174 ranked the found combinations according to their t-values 175 obtained in the previous step. Only the 1 best ranked 176 features were retained in this way. Thus, for each of the two 177 electrodes considered, we have those 1 best (according to 178 the previous procedure) separating features in terms of scale 179 and time combinations and the local extrema. During the 18 classification stage, we do not need to calculate the whole 181 CWT, we only need to perform a template-matching in the 182 selected time and scale combinations. As a result, this does not take much computational effort, and can be done online For an online application, additional attention should be paid to the selection of the length of the buffer as well as the size of the mother wavelet s support. The feature extraction itself needs the calculation of the whole 184 CWT, but this can be done offline. 185 IV. PHASE SYNCHRONY 186 Besides the stimulus-dependent differences between the 187 recordings in the time frequency domain, the interaction be- 188 tween pairs of electrodes can also be helpful in distinguishing 189 between stimuli. Some fundamental considerations to support 19 this were discussed in [17]. Indeed, by studying the relation 191 between channels, we can gain additional information because 192 a population of neural responses can have a temporal dimen- 193 sion not explicitly present in the dynamics of a single neuron s 194 response, and the population s synchrony can emerge even 195 without any modulation present in the firing rates of individual 196 neurons. As a proof of this concept, in [18] it was shown that 197 the monkey s neocortical neurons change their coherent firing 198 in response to different stimuli, even though the firing rates of 199 the individual neurons did not necessarily change. In addition, 2 there also exists evidence for different degrees of coupling in 21 the visual cortex during information processing [19], [2]. In 22 light of these findings, we examined the synchrony between 23 pairs of simultaneous LFP recordings in the array. In particular, 24 we examined phase synchrony. 25 The phases of two coupled nonlinear oscillators may syn- 26 chronize even if their amplitudes are uncorrelated [21]. Unlike 27 coherence, which computes the linear correlation between 28 two stationary signals as a function of frequency [22] but 29 does not separate the effects of amplitude and phase in the 21 correspondence between these signals [23], phase synchrony 211 describes exactly the similarity of their rhythmicities. Two 212 signals x and y are synchronized when the phase locking 213 condition ϕ xy (t) = ϕ x (t) ϕ y (t) constant, with ϕ x 214 and ϕ y the phases of x and y, applies for any time t. For 215 estimating the instantaneous phases, several approaches have 216 been described in the literature. More common are the methods 217 based on the wavelet transform (see [23]) and the Hilbert 218 transform. We used the latter one in our study, because for 219 the wavelet-based method we need to estimate all wavelet 22

16 4 TRANSACTIONS ON NEURAL NETWORKS level of synchronization Fig. 4. Temporal evolution of the difference in average synchrony between the rewarded and unrewarded stimuli after stimulus onset (vertical axes in millseconds) for each day of training (horizontal axis). The two curves in the panels trace the peaks in the absolute difference in average synchrony for two different time intervals. The blue curve is for the early interval, the pink curve is for the late interval. The black line indicates the reversal moment. Time course of the average level in synchrony (vertical axis) in the Utah array for the rewarded (green line) and unrewarded (blue) stimuli for all of training (horizontal axis) for one of the two LFP groups defined in Fig. 2(d). Yellow strips indicate weeks. Red line indicates the reversal moment. coefficients, for all scaling parameters, within a frequency band of interest, and this will be computationally too intensive a task, if a real-time application is envisaged. When deriving the phases from the recorded signal s(t), an analytical signal ζ(t) is constructed, such as ζ(t) = s(t) + ih(s(t)) = A s (t)e iϕ s (t), where the analytic amplitude is A s (t) = s 2 (t) + H 2 (s(t)) and the analytic phase is ϕ s (t) = arctan(h (s(t))/s(t)). Here, H (s(t)) is the Hilbert transform of signal s(t), definedas H (s(t)) = 1 π P.V. s(τ) t τ dτ where P.V. denotes that the integral is taken in the sense of the Cauchy principal value. Although the phase ϕ s (t) and amplitude A s (t) can be unambiguously estimated for arbitrary signals, they have a clear physical meaning only for narrowband signals [24]. The aforementioned difference ϕ xy (t) between two phases is rarely analyzed directly. Instead, indices of bivariate phase synchrony, such as the mean phase coherence [25] (also called phase locking value [23]), the Shannon synchronization index [26] and the stroboscopic approach [27], are often used for estimating the level of synchrony (for a review see [28]). Here, we consider the mean phase coherence as our synchrony measure. This index, estimated as γ x,y = e iϕxy(t) = cosϕ xy (t) 2 + sinϕ xy (t) 2 where denotes the average over time, measures how the relative phase difference is distributed over the unit circle. The result is in the interval [, 1], where 1 corresponds to perfect synchrony. Previously, Montemurro and co-workers [29] have shown that the low-frequency LFP phase yields additional information, compared to the neural spike counts, when recording from the primary visual cortex of anesthetized macaques observing natural movies. The additional amount of information decreases from 54% in the 1 4 Hz band of the LFP phase to become equal to the spike count information 255 for LFP phase frequencies greater than 24 Hz. Hence, for 256 our analysis, we also take the low-frequency component of 257 the LFP phase up to 3 Hz. Assume that we are looking into 258 the evolution (in ms) of the difference in the average phase 259 synchrony (averaged across pairs of electrodes and all trials 26 within the same day) between recordings for the rewarded 261 and unrewarded stimuli, after stimulus onset [3] (calculated 262 within a sliding window of length 4 ms). From this, we can 263 conclude that a difference in synchrony is present which is 264 more prominent in two intervals (at least after a number of 265 training have elapsed), between and 12 ms and and 26 ms for monkey 1 [Fig. 4] and between 8 and ms and 2 and 26 ms for monkey 2 [Fig. 5]. These 268 two intervals could correspond to two stages in learning, 269 a sensorial one and an attentive one [14]. Between the 27 monkeys, the difference in timing of these intervals can be 271 due to differences in stimulus size or eccentricity, to the 272 recorded cortical layers, or to slightly different electrical 273 properties of the Utah array. The difference is noticeable by 274 the increase in the level of synchrony between the two groups 275 of electrodes mentioned above, but we can also see the same 276 effect on a smaller scale within only one group [Fig. 4]. 277 We observe that, during training, the phase synchrony for 278 both the rewarded and unrewarded stimuli increases, and 279 from 15 onward a significant difference between the 28 two types of stimuli appears (responses to the background 281 stimuli are not displayed, but they possess the same level 282 of synchrony as the responses to the unrewarded stimuli). 283 We should also mention that, immediately after the reversal 284 of the stimulus reward association, the difference in phase 285 synchrony is related to the stimulus, not the reward (which 286 indicates the effect of learning), however, after some of 287 further training, the difference in synchrony is re-established 288 but now with respect to the stimulus that is associated with 289 the reward. All these results recommend phase synchrony as 29 a discriminative feature for the classifier. 291

17 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX day Rewarded angle Fig. 5. Temporal evolution in the difference in average synchrony between rewarded and unrewarded stimuli after stimulus onset (vertical axes in ms) for each day of training (horizontal axes) for the second monkey. The two curves in the panels trace the peaks in the absolute difference in average synchrony for two different time intervals. The blue curve is for the early interval, the pink curve is for the late interval. The black line indicates the reversal moment, the dashed lines indicate the breaks in training (when no recording was done). Changes in wave propagation directions as a function of training (vertical axes) for the rewarded stimulus and for the second monkey. The red line indicates the reversal moment, the dashed lines indicate breaks in training. V. PROPAGATING WAVES Recently, Rubino and co-workers [15] discovered the occurrence of LFP waves in the monkey s motor cortex. In our case, when plotting phase synchrony as a function of the distance between the electrodes, we found that synchrony decreases with increasing distance (result not shown). This might be a sign of propagating waves within the array. To check this, we use an approach similar to Rubino and coworkers, and compute a phase-based estimate of optical flow [31], in which the electrode array is regarded as a ten by ten pixel image. By coupling the grey level of each pixel to the LFPs of the corresponding electrode at time t, the array becomes a frame in an image sequence. To estimate the optical flow, the array s LFP recordings become a movie in the form s(x, y, t), wherex and y refer to the pixel/electrode position. The previously derived phases are now referred to as ϕ(x, y, t). The velocity of the coherent activity in the array is defined as the velocity of the contours of constant phase [31]. This velocity v = (dx/dt, dy/dt) is computed by taking the total derivative of ϕ(x, y, t) = C with respect to time dϕ = ϕ v + ϕ dt t. The velocity direction, which is perpendicular to the contours, is ϕ = ( ϕ/ x, ϕ/ y). This velocity can be estimated for each electrode in the array at every moment of time t. A propagating wave requires that the individual directions of the electrodes are aligned. Hence, as a measure of alignment, ϕ / ϕ is used, where. denotes spatial averaging across all electrodes at a particular moment in time (thus ϕ is the norm of the average velocity vector, averaged over all electrodes, ϕ is the average of the norms of the velocity vectors of all electrodes). Only for well-aligned cases, where the above-described measure is higher than.5, the direction of the wave velocity vector of the whole array is estimated as ϕ with magnitude ϕ/ t / ϕ, where ϕ/ t is the absolute value of the spatial average taken over all electrodes for ϕ/ t at a 327 particular moment in time. 328 The histogram of Fig. 6 shows the distribution of the 329 wave directions for the different trials of day 33, for mon- 33 key 1, and for each stimulus type. The bimodal shape of 331 the distribution indicates that two principal directions are 332 present. Throughout training, these two principal directions 333 change slightly in angle [Fig. 6 (d) for monkey 1]. After 334 the stimulus reversal, we detect a shift in these directions 335 [Fig. 6 and (c)]. Suppose we take from Fig. 6 the 336 section above the reversal moment and use it to replace the 337 corresponding section of Fig. 6(c), or vice versa, then we 338 obtain a smooth transition for the reversal moment, as in Fig (d) (background stimulus case). Because of this, we can use 34 wave direction as a feature for decoding which stimulus is 341 associated with the reward. Indeed, the distribution of the wave 342 propagation direction for the rewarded and the unrewarded 343 stimuli differ significantly ( p<.1 permutation test), starting 344 from day The same holds for the second monkey 345 [Fig. 5]. For this monkey, we can clearly see that one wave 346 direction prevails over the other [the same effect is present in 347 the first monkey as well, but on a smaller scale, see Fig and (d)]. On the other hand, the shift in the wave direction, 349 after stimuli reversal, is more prominent in monkey 2. 3 We also estimated the timing of the propagating wave. 351 Fig. 7 and (d) shows the timing of the two wave directions 352 (vertical axis) as a function of training (horizontal axis) 353 for monkey 1, for the rewarded and the unrewarded stimuli. 354 The wave direction shown in Fig. 7 is indicated in white and 355 that of Fig. 7(c) is in black. Comparing these figures with the 356 synchrony results in Fig. 4 reveals that the time interval 357 during which a wave with a constant direction is present 358 (around ms after stimuli onset) corresponds to the Note that a significant difference between the responses to the rewarded and background stimuli appears only after day 16, and disappears for 3 after the reversal moment. The responses to the unrewarded and background stimuli do not differ till the reversal moment.

18 6 TRANSACTIONS ON NEURAL NETWORKS relative frequency.12 day angle Unrewarded angle (c) day day Rewarded angle Background angle (d) Fig. 6. Histograms of directions of propagating waves within the array, for all trials of day 33, and for all time moments in a 3-ms time interval after stimulus onset, for the rewarded (green) and unrewarded (red) stimuli., (c), and (d) Changes in wave propagation directions as a function of training (vertical axes) for rewarded, unrewarded, and background stimuli, respectively. Red line indicates the reversal moment. early time interval of a prominent difference in average synchrony between the responses to the rewarded and unrewarded stimuli. This interval also corresponds to the one described in [14] and was attributed to sensorial learning in V4. Another source of information for distinguishing between the two stimuli is the magnitude of the speed of the wave propagation vector. If we plot the difference in the speed magnitudes for the rewarded and unrewarded stimuli for each time instant after stimulus onset and for each training day, we obtain a significant difference in the later stage of the response (i.e., 17 ms after stimuli onset) after some training have elapsed (Fig. 8 for monkey 1). This difference also follows the reward, not the stimuli, even after the reversal, but only after some of further training. Based on the above, we use the direction and magnitude of the propagating waves in the array as features for discriminating between the two stimuli. VI. FEATURE SELECTION Our feature selection procedure consists of two stages. First, we perform a filter-based feature selection. This is done for the time frequency features, the synchrony features, and the wave propagation features, separately. For this, the entire set of available recordings from the previous day was considered. This was required for our case because it allowed us to reduce the number of features for further assessment and to save computational time in the next stage. As time frequency features, we took the local extrema in the difference between the average rewarded and unrewarded responses in the wavelet scalogram (for the 3-ms interval after stimulus onset, we applied a transformation for the scale factor a, ranging from 1 to 3) for electrodes #18 and #8 and, additionally, the 1 best separating features (in the (c) Fig. 7. and (c) Wave propagation in the Utah array after rewarded stimulus onset, for the last day before the reversal (day 37), centered at 7 ms for the go direction of the wave, and centered at 16 ms for the back direction (c). Grey scales indicate delays in ms of wave propagation, the scale is shown to the right of each panel (note the differences in range). Crosses indicate nonconnected electrodes. and (d) Changes in the direction of wave propagation in the Utah array during presentation of the rewarded and the unrewarded (d) stimuli as a function of time after stimulus onset (in ms) (vertical axis), and as a function of training (horizontal axis). White and black indicate the two propagation directions (go and back). Red vertical line indicates the reversal moment. sense of the t-statistic), expressed in terms of scale and time 392 combinations for each electrode (see Section III). 393 As phase synchrony features, we took the level of synchrony 394 between all electrode pairs within nonoverlapping windows 395 of 2 ms (so for each electrode pair we had 3/2 = synchrony features). These features were sorted in terms of 397 their contribution in distinguishing between the two stimuli 398 using the t-statistic. Only the 1 best (according to previous 399 method) phase synchrony features were retained. 4 As wave propagation features, the direction and magnitude 41 of optical flow in each electrode, for each moment after the 42 stimulus onset, were calculated. Among these also, the 1 43 best separating ones were taken according to t-statistic, as 44 above. 45 The second stage of our feature selection consist of a 46 wrapper based on the group method of data handling (GMDH) 47 [32], which is a breadth-first search algorithm that minimizes 48 the hold-out error. This algorithm constructs, for each iteration 49 i, thesets i with cardinality n of the best subsets C ij (where 41 j = 1,...,n). This means that S i ={C i1, C i2,...,c in } (in 411 the first step S 1 consist of the n best single discriminative 412 features). Each of these subsets C ij consists of i features from 413 the whole feature space with dimension N. The transition 414 from one iteration (i) to another (i + 1) causes a new set 415 of n(n i) subsets (where from each subset C ij we have 416 (N i) new subsets, which is obtained by adding to C ij one 417 from all possible (N i) single features that are not yet in 418 this subset). From all n(n i) subsets obtained in this way, 419 the best n subsets are chosen by an external criterion (we use 42 k-fold cross-validation with some classifier) to generate a new 421 (d)

19 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX Fig. 8. Temporal evolution in the difference in speed magnitude between average brain responses for rewarded and unrewarded stimuli after stimulus onset (vertical axis in ms), and for each day of training (horizontal axis). Black vertical line indicates the reversal moment. set S i+1. This means that all data from the previous day are divided into k nonoverlapping intervals. And we construct a classifier based on feature subset C ij using (k 1) of these k intervals and estimate the prediction accuracy on the remaining kth interval. This can be done in k different ways (depending on which parts to take for the construction of the classifier and its validation). Thus, for each feature subset C ij we have k values for the prediction accuracies r ij as a result of k-fold cross-validation. We assign to each such feature subset C ij the mean of these accuracies r ij. After ranking these feature subsets according to their mean accuracies, we select the n top subsets that form a new set S i+1. As a stopping criterion, we verify the absence of an increase in performance in d subsequent iterations and take the best subset in the latter d iterations. Thus, we determine how many and which features we have to take. After that, we construct the classifier for this best subset of features, but now on the whole data from the previous day (i.e., all k intervals). And this classifier is then applied on the data of next day to estimate the decoding performance. GMDH-based feature selection was done in three different modes: 1) for the whole set of previously selected (based on the filter procedure) features; 2) only for single electrode time frequency features; and 3) only for the synchrony and the wave features. Feature selection was done on the basis of the recordings of the previous day. As an external criterion, a fivefold cross validation was used by partitioning the previous day into training and validation sets. Around 15 2 features were selected for each day. As a classifier, linear discriminate analysis (LDA) was chosen. The classifier was trained on data from the previous day and tested on data of the current day. In this way, we obtain only one classification accuracy result for the current day. VII. RESULTS The accuracy of the classification was determined for all pre and post reversal of monkey training. Fig. 9 shows the accuracy of the single-trial classification as a function of training for the three feature set combinations mentioned above, for monkey 1 and for monkey 2. We can see that the spatial features (waves and phase synchrony) yield a 46 significant gain in the classification performance (p <.1 for 461 the Friedman test) after several of training. This supports 462 our conjecture that they should be considered for decoding 463 microelectrode array recordings. The classification accuracy 464 also enables us to monitor the monkey s training progress. 465 Before the stimulus rewarded pairing reversal, the single-trial 466 accuracy increases from chance level toward a stable 87% for 467 monkey 1 (82% for monkey 2). 468 The question is whether the activity distribution in the 469 microelectrode array is based on a representation related to the 47 stimulus feature (orientation) or to the learned reward-related 471 valence of the stimulus feature. Note that we cannot rely 472 on the performance of the classifiers trained on the previous 473 day s activities to answer this question. Indeed, the classifier 474 adopts whatever features support the classification. In any case, 475 a drop in the classification performance will occur for the 476 day just after the reversal. To come back to the question, 477 we have applied the classifier trained for the day before 478 the reversal to all after the reversal. We observe that, 479 for several after the reversal moment, the performance 48 is restored (see magenta curve in Fig. 9, considering all 481 features). This means that the activity distribution that we had 482 before the reversal is restored. This convincingly shows that 483 we are classifying stimulus reward pairing and not individual 484 stimuli. 485 We also applied an exhaustive feature selection for each 486 of the aforementioned feature subsets (wavelet coefficients 487 from electrodes #18, #8, phase synchrony, magnitude and 488 direction of speed of the propagating waves) for monkey 489 1, using LDA as a classifier for day 36. These four classi- 49 fiers were grouped using AdaBoost [33]. The classification 491 accuracy for the recordings of day 37 (i.e., the classifiers 492 trained and boosted from the recordings of day 36) were 493 around 83.7% on average (different results were obtained 494 for different sequences of classifier inclusion in greedy Ad- 495 aboost). This performance is lower than what was obtained 496 with the previous method. On the other hand, AdaBoost 497 allows us to assess the influence of each classifier (i.e., 498 for each feature subset) in the decoding process by their 499 differential contributions in the classification result, we ob- tained on average.5893 for electrode #18,.3489 for elec- 1 trode #8,.1834 for phase synchrony, and.1728 for the 2 waves. This shows once more the necessity of the spatial 3 features. 4 VIII. DISCUSSION 5 A. Decoding Based on LFPs 6 We developed a strategy for decoding rewarded from un- 7 rewarded stimuli from LFPs recorded with a microelectrode 8 array implanted in area V4 of the rhesus monkey. The use of 9 LFPs was justified by several reports in the literature, showing 51 equal or even slightly better decoding accuracy than decoders 511 based on single (multi) unit activity [9] [13]. In [9], it was 512 shown that hand movement target and velocity can be inferred 513 from multiple LFPs in single trials nearly as efficiently as from 514 multiple single-unit activity recorded from the same electrodes 515

20 8 TRANSACTIONS ON NEURAL NETWORKS 8 8 accuracy (%) 6 4 accuracy (%) Fig. 9. Classification accuracy as a function of training (horizontal axis) for monkey 1 (left) and monkey 2 (right panel). Blue, red, and black curves show the classification accuracy for the current day for the classifier trained on recordings from the previous day. The blue curve is for the classifier based on wave and synchrony features, the red curve for only time frequency features, and the black curve for wave, synchrony, and time frequency features. The magenta curves show the classification accuracy, after the reversal moment, of the classifier built on the last day before the reversal (thus, without retraining the classifier afterwards), the classifier uses the whole set of features (for wave, synchrony, and time frequency features). The red vertical line indicates the reversal moment. Yellow strips indicate the when recordings were made. in the monkey motor cortex. Pesaran and co-workers [1] found that LFP activity in the parietal cortex can be used for discriminating between preferred and anti-preferred directions of hand movement with the same accuracy as the spike rate (LFPs in a frequency band 25 9 Hz were used), and that it predicts the time of a planned movement with better accuracy than with the spike rate (with use of LFPs in a band 2 Hz). Encouraged by these findings, we also considered the LFP signal for decoding. It can be used as the sole source for decoding, or as an additional one to spikes (if they are available). In other LFP decoding studies, recordings were considered that were done sequentially with a single electrode at different recording sites [12], tetrode [1], or with four electrodes [9], or 2 5 electrodes [13], or eight electrodes [11] temporally inserted transdurally in implanted chambers. This is all different from our experiment, where we had a chronically implanted electrode grid (Utah array) with 96 recording sites. Chronic implantations are quite different from acute ones, where the recordings are done in one session, and whereby spikes are likely to be omnipresent. Mehring and co-workers [9] tested the influence of the correlation between simultaneously recorded LFPs on the decoding of hand movements. They observed that there was essentially no difference between the decoding power of simultaneously recorded LFPs (correlation present) and LFPs recorded on different but viewed as being simultaneously acquired (correlation absent). We argue against treating the sequential recordings as simultaneous ones, since it is not appropriate for real-time single-trial decoding. On the other hand, Mehring and co-workers used classifiers that are based on single electrode features that only rely on amplitude changes. This means that, contrary to our case, the classifier does not take into account the relation between electrode recordings. In our research, we have shown that spatial information in the array yields a gain in classification accuracy. B. Features for Decoding 552 In decoding stimulus-associated reward LFPs, we are the 553 first to use wavelet coefficients. The latter can be viewed as a 554 generalization of the previously used time frequency methods 555 in the decoding of invasive recordings, for LFP decoding, the 556 peak-to-peak amplitude between the negative and successive 557 positive peak [9], the subsampled time series amplitudes [9], 558 and the amplitudes in different frequency bands [11], [12] 559 have already been used. But instead of subtracting amplitudes 56 in particular frequency bands, we considered all coefficients 561 of the CWT, which yields a much better localization both 562 in the time and frequency domains [16], and extracted par- 563 ticular locations in the scalogram as features for decoding. 564 The wavelet-based features used in this paper are similar 565 to those used in noninvasive brain computer interfaces for 566 detecting event-related potentials in EEG recordings [34]. 567 We also share the application of the Student s t-statistics to 568 the coefficients of CWT, but we differ in the way feature 569 extraction is performed, we took the best separable locations 57 in the wavelet scalogram based on the t-value in every square 571 obtained after dividing the whole scalogram. Moreover, we 572 took the difference between the scalograms of the averaged 573 responses to the two stimuli and identified extremes in the 574 resulting scalogram. In addition, we also performed feature 575 selection to eliminate redundancy and to simplify the decoding 576 process afterwards, which again distinguishes our work from 577 [34]. From the single electrode features we constructed, we 578 can assert that we have a proper design, since in [34] the 579 authors show a better performance of their t-cwt classifier 58 compared to other methods [peak picking, area computa- 581 tion, discrete wavelet transform, and principal component 582 analysis (PCA)]. 583 We have shown a significant gain in classification accuracy, 584 compared to single electrode features, when additionally using 585 spatial features. Here, we should make a distinction be- 586 tween spatial filtering techniques and spatial features. Spatial 587

21 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX filtering techniques, such as PCA, independent component analysis, common spatial pattern (CSP) (for their comparison in connection with brain signal decoding, see [35]), construct linear spatial combinations of the electrode recordings, so that from these transformed signals features can be defined (for example, variances, as in CSP) for the classifier to work on. This is different from the use of spatial features that directly characterize the relations between several electrode recordings for the classifier to work on. In this paper, we considered the latter strategy. As a first spatial feature, phase synchronization was used. In spite of several studies on neural synchronization in brain areas (see [36], [37]), to our knowledge the index of phase synchrony has not yet been applied to LFP classification. For mental task decoding from EEG data, phase locking was applied in [38], and was shown to yield an increase in performance when used simultaneously with features from the power spectral density in the alpha band. Compared to the latter, we have justified our choice for selecting phase synchrony as a feature, by showing the time intervals where a difference in phase synchrony occurs in the responses to the two stimuli. These intervals correspond, putatively, to sensorial and attentional learning, respectively, as hypothesized in [14], based on the amplitudes of averaged recordings. As a second spatial feature, we have used wave propagation in the array. To the best of our knowledge, this is the first time this feature is used for decoding brain signals. The estimation of propagating waves in the Utah array was introduced in [15], which led to the discovery of waves (in the beta range) propagating in two opposite directions in the motor cortex. Similar to that study, we also have found two principal directions of propagating waves but now in a visual area (area V4) during stimulus reward association learning. But in addition to that, we tracked the waves behavior during training. We have found, again for the first time, that the direction of the propagating waves changes during training in a such way that after several it is possible to use wave direction for discriminating between stimuli differing in associated reward. We have also found that, for the rewarded stimulus, the velocity of the propagating wave becomes larger than for the unrewarded stimulus, after about 2 ms after stimuli onset. These two findings justify wave propagation as a feature for decoding brain signals. Complementary to that, we showed that the direction of propagating waves depends on the time after stimuli onset. In the interval between and ms after stimuli onset (i.e., the time interval possibly related to sensorial learning), we have a constant direction for both stimuli, before and after the stimulus reversal in the training paradigm (see Fig. 8). It is tempting to speculate that the propagating waves especially those during the later interval are related to the propagating feedback depolarization waves that were observed by Roland et al. [39] in the ferret s visual cortex. As a classifier, we have used LDA. Based on that, we have performed feature selection as a wrapper. It is quite possible that, with a nonlinear classifier (for example, kernel support vector machine), we would obtain a better classification performance. But the training of such a classifier, together with feature selection, will take much more time. 646 Hence, we have restricted ourselves to a simple LDA, leav- 647 ing open the possibility to obtain an even better single-trial 648 performance. 649 C. Perceptual Learning 6 With our study, we also confirmed previous findings [14] 651 on the effect of stimulus reward training on visual responses 652 in area V4. But, in addition to the mentioned study, where the 653 effect of such training was discovered in the time frequency 654 domain, we have shown the effects of stimulus reward training 655 in terms of changes in velocities and directions of propagating 656 waves, and in changes in the relation between simultaneous 657 recordings from different electrodes (phase synchrony). Frankó 658 et al. [14] left open the question whether the observed changes 659 in LFPs with stimulus reward training were local to V4 or 66 instead were due to volume conduction from other brain 661 areas. Several results of the present analysis strongly argue 662 against volume conduction but, in contrast, point to changes 663 of the local V4 response with stimulus reward associative 664 learning. Both the decrease of phase synchrony with increasing 665 interelectrode distance and the LFP wave propagation across 666 cortical surface suggest a local origin of the LFP. The latter 667 would not be present when the LFP measured in V4 were due 668 to volume conduction. Frankó et al. also stressed that, during 669 the course of the response, two apparent dissociable processes 67 occur, one, which they related to sensorial learning, is apparent 671 shortly after stimulus appearance, and a second one, which 672 they related to attentional effects, occurs later. The present 673 analysis support this distinction by showing that synchrony 674 and wave propagation features also differ between these two 675 response phases. 676 IX. CONCLUSION 677 We have developed a new strategy for decoding invasive 678 recordings made with a microelectrode array. Our strategy 679 was based on three types of features applied to LFPs: wavelet- 68 based features of individual electrodes; and two types of spatial 681 features, i.e., phase synchrony between electrode pairs and 682 wave propagation on the array. We have shown that the latter 683 two new features yield a significant improvement in single- 684 trial decoding accuracy and, therefore, could be candidate 685 LFP features for other decoding applications (for example, for 686 brain machine interfacing). From a neuroscience perspective, 687 the excellent single-trial classification performance of the 688 rewarded versus unrewarded stimuli indicates that stimulus 689 reward association has a profound effect on LFPs in the 69 extrastriate area V4. How these effects are related to spiking 691 activity requires further study. 692 ACKNOWLEDGMENT 693 The authors are deeply indebted to Dr. E. Frankó of the 694 Laboratorium voor Neuro- en Psychofysiologie, Katholieke 695 Universiteit Leuven, Leuven, Belgium, for sharing her exper- 696 imental data. 697

22 1 TRANSACTIONS ON NEURAL NETWORKS REFERENCES [1] M. A. Nicolelis, Brain-machine interface to restore motor function and probe neural circuits, Nat. Rev. Neurosci., vol. 4, no. 5, pp , May 23. [2] G.Santhanam,S.I.Ryu,B.M.Yu,A.Afshar,andK.V.Shenoy, A high-performance brain-computer interface, Nature, vol. 442, no. 799, pp , Jul. 26. [3] M. Velliste, S. Perel, M. C. Spalding, A. S. Whitford, and A. B. Schwartz, Cortical control of a prosthetic arm for self-feeding, Nature, vol. 453, no. 7198, pp , Jun. 28. [4] N. V. Manyakov and M. M. Van Hulle, Decoding grating orientation from microelectrode array recordings in monkey cortical area V4, Int. J. Neural Syst., vol. 2, no. 2, pp , Apr. 21. [5] G. B. Stanley, F. L. Fei, and Y. Dan, Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus, J. Neurosci., vol. 19, no. 18, pp , Sep [6] S. Musallam, B. D. Corneil, B. Greger, H. Scherberger, and R. A. Andersen, Cognitive control signals for neural prosthetics, Science, vol. 35, no. 5681, pp , Jul. 24. [7] B. Pesaran, S. Musallam, and R. A. Andersen, Cognitive neural prosthetics, Current Biol., vol. 16, no. 3, pp. R77 R8, Feb. 26. [8] G. Buzsáki, Large-scale recording of neuronal ensembles, Nat. Neurosci., vol. 7, no. 5, pp , May 24. [9] C. Mehring, J. Rickert, E. Vaadia, S. C. de Oliveira, A. Aertsen, and S. Rotter, Inference of hand movements from local field potentials in monkey motor cortex, Nat. Neurosci., vol. 6, no. 12, pp , Dec. 23. [1] B. Pesaran, J. S. Pezaris, M. Sahani, P. P. Mitra, and R. A. Anderson, Temporal structure in neural activity during working memory in macaque parietal cortex, Nat. Neurosci., vol. 5, no. 8, pp , Aug. 22. [11] J. Rickert, S. C. de Oliveira, E. Vaadia, A. Aertsen, S. Rotter, and C. Mehring, Encoding of movement direction in different frequency ranges of motor cortical local field potentials, J. Neurosci., vol. 25, no. 39, pp , Sep. 25. [12] H. Scherberger, M. R. Jarvis, and R. A. Anderson, Cortical local field potential encodes movement intentions in the posterior parietal cortex, Neuron, vol. 46, no. 2, pp , Apr. 25. [13] R. L. Spinks, A. Kraskov, T. Brochier, M. A. Umilta, and R. N. Lemon, Selectivity for grasp in local field potential and single neuron activity recorded simultaneously from M1 and F5 in the awake macaque monkey, J. Neurosci., vol. 28, no. 43, pp , Oct. 28. [14] E. Frankó, A. R. Seiz, and R. Vogels, Dissociable neural effects of long term stimulus-reward pairing in macaque visual cortex, J. Cogn. Neurosci., vol. 22, no. 7, pp , Jul. 21. [15] D. Rubino, K. A. Robbins, and N. G. Hatsopoulos, Propagating waves mediate information transfer in the motor cortex, Nat. Neurosci., vol. 9, no. 12, pp , Nov. 26. [16] I. Daubechies, Ten Lectures on Wavelets. Philadelphia, PA: SIAM, [17] G. Buzsáki, Rhythms of the Brain. London, U.K.: Oxford Univ. press, 26. [18] E. Vaadia, I. Haalman, M. Abeles, H. Bergman, Y. Prut, H. Slovin, and A. Aertsen, Dynamics of neuronal interactions in monkey cortex in relation to behavioural events, Nature, vol. 373, no. 6514, pp , Feb [19] R. Eckhorn, A. M. Gail, A. Bruns, A. Gabriel, B. Al-Shaikhli, and M. Saam, Different types of signal coupling in the visual cortex related to neural mechanisms of associative processing and perception, Trans. Neural Netw., vol. 15, no. 5, pp , Sep. 24. [2] R. Pashaie and N. H. Farhat, Self-organization in a parametrically coupled logistic map network: A model for information processing in the visual cortex, Trans. Neural Netw., vol. 2, no. 4, pp , Apr. 29. [21] A. Pikovsky, M. G. Rosenblum, and J. Kurths, Synchronization: A Universal Concept in Nonlinear Science. Cambridge, U.K.: Cambridge Univ. Press, 21. [22] G. C. Carter, Coherence and time delay estimation, Proc., vol. 75, no. 2, pp , Feb [23] J.-P. Lachaux, E. Rogriguez, J. Martinerie, and F. J. Varela, Measuring phase synchrony in brain signals, Human Brain Mapping, vol. 8, no. 4, pp , [24] B. Boashash, Estimating and interpreting the instantaneous frequency 769 of a signal. II. Algorithms and applications, Proc., vol. 8, no. 77 4, pp , Apr [25] F. Mormann, K. Lehnertz, P. David, and C. E. Elger, Mean phase 772 coherence as a measure for phase synchronization and its application 773 to the EEG of epilepsy patients, Physica D, vol. 144, nos. 3 4, pp , Oct [26] P. Tass, M. G. Rosenblum, J. Weule, J. Kurths, A. Pikovsky, J. Volk- 776 mann, A. Schnitzler, and H. J. Freund, Detection of n:m phase locking 777 from noisy data: Application to magnetoencephalography, Phys. Rev. 778 Lett., vol. 81, no. 15, pp , Oct [27] M. G. Rosenblum and A. Pikovsky, Detecting direction of coupling 78 in interacting oscillators, Phys. Rev. E, vol. 64, no. 4, pp , Oct [28] E. Pereda, R. Q. Quiroga, and J. Bhattacharya, Nonlinear multivariate 783 analysis of neurophysiological signals, Prog. Neurobiol., vol. 77, nos , pp. 1 37, Sep. Oct [29] M. A. Montemurro, M. J. Rasch, Y. Murayama, N. K. Logothetis, and 786 S. Panzeri, Phase-of-firing coding of natural visual stimuli in primary 787 visual cortex, Current Biol., vol. 18, no. 5, pp , Mar [3] N. V. Manyakov and M. M. Van Hulle, Synchronization in monkeys 789 visual cortex analyzed with information-theoretic measure, Chaos, vol , no. 3, pp , Sep [31] D. J. Fleet and A. D. Jepson, Computation of component image velocity 792 from local phase information, Int. J. Comput. Vis., vol. 5, no. 1, pp , Aug [32] A. G. Ivakhnenko and G. A. Ivakhnenko, The review of problem 795 solvable by algorithms of the group methods of data handling (GMDH), 796 Pattern Recogn. Image Anal., vol. 5, no. 4, pp , [33] Y. Freund and R. E. Schapire, A decision-theoretic generalization of 798 on-line learning and an application to boosting, J. Comput. Syst. Sci., 799 vol. 55, no. 1, pp , Aug [34] V. Bostanov and B. Kotchoubey, The t-cwt: A new ERP detection and 81 quantification method based on the continuous wavelet transform and 82 Student s t-statistics, Clin. Neurophysiol., vol. 117, no. 12, pp , Dec [35] G. Dornhege, M. Krauledat, K.-R. Müller, and B. Blankertz, General 85 signal processing and machine learning tools for BCI analysis, in 86 Toward Brain-Computer Interfacing, G. Dornhege, J. R. Millán, T. 87 Hinterberger, D. J. McFarland, K.-R. Müller, Eds. Cambridge, MA: MIT 88 Press, 27, pp [36] P. Fries, J.-H. Schroeder, P. R. Roelfsema, W. Singer, and A. K. 81 Engel, Oscillatory neural synchronization in primary visual cortex as a 811 correlate of stimulus selection, J. Neurosci., vol. 22, no. 9, pp , May [37] L. Melloni, C. Molina, M. Pena, D. Torres, W. Singer, and E. Rodriguez, 814 Synchronization of neural activity across cortical areas correlates with 815 conscious perception, J. Neurosci., vol. 27, no. 11, pp , Mar [38] E. Gysels and P. Celka, Phase synchronization for the recognition of 818 mental tasks in a brain-computer interface, Trans. Neural Syst. 819 Rehabil. Eng., vol. 12, no. 4, pp , Dec [39] P. E. Roland, A. Hanazawa, C. Undeman, D. Eriksson, T. Tompa, 821 H. Nakamura, S. Valentiniene, and B. Ahmed, Cortical feedback 822 depolarization waves: A mechanism of top-down influence on early 823 visual areas, Proc. Natl. Acad. Sci., vol. 13, no. 33, pp , 824 Aug Nikolay V. Manyakov received the M.Sc. degree in 826 mathematics from the Belarussian State University, 827 Minsk, Belarus, in 1998, and the Ph.D. degree in the- 828 oretical informatics from the Belarussian State Uni- 829 versity of Informatics and Radioelectronics, Minsk, 83 in He is a Research Fellow at the Computational 832 Neuroscience Group, Laboratorium voor Neuro- en 833 Psychofysiologie, Katholieke Universiteit Leuven 834 Medical School, Leuven, Belgium. His current re- 835 search interests include computational neuroscience, 836 neural networks, machine learning, data mining, and signal processing. 837

23 MANYAKOV et al.: DECODING STIMULUS REWARD PAIRING FROM LOCAL FIELD POTENTIALS RECORDED FROM MONKEY VISUAL CORTEX Rufin Vogels received the Ph.D. degree in experimental and clinical psychology from the Katholieke Universiteit Leuven (K. U. Leuven), Leuven, Belgium, in His Ph.D. thesis was on psychophysical studies on human visual orientation discrimination. He was a Post-Doctoral Research Fellow in Peter Schiller s Laboratory at Massachusetts Institute of Technology, Boston, recording instriate and extrastriate cortexes of behaving monkeys. Currently, he is a Professor at the Division of Neurophysiology, K. U. Leuven Medical School. His current research interests include visual cortical processing in awake behaving monkeys using single-cell recording and other invasive techniques. Marc M. Van Hulle (M 97 SM ) received the 852 M.Sc. degree in electrotechnical engineering and 853 the Ph.D. degree in applied sciences from the 854 Katholieke Universiteit Leuven (K. U. Leuven), 855 Leuven, Belgium. He also received the B.Sc.Econ. 856 and M.B.A. degrees. He received the Doctor 857 Technices degree from Queen Margrethe II of 858 Denmark, in 23, and an Honorary Doctoral 859 degree from Brest State University, Brest, Belarus, 86 in He is currently a Full Professor at the K. U. 862 Leuven Medical School, where he heads the Computational Neuroscience 863 Group of the Laboratorium voor Neuro- en Psychofysiologie. In 1992, he 864 was with the Brain and Cognitive Sciences Department, Massachusetts 865 Institute of Technology, Boston, as a Post-Doctoral Scientist. He has authored 866 a monograph titled Faithful Representations and Topographic Maps: From 867 Distortion- to Information-Based Self-Organization (John Wiley, 2; also 868 translated into Japanese) and 2 technical publications. His current research 869 interests include computational neuroscience, neural networks, computer 87 vision, data mining, and signal processing. 871 Dr. Van Hulle is an Executive Member of the Signal Processing 872 Society, Associate Editor of TRANSACTIONS ON NEURAL NETWORKS, 873 Computational Intelligence, and Neuroscience and International Journal 874 of Neural Systems. He is a member of the program committees of several 875 international conferences on machine learning, informatics, information 876 processing, etc. 877