arxiv: v1 [cs.ne] 10 Sep 2018

Transcription

1 Fast and Efficient Information Transmission with Burst Spikes in Deep Spiking Neural Networks Seongsik Park, Seijoon Kim, Hyeokjun Choe, Sungroh Yoon Department of Electrical and Computer Engineering Seoul National University Seoul 08826, Korea arxiv: v1 [cs.ne] 10 Sep 2018 Abstract The spiking neural networks (SNNs), the 3rd generation of neural networks, are considered as one of the most promising artificial neural networks due to their energy-efficient computing capability. Despite their potential, the SNNs have a limited applicability owing to difficulties in training. Recently, conversion of a trained deep neural network (DNN) model to an SNN model has been extensively studied as an alternative approach. The result appears to be comparable to that of the DNN in image classification tasks. However, rate coding, one of the techniques used in modeling the SNNs, suffers from long latency due to its inability to transmit sufficient information to a subsequent neuron and this could have a catastrophic effect on a deeper SNN model. Another type of neural coding, called phase coding, also determines the amount of information being transmitted according to a global reference oscillator, and therefore, is inefficient in hidden layers where dynamics of neurons can change. In this paper, we propose a deep SNN model that can transmit information faster, and more efficiently between neurons by adopting a notion of burst spiking. Furthermore, we introduce a novel hybrid neural coding scheme that uses different neural coding schemes for different types of layers. Our experimental results for various image classification datasets, such as MNIST, CIFAR-10 and CIFAR-100, showed that the proposed methods can improve inference efficiency and shorten the latency while preserving high accuracy. Lastly, we validated the proposed methods through firing pattern analysis. Introduction With the recent development of deep learning, deep neural networks (DNNs) have shown outstanding performance in various applications. The DNNs, however, demand high computational power of GPUs which raises concerns for its high energy consumption when deployed in a mobile environment. Recent extensive studies performed to overcome such challenges have focused primarily on two approaches; one is reducing size of the model by quantization (Han, Mao, and Dally 2015; Park et al. 2018b) or pruning (Han et al. 2015), and another is improving the efficiency of computations by designing a dedicated hardware accelerator (Han et al. 2016; Park et al. 2018a). Copyright c 2019, Association for the Advancement of Artificial Intelligence ( All rights reserved. Although these efforts have been able to increase computational efficiency of the DNNs, they have failed to fully take into account the efficiency and capability of information processing which takes place in the human brain. To bridge the gap between the human brain and artificial neural networks, spiking neural networks (SNNs) have been studied as the 3rd generation of the artificial neural networks (Maass 1997). Unlike the DNNs, neurons in the SNNs can transmit information through a spike train, which is a series of spikes that contains temporal information. In addition, a neuron integrates all the information received from synapses, and then generates a spike if the integrated value exceed a certain threshold. This is recognized as one of the most common types of neurons, called integrate-and-fire (IF) neuron model. The SNNs can exploit event-driven computation with IF neurons, which leads to much higher computational efficiency of the neuromorphic architecture (Merolla et al. 2011; Merolla et al. 2014). Thus, SNNs have potentials to process information rapidly and efficiently. Despite their possibilities, the SNNs have not shown comparable results with the DNNs in various tasks. One of the major reasons that SNNs have a limited applicability, is the difficulty in training. Spike-timing dependent plasticity (STDP) is a well-known training algorithm for the SNNs. It is possible to train shallow SNNs consisting of 2 or 3 layers, using the STDP (Diehl and Cook 2015). However, there have been only a few studies regarding training deep SNNs using the STDP. In order to train deep SNNs, a training method for SNNs with error back-propagation, which is a well-known training algorithm for DNNs, has been proposed (Lee, Delbruck, and Pfeiffer 2016). However, the applicability of this method is limited to relatively simple image classification tasks, such as that involving the MNIST dataset. Since it is challenging to directly train deep SNNs, several studies have proposed methods of converting the trained DNN model to the SNN model (Cao, Chen, and Khosla 2015; Diehl et al. 2015; Rueckauer et al. 2016; Rueckauer et al. 2017; Kim et al. 2018). These approaches aim at energy-efficiency inference in the SNNs by utilizing welltrained weights in the DNNs. Through these methods, the SNNs have shown comparable inference accuracy with that of DNNs in various tasks, especially image classification task.

2 Most of the related studies have focused on reducing the accuracy loss caused by the conversion of DNNs to SNNs. A few studies have considered the latency and efficiency of inference in SNNs (Kim et al. 2018; Park et al. 2016; Diehl et al. 2015). In addition, to the best of our knowledge, deep SNNs (Kim et al. 2018) is the only relevant study utilizing the temporal information of spike trains in deep SNNs. Their method was successful in delivering information quickly, but the large number of spikes lowered the efficiency of inference. In this paper, we propose a fast and efficient method to transmit the information between subsequent neurons in deep SNNs. The proposed method consists of a neural model for generating burst spikes and a layer-wise hybrid neural coding scheme. We were inspired by the neuroscience research that burst spikes enhance the efficiency of the information transmission (Connors and Gutnick 1990; Izhikevich et al. 2003; Lisman 1997), and the human brain generated different spike patterns depending on the role of the neurons (Mochizuki et al. 2016). We analyzed the effect of the hybrid neural coding scheme through firing pattern analysis. By using our method, we can improve the accuracy and efficiency of the deep SNNs in terms of latency and number of spikes in image classification tasks. The contributions of this paper can be summarized as follows: We propose a spiking neuron model that exploits burst spikes to improve the efficiency of information transmission between subsequent neurons in the deep SNNs. This is the first attempt to apply the concept of the burst spiking to the deep SNNs. We suggest a layer-wise hybrid neural coding scheme depending on the functions of different types of layer. We demonstrate the effects of the hybrid neural coding scheme through analysis of spike patterns in the deep SNN. Background and Related Work Spiking Neural Networks The SSNs are composed of neurons which generate spike trains and synapses which allow a neuron to transmit spike trains to another neuron. Among the various neuron models, the IF model is one of the most widely used models in the SNNs. An IF neuron model integrates post-synaptic potential (PSP) into the membrane potential (V mem ). The integrated PSP of the jth neuron in the lth layer at time t is described as zj(t) l = wijθ l l-1 i (t) + b l j, (1) i where zj l(t) is the sum of PSPs, wl ij is the synapse weight between ith neuron in l-1th layer and jth neuron in lth layer, (t) is a spike in the ith neuron in l-1th layer, and b l j is the bias. If the membrane potential exceeds a certain threshold (V th ), the neuron generates a spike. The spike generating process of the ith neuron in the lth layer at time t can be formulated as follows: Θ l i(t) = U(Vmem,i(t l 1) + zi(t) l V th (t)), (2) Θ l-1 i where U(x) is a unit step function for representing the output spike, z is the sum of PSPs, and V th is the threshold. If an IF neuron fires a spike, then the membrane potential is set to the resting potential (V rest ), otherwise the original potential is maintained. This dynamics of the membrane potential is described as { V Vmem,j(t) l l = rest,j (t) if Θ l j (t) = 1 Vmem,j l (t 1) + zl j (t) otherwise. (3) In SNNs, information is transmitted through spike train and processed only when spike occurs. This event-driven feature of SNNs improves computational energy efficiency of the neuromorphic architecture (Merolla et al. 2011; Merolla et al. 2014). Despite the efficient processing ability, the SNNs have not yet been widely applied due to the difficulty in training. Neural Coding Neural coding is an encoding scheme of the information transmitted from one neuron to another. Various types of neural coding have been extensively studied for decades, with rate and temporal coding (Adrian 1926; Gautrais and Thorpe 1998; Thorpe, Delorme, and Van Rullen 2001; Thorpe 1990; Kayser et al. 2009) being the well-known ones from recent years. Rate coding is based on spike firing rate and has been widely used in several studies recently (Diehl et al. 2015; Rueckauer et al. 2017; Rueckauer et al. 2016). It relies solely on the number of spikes over time and thus, is highly inefficient in terms of information capacity and transmission speed (Gautrais and Thorpe 1998). For example, in order to represent 8 bits, 2 8 = 256 time steps are required. Consequently, not only a higher stimulus intensity generates more spikes, but also neurons must wait until the last spike of the information arrives, before they transmit the received information to the next layer. In addition, it induces the same size of PSP on each spike as shown in Fig. 1-B1. This leads to a higher latency and energy consumption in deep SNNs. Temporal coding is based on the temporal information in spike trains and includes time-to-first-spike (Thorpe, Delorme, and Van Rullen 2001), rank-order coding (Thorpe 1990), and phase coding (Kayser et al. 2009). It has been proven to be effective in information transmission. However, to the best our knowledge, phase coding has only been successfully applied to deep SNNs (Kim et al. 2018). Phase coding uses a periodic oscillation function as a global reference (Fig. 1-A2). It produces PSP according to the phase of the oscillation function (Fig. 1-B2). Burst spiking is an extremely diverse spike pattern that contains a group of short inter-spike interval (ISI) spikes as shown in Fig. 1-A3. Typically, burst spiking is known to be more reliable and carries more information than a typical spike train. For example, it can dynamically induce synaptic potentiation which can improve information transmission efficiency as described in Fig. 1-B3. Given this context, burst spikes can be applied to deep SNN to improve efficiency of inference and reduce inference time.

3 Spike Train Rate coding Phase coding A1 N spikes firing rate=n/t A2 Oscillator A3 time time T Burst spiking Burst spikes time B1 B2 B3 PSP C1 C2 C3 ISIH Count Count Count Inter-spike interval (ISI) Inter-spike interval (ISI) Inter-spike interval (ISI) Figure 1: (A) Spike train, (B) post-synaptic potentiation (PSP), and (C) inter-spike interval histogram (ISIH) of the integrateand-fire (IF) neurons depending on the various types of neural coding. Conversion Methods of DNNs to SNNs The SNNs trained by STDP or error back-propagation have not shown comparable results to those of DNNs (Lee, Delbruck, and Pfeiffer 2016; Diehl and Cook 2015). To take advantage of the energy-efficient information processing capability of the deep SNNs, DNN to SNN conversion methods have been studied recently (Cao, Chen, and Khosla 2015; Diehl et al. 2015; Rueckauer et al. 2016; Rueckauer et al. 2017; Kim et al. 2018). The conversion method is based on a concept that imports weights of back-propagation DNNs into the SNNs. It benefits from the training ability of the DNNs and combines this with the computational efficiency of the SNNs. To minimize accuracy degradation of the converted SNN, (Cao, Chen, and Khosla 2015) proposed a constrained model that is a suitable fit for the SNN model. The constrained model removed biases and used average-pooling instead of max-pooling. The authors then retrained the constrained model and applied retrained weights to SNN. This method, however, is not generally applicable to DNNs containing a bias term or max-pooling layers. Weight normalization was introduced to improve performance of deep SNNs in terms of accuracy and latency (Diehl et al. 2015). The authors approximated DNN s activation value to SNN s firing rate through ReLU activation and data-based weight normalization. This approach showed improvement in inference speed and accuracy for the MNIST dataset, however, it is only applicable to constrained models. To resolve such issues, the authors of (Rueckauer et al. 2016; Rueckauer et al. 2017) proposed normalized bias and spiking max pooling. In addition, they proposed a novel normalization method that is robust to outliers. They used resetby-subtraction approach for preventing performance degradation caused by reset function of membrane potential. The resting potential (V rest ) is stated as V l rest,j(t) = V l mem,j(t 1) + z l j(t) V l th,j(t)θ l j(t), (4) and the sum of PSPs to reduce information loss between neurons is represented as zj(t) l = wijv l th,j(t)θ l l-1 i (t) + b l j. (5) i The results showed similar accuracy to that of DNN for on MNIST and CIFAR-10 datasets. However, their approach of using rate coding may not fully exploit the temporal information in spike trains for faster information transmission and they might have undermined the importance of the efficiency in relation to the number of spikes. A recent study emphasized the efficiency (latency and number of spikes) of deep SNNs while achieving DNNs accuracy (Kim et al. 2018). Most of the prior studies suffered from long inference latency because of rate coding such that they failed to exploit the temporal information in spike train. In the study by (Kim et al. 2018), weighted spike has been proposed to utilize temporal information (we will refer to this method as phase coding in the rest of the paper). The oscillation function of their phase coding is given by Π(t) = 2 (1+mod(t,k)), (6) where k is period of the phase coding. They used the PSP function shown in Eq. 5 and adjusted V th (t) according to the oscillation function as V l th,i(t) = Π(t)v th, (7)

4 70% 60% 50% 40% 30% 20% 10% 0% burst length = 2 burst length = 3 burst length = 4 burst length = 5 burst length > 5 % of burst spikes (v th ) Figure 2: Percentage of burst spikes for various v th as shown in Eq. 9, and their composition by different burst lengths Table 1: Experimental results on CIFAR-10 with various configurations of neural coding in input and hidden layers (model: VGG-16, and test accuracy of DNN model: %) Input Hidden Accuracy (%) Latency # of spikes (10 6 ) real rate , real phase , real burst , rate rate , rate phase , rate burst , phase rate , phase phase , phase burst , where v th is a threshold constant, which exerts the same effect on synaptic plasticity that shown in Eq. 5. As a result, this method was able to perform image classification with lower latency than other SNN models with rate coding. However, their method can only transmit bounded information which can cause bottleneck in hidden layers where neural dynamics change. In addition, more spikes are generated because this method cannot change the transmission amount adaptively. This increases computational overhead especially when executed in an event-driven manner. Thus, there is still a need for novel inference methods that can reduce latency and number of spikes, while improving inference accuracy. Proposed Methods Neural Model for Burst Spikes As mentioned before, the rate and phase coding lack efficiency in transmitting information such that they do not fit well in the deep SNNs. In this paper, we propose a methodology for achieving fast and efficient inference in the deep SNNs with burst spikes. Our proposed method is motivated by the fact that burst spikes are adequate for transmitting information efficiently (Connors and Gutnick 1990; Izhikevich et al. 2003; Lisman 1997). To implement the functionality of burst spiking, we defined the burst function as follows: { βg gi(t) l l = i (t 1) if Θ l i (t 1) = 1 (8) 1.0 otherwise, where β is a burst constant. To obtain the effect of burst spikes, we applied the burst function to Eq. 5 through threshold adaptation which is depicted as V l th,i(t) = g l i(t)v th. (9) The burst spikes induce synaptic potentiation, which improves transmitting efficiency (Krahe and Gabbiani 2004). Through Eqs. 5 and 9, we can determine effective weights (ŵij l (t)) due to burst spikes as ŵ l ij(t) = w l ijg l i(t). (10) The effective weights can be interpreted as the synaptic potentiation induced by the burst spikes. A burst is composed of a group of short-isi spikes. Thus, we can verify the occurrence of the burst spikes indirectly through the ISI histogram (ISIH). In Fig. 1, C1, C2, and C3 illustrate the ISIH of rate coding, phase coding (Kim et al. 2018), and proposed method for generating burst spikes, respectively. As presented in Fig. 1-C1 (rate coding) and C3 (proposed method), the ratio of short-isi spikes was considerably increased when the proposed method was applied. In the case of phase coding (Fig. 1-C2), the ratio of short- ISI spikes was higher than that of the proposed method. The phase coding is, however, inefficient in the information transmission between neurons, because it cannot utilize the ISI information. The proposed approach can control the precision of the information being transmitted according to the value of v th. Fig. 2 shows the total number of burst spikes and their composition by different burst lengths as v th varies. We observed that, as the v th decreases, the burst spike rate increases and long periods of bursts occur more frequently. To transmit more precise information, significantly more number of spikes would be required for the same amount of information. Thus, in the proposed method, a trade-off exists between precision and total number of spikes. Throughout the rest of the paper, we will refer to this method as burst coding. Hybrid Neural Coding Scheme in Deep SNNs for Image Classification Most of the previous studies have used only one neural coding method in the entire deep SNN (Cao, Chen, and Khosla 2015; Diehl et al. 2015; Kim et al. 2018). Only (Rueckauer et al. 2016; Rueckauer et al. 2017) used different neural coding schemes in input and hidden layers, but they failed to provide adequate analysis of the neural coding used in their work. Using one neural coding may lead to difficulty in generating a spike pattern that is optimized for different layers in the deep SNNs. According to (Mochizuki et al. 2016), neurons occasionally use different neural coding schemes depending on their role and location in the brain. Motivated by this observation, we propose a hybrid coding scheme that exploits different characteristic of the layers in the deep SNNs. To find an appropriate combination of neural coding for image classification, we divided the layers in the deep SNNs

5 Time step Hidden Input Target Acc. Latency # of spikes (M) / image rate burst phase burst rate phase burst phase burst rate phase burst phase burst real phase real phase real phase 91 % % % Figure 3: Latency and number of generated spikes to reach the target accuracy of various neural coding schemes into two types, namely, input and hidden layers. The input layer converts continuous values of input image to discrete spikes. For an image classification task, the values of inputs are static and bounded. Thus, neural coding schemes suitable for the input layer are real and phase coding which are capable of delivering static and bounded information quickly and accurately. On the other hand, rate coding is not a suitable neural coding scheme for the input layer for fast classification. Theoretically, rate coding needs 2 k time steps for representing a k-bit fixed-point value, whereas phase coding can convert it within a given period of k. Table 1 describes the impact of neural coding scheme in the input and hidden layer on image classification. In input layer s neural coding, real and phase coding accomplished maximum accuracy of 91.34% and 91.41%, respectively. Rate coding failed to reach the DNN s accuracy in 1,500 time steps, because the input layer became a bottleneck while transmitting information. The difference between the neurons in the input layer and hidden layers lies in whether the information transmitted changes dynamically. The important property of burst coding is the ability to dynamically adjust to the different types of information being transmitted. Consequently, the burst coding is a suitable neural coding scheme for the neurons in hidden layers. Burst coding can dynamically determine the capacity of the transmission in an unbounded range, and thus, is the appropriate coding scheme for hidden layers. As was noted earlier, rate coding struggles to transmit information as more hidden layers are stacked in the deep SNNs. In our experiment, the SNN model with the rate coding in hidden layer failed to reach a comparable accuracy of the DNN model in the given time steps. Phase coding has a limit on the amount of information that it can transmit, and requires more spikes for the same amount of information. This disadvantage of the phase coding is illustrated in the # of spikes column in Table. 1. Thus, rate and phase coding lead to degradation of classification efficiency when they are applied to the hidden layers # of spikes (10 6 ) Accuracy (%) real-burst real-rate real-phase rate-burst rate-rate rate-phase phase-burst phase-rate phase-phase Time step Figure 4: Inference curve of various neural coding schemes Applying burst coding to the hidden layer yields the highest accuracy, regardless of the neural coding scheme used in the input layer (the accuracy column in Table. 1). This proves that the hidden neurons in burst coding scheme can generate spike patterns adaptively in order to transmit information efficiently. Overall, the best combination is phase coding for the input layer and burst coding for the hidden layer, which achieves the same accuracy as DNN (91.41%) while taking less time steps and generating less spikes than other approaches achieving similar accuracy. For the rest of the paper, we use a notation, input-hidden to indicate neural coding of input layer and hidden layer. For instance, phase-burst denotes that the phase and burst coding are used in input and hidden layers, respectively. Experimental Results We evaluated various neural coding schemes using the image classification tasks, such as MNIST, CIFAR-10, and CIFAR-100, with deep SNNs. We set accuracy, number of spikes, and latency of inference as the evaluation criteria. All the implementations are based on TensorFlow and SNN library 1. Experimental Results for Various Neural Coding Schemes To assess the proposed methods, we measured the inference latency in time steps and the number of generated spikes to reach target accuracy for various configurations of neural coding. We set three types of the target accuracy: 91.0%, 90.49%, and 86.83%. In these experiments, we trained VGG-16 on the CIFAR-10 dataset, and then converted the trained model to the SNN. Fig. 3 presents the latency in time steps and number of spikes required to reach the accuracy for various coding schemes. It should be noted that we excluded the results of some neural coding schemes from the graph because they failed to reach the target accuracy in 1,500 time steps. The input-hidden configurations for reaching target accuracy are in the orders of real-burst, phase-burst, and phase- 1 A public version of the code will be available.

6 Table 2: Comparison of inference results with other SNN methods on various image classification datasets (MNIST, CIFAR-10, and CIFAR-100) Methods MNIST Neural Coding Model Number of Accuracy (%) Latency Spikes/image Spiking Input Hidden neurons DNN SNN # (10 6 ) overhead (Diehl et al. 2015) rate rate CNN c 22, (Kim et al. 2018) phase phase CNN c 22, Our Method (v th = 0.125) real burst CNN c 22, Our Method (v th = 0.125) real burst CNN c 22, CIFAR-10 (Cao, Chen, and Khosla 2015) rate rate CNN 38, (Rueckauer et al. 2016) real rate CNN 118, (Rueckauer et al. 2016) b real rate VGG , , (Kim et al. 2018) phase phase CNN 118, (Kim et al. 2018) phase phase ResNet (Kim et al. 2018) b phase phase VGG , , Our Method (v th = 0.125) phase burst VGG , , Our Method (v th = ) phase burst VGG , CIFAR-100 (Kim et al. 2018) phase phase ResNet (Kim et al. 2018) b Our Method (v th = 0.125) phase phase phase burst VGG-16 VGG , , ,000 3, a Spiking overhead := # of spikes per image / (# of Neurons Latency) b our experimental results c CNN: 12c5-2s-64c5-2s-10 phase, and for the number of spikes generated, phase-burst, real-rate, real-burst, phase-phase, and real-phase. The phaseburst had the least number of spikes to reach target accuracy. Thus, the hidden neurons showed the best performance overall when burst coding was applied, while the maximum number of spikes were generated when phase coding was used. The one thing to be noted here is that the real-rate configuration increased the latency rapidly as the target accuracy increased. Another finding suggests that there is a trade-off between the latency and the number of spikes when a hidden neuron uses burst coding depending on the configurations between real-burst and phase-burst. The inference curves for various configurations are illustrated in Fig. 4. The deep SNNs with rate coding in the input layer showed slower convergence speed than the others. In the hidden layer, burst coding yields rapid convergence speed. The combination of phase coding and rate coding resulted in poor performance. The hybrid coding scheme (real-burst) showed the fastest convergence speed. This led us to posit that the burst coding and hybrid coding scheme empower the deep SNNs in many aspects, namely, speed, accuracy, and efficiency. Comparison with Other Deep SNN Algorithms To validate our proposed burst coding and hybrid coding scheme, we compared them with previous works on MNIST, CIFAR-10, and CIFAR-100 datasets in terms of classification accuracy, latency and number of spikes, as shown in Table. 2. Regardless of which neural coding scheme is used, the SNN achieved comparable results with that of DNN for the MNIST dataset. We applied rate coding combined with data-based normalization (Rueckauer et al. 2016) to the VGG-16 model, and achieved 91.06% accuracy. When the burst coding and hybrid neural coding scheme are applied to the VGG-16 model, 91.41% classification accuracy is achieved in only 1,125 time steps. This is faster and more accurate than the model with phase coding in both input and hidden layers. The number of spikes generated using phase coding was greater than that obtained using any other method, which has also been shown by (Kim et al. 2018). These results confirm the shortcomings of phase coding. Unlike the DNN, the SNN enables temporal processing and requires a certain amount of time to transmit spikes from one neuron to another. Eventually, the inference time step plays a critical role on in the classification accuracy of the model. It is also challenging to fairly compare SNNs efficiency because each method uses different DNN models, resulting in different accuracy. Thus, we defined spiking overhead to evaluate the efficiency. We obtained the spiking overhead by calculating the expected value of the number of spikes generated in a neuron per time step. The result indicated that burst coding has the lowest spiking overhead of , while achieving roughly 29 in the study by (Kim et al. 2018). In the our experiment with the method of (Rueckauer et al. 2016), the spiking overhead (0.0222) was similar to that of the proposed method, but the latency was much longer and the accuracy was lower than those of our method. Thus, our findings suggest that the proposed method can transmit information more efficiently. Discussion The neural spike pattern analysis typically aims to analyze neurons characteristic and their neural coding scheme in

7 Firing regularity (<κ>) real-burst rate-burst phase-burst real-rate rate-rate phase-rate real-phase rate-phase phase-phase Firing rate (<log λ>) Figure 5: Firing rate-regularity graph for various neural coding schemes depth (Mochizuki et al. 2016). To analyze spike patterns of the proposed burst coding and hybrid neural coding scheme, firing characteristics were determined in our experiments. We calculated firing rate and regularity of spike trains as the firing characteristics for each neural coding scheme. The spike trains were measured from randomly sampled 10% of the neurons in each layers for 10,000 time steps. The firing rate of a neuron (λ) can be defined as n λ = n i=1 I, (11) i where I i is the duration of the ith ISI, and n is the number of ISIs in a spike train. We calculated the average log firing rate over sampled neurons in the layer as follows: m j=1 < logλ >= logλ j, (12) m where λ j is the firing rate of the jth neuron in the layer, and m is the number of sampled neurons in the layer. Firing regularity is defined by the regularity of the spike train generated. We used the coefficient of variation (CV) to obtain the firing regularity in a spike train. It is 0 for a completely regular spike train and becomes larger if spike train is more irregular. The firing regularity of a neuron (κ) is obtained as follows: κ = std(i) mean(i), (13) and the regularity of a layer is given by m j=1 < κ >= κ j m, (14) which is the averaged firing regularity over sampled neurons in the layer. A firing rate-regularity graph for various neural coding schemes is shown in Fig. 5. The colors and shapes of each dot indicate various neural coding schemes of the input and hidden layers, respectively. Assuming that the same colored dots form a cluster, there are three clusters in our experiments (red, yellow, and blue clusters). Fig. 5 reveals that each sub-cluster (shapes) tends to be plotted at a similar location in each cluster (colored). Sub-clusters of circle dots (the real value in input layer) are located on the bottom-right of each colored cluster. This can be interpreted to imply that the real value in the input layer can promote spiking of the neurons in the hidden layer. In addition, sub-clusters of triangle dots (phase coding in the input layer) are located on the bottom-left of each colored cluster, which means that phase coding in the input layer inhibits spike firing of the hidden neurons. Unlike the above two cases the rate coding in the input layer does not affect the spiking. Instead, it tends to make the spike train irregular. Based on this analysis, we can conclude that the neural coding of the input layer influences the spike pattern generation in the hidden neurons. When analyzing the characteristics of the colored clusters, we found that the phase coding in the hidden layer (blue dots) has the highest firing rate regardless of the neural coding method used in the input layer. This illustrates the low flexibility of phase coding; consequently, the phase coding is not suitable in the hidden layer for the hybrid neural coding method. On the contrary, the rate coding in the hidden layer (yellow dots) can lead to flexibility depending on the coding scheme in the input layer. It is not able to transmit information quickly at a low firing rate, such that classification latency may be longer than that for other neural coding schemes. The burst coding in the hidden layer (red dots) tends to have the largest flexibility depending on the coding method in the input layer. This means that burst coding is well-adapted to different coding schemes in the input layer. Thus, it is a suitable coding scheme for the hidden layer of the hybrid neural coding scheme. Conclusion In this paper, we analyzed the effect of various neural coding schemes in deep SNNs on the performance in an image classification task. Our analysis revealed that rate coding lacks the ability to transmit information efficiently while phase coding has low computational efficiency due to the generation of an excessive number of spikes. Based on our analysis, we propose burst coding and the hybrid neural coding scheme, which are methods inspired by human brain, to improve the accuracy and efficiency of image classification. The proposed methods not only achieve reasonable accuracy, but also substantially reduce the number of spikes generated by a neuron per hour, which leads to higher inference efficiency of the SNN model. We expect that the proposed methods can be the applied in deep SNNs for other tasks to improve the inference efficiency.

8 References [Adrian 1926] Adrian, E. D The impulses produced by sensory nerve endings. The Journal of physiology 61(1): [Cao, Chen, and Khosla 2015] Cao, Y.; Chen, Y.; and Khosla, D Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision 113(1): [Connors and Gutnick 1990] Connors, B. W., and Gutnick, M. J Intrinsic firing patterns of diverse neocortical neurons. Trends in neurosciences 13(3): [Diehl and Cook 2015] Diehl, P. U., and Cook, M Unsupervised learning of digit recognition using spike-timingdependent plasticity. Frontiers in computational neuroscience 9:99. [Diehl et al. 2015] Diehl, P. U.; Neil, D.; Binas, J.; Cook, M.; Liu, S.-C.; and Pfeiffer, M Fast-classifying, highaccuracy spiking deep networks through weight and threshold balancing. In Neural Networks (IJCNN), 2015 International Joint Conference on, 1 8. IEEE. [Gautrais and Thorpe 1998] Gautrais, J., and Thorpe, S Rate coding versus temporal order coding: a theoretical approach. Biosystems 48(1-3): [Han et al. 2015] Han, S.; Pool, J.; Tran, J.; and Dally, W Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, [Han et al. 2016] Han, S.; Liu, X.; Mao, H.; Pu, J.; Pedram, A.; Horowitz, M. A.; and Dally, W. J Eie: efficient inference engine on compressed deep neural network. In Computer Architecture (ISCA), 2016 ACM/IEEE 43rd Annual International Symposium on, IEEE. [Han, Mao, and Dally 2015] Han, S.; Mao, H.; and Dally, W. J Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arxiv preprint arxiv: [Izhikevich et al. 2003] Izhikevich, E. M.; Desai, N. S.; Walcott, E. C.; and Hoppensteadt, F. C Bursts as a unit of neural information: selective communication via resonance. Trends in neurosciences 26(3): [Kayser et al. 2009] Kayser, C.; Montemurro, M. A.; Logothetis, N. K.; and Panzeri, S Spike-phase coding boosts and stabilizes information carried by spatial and temporal spike patterns. Neuron 61(4): [Kim et al. 2018] Kim, J.; Kim, H.; Huh, S.; Lee, J.; and Choi, K Deep neural networks with weighted spikes. Neurocomputing. [Krahe and Gabbiani 2004] Krahe, R., and Gabbiani, F Burst firing in sensory systems. Nature Reviews Neuroscience 5(1):13. [Lee, Delbruck, and Pfeiffer 2016] Lee, J. H.; Delbruck, T.; and Pfeiffer, M Training deep spiking neural networks using backpropagation. Frontiers in neuroscience 10:508. [Lisman 1997] Lisman, J. E Bursts as a unit of neural information: making unreliable synapses reliable. Trends in neurosciences 20(1): [Maass 1997] Maass, W Networks of spiking neurons: the third generation of neural network models. Neural networks 10(9): [Merolla et al. 2011] Merolla, P.; Arthur, J.; Akopyan, F.; Imam, N.; Manohar, R.; and Modha, D. S A digital neurosynaptic core using embedded crossbar memory with 45pj per spike in 45nm. In Custom Integrated Circuits Conference (CICC), 2011 IEEE, 1 4. IEEE. [Merolla et al. 2014] Merolla, P. A.; Arthur, J. V.; Alvarez- Icaza, R.; Cassidy, A. S.; Sawada, J.; Akopyan, F.; Jackson, B. L.; Imam, N.; Guo, C.; Nakamura, Y.; et al A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345(6197): [Mochizuki et al. 2016] Mochizuki, Y.; Onaga, T.; Shimazaki, H.; Shimokawa, T.; Tsubo, Y.; Kimura, R.; Saiki, A.; Sakai, Y.; Isomura, Y.; Fujisawa, S.; et al Similarity in neuronal firing regimes across mammalian species. Journal of Neuroscience 36(21): [Park et al. 2016] Park, S.; Lee, S.-g.; Nam, H.; and Yoon, S An efficient approach to boosting performance of deep spiking network training. In Proceedings of the NIPS Workshop on Computing with Spikes. [Park et al. 2018a] Park, S.; Jang, J.; Kim, S.; and Yoon, S. 2018a. Streaming mann: A streaming-based inference for energy-efficient memory-augmented neural networks. arxiv preprint arxiv: [Park et al. 2018b] Park, S.; Kim, S.; Lee, S.; Bae, H.; and Yoon, S. 2018b. Quantized memory-augmented neural networks. In AAAI Conference on Artificial Intelligence, [Rueckauer et al. 2016] Rueckauer, B.; Lungu, I.-A.; Hu, Y.; and Pfeiffer, M Theory and tools for the conversion of analog to spiking convolutional neural networks. arxiv preprint arxiv: [Rueckauer et al. 2017] Rueckauer, B.; Lungu, I.-A.; Hu, Y.; Pfeiffer, M.; and Liu, S.-C Conversion of continuousvalued deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience 11:682. [Thorpe, Delorme, and Van Rullen 2001] Thorpe, S.; Delorme, A.; and Van Rullen, R Spike-based strategies for rapid processing. Neural networks 14(6-7): [Thorpe 1990] Thorpe, S. J Spike arrival times: A highly efficient coding scheme for neural networks. Parallel processing in neural systems