Deep Compression and EIE: Efficient Inference Engine on Compressed Deep Neural Network

Transcription

1 Deep Compression and EIE: Efficient Inference Engine on Deep Neural Network Song Han*, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, Bill Dally Stanford University

2 Our Prior Work: Deep Compression Memory reference is expensive. Small DNN models are critical. [1]. Han et al. NIPS 2015 [2]. Han et al. ICLR 2016, best paper award (2 bit uint) pruning synapses cluster : 2: pruning neurons : 0.00 pruning weight sharing 0: Network Original Size Size Compression Ratio Original Accuracy Accuracy AlexNet 240MB 6.9MB % 80.30% VGGNet 550MB 11.3MB % 89.09% GoogleNet 28MB 2.8MB 88.90% 88.92% SqueezeNet 4.8MB 0.47MB 80.32% 80.35%

3 EIE: First Accelerator for Sparse DNN Deep Compression solves the model size problem. But it creates another problem: irregular computation pattern. CPU/GPU are only good at dense linear algebra. So we create EIE that supports: static-sparse M, dynamic-sparse V, indirect indexing, weight sharing. Sparse Matrix 90% static sparsity in the weights, less computation, less memory footprint Sparse Vector 70% dynamic sparsity in the activation less computation Fully fits in SRAM 120x less energy than DRAM Weight Sharing 4bits weights 8x less memory footprint Savings are multiplicative: 8x120=14,400 theoretical energy improvement. Dally. NIPS tutorial 2015; Han et al. ISCA 2016

4 EIE: First Accelerator for Sparse DNN Technology 45 nm # PEs 64 on-chip SRAM Max Model Size Static Sparsity Dynamic Sparsity Quantization 8 MB 84 Million 4-bit DNN Model Input Image Encoded Weight Relative Index Sparse Format 4-bit Virtual weight 4-bit Relative Index Weight Look-up Index Accum 16-bit Real weight ALU 16-bit Mem Absolute Index Dally. NIPS tutorial 2015; Han et al. ISCA 2016 Prediction Result 4 ALU Width Area MxV Throughput Power 16-bit 40.8 mm^2 81,967 layers/s 586 mw 1. Post layout result 2. Throughput measured on AlexNet FC-7

5 FC Layers: Speedup / Energy Efficiency CPU Dense (Baseline) Speedup 1000x 56x 0.6x 24x x 1.0x mgpu Dense mgpu EIE 618x GPU 1018x 94x GPU Dense 507x 248x 100x CPU 1.0x 34x 16x 0. 48x 2 60x x x 0. Alex-6 Figure 6. Alex-7 Alex-8 VGG-7 VGG-8 NT-We NT-Wd NT-LSTM Geo Mean Speedups of GPU, mobile GPU and EIE compared with CPU running uncompressed DNN model. There is no batching in all cases. CPU Dense (Baseline) Energy Efficiency VGG x CPU GPU Dense 119,797x 61,53 34,52 GPU mgpu Dense mgpu EIE 76,784x 14,826x 11,828x 10000x 10,904x 9,48 24,207x 8, x 100x Alex-6 Figure 7. 37x 1 26x 37x 7x 78x Alex-7 7x 18x 7x Alex x 17x 6 1 VGG-6 VGG-7 8x 2 VGG-8 3 6x 6x 6x 6x 8x NT-We 2 7x NT-Wd 1 20x 4x 7x NT-LSTM 2 6x 7x 36x Geo Mean Energy efficiency of GPU, mobile GPU and EIE compared with CPU running uncompressed DNN model. There is no batching in all cases. Compared to CPU and GPU: Table III corner. We placed and routed the PE using the Synopsys IC B ENCHMARK FROM STATE - OF - THE - ART DNN MODELS 18We and compiler (ICC). used1 Cactifaster [25] to get SRAM area and Layer Size Weight% Act% FLOP% Description energy numbers. We annotated the toggle rate from the RTL 9216, 24,000x and 3,400x more energy efficient Alex-6 9% 35.1% 3% 4096 simulation to the gate-level netlist, which was dumped to 4096, AlexNet [1] for switching activity interchange format (SAIF), and estimated Alex-7 9% 35.3% 3% 4096 large scale image the power using Prime-Time PX. 4096, classification Alex-8 25% 37.5% 10% 1000 Comparison Baseline. We compare EIE with three dif25088,

6 Beyond EIE: a Multi-Dimension Sparse Recipe for Deep Learning Faster Speed: EIE accelerator sparsity Smaller Size: Deep Compression, SqueezeNet++ Higher Accuracy: DSD regularization [1]. Han et al. Learning both Weights and Connections for Efficient Neural Networks, NIPS 2015 [2]. Han et al. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, Deep Learning Symposium 2015, ICLR 2016 (best paper award) [3]. Han et al. EIE: Efficient Inference Engine on Deep Neural Network, ISCA 2016 [4]. Han et al. DSD: Regularizing Deep Neural Networks with Dense-Sparse-Dense Training Flow, arxiv 2016 [5]. Iandola, Han,et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size, arxiv 16 [6]. Yao, Han, et.al, Hardware-friendly convolutional neural network with even-number filter size, ICLR workshop