Bitwise Neural Networks
Minje Kim Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801 USA Paris Smaragdis University of Illinois at Urbana-Champaign, Urbana, IL 61801 USA Adobe Research, Adobe Systems Inc., San Francisco, CA 94103, USA
Abstract
Based on the assumption that there exists a neural network that efficiently represents a set of Boolean functions between all binary inputs and outputs, we propose a process for developing and deploying neural networks whose weight parameters, bias terms, input, and intermediate hidden layer output signals, are all binary-valued, and require only basic bit logic for the feedforward pass. The proposed Bitwise Neural Network (BNN) is especially suitable for resource-constrained environments, since it replaces either floating or fixed-point arithmetic with significantly more efficient bitwise operations. Hence, the BNN requires for less spatial complexity, less memory bandwidth, and less power consumption in hardware. In order to design such networks, we propose to add a few training schemes, such as weight compression and noisy backpropagation, which result in a bitwise network that performs almost as well as its corresponding real-valued network. We test the proposed network on the MNIST dataset, represented using binary features, and show that BNNs result in competitive performance while offering dramatic computational savings.
中文速览
传统深度神经网络在资源受限的嵌入式设备上运行时，浮点或定点数的乘加运算会带来巨大的计算开销和功耗压力。为此，研究者提出了一种"逐位神经网络"（Bitwise Neural Network，BNN），将网络中所有的权重、偏置、输入及隐层激活值全部压缩为单个二进制位（±1），前向推理时只需 XNOR 和位计数操作，彻底取代了浮点乘法和加法。训练方面采用两阶段策略：先用 tanh 权重压缩技术训练一个实值网络作为初始化，再以"噪声反向传播"对全二值网络进行微调，同时引入稀疏性参数允许部分权重为零以进一步减少计算量。在 MNIST 手写数字识别任务上，BNN 取得了与对应实值网络相当的分类性能，同时在存储占用、内存带宽和功耗上均实现了数量级级别的节省，为在手机、可穿戴设备等资源受限场景中部署神经网络提供了一条切实可行的路径。
原文 arXiv:1601.06071;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1601.06071v1