Neural Networks with Few Multiplications
Zhouhan Lin Université de Montréal Canada、Matthieu Courbariaux Université de Montréal Canada \ANDRoland Memisevic Université de Montréal Canada、Yoshua Bengio Université de Montréal Canada
Abstract
For most deep learning algorithms training is notoriously time consuming. Since most of the computation in training neural networks is typically spent on floating point multiplications, we investigate an approach to training that eliminates the need for most of these. Our method consists of two parts: First we stochastically binarize weights to convert multiplications involved in computing hidden states to sign changes. Second, while back-propagating error derivatives, in addition to binarizing the weights, we quantize the representations at each layer to convert the remaining multiplications into binary shifts. Experimental results across 3 popular datasets (MNIST, CIFAR10, SVHN) show that this approach not only does not hurt classification performance but can result in even better performance than standard stochastic gradient descent training, paving the way to fast, hardware-friendly training of neural networks.
中文速览
神经网络训练速度慢的一大根源在于大量浮点乘法运算，这项工作专门针对这一瓶颈提出了两步消减策略：在前向传播中，把网络权重随机二值化（Binary/Ternary Connect）为±1或含0的三值，从而把矩阵乘法变成符号取反；在反向传播中，把每层的激活值量化为2的整数次幂，把梯度更新中的乘法变成移位操作（Quantized Back Propagation）。在MNIST、CIFAR10、SVHN三个标准数据集上的实验表明，这套"几乎无乘法"的训练方式不仅没有损害分类精度，在部分设置下反而因随机采样带来的正则化效果而略优于全精度的标准SGD训练。这项工作直接大幅削减了训练阶段的计算量，为在低功耗硬件（如FPGA、专用芯片）上高效训练神经网络铺平了道路。
原文 arXiv:1510.03009;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1510.03009v3