WRPN: Wide Reduced-Precision Networks
Asit Mishra Eriko Nurvitadhi Jeffrey J Cook Debbie Marr Accelerator Architecture Lab, Intel
Abstract
For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies
中文速览
深度神经网络在推理和训练时，激活图（activation map）占据的内存往往远超模型权重，尤其在批量输入场景下这一问题更为突出，但直接降低激活值的数值精度会显著损害模型精度。为此，研究者提出了"宽化低精度网络"（WRPN，Wide Reduced-Precision Networks）方案：同时降低激活图和权重的数值精度，并相应增加每层的滤波器数量，以更多但更轻量的低精度运算来弥补精度损失。实验在AlexNet、ResNet-34和Inception网络上的ImageNet（ILSVRC-12）基准测试中表明，采用4位激活和2位权重配合滤波器数量翻倍，可以达到甚至超越全精度基线模型的精度，例如2倍宽的二值化ResNet-34达到69.85%的top-1准确率，整体计算开销仅为全精度基线的15%。这项工作的意义在于，它首次系统验证了"降低激活精度+拓宽网络"的组合策略能在不牺牲准确率的前提下大幅节省内存、带宽和计算能量，为在FPGA、ASIC等硬件上高效部署深度学习模型提供了切实可行的路径。
原文 arXiv:1709.01134;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.01134v1