The Power of Sparsity in Convolutional Neural Networks
Soravit Changpinyo Department of Computer Science University of Southern California Los Angeles, CA 90020, USA、Mark Sandler and Andrey Zhmoginov Google Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043, USA The work was done while the author was doing an internship at Google Research.
Abstract
Deep convolutional networks are well-known for their high computational and memory demands. Given limited resources, how does one design a network that balances its size, training time, and prediction accuracy? A surprisingly effective approach to trade accuracy for size and speed is to simply reduce the number of channels in each convolutional layer by a fixed fraction and retrain the network. In many cases this leads to significantly smaller networks with only minimal changes to accuracy. In this paper, we take a step further by empirically examining a strategy for deactivating connections between filters in convolutional layers in a way that allows us to harvest savings both in run-time and memory for many network architectures. More specifically, we generalize 2D convolution to use a channel-wise sparse connection structure and show that this leads to significantly better results than the baseline approach for large networks including VGG and Inception V3.
中文速览
卷积神经网络（CNN）往往因参数量巨大而难以部署到移动端或嵌入式设备，已有的"深度乘数"压缩法虽然简单有效，但它只是均匀地砍掉每层的通道数。这篇论文在此基础上更进一步，提出用随机稀疏连接（sparse random connections）替代全连接的通道关系——不再要求每个输出通道与所有输入通道相连，而是让每个输出通道随机连接一小部分输入通道，同时保持空间卷积的连续内存布局以便高效计算。在 VGG-16 和 Inception V3 等大型网络上的实验表明，相同压缩比下稀疏连接网络的精度明显优于深度乘数基线，例如 Inception V3 在参数量压缩到约 30 万的极端情况下仍能达到 52% 的 Top-1 准确率，而深度乘数方案只有约 41%。这项工作的意义在于提供了一种结构规整、无需复杂剪枝或量化流程的压缩思路，既能节省运行时计算量和内存，又能为渐进式训练提供天然支撑，对资源受限场景下的模型部署具有实用价值。
原文 arXiv:1702.06257;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1702.06257v1