Data-free Parameter Pruning for Deep Neural Networks
Abstract
Deep Neural nets (NNs) with millions of parameters are at the heart of many state-of-the-art computer vision systems today. However, recent works have shown that much smaller models can achieve similar levels of performance. In this work, we address the problem of pruning parameters in a trained NN model. Instead of removing individual weights one at a time as done in previous works, we remove one neuron at a time. We show how similar neurons are redundant, and propose a systematic way to remove them. Our experiments in pruning the densely connected layers show that we can remove upto 85% of the total parameters in an MNIST-trained network, and about 35% for AlexNet without significantly affecting performance. Our method can be applied on top of most networks with a fully connected layer to give a smaller network.
中文速览
深度神经网络往往拥有数百万个参数，但其中存在大量冗余——当两个神经元的权重向量高度相似时，它们实际上在计算几乎相同的特征。针对这一问题，作者提出了一种无需训练数据的神经元级剪枝方法：通过定义一个"显著性"指标来衡量任意两个神经元之间的相似程度，每次找出最相似的一对并将其合并（删除其中一个，同时把它在后一层的连接权重累加到保留的那个神经元上），如此反复直至网络足够精简。实验结果表明，在MNIST训练的网络上可删除高达85%的全连接层参数，在AlexNet上也能减少约35%，且分类精度几乎不受影响。该方法不依赖任何数据、计算开销低、不改变网络整体结构，因此剪枝后仍可直接进行微调，对需要部署在资源受限设备上的实际应用具有重要的实用价值。
原文 arXiv:1507.06149;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1507.06149v1