Sparsifying Neural Network Connections for Face Recognition
Yi Sun1 Xiaogang Wang2,4 Xiaoou Tang3,4 1SenseTime Group 2Department of Electronic Engineering, The Chinese University of Hong Kong 3Department of Information Engineering, The Chinese University of Hong Kong 4Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
Abstract
This paper proposes to learn high-performance deep ConvNets with sparse neural connections, referred to as sparse ConvNets, for face recognition. The sparse ConvNets are learned in an iterative way, each time one additional layer is sparsified and the entire model is re-trained given the initial weights learned in previous iterations. One important finding is that directly training the sparse ConvNet from scratch failed to find good solutions for face recognition, while using a previously learned denser model to properly initialize a sparser model is critical to continue learning effective features for face recognition. This paper also proposes a new neural correlation-based weight selection criterion and empirically verifies its effectiveness in selecting informative connections from previously learned models in each iteration. When taking a moderately sparse structure ( $26\%$ - $76\%$ of weights in the dense model), the proposed sparse ConvNet model significantly improves the face recognition performance of the previous state-of-the-art DeepID2+ models given the same training data, while it keeps the performance of the baseline model with only $12\%$ of the original parameters.
中文速览
人脸识别深度卷积网络（ConvNet）的参数量庞大，既占内存又容易过拟合，如何在压缩模型的同时还能提升识别精度，是一个核心挑战。研究者提出了一种"稀疏卷积网络"（Sparse ConvNet）训练方法：先训练一个高性能的稠密基线模型，再逐层从后往前裁剪神经连接，每次只稀疏化一层并用上一轮更稠密的模型权重来初始化、重新训练整个网络，如此迭代直至达到目标稀疏度。裁剪连接的依据是一种新提出的"神经相关性"准则——激活值相关性越强的连接越值得保留，灵感来自"共同激发的神经元倾向于相互连接"的赫布理论。实验表明，在相同的约30万张人脸训练数据下，该方法将LFW人脸验证准确率从98.95%提升至99.30%（错误率降低33%），而仅保留12%参数时仍能维持基线精度不下降。这项工作不仅给出了一套可控、可复现的模型压缩流程，还揭示了"用稠密模型初始化稀疏模型"这一关键训练策略，对移动端等资源受限场景下的人脸识别落地具有重要实用价值。
原文 arXiv:1512.01891;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1512.01891v1