Group Sparse Regularization for Deep Neural Networks
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini Simone Scardapane, Danilo Comminiello and Aurelio Uncini are with the Department of Information Engineering, Electronics and Telecommunications (DIET), “Sapienza” University of Rome, Via Eudossiana 18, 00184, Rome. Amir Hussain is with the Division of Computing Science、Maths, School of Natural Sciences, University of Stirling, Stirling FK9 4LA, Scotland, UK. Email: Author: Simone Scardapane.
Abstract
In this paper, we consider the joint task of simultaneously optimizing (i) the weights of a deep neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are generally dealt with separately, we present a simple regularized formulation allowing to solve all three of them in parallel, using standard optimization routines. Specifically, we extend the group Lasso penalty (originated in the linear regression literature) in order to impose group-level sparsity on the network’s connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We perform an extensive experimental evaluation, by comparing with classical weight decay and Lasso penalties. We show that a sparse version of the group Lasso penalty is able to achieve competitive performances, while at the same time resulting in extremely compact networks with a smaller number of input features. W
中文速览
神经网络在实际部署时面临三个相互缠绕的难题：冗余神经元太多、输入特征维度太高、网络权重过于庞大，而现有方法往往要分开多步处理这些问题。这项研究把线性回归中的"群 Lasso（group Lasso）"惩罚项引入深度神经网络，将每个神经元的所有出边权重定义为一个"组"，用一个统一的正则化目标函数同时完成隐层神经元剪枝、输入特征选择和偏置筛选，整个过程只需在标准随机梯度下降中加一项惩罚，无需额外训练步骤。实验表明，其中"稀疏群 Lasso（sparse group Lasso）"变体在手写数字识别玩具任务和多个大规模分类基准上都能达到与 L2、L1 正则化相当的精度，同时将每层神经元数量大幅压缩、显著减少所用输入特征数。这项工作的价值在于提供了一个简洁、原则性的一体化框架，让网络在训练结束时就自动变得紧凑，既降低了过拟合风险，也为在资源受限设备上部署深度模型提供了实用的解决路径。
原文 arXiv:1607.00485;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1607.00485v1