Theory IIIb: Generalization in Deep Networks
Tomaso Poggio 111To whom correspondence should be addressed Alphabet (Google) X Qianli Liao Alphabet (Google) X Brando Miranda Alphabet (Google) X Andrzej Banburski Alphabet (Google) X Xavier Boix Alphabet (Google) X Jack Hidary Alphabet (Google) X
Abstract
A main puzzle of deep neural networks (DNNs) revolves around the apparent absence of “overfitting”, defined in this paper as follows: the expected error does not get worse when increasing the number of neurons or of iterations of gradient descent. This is surprising because of the large capacity demonstrated by DNNs to fit randomly labeled data and the absence of explicit regularization. Recent results by [1] provide a satisfying solution of the puzzle for linear networks used in binary classification. They prove that minimization of loss functions such as the logistic, the cross-entropy and the exp-loss yields asymptotic, “slow” convergence to the maximum margin solution for linearly separable datasets, independently of the initial conditions. Here we prove a similar result for nonlinear multilayer DNNs near zero minima of the empirical loss. The result holds for exponential-type losses but not for the square loss. In particular, we prove that the weight matrix at each layer of a deep network converges to a minimum norm solution up to a scale factor (in the separable case). Our analysis of the dynamical system corresponding to gradient descent of a multilayer network suggests a si
中文速览
深度神经网络在没有任何显式正则化的情况下，参数数量远超训练样本却依然不出现过拟合——这是深度学习领域的核心谜题之一。作者将这一谜题归结为：为何增加神经元数量或梯度下降迭代次数，测试误差并不上升？针对这个问题，论文将此前仅适用于线性网络的理论推广到非线性多层深度网络：通过将梯度下降诱导的训练过程建模为动力系统，在经验损失的零极小值附近对非线性系统做线性化，再借助经典的 Hartman-Grobman 定理证明线性化结果对原非线性系统同样成立，从而说明在指数型损失函数（如交叉熵、logistic 损失）下，每层权重矩阵经过归一化后会收敛到最小范数解，即最大间隔（max margin）解，且这一收敛与初始条件无关。这一结果揭示了梯度下降本身就扮演了隐式正则化（implicit regularization）的角色，从理论上解释了深度网络"不过拟合"的根源，同时也为比较不同零极小值点的泛化性能提供了一个简洁的判别准则，对深度学习理论的完善具有重要意义。
原文 arXiv:1806.11379;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.11379v1