Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss
Abstract
Neural networks trained to minimize the logistic (a.k.a. cross-entropy) loss with gradient-based methods are observed to perform well in many supervised classification tasks. Towards understanding this phenomenon, we analyze the training and generalization behavior of infinitely wide two-layer neural networks with homogeneous activations. We show that the limits of the gradient flow on exponentially tailed losses can be fully characterized as a max-margin classifier in a certain non-Hilbertian space of functions. In presence of hidden low-dimensional structures, the resulting margin is independent of the ambiant dimension, which leads to strong generalization bounds. In contrast, training only the output layer implicitly solves a kernel support vector machine, which a priori does not enjoy such an adaptivity. Our analysis of training is non-quantitative in terms of running time but we prove computational guarantees in simplified settings by showing equivalences with online mirror descent. Finally, numerical experiments suggest that our analysis describes well the practical behavior of two-layer neural networks with ReLU activations and confirm the statistical benefits of this impli
中文速览
训练两层神经网络做分类时，即便参数远多于样本数量、没有任何显式正则化，梯度下降也往往能学出泛化良好的分类器——这背后隐藏着一种"隐式偏置"（implicit bias），但其数学本质长期不清晰。本文对无限宽两层神经网络进行理论分析，证明用指数尾损失（如交叉熵损失）做梯度流训练，最终收敛到一个以"变分范数"（variation norm）为度量的最大间隔分类器（max-margin classifier），而非普通核方法对应的希尔伯特空间最大间隔。关键发现在于，当数据存在隐藏的低维结构时，这个变分范数意义下的间隔与环境空间维度无关，从而给出维度无关的泛化误差界，统计效率显著优于仅训练输出层所对应的核支持向量机。数值实验在ReLU网络上验证了上述理论预测，表明全量训练神经网络的隐式偏置在高维场景下具有真实可见的统计优势，为理解深度学习为何能在过参数化条件下仍然泛化提供了严格的理论支撑。
原文 arXiv:2002.04486;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.04486v4