Overparameterization without Overfitting: Jacobian-based Generalization Guarantees for Neural Networks
Samet Oymak Zalan Fabian α Mingchen Li α Mahdi Soltanolkotabi Department of Electrical and Computer Engineering, University of California, Riverside, CAMing Hsieh Department of Electrical Engineering, University of Southern California, Los Angeles, CADepartment of Computer Science and Engineering, University of California, Riverside, CA
Abstract
Modern neural network architectures often generalize well despite containing many more parameters than the size of the training dataset. This paper explores the generalization capabilities of neural networks trained via gradient descent. We develop a data-dependent optimization and generalization theory which leverages the low-rank structure of the Jacobian matrix associated with the network. Our results help demystify why training and generalization is easier on clean and structured datasets and harder on noisy and unstructured datasets as well as how the network size affects the evolution of the train and test errors during training. Specifically, we use a control knob to split the Jacobian spectum into “information" and “nuisance" spaces associated with the large and small singular values. We show that over the information space learning is fast and one can quickly train a model with zero training loss that can also generalize well. Over the nuisance space training is slower and early stopping can help with generalization at the expense of some bias. We also show that the overall generalization capability of the network is controlled by how well the label vector is aligned with
中文速览
深度神经网络参数数量远超训练样本，却仍能在真实数据上良好泛化，这一现象长期缺乏清晰解释。该研究借助神经网络雅可比矩阵（Jacobian matrix）的低秩结构，将学习过程分解为"信息空间"（少数大奇异值对应的方向）和"干扰空间"（大量小奇异值对应的方向）：在信息空间上梯度下降收敛极快且泛化性好，在干扰空间上学习缓慢且容易过拟合，早停（early stopping）能有效抑制后者带来的方差膨胀。理论结果表明，泛化能力的关键在于标签向量与信息空间的对齐程度——真实结构化数据的标签高度集中在信息空间，而标签噪声则落入干扰空间从而阻碍泛化；此外，即便是固定宽度的小网络，只要数据足够结构化也能获得泛化保证，无需像以往工作那样要求网络宽度随样本规模无限增长。这项工作为理解"为何干净数据上训练容易而噪声数据上训练难""网络大小如何影响偏差-方差权衡"以及预训练模型的迁移泛化提供了统一的理论框架。
原文 arXiv:1906.05392;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.05392v2