Sensitivity and Generalization in Neural Networks: an Empirical Study
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein Google Brain {romann, yasamanb, danabo, jpennin, Work done as a member of the Google Brain Residency program (g.co/brainresidency)
Abstract
In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classical notions of function complexity, which typically favor smaller models. In this work, we investigate this tension between complexity and generalization through an extensive empirical exploration of two natural metrics of complexity related to sensitivity to input perturbations. Our experiments survey thousands of models with various fully-connected architectures, optimizers, and other hyper-parameters, as well as four different image classification datasets. We find that trained neural networks are more robust to input perturbations in the vicinity of the training data manifold, as measured by the norm of the input-output Jacobian of the network, and that it correlates well with generalization. We further establish that factors associated with poor generalization – such as full-batch training or using random labels – correspond to lower robustness, while factors associated with good generalization – such as data augmentation and ReLU non-linearities – give rise to more robust functions. Finally, we demons
中文速览
大型神经网络明明参数更多、按传统理论应该更容易过拟合，却往往比小网络泛化得更好——这篇论文就是为了解释这个矛盾。研究者通过对数千个全连接网络（涵盖不同深度、宽度、优化器、超参数和四个图像分类数据集）的大规模实验，发现了一个关键现象：泛化好的网络在训练数据附近对输入扰动更不敏感，而衡量这种敏感性的核心指标——输入输出雅可比矩阵（input-output Jacobian）的范数——与泛化性能高度相关。进一步分析表明，数据增强、ReLU激活函数、小批量随机梯度下降等有助于泛化的因素都能让网络变得更"鲁棒"，而全批量训练、随机标签等有损泛化的因素则对应更高的敏感性；该指标甚至能在单个测试样本层面预测其是否会被正确分类。这项工作为理解深度学习的泛化之谜提供了一个简洁、实用的复杂度度量视角，也为未来设计正则化方法和理解隐式偏置提供了实证基础。
原文 arXiv:1802.08760;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.08760v3