Sharp Minima Can Generalize For Deep Nets
Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio
Abstract
Despite their overwhelming capacity to overfit, deep learning architectures tend to generalize relatively well to unseen data, allowing them to be deployed in practice. However, explaining why this is the case is still an open area of research. One standing hypothesis that is gaining popularity, e.g. Hochreiter & Schmidhuber (1997); Keskar et al. (2017), is that the flatness of minima of the loss function found by stochastic gradient based methods results in good generalization. This paper argues that most notions of flatness are problematic for deep models and can not be directly applied to explain generalization. Specifically, when focusing on deep networks with rectifier units, we can exploit the particular geometry of parameter space induced by the inherent symmetries that these architectures exhibit to build equivalent models corresponding to arbitrarily sharper minima. Furthermore, if we allow to reparametrize a function, the geometry of its parameters can change drastically without affecting its generalization properties.
中文速览
深度学习模型即便过度参数化也能泛化到新数据，"平坦极小值（flat minima）导致好泛化"是目前颇受关注的一种解释——直觉是：损失函数在最优解附近越平坦，模型对参数扰动越不敏感，泛化就越好。然而这篇论文指出，这一论断在深度神经网络上存在根本性缺陷：利用ReLU网络固有的尺度对称性（non-negative homogeneity），只需对相邻两层权重做一个简单的缩放变换（一层乘以α、另一层除以α），就能得到一个在数学上完全等价、泛化性能丝毫不变的模型，却可以让任意"平坦"的极小值变得任意"尖锐"，反之亦然。实验与理论分析均表明，文献中常用的几种平坦度度量——包括基于邻域体积、Hessian特征值和ε-锐度的定义——都会因参数空间的重参数化而被随意操控，无法真正区分泛化能力的好坏。这一结论说明，单纯用损失曲面几何来解释泛化是不充分的，社区需要寻找对参数变换保持不变的泛化度量，才能建立更坚实的理论基础。
原文 arXiv:1703.04933;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.04933v2