Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
Charles H. Martin Calculation Consulting, 8 Locksley Ave, 6B, San Francisco, CA 94122, Michael W. Mahoney ICSI and Department of Statistics, University of California at Berkeley, Berkeley, CA 94720,
Abstract
We describe an approach to understand the peculiar and counterintuitive generalization properties of deep neural networks. The approach involves going beyond worst-case theoretical capacity control frameworks that have been popular in machine learning in recent years to revisit old ideas in the statistical mechanics of neural networks. Within this approach, we present a prototypical Very Simple Deep Learning (VSDL) model, whose behavior is controlled by two control parameters, one describing an effective amount of data, or load, on the network (that decreases when noise is added to the input), and one with an effective temperature interpretation (that increases when algorithms are early stopped). Using this model, we describe how a very simple application of ideas from the statistical mechanics theory of generalization provides a strong qualitative description of recently-observed empirical results regarding the inability of deep neural networks not to overfit training data, discontinuous learning and sharp transitions in the generalization properties of learning algorithms, etc.
中文速览
深度神经网络（DNN）存在一个令人费解的现象：即便训练数据被大量随机噪声污染，网络仍能把训练误差压到近乎为零，而常见的正则化手段（dropout、权重惩罚、输入加噪等）几乎无法阻止这种过拟合，只有早停（early stopping）才真正管用。作者借助统计力学（statistical mechanics）中的泛化理论，提出了一个"极简深度学习"（Very Simple Deep Learning, VSDL）模型，把DNN的行为归结为两个可操作的控制参数：一个类似"负载"（load）的参数α——代表有效数据量与模型复杂度之比，向训练数据加噪会使α降低；一个类似"温度"（temperature）的参数τ——代表训练算法的随机性程度，提前停止迭代相当于升高τ。在这两个参数构成的相图（phase diagram）中，泛化性能会发生急剧的相变（phase transition），由此自然地解释了为何DNN"无法不过拟合"以及为何早停是唯一有效的正则化手段。这一视角的重要性在于，它跳出了PAC/VC等最坏情形界的框架，用物理学中相变的语言为深度学习中的反直觉现象提供了简洁而统一的定性解释，并指出未来可以沿此路径给出定量预测。
原文 arXiv:1710.09553;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1710.09553v2