\FontlukasFantastic Generalization Measures and Where to Find Them
Yiding Jiang , Behnam Neyshabur , Hossein Mobahi Dilip Krishnan, Samy Bengio Google Contributed equally.
Abstract
Generalization of deep networks has been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures. However, most papers proposing such measures study only a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings. We present the first large scale study of generalization in deep networks. We investigate more then 40 complexity measures taken from both theoretical bounds and empirical studies. We train over 10,000 convolutional networks by systematically varying commonly used hyperparameters. Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.
中文速览
深度神经网络为什么能泛化、如何衡量其泛化能力，至今仍是未解之谜，而现有研究往往只在少数模型上验证某几个复杂度度量（complexity measure），结论的普适性存疑。为此，研究者系统训练了超过10,000个卷积网络，覆盖批大小、学习率、网络深度/宽度、Dropout、权重衰减及优化器等7类超参数的大量组合，并在CIFAR-10和SVHN两个图像分类数据集上，对40余种来自理论界和实验界的复杂度度量进行了大规模评估。结果发现：许多基于参数范数（norm）的度量不仅预测效果差，在随机性较强的优化过程中甚至与泛化负相关；而基于"锐度"（sharpness）的度量（如PAC-Bayes界和Keskar等人提出的锐度指标）以及与优化过程相关的度量（如梯度噪声、优化速度）表现最为突出。这项研究是迄今规模最大的深度网络泛化实证分析，其结论有助于筛选出真正值得深入研究的泛化度量，为理论和实践双向推进提供可靠依据。
原文 arXiv:1912.02178;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1912.02178v1