Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach
Wenda Zhou Columbia University New York, NY、Victor Veitch Columbia University New York, NY、Morgane Austern Columbia University New York, NY、Ryan P. Adams Princeton University Princeton, NJ、Peter Orbanz Columbia University New York, NY
Abstract
Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be “compressed” to much smaller representations. The purpose of this paper is to connect these two empirical observations. Our main technical result is a generalization bound for compressed networks based on the compressed size that, combined with off-the-shelf compression algorithms, leads to state-of-the-art generalization guarantees. In particular, we provide the first non-vacuous generalization guarantees for realistic architectures applied to the ImageNet classification problem. Additionally, we show that compressibility of models that tend to overfit is limited. Empirical results show that an increase in overfitting increases the number of bits required to describe a trained network.
中文速览
过参数化深度神经网络为什么在训练数据上能过拟合、在测试数据上却仍然泛化良好，这一问题长期缺乏令人信服的理论解释。这篇论文把"网络可压缩"这一工程现象与泛化性能挂钩，利用PAC-Bayes框架推导出一个以网络压缩后比特数为核心的泛化误差上界，并进一步把权重对扰动的鲁棒性纳入考量来收紧这个界。将该界与现成的神经网络压缩算法结合后，研究者在ImageNet图像分类这一主流基准上首次给出了非平凡（non-vacuous）的泛化保证——即理论上界实际上小于1，真正有意义。此外，论文从理论和实验两方面证明，越容易过拟合的模型可压缩性越差，进一步揭示了压缩率与泛化能力之间的内在联系，为理解深度学习泛化之谜提供了一个量化且实用的新视角。
原文 arXiv:1804.05862;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1804.05862v3