The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
Preetum Nakkiran Harvard University、Behnam Neyshabur Google、Hanie Sedghi Google Brain
Abstract
We propose a new framework for reasoning about generalization in deep learning. The core idea is to couple the Real World, where optimizers take stochastic gradient steps on the empirical loss, to an Ideal World, where optimizers take steps on the population loss. This leads to an alternate decomposition of test error into: (1) the Ideal World test error plus (2) the gap between the two worlds. If the gap (2) is universally small, this reduces the problem of generalization in offline learning to the problem of optimization in online learning. We then give empirical evidence that this gap between worlds can be small in realistic deep learning settings, in particular supervised image classification. For example, CNNs generalize better than MLPs on image distributions in the Real World, but this is “because” they optimize faster on the population loss in the Ideal World. This suggests our framework is a useful tool for understanding generalization in deep learning, and lays a foundation for future research in the area.
中文速览
深度学习中的泛化问题一直难以用经典"训练误差+泛化差距"框架解释，尤其是当模型已把训练误差压到零时，经典分解毫无信息量。本文提出"深度自举（Deep Bootstrap）"框架，把测试误差重新拆解为"理想世界误差"与"自举误差"之和：前者衡量模型在每步都能看到全新样本（即直接优化总体损失）时的表现，后者衡量有限数据复用与无限新鲜数据之间的差距。作者通过构建600万张合成CIFAR-10图像数据集（CIFAR-5m）及ImageNet二分类子集，在ResNet、MLP、Vision Transformer等多种架构上系统验证：自举误差在训练收敛前始终很小，也就是说现实有限数据训练的表现几乎等同于无限数据的在线优化表现。这一发现揭示了一个简洁结论——卷积网络比MLP泛化更好，根本原因在于它在总体损失上优化得更快，而非依赖某种神秘的归纳偏置——从而把离线学习的泛化问题转化为在线学习的优化问题，为未来理论研究指明了一条更有操作性的新路径。
原文 arXiv:2010.08127;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.08127v2