Distributional Generalization: A New Kind of Generalization
Preetum Nakkiran Harvard University、Yamini Bansal Harvard University Co-first authors. Author contributions in Appendix A.
Abstract
We introduce a new notion of generalization— Distributional Generalization— which roughly states that outputs of a classifier at train and test time are close as distributions, as opposed to close in just their average error. For example, if we mislabel 30% of dogs as cats in the train set of CIFAR-10, then a ResNet trained to interpolation will in fact mislabel roughly 30% of dogs as cats on the test set as well, while leaving other classes unaffected. This behavior is not captured by classical generalization, which would only consider the average error and not the distribution of errors over the input domain. Our formal conjectures, which are much more general than this example, characterize the form of distributional generalization that can be expected in terms of problem parameters: model architecture, training procedure, number of samples, and data distribution. We give empirical evidence for these conjectures across a variety of domains in machine learning, including neural networks, kernel machines, and decision trees. Our results thus advance our empirical understanding of interpolating classifiers.
中文速览
训练好的深度神经网络在测试集上的表现，远比"平均错误率"所揭示的更有规律：如果训练集里30%的猫被错贴成"物体"标签，那么用 ResNet 拟合到零训练误差后，它在测试集上也会把大约30%的猫错分为"物体"，而其他类别几乎不受影响——这种精细的行为用传统泛化理论根本无法描述。为此，作者提出"分布式泛化"（Distributional Generalization）这一新概念：不只要求训练集和测试集上的平均错误率接近，而是要求模型输出的整体联合分布在两者上都近似相同。基于这一框架，他们进一步提出两个具体猜想——"特征校准猜想"（Feature Calibration Conjecture）和"一致性猜想"（Agreement Conjecture）——并在神经网络、核方法、决策树等多种模型和真实数据集上给出了大量实验支持，还对一近邻分类器给出了严格证明。这项工作为理解过参数化、插值型分类器的内在结构提供了全新视角，也表明神经网络对标签噪声的"敏感性"并非缺陷，而是一种更深层泛化性质的自然体现。
原文 arXiv:2009.08092;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2009.08092v2