Deep Double Descent: Where Bigger Models and More Data Hurt
Preetum Nakkiran Harvard University、Gal Kaplun Harvard University、Yamini Bansal Harvard University、Tristan Yang Harvard University、Boaz Barak Harvard University、Ilya Sutskever OpenAI Work performed in part while Preetum Nakkiran was interning at OpenAI, with Ilya Sutskever. We especially thank Mikhail Belkin and Christopher Olah for helpful discussions throughout this work. Correspondence Email: contribution
Abstract
We show that a variety of modern deep learning tasks exhibit a “double-descent” phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.
中文速览
深度学习模型存在一个反直觉的"双重下降"（double descent）现象：随着模型规模增大，测试误差并非单调下降，而是先降、后升、再降，在模型恰好能"刚好拟合"训练数据的临界点附近出现一个性能低谷。研究者提出"有效模型复杂度"（Effective Model Complexity, EMC）这一统一度量——它不仅取决于模型参数量，还取决于训练轮数——并用它将模型规模、训练时长、训练样本量三个维度的异常行为纳入同一框架：增大模型、训练更久、甚至增加数据量，都可能在临界区附近导致性能变差，例如将训练样本增加4.5倍反而使测试误差上升。研究团队在ResNet、CNN、Transformer等多种主流架构以及CIFAR-10、CIFAR-100、机器翻译等任务上做了大量实验，系统验证了这一"广义双重下降"假说，并在有无标签噪声的真实场景下均观察到该现象。这项工作调和了经典统计学"模型越大越易过拟合"与深度学习实践"模型越大越好"这两种看似矛盾的直觉，同时警示从业者：在临界参数化区域盲目扩大数据集或延长训练并不可靠，模型选择和训练策略需要综合考量有效复杂度与样本规模的相对关系。
原文 arXiv:1912.02292;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1912.02292v1