Reconciling modern machine learning practice and the bias-variance trade-off
Mikhail Belkin The Ohio State University, Columbus, OH Daniel Hsu Columbia University, New York, NY Siyuan Ma The Ohio State University, Columbus, OH Soumik Mandal The Ohio State University, Columbus, OH
Abstract
Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners.
中文速览
现代机器学习中那些超大模型（如神经网络）往往被训练到把训练数据"背得一字不差"，按教科书里的偏差-方差权衡（bias-variance trade-off）理论，这样的模型理应严重过拟合、在新数据上表现很差——但现实恰恰相反，它们的测试精度常常极高。这篇论文用大量实验和理论分析提出了一条"双重下降"（double descent）风险曲线来统一解释这个矛盾：随着模型参数量增加，测试误差先呈现经典的U形下降再上升，在恰好能完美拟合训练集的"插值阈值"处达到峰值，此后继续增大模型容量，误差反而再次下降，并能低于U形曲线的最优点。背后的机制在于：更大的函数类让学习算法能够从众多可完美拟合数据的候选函数中挑出范数最小、最"平滑"的那个，这种隐式的简洁性偏好（类似奥卡姆剃刀）才是超大模型泛化能力强的根本原因。这一发现不仅为深度学习的实践经验提供了理论依据，也划定了经典统计学习理论的适用边界，对模型选择和机器学习基础理论都有深远影响。
原文 arXiv:1812.11118;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1812.11118v2