Two models of double descent for weak features
Mikhail Belkin Halıcıoğlu Data Science Institute, UC San Diego, La Jolla, CA Daniel Hsu Department of Computer Science, Columbia University, New York, NY Data Science Institue, Columbia University, New York, NY Ji Xu Department of Computer Science, Columbia University, New York, NY
Abstract
The “double descent” risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models. This article provides a precise mathematical analysis for the shape of this curve in two simple data models with the least squares/least norm predictor. Specifically, it is shown that the risk peaks when the number of features $p$ is close to the sample size $n$ , but also that the risk decreases towards its minimum as $p$ increases beyond $n$ . This behavior is contrasted with that of “prescient” models that select features in an a priori optimal order.
中文速览
过参数化模型（overparameterized model）明明在训练集上把误差拟合到零，为什么在测试集上反而能表现良好？这正是"双下降"（double descent）风险曲线想解释的现象——随着模型参数数量 p 从少到多，测试误差先下降、在 p 接近样本量 n 时急剧飙升，随后随着 p 继续增大又再次下降，甚至低于经典"最优点"处的误差。本文针对两个简单数据模型（高斯随机特征模型和圆上的傅里叶级数模型），在最小二乘/最小范数预测器下给出了这条曲线的精确数学推导：严格证明了风险在 p≈n 时趋于无穷，并在 p 超过 n 后单调下降，当信噪比足够高时，使用全部特征（p 最大）可以达到全局最低风险。与此形成对比的是"先知式"特征选择——如果能预先按重要性排序选特征，最优点始终出现在 p≤n 的经典区间内，不会出现双下降。这项工作首次以有限样本的闭合公式把"双下降"现象从定性描述落实为严格数学结论，为理解过参数化深度学习模型的泛化能力提供了坚实的理论基础。
原文 arXiv:1903.07571;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1903.07571v2