More Data Can Hurt for Linear Regression: Sample-wise Double Descent
Preetum Nakkiran Harvard University
Abstract
In this expository note we describe a surprising phenomenon in overparameterized linear regression, where the dimension exceeds the number of samples: there is a regime where the test risk of the estimator found by gradient descent increases with additional samples. In other words, more data actually hurts the estimator. This behavior is implicit in a recent line of theoretical works analyzing “double descent” phenomena in linear models. In this note, we isolate and understand this behavior in an extremely simple setting: linear regression with isotropic Gaussian covariates. In particular, this occurs due to an unconventional type of bias-variance tradeoff in the overparameterized regime: the bias decreases with more samples, but variance increases.
中文速览
过度参数化线性回归（即特征维度 d 远超样本量 n）中存在一个反直觉现象：随着训练样本的增加，模型的测试误差并不单调下降，而是先降、后升、再降，呈现出"双重下降"（double descent）的形状，其中测试误差的峰值恰好出现在 n = d 的临界点。这篇文章用最简洁的设置——各向同性高斯协变量的线性回归，配合梯度下降找到的最小范数解——把上述现象孤立出来，逐一拆解偏差与方差的贡献：在过参数化区间，增加样本会减小偏差，但同时会让数据矩阵趋近奇异、方差急剧膨胀，两者此消彼长导致误差在 n ≈ d 时激增。理论近似公式与有限样本实验高度吻合，从而清晰地解释了"更多数据反而有害"这一现象背后的数学机制，为理解现代深度学习中的类似行为提供了可解释的理论基础。
原文 arXiv:1912.07242;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1912.07242v1