Harmless interpolation of noisy data in regression
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian and Anant Sahai BLISS、ML4Wireless, EECS, UC Berkeley {vidya.muthukumar, kailasv, vignesh.subramanian,
Abstract
A continuing mystery in understanding the empirical success of deep neural networks is their ability to achieve zero training error and generalize well, even when the training data is noisy and there are more parameters than data points. We investigate this overparameterized regime in linear regression, where all solutions that minimize training error interpolate the data, including noise. We characterize the fundamental generalization (mean-squared) error of any interpolating solution in the presence of noise, and show that this error decays to zero with the number of features. Thus, overparameterization can be explicitly beneficial in ensuring harmless interpolation of noise. We discuss two root causes for poor generalization that are complementary in nature – signal “bleeding" into a large number of alias features, and overfitting of noise by parsimonious feature selectors. For the sparse linear model with noise, we provide a hybrid interpolating scheme that mitigates both these issues and achieves order-optimal MSE over all possible interpolating solutions.
中文速览
深度神经网络在训练数据有噪声、参数远多于数据点的情况下依然能泛化，这一反直觉现象长期缺乏理论解释。研究者在过参数化线性回归（overparameterized linear regression）框架下系统分析了"插值噪声"这一行为：当模型参数数量远超训练样本时，所有使训练误差为零的解都必然把噪声也一并拟合进去。他们证明，在温和条件下，任意插值解的超额均方误差（excess MSE）会随特征数量的增加而趋于零，从而从理论上说明过参数化本身可以是无害甚至有益的；同时指出泛化表现差的两种互补根源——信号"扩散"到大量冗余特征，以及稀疏选择器对噪声的过度拟合——并由此构造出一种混合插值方案，在所有插值解中达到阶最优的测试误差。这一结果为理解"双下降"（double descent）现象和深度学习的泛化能力提供了清晰的线性模型视角。
原文 arXiv:1903.09139;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1903.09139v2