Analytic Study of Double Descent in Binary Classification: The Impact of Loss
Ganesh Kini and Christos Thrampoulidis University of California, Santa Barbara, Department of Electrical and Computer Engineering
Abstract
Extensive empirical evidence reveals that, for a wide range of different learning methods and datasets, the risk curve exhibits a double-descent (DD) trend as a function of the model size. In our recent coauthored paper [DKT19], we studied binary linear classification models and showed that the test error of gradient descent (GD) with logistic loss undergoes a DD. In this paper, we complement these results by extending them to GD with square loss. We show that the DD phenomenon persists, but we also identify several differences compared to logistic loss. This emphasizes that crucial features of DD curves (such as their transition threshold and global minima) depend both on the training data and on the learning algorithm. We further study the dependence of DD curves on the size of the training set. Similar to our earlier work, our results are analytic: we plot the DD curves by first deriving sharp asymptotics for the test error under Gaussian features. Albeit simple, the models permit a principled study of DD features, the outcomes of which theoretically corroborate related empirical findings occurring in more complex learning tasks.
中文速览
当模型参数量远超训练样本量时，测试误差并不会单调上升，而是会出现"双下降"（double descent，DD）现象——先降、再升、再降。本文针对二元线性分类任务，用梯度下降配合**平方损失**进行理论分析，在高斯特征假设下推导出测试误差关于模型规模的精确渐近公式，从而将此前同一框架下针对逻辑损失得到的双下降结论扩展到平方损失。研究发现，双下降现象在平方损失下同样存在，但曲线的"峰值"位置、形状以及全局最优点与逻辑损失明显不同，说明DD曲线的关键特征同时取决于数据分布和所用的训练算法；此外，论文还从理论上解释了一个反直觉的经验观察：由于DD曲线的W形结构，在某些"糟糕"的模型规模下，增加训练样本反而会让测试误差变差。这些结果为近年来在深度神经网络等复杂模型上观察到的双下降现象提供了严格的理论佐证。
原文 arXiv:2001.11572;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2001.11572v1