The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
Vardan Papyan Department of Statistics Stanford University Stanford, CA 94305
Abstract
We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data. Our results corroborate previous findings, based on small-scale networks, that the Hessian exhibits ‘spiked’ behavior, with several outliers isolated from a continuous bulk. We decompose the Hessian into different components and study the dynamics with training and sample size of each term individually.
中文速览
深度神经网络（deep neural network）在训练时所用的损失函数曲率结构——也就是黑塞矩阵（Hessian matrix）的谱——对理解泛化、优化和损失曲面都至关重要，但此前的研究只在参数量极少的小网络上做过验证，完全不清楚这些规律在拥有数千万参数的现代大网络上是否依然成立。本文将高维数值线性代数中的 Lanczos 谱密度估计方法引入深度学习，开发了一套可在 VGG、ResNet 等大规模网络上高效计算黑塞矩阵及其各分解项谱密度的软件工具。实验在多个数据集和网络上系统确认了"尖峰加连续体（bulk-and-outliers）"的谱结构：少数离群特征值由高斯-牛顿分解中的低秩项 G 贡献，而连续主体则主要来自残差项 H，且 H 的谱服从幂律而非此前理论所假设的 Marchenko-Pastur 分布；此外，研究还揭示了 G 的三层层级结构及其随训练轮次和样本量的动态演变规律。这些发现为深度学习优化算法设计、泛化理解以及随机矩阵理论假设的验证提供了坚实的大规模实证基础。
原文 arXiv:1811.07062;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.07062v2