An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
Behrooz Ghorbani Department of Electrical Engineering Stanford University \AndShankar Krishnan Machine Perception, Google Inc. \AndYing Xiao Machine Perception, Google Inc. Work was done while author was an intern at Google.
Abstract
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications.
中文速览
深度神经网络训练过程中，损失函数的黑塞矩阵（Hessian）决定了参数空间的局部曲率，但因为参数量动辄千万，几乎无法直接计算其全部特征值。这篇文章借鉴数值线性代数里已有数十年历史的随机Lanczos二次型方法，开发出一套可扩展到ImageNet级别大模型的黑塞谱估计工具，精度可达双精度数值误差量级。借助这一工具，作者追踪了整个训练过程中黑塞谱的动态演变，发现在没有批归一化（Batch Normalization，BN）的网络中，训练初期会迅速冒出少数极大的孤立特征值，而梯度能量也高度集中在这些方向上，导致其余方向的参数几乎无法有效更新，从而拖慢训练；而加入BN之后，这些孤立特征值几乎消失，谱分布更加集中，训练因此大幅加速。这一发现为批归一化为何能加速优化提供了迄今最直接的谱层面解释，同时也为理解深度学习中平坦/尖锐极小值、残差连接等经典问题提供了新的实验证据和分析框架。
原文 arXiv:1901.10159;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1901.10159v1