Practical Gauss-Newton Optimisation for Deep Learning
Aleksandar Botev Hippolyt Ritter David Barber
Abstract
We present an efficient block-diagonal approximation to the Gauss-Newton matrix for feedforward neural networks. Our resulting algorithm is competitive against state-of-the-art first-order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a laborious process, our approach can provide good performance even when used with default settings. A side result of our work is that for piecewise linear transfer functions, the network objective function can have no differentiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.
中文速览
训练神经网络时调超参数既费时又费力，而二阶优化方法虽能利用曲率信息、每步走得更远，却因计算和存储海森矩阵（Hessian matrix）代价太高而难以实用。本文提出了一种针对前馈神经网络的高效块对角高斯-牛顿（Gauss-Newton）矩阵近似方法：将每层权重对应的矩阵块分解为两个小矩阵的克罗内克积（Kronecker product），使得计算和求逆都可以在一次反向传播中完成。实验表明，这套方法在三个标准基准测试上即使不调参也能媲美甚至超过精心调过超参的一阶方法（如Adam）。此外，作者还发现了一个理论结论：对于使用分段线性激活函数（如ReLU）的网络，目标函数不存在可微的局部极大值点，这在一定程度上解释了为什么此类激活函数更有利于优化。
原文 arXiv:1706.03662;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.03662v2