On the convergence of Adam and Beyond
Sashank J. Reddi, Satyen Kale、Sanjiv Kumar Google New York New York, NY 10011, USA
Abstract
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSprop, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with “long-term memory” of past gradients, and propose new variants of the Adam algorithm which not only fix the convergence issues but often also lead to improved empirical performance.
中文速览
自适应优化算法 Adam、RMSprop 等在深度学习中广受欢迎，但在某些任务中会出现无法收敛到最优解的奇怪现象，这篇论文正是要找出背后的根本原因并给出修复方案。作者证明，问题的症结在于这些算法用指数移动平均（exponential moving average）来累积历史梯度信息——这种机制会让偶发但信息量大的梯度迅速被"遗忘"，导致算法在某些简单凸优化问题上都会收敛到最差的点，而非最优解。为此，他们提出了一种具备"长期记忆"的改进算法 AMSGrad，它在每一步都保留历史上最大的二阶矩估计，从而保证学习率单调不增，并给出了严格的收敛性证明和与 Adagrad 类似的遗憾界。实验结果表明，AMSGrad 在多个常见机器学习任务上表现与 Adam 持平或更优。这项工作不仅揭示了当前最流行优化器的理论缺陷，还提供了一个兼具理论保证与实用性的改进方向，对理解和设计自适应优化算法具有重要参考价值。
原文 arXiv:1904.09237;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.09237v1