LaProp: Separating Momentum and Adaptivity in Adam
Liu Ziyin111Department of Physics and Institute for Physics of Intelligence, University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan 222Correspondence to: Liu Ziyin Zhikang T. Wang††Masahito Ueda††RIKEN Center for Emergent Matter Science (CEMS), Wako, Saitama 351-0198, Japan
Abstract
We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, LaProp, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that LaProp has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of LaProp on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.
中文速览
动量（momentum）与自适应学习率（adaptivity）在 Adam 优化器内部的不合理耦合，会导致当两者参数不匹配时出现训练不稳定乃至发散的问题。为此，本文提出了 LaProp 优化器，其核心思路是将参数空间中的动量更新与自适应梯度缩放彻底解耦——先对梯度做自适应归一化，再累积动量，而不是像 Adam 那样先累积梯度再整体除以预处理因子。这一改动使得超参数 μ（动量系数）和 ν（自适应系数）的作用相互独立，LaProp 的单步更新幅度始终有界且仅取决于 ν，同时还能通过调节 ν→0 平滑过渡到符号随机梯度（signed stochastic gradient）方法，而 Adam 在动量与自适应参数不匹配时会直接发散。实验表明，LaProp 在多类任务上的收敛速度和稳定性均优于或持平于 Adam，尤其在噪声大、训练不稳定的场景下优势更为明显，且超参数数量与 Adam 完全相同、Adam 的默认参数设置可直接迁移使用。这项工作为理解自适应优化器的内在缺陷提供了新视角，并给出了一种几乎零额外代价即可获得更强稳定性与灵活性的实用替代方案。
原文 arXiv:2002.04839;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.04839v3