Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
\namePeter Henderson \addrStanford University \AND\nameJoshua Romoff \addrMILA, McGill University, Facebook \AND\nameJoelle Pineau \addrMILA, McGill University, Facebook
Abstract
Recent analyses of certain gradient descent optimization methods have shown that performance can degrade in some settings – such as with stochasticity or implicit momentum. In deep reinforcement learning (Deep RL), such optimization methods are often used for training neural networks via the temporal difference error or policy gradient. As an agent improves over time, the optimization target changes and thus the loss landscape (and local optima) change. Due to the failure modes of those methods, the ideal choice of optimizer for Deep RL remains unclear. As such, we provide an empirical analysis of the effects that a wide range of gradient descent optimizers and their hyperparameters have on policy gradient methods, a subset of Deep RL algorithms, for benchmark continuous control tasks. We find that adaptive optimizers have a narrow window of effective learning rates, diverging in other cases, and that the effectiveness of momentum varies depending on the properties of the environment. Our analysis suggests that there is significant interplay between the dynamics of the environment and Deep RL algorithm properties which aren’t necessarily accounted for by traditional adaptive gradie
中文速览
深度强化学习（Deep RL）训练神经网络时普遍借用监督学习中的梯度下降优化器，但这些优化器是否真的适合强化学习这个目标持续变化、梯度天然带有随机性的场景，一直缺乏系统验证。本文对十种主流梯度优化器（包括 Adam、RMSProp、SGD with Nesterov momentum 等）在 A2C 和 PPO 两种策略梯度算法上展开大规模实证分析，覆盖多个连续控制基准任务。结果发现，Adam、RMSProp 等自适应方法对学习率极为敏感，只在极窄的范围内表现良好，稍高则直接发散；而动量的有效性则因环境动力学特性不同而差异显著，在某些环境中高动量反而有害，且 PPO 与 A2C 对动量的响应也不一致，暗示轨迹长度和并行工作者数量等算法结构本身可能引入隐式动量。这项工作揭示了环境动态、算法结构与优化器特性之间存在深层交互，为从业者选择和调优强化学习优化器提供了实用指导，也指出了设计专为强化学习场景定制的优化方法这一重要研究方向。
原文 arXiv:1810.02525;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.02525v1