An Alternative Softmax Operator for Reinforcement Learning
Kavosh Asadi Michael L. Littman
Abstract
A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one’s weight behind a single maximum utility decision. The Boltzmann softmax operator is the most commonly used softmax operator in this setting, but we show that this operator is prone to misbehavior. In this work, we study a differentiable softmax operator that, among other properties, is a non-expansion ensuring a convergent behavior in learning and planning. We introduce a variant of SARSA algorithm that, by utilizing the new operator, computes a Boltzmann policy with a state-dependent temperature parameter. We show that the algorithm is convergent and that it performs favorably in practice.
中文速览
强化学习中常用的 Boltzmann softmax 算子虽然可微、能近似最大化，却并非"非扩张算子"，这会导致价值迭代出现多个不动点、SARSA 算法在表格环境下也无法收敛——本文首次给出了这一失败的具体 MDP 案例。为此，作者提出了一种新的 softmax 算子"mellowmax"，定义为对动作价值做指数加权平均后取对数再除以温度参数，并严格证明它满足非扩张性、可微性、趋近最大值和趋近均值等四个理想性质。在此基础上，他们将 mellowmax 与最大熵原则结合，推导出一种形式上等同于 Boltzmann 策略、但温度参数随状态自适应调整的新策略，并将其嵌入 SARSA 算法，证明了算法收敛性。在随机 MDP、表格 SARSA 以及深度策略梯度的实验中，mellowmax 均优于或持平于 Boltzmann softmax，且彻底消除了后者的不稳定问题，为序列决策中的 softmax 设计提供了理论更扎实的替代方案。
原文 arXiv:1612.05628;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1612.05628v5