A Unified View of Entropy-Regularized Markov Decision Processes
\NameGergely Neu \addrUniversitat Pompeu Fabra Barcelona Spain \NameVicenç Gómez \addrUniversitat Pompeu Fabra Barcelona Spain \NameAnders Jonsson \addrUniversitat Pompeu Fabra Barcelona Spain
Abstract
We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimization in MDPs to accommodate convex regularization functions. Our key result is showing that using the conditional entropy of the joint state-action distributions as regularization yields a dual optimization problem closely resembling the Bellman optimality equations. This result enables us to formalize a number of state-of-the-art entropy-regularized reinforcement learning algorithms as approximate variants of Mirror Descent or Dual Averaging, and thus to argue about the convergence properties of these methods. In particular, we show that the exact version of the TRPO algorithm of Schulman et al. (2015) actually converges to the optimal policy, while the entropy-regularized policy gradient methods of Mnih et al. (2016) may fail to converge to a fixed point. Finally, we illustrate empirically the effects of using various regularization techniques on learning performance in a simple reinforcement learning setup.
中文速览
强化学习里有一类做法是在优化目标里加入"熵正则化"来鼓励探索、防止过拟合，但理论上为什么这样做有效、不同算法之间有何联系，一直缺乏统一解释。研究者将马尔可夫决策过程的线性规划框架推广到带凸正则化项的形式，并发现：当用"条件熵"作为正则化函数时，对偶优化问题在结构上与经典的贝尔曼最优方程高度吻合，从而在值函数方法和策略搜索方法之间建立起严格的拉格朗日对偶桥梁。基于这一统一框架，研究者证明了TRPO算法的精确版本等价于镜像下降法并能收敛到最优策略，而Mnih等人提出的熵正则化策略梯度方法则因近似步骤破坏了目标函数的凸性，可能无法收敛到任何不动点。这项工作不仅为一批启发式算法提供了坚实的理论依据，还为设计可证明收敛的熵正则化强化学习方法指明了方向。
原文 arXiv:1705.07798;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1705.07798v1