Reinforcement Learning with Deep Energy-Based Policies
Tuomas Haarnoja Haoran Tang Pieter Abbeel Sergey Levine
Abstract
We propose a method for learning expressive energy-based policies for continuous states and actions, which has been feasible only in tabular domains before. We apply our method to learning maximum entropy policies, resulting into a new algorithm, called soft Q-learning, that expresses the optimal policy via a Boltzmann distribution. We use the recently proposed amortized Stein variational gradient descent to learn a stochastic sampling network that approximates samples from this distribution. The benefits of the proposed algorithm include improved exploration and compositionality that allows transferring skills between tasks, which we confirm in simulated experiments with swimming and walking robots. We also draw a connection to actor-critic methods, which can be viewed performing approximate inference on the corresponding energy-based model.
中文速览
深度强化学习长期依赖确定性策略或简单高斯策略，无法捕捉多峰（multimodal）行为、也难以把学到的技能迁移到新任务。本文提出"软Q学习"（soft Q-learning）算法，把最大熵强化学习目标与基于能量的模型（energy-based model）结合，用"软Q函数"（soft Q-function）定义一个玻尔兹曼分布作为最优策略，再借助摊销斯坦因变分梯度下降（amortized Stein variational gradient descent）训练一个采样网络来高效地从这个复杂分布中取样，从而把原本只在离散表格域可行的方法推广到连续高维空间。在游泳和行走机器人仿真实验中，该方法在多峰奖励场景下探索能力更强，并且预训练得到的策略可以作为学习新技能的良好初始化，实现技能的"可组合迁移"。这项工作同时揭示了演员-评论家（actor-critic）方法与能量模型近似推断之间的内在联系，为理解和设计新一代随机策略算法提供了统一框架。
原文 arXiv:1702.08165;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1702.08165v2