Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Tuomas Haarnoja Aurick Zhou Pieter Abbeel Sergey Levine
Abstract
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar
中文速览
深度强化学习（deep RL）在连续控制任务中面临两大顽疾：样本效率低下（往往需要数百万步才能学会一个技能）和训练过程脆弱（对超参数极度敏感，换个随机种子结果就天差地别）。为此，研究者提出了软演员-评论家算法（Soft Actor-Critic，SAC），其核心思想是在最大化累积奖励的同时，也最大化策略的熵（entropy），也就是让智能体在完成任务的前提下尽可能"保持随机性"，从而兼顾探索与鲁棒性。SAC将这一最大熵框架与离策略（off-policy）的演员-评论家结构结合，既能复用历史经验以提升样本效率，又通过随机策略避免了DDPG等确定性算法的不稳定问题，论文还给出了策略迭代在该框架下的收敛性证明。在多个连续控制标准测试（包括拥有21个动作维度的Humanoid任务）上，SAC的最终性能和样本效率均大幅超越PPO、DDPG、TD3等主流方法，且在不同随机种子下表现高度一致，意味着它有望成为一个真正稳健、实用的深度强化学习基础算法。
原文 arXiv:1801.01290;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1801.01290v2