Boosting the Actor with Dual Critic
∗Bo Dai1 111The first two authors equally contributed. Albert Shaw1 Niao He2 Lihong Li3 Le Song1 1Georgia Institute of Technology {bodai 2University of Illinois at Urbana-Champaign 3Google AI
Abstract
This paper proposes a new actor-critic-style algorithm called Dual Actor-Criticor Dual-AC. It is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation, which can be viewed as a two-player game between the actor and a critic-like function, which is named as dual critic. Compared to its actor-critic relatives, Dual-AC has the desired property that the actor and dual critic are updated cooperatively to optimize the same objective function, providing a more transparent way for learning the critic that is directly related to the objective function of the actor. We then provide a concrete algorithm that can effectively solve the minimax optimization problem, using techniques of multi-step bootstrapping, path regularization, and stochastic dual ascent algorithm. We demonstrate that the proposed algorithm achieves the state-of-the-art performances across several benchmarks.
中文速览
深度强化学习中，Actor-Critic（演员-评论家）方法虽然兼顾了策略搜索与价值估计的优点，但演员和评论家各有各的优化目标，评论家究竟该怎么训练才能真正帮助演员提升，一直缺乏清晰的理论依据。这篇论文从贝尔曼最优方程的拉格朗日对偶形式出发，推导出一种新算法 Dual Actor-Critic（Dual-AC），将强化学习问题重新表述为演员与"对偶评论家"之间的极小极大博弈，两者共同优化同一个目标函数，从原理上解决了评论家与演员目标不一致的问题。为了应对函数近似带来的训练不稳定性，论文引入了多步自举、路径正则化和随机对偶上升等技术，使算法在实践中稳定可用。在 MuJoCo 连续控制基准任务上的实验表明，Dual-AC 达到了当时的最优水平，同时为 Actor-Critic 类方法提供了更严谨的理论基础。
原文 arXiv:1712.10282;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1712.10282v1