Dual Policy Iteration
Wen Sun1, Geoffrey J. Gordon1, Byron Boots2, and J. Andrew Bagnell3 1School of Computer Science, Carnegie Mellon University, USA 2College of Computing, Georgia Institute of Technology, USA 3Aurora Innovation, USA {wensun, ggordon,
Abstract
A novel class of Approximate Policy Iteration (API) algorithms have recently demonstrated impressive practical performance (e.g., ExIt [2], AlphaGo-Zero [30]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at test time, and a slow, non-reactive policy (e.g., Tree Search), that can plan multiple steps ahead. The reactive policy is updated under supervision from the non-reactive policy, while the non-reactive policy is improved via guidance from the reactive policy. In this work we study this class of Dual Policy Iteration (DPI) strategy in an alternating optimization framework and provide a convergence analysis that extends existing API theory. We also develop a special instance of this framework which reduces the update of non-reactive policies to model-based optimal control using learned local models, and provides a theoretically sound way of unifying model-free and model-based RL approaches with unknown dynamics. We demonstrate the efficacy of our approach on various continuous control Markov Decision Processes.
中文速览
双策略迭代（Dual Policy Iteration, DPI）框架要解决的核心问题是：传统强化学习方法依赖随机试探，导致样本效率低下，而现有的模型自由与模型驱动方法又难以有机结合。作者提出了一套交替优化框架，同时维护两个策略——一个在测试时快速响应的"反应式策略"（如深度神经网络）和一个在训练时借助局部学习动力学模型做多步规划的"中间专家策略"，后者通过基于模型的最优控制（Model-Based Optimal Control）在当前策略邻域内求解局部最优，再用模仿学习的方式将改进传递给反应式策略。理论上，作者证明了该框架在每次迭代中能获得比保守策略迭代（CPI）更大的性能提升，且只需局部动力学模型在当前策略的状态-动作分布下准确即可保证策略改进。在连续控制等基准任务上的实验表明，DPI 在样本效率上显著优于经典 API 和 actor-critic 基线，这一工作为理解 ExIt、AlphaGo Zero 等成功算法提供了统一的理论基础，也为模型自由与模型驱动强化学习的融合指出了一条有据可查的路径。
原文 arXiv:1805.10755;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1805.10755v2