Constrained Policy Optimization
Joshua Achiam David Held Aviv Tamar Pieter Abbeel
Abstract
For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016; Schulman et al., 2015; Lillicrap et al., 2016; Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting.
中文速览
强化学习（RL）在机器人控制等高风险场景中落地时，仅靠设计奖励函数很难保证智能体始终满足安全要求。这篇文章提出了一种名为约束策略优化（Constrained Policy Optimization，CPO）的算法，将"约束马尔可夫决策过程"（Constrained MDP，CMDP）框架与信任域（trust region）策略搜索结合起来，在每次策略更新时都能以理论保证的方式同时提升累计奖励并满足辅助代价约束。算法的核心是一个新的理论定理——用两个策略之间的平均KL散度来界定它们回报之差的上下界——从而将代价约束转化为可用采样数据近似求解的二次规划问题。在模拟机器人运动任务上的实验表明，CPO能够训练拥有数千个参数的神经网络策略，在整个训练过程中持续满足安全约束，而无需像以往方法那样在"完全不违约"与"实用性"之间做出妥协，这是首个对任意参数化策略类（包括深度神经网络）在训练全程提供约束满足保证的通用策略搜索算法。
原文 arXiv:1705.10528;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1705.10528v1