Safe Exploration in Continuous Action Spaces
Gal Dalal Krishnamurthy Dvijotham Matej Vecerik Todd Hester Cosmin Paduraru Yuval Tassa
Abstract
We address the problem of deploying a reinforcement learning (RL) agent on a physical system such as a datacenter cooling unit or robot, where critical constraints must never be violated. We show how to exploit the typically smooth dynamics of these systems and enable RL algorithms to never violate constraints during learning. Our technique is to directly add to the policy a safety layer that analytically solves an action correction formulation per each state. The novelty of obtaining an elegant closed-form solution is attained due to a linearized model, learned on past trajectories consisting of arbitrary actions. This is to mimic the real-world circumstances where data logs were generated with a behavior policy that is implausible to describe mathematically; such cases render the known safety-aware off-policy methods inapplicable. We demonstrate the efficacy of our approach on new representative physics-based environments, and prevail where reward shaping fails by maintaining zero constraint violations.
中文速览
把强化学习（RL）安全地部署到数据中心冷却系统或机器人等真实物理设备上，核心难题是如何在整个学习过程中从不违反温度、压力、角度等安全约束——哪怕一次也不行。作者提出了一个"安全层"（safety layer）方法：在策略网络输出动作之后，再叠加一层解析求解的动作校正模块，将原始动作微调至满足所有安全约束的最近点；这一校正的关键在于用一个轻量神经网络对安全信号关于动作做线性化近似，从而把约束优化问题化简为可闭合求解的凸二次规划，实现起来只需三行代码。与已有方法不同，该方法不依赖任何已知的行为策略，只需用随机动作采集的历史轨迹数据预训练线性模型即可，更贴近工业场景中日志数据来源不明的现实。在多个基于物理引擎的仿真环境中，该方法实现了零约束违反，而奖励惩罚（reward shaping）等常规手段则做不到这一点，证明了其在安全关键物理系统中的实用价值。
原文 arXiv:1801.08757;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1801.08757v1