Safe Reinforcement Learning in Constrained Markov Decision Processes
Akifumi Wachi Yanan Sui
Abstract
Safe reinforcement learning has been a promising approach for optimizing the policy of an agent that operates in safety-critical applications. In this paper, we propose an algorithm, SNO-MDP, that explores and optimizes Markov decision processes under unknown safety constraints. Specifically, we take a stepwise approach for optimizing safety and cumulative reward. In our method, the agent first learns safety constraints by expanding the safe region, and then optimizes the cumulative reward in the certified safe region. We provide theoretical guarantees on both the satisfaction of the safety constraint and the near-optimality of the cumulative reward under proper regularity assumptions. In our experiments, we demonstrate the effectiveness of SNO-MDP through two experiments: one uses a synthetic data in a new, openly-available environment named GP-Safety-Gym, and the other simulates Mars surface exploration by using real observation data.
中文速览
在安全关键场景（如火星探测）中，智能体往往事先不知道哪些区域安全、奖励函数如何，但又必须在探索过程中始终保证安全，同时还要尽可能积累高回报——现有方法要么只保证安全却不管回报是否接近最优，要么反过来。本文提出 SNO-MDP 算法，将整个学习过程分为两个阶段：先用高斯过程（Gaussian Process）建模未知安全函数，通过不断扩展"经过概率认证的安全区域"来探索环境；再在该安全区域内优化累积回报，并引入"安全探索提前终止"（ES²）机制加速收敛。理论上，作者在适当的正则性假设下证明了算法以高概率同时满足安全约束并获得近似最优的累积回报；实验上，算法在自建合成环境 GP-Safety-Gym 以及真实火星地形数据上均验证了其安全性与高效性。这项工作的意义在于，它首次在未知安全约束的 MDP 框架下同时给出了安全保证与累积回报近似最优性的理论承诺，为行星探测等现实安全关键任务提供了可靠的算法基础。
原文 arXiv:2008.06626;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2008.06626v1