Safe Policy Improvement with Soft Baseline Bootstrapping
Kimia Nadjahi1†∗ Romain Laroche2∗ Rémi Tachet des Combes2
Abstract
Batch Reinforcement Learning (Batch RL) consists in training a policy using trajectories collected with another policy, called the behavioural policy. Safe policy improvement (SPI) provides guarantees with high probability that the trained policy performs better than the behavioural policy, also called baseline in this setting. Previous work shows that the SPI objective improves mean performance as compared to using the basic RL objective, which boils down to solving the MDP with maximum likelihood (Laroche et al.,, 2019). Here, we build on that work and improve more precisely the SPI with Baseline Bootstrapping algorithm (SPIBB) by allowing the policy search over a wider set of policies. Instead of binarily classifying the state-action pairs into two sets (the uncertain and the safe-to-train-on ones), we adopt a softer strategy that controls the error in the value estimates by constraining the policy change according to the local model uncertainty. The method can take more risks on uncertain actions all the while remaining provably-safe, and is therefore less conservative than the state-of-the-art methods. We propose two algorithms (one optimal and one approximate) to solve this c
中文速览
批量强化学习（Batch RL）面临一个核心难题：如何用有限的历史数据训练出一个新策略，同时以高概率保证它不比原来收集数据的"基准策略"差。现有的SPIBB算法通过把状态-动作对"二元分类"来解决这一问题——数据太少的对就完全照搬基准策略，数据够多的才允许自由优化，但这种非此即彼的做法过于保守。本文提出了Soft-SPIBB方法，核心思路是用局部模型不确定性来"软性"约束策略变化的幅度：数据越少的动作允许偏离基准的程度越小，而不是直接锁死，从而在保留理论安全保证的前提下大幅扩展了可搜索的策略空间。研究者为此设计了一个精确求解算法和一个高效近似算法，并在有限状态MDP的网格世界任务和结合神经网络函数近似的无限状态导航任务上均验证了Soft-SPIBB在安全性和平均性能上显著优于现有所有竞争算法。这一工作对对话系统、医疗决策等现实场景意义重大，因为这些场景既无法在线试错、又无法容忍策略退化带来的实际风险。
原文 arXiv:1907.05079;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1907.05079v1