Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations
Daniel S. Brown Wonjoon Goo Prabhat Nagarajan Scott Niekum
Abstract
A critical flaw of existing inverse reinforcement learning (IRL) methods is their inability to significantly outperform the demonstrator. This is because IRL typically seeks a reward function that makes the demonstrator appear near-optimal, rather than inferring the underlying intentions of the demonstrator that may have been poorly executed in practice. In this paper, we introduce a novel reward-learning-from-observation algorithm, Trajectory-ranked Reward EXtrapolation (T-REX), that extrapolates beyond a set of (approximately) ranked demonstrations in order to infer high-quality reward functions from a set of potentially poor demonstrations. When combined with deep reinforcement learning, T-REX outperforms state-of-the-art imitation learning and IRL methods on multiple Atari and MuJoCo benchmark tasks and achieves performance that is often more than twice the performance of the best demonstration. We also demonstrate that T-REX is robust to ranking noise and can accurately extrapolate intention by simply watching a learner noisily improve at a task over time.
中文速览
逆强化学习（IRL）长期存在一个致命缺陷：学出来的智能体顶多和示范者一样好，因为传统方法的目标只是找一个让示范者"看起来最优"的奖励函数，而不是真正推断示范者的真实意图。为了突破这一瓶颈，论文提出了 T-REX（Trajectory-ranked Reward EXtrapolation，轨迹排序奖励外推），核心思路是：给定一批按质量排好序的示范轨迹（不需要动作标签，只需观测），训练一个神经网络奖励函数，使其给排名更高的轨迹打出更高的总回报分数，再用这个外推出的奖励函数去跑强化学习。在多个 Atari 和 MuJoCo 标准测试任务上，T-REX 不仅超越了所有对比的模仿学习和 IRL 方法，最终策略的表现往往达到最佳示范者的两倍以上，而且对排序噪声具有良好鲁棒性，甚至只靠观看一个"边学边犯错"的新手视频就能奏效——这是首个无需真实奖励信号或人工实时监督便能显著超越示范者的方法，对于奖励难以手动设计的真实场景具有重要意义。
原文 arXiv:1904.06387;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.06387v5