Inverse Reward Design
Dylan Hadfield-Menell Smitha Milli Pieter Abbeel Stuart Russell Anca D. Dragan Department of Electrical Engineering and Computer Science University of California, Berkeley Berkeley, CA 94709 {dhm, smilli, pabbeel, russell, OpenAI, International Computer Science Institute (ICSI)
Abstract
Autonomous agents optimize the reward function we give them. What they don’t know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.
中文速览
给机器人设计奖励函数（reward function）时，设计者往往只考虑了训练场景，一旦机器人遭遇新环境，原本的奖励函数就可能导致灾难性的错误行为——比如导航机器人把岩浆地形当成普通路面径直冲过去。这篇论文的核心洞察是：奖励函数不应被当作真实目标的精确定义，而只是设计者意图的一个"观测信号"，必须结合它被设计时所处的训练环境来理解。为此，作者提出了"逆向奖励设计"（Inverse Reward Design, IRD）框架，把推断设计者真实意图这件事建模为贝叶斯推断问题，并借鉴逆强化学习的思路给出了可计算的近似算法；在此基础上，机器人对不熟悉的新场景保持奖励的不确定性，并采用风险规避策略来行动。实验结果表明，这一方法能有效缓解奖励函数设计不当带来的"负面副作用"和"奖励作弊"问题，让机器人在遭遇分布外场景时懂得谨慎行事，而不是盲目执行一个可能已经失效的奖励信号。
原文 arXiv:1711.02827;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.02827v2