Simplifying Reward Design through Divide-and-Conquer
Ellis Ratner Dylan Hadfield-Menell Anca D. Dragan {eratner, dhm,
Abstract
Designing a good reward function is essential to robot planning and reinforcement learning, but it can also be challenging and frustrating. The reward needs to work across multiple different environments, and that often requires many iterations of tuning. We introduce a novel divide-and-conquer approach that enables the designer to specify a reward separately for each environment. By treating these separate reward functions as observations about the underlying true reward, we derive an approach to infer a common reward across all environments. We conduct user studies in an abstract grid world domain and in a motion planning domain for a 7-DOF manipulator that measure user effort and solution quality. We show that our method is faster, easier to use, and produces a higher quality solution than the typical method of designing a reward jointly across all environments. We additionally conduct a series of experiments that measure the sensitivity of these results to different properties of the reward design task, such as the number of environments, the number of feasible solutions per environment, and the fraction of the total features that vary within each environment. We find that inde
中文速览
为机器人设计奖励函数（reward function）时，工程师通常要同时兼顾多个训练环境，反复调参却顾此失彼，既费时又容易出错。本文提出一种"分而治之"的独立奖励设计（Independent Reward Design）方法：让设计者为每个环境单独指定一个奖励函数，再借助贝叶斯推断——将各环境下的奖励函数视为对"真实奖励"的观测——将它们整合成一个可泛化到新环境的通用奖励分布。在网格世界导航和七自由度机械臂运动规划两项用户实验中，与传统的联合设计方法相比，该方法平均节省51%的设计时间、让用户感受到84%的易用性提升，同时在测试环境上的决策质量（以后悔值衡量）降低约70%。这项工作表明，把复杂的多环境奖励设计问题拆解成若干单环境子问题，不仅大幅降低了人的认知负担，还能让机器人学到更可靠的行为策略，对实际机器人系统的落地具有重要意义。
原文 arXiv:1806.02501;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.02501v1