From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
Justin Fu , Anoop Korattikara, Sergey Levine, Sergio Guadarrama Google AI Work done during an internship at Google AI Research
Abstract
Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a language-conditioned pol
中文速览
让机器人理解人类的自然语言指令并完成任务，一直是个难题——以往的做法是直接训练一个"语言条件策略"，让模型把语言指令映射到动作序列，但这要求模型在新环境中做到零样本泛化，往往表现很差。这篇论文换了一个思路：与其让模型直接输出动作，不如先把语言指令转化成一个奖励函数（reward function），再让智能体在新环境里自己用强化学习去优化这个奖励——提出了语言条件奖励学习方法（Language-Conditioned Reward Learning，LC-RL），核心是用最大因果熵逆强化学习（MaxEnt IRL）训练一个深度神经网络，将语言指令和视觉观测联合映射为奖励信号。在基于真实室内场景（SUNCG数据集）的导航和物体摆放任务上，实验证明LC-RL学到的奖励函数能有效迁移到全新的任务和从未见过的场景，而直接学习语言条件策略的基线方法则泛化能力很差。这项工作的意义在于：奖励函数比策略更容易跨环境复用，为构建能听懂人话、又能自主适应新环境的机器人系统提供了一条更实用的路径。
原文 arXiv:1902.07742;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.07742v1