Mapping Instructions and Visual Observations to Actions with Reinforcement Learning
Dipendra Misra Dept. of Computer Science and Cornell Tech Cornell University New York, NY 10044 {dkm, John Langford Microsoft Research New York, NY 10011 Yoav Artzi Dept. of Computer Science and Cornell Tech Cornell University New York, NY 10044 {dkm,
Abstract
We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input. We use reinforcement learning in a contextual bandit setting to train a neural network agent. To guide the agent’s exploration, we use reward shaping with different forms of supervision. Our approach does not require intermediate representations, planning procedures, or training different models. We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.
中文速览
让神经网络智能体直接读取原始图像和自然语言指令、输出一系列操作动作，是指令跟随（instruction following）领域的核心挑战，此前方法要么依赖结构化的环境表示，要么将视觉理解和语言理解拆成独立模块分别训练。本文提出用一个统一的神经网络模型同时处理图像和文本，通过强化学习中的上下文赌博机（contextual bandit）框架进行训练，并借助奖励塑形（reward shaping）将演示数据或目标状态标注转化为即时奖励信号，以引导智能体在有限数据下高效探索。在积木世界（Blocks）仿真环境中，该方法相比监督学习将执行误差降低了24%，相比常见强化学习变体降低了34–39%。这项工作表明，无需手工设计中间表示或多模块流水线，单一端到端模型结合合理的探索策略就能有效完成复杂的视觉语言指令执行任务，为更通用的具身语言理解奠定了基础。
原文 arXiv:1704.08795;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1704.08795v2