Gated-Attention Architectures for Task-Oriented Language Grounding
Devendra Singh Chaplot School of Computer Science Carnegie Mellon University \AndKanthashree Mysore Sathyendra School of Computer Science Carnegie Mellon University \ANDRama Kumar Pasumarthi School of Computer Science Carnegie Mellon University \AndDheeraj Rajagopal School of Computer Science Carnegie Mellon University \AndRuslan Salakhutdinov School of Computer Science Carnegie Mellon University Equal contribution
Abstract
To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and actions in the environment. This problem is called task-oriented language grounding. We propose an end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input. The proposed model combines the image and text representations using a Gated-Attention mechanism and learns a policy to execute the natural language instruction using standard reinforcement and imitation learning methods. We show the effectiveness of the proposed model on unseen instructions as well as unseen maps, both quantitatively and qualitatively. We also introduce a novel environment based on a 3D game engine to simulate the challenges of task-oriented language grounding over a rich set of instructions and environment states.
中文速览
让AI智能体理解自然语言指令并在3D环境中找到正确目标物体，是视觉语言对齐（task-oriented language grounding）领域的核心难题。研究者提出了一种端到端可训练的神经网络架构，仅凭原始像素画面和自然语言指令作为输入，通过一种名为"门控注意力"（Gated-Attention）的多模态融合机制，让文字描述像"开关"一样筛选视觉特征图，使模型能够按指令关注物体的颜色、形状等关键属性，再结合强化学习与模仿学习训练出导航策略。实验在基于ViZDoom构建的新3D游戏环境中进行，结果表明该模型不仅能泛化到从未见过的指令和地图，在定量与定性评估上均优于简单拼接特征的基线方法。这项工作的价值在于，它无需任何预设的语言或视觉先验知识，为真实场景中的具身智能体理解人类指令提供了一条可行路径。
原文 arXiv:1706.07230;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.07230v2