Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
Hongyuan Mei Mohit Bansal Matthew R. Walter Toyota Technological Institute at Chicago Chicago, IL 60637
Abstract
We propose a neural sequence-to-sequence model for direction following, a task that is essential to realizing effective autonomous agents. Our alignment-based encoder-decoder model with long short-term memory recurrent neural networks (LSTM-RNN) translates natural language instructions to action sequences based upon a representation of the observable world state. We introduce a multi-level aligner that empowers our model to focus on sentence “regions” salient to the current world state by using multiple abstractions of the input sentence. In contrast to existing methods, our model uses no specialized linguistic resources (e.g., parsers) or task-specific annotations (e.g., seed lexicons). It is therefore generalizable, yet still achieves the best results reported to-date on a benchmark single-sentence dataset and competitive results for the limited-training multi-sentence setting. We analyze our model through a series of ablations that elucidate the contributions of the primary components of our model.
中文速览
让机器人听懂人类的自然语言导航指令并自动执行动作序列，一直是自主智能体领域的核心难题。研究者提出了一种基于双向长短期记忆网络（LSTM-RNN）的序列到序列（sequence-to-sequence）模型，将自由形式的自然语言指令直接翻译为可执行的动作序列，同时引入"多层级对齐器"（multi-level aligner），让模型在解码每一步动作时既能关注句子的高层语义表示，也能直接回溯原始词向量，从而更精准地把"easel（画架）"这类关键词与当前环境状态对应起来。与以往依赖语义解析器、种子词典或人工标注的方法不同，该模型完全端到端训练，无需任何专门语言资源，在仅有2000条训练样本的单句导航基准测试上刷新了最优成绩，在训练数据更少的多句段任务上也达到了有竞争力的水平。这项工作表明，纯数据驱动的神经网络方法在低资源、无先验语言知识的条件下同样能超越依赖繁重人工工程的传统方案，为构建真正通用的语言驱动自主导航系统提供了新思路。
原文 arXiv:1506.04089;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1506.04089v4