Representation Learning for Grounded Spatial Reasoning
Michael Janner, Karthik Narasimhan, and Regina Barzilay Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {janner, karthikn,
Abstract
The interpretation of spatial references is highly contextual, requiring joint inference over both language and the environment. We consider the task of spatial reasoning in a simulated environment, where an agent can act and receive rewards. The proposed model learns a representation of the world steered by instruction text. This design allows for precise alignment of local neighborhoods with corresponding verbalizations, while also handling global references in the instructions. We train our model with reinforcement learning using a variant of generalized value iteration. The model outperforms state-of-the-art approaches on several metrics, yielding a 45% reduction in goal localization error. 111Code and dataset are available at https://github.com/JannerM/spatial-reasoning
中文速览
智能体需要理解自然语言指令中的空间关系（比如"最西边的岩石上方那格"），并在地图环境中导航到正确位置——难点在于同一句话在不同地图上对应不同位置，模型必须同时理解语言和环境才能泛化。研究者提出了一种新架构：先用LSTM把指令编码成向量，再将其拆分为"局部分量"和"全局分量"，局部分量作为卷积核直接与地图表示做卷积，捕捉"往左两格"这类邻域关系；全局分量则生成覆盖整张地图的梯度场，处理"最北边的树"这类全局最值引用。整个模型只靠强化学习的环境奖励信号（智能体到达目标时才给反馈）来训练，无需任何路径示范。在包含200张地图、逾3300个任务的测试集上，该模型将目标定位的曼哈顿距离误差比最强基线降低了45%，且能稳健地泛化到训练中未见过的地图布局。这项工作表明，将语言与地图在空间上精准对齐——而非简单拼接两种模态的表示——是实现空间语言理解的关键，对机器人导航和人机交互具有重要实践意义。
原文 arXiv:1707.03938;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.03938v2