FILM: Following Instructions in Language with Modular Methods
So Yeon Min1 Devendra Singh Chaplot2 Pradeep Ravikumar1 \ANDYonatan Bisk1 Ruslan Salakhutdinov1 1 Carnegie Mellon University 2 Facebook AI Research {soyeonm, pradeepr, ybisk,
Abstract
Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46%) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49%).111The official ALFRED leaderboard: https://leaderboard.allenai.org/alfred/submissions/public. Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions
中文速览
让具身智能体（embodied agent）按照自然语言指令完成家庭任务极具挑战，因为它需要同时处理导航、物体交互、空间记忆和长程规划等多个难题，而以往的端到端方法要靠大量专家示范轨迹和逐步细化的低级指令才能勉强应对。为此，研究者提出了模块化方法 FILM（Following Instructions in Language with Modular methods），将任务拆分为语言解析、语义地图构建、语义搜索策略和确定性控制四个独立模块，用结构化的空间表示替代神经网络隐式记忆来追踪状态和引导探索。在 ALFRED 基准测试上，FILM 无需专家轨迹和低级指令便达到 24.46% 的成功率，比此前最优方法高出约 8 个百分点，加入低级指令后可进一步提升至 26.49%。这项工作表明，显式的空间地图加上语义搜索策略是一种更强、更通用的具身任务解决范式，也为后续改进提供了清晰可解释的模块化路径。
原文 arXiv:2110.07342;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.07342v3