On the Evaluation of Vision-and-Language Navigation Instructions
Ming Zhao Peter Anderson Vihan Jain Su Wang Alexander Ku Jason Baldridge Eugene Ie Google Research {astroming, pjand, vihan, wangsu, alexku, jridge, eugeneie} @google.com
Abstract
Vision-and-Language Navigation wayfinding agents can be enhanced by exploiting automatically generated navigation instructions. However, existing instruction generators have not been comprehensively evaluated, and the automatic evaluation metrics used to develop them have not been validated. Using human wayfinders, we show that these generators perform on par with or only slightly better than a template-based generator and far worse than human instructors. Furthermore, we discover that BLEU, ROUGE, METEOR and CIDEr are ineffective for evaluating grounded navigation instructions. To improve instruction evaluation, we propose an instruction-trajectory compatibility model that operates without reference instructions. Our model shows the highest correlation with human wayfinding outcomes when scoring individual instructions. For ranking instruction generation systems, if reference instructions are available we recommend using SPICE.
中文速览
视觉语言导航（Vision-and-Language Navigation）领域依赖自动生成的路径指令来训练智能体，但这些指令生成器究竟好不好用，一直缺乏系统验证。研究者招募真实的人类路径跟随者，发现现有两大主流指令生成模型（Speaker-Follower 和 EnvDrop）的表现竟与一个基于模板的简单系统相当，远不如真人撰写的指令，说明这一领域还有巨大的提升空间。在自动评估指标方面，研究者发现 BLEU、ROUGE、METEOR、CIDEr 这四种常用指标与人类实际导航结果几乎没有相关性，不适合评估有视觉场景依托的路径指令；对于系统级排名，推荐使用 SPICE；对于单条指令的质量打分，研究者提出了一种无需参考指令的"指令-轨迹兼容性模型"，该模型与人类导航结果的相关性优于所有对比方法。这项工作提醒我们：文本生成评估指标用到新领域时必须经过人工验证，同时也为导航指令生成的后续研究提供了更可靠的评估工具。
原文 arXiv:2101.10504;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2101.10504v1