Learning to Execute Actions or Ask Clarification Questions
Zhengxiang Shi Yue Feng Aldo Lipani University College London, London, UK
Abstract
Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also define two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly.
中文速览
在Minecraft虚拟建造场景中，如何让AI助手不仅能听懂用户指令并执行操作，还能在指令不清晰时主动提问寻求澄清——这正是本文试图解决的核心问题。研究者首先对Minecraft对话语料库中所有建造者发言进行了细粒度标注，将其分为八种对话行为类型（含"澄清提问"这一关键类型），在此基础上提出了一个新的建造者智能体模型，该模型通过跨模态注意力机制（cross-attention）融合对话文本与三维世界状态，并以"动作类型、位置、颜色"三个槽位（slot）简化了原本庞大的动作空间。实验结果表明，该模型在协作建造任务上达到了新的最优性能，且大幅优于已有方法，同时为"何时提问"及"建造与提问联合学习"两个新任务提供了基准。这项工作的意义在于，它将主动澄清能力引入了具身对话智能体的研究框架，为构建真正能与人自然协作、而非只会被动执行命令的AI系统奠定了基础。
原文 arXiv:2204.08373;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.08373v2