All-in-One Image-Grounded Conversational Agents
Da Ju111Contact Author Kurt Shuster Y-Lan Boureau、Jason Weston Facebook AI Research {daju, kshuster, ylan,
Abstract
As single-task accuracy on individual language and image tasks has improved substantially in the last few years, the long-term goal of a generally skilled agent that can both see and talk becomes more feasible to explore. In this work, we focus on leveraging individual language and image tasks, along with resources that incorporate both vision and language towards that objective. We design an architecture that combines state-of-the-art Transformer and ResNeXt modules fed into a novel attentive multimodal module to produce a combined model trained on many tasks. We provide a thorough analysis of the components of the model, and transfer performance when training on one, some, or all of the tasks. Our final models provide a single system that obtains good results on all vision and language tasks considered, and improves the state of the art in image-grounded conversational applications.
中文速览
视觉与语言的融合一直是人工智能的长期目标，但如何让一个模型同时在图文对话、看图问答、图像描述等多项任务上都表现出色仍是难题。研究者设计了一种结合顶尖Transformer文本编码器与ResNeXt图像编码器的新架构，核心创新是"注意力多模态融合模块"（Attentive Multimodal Combiner，AMMC）——它能根据不同输入动态决定更多依赖语言还是视觉信息，比普通注意力机制更灵活。团队在涵盖图像描述、个性化对话、视觉问答等多个视觉+语言任务上同时训练这一模型，并通过大量消融实验分析了各模块的贡献与最优配置。最终得到的单一系统在所有考察任务上均取得良好成绩，并在以图像为基础的对话类应用上刷新了当时的最优结果，朝着"能看会说"的通用对话智能体迈出了重要一步。
原文 arXiv:1912.12394;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1912.12394v2