Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
Abhishek Das1, Satwik Kottur, José M.F. Moura2, Stefan Lee3, Dhruv Batra1 1Georgia Institute of Technology, 2Carnegie Mellon University, 3Virginia Tech visualdialog.org The first two authors (AD, SK) contributed equally.
Abstract
We introduce the first goal-driven training for visual question answering and dialog agents. Specifically, we pose a cooperative ‘image guessing’ game between two agents – Q-bot and A-bot– who communicate in natural language dialog so that Q-bot can select an unseen image from a lineup of images. We use deep reinforcement learning (RL) to learn the policies of these agents end-to-end – from pixels to multi-agent multi-round dialog to game reward.
中文速览
视觉问答与对话系统长期依赖监督学习，只会被动模仿人类对话数据，根本无法主动引导对话走向。这篇论文设计了一个合作性的"猜图游戏"：提问机器人（Q-bot）看不到图，只凭一句描述向看图的回答机器人（A-bot）连续追问，最终从一组候选图中认出目标图片，整个过程用深度强化学习（deep reinforcement learning）端到端地训练两个智能体的对话策略。在合成小世界的实验中，两个机器人从零开始自发约定出一套符号协议，比如用"X"问颜色、用"1"答红色，证明了视觉语言的自发涌现；在真实图像的大规模实验（VisDial数据集）上，先用人类对话数据预训练再用强化学习微调的方案，让Q-bot学会专门问A-bot擅长回答的问题，对话质量和图像检索准确率都显著优于纯监督学习基线。这项工作是视觉问答领域首次实现目标驱动训练，打破了该领域数百篇论文沿用监督学习的惯例，为构建真正能主动交流的视觉对话智能体开辟了新路径。
原文 arXiv:1703.06585;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.06585v2