Hierarchical Question-Image Co-Attention for Visual Question Answering
Jiasen Lu∗, Jianwei Yang∗, Dhruv Batra∗† , Devi Parikh∗† ∗* Virginia Tech, ††\dagger Georgia Institute of Technology {jiasenlu, jw2yang, dbatra,
Abstract
A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling “where to look” or visual attention, it is equally important to model “what words to listen to” or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.111The source code can be downloaded from https://github.com/jiasenlu/HieCoAttenVQA.
中文速览
视觉问答（Visual Question Answering, VQA）任务要求机器同时理解图像和问题才能给出正确答案，已有方法主要关注"看图像的哪个区域"（视觉注意力），却忽视了"问题中哪些词更重要"（问题注意力）。这项工作提出了一种层次化协同注意力（Hierarchical Co-Attention）模型，通过图像与问题的双向引导，同时生成视觉注意力图和问题注意力图，并借助一维卷积神经网络在词级、短语级、句子级三个层次上对问题进行逐层编码和联合推理。实验结果显示，该模型在VQA数据集上将准确率从60.3%提升至60.5%，在COCO-QA数据集上从61.6%大幅提升至63.3%，换用ResNet后进一步达到62.1%和65.4%。这项工作的意义在于，它首次将问题注意力与视觉注意力置于同等重要的地位并加以联合建模，为多模态理解提供了更对称、更鲁棒的框架。
原文 arXiv:1606.00061;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.00061v5