Simple Baseline for Visual Question Answering
Bolei Zhou Massachusetts Institute of Technology Yuandong Tian Facebook AI Research Sainbayar Sukhbaatar Facebook AI Research Arthur Szlam Facebook AI Research Rob Fergus Facebook AI Research
Abstract
We describe a very simple bag-of-words baseline for visual question answering. This baseline concatenates the word features from the question and CNN features from the image to predict the answer. When evaluated on the challenging VQA dataset [2], it shows comparable performance to many recent approaches using recurrent neural networks. To explore the strength and weakness of the trained model, we also provide an interactive web demo111http://visualqa.csail.mit.edu, and open-source code222https://github.com/metalbubble/VQAbaseline.
中文速览
视觉问答（Visual Question Answering, VQA）要求模型同时理解图像内容和自然语言问题并给出正确答案，现有方法大多依赖循环神经网络（如LSTM）来编码问题。本文提出了一个极其简单的基线方法 iBOWIMG：直接把问题的词袋（Bag-of-Words）特征与 CNN 提取的图像特征拼接，再接一个 softmax 分类器预测答案，整个模型用不超过10行 Torch 代码即可实现。在大规模 COCO VQA 数据集上的评测表明，这一简单基线的准确率与多个基于 LSTM 的复杂模型相当，与最优方法的差距也仅约1.5%。这一结果说明当前 VQA 数据集中存在较强的语言偏置——模型更多依赖问题关键词而非真正的图文推理，也为后续研究指出了从"记忆相关性"迈向"真正理解与推理"的改进方向。
原文 arXiv:1512.02167;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1512.02167v2