Quantifying and Alleviating the Language Prior Problem in Visual Question Answering
Yangyang Guo†, Zhiyong Cheng§, Liqiang Nie†, Yibing Liu†, Yinglong Wang§, Mohan Kankanhalli‡ †School of Computer Science and Technology, Shandong University §Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences) ‡School of Computing, National University of Singapore guoyang.eric, jason.zy.cheng, nieliqiang,
Abstract
Benefiting from the advancement of computer vision, natural language processing and information retrieval techniques, visual question answering (VQA), which aims to answer questions about an image or a video, has received lots of attentions over the past few years. Although some progress has been achieved so far, several studies have pointed out that current VQA models are heavily affected by the language prior problem, which means they tend to answer questions based on the co-occurrence patterns of question keywords (e.g., how many) and answers (e.g., 2) instead of understanding images and questions. Existing methods attempt to solve this problem by either balancing the biased datasets or forcing models to better understand images. However, only marginal effects and even performance deterioration are observed for the first and second solution, respectively. In addition, another important issue is the lack of measurement to quantitatively measure the extent of the language prior effect, which severely hinders the advancement of related techniques.
中文速览
视觉问答（Visual Question Answering, VQA）模型在回答图片相关问题时，往往偷懒地依赖训练数据中"问题关键词→答案"的统计规律（即语言先验问题），而非真正理解图像内容——比如看到"有多少个"就直接回答"2"，不管图片里实际有几个。针对这一问题，现有方法要么通过扩充数据集来减少偏差（效果有限），要么强迫模型关注图像（反而导致性能下降），且始终缺乏一个能定量衡量语言先验程度的评估指标。本文从两个角度入手：一是提出了一个叫"LP分数"的量化指标，能够同时考虑数据集偏差和模型自身缺陷，首次给语言先验效应打上可比较的数字标签；二是设计了一个"分数正则化模块"，通过成对学习策略迫使模型在预测答案前必须结合图像内容进行推理，而非依赖问答的共现模式，且该模块可以即插即用地嵌入多种现有VQA主干网络并端到端训练。在VQA 1.0和VQA 2.0两个主流数据集上的大量实验表明，该模块不仅有效缓解了语言先验问题，还同步提升了主干模型的问答准确率，兼顾了"纠偏"与"提性能"两个目标，为后续研究提供了可量化的评估基准和实用的改进工具。
原文 arXiv:1905.04877;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.04877v1