CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Justin Johnson1, Li Fei-Fei1 Bharath Hariharan2 C. Lawrence Zitnick2 Laurens van der Maaten2 Ross Girshick2 1Stanford University 2Facebook AI Research
Abstract
When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.
中文速览
视觉问答（Visual Question Answering, VQA）系统到底有没有真正学会"推理"，还是只是在利用数据集里的统计偏差走捷径，这是一个很难诊断的问题。为此，研究者推出了CLEVR这个专门用于诊断视觉推理能力的数据集：它包含10万张由3D简单几何体渲染的图像和近100万个自动生成的问题，每个问题都附带可执行的"功能程序"标注，精确记录回答该问题需要哪些推理步骤，同时通过拒绝采样等手段把答案分布偏差压到最低。用CLEVR对多种主流VQA模型进行测试后发现，这些模型在需要短期记忆（如比较两个物体的属性）和组合推理（如识别属性的新颖组合）的任务上表现很差，揭示出了以往基准测试难以发现的具体缺陷。这项工作的意义在于，它为研究者提供了一把"手术刀"而非"体温计"——不只告诉你模型整体得分高低，而是精确定位其推理能力的短板，为后续改进指明方向。
原文 arXiv:1612.06890;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1612.06890v1