FigureQA: An Annotated Figure Dataset for Visual Reasoning
Samira Ebrahimi Kahou1, Vincent Michalski2∗, Adam Atkinson1, Ákos Kádár3†, Adam Trischler1, Yoshua Bengio3 1Microsoft Research Montréal 2Université de Montréal, MILA 3Tilburg University Equal contribution.Part of this work was completed while the author interned at Microsoft Research Montréal.
Abstract
We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over $100,000$ images. The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection. To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure. To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives. In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements. We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as a strong baseline. Preliminary results indicate that the task poses a significant machine learning challenge. We envision Figur
中文速览
科学图表里藏着大量结构化信息，但让机器像人一样"看懂"折线图、柱状图、饼图并回答关于它们的推理问题，至今仍是难题。为此，研究者构建了 FigureQA 这一大规模合成视觉推理数据集，包含超过 10 万张五类科学风格图表和逾百万对问答，问题涵盖最大值、最小值、面积、平滑度、曲线交叉等 15 种关系类型，答案均为"是/否"二选一。他们以此训练了包括关系网络（Relation Network）在内的多个神经网络基线模型，最强的关系网络在测试集上准确率约为 72–77%，远未达到人类水平，表明该任务对现有模型仍构成显著挑战。这项工作的价值在于：它提供了一个可控、可扩展、带有边界框和数值标注的基准平台，推动研究者开发真正能从图表中进行直觉式视觉推理的模型，而不仅仅是把图像还原成原始数据。
原文 arXiv:1710.07300;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1710.07300v2