From Recognition to Cognition: Visual Commonsense Reasoning
Rowan Zellers♠ Yonatan Bisk♠ Ali Farhadi♠♡ Yejin Choi♠♡ ♠Paul G. Allen School of Computer Science、Engineering, University of Washington ♡Allen Institute for Artificial Intelligence visualcommonsense.com
Abstract
Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people’s actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today’s vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer.
中文速览
视觉常识推理（Visual Commonsense Reasoning）要求机器不只是认出图里有什么，还要像人一样推断人物的意图、目标和社会关系——这对现有视觉系统来说极具挑战性。研究者为此构建了大规模数据集 VCR，包含来自 11 万个电影场景的 29 万道多项选择题，并提出了"对抗匹配"（Adversarial Matching）算法，通过把正确答案循环复用为其他问题的干扰选项，在大规模自动生成题目的同时有效消除答案偏差。在模型层面，他们提出了从识别到认知网络（R2C），依次完成视觉指代定位、上下文语义融合和跨模态推理三个步骤，将机器在该任务上的准确率从约 45% 提升到约 65%，但与人类 90% 以上的水准仍有明显差距。这项工作首次以大规模、可量化的方式将"看懂图片背后在发生什么"变成可评测的基准，为推动机器走向真正的认知级视觉理解提供了数据、方法和分析基础。
原文 arXiv:1811.10830;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.10830v2