Visual Commonsense in Pretrained Unimodal and Multimodal Models
Chenyu Zhang Benjamin Van Durme Zhuowan Li Elias Stengel-Eskin Johns Hopkins University {czhan105, vandurme, zli110, *Joint Advising
Abstract
Our commonsense knowledge about objects includes their typical visual attributes; we know that bananas are typically yellow or green, and not purple. Text and image corpora, being subject to reporting bias, represent this world-knowledge to varying degrees of faithfulness. In this paper, we investigate to what degree unimodal (language-only) and multimodal (image and language) models capture a broad range of visually salient attributes. To that end, we create the Visual Commonsense Tests (ViComTe) dataset covering 5 property types (color, shape, material, size, and visual co-occurrence) for over 5000 subjects. We validate this dataset by showing that our grounded color data correlates much better than ungrounded text-only data with crowdsourced color judgments provided by Paik et al. (2021). We then use our dataset to evaluate pretrained unimodal models and multimodal models. Our results indicate that multimodal models better reconstruct attribute distributions, but are still subject to reporting bias. Moreover, increasing model size does not enhance performance, suggesting that the key to visual commonsense lies in the data.111The dataset and code is available at https://github.co
中文速览
人类对物体外观的常识性认知（比如香蕉通常是黄色或绿色而非紫色）在文字和图像数据中因"报告偏差"（reporting bias）而被不同程度地失真，这篇论文正是要系统评估语言模型和多模态模型在捕捉这类视觉常识方面的能力。为此，研究者构建了一个名为 ViComTe 的大规模测评数据集，涵盖颜色、形状、材质、大小和视觉共现五类属性，共涉及逾五千个物体，并借助软提示调优（soft prompt tuning）等方法对多种单模态和多模态预训练模型进行系统测试。结果显示，多模态模型在还原物体属性的真实分布上明显优于纯文本模型，但同样难逃报告偏差的影响；更关键的是，单纯扩大模型参数量并不能提升视觉常识能力，真正的瓶颈在于训练数据的质量与来源。这一发现为如何让语言模型真正"理解"物理世界的视觉规律提供了重要方向——即数据本身比模型规模更为关键。
原文 arXiv:2205.01850;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.01850v1