VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Weijie Su1,2∗, Xizhou Zhu1,2 , Yue Cao2, Bin Li1, Lewei Lu2, Furu Wei2, Jifeng Dai2 1University of Science and Technology of China 2Microsoft Research Asia Equal contribution. This work is done when Weijie Su and Xizhou Zhu are interns at Microsoft Research Asia.
Abstract
We introduce a new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT for short). VL-BERT adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input. In it, each element of the input is either of a word from the input sentence, or a region-of-interest (RoI) from the input image. It is designed to fit for most of the visual-linguistic downstream tasks. To better exploit the generic representation, we pre-train VL-BERT on the massive-scale Conceptual Captions dataset, together with text-only corpus. Extensive empirical analysis demonstrates that the pre-training procedure can better align the visual-linguistic clues and benefit the downstream tasks, such as visual commonsense reasoning, visual question answering and referring expression comprehension. It is worth noting that VL-BERT achieved the first place of single model on the leaderboard of the VCR benchmark. Code is released at https://github.com/jackroos/VL-BERT.
中文速览
视觉与语言的联合理解任务（如视觉问答、视觉常识推理）长期缺乏像BERT那样可通用预训练的特征表示，研究者提出了VL-BERT，将Transformer骨干网络同时接收图像感兴趣区域（Region-of-Interest, RoI）和文字词元作为输入，让视觉与语言信息在统一框架内自由交互对齐。预训练阶段在大规模图文对数据集Conceptual Captions和纯文本语料库上联合进行，通过随机遮掩词语或图像区域并要求模型预测来学习跨模态语义对齐，同时保留了Fast R-CNN参数的端到端更新以避免视觉信息泄露。实验表明VL-BERT在视觉常识推理（VCR）、视觉问答（VQA）和指代表达理解等多个基准上达到当时最优水平，并在VCR单模型榜单上排名第一。这项工作证明了为视觉语言任务构建统一预训练通用表示的可行性，为后续多模态大模型奠定了重要基础。
原文 arXiv:1908.08530;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1908.08530v4