UNITER: UNiversal Image-TExt Representation Learning
Yen-Chun Chen Equal contribution. Linjie Li⋆ Licheng Yu⋆ Ahmed El Kholy Faisal Ahmed Zhe Gan Yu Cheng Jingjing Liu
Abstract
Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are simultaneously processed for joint visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design four pre-training tasks: Masked Language Modeling (MLM), Masked Region Modeling (MRM, with three variants), Image-Text Matching (ITM), and Word-Region Alignment (WRA). Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). In addition to ITM for global image-text alignment, we also propose WRA via the use of Optimal Transport (OT) to explicitly encourage fine-grained alignment between words and image regions during pre-training. Comprehensive analysis shows that both conditional masking and OT-based WRA contribute to better pre-training. We
中文速览
图文理解领域长期面临一个难题：针对视觉问答、图文检索等不同任务训练出的视觉-语言模型往往高度专用，无法在任务间通用。研究者提出了UNITER（通用图文表示模型），以Transformer为核心，在四个大规模图文数据集上通过四种预训练任务联合优化：遮蔽语言建模、遮蔽区域建模、图文匹配，以及基于最优传输（Optimal Transport）的词-区域对齐——其中两大关键创新是"条件遮蔽"（每次只遮蔽一个模态、保留另一个完整，避免同时遮蔽带来的语义错位）和用最优传输显式拉近词与图像区域之间的细粒度对应关系。在视觉问答、图文检索、指代表达理解等六项任务、九个数据集上的测试中，UNITER均刷新了当时的最优结果，证明一个统一预训练模型完全可以在异构视觉-语言任务上全面超越任务专用方案，为后续多模态大模型的发展奠定了重要基础。
原文 arXiv:1909.11740;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.11740v3