Supervised Multimodal Bitransformers for Classifying Images and Text
Douwe Kiela†, Suvrat Bhooshan†, Hamed Firooz†, Ethan Perez‡, Davide Testuggine† †Facebook AI; ‡NYU
Abstract
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a simple yet effective baseline for multimodal BERT-like architectures, a supervised multimodal bitransformer that jointly finetunes unimodally pretrained text and image encoders by projecting image embeddings to text token space. We approach or match state-of-the-art accuracy on several text-heavy multimodal classification tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal performance. Surprisingly, our method is competitive with ViLBERT, a self-supervised multimodal “BERT for vision-and-language” approach, while being much simpler and more easily extendible.
中文速览
现代互联网上的信息往往是图文并存的，如何让机器同时理解文字和图像、做出准确分类，是一个亟待解决的多模态问题。研究者提出了一种简单但有效的方案——监督式多模态双向Transformer（supervised multimodal bitransformer），核心做法是把图像编码器输出的向量通过线性投影映射到BERT的词嵌入空间，让原本只处理文字的BERT直接把图像特征当作"词"来联合建模。在MM-IMDB、Food101和V-SNLI三个以文本为主的多模态分类任务上，该方法达到甚至追平了当时最先进水平，尤其在专门考验多模态融合能力的困难测试集上表现突出，且出人意料地能与需要大规模多模态预训练的ViLBERT相媲美。这一结果表明，无需昂贵的多模态预训练，只需将各自单模态预训练好的文本和图像编码器联合微调，就能构建出强大的多模态基线，为后续研究提供了一个简单、灵活、易于扩展的起点。
原文 arXiv:1909.02950;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.02950v2