Multimodal Few-Shot Learning with Frozen Language Models
Maria Tsimpoukelli∗ DeepMind、Jacob Menick∗ DeepMind University College London、Serkan Cabi∗ DeepMind、S. M. Ali Eslami DeepMind、Oriol Vinyals DeepMind、Felix Hill DeepMind
Abstract
When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we train a vision encoder to represent each image as a sequence of continuous embeddings, such that a pre-trained, frozen language model prompted with this prefix generates the appropriate caption. The resulting system is a multimodal few-shot learner, with the surprising ability to learn a variety of new tasks when conditioned on examples, represented as a sequence of multiple interleaved image and text embeddings. We demonstrate that it can rapidly learn words for new objects and novel visual categories, do visual question-answering with only a handful of examples, and make use of outside knowledge, by measuring a single model on a variety of established and new benchmarks.
中文速览
大规模自回归语言模型已经能用少量示例学习新任务，但它们对图像"视而不见"——Frozen 就是为了解决这个问题而生的。方法思路很简洁：训练一个视觉编码器，把图像映射成几个连续的"视觉前缀"向量，直接拼在语言模型的输入序列最前面，而语言模型本身的权重全程冻结不动，只靠反向传播更新视觉编码器。训练只用了图文配对的说明文字数据（Conceptual Captions），但推理时可以把任意数量的图像和文字交错排列作为提示，让模型完成它从未专门训练过的任务——包括零样本视觉问答、少样本图像分类，以及看了几张图片就能掌握新词含义的"快速概念绑定"。实验结果表明，仅凭少量上下文示例，Frozen 在多项视觉语言基准上的表现就能远超随机基线，并随着示例数量增加持续提升，证明了一种极其轻量、模块化的方式就能让已有的大型语言模型获得真正的多模态少样本学习能力，为未来更强的视觉语言系统提供了清晰的设计范式。
原文 arXiv:2106.13884;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.13884v2