Flamingo: a Visual Language Model for Few-Shot Learning
Jean-Baptiste Alayrac∗,‡、、Jeff Donahue∗、、Pauline Luc∗、、Antoine Miech∗、Iain Barr†、Yana Hasson†、Karel Lenc†、Arthur Mensch†、Katie Millican†、Malcolm Reynolds†、Roman Ring†、Eliza Rutherford†、Serkan Cabi、Tengda Han、Zhitao Gong、Sina Samangooei、Marianne Monteiro、Jacob Menick、Sebastian Borgeaud、Andrew Brock、Aida Nematzadeh、Sahand Sharifzadeh、Mikolaj Binkowski、Ricardo Barreira、Oriol Vinyals、Andrew Zisserman、Karen Simonyan∗,‡、∗ Equal contributions, ordered alphabetically, † Equal contributions, ordered alphabetically, ‡ Equal senior contributions DeepMind
Abstract
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability to describe a scene or an event; and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flam
中文速览
快速适应新任务、只用少量样本就能完成图文理解，是多模态机器学习长期面临的难题。Flamingo 提出了一套视觉语言模型（Visual Language Model，VLM）架构，通过引入 Perceiver Resampler 模块和门控交叉注意力层，将预训练好的视觉编码器与大型语言模型"冻结"后桥接在一起，使模型能够处理图片、视频与文字任意交错的输入序列。训练时，Flamingo 利用从网页中抓取的大规模多模态数据（包括交错图文网页、图文对和视频文本对），无需任何人工标注，从而获得强大的上下文少样本学习能力（in-context few-shot learning）——推理时只需在提示词中插入几个示例，模型即可直接生成答案。在 16 项图文理解基准测试上，Flamingo 刷新了少样本学习的最优结果，其中 6 项甚至超越了使用数千倍更多标注数据进行微调的专用模型，这表明将强大的预训练语言模型与视觉感知能力有机融合，是迈向通用多模态智能的一条极具潜力的路径。
原文 arXiv:2204.14198;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.14198v2