Grounding Language Models to Images for Multimodal Inputs and Outputs
Jing Yu Koh Ruslan Salakhutdinov Daniel Fried
Abstract
We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images. Our method leverages the abilities of language models learnt from large scale text-only pretraining, such as in-context learning and free-form text generation. We keep the language model frozen, and finetune input and output linear layers to enable cross-modality interactions. This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images. We achieve strong zero-shot performance on grounded tasks such as contextual image retrieval and multimodal dialogue, and showcase compelling interactive abilities. Our approach works with any off-the-shelf language model and paves the way towards an effective, general solution for leveraging pretrained language models in visually grounded settings.
中文速览
纯文本大语言模型（LLM）看不懂图、也生成不了图，但从头训练一个多模态模型又极其昂贵——FROMAGe 就是为了用极低代价打通这道壁垒而生的。它的做法是把预训练好的 LLM（OPT-6.7B）和视觉编码器（CLIP）全部冻住，只训练两个轻量线性层和一个新增的特殊词元 [RET]：线性层负责把图像嵌入映射进文本空间（让模型读图）、把文本嵌入映射进视觉空间（让模型检索图），而 [RET] 词元则让模型在生成文本时自然地插入"此处取一张图"的指令。整个训练只需一块 GPU 跑不到 24 小时，用的数据也仅是 330 万张图文对，而非竞品所依赖的网络级交错图文语料。实验表明，FROMAGe 在零样本多模态对话、上下文图像检索等任务上表现优异，还能处理任意交错的图文输入并输出同样交错的图文内容，证明冻结 LLM 的语言能力完全可以被迁移到视觉落地场景，为高效构建通用多模态模型提供了一条切实可行的新路径。
原文 arXiv:2301.13823;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.13823v4