Retrieval-Augmented Multimodal Language Modeling
Michihiro Yasunaga Armen Aghajanyan Weijia Shi Rich James Jure Leskovec Percy Liang Mike Lewis Luke Zettlemoyer Wen-tau Yih
Abstract
Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in text-to-image and image-to-text generation. However, these models store all their knowledge (e.g., the appearance of the Eiffel Tower) in the model parameters, requiring increasingly larger models and training data to capture more knowledge. To integrate knowledge in a more scalable and modular way, we propose a retrieval-augmented multimodal model, which enables a base multimodal model (generator) to refer to relevant text and images fetched by a retriever from external memory (e.g., documents on the web). Specifically, for the retriever, we use a pretrained CLIP, and for the generator, we train a CM3 Transformer on the LAION dataset. Our resulting model, named Retrieval-Augmented CM3 (RA-CM3), is the first multimodal model that can retrieve and generate both text and images. We show that RA-CM3 significantly outperforms baseline multimodal models such as DALL-E and CM3 on both image and caption generation tasks (12 FID and 17 CIDEr improvements on MS-COCO), while requiring much less compute for training ( $<\!30\%$ of DALL-E). Moreover, we show that RA-CM3 exhibits novel capabilities, such as fai
中文速览
多模态大模型（如DALL-E、CM3）通常把所有知识"死记"在参数里，随着知识量增长不得不堆砌越来越多的参数和训练数据，扩展性很差。本文提出检索增强多模态模型RA-CM3，让模型在生成图像或文字时能主动从外部记忆库（如网页文档）中检索相关的图文内容作为参考，核心做法是以预训练CLIP作为检索器、以CM3 Transformer作为生成器，将检索回来的多模态文档拼接在输入前作为上下文示例，再联合优化主文档和检索文档的token预测损失。实验表明，RA-CM3在MS-COCO图像生成和图像描述任务上分别比基线提升了12 FID和17 CIDEr，同时只用不到DALL-E 30%的训练算力；此外，该模型还能生成与实体外观高度一致的图像，并首次实现针对图文双模态的上下文学习（in-context learning），为构建可扩展、可更新的多模态知识系统开辟了新路径。
原文 arXiv:2211.12561;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.12561v2