Generating Images with Multimodal Language Models
Jing Yu Koh Carnegie Mellon University、Daniel Fried Carnegie Mellon University、Ruslan Salakhutdinov Carnegie Mellon University
Abstract
We propose a method to fuse frozen text-only large language models (LLMs) with pre-trained image encoder and decoder models, by mapping between their embedding spaces. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue. Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs. To achieve strong performance on image generation, we propose an efficient mapping network to ground the LLM to an off-the-shelf text-to-image generation model. This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs. Our approach outperforms baseline generation models on tasks with longer and more complex language. In addition to novel image generation, our model is also capable of image retrieval from a prespecified dataset, and decides whether to retrieve or generate at inference time. This is done with a learnt decision module which conditions on the hidden representations of the LLM. Our model exhibits
中文速览
让冻结的纯文本大语言模型（LLM）同时"看图"又"画图"，一直是多模态研究的难题。GILL（Generating Images with Large Language Models）通过训练一个轻量级映射网络 GILLMapper，把冻结 LLM 的隐藏表示翻译到 Stable Diffusion 文本编码器的嵌入空间，从而把语言模型的强大语义理解能力"嫁接"到图像生成模型上，整个训练只用图文标注对、无需在训练时运行图像生成模型，两块 GPU 跑两天即可完成。该模型不仅能生成新图，还能从候选集中检索图片，并通过一个学习到的决策模块在推理时自动选择"生成"还是"检索"，最终实现对任意交错图文输入的统一处理与多模态对话输出。实验表明，面对较长、较复杂的自然语言描述（如对话和叙事场景），GILL 在文本-图像匹配度上显著超越作为骨干的 Stable Diffusion，也是迄今第一个能在输出序列中同时交错生成文本、检索图像和合成新图的多模态语言模型，为低算力条件下构建通用多模态系统提供了可行路径。
原文 arXiv:2305.17216;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.17216v3