LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models
Long Lian UC Berkeley Boyi Li UC Berkeley Adam Yala UC Berkeley, UCSF Trevor Darrell UC Berkeley
Abstract
Recent advancements in text-to-image diffusion models have yielded impressive results in generating realistic and diverse images. However, these models still struggle with complex prompts, such as those that involve numeracy and spatial reasoning. This work proposes to enhance prompt understanding capabilities in diffusion models. Our method leverages a pretrained large language model (LLM) for grounded generation in a novel two-stage process. In the first stage, the LLM generates a scene layout that comprises captioned bounding boxes from a given prompt describing the desired image. In the second stage, a novel controller guides an off-the-shelf diffusion model for layout-grounded image generation. Both stages utilize existing pretrained models without additional model parameter optimization. Our method significantly outperforms the base diffusion model and several strong baselines in accurately generating images according to prompts that require various capabilities, doubling the generation accuracy across four tasks on average. Furthermore, our method enables instruction-based multi-round scene specification and can handle prompts in languages not supported by the underlying dif
中文速览
现有的文字生成图片模型（如 Stable Diffusion）在处理涉及数量、空间关系、属性绑定等复杂提示语时经常"翻车"，比如让它生成"三只猫"或"左边一只狗、右边一棵树"往往结果不对。这项工作提出了一个无需额外训练的两阶段方法：先用大语言模型（LLM）把文字提示"翻译"成带坐标的场景布局（每个前景物体对应一个标注了描述文字的边界框），再用一个新设计的控制器把这个布局作为先验，逐个生成每个物体的带遮罩潜变量，最后组合引导现成的扩散模型完成精确的图像生成。在四项测试任务上，该方法的平均生成准确率比基础扩散模型翻了一番，也超过了多个强基线方法。由于整个流程只调用现成的预训练模型，不改动任何模型参数，因此还能顺带支持多轮对话式场景修改以及非英语提示语，极大拓展了文字生成图片模型的实用边界。
原文 arXiv:2305.13655;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.13655v3