Gligen: Open-Set Grounded Text-to-Image Generation
Yuheng Li1§, Haotian Liu1§, Qingyang Wu2, Fangzhou Mu1, Jianwei Yang3, Jianfeng Gao3, Chunyuan Li3¶, Yong Jae Lee1¶ 1University of Wisconsin-Madison 2Columbia University 3Microsoft https://gligen.github.io/
Abstract
Large-scale text-to-image diffusion models have made amazing advances. However, the status quo is to use text input alone, which can impede controllability. In this work, we propose Gligen, Grounded-Language-to-Image Generation, a novel approach that builds upon and extends the functionality of existing pre-trained text-to-image diffusion models by enabling them to also be conditioned on grounding inputs. To preserve the vast concept knowledge of the pre-trained model, we freeze all of its weights and inject the grounding information into new trainable layers via a gated mechanism. Our model achieves open-world grounded text2img generation with caption and bounding box condition inputs, and the grounding ability generalizes well to novel spatial configurations and concepts. Gligen’s zero-shot performance on COCO and LVIS outperforms existing supervised layout-to-image baselines by a large margin. †† $\S$ Part of the work performed at Microsoft; ${\P}$ Co-senior authors
中文速览
文生图扩散模型（text-to-image diffusion model）虽然效果惊人，但仅靠文字描述很难精确控制生成图像中物体的位置和布局。本文提出 GLIGEN（Grounded-Language-to-Image Generation），在冻结已有预训练模型全部权重的前提下，通过新增可训练的门控 Transformer（gated Transformer）层，把边界框（bounding box）、关键点、参考图像等空间定位信息注入生成过程，从而让模型同时接受文字说明和位置条件。由于复用了预训练文本编码器来表示待定位的实体短语，模型天然具备开放词汇（open-world）泛化能力，即便只在 COCO 数据集上训练，也能在 LVIS 等新类别上大幅超越已有的全监督布局生成基线。这项工作证明，在预训练大模型之上用轻量化的方式注入新条件，是一条兼顾知识保留与灵活可控的高效路径，为可控图像生成提供了重要参考。
原文 arXiv:2301.07093;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.07093v2