T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
Chong Mou∗1,2 Xintao Wang†2 Liangbin Xie∗2,3,4 Yanze Wu2 Jian Zhang†1 Zhongang Qi2 Ying Shan2 Xiaohu Qie2 1Peking University Shenzhen Graduate School 2ARC Lab, Tencent PCG 3University of Macau 4Shenzhen Institute of Advanced Technology https://github.com/TencentARC/T2I-Adapter
Abstract
The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics. However, relying solely on text prompts cannot fully take advantage of the knowledge learned by the model, especially when flexible and accurate controlling (e.g., color and structure) is needed. In this paper, we aim to “dig out” the capabilities that T2I models have implicitly learned, and then explicitly use them to control the generation more granularly. Specifically, we propose to learn simple and lightweight T2I-Adapters to align internal knowledge in T2I models with external control signals, while freezing the original large T2I models. In this way, we can train various adapters according to different conditions, achieving rich control and editing effects in the color and structure of the generation results. Further, the proposed T2I-Adapters have attractive properties of practical value, such as composability and generalization ability. Extensive experiments demonstrate that our T2I-Adapter has promising generation quality and a wide range of applications.
中文速览
大规模文生图模型（text-to-image, T2I）其实已经在内部隐式地学会了颜色、边缘、结构等丰富知识，但单靠文字提示词却很难把这些能力精准地"激活"出来。为此，研究者提出了一种轻量级的"T2I-Adapter"，只需训练一个约77M参数的小网络，让它把外部的控制信号（如草图、深度图、语义分割、关键点、颜色调色板）映射到冻结的Stable Diffusion内部特征上，从而实现对生成图像在颜色和结构层面的精细控制。不同适配器可以即插即用，也可以多个叠加组合，还能直接迁移到从同一基础模型微调出的定制模型上，泛化能力强。实验表明，这一方案在几乎不改动原始大模型的前提下，显著提升了生成的可控性和精确度，为普通用户和创作者提供了低成本、高灵活性的图像定制手段。
原文 arXiv:2302.08453;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2302.08453v2