IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models
Hu Ye, Jun Zhang , Sibo Liu, Xiao Han, Wei Yang Tencent AI Lab {huye, junejzhang, siboliu, haroldhan, Corresponding author
Abstract
Recent years have witnessed the strong power of large text-to-image diffusion models for the impressive generative capability to create high-fidelity images. However, it is very tricky to generate desired images using only text prompt as it often involves complex prompt engineering. An alternative to text prompt is image prompt, as the saying goes: "an image is worth a thousand words". Although existing methods of direct fine-tuning from pretrained models are effective, they require large computing resources and are not compatible with other base models, text prompt, and structural controls. In this paper, we present IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pretrained text-to-image diffusion models. The key design of our IP-Adapter is decoupled cross-attention mechanism that separates cross-attention layers for text features and image features. Despite the simplicity of our method, an IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fully fine-tuned image prompt model. As we freeze the pretrained diffusion model, the proposed IP-Adapter can be generalized not only to other custom models fin
中文速览
用图像作为生成提示词（image prompt）来控制文生图扩散模型，比写文字提示词更直观，但现有方法要么需要对整个模型大规模微调、破坏原有文字生成能力，要么只是简单地把图像特征拼接到文字特征里、效果远不如微调模型。IP-Adapter 提出了一种"解耦交叉注意力"机制，在冻结原始扩散模型的前提下，为每个交叉注意力层单独增加一路专门处理图像特征的注意力层，只训练这22M新增参数。实验结果表明，IP-Adapter 的生成质量可以媲美甚至超越完整微调的图像提示模型，同时还能无缝兼容同底座的其他定制模型、ControlNet 等结构控制工具，以及与文字提示联合使用实现多模态生成。这项工作的价值在于，它以极低的训练成本和极强的灵活性，把图像提示能力"插件化"地赋予了现有文生图模型，为内容创作提供了一个轻量、通用的新范式。
原文 arXiv:2308.06721;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.06721v1