PEA-Diffusion: Parameter-Efficient Adapter with Knowledge Distillation in non-English Text-to-Image Generation
Jian Ma\orcidlink0009-0004-0057-3033 Chen Chen\orcidlink0000-0003-3498-2527† Qingsong Xie\orcidlink0000-0001-6974-2019 Haonan Lu\orcidlink0000-0001-6332-2785†
Abstract
Text-to-image diffusion models are well known for their ability to generate realistic images based on textual prompts. However, the existing works have predominantly focused on English, lacking support for non-English text-to-image models. The most commonly used translation methods cannot solve the generation problem related to language culture, while training from scratch on a specific language dataset is prohibitively expensive. In this paper, we are inspired to propose a simple plug-and-play language transfer method based on knowledge distillation. All we need to do is train a lightweight MLP-like parameter-efficient adapter (PEA) with only 6M parameters under teacher knowledge distillation along with a small parallel data corpus. We are surprised to find that freezing the parameters of UNet can still achieve remarkable performance on the language-specific prompt evaluation set, demonstrating that PEA can stimulate the potential generation ability of the original UNet. Additionally, it closely approaches the performance of the English text-to-image model on a general prompt evaluation set. Furthermore, our adapter can be used as a plugin to achieve significant results in downstr
中文速览
主流的文字生成图像（text-to-image）模型几乎只支持英文提示词，非英语用户若直接翻译使用，不仅容易出错，更会丢失本民族特有的文化概念。为此，本文提出 PEA-Diffusion，其核心是在语言特定的 CLIP 文字编码器后面插入一个仅有 600 万参数的轻量 MLP 适配器（parameter-efficient adapter，PEA），并通过知识蒸馏（knowledge distillation）让预训练英文扩散模型充当"教师"，在特征图和输出层两个层面监督"学生"模型对齐，从而把英文 Stable Diffusion 迁移到中文等其他语言，整个过程只需冻结 UNet 权重、用少量平行语料训练这个小适配器。实验表明，PEA-Diffusion 在语言文化特定提示词的评测上达到了同类模型的最优效果，同时在通用提示词上几乎保留了原英文模型的生成能力，还能即插即用地与 LoRA、ControlNet、Inpainting 等下游任务无缝结合。这项工作以极低的训练代价打通了英文扩散模型生态与非英语社区之间的壁垒，让更多语言的用户可以直接受益于已有的庞大英文模型社区资源。
原文 arXiv:2311.17086;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.17086v2