AltDiffusion: A Multilingual Text-to-Image Diffusion Model
Fulong Ye 111Equal contribution. 222Work done during internship with Beijing Academy of Artificial Intelligence. 1 2, Guang liu 111Equal contribution. 2, Xinya Wu 2, Ledell Wu 2
Abstract
Large Text-to-Image(T2I) diffusion models have shown a remarkable capability to produce photorealistic and diverse images based on text inputs. However, existing works only support limited language input, e.g., English, Chinese, and Japanese, leaving users beyond these languages underserved and blocking the global expansion of T2I models. Therefore, this paper presents AltDiffusion, a novel multilingual T2I diffusion model that supports eighteen different languages111Eighteen languages: English, Chinese, Japanese, Thai, Korean, Hindi, Ukrainian, Arabic, Turkish, Vietnamese, Polish, Dutch, Portuguese, Italian, Spanish, German, French, and Russian.. Specifically, we first train a multilingual text encoder based on the knowledge distillation. Then we plug it into a pretrained English-only diffusion model and train the model with a two-stage schema to enhance the multilingual capability, including concept alignment and quality improvement stage on a large-scale multilingual dataset. Furthermore, we introduce a new benchmark, which includes Multilingual-General-18(MG-18) and Multilingual-Cultural-18(MC-18) datasets, to evaluate the capabilities of T2I diffusion models for generating hig
中文速览
现有的文生图（Text-to-Image）扩散模型大多只支持英语，导致全球大量非英语用户不得不借助翻译工具来使用，这不仅带来翻译误差，还会丢失文化特有概念的语义信息。为此，研究者提出了 AltDiffusion，先通过知识蒸馏训练一个支持18种语言的多语言文本编码器，再将其接入预训练的英语扩散模型，并经过"概念对齐"和"质量提升"两阶段训练，使模型真正具备多语言理解和高质量生图能力。为衡量模型表现，他们还构建了两个评测数据集：面向通用生成质量的 MG-18 和专门考察文化特有概念的 MC-18（后者是该领域首个此类数据集）。实验结果表明，AltDiffusion 在多语言理解上全面超越基于翻译的 Stable Diffusion，尤其在文化特有概念的识别上优势显著，同时图像生成质量与现有最优模型持平，为全球用户平等使用文生图技术迈出了重要一步。
原文 arXiv:2308.09991;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.09991v2