MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Sen Xing Muyan Zhong Zeqiang Lai Liangchen Li Jiawen Liu Yaohui Wang Jifeng Dai Wenhai Wang
Abstract
In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages. Based on this insight, we introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases.
中文速览
用一个轻量级"语言适配器"插件，就能让原本只懂英文的图像生成模型（如 Stable Diffusion）流畅支持110多种语言——这是 MuLan 想解决的核心难题。研究者发现，用在海量嘈杂网络图文对上预训练的多语言文本编码器（如 InternVL），比依赖精心标注的多语言图像数据效率高得多；基于此，他们设计了一个不足2000万参数的可插拔适配器，只需冻结原有扩散模型和文本编码器，仅用约1700万条英文图文数据、在8张A100上训练约12小时，便让模型在其余110余种语言上取得了与英文几乎持平的生成质量（CLIP相似度英文39.57，其他语言均值39.61）。更关键的是，MuLan 无需为每种语言单独准备数据，还能无缝接入 LoRA、ControlNet、IP-Adapter 等社区工具，大幅降低了多语言图像生成的门槛，对全球化内容创作具有重要实用价值。
原文 arXiv:2412.01271;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2412.01271v2