One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale
Fan Bao Shen Nie Kaiwen Xue Chongxuan Li Shi Pu Yaole Wang Gang Yue Yue Cao Hang Su Jun Zhu
Abstract
This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Our key insight is – learning diffusion models for marginal, conditional, and joint distributions can be unified as predicting the noise in the perturbed data, where the perturbation levels (i.e. timesteps) can be different for different modalities. Inspired by the unified view, UniDiffuser learns all distributions simultaneously with a minimal modification to the original diffusion model – perturbs data in all modalities instead of a single modality, inputs individual timesteps in different modalities, and predicts the noise of all modalities instead of a single modality. UniDiffuser is parameterized by a transformer for diffusion models to handle input types of different modalities. Implemented on large-scale paired image-text data, UniDiffuser is able to perform image, text, text-to-image, image-to-text, and image-text pair generation by setting proper timesteps without additional overhead. In particular, UniDiffuser is able to produce perceptually realistic samples in all tasks and its quantitative results (e.g., the FID and CLIP sc
中文速览
多模态生成模型长期存在一个痛点：文生图、图生文、无条件生成等任务各自需要独立训练专用模型，资源消耗大且难以统一。UniDiffuser 的核心发现是：对多模态数据的边缘分布、条件分布和联合分布进行扩散模型学习，可以统一为同一个噪声预测问题——只需允许不同模态拥有各自独立的加噪时间步。基于这一洞察，研究者在标准扩散模型上仅做最小改动，训练出一个以Transformer为骨干的统一模型UniDiffuser：在图文大规模数据上，只需在推理时调整各模态的时间步，就能无缝切换图像生成、文本生成、文生图、图生文以及图文对联合生成五项任务，且无需任何额外开销。实验表明，UniDiffuser在所有任务上均能生成视觉逼真的结果，定量指标（FID、CLIP分数）不仅超过现有通用生成模型，在文生图等代表性任务上甚至可与Stable Diffusion、DALL·E 2等专用模型媲美，为构建真正统一的多模态生成系统提供了简洁而有力的解决方案。
原文 arXiv:2303.06555;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.06555v2