Latent Consistency Models: Synthesizing High-Resolution Images with Few-step Inference
Simian Luo*{}^{*}start_FLOATSUPERSCRIPT * end_FLOATSUPERSCRIPT Yiqin Tan Longbo Huang††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT Jian Li††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT Hang Zhao††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT Institute for Interdisciplinary Information Sciences, Tsinghua University {luosm22, {longbohuang, lijian83, Equal Contribution ††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPTCorresponding Authors
Abstract
Latent Diffusion models (LDMs) have achieved remarkable results in synthesizing high-resolution images. However, the iterative sampling process is computationally intensive and leads to slow generation. Inspired by Consistency Models (Song et al., 2023), we propose Latent Consistency Models (LCMs), enabling swift inference with minimal steps on any pre-trained LDMs, including Stable Diffusion (Rombach et al., 2022). Viewing the guided reverse diffusion process as solving an augmented probability flow ODE (PF-ODE), LCMs are designed to directly predict the solution of such ODE in latent space, mitigating the need for numerous iterations and allowing rapid, high-fidelity sampling. Efficiently distilled from pre-trained classifier-free guided diffusion models, a high-quality 768 $\times$ 768 2 $\sim$ 4-step LCM takes only 32 A100 GPU hours for training. Furthermore, we introduce Latent Consistency Fine-tuning (LCF), a novel method that is tailored for fine-tuning LCMs on customized image datasets. Evaluation on the LAION-5B-Aesthetics dataset demonstrates that LCMs achieve state-of-the-art text-to-image generation performance with few-step inference. Project Page: https://latent-consi
中文速览
扩散模型（Latent Diffusion Models）虽然能生成高质量图像，但每次都要跑几十上百步迭代采样，速度极慢、难以实时应用。本文提出潜在一致性模型（Latent Consistency Models，LCM），核心思路是把带分类器自由引导的反向扩散过程重新理解为一个增广的概率流常微分方程（PF-ODE），然后训练一个"一致性函数"直接在潜在空间预测该方程的终点解，跳过繁琐的逐步迭代。通过一阶段引导蒸馏和"跳步"加速技巧，只需32小时A100训练，就能把Stable Diffusion压缩成仅用2至4步即可生成768×768高清图像的LCM，并在LAION-5B美学数据集上达到当时最优的少步文生图效果；此外还提出了配套的潜在一致性微调方法（LCF），让LCM能高效适配定制数据集同时保留快速推理能力。这项工作首次将一致性模型的优势延伸到高分辨率文本到图像生成场景，大幅降低了高质量图像实时生成的门槛。
原文 arXiv:2310.04378;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.04378v1