SinDiffusion: Learning a Diffusion Model from a Single Natural Image
Weilun Wang11~{}^{1}, Jianmin Bao22~{}^{2}, Wengang Zhou1,414~{}^{1,4}, Dongdong Chen33~{}^{3}, Dong Chen22~{}^{2}, Lu Yuan33~{}^{3}, Houqiang Li1,414~{}^{1,4} 1 CAS Key Laboratory of GIPAS, EEIS Department, University of Science and Technology of China (USTC) 2 Microsoft Research Asia 3 Microsoft Cloud+AI 4 Institute of Artificial Intelligence, Hefei Comprehensive National Science Center Corresponding author: Jianmin Bao.
Abstract
We present SinDiffusion, leveraging denoising diffusion models to capture internal distribution of patches from a single natural image. SinDiffusion significantly improves the quality and diversity of generated samples compared with existing GAN-based approaches. It is based on two core designs. First, SinDiffusion is trained with a single model at a single scale instead of multiple models with progressive growing of scales which serves as the default setting in prior work. This avoids the accumulation of errors, which cause characteristic artifacts in generated results. Second, we identify that a patch-level receptive field of the diffusion network is crucial and effective for capturing the image’s patch statistics, therefore we redesign the network structure of the diffusion model. Coupling these two designs enables us to generate photorealistic and diverse images from a single image. Furthermore, SinDiffusion can be applied to various applications, i.e., text-guided image generation, and image outpainting, due to the inherent capability of diffusion models. Extensive experiments on a wide range of images demonstrate the superiority of our proposed method for modeling the patch d
中文速览
从单张自然图像中学习并生成多样化新图像，是图像生成领域的一个经典难题，现有基于GAN的方法（如SinGAN）通过多尺度渐进式训练来解决，但各尺度累积的细节误差会在最终结果中留下明显的视觉瑕疵。SinDiffusion提出用去噪扩散概率模型（DDPM）替代多尺度GAN架构，核心做法有两点：一是只在单一尺度上训练单个扩散模型，从根本上消除误差累积问题；二是重新设计去噪网络结构，减少下采样层数和残差块数量，将感受野限制在图块（patch）级别，迫使模型学习图像的局部统计规律而非死记整张训练图。实验表明，SinDiffusion在生成质量与多样性上均显著优于现有GAN方案，还可无需重新训练即扩展到文本引导图像生成、图像外推（outpainting）等多种应用，为单图生成任务提供了更稳健、更灵活的新基准。
原文 arXiv:2211.12445;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.12445v1