Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance
Bahjat Kawar Computer Science Department Technion, Israel Roy Ganz Electrical Engineering Department Technion, Israel Michael Elad Computer Science Department Technion, Israel
Abstract
Denoising diffusion probabilistic models (DDPMs) are a recent family of generative models that achieve state-of-the-art results. In order to obtain class-conditional generation, it was suggested to guide the diffusion process by gradients from a time-dependent classifier. While the idea is theoretically sound, deep learning-based classifiers are infamously susceptible to gradient-based adversarial attacks. Therefore, while traditional classifiers may achieve good accuracy scores, their gradients are possibly unreliable and might hinder the improvement of the generation results. Recent work discovered that adversarially robust classifiers exhibit gradients that are aligned with human perception, and these could better guide a generative process towards semantically meaningful images. We utilize this observation by defining and training a time-dependent adversarially robust classifier and use it as guidance for a generative diffusion model. In experiments on the highly challenging and diverse ImageNet dataset, our scheme introduces significantly more intelligible intermediate gradients, better alignment with theoretical findings, as well as improved generation results under several e
中文速览
扩散模型（diffusion model）生成图像时可以借助分类器的梯度来控制生成内容的类别，但普通分类器天生容易被对抗攻击欺骗，其梯度往往指向无意义的方向，导致生成质量受限。这篇论文指出了这一根本缺陷，并提出用对抗鲁棒分类器（adversarially robust classifier）来替代普通分类器提供引导梯度——鲁棒分类器的梯度在感知上与人类视觉对齐，能真正指向目标类别的语义特征。作者专门训练了一个接受扩散时间步作为输入的鲁棒时间相关分类器，并将其嵌入扩散采样过程，在极具挑战性的 ImageNet 数据集上进行验证，结果在 FID 等多项客观指标上超过了原始分类器引导方法，人工评估调查也显示人类评测者明显偏好新方法生成的图像。这项工作揭示了分类器梯度质量对扩散生成至关重要，为提升大规模类别条件图像生成提供了一条简洁有效的路径。
原文 arXiv:2208.08664;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2208.08664v2