Null-text Inversion for Editing Real Images using Guided Diffusion Models
Ron Mokady*** ††† 1,2, Amir HertzNull-text Inversion for Editing Real Images using Guided Diffusion Models Null-text Inversion for Editing Real Images using Guided Diffusion Models 1,2, Kfir Aberman1, Yael Pritch1, and Daniel Cohen-OrNull-text Inversion for Editing Real Images using Guided Diffusion Models 1,2 1Google Research, 2The Blavatnik School of Computer Science, Tel Aviv University
Abstract
Recent text-guided diffusion models provide powerful image generation capabilities. Currently, a massive effort is given to enable the modification of these images using text only as means to offer intuitive and versatile editing. To edit a real image using these state-of-the-art tools, one must first invert the image with a meaningful text prompt into the pretrained model’s domain. In this paper, we introduce an accurate inversion technique and thus facilitate an intuitive text-based modification of the image. Our proposed inversion consists of two novel key components: (i) Pivotal inversion for diffusion models. While current methods aim at mapping random noise samples to a single input image, we use a single pivotal noise vector for each timestamp and optimize around it. We demonstrate that a direct inversion is inadequate on its own, but does provide a good anchor for our optimization. (ii) null-text optimization, where we only modify the unconditional textual embedding that is used for classifier-free guidance, rather than the input text embedding. This allows for keeping both the model weights and the conditional embedding intact and hence enables applying prompt-based editin
中文速览
想对真实照片做文字驱动的精准编辑，关键难题在于如何把一张真实图像"反推"回扩散模型（diffusion model）的噪声空间，同时还要保住模型的编辑能力。现有的DDIM反演方法在加入无分类器引导（classifier-free guidance）后会积累误差、导致重建失真。为此，本文提出"null-text inversion"方法，结合两个核心设计：一是以DDIM反演轨迹为锚点（pivot）做局部优化，避免从随机噪声出发的低效搜索；二是只优化无条件文本嵌入（即"空文本"对应的embedding），而不改动模型权重或条件文本嵌入，从而在精确重建图像的同时完整保留模型的编辑先验。基于公开的Stable Diffusion模型，该方法在多种真实图像上实现了近乎完美的重建，并首次让Prompt-to-Prompt这类仅靠改写提示词就能局部或全局修改图像的编辑技术成功适用于真实照片，为直觉式文字图像编辑提供了一条无需微调模型、轻量高效的新路径。
原文 arXiv:2211.09794;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.09794v1