Swapping Autoencoder for Deep Image Manipulation
Taesung Park12 Jun-Yan Zhu23 Oliver Wang2 Jingwan Lu2 Eli Shechtman2 Alexei A. Efros12 Richard Zhang2 1UC Berkeley 2Adobe Research 3CMU
Abstract
Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging. We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation, rather than random sampling. The key idea is to encode an image into two independent components and enforce that any swapped combination maps to a realistic image. In particular, we encourage the components to represent structure and texture, by enforcing one component to encode co-occurrent patch statistics across different parts of the image. As our method is trained with an encoder, finding the latent codes for a new input image becomes trivial, rather than cumbersome. As a result, our method enables us to manipulate real input images in various ways, including texture swapping, local and global editing, and latent code vector arithmetic. Experiments on multiple datasets show that our model produces better results and is substantially more efficient compared to recent generative models.
中文速览
交换自编码器（Swapping Autoencoder）针对的是一个核心难题：现有深度生成模型擅长随机生成逼真图像，却很难用来精确操控一张已有的真实图片。方法的核心思路是训练一个自编码器，把输入图像分解成两个独立的潜在编码——一个带空间维度的"结构码"和一个全局向量形式的"纹理码"，并通过对抗训练强制任意组合这两个编码都能生成真实图像；同时引入一个"图块共现判别器"，要求生成图像中任意局部图块与原始纹理图像的多处图块在统计上保持一致，从而让两个编码真正分别捕捉结构与纹理。在教堂、卧室、人脸等多个数据集上的实验表明，该方法不仅在纹理迁移质量上可与专门的风格迁移方法媲美，还支持局部/全局编辑、潜在码向量运算等多种操作，且因为自带编码器，对新图像的嵌入速度比需要优化反推的GAN方法快了数个数量级。这项工作的意义在于，它提供了一个无需任何配对标注、任务无关的统一框架，让"用数据集知识来编辑真实图片"这件事变得既灵活又高效。
原文 arXiv:2007.00653;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2007.00653v2