Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks
Peihao Zhu KAUST、Rameen Abdal KAUST、John Femiani Miami University、Peter Wonka KAUST
Abstract
We present a new method for one shot domain adaptation. The input to our method is a trained GAN that can produce images in domain $A$ and a single reference image $I_{B}$ from domain $B$ . The proposed algorithm can translate any output of the trained GAN from domain $A$ to domain $B$ . There are two main advantages of our method compared to the current state of the art: First, our solution achieves higher visual quality, e.g. by noticeably reducing overfitting. Second, our solution allows for more degrees of freedom to control the domain gap, i.e. what aspects of the image $I_{B}$ are used to define the domain $B$ . Technically, we realize the new method by building on a pre-trained StyleGAN generator as GAN and a pre-trained CLIP model for representing the domain gap. We propose several new regularizers for controlling the domain gap to optimize the weights of the pre-trained StyleGAN generator so that it will output images in domain $B$ instead of domain $A$ . The regularizers prevent the optimization from taking on too many attributes of the single reference image. Our results show significant visual improvements over the state of the art as well as multiple applications that
中文速览
单张参考图就能完成风格域迁移（one-shot domain adaptation）一直面临过拟合严重、生成结果千篇一律的痛点——新方法针对这一问题，将预训练的 StyleGAN2 与 CLIP 图像嵌入相结合，通过"反向域投影"先把参考图 $I_B$ 映射回源域 $A$，从而精准计算两个域之间的语义差距向量，而非像前人方法那样粗糙地用域均值来代替。在此基础上，研究者提出了多个新正则化项来约束微调过程，防止模型把所有生成结果都拉向同一张参考图的外观，同时改进了 StyleGAN-NADA 中层选择的启发式策略，使微调更稳健。实验结果表明，该方法在人脸等多个场景下显著优于现有最优方法，伪影更少、身份保留更好，还支持姿态、光照、表情、纹理等多种下游编辑应用，整个流程仅需在单张 GPU 上运行几分钟，具备较强的实用价值。
原文 arXiv:2110.08398;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.08398v2