One-Shot Adaptation of GAN in Just One CLIP
Gihyun Kwon, and Jong Chul Ye Gihyun Kwon is with the Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, South Korea. E-mail: Jong Chul Ye is with the Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, South Korea. E-mail: Manuscript received December 5, 2022;
Abstract
There are many recent research efforts to fine-tune a pre-trained generator with a few target images to generate images of a novel domain. Unfortunately, these methods often suffer from overfitting or under-fitting when fine-tuned with a single target image. To address this, here we present a novel single-shot GAN adaptation method through unified CLIP space manipulations. Specifically, our model employs a two-step training strategy: reference image search in the source generator using a CLIP-guided latent optimization, followed by generator fine-tuning with a novel loss function that imposes CLIP space consistency between the source and adapted generators. To further improve the adapted model to produce spatially consistent samples with respect to the source generator, we also propose contrastive regularization for patchwise relationships in the CLIP space. Experimental results show that our model generates diverse outputs with the target texture and outperforms the baseline models both qualitatively and quantitatively. Furthermore, we show that our CLIP space manipulation strategy allows more effective attribute editing. Our Github source is as follows: https://github.com/cyclomo
中文速览
用一张图适配生成模型（GAN）时，既容易过拟合到这张图、又容易欠拟合而无法捕捉目标风格，现有方法在这一极端单样本场景下表现不稳定。本文提出一种基于 CLIP 语义空间的两阶段适配方法：第一步用 CLIP 相似度而非像素距离在源域生成器中搜索与目标图最接近的参考图，第二步以这张参考图为锚点微调生成器，同时引入样本级与图像块级（patchwise）的 CLIP 一致性损失，让适配后的模型在继承源域多样性的同时学到目标图的纹理和风格。实验表明，该方法在定性和定量指标上均优于现有少样本 GAN 适配基线，还能灵活支持文本引导的属性编辑，证明了把 CLIP 空间统一用于参考图搜索和训练正则化这一思路的有效性。
原文 arXiv:2203.09301;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.09301v4