Repurposing GANs for One-shot Semantic Part Segmentation
Nontawat Tritrong111Authors contributed equally to this work. Pitchaporn Rewatbowornwong111Authors contributed equally to this work. Supasorn Suwajanakorn VISTEC, Thailand {nontawat.t_s19, pitchaporn.r_s18,
Abstract
While GANs have shown success in realistic image generation, the idea of using GANs for other tasks unrelated to synthesis is underexplored. Do GANs learn meaningful structural parts of objects during their attempt to reproduce those objects? In this work, we test this hypothesis and propose a simple and effective approach based on GANs for semantic part segmentation that requires as few as one label example along with an unlabeled dataset. Our key idea is to leverage a trained GAN to extract a pixel-wise representation from the input image and use it as feature vectors for a segmentation network. Our experiments demonstrate that this GAN-derived representation is “readily discriminative” and produces surprisingly good results that are comparable to those from supervised baselines trained with significantly more labels. We believe this novel repurposing of GANs underlies a new class of unsupervised representation learning, which can generalize to many other tasks. More results are available at https://RepurposeGANs.github.io/.
中文速览
训练一个能用极少标注样本完成语义部件分割（semantic part segmentation）的模型，一直是计算机视觉的难题。研究者提出了一种巧妙的思路：利用已训练好的生成对抗网络（GAN）来提取每个像素的特征向量——GAN在学习合成逼真图像的过程中，其内部激活图已经隐式地编码了对象各部件的结构信息，把这些激活值拼接起来就能得到"天然具有判别性"的像素级表示，再用极少的带标注样本训练一个轻量分割网络即可完成部件分割。实验表明，仅凭一张标注图像，该方法就能在人脸、猫、汽车等多类对象上取得可与使用大量标注的监督方法媲美的分割精度，而且进一步提出的"auto-shot"扩展方案省去了耗时的隐空间优化，还能应对现实场景中的尺度和角度变化。这项工作揭示了GAN作为无监督表示学习工具的全新潜力，为数据标注昂贵的视觉任务提供了一条几乎不依赖人工标注的解决路径。
原文 arXiv:2103.04379;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.04379v5