Ensembling with Deep Generative Views
Lucy Chai1,2 Jun-Yan Zhu2,3 Eli Shechtman2 Phillip Isola1 Richard Zhang2 1MIT 2Adobe Research 3CMU {lrchai, {elishe,
Abstract
Recent generative models can synthesize “views” of artificial images that mimic real-world variations, such as changes in color or pose, simply by learning from unlabeled image collections. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. Using a pretrained generator, we first find the latent code corresponding to a given real input image. Applying perturbations to the code creates natural variations of the image, which can then be ensembled together at test-time. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars. Critically, we find that several design decisions are required towards making this process work; the perturbation procedure, weighting between the augmentations and original image, and training the classifier on synthesized images can all impact the result. Currently, we find that while test-time ensembling with GAN-based augmentations can offer some small improvements, the remaining bottlenecks are the efficiency and accuracy of the GAN reconstructions, coupled with classifier
中文速览
生成对抗网络（GAN）已经能合成逼真图像，但能否把它产生的图像变体直接用于提升真实图像的分类性能，目前还缺乏系统研究。研究者提出了一套基于 StyleGAN2 的测试时集成（test-time ensembling）方案：先把待分类的真实图像"投影"回 GAN 的隐空间，再对隐编码施加微小扰动，从而生成该图像的多个自然变体，最后将分类器对这些变体的预测结果取平均以提升准确率。实验覆盖人脸属性、猫脸和汽车三类任务，结果显示该方法确实能带来一定改进，但提升幅度有限，主要瓶颈在于 GAN 重建的精度与效率不足，以及分类器对 GAN 生成图像中残留伪影的敏感性。这项工作揭示了将生成模型作为数据增强工具的潜力与现实局限，为后续如何缩小生成图像与真实图像之间的域差距指明了方向。
原文 arXiv:2104.14551;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.14551v1