Learning What and Where to Draw
Scott Reed1,、Zeynep Akata2、Santosh Mohan1 \ANDSamuel Tenka1、Bernt Schiele2、Honglak Lee1 \AFF1University of Michigan, Ann Arbor, USA \AFF2Max Planck Institute for Informatics, Saarbrücken, Germany Majority of this work was done while first author was at U. Michigan, but completed while at DeepMind.
Abstract
Generative Adversarial Networks (GANs) have recently demonstrated the capability to synthesize compelling real-world images, such as room interiors, album covers, manga, faces, birds, and flowers. While existing models can synthesize images based on global constraints such as a class label or caption, they do not provide control over pose or object location. We propose a new model, the Generative Adversarial What-Where Network (GAWWN), that synthesizes images given instructions describing what content to draw in which location. We show high-quality $128\times 128$ image synthesis on the Caltech-UCSD Birds dataset, conditioned on both informal text descriptions and also object location. Our system exposes control over both the bounding box around the bird and its constituent parts. By modeling the conditional distributions over part locations, our system also enables conditioning on arbitrary subsets of parts (e.g. only the beak and tail), yielding an efficient interface for picking part locations. We also show preliminary results on the more challenging domain of text- and location-controllable synthesis of images of human actions on the MPII Human Pose dataset.
中文速览
现有的图像生成模型虽然能根据文字描述合成图像，却无法控制物体出现在画面的哪个位置，也没法指定鸟嘴、翅膀等局部关键点的坐标。为此，论文提出了「生成对抗式什么-哪里网络」（Generative Adversarial What-Where Network，GAWWN），通过空间变换器将边界框或关键点坐标与文字描述一同送入生成器和判别器，让网络同时学会"画什么"和"画在哪里"。在 Caltech-UCSD 鸟类数据集上，该方法实现了 128×128 高质量图像合成，用户既可以框定鸟的大致位置，也可以只指定少数几个部位（比如仅给出嘴和尾巴的坐标），系统会用一个额外的条件 GAN 自动补全其余关键点并生成完整图像；此外，论文还初步验证了该框架在人体动作图像合成任务上的可行性。这项工作的意义在于，它首次将空间位置控制与文本语义控制结合进一个统一的生成模型，为更灵活、更可控的图像创作工具奠定了基础。
原文 arXiv:1610.02454;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1610.02454v1