Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artwork
Alex J. Champandard nucl.ai Research Laboratory、nucl.ai Conference 2016 Artificial Intelligence in Creative Industries July 18-20, Vienna/Austria. http://events.nucl.ai/ This research was funded out of the marketing budget.
Abstract
Convolutional neural networks (CNNs) have proven highly effective at image synthesis and style transfer. For most users, however, using them as tools can be a challenging task due to their unpredictable behavior that goes against common intuitions. This paper introduces a novel concept to augment such generative architectures with semantic annotations, either by manually authoring pixel labels or using existing solutions for semantic segmentation. The result is a content-aware generative algorithm that offers meaningful control over the outcome. Thus, we increase the quality of images generated by avoiding common glitches, make the results look significantly more plausible, and extend the functional range of these algorithms—whether for portraits or landscapes, etc. Applications include semantic style transfer and turning doodles with few colors into masterful paintings!
中文速览
风格迁移（style transfer）算法虽然效果惊艳，却常常产生令人抓狂的"错位"——人像的背景纹理跑到皮肤上、植被出现在天空里，普通用户根本无法预测和控制结果。为此，作者提出在现有的基于图块匹配（patch-based）的卷积神经网络生成架构中，额外拼接一组语义标注通道：用户只需手工涂几种颜色（或接入现成的语义分割网络）为图像的不同区域打上标签，算法在寻找最近邻图块时就会优先在语义相符的区域之间进行风格匹配，从而让头发对头发、皮肤对皮肤、天空对天空地完成迁移。实验表明，加入语义图后，人像和风景的风格迁移质量显著提升，常见的纹理错位和重复图块伪影大幅减少，同时风格权重参数的可调范围也更宽、更稳健，甚至能把几笔涂鸦直接转化为风格统一的绘画作品。这项工作的意义在于，它用极低的额外成本（一张简单的色块标注图）就把语义理解引入了生成式算法，让普通用户也能对神经风格迁移的结果进行有意义的精确控制。
原文 arXiv:1603.01768;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1603.01768v1