Universal Style Transfer via Feature Transforms
Yijun Li UC Merced、Chen Fang Adobe Research、Jimei Yang Adobe Research、Zhaowen Wang Adobe Research、Xin Lu Adobe Research、Ming-Hsuan Yang UC Merced, NVIDIA Research
Abstract
Universal style transfer aims to transfer arbitrary visual styles to content images. Existing feed-forward based methods, while enjoying the inference efficiency, are mainly limited by inability of generalizing to unseen styles or compromised visual quality. In this paper, we present a simple yet effective method that tackles these limitations without training on any pre-defined styles. The key ingredient of our method is a pair of feature transforms, whitening and coloring, that are embedded to an image reconstruction network. The whitening and coloring transforms reflect a direct matching of feature covariance of the content image to a given style image, which shares similar spirits with the optimization of Gram matrix based cost in neural style transfer. We demonstrate the effectiveness of our algorithm by generating high-quality stylized images with comparisons to a number of recent methods. We also analyze our method by visualizing the whitened features and synthesizing textures via simple feature coloring.
中文速览
把任意风格迁移到内容图像上，现有的前向网络方法要么泛化能力差（只能处理训练时见过的风格），要么视觉质量差，难以兼顾。本文提出了一种无需在任何预定义风格上训练的通用风格迁移方法，核心手段是将"白化与着色变换"（Whitening and Coloring Transforms，WCT）嵌入一个图像重建网络：白化步骤把内容图像的特征协方差矩阵"归零"，去除其原有风格信息；着色步骤再将风格图像的特征协方差注入进来，从而实现统计意义上的精确风格匹配，并进一步在五个不同层级上依次施加WCT，形成由粗到细的多级风格化流水线。实验表明，该方法在任意风格上均能生成视觉质量高、细节丰富的风格化图像，同时兼具前向推理的效率优势，还可推广到通用纹理合成任务，为真正意义上的"无限制风格迁移"提供了一个简洁而有效的解决方案。
原文 arXiv:1705.08086;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1705.08086v2