AUDIO STYLE TRANSFER
Abstract
“Style transfer” among images has recently emerged as a very active research topic, fuelled by the power of convolution neural networks (CNNs), and has become fast a very popular technology in social media. This paper investigates the analogous problem in the audio domain: How to transfer the style of a reference audio signal to a target audio content? We propose a flexible framework for the task, which uses a sound texture model to extract statistics characterizing the reference audio style, followed by an optimization-based audio texture synthesis to modify the target content. In contrast to mainstream optimization-based visual transfer method, the proposed process is initialized by the target content instead of random noise and the optimized loss is only about texture, not structure. These differences proved key for audio style transfer in our experiments. In order to extract features of interest, we investigate different architectures, whether pre-trained on other tasks, as done in image style transfer, or engineered based on the human auditory system. Experimental results on different types of audio signal confirm the potential of the proposed approach.
中文速览
如何把一段音频的"风格"迁移到另一段音频的内容上，是近年来随着图像风格迁移流行而自然引出的新问题，但直接套用图像方法效果很差，结果往往像两段声音的混合而非风格化的内容。为此，研究者提出了一套灵活的音频风格迁移框架：先用声音纹理模型（sound texture model）从参考音频中提取刻画风格的统计特征，再以目标内容音频作为初始点（而非随机噪声），仅优化纹理风格损失，通过梯度下降逐步将风格融入内容。框架兼容多种特征提取方式，涵盖在其他任务上预训练的深度网络（VGG-19、SoundNet）、随机浅层网络，以及仿人类听觉系统手工设计的 McDermott 听觉模型，实验在语音、音乐、自然声等多类音频上均验证了方法的有效性。这项工作首次系统性地解决了音频风格迁移的核心难题，为专业音频编辑、音乐创作和影视声音设计等应用场景提供了重要基础。
原文 arXiv:1710.11385;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1710.11385v3