Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
Jesse Engel Cinjon Resnick Adam Roberts Sander Dieleman Douglas Eck Karen Simonyan Mohammad Norouzi
Abstract
Generative models in vision have seen rapid progress due to algorithmic improvements and the availability of high-quality image datasets. In this paper, we offer contributions in both these areas to enable similar progress in audio modeling. First, we detail a powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality dataset of musical notes that is an order of magnitude larger than comparable public datasets. Using NSynth, we demonstrate improved qualitative and quantitative performance of the WaveNet autoencoder over a well-tuned spectral autoencoder baseline. Finally, we show that the model learns a manifold of embeddings that allows for morphing between instruments, meaningfully interpolating in timbre to create new types of sounds that are realistic and expressive.
中文速览
音频合成长期依赖手工设计的合成器，缺乏能从数据中自动学习声音表示的大规模基准。研究者提出了一种基于WaveNet的自编码器（WaveNet Autoencoder），让模型先用编码器从原始音频波形中提取带时间结构的隐编码，再用解码器依据这些编码重建音频，从而无需外部条件信号就能捕捉较长时程的音色与动态特征。与此同时，他们发布了迄今规模最大的乐器音符数据集NSynth，包含约30万条来自千余种乐器的4秒标注音频，比同类公开数据集大一个数量级。实验表明，WaveNet自编码器在重建质量上优于基于频谱图的基线模型，并且学到的嵌入空间具有语义连续性——在两种乐器的嵌入之间做插值，能生成音色平滑过渡、听感真实的全新声音。这项工作为数据驱动的神经音频合成提供了可复现的模型框架和标准数据集，有望推动音乐生成与语音合成领域取得类似图像生成领域的突破性进展。
原文 arXiv:1704.01279;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1704.01279v1