Diffsound: Discrete Diffusion Model for Text-to-sound Generation
Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu, Dongchao Yang, Helin Wang, Wen Wang and Yuexian Zou are with the Advanced Data and Signal Processing laboratory, School of Electronic and Computer Engineering, Peking University, China. This work was done when Dongchao Yang was an intern at Tencent AI Lab.Jianwei Yu, Chao Weng and Dong Yu are with Tencent AI Lab.Yuexian Zou and Jianwei Yu are the corresponding authors.
Abstract
Generating sound effects that people want is an important topic. However, there are limited studies in this area for sound generation. In this study, we investigate generating sound conditioned on a text prompt and propose a novel text-to-sound generation framework that consists of a text encoder, a Vector Quantized Variational Autoencoder (VQ-VAE), a token-decoder, and a vocoder. The framework first uses the token-decoder to transfer the text features extracted from the text encoder to a mel-spectrogram with the help of VQ-VAE, and then the vocoder is used to transform the generated mel-spectrogram into a waveform. We found that the token-decoder significantly influences the generation performance. Thus, we focus on designing a good token-decoder in this study. We begin with the traditional autoregressive (AR) token-decoder, which has shown state-of-the-art performance in previous sound generation works. However, the AR token-decoder always predicts the mel-spectrogram tokens one by one in order, which may introduce the unidirectional bias and accumulation of errors problems. Moreover, with the AR token-decoder, the sound generation time increases linearly with the sound duration.
中文速览
想让AI根据一句文字描述自动生成对应的音效，但现有方法大多依赖自回归（autoregressive）解码器，存在单向预测偏差、误差逐步累积、生成速度慢等缺陷。为此，研究者提出了一套"文字转音效"框架，核心创新是用基于离散扩散模型（discrete diffusion model）的非自回归解码器 Diffsound 来替代传统自回归解码器——它能一次预测所有梅尔频谱（mel-spectrogram）的离散token，再通过多步迭代逐步精炼，从而同时利用全局上下文信息并避免误差积累。针对文本-音频配对数据稀缺的问题，团队还提出了基于掩码的文本生成策略，将大规模AudioSet数据集的事件标签自动转化为文字描述，并结合课程学习（curriculum learning）先易后难地预训练模型。实验结果表明，Diffsound在主观评分（MOS 3.56 vs. 2.786）和生成速度（快5倍）上均显著优于自回归基线，为音效自动生成这一几乎空白的研究方向提供了切实可行的解决方案。
原文 arXiv:2207.09983;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2207.09983v2