Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chengyi Wang∗ Sanyuan Chen∗ Yu Wu∗ Ziqiang Zhang Long Zhou Shujie Liu Zhuo Chen Yanqing Liu Huaming Wang Jinyu Li Lei He Sheng Zhao Furu Wei Microsoft https://github.com/microsoft/unilm
Abstract
We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called VALL-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. VALL-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that VALL-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find VALL-E could preserve the speaker’s emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.
中文速览
零样本语音合成（zero-shot TTS）长期面临一个难题：如果说话人从未出现在训练数据中，合成出的声音就很难还原其音色和自然度。VALL-E 把这个问题重新定义为一个语言模型任务——先用现成的神经音频编解码器（EnCodec）把语音压缩成离散的"声学 token"，再训练一个大型语言模型，像 GPT 预测下一个词那样，根据文本音素序列和仅 3 秒的参考录音来预测这些声学 token，最后再用解码器还原成波形。为了让模型真正"见多识广"，研究者用了足足 6 万小时的英语语音（LibriLight）进行预训练，规模比同类系统大出数百倍。实验结果显示，VALL-E 在说话人相似度和语音自然度上大幅超越此前最强的零样本 TTS 系统，甚至在部分测试集上与真人录音旗鼓相当，还能自动保留参考音频里的情绪和混响环境——这说明把海量半监督数据与语言模型范式结合起来，是打破零样本语音合成瓶颈的一条切实可行的新路。
原文 arXiv:2301.02111;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.02111v1