Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
Ziqiang Zhang∗ Long Zhou Chengyi Wang Sanyuan Chen Yu Wu Shujie Liu Zhuo Chen Yanqing Liu Huaming Wang Jinyu Li Lei He Sheng Zhao Furu Wei Microsoft https://github.com/microsoft/unilm Both authors contributed equally to this work. Correspondence:
Abstract
We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis. Specifically, we extend VALL-E (Wang et al., 2023) and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks. Experimental results show that it can generate high-quality speech in the target language via just one speech utterance in the source language as a prompt while preserving the unseen speaker’s voice, emotion, and acoustic environment. Moreover, VALL-E X effectively alleviates the foreign accent problems, which can be controlled by a language ID. Audio samples are available at https://aka.ms/vallex.
中文速览
跨语言语音合成长期面临说话人音色迁移难、外语口音重、零样本泛化差等痛点，微软研究团队提出了 VALL-E X——一个跨语言神经编解码语言模型（cross-lingual neural codec language model）。它在 VALL-E 的基础上，用英文语音（LibriLight，约6万小时）和中文语音（WenetSpeech，逾1万小时）联合训练一个多语言条件语言模型：把源语言语音和目标语言文本同时作为提示（prompt），让模型直接预测目标语言的声学离散编码（acoustic token），再由编解码器还原为波形。实验表明，只需一句源语言语音作为示例，VALL-E X 就能在目标语言中复现说话人的音色、情感和声学环境，同时借助语言ID有效消除外语口音；在零样本跨语言TTS和语音到语音翻译（S2ST）任务上，说话人相似度、语音自然度和翻译质量均大幅超越已有最强基线。这项工作的意义在于，它首次将大规模语言模型的上下文学习能力真正引入跨语言语音合成，为让任何人用母语腔调"说"另一种语言提供了切实可行的零样本方案。
原文 arXiv:2303.03926;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.03926v1