AdaSpeech: Adaptive Text to Speech for Custom Voice
Mingjian Chen∗, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, Sheng Zhao, Tie-Yan Liu Microsoft Research Asia, Microsoft Azure Speech The first two authors contribute equally to this work. Corresponding author: Xu Tan,
Abstract
Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech from her/him. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions which could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we model the acoustic information in both utterance and phoneme level. Specifically, we use one acoustic encoder to extract an utterance-level vector and another one to extract a sequence of phoneme-level vectors from the target speech during pre-training and fine-tuning; in inference, we extract the utterance-level vector from a reference speech
中文速览
定制语音（Custom Voice）系统面临两大难题：一是用户录音的音色、语调、录音环境等声学条件五花八门，与训练源模型的数据差距极大；二是要服务海量用户，每位用户占用的模型参数必须尽可能小，否则存储成本会爆炸式增长。为此，研究者提出了 AdaSpeech，通过两项核心设计来应对挑战：一是"声学条件建模"，在预训练和微调阶段同时提取话语级（utterance-level）和音素级（phoneme-level）声学向量，让模型更好地理解各种声学差异，推理时再用预测器补全这些信息；二是"条件层归一化"（Conditional Layer Normalization），用说话人嵌入动态生成解码器中层归一化的缩放和偏置参数，使得微调时只需更新这少量参数。实验在 LibriTTS 上预训练、在 VCTK 和 LJSpeech 上用约 20 句话微调，结果显示 AdaSpeech 在自然度和相似度评分（MOS/SMOS）上均显著优于基线方法，每位说话人仅需约 5K 专属参数。这项工作的重要意义在于，它首次在保证高质量个性化语音的同时，将每用户的额外存储压缩到几乎可忽略的量级，为大规模商业定制语音服务提供了切实可行的技术路径。
原文 arXiv:2103.00993;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.00993v1