Neural Voice Cloning with a Few Samples
Sercan Ö. Arık、Jitong Chen∗、Kainan Peng∗、Wei Ping∗、Yanqi Zhou Baidu Research 1195 Bordeaux Dr. Sunnyvale, CA 94089 Equal contribution
Abstract
Voice cloning is a highly desired feature for personalized speech interfaces. We introduce a neural voice cloning system that learns to synthesize a person’s voice from only a few audio samples. We study two approaches: speaker adaptation and speaker encoding. Speaker adaptation is based on fine-tuning a multi-speaker generative model. Speaker encoding is based on training a separate model to directly infer a new speaker embedding, which will be applied to a multi-speaker generative model. In terms of naturalness of the speech and similarity to the original speaker, both approaches can achieve good performance, even with a few cloning audios. 111Cloned audio samples can be found in https://audiodemos.github.io While speaker adaptation can achieve slightly better naturalness and similarity, cloning time and required memory for the speaker encoding approach are significantly less, making it more favorable for low-resource deployment.
中文速览
只需要少量音频样本就能"克隆"一个人的声音——这是语音个性化交互长期追求的目标，但现有方法要么需要大量数据，要么克隆速度太慢。研究者提出了一套神经网络声音克隆系统，并对比了两种路线：一是"说话人自适应"（speaker adaptation），即用少量目标音频对已训练好的多说话人模型做微调；二是"说话人编码"（speaker encoding），即单独训练一个编码器网络，直接从目标音频中推断出代表其音色的低维向量，无需对主模型做任何微调。实验表明，两种方法仅凭几条甚至几秒钟的音频就能产生自然度和相似度都不错的克隆语音，其中自适应方式质量略优，但编码方式所需的克隆时间和计算资源大幅减少，更适合资源受限的实际部署场景。这项工作不仅推动了个性化语音合成的落地，还提出了基于说话人分类和说话人验证的自动评估方法，并展示了通过向量插值实现性别和口音渐变的有趣应用。
原文 arXiv:1802.06006;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.06006v3