On Generative Spoken Language Modeling from Raw Audio
Kushal Lakhotia , Eugene Kharitonov∗, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte§, Tu-Anh Nguyen†, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, Emmanuel Dupoux‡ Facebook AI Research Author’s version of Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A.,、Dupoux, E. (2022). On Generative Spoken Language Modeling from Raw Audio, Transactions of the Association for Computational Linguistics equal contribution. ‡ Also at EHESS. † Also at INRIA. § Work done while at FAIR.
Abstract
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo-text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder-dependent way, and that some combinations approach text-based systems.111Evaluation code and trained models are here: https://github.com/pytorch/fairseq/tree/master/examples/textless_nlp/gslm. Sample audios are here: https://speechbot.github.io/gslm.
中文速览
不依赖任何文字或标注，直接从原始音频中学习一门语言的声学和语言规律——这就是"生成式口语语言建模"（Generative Spoken Language Modeling，GSLM）要攻克的核心难题。研究者搭建了一套完全无监督的流水线：先用离散语音编码器（CPC、wav2vec 2.0、HuBERT）把语音压缩成"伪文本"单元，再在这些单元上训练生成式语言模型，最后用语音合成器把单元还原为可听的波形；为了衡量系统好坏，他们还提出了一套基于现成 ASR 的自动评测指标，并用人工评测加以验证。实验系统地比较了三种编码器与三种码本大小（50/100/200个单元）的组合，发现单元数量的影响因任务和编码器而异，某些组合的表现已接近基于文字的系统。这项工作意义在于，它为"无文本 NLP"提供了可复现的基准与评测框架，有望让那些缺乏大规模文本资源的语言也能受益于现代语言技术。
原文 arXiv:2102.01192;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.01192v2