SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
Abstract
While various end-to-end models for spoken language understanding tasks have been explored recently, this paper is probably the first known attempt to challenge the very difficult task of end-to-end spoken question answering (SQA). Learning from the very successful BERT model for various text processing tasks, here we proposed an audio-and-text jointly learned SpeechBERT model. This model outperformed the conventional approach of cascading ASR with the following text question answering (TQA) model on datasets including ASR errors in answer spans, because the end-to-end model was shown to be able to extract information out of audio data before ASR produced errors. When ensembling the proposed end-to-end model with the cascade architecture, even better performance was achieved. In addition to the potential of end-to-end SQA, the SpeechBERT can also be considered for many other spoken language understanding tasks just as BERT for many text processing tasks.
中文速览
语音问答（Spoken QA）的传统做法是先用语音识别把音频转成文字、再套用文本问答模型，但识别错误会让答案丢失且无从弥补。为此，研究者提出 SpeechBERT——一个将音频与文本联合训练的预训练模型，让模型直接从语音信号中提取语义，不再完全依赖识别结果是否正确。具体方法是先用文本 BERT 学好词语的语义表示，再训练一个自编码器让音频词的嵌入向量同时保留发音结构和文本语义，最后在混合音频与文本的掩码语言模型任务上预训练，并在语音问答数据集上微调。实验表明，在答案片段存在识别错误的测试集上，端到端的 SpeechBERT 优于级联方式，两者集成后效果更佳，证明了直接从语音中"读"出答案的可行性。这项工作是首个将端到端方法应用于如此复杂的语音问答任务的尝试，为后续语音语言理解研究提供了重要参考。
原文 arXiv:1910.11559;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.11559v4