Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
Yonatan Belinkov and James Glass Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA 02139 {belinkov,
Abstract
Neural models have become ubiquitous in automatic speech recognition systems. While neural networks are typically used as acoustic models in more complex systems, recent studies have explored end-to-end speech recognition systems based on neural networks, which can be trained to directly predict text from input acoustic features. Although such systems are conceptually elegant and simpler than traditional systems, it is less obvious how to interpret the trained models.
中文速览
端到端语音识别（end-to-end ASR）系统能直接把声学特征映射成文字，但它在内部到底学到了什么语音学知识，一直不够清楚。研究者选取基于卷积和循环网络、以CTC为训练目标的DeepSpeech2模型，把它各层输出的帧级特征送入一个独立的音素分类器，用分类准确率来衡量不同层所携带的语音学信息质量。实验发现：第一层卷积能提升音素表示质量，第二层卷积反而大幅降低准确率；前几层循环网络持续改善表示，但更高层的准确率又开始下滑，说明顶层更关注长距离的字符序列建模，而非局部音素信息。这项分析揭示了端到端模型各层分工的内在规律，为多任务学习、网络结构设计等方向提供了可量化的参考依据。
原文 arXiv:1709.04482;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.04482v1