Feedforward Sequential Memory Neural Networks without Recurrent Feedback
Shiliang Zhang1, Hui Jiang2, Si Wei3, Lirong Dai1 1National Engineering Laboratory for Speech and Language Information Processing University of Science and Technology of China, Hefei, Anhui, China 2Department of Electrical Engineering and Computer Science York University, 4700 Keele Street, Toronto, Ontario, M3J 1P3, Canada 3IFLYTEK Research, Hefei, Anhui, China
Abstract
We introduce a new structure for memory neural networks, called feedforward sequential memory networks (FSMN), which can learn long-term dependency without using recurrent feedback. The proposed FSMN is a standard feedforward neural networks equipped with learnable sequential memory blocks in the hidden layers. In this work, we have applied FSMN to several language modeling (LM) tasks. Experimental results have shown that the memory blocks in FSMN can learn effective representations of long history. Experiments have shown that FSMN based language models can significantly outperform not only feedforward neural network (FNN) based LMs but also the popular recurrent neural network (RNN) LMs.
中文速览
长期依赖（long-term dependency）是序列数据建模的核心难题，循环神经网络（RNN）虽能捕捉历史信息，却因反向传播中的梯度消失/爆炸问题而难以训练。本文提出一种名为前馈序列记忆网络（Feedforward Sequential Memory Network，FSMN）的新结构，在标准前馈网络的隐藏层中引入可学习的记忆块，用类似有限冲激响应（FIR）滤波器的方式对历史隐层状态做加权求和，从而在不使用任何循环连接的情况下编码长程历史信息。在 Penn Treebank 和大规模英文维基百科两个语言模型基准上，FSMN 的困惑度（perplexity）显著低于传统前馈网络和主流 RNN 语言模型，且由于所有运算均可转化为矩阵乘法，训练效率与普通前馈网络相当。这一工作表明，无需循环结构也能有效捕获序列长期依赖，为语音识别、自然语言处理等领域提供了一种更稳定、更高效的序列建模选择。
原文 arXiv:1510.02693;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1510.02693v1