Aha.
正在载入中英对照阅读…

arXiv:1503.01007 · 中英对照阅读

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

Armand Joulin、Tomas Mikolov

中文速览

普通循环神经网络很难学会需要持续计数或记住很长序列的规律,因此研究者想让它不只靠有限的隐藏状态做预测。为此,他们给循环网络接上可训练的外部记忆,例如像牌堆一样后进先出的栈,或能左右移动、插入内容的列表,并让网络从数据中学会何时读写这些记忆。实验显示,这类模型能学会一些普通循环网络难以掌握的算法生成序列,也能用于语言建模。它说明,给神经网络配上可扩展的结构化记忆,可能帮助机器学会更复杂、需要长程记忆和计数的规律。

摘要

Despite the recent achievements in machine learning, we are still very far from achieving real artificial intelligence. In this paper, we discuss the limitations of standard deep learning approaches and show that some of these limitations can be overcome by learning how to grow the complexity of a model in a structured way. Specifically, we study the simplest sequence prediction problems that are beyond the scope of what is learnable with standard recurrent networks, algorithmically generated sequences which can only be learned by models which have the capacity to count and to memorize sequences. We show that some basic algorithms can be learned from sequential data using a recurrent network associated with a trainable memory.

术语表

artificial intelligence (AI)
人工智能(AI)
deep learning
深度学习
recurrent network
循环网络
sequence prediction
序列预测
algorithmically generated sequences
算法生成序列
trainable memory
可训练记忆
neural network
神经网络
computer vision
计算机视觉
speech recognition
语音识别
XOR problem
异或问题(XOR)
linear classifier
线性分类器
non-linear hidden layer
非线性隐藏层
kernel method
核方法
parity problem
奇偶性问题
pushdown stack
下推栈
structured memory
结构化记忆
multiplicative gating mechanism
乘法门控机制
PUSH
入栈(PUSH)
POP
出栈(POP)
Neural Turing Machine
神经图灵机(Neural Turing Machine)
Memory Networks
记忆网络(Memory Networks)
algorithmic patterns
算法模式
context-free grammar
上下文无关文法
context-sensitive grammar
上下文有关文法
simple recurrent network
简单循环网络
Long Short-Term Memory (LSTM)
长短期记忆网络(LSTM)
vanishing and exploding gradient problems
梯度消失与梯度爆炸问题
Gated Recurrent Unit (GRU)
门控循环单元(GRU)
attention mechanism
注意力机制
external memory module
外部记忆模块