arXiv:1503.01007 · 中英对照阅读
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets
中文速览
普通循环神经网络很难学会需要持续计数或记住很长序列的规律,因此研究者想让它不只靠有限的隐藏状态做预测。为此,他们给循环网络接上可训练的外部记忆,例如像牌堆一样后进先出的栈,或能左右移动、插入内容的列表,并让网络从数据中学会何时读写这些记忆。实验显示,这类模型能学会一些普通循环网络难以掌握的算法生成序列,也能用于语言建模。它说明,给神经网络配上可扩展的结构化记忆,可能帮助机器学会更复杂、需要长程记忆和计数的规律。
摘要
Despite the recent achievements in machine learning, we are still very far from achieving real artificial intelligence. In this paper, we discuss the limitations of standard deep learning approaches and show that some of these limitations can be overcome by learning how to grow the complexity of a model in a structured way. Specifically, we study the simplest sequence prediction problems that are beyond the scope of what is learnable with standard recurrent networks, algorithmically generated sequences which can only be learned by models which have the capacity to count and to memorize sequences. We show that some basic algorithms can be learned from sequential data using a recurrent network associated with a trainable memory.
术语表
- artificial intelligence (AI)
- 人工智能(AI)
- deep learning
- 深度学习
- recurrent network
- 循环网络
- sequence prediction
- 序列预测
- algorithmically generated sequences
- 算法生成序列
- trainable memory
- 可训练记忆
- neural network
- 神经网络
- computer vision
- 计算机视觉
- speech recognition
- 语音识别
- XOR problem
- 异或问题(XOR)
- linear classifier
- 线性分类器
- non-linear hidden layer
- 非线性隐藏层
- kernel method
- 核方法
- parity problem
- 奇偶性问题
- pushdown stack
- 下推栈
- structured memory
- 结构化记忆
- multiplicative gating mechanism
- 乘法门控机制
- PUSH
- 入栈(PUSH)
- POP
- 出栈(POP)
- Neural Turing Machine
- 神经图灵机(Neural Turing Machine)
- Memory Networks
- 记忆网络(Memory Networks)
- algorithmic patterns
- 算法模式
- context-free grammar
- 上下文无关文法
- context-sensitive grammar
- 上下文有关文法
- simple recurrent network
- 简单循环网络
- Long Short-Term Memory (LSTM)
- 长短期记忆网络(LSTM)
- vanishing and exploding gradient problems
- 梯度消失与梯度爆炸问题
- Gated Recurrent Unit (GRU)
- 门控循环单元(GRU)
- attention mechanism
- 注意力机制
- external memory module
- 外部记忆模块