Decoupled Neural Interfaces using Synthetic Gradients
Max Jaderberg Wojciech Marian Czarnecki Simon Osindero Oriol Vinyals Alex Graves David Silver Koray Kavukcuoglu
Abstract
Training directed neural networks typically requires forward-propagating data through a computation graph, followed by backpropagating error signal, to produce weight updates. All layers, or more generally, modules, of the network are therefore locked, in the sense that they must wait for the remainder of the network to execute forwards and propagate error backwards before they can be updated. In this work we break this constraint by decoupling modules by introducing a model of the future computation of the network graph. These models predict what the result of the modelled subgraph will produce using only local information. In particular we focus on modelling error gradients: by using the modelled synthetic gradient in place of true backpropagated error gradients we decouple subgraphs, and can update them independently and asynchronously i.e. we realise decoupled neural interfaces. We show results for feed-forward models, where every layer is trained asynchronously, recurrent neural networks (RNNs) where predicting one’s future gradient extends the time over which the RNN can effectively model, and also a hierarchical RNN system with ticking at different timescales. Finally, we de
中文速览
神经网络训练时每一层都必须等前面的层算完正向传播、后面的层算完反向传播，才能更新自己的参数，这种"层层等待"的锁死机制严重制约了并行化和异步训练。本文提出"解耦神经接口"（Decoupled Neural Interfaces, DNI）：给每个模块配备一个小型"梯度预测器"，让它在真实梯度还没算出来之前就预测出合成梯度（synthetic gradient），从而让每层可以立刻更新，无需等待后续模块完成计算。实验表明，这一方法不仅能让前馈网络的每一层异步独立训练，还能帮助循环神经网络（RNN）突破截断反向传播（truncated BPTT）的时间限制、学到更长期的依赖关系，并显著加速多时间尺度层级RNN系统的训练。这项工作从根本上打破了深度学习训练的同步约束，为大规模分布式、异步神经网络系统的实现提供了可行路径。
原文 arXiv:1608.05343;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1608.05343v2