Deep Information Propagation
Samuel S. Schoenholz Google Brain、Justin Gilmer∗ Google Brain、Surya Ganguli Stanford University、Jascha Sohl-Dickstein Google Brain Work done as a member of the Google Brain Residency program (g.co/brainresidency)
Abstract
We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be trained precisely when information can travel through them. Thus, the depth scales that we identify provide bounds on how deep a network may be trained for a specific choice of hyperparameters. As a corollary to this, we argue that in networks at the edge of chaos, one of these depth scales diverges. Thus arbitrarily deep networks may be trained only sufficiently close to criticality. We show that the presence of dropout destroys the order-to-chaos critical point and therefore strongly limits the maximum trainable depth for random networks. Finally, we develop a mean field theory for backpropagation and we show that the ordered and chaotic phases correspond to regions of vanishing and exploding gradient respectively.
中文速览
随机初始化神经网络中信号能传播多深，本质上决定了这个网络能被训练多深——这正是本文要回答的核心问题。作者用平均场理论（mean field theory）分析了权重和偏置随机分布的深度网络，发现存在两个天然的"深度尺度"（depth scale），分别控制单个输入幅度信息和两个输入相关性信息的传播距离，其中描述相关性传播的深度尺度 ξ_c 在有序相与混沌相的临界边界（"混沌边缘"）处会发散到无穷大。他们进一步将该框架延伸到梯度反向传播，严格证明有序相对应梯度消失、混沌相对应梯度爆炸，从理论上首次为这两个长期困扰深度学习的现象给出了统一解释。在 MNIST 和 CIFAR10 上的训练实验证实：网络恰好在深度不超过 ξ_c 时才可以被成功训练，而一旦引入 dropout，临界点被彻底破坏、ξ_c 不再发散，因此 dropout 会严格限制可训练的最大深度。这项工作的重要性在于，它将"网络能训多深"这一经验性难题转化为可计算的理论预测，为超参数选择和网络架构设计提供了原则性依据。
原文 arXiv:1611.01232;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.01232v2