Self-Normalizing Neural Networks
Günter Klambauer、Thomas Unterthiner、Andreas Mayr、Sepp Hochreiter LIT AI Lab、Institute of Bioinformatics, Johannes Kepler University Linz A-4040 Linz, Austria
Abstract
Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are “scaled exponential linear units” (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance — even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization schemes, and (3) to make learning highly robust. Furthermore, for activations not close to u
中文速览
普通前馈神经网络（FNN）一直难以像卷积网络或循环网络那样堆叠很多层、学到高层抽象特征，根本原因在于批归一化等外部归一化手段在随机梯度下降和dropout的扰动下训练方差极大、容易不稳定。本文提出"自归一化神经网络"（Self-Normalizing Neural Networks，SNN），其核心是一种名为SELU（Scaled Exponential Linear Unit）的新激活函数——通过数学推导确定其两个参数值，使得每一层的神经元激活均值和方差在层间传播时自动收敛到零均值、单位方差，无需任何外部归一化模块，并用Banach不动点定理严格证明了这一收敛性，同时排除了梯度爆炸和消失的可能。在UCI 121个分类/回归任务、药物毒性预测（Tox21）以及天文数据集上，SNN显著优于ReLU网络、批归一化、残差网络、随机森林、支持向量机等多种基线方法，且胜出的SNN架构往往层数很深。这项工作填补了深度学习在表格数据和通用机器学习任务上长期缺席的空白，为FNN真正走向"深度"提供了理论保障和实用工具。
原文 arXiv:1706.02515;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.02515v5