Gaussian Error Linear Units (GELUs)
Dan Hendrycks University of California, Berkeley、Kevin Gimpel Toyota Technological Institute at Chicago Work done while the author was at TTIC. Code available at github.com/hendrycks/GELUs
Abstract
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$ , where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ( $x\mathbf{1}_{x>0}$ ). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks.
中文速览
神经网络的激活函数长期被ReLU统治，但ReLU只根据输入的正负号做硬性门控，缺乏概率意义上的平滑性。作者提出了高斯误差线性单元（Gaussian Error Linear Unit，GELU），其形式为 x 乘以标准正态分布的累积分布函数 Φ(x)，本质上是把"随机丢弃"这类正则化思想融入激活函数本身——输入值越大被保留的概率越高，从而以连续平滑的方式对输入加权，而非粗暴地置零。在计算机视觉、自然语言处理和语音识别等多项任务上，GELU在MNIST、CIFAR-10/100、TIMIT、Twitter词性标注等基准上全面超越ReLU和ELU，且无需引入额外超参数。这一激活函数随后被BERT、GPT等大规模预训练模型广泛采用，成为现代深度学习架构的标配组件，其影响力远超论文发表时的预期。
原文 arXiv:1606.08415;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.08415v5