FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks
Suo Qiu, Xiangmin Xu and Bolun Cai School of Electronic and Information Engineering, South China University of Technology Wushan RD., Tianhe District, Guangzhou, P.R.China Email:
Abstract
Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. However, because of the zero-hard rectification, ReLU networks miss the benefits from negative values. In this paper, we propose a novel activation function called flexible rectified linear unit (FReLU) to further explore the effects of negative values. By redesigning the rectified point of ReLU as a learnable parameter, FReLU expands the states of the activation output. When the network is successfully trained, FReLU tends to converge to a negative value, which improves the expressiveness and thus the performance. Furthermore, FReLU is designed to be simple and effective without exponential functions to maintain low cost computation. For being able to easily used in various network architectures, FReLU does not rely on strict assumptions by self-adaption. We evaluate FReLU on three standard image classification datasets, including CIFAR-10, CIFAR-100, and ImageNet. Experimental results show that the proposed method achieves fast convergence and higher performances on both plain and residual networks.
中文速览
ReLU激活函数因为把所有负值强制归零，导致网络损失了负值所携带的有用信息。为此，研究者提出了一种名为"灵活修正线性单元"（Flexible Rectified Linear Unit，FReLU）的新激活函数，其核心思路是把ReLU的"截断点"从固定的零改为一个可学习的参数$b$，让网络在训练过程中自适应地决定是否利用负值区间。当$b$收敛到负数时，激活函数的输出状态从$2^n$种扩展到$3^n$种，显著提升了网络表达能力，同时整个计算过程无需指数运算，开销低，还与批归一化（Batch Normalization）保持良好兼容性。在CIFAR-10、CIFAR-100和ImageNet三个标准图像分类基准上，FReLU在普通网络和残差网络中均实现了更快的收敛速度和更高的分类精度。这项工作的价值在于：仅凭一个极简的改动——把截断点变成可学习参数——就在几乎不增加计算成本的前提下系统性地弥补了ReLU丢失负值信息的固有缺陷，为激活函数设计提供了一条实用且易于推广的新思路。
原文 arXiv:1706.08098;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.08098v2