Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks
Luo Chunjie Zhan Jianfeng Wang Lei Yang Qiang
Abstract
Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks as well as convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100 and SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.
中文速览
传统神经网络用点积（dot product）计算神经元的预激活值，但点积结果无界，容易造成神经元方差过大，导致模型泛化能力差、训练不稳定。为此，研究者提出用余弦相似度（cosine similarity）乃至皮尔逊相关系数（Pearson Correlation Coefficient）来替代点积，将预激活值严格限制在 -1 到 1 之间，即"余弦归一化（cosine normalization）"。在 MNIST、20NEWS GROUP、CIFAR-10/100 和 SVHN 等多个数据集上，与批归一化、权重归一化、层归一化的对比实验表明，余弦归一化的测试误差更低、训练更稳定、收敛速度更快，且不依赖批统计量，既可用批梯度下降也可用随机梯度下降训练。这项工作为深度网络的归一化设计提供了一个简洁而有效的新思路，对提升模型鲁棒性和训练效率具有实用价值。
原文 arXiv:1702.05870;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1702.05870v5