Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
Mengye Ren∗†, Renjie Liao∗†, Raquel Urtasun†, Fabian H. Sinz‡, Richard S. Zemel†⋇ †University of Toronto, Toronto ON, CANADA ‡Baylor College of Medicine, Houston TX, USA ⋇Canadian Institute for Advanced Research (CIFAR) {mren, rjliao, indicates equal contribution
Abstract
Normalization techniques have only recently begun to be exploited in supervised learning tasks. Batch normalization exploits mini-batch statistics to normalize the activations. This was shown to speed up training and result in better models. However its success has been very limited when dealing with recurrent neural networks. On the other hand, layer normalization normalizes the activations across all activities within a layer. This was shown to work well in the recurrent setting. In this paper we propose a unified view of normalization techniques, as forms of divisive normalization, which includes layer and batch normalization as special cases. Our second contribution is the finding that a small modification to these normalization schemes, in conjunction with a sparse regularizer on the activations, leads to significant benefits over standard normalization techniques. We demonstrate the effectiveness of our unified divisive normalization framework in the context of convolutional neural nets and recurrent neural networks, showing improvements over baselines in image classification, language modeling as well as super-resolution.
中文速览
神经网络训练困难的一个核心原因是各层激活值的分布不稳定，批归一化（Batch Normalization）和层归一化（Layer Normalization）虽然能缓解这一问题，但前者在小批量和循环神经网络中表现欠佳，后者用于卷积网络时又容易掉点。本文提出将这两种方法统一纳入"除法归一化"（Divisive Normalization，DN）框架，揭示它们本质上都是在张量的不同维度上做同一种归一化操作，只是"邻域"范围各异。在此基础上，作者引入两个小改动：在归一化分母中加入平滑项 σ，并对激活值施加 L1 稀疏正则，从而促使特征表示更加解耦。实验涵盖图像分类、语言建模和超分辨率三类任务，结果显示改进后的 DN 框架在卷积网络上与批归一化持平，在循环网络和超分辨率任务上明显优于现有方法，且无需存储批统计量。这项工作不仅为理解各类归一化方法提供了统一视角，也为实际应用中选择和改进归一化策略提供了可操作的指导。
原文 arXiv:1611.04520;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.04520v2