Regularizing Neural Networks by Penalizing Confident Output Distributions
Gabriel Pereyra Google Brain、George Tucker Google Brain、Jan Chorowski Google Brain、Łukasz Kaiser Google Brain、Geoffrey Hinton University of Toronto、Google Brain Work done as part of the Google Brain Residency ProgramEqual Contribution
Abstract
We systematically explore regularizing neural networks by penalizing low entropy output distributions. We show that penalizing low entropy output distributions, which has been shown to improve exploration in reinforcement learning, acts as a strong regularizer in supervised learning. Furthermore, we connect a maximum entropy based confidence penalty to label smoothing through the direction of the KL divergence. We exhaustively evaluate the proposed confidence penalty and label smoothing on 6 common benchmarks: image classification (MNIST and Cifar-10), language modeling (Penn Treebank), machine translation (WMT’14 English-to-German), and speech recognition (TIMIT and WSJ). We find that both label smoothing and the confidence penalty improve state-of-the-art models across benchmarks without modifying existing hyperparameters, suggesting the wide applicability of these regularizers.
中文速览
神经网络在分类任务中往往过度自信——把几乎所有概率都压在一个类别上，这是过拟合的典型症状。针对这一问题，研究者系统地探索了两种"输出分布正则化"方法：置信度惩罚（confidence penalty，即在训练目标中加入负熵项，阻止模型输出过于尖锐的概率分布）和标签平滑（label smoothing，把硬标签按一定比例替换为均匀分布），并从KL散度方向的角度揭示了二者之间的理论联系。研究者在图像分类、语言模型、机器翻译和语音识别共6个基准任务上对这两种方法进行了全面评测，结果表明，无需调整现有超参数，两种方法均能稳定提升当时最优模型的性能。这项工作的意义在于：它提供了一类简单、通用、几乎"即插即用"的正则化思路，直接作用于模型的输出分布，而非内部权重，理论上对任何深度神经网络架构都适用。
原文 arXiv:1701.06548;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1701.06548v1