When Does Label Smoothing Help?
Rafael Müller, Simon Kornblith, Geoffrey Hinton Google Brain Toronto This work was done as part of the Google AI Residency.
Abstract
The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition. Despite its widespread use, label smoothing is still poorly understood. Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. To explain these observations, we visualize how label smoothing changes the representations learned by the penultimate layer of the network. We show that label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different
中文速览
标签平滑（label smoothing）是训练神经网络时广泛使用的一个技巧——把原本非0即1的硬标签替换成掺了少量均匀分布的软标签——但人们对它为何有效知之甚少。研究者通过可视化网络倒数第二层的表示发现，标签平滑会把同类样本的特征向量压缩成非常紧密的簇，使每个样本到各类别模板的距离趋于均等，从而揭示了其防止模型过度自信的几何原因。实验进一步证明，标签平滑能显著改善模型校准（calibration），让模型的置信度更准确地反映其真实正确率，在机器翻译任务中还因此提升了束搜索（beam search）的效果。然而，这种紧密聚类的代价是抹去了不同类别之间相似度的细粒度信息，导致以标签平滑训练的教师网络在知识蒸馏（knowledge distillation）时无法将有效的"暗知识"传递给学生网络，使蒸馏效果大打折扣。这一发现揭示了标签平滑的双面性，提示从业者在需要做知识蒸馏时应谨慎使用标签平滑训练教师模型。
原文 arXiv:1906.02629;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.02629v3