Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels
Pengfei Chen Benben Liao Guangyong Chen Shengyu Zhang
Abstract
Noisy labels are ubiquitous in real-world datasets, which poses a challenge for robustly training deep neural networks (DNNs) as DNNs usually have the high capacity to memorize the noisy labels. In this paper, we find that the test accuracy can be quantitatively characterized in terms of the noise ratio in datasets. In particular, the test accuracy is a quadratic function of the noise ratio in the case of symmetric noise, which explains the experimental findings previously published. Based on our analysis, we apply cross-validation to randomly split noisy datasets, which identifies most samples that have correct labels. Then we adopt the Co-teaching strategy which takes full advantage of the identified samples to train DNNs robustly against noisy labels. Compared with extensive state-of-the-art methods, our strategy consistently improves the generalization performance of DNNs under both synthetic and real-world training noise.
中文速览
真实世界的训练数据普遍存在标签噪声（noisy labels），深度神经网络（DNN）因记忆能力强而容易过拟合这些错误标签，导致泛化性能下降。这篇论文首先从理论上推导出：在对称噪声（symmetric noise）条件下，DNN的测试准确率是噪声比例的二次函数，从而定量解释了前人在实验中观察到但未能说清的现象。在此理论基础上，作者提出了迭代噪声交叉验证（Iterative Noisy Cross-Validation, INCV）方法——通过交叉验证将含噪数据集反复拆分，筛选出大概率具有正确标签的样本子集，大幅降低子集的噪声比例，同时自动估计噪声率，再结合联合训练两个网络互相监督的Co-teaching策略来鲁棒地训练模型。在CIFAR-10合成噪声数据集和含真实噪声的WebVision大规模图像数据集上，该方法均显著优于现有主流方法，对需要用低成本、带噪标注数据训练可靠模型的实际应用具有重要意义。
原文 arXiv:1905.05040;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.05040v1