Self-training with Noisy Student improves ImageNet classification
Qizhe Xie 1, Minh-Thang Luong1, Eduard Hovy2, Quoc V. Le1 1Google Research, Brain Team, 2Carnegie Mellon University {qizhex, thangluong, This work was conducted at Google.
Abstract
We present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant. Noisy Student Training achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. On robustness test sets, it improves ImageNet-A top-1 accuracy from 61.0% to 83.7%, reduces ImageNet-C mean corruption error from 45.7 to 28.3, and reduces ImageNet-P mean flip rate from 27.8 to 12.2.
中文速览
用3亿张无标签图片让图像识别模型变得更准、更鲁棒，是这项工作的核心目标。研究者提出了"噪声学生训练"（Noisy Student Training）方法：先用有标签数据训练一个教师模型，再让它为海量无标签图片生成伪标签，然后训练一个更大的学生模型同时学习真实标签和伪标签——关键在于训练学生时故意加入数据增强（RandAugment）、随机丢弃（Dropout）和随机深度（Stochastic Depth）等噪声，迫使学生比教师学得更扎实，之后再把学生升级为新教师反复迭代。最终，基于EfficientNet架构的模型在ImageNet上达到88.4%的Top-1准确率，比此前需要35亿张弱标注Instagram图片才能达到的最优结果还高2%，同时在专门考验模型鲁棒性的ImageNet-A、ImageNet-C、ImageNet-P等测试集上也大幅领先。这项工作的意义在于证明：利用廉价的无标签数据配合"让学生在噪声中磨练"的训练策略，不仅能刷新准确率纪录，还能显著提升模型面对真实世界图像扰动时的泛化能力。
原文 arXiv:1911.04252;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.04252v4