Do ImageNet Classifiers Generalize to ImageNet?
Benjamin Recht UC Berkeley Authors ordered alphabetically. Ben did none of the work. Rebecca Roelofs UC Berkeley Ludwig Schmidt UC Berkeley Vaishaal Shankar UC Berkeley
Abstract
We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% – 15% on CIFAR-10 and 11% – 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models’ inability to generalize to slightly “harder” images than those found in the original test sets.
中文速览
图像分类领域长期依赖CIFAR-10和ImageNet两个基准测试集衡量模型进步，但这些测试集被反复使用近十年，存在模型"过度适应"测试集的隐患。研究者严格复现了两个数据集的原始构建流程，重新采集并标注了全新的测试集，然后在数十个主流分类模型上对比新旧测试集的准确率。结果令人警醒：模型在新测试集上的准确率普遍下降3%–15%（CIFAR-10）和11%–14%（ImageNet），相当于抹去了ImageNet上约五年的研究进展。然而，在旧测试集上表现更好的模型，在新测试集上同样领先、甚至优势更大，说明准确率下滑并非源于模型对旧测试集的过拟合，而是因为新测试集中图像稍"难"一些——数据清洗流程的细微差异就能让当前最优模型的表现大幅波动，这也提醒我们应审慎看待那些宣称达到"人类水平"的结论。
原文 arXiv:1902.10811;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.10811v2