Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
Dan Hendrycks University of California, Berkeley、Thomas Dietterich Oregon State University
Abstract
In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier’s robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.
中文速览
现有图像分类器在面对雪、模糊、噪声等日常图像失真时表现脆弱，但学界缺乏统一、严格的评测基准来衡量这类"常见干扰"下的鲁棒性。研究者构建了两个基准数据集：ImageNet-C，将75种常见图像损坏（噪声、模糊、天气、数字压缩等）以五个强度等级施加于ImageNet验证集，用于测试模型对损坏图像的准确率；ImageNet-P，则为每张验证图像生成连续微小扰动序列，用于衡量模型预测的稳定性。评测结果显示，从AlexNet到ResNet，各代模型的相对鲁棒性几乎没有实质性进步，说明架构迭代并未系统性地解决这一问题；而某些新型架构和训练策略（包括一种对抗训练防御方法）能够显著提升两类鲁棒性。这项工作填补了对抗样本研究与真实世界部署需求之间的空白，为开发能够可靠泛化、不依赖无关统计线索的视觉模型提供了标准化的评测工具。
原文 arXiv:1903.12261;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1903.12261v1