The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Dan Hendrycks1 Steven Basart2 Equal contribution. 1UC Berkeley, 2UChicago, 3Google. Code is available at https://github.com/hendrycks/imagenet-r. Norman Mu Saurav Kadavath1 Frank Wang3 Evan Dorundo3 Rahul Desai1 Tyler Zhu1 Samyak Parajuli1 Mike Guo1 Dawn Song1 Jacob Steinhardt1 Justin Gilmer3
Abstract
We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with $1000\times$ more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated metho
中文速览
真实世界里的图像识别模型常常在遇到"没见过的数据分布"时性能大幅下滑，但学界对哪些方法真正有效一直众说纷纭。研究者为此新建了四个涵盖风格渲染、图像模糊、地理位置变化、相机参数差异等真实场景的分布偏移数据集（包括 ImageNet-R、SVSF、DFR 和 Real Blurry Images），并在这些数据集上系统评测了更大模型、自注意力机制、多样化数据增强、大规模预训练四类主流鲁棒性方法。同时提出了一种新的数据增强方法 DeepAugment——通过在图像到图像网络的前向传播中随机扰动权重和中间特征来生成多样化的训练样本，仅用这一方法就超越了使用多一千倍标注数据进行预训练的模型。结果显示，数据增强和更大模型确实能提升纹理和局部统计特征相关的分布偏移鲁棒性，合成基准上的改进也能迁移到真实世界场景，但没有任何一种方法能在所有分布偏移类型上持续奏效，说明鲁棒性研究必须同时考察多种真实分布偏移，而非依赖单一基准得出笼统结论。
原文 arXiv:2006.16241;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.16241v3