The Effects of Regularization and Data Augmentation are Class Dependent
Randall Balestriero1 1Meta AI Research, 2NYU Léon Bottou1 1Meta AI Research, 2NYU Yann LeCun1,2 1Meta AI Research, 2NYU
Abstract
Regularization is a fundamental technique to prevent over-fitting and to improve generalization performances by constraining a model’s complexity. Current Deep Networks heavily rely on regularizers such as Data-Augmentation (DA) or weight-decay, and employ structural risk minimization, i.e. cross-validation, to select the optimal regularization hyper-parameters. In this study, we demonstrate that techniques such as DA or weight decay produce a model with a reduced complexity that is unfair across classes. The optimal amount of DA or weight decay found from cross-validation leads to disastrous model performances on some classes e.g. on Imagenet with a resnet50, the “barn spider” classification test accuracy falls from $68\%$ to $46\%$ only by introducing random crop DA during training. Even more surprising, such performance drop also appears when introducing uninformative regularization techniques such as weight decay. Those results demonstrate that our search for ever increasing generalization performance -averaged over all classes and samples- has left us with models and regularizers that silently sacrifice performances on some classes. This scenario can become dangerous when depl
中文速览
追求整体平均准确率提升的正则化手段，其实在悄悄"牺牲"某些类别的识别性能——这篇论文专门揭示了这一问题。研究者系统分析了数据增强（Data Augmentation, DA）和权重衰减（weight decay）两种主流正则化技术，发现它们虽然能提升模型在所有类别上的平均测试准确率，却会对特定类别造成显著的性能下滑：例如在 ImageNet 上用 ResNet50 训练时，仅引入随机裁剪数据增强，"仓蛛"类的测试准确率就从 68% 骤降至 46%，而且就连不依赖任何语义知识的权重衰减也会产生类似的类别偏差。更危险的是，这种偏差还会随迁移学习传导到下游任务，比如在 INaturalist 数据集上，某个类别的准确率因上游预训练阶段引入数据增强而从 70% 跌至 30%。这项工作提醒我们：用交叉验证选出的"最优"正则化超参数，只是在平均意义上最优，却可能对特定类别极不公平，如何设计无类别偏差的正则化方法仍是一个亟待解决的开放性问题。
原文 arXiv:2204.03632;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.03632v2