Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement
Fartash Faghri , Hadi Pouransari, Sachin Mehta, Mehrdad Farajtabar, Ali Farhadi, Mohammad Rastegari, Oncel Tuzel Apple Correspondence to
Abstract
We propose Dataset Reinforcement, a strategy to improve a dataset once such that the accuracy of any model architecture trained on the reinforced dataset is improved at no additional training cost for users. We propose a Dataset Reinforcement strategy based on data augmentation and knowledge distillation. Our generic strategy is designed based on extensive analysis across CNN- and transformer-based models and performing large-scale study of distillation with state-of-the-art models with various data augmentations. We create a reinforced version of the ImageNet training dataset, called ImageNet+, as well as reinforced datasets CIFAR-100+, Flowers-102+, and Food-101+. Models trained with ImageNet+ are more accurate, robust, and calibrated, and transfer well to downstream tasks (e.g., segmentation and detection). As an example, the accuracy of ResNet-50 improves by 1.7% on the ImageNet validation set, 3.5% on ImageNetV2, and 10.0% on ImageNet-R. Expected Calibration Error (ECE) on the ImageNet validation set is also reduced by 9.9%. Using this backbone with Mask-RCNN for object detection on MS-COCO, the mean average precision improves by 0.8%. We reach similar gains for MobileNets, Vi
中文速览
大规模模型的知识其实可以"一次性"烧录进训练数据集，从而让任何人训练任何模型时都能免费获益——这正是本文提出的「数据集强化」（Dataset Reinforcement）的核心思路。具体做法是：用一个由多个顶尖预训练模型组成的集成教师，对原始训练集中每张图片在多种数据增强下预先计算软标签，将这些软标签连同增强参数一起存入磁盘，形成强化版数据集（如 ImageNet+），用户只需换一个数据加载器即可使用，无需改动模型或增加训练计算量。大规模实验表明，用 ImageNet+ 训练 ResNet-50 的 ImageNet 准确率提升 1.7%，在分布外测试集 ImageNet-R 上提升高达 10%，预期校准误差（ECE）下降 9.9%，迁移到目标检测和语义分割任务也有稳定增益；MobileNet、ViT、Swin-Transformer 等架构同样受益，在鲁棒性数据集上最高提升 20%。这项工作的价值在于它把昂贵的知识蒸馏成本从"每次训练都要付"变成"只需付一次"，让小模型、小数据场景也能真正用上大模型的知识红利。
原文 arXiv:2303.08983;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.08983v3