How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
Andreas Steiner∗ Alexander Kolesnikov∗ Xiaohua Zhai∗ Ross Wightman† Jakob Uszkoreit Lucas Beyer∗ Google Research, Brain Team, Zürich ∗ Equal technical contribution, † independent researcher
Abstract
Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer’s weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation (“AugReg” for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget.111We release more than 50 000 ViT models trained under diverse settings on various datasets. We believe this to be a treasure trove for model analysis. Available at https://github.com/google-research/vision_transformer and https://github.com/rwightman/pytorch-image-models. The code for full reproduction of model training is available at https://github.com/google-research/big_vision. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data:
中文速览
视觉变换器（Vision Transformer, ViT）在图像分类等任务上表现出色，但因缺乏卷积网络内置的归纳偏置，在小数据集上容易过拟合，通常需要更多数据或更强的数据增强与正则化（AugReg）才能训练好——然而这两种补救手段之间的权衡关系此前从未被系统研究过。为此，研究者在公开的 ImageNet-1k 和 ImageNet-21k 数据集上，系统地训练了超过五万个不同规模的 ViT 模型，全面对比了训练数据量、AugReg 强度、模型大小和计算量这四个因素的相互影响。核心发现是：精心设计的 AugReg 方案在效果上大约等价于将训练数据量扩大十倍，而在 ImageNet-21k 上配合更多计算量训练的 ViT，其性能可以达到甚至超过在规模大得多的私有数据集 JFT-300M 上训练的同类模型。这项研究的意义在于，它让普通研究者和开发者仅凭公开数据和合理计算预算，就能复现此前只有拥有海量私有数据才能达到的模型性能，极大降低了高性能 ViT 的使用门槛。
原文 arXiv:2106.10270;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.10270v2