Scaling Vision Transformers
Xiaohua Zhai⋆, Alexander Kolesnikov⋆, Neil Houlsby, Lucas Beyer⋆ Google Research, Brain Team, Zürich {xzhai, akolesnikov, neilhoulsby,
Abstract
Attention-based neural networks such as the Vision Transformer (ViT) have recently attained state-of-the-art results on many computer vision benchmarks. Scale is a primary ingredient in attaining excellent results, therefore, understanding a model’s scaling properties is a key to designing future generations effectively. While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale. To address this, we scale ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute. Along the way, we refine the architecture and training of ViT, reducing memory consumption and increasing accuracy of the resulting models. As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of $90.45\%$ top-1 accuracy. The model also performs well for few-shot transfer, for example, reaching $84.86\%$ top-1 accuracy on ImageNet with only 10 examples per class.
中文速览
视觉Transformer（Vision Transformer，ViT）在图像识别上表现出色，但研究者此前并不清楚它在规模扩大时性能究竟如何变化——而这正是设计下一代模型的关键。为此，本文系统地对ViT的模型参数量（500万到20亿）、训练数据量（100万到30亿图片）以及计算量进行了大范围缩放实验，并总结出误差率与计算资源之间遵循一种"双饱和幂律"关系：中间区间符合幂律，两端各有一个饱和上限。研究同时发现，大模型比小模型更"样本高效"——用更少的图片就能达到同等表现，且持续增加数据对大模型仍有帮助。在方法层面，作者通过对最终预测层单独施加强L2正则化、用多头注意力池化替换[class]token等改进，大幅降低了显存占用并提升了少样本迁移能力，最终成功训练出拥有20亿参数的ViT-G/14模型，在ImageNet上取得90.45%的顶级精度，并在每类仅10张样本的少样本设置下达到84.86%，刷新了多项基准记录。这项工作为未来大规模视觉模型的设计提供了可量化的缩放规律，意义类似于NLP领域的Chinchilla缩放定律。
原文 arXiv:2106.04560;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.04560v2