Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Li Yuan1, Yunpeng Chen2, Tao Wang1,3∗, Weihao Yu1, Yujun Shi1, Zihang Jiang1, Francis E.H. Tay1, Jiashi Feng1, Shuicheng Yan1 1 National University of Singapore 2 YITU Technology 3 Institute of Data Science, National University of Singapore Work done during an internship at Yitu Tech.
Abstract
Transformers, which are popular for language modeling, have been explored for solving vision tasks recently, e.g., the Vision Transformer (ViT) for image classification. The ViT model splits each image into a sequence of tokens with fixed length and then applies multiple Transformer layers to model their global relation for classification. However, ViT achieves inferior performance to CNNs when trained from scratch on a midsize dataset like ImageNet. We find it is because: 1) the simple tokenization of input images fails to model the important local structure such as edges and lines among neighboring pixels, leading to low training sample efficiency; 2) the redundant attention backbone design of ViT leads to limited feature richness for fixed computation budgets and limited training samples. To overcome such limitations, we propose a new Tokens-To-Token Vision Transformer (T2T-ViT), which incorporates 1) a layer-wise Tokens-to-Token (T2T) transformation to progressively structurize the image to tokens by recursively aggregating neighboring Tokens into one Token (Tokens-to-Token), such that local structure represented by surrounding tokens can be modeled and tokens length can be red
中文速览
视觉Transformer（ViT）直接把图像切成固定大小的图块再送入Transformer，导致无法捕捉边缘、纹理等局部结构，在ImageNet这样的中等规模数据集上从头训练时表现明显弱于CNN；此外ViT的主干网络存在大量冗余通道，特征丰富度不足。为解决这两个问题，作者提出了T2T-ViT：先用"逐层Token聚合"模块（T2T module）将相邻Token反复合并，把局部结构信息逐步编码进Token序列并缩短序列长度；再借鉴CNN的"深窄结构"设计，用更少的通道数搭配更多的层数来降低冗余、提升特征多样性。实验结果表明，T2T-ViT参数量和计算量约为原始ViT的一半，在ImageNet上从头训练的Top-1准确率却提升了3%以上，同时超越同等规模的ResNet，并与MobileNet相当——这是首次纯Transformer视觉模型无需在超大规模数据集上预训练就能全面媲美甚至超越CNN，对轻量化视觉Transformer的设计具有重要参考价值。
原文 arXiv:2101.11986;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2101.11986v3