Augmented Shortcuts for Vision Transformers
Yehui Tang1,2, Kai Han2, Chang Xu3, An Xiao2, Yiping Deng4, Chao Xu1, Yunhe Wang2 1Key Lab of Machine Perception (MOE), Dept. of Machine Intelligence, Peking University. 2Noah’s Ark Lab, Huawei Technologies. 4Central Software Institution, Huawei Technologies. 3School of Computer Science, Faculty of Engineering, University of Sydney. {kai.han, yunhe.wang, an.xiao,
Abstract
Transformer models have achieved great progress on computer vision tasks recently. The rapid development of vision transformers is mainly contributed by their high representation ability for extracting informative features from input images. However, the mainstream transformer models are designed with deep architectures, and the feature diversity will be continuously reduced as the depth increases, i.e., feature collapse. In this paper, we theoretically analyze the feature collapse phenomenon and study the relationship between shortcuts and feature diversity in these transformer models. Then, we present an augmented shortcut scheme, which inserts additional paths with learnable parameters in parallel on the original shortcuts. To save the computational costs, we further explore an efficient approach that uses the block-circulant projection to implement augmented shortcuts. Extensive experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method, which brings about 1% accuracy increase of the state-of-the-art visual transformers without obviously increasing their parameters and FLOPs.
中文速览
视觉Transformer（Vision Transformer）在越叠越深的网络结构中会出现"特征坍塌"问题——不同图像块的特征越来越相似、失去区分度，导致模型表达能力下降。为此，研究者从理论上证明了这一现象的根源在于注意力机制的逐层特征聚合，同时揭示了原有"恒等捷径连接（shortcut）"虽能缓解但仍不足以充分保持特征多样性。他们提出"增强捷径（Augmented Shortcut）"方案：在原有捷径旁并联多条带可学习参数的变换路径，并进一步用块循环矩阵（block-circulant matrix）替代稠密矩阵以大幅降低计算量。在ImageNet等基准上的实验表明，将该方案嵌入ViT、PVT、T2T等主流视觉Transformer后，Top-1准确率平均提升约1%，而额外引入的参数量和计算量可忽略不计。这项工作为改善深层Transformer的特征多样性提供了轻量且有理论保障的通用解法，对后续Transformer架构设计具有重要参考价值。
原文 arXiv:2106.15941;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.15941v1