Transformer in Transformer
Kai Han1,2 An Xiao2 Enhua Wu1,3 Jianyuan Guo2 Chunjing Xu2 Yunhe Wang2∗ 1State Key Lab of Computer Science, ISCAS、UCAS 2Noah’s Ark Lab, Huawei Technologies 3University of Macau Corresponding author.
Abstract
Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (e.g., 16 $\times$ 16) as “visual sentences” and present to further divide them into smaller patches (e.g., 4 $\times$ 4) as “visual words”. The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of th
中文速览
针对视觉Transformer（Visual Transformer）在处理图像时"一刀切"地把图像分成固定大块、忽略块内部细节结构的问题，研究者提出了一种嵌套式架构——Transformer中套Transformer（TNT，Transformer iN Transformer）：先把图像切成较大的"视觉句子"，再把每个句子进一步细分为更小的"视觉词"，然后在词级别和句子级别分别运行注意力机制，最终将词特征融合进句子表示，从而同时捕获局部细节和全局关系。由于词级别的内层Transformer参数是跨句子共享的，额外引入的计算量（约多14%的FLOPs）和参数量（约多8%）都极为有限。在ImageNet图像分类基准上，TNT以相近的计算代价比当时最先进的视觉Transformer高出约1.7个百分点（81.5% top-1准确率），在下游迁移学习任务上也表现出色，为如何更精细地挖掘图像的多尺度局部结构提供了一条高效可行的路径。
原文 arXiv:2103.00112;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.00112v3