Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding
Zizhao Zhang1 Han Zhang2 Long Zhao2 Ting Chen2 Sercan Ö. Arık1 Tomas Pfister1
Abstract
Hierarchical structures are popular in recent vision transformers, however, they require sophisticated designs and massive datasets to work well. In this paper, we explore the idea of nesting basic local transformers on non-overlapping image blocks and aggregating them in a hierarchical way. We find that the block aggregation function plays a critical role in enabling cross-block non-local information communication. This observation leads us to design a simplified architecture that requires minor code changes upon the original vision transformer. The benefits of the proposed judiciously-selected design are threefold: (1) NesT converges faster and requires much less training data to achieve good generalization on both ImageNet and small datasets like CIFAR; (2) when extending our key ideas to image generation, NesT leads to a strong decoder that is 8 $\times$ faster than previous transformer-based generators; and (3) we show that decoupling the feature learning and abstraction processes via this nested hierarchy in our design enables constructing a novel method (named GradCAT) for visually interpreting the learned model. Source code is available https://github.com/google-research/ne
中文速览
视觉变换器（Vision Transformer, ViT）在大规模数据集上表现出色，但因缺乏局部性归纳偏置，在小数据集上训练效率低下且泛化差。NesT通过把标准Transformer块限定在不重叠的图像局部区域独立运行，再借助一个关键的"块聚合（block aggregation）"函数将多个局部块逐层合并成嵌套层级结构，从而以极简的代码改动实现了跨块的全局信息交流。实验结果表明，NesT在ImageNet上用更少的训练轮次就达到了83%以上的准确率，在CIFAR等小数据集上也能媲美卷积网络，同时将这一思路"转置"后还能构建出比以往基于Transformer的图像生成器快8倍的解码器，并衍生出一种树状梯度可视化方法GradCAT，直观解释模型的决策过程。这项工作的价值在于，它用最简单的架构变动同时改善了数据效率、生成速度和模型可解释性，为层级式视觉Transformer的设计提供了新思路。
原文 arXiv:2105.12723;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.12723v4