Efficient Self-supervised Vision Transformers for Representation Learning
Chunyuan Li1 Jianwei Yang1 Pengchuan Zhang1 Mei Gao2 Bin Xiao2 Xiyang Dai2 Lu Yuan2 Jianfeng Gao1 1Microsoft Research at Redmond, 2Microsoft Cloud + AI
Abstract
This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose a new pre-training task of region matching which allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 accuracy on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring to downstream linear classification tasks, EsViT outperforms its supervised counterpart on 17 out of 18 datasets. The code and pre-trained models are released at: https://github.com/microsoft/esvit
中文速览
多阶段稀疏自注意力视觉Transformer在自监督预训练中计算效率很高，但会严重损失模型捕捉图像局部区域间细粒度对应关系的能力——单体ViT在区域匹配准确率上达到95%，换成多阶段结构后骤降至66%。为此，作者提出了一种区域匹配（region matching）预训练任务：在师生自蒸馏框架中，让模型不仅对齐两个增强视图的整体表示，还显式匹配两个视图中余弦相似度最高的局部区域特征对，从而弥补多阶段结构丢失的细粒度感知能力。将高效的多阶段架构与该区域级预训练任务结合，所得模型EsViT在ImageNet线性探测上达到81.3% top-1精度，超越此前最优方法的同时吞吐量提升约10倍、参数量减少约3.5倍，并在18个下游分类数据集中的17个上超过了有监督的Swin Transformer。这项工作表明，架构效率与表示质量并非不可兼得，关键在于用合适的预训练任务补偿架构简化所带来的信息损失。
原文 arXiv:2106.09785;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.09785v2