Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Sixiao Zheng1 Jiachen Lu1 Hengshuang Zhao2 Xiatian Zhu3 Zekun Luo4 Yabiao Wang4 Yanwei Fu1 Jianfeng Feng1 Tao Xiang3, 5 Philip H.S. Torr2 Li Zhang1 1Fudan University 2University of Oxford 3University of Surrey 4Tencent Youtu Lab 5Facebook AI https://fudan-zvg.github.io/SETR Work done while Sixiao Zheng was interning at Tencent Youtu Lab.Li Zhang is the corresponding author with School of Data Science, Fudan University.
Abstract
Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (i.e., without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the fir
中文速览
语义分割长期依赖全卷积网络（FCN），其编码器靠逐步压缩空间分辨率来扩大感受野、建模上下文，但这种设计天然限制了对远程依赖关系的捕捉能力。针对这一瓶颈，研究者提出将语义分割重新定义为序列到序列的预测任务，用纯Transformer（SEgmentation TRansformer，SETR）作为编码器：把输入图像切成固定大小的图块并展平为序列，再通过多层全局自注意力机制学习特征表示，整个编码过程始终保持空间分辨率不变、每一层都具有全局感受野，最后配合轻量级解码器还原像素级分类结果。在多个权威基准上，SETR均取得了当时最优成绩，包括ADE20K（mIoU 50.28%）和Pascal Context（mIoU 55.83%），并在ADE20K测试集排行榜上位列第一。这项工作的意义在于彻底打破了FCN框架对语义分割的垄断，证明无需任何卷积操作、仅靠纯Transformer编码器就能在密集预测任务上超越精心设计的卷积网络，为视觉感知模型的架构创新开辟了新方向。
原文 arXiv:2012.15840;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.15840v3