Conditional Positional Encodings for Vision Transformers
Xiangxiang Chu1, Zhi Tian1, Bo Zhang1, Xinlong Wang2, Chunhua Shen3 1 Meituan Inc. 2 Beijing Academy of AI 3 Zhejiang University, China {chuxiangxiang, tianzhi02, Corresponding author.
Abstract
We propose a conditional positional encoding (CPE) scheme for vision Transformers (Dosovitskiy et al., 2021; Touvron et al., 2020). Unlike previous fixed or learnable positional encodings that are predefined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during the training. Besides, CPE can keep the desired translation equivalence in vision tasks, resulting in improved performance. We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework. Built on PEG, we present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results. Our Code is available at: https://git.io/CPVT.
中文速览
视觉Transformer在处理图像时面临一个棘手问题：传统的固定或可学习位置编码（positional encoding）在训练结束后就"定死"了，测试时遇到比训练图像更大的输入就会性能下滑，而且这类编码会破坏平移等变性（translation equivariance），影响模型对移动目标的识别能力。为此，作者提出了条件位置编码（Conditional Positional Encoding, CPE）——用一个轻量的位置编码生成器（Position Encoding Generator, PEG）根据输入token的局部邻域动态生成位置编码，而不是预先固定一套编码加到每个token上。实验表明，基于CPE构建的CPVT模型在ImageNet分类上超越了DeiT等基线，在更高分辨率输入（如384×384）下无需微调性能反而继续提升，而DeiT在同等情况下会明显下降；在目标检测和语义分割任务中，将CPE引入金字塔Transformer同样带来了显著增益。这项工作以极小的改动代价解决了视觉Transformer的位置编码泛化难题，对需要处理任意分辨率图像的下游任务具有重要的实用价值。
原文 arXiv:2102.10882;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.10882v3