A Survey of Transformers
Tianyang Lin 0000-0003-1193-6472 , Yuxin Wang , Xiangyang Liu and Xipeng Qiu 0000-0001-7163-5247 School of Computer Science, Fudan UniversityShanghaiChina200433 Shanghai Key Laboratory of Intelligent Information Processing, Fudan UniversityShanghaiChina200433
Abstract
Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.
中文速览
Transformer自2017年提出以来迅速席卷自然语言处理、计算机视觉、语音等众多AI领域，由此催生了海量改进变体（X-formers），但学界一直缺乏一份系统全面的综述来梳理这些工作。这篇论文从架构改进、预训练和应用三个维度构建了一套新的分类体系，重点深入剖析了注意力机制（如稀疏注意力、线性化注意力）、位置编码、层归一化、前馈网络等模块层面的改进，同时也概览了预训练模型（如BERT、GPT系列）和跨领域应用的进展。调研发现，现有工作主要围绕降低自注意力的二次方复杂度、提升小数据场景的泛化能力以及适配特定下游任务三条主线展开，并归纳了若干值得继续探索的未来方向。这份综述为研究者快速掌握Transformer变体全貌、选择适合自身场景的模型改进思路提供了系统性参考。
原文 arXiv:2106.04554;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.04554v2