Long Range Arena: A Benchmark for Efficient Transformers
Yi Tay1 , Mostafa Dehghani1∗, Samira Abnar1, Yikang Shen1, Dara Bahri1, Philip Pham1 Jinfeng Rao1, Liu Yang1, Sebastian Ruder2, Donald Metzler1 1Google Research 2Google DeepMind {yitay, First two authors contributed equally.
Abstract
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, Long-Range Arena, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, a
中文速览
标准Transformer的自注意力机制计算量随序列长度平方级增长，催生了大量声称能高效处理长序列的改进模型，但这些模型各自在不同任务和数据集上评测，彼此之间根本无从比较。为此，研究者提出了Long-Range Arena（LRA）这套统一基准，专门衡量模型在长上下文场景下的能力，包含从1K到16K token不等的六项任务，覆盖文本分类、文档检索、层次结构推理、图像像素序列分类以及视觉路径判断等多种类型。他们在这套基准上系统地评测了十种主流高效Transformer变体，结果发现各模型表现参差不齐，没有哪款能在所有任务上稳定领先，而且最具挑战性的极长序列任务（Pathfinder-X，16K token）几乎让所有模型都接近随机猜测水平。这项工作填补了高效Transformer评估体系的空白，为社区提供了公平、轻量、可复现的对比平台，也揭示出现有架构在真正长程依赖建模上仍存在巨大提升空间。
原文 arXiv:2011.04006;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.04006v1