General-purpose, long-context autoregressive modeling with Perceiver AR
Curtis Hawthorne Andrew Jaegle Cătălina Cangea Sebastian Borgeaud Charlie Nash Mateusz Malinowski Sander Dieleman Oriol Vinyals Matthew Botvinick Ian Simon Hannah Sheahan Neil Zeghidour Jean-Baptiste Alayrac João Carreira Jesse Engel
Abstract
Real-world data is high-dimensional: a book, image, or musical performance can easily contain hundreds of thousands of elements even after compression. However, the most commonly used autoregressive models, Transformers, are prohibitively expensive to scale to the number of inputs and layers needed to capture this long-range structure. We develop Perceiver AR, an autoregressive, modality-agnostic architecture which uses cross-attention to map long-range inputs to a small number of latents while also maintaining end-to-end causal masking. Perceiver AR can directly attend to over a hundred thousand tokens, enabling practical long-context density estimation without the need for hand-crafted sparsity patterns or memory mechanisms. When trained on images or music, Perceiver AR generates outputs with clear long-term coherence and structure. Our architecture also obtains state-of-the-art likelihood on long-sequence benchmarks, including 64 $\times$ 64 ImageNet images and PG-19 books.
中文速览
真实世界的数据（图像、音乐、书籍）动辄包含数十万个元素，而主流的自回归模型 Transformer 的计算量会随序列长度平方级增长，根本撑不住这么长的上下文。为此，研究者提出了 Perceiver AR：它借助交叉注意力（cross-attention）将超长输入压缩到少量"潜变量"（latents）再做深层处理，同时通过对交叉注意力和自注意力全程施加因果掩码，保证自回归生成所必须的前因后果顺序。这样一来，模型的计算量与输入长度解耦，可以直接处理超过十万个词元，既不需要手工设计稀疏模式，也不需要额外的记忆机制。实验表明，Perceiver AR 在 64×64 ImageNet 图像密度估计和 PG-19 长篇书籍语言建模上达到当时最优水平，生成的图像和音乐也呈现出明显的长程连贯结构，证明了用一套统一架构高效捕捉多模态长程依赖的可行性。
原文 arXiv:2202.07765;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2202.07765v2