ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models
Linting Xue , Aditya Barua , Noah Constant , Rami Al-Rfou , Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel Google Research {lintingx, adityabarua, nconstant, rmyeid, sharannarang, mihirkale, adarob} @google.com, Equal contribution.
Abstract
Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: they can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5
中文速览
用标准 Transformer 直接"吃"原始字节流来处理文本，长期以来被认为效率太低，因为字节序列比词/子词序列长得多。本文提出 ByT5，对现有的多语言预训练模型 mT5 做极少量改动——去掉分词器、直接输入 UTF-8 字节、加重编码器深度、拉长掩码跨度——就实现了无需任何文本预处理的端到端字节级文本模型。实验表明，ByT5 在仅用 mT5 四分之一文本数据预训练的情况下，在多项英文和多语言任务上与子词基线持平甚至更优，尤其在拼写噪声和语音敏感任务上表现显著更强。这项工作证明了无需设计全新架构也能让字节级模型达到实用水准，同时为语言多样性、鲁棒性和流水线简化提供了一条切实可行的路径。
原文 arXiv:2105.13626;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.13626v3