UL2: Unifying Language Learning Paradigms
Yi Tay Mostafa Dehghani∗ Vinh Q. Tran♯ Xavier Garcia♯ Jason Wei♯ Xuezhi Wang♯ Hyung Won Chung♯ Siamak Shakeri♯ Dara Bahri♭ Tal Schuster♭ Huaixiu Steven Zheng△ Denny Zhou△ Neil Houlsby△ Donald Metzler△ Google Brain Yi and Mostafa are co-leads of this project and are denoted with ∗. ♯♯\sharp denotes technical research contributors. ♭♭\flat denotes data、infrastructure contributions. △ denotes advising contributions. Don, denoted with □ is the last author. Full contributions of all authors at the end of paper. Correspondence to or
Abstract
Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives – two concepts that are commonly conflated. Next, we present a generalized and unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 and/or GPT-like models across multiple diverse setups. Finally, by scaling our model up to 20B parameters, we a
中文速览
训练一个在所有NLP任务上都表现出色的通用语言模型，一直是个难题——现有模型往往在生成任务或理解任务上各有偏长，很难两全。谷歌研究团队提出了UL2框架，核心创新是"去噪混合"（Mixture-of-Denoisers, MoD）预训练目标：把标准片段掩码、前缀语言模型、极端长跨度破坏等多种预训练范式统一为一套机制，并引入"模式切换"（mode switching）让模型在微调时能按需激活不同的预训练行为。实验结果显示，20B参数的UL2在涵盖文本生成、问答、常识推理、结构化知识、信息检索等50余项NLP任务上达到最先进水平，零样本SuperGLUE表现超越拥有175B参数的GPT-3，单样本摘要成绩是同规模T5-XXL的三倍；在此基础上叠加FLAN指令微调后，Flan-UL2 20B的表现甚至能与FLAN-PaLM 62B媲美。这一工作的意义在于打破了"不同任务必须选择不同架构或预训练方案"的惯例，为构建真正通用的语言模型提供了一条切实可行的路径，且全部模型权重已公开发布。
原文 arXiv:2205.05131;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.05131v3