Language Models are General-Purpose Interfaces
Yaru Hao, Haoyu Song, Li Dong Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, Furu Wei† Microsoft Research https://github.com/microsoft/unilm Equal contribution. ††\dagger Corresponding author.
Abstract
Foundation models have received much attention due to their effectiveness across a broad range of downstream applications. Though there is a big convergence in terms of architecture, most pretrained models are typically still developed for specific tasks or modalities. In this work, we propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language), and they dock with a language model that plays the role of a universal task layer. We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders. We subsume the advantages and capabilities from both causal and non-causal modeling, thereby combining the best of two worlds. Specifically, the proposed method not only inherits the capabilities of in-context learning and open-ended generation from causal language modeling, but also is conducive to finetuning because of the bidirectional encoders. More importantly, our approach seamlessly unlocks the combinations of the above capabilities, e.g., enabling in-context learning or instruction following with finetuned encoders.
中文速览
大型语言模型在多种任务上表现出色，但现有预训练模型通常只针对特定任务或模态，难以同时兼顾"微调效果好"和"零样本/少样本泛化强"这两个目标。MetaLM提出用一个因果语言模型（causal language model）作为通用任务接口，让多个双向编码器（bidirectional encoder）负责感知不同模态的输入（如图像、文本），再通过一种新的"半因果语言建模"（semi-causal language modeling）预训练目标把两者联合训练起来——编码器以双向方式理解输入，语言模型以自回归方式生成输出。这样设计既继承了双向编码器微调性能强的优势，又保留了因果语言模型上下文学习（in-context learning）和开放式生成的能力，还能将两者自由组合，例如先微调再用示例提示进行上下文定制。在纯语言和视觉-语言多项基准测试上，MetaLM的微调、零样本泛化和少样本学习表现均与甚至超过专用模型，为构建真正通用的多模态AI接口提供了一条可行路径。
原文 arXiv:2206.06336;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.06336v1