A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Nikunj Saunshi Department of Computer Science, Princeton University Sadhika Malladi Department of Computer Science, Princeton University Sanjeev Arora Department of Computer Science, Princeton University Institute for Advanced Study
Abstract
Autoregressive language models, pretrained using large text corpora to do well on next word prediction, have been successful at solving many downstream tasks, even with zero-shot usage. However, there is little theoretical understanding of this success. This paper initiates a mathematical study of this phenomenon for the downstream task of text classification by considering the following questions: (1) What is the intuitive connection between the pretraining task of next word prediction and text classification? (2) How can we mathematically formalize this connection and quantify the benefit of language modeling? For (1), we hypothesize, and verify empirically, that classification tasks of interest can be reformulated as sentence completion tasks, thus making language modeling a meaningful pretraining task. With a mathematical formalization of this hypothesis, we make progress towards (2) and show that language models that are $\epsilon$ -optimal in cross-entropy (log-perplexity) learn features that can linearly solve such classification tasks with ${\mathcal{O}}(\sqrt{\epsilon})$ error, thus demonstrating that doing well on language modeling can be beneficial for downstream tasks.
中文速览
用"下一个词预测"来预训练语言模型（language model）已被证明对文本分类等下游任务很有用，但背后的数学原理从未被严格解释过。这篇论文的核心观察是：文本分类任务往往可以被改写成"句子补全"任务——比如判断情感时，可以看模型预测"The movie was ___"后面更可能出现"great"还是"terrible"——由此将语言模型预训练与分类任务自然地联系起来。在此基础上，作者提出"自然任务（natural task）"的数学定义，并从理论上证明：一个在交叉熵（cross-entropy）损失上达到ε最优的语言模型，其学到的特征可以用线性分类器以O(√ε)的误差解决这类分类任务，即语言模型预训练越好，下游分类性能就有保证地越好。此外，受理论分析启发，作者还提出了一种新的训练目标函数（Quad）以及"条件均值特征（conditional mean features）"，在标准基准测试上验证了理论预测，为理解大规模语言模型为何能零样本泛化到下游任务提供了首个严格的数学框架。
原文 arXiv:2010.03648;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.03648v2