Confident Adaptive Language Modeling
Tal Schuster1, Adam Fisch2,∗ Jai Gupta1 Mostafa Dehghani1 Dara Bahri1 Vinh Q. Tran1 Yi Tay1 Donald Metzler1 1Google Research 2CSAIL, MIT Project leads. Correspondence to:
Abstract
Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in the models’ size, potentially leading to slow and costly use at inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly benefit from the models’ full capacity, other continuations are more trivial and can be solved with reduced compute. In this work, we introduce Confident Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of compute per input and generation timestep. Early exit decoding involves several challenges that we address here, such as: (1) what confidence measure to use; (2) connecting sequence-level constraints to local per-token exit decisions; and (3) attending back to missing hidden representations due to early exits in previous tokens. Through theoretical analysis and empirical experiments on three diverse text generation tasks, we demonstrate the efficacy of our framework in reducing compute—speedup of up to $\times 3$ —while provably maintaining high per
中文速览
大型语言模型（LLM）每次生成一个词时都要跑完所有层，但实际上很多词的预测根本不需要那么多计算——"the""is"这类简单词在浅层就已经有把握了。为此，研究者提出了 CALM（Confident Adaptive Language Modeling，自信自适应语言建模）框架：在自回归解码时，每生成一个词都动态决定"走到第几层就够了"，而不是每次都跑完全部层，从而节省计算量。难点在于如何把"每个词局部的提前退出决策"与"整段输出的质量保证"联系起来，论文为此引入了无分布风险控制理论，用少量校准数据就能在统计上证明：提前退出后的输出与完整模型的输出在 ROUGE、BLEURT 等指标上的差距可被严格控制在设定容忍范围内。在文本摘要、机器翻译和问答三项任务上，CALM 在几乎不损失性能的前提下实现了最高 3 倍的推理加速，为让 LLM 在现实场景中既快又可靠地部署提供了一条有理论保证的实用路径。
原文 arXiv:2207.07061;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2207.07061v2