Aha.
正在载入中英对照阅读…

arXiv:2611.23229 · 中英对照阅读

Ambient LLM:通过温度匹配实现数据高效的监督微调

Ambient LLMs: Data-Efficient Supervised Fine-tuning via Temperature Matching

Giannis Daras、Marios Papachristou、Adam Klivans、Antonio Torralba、Constantinos Daskalakis

中文速览

小规模的后训练数据容易让语言模型过度贴合新数据,忘掉预训练时学到的广泛知识,研究者想找一种不必大量收集新数据的解决办法。他们提出 Ambient SFT:微调时,除了学习新数据里的正确答案,还让新模型在高温(也就是输出更随机、更平滑)下保留与冻结的预训练模型相似的预测,并用温度参数调节对旧模型的信任程度。实验显示,这种方法在领域适配、指令微调和推理任务上都优于普通微调,例如领域适配的困惑度最多降低 26.7%,推理评测中赢下六项里的五项。它的意义在于,模型能更充分地利用有限的后训练数据,在适应新任务的同时减少遗忘,而且训练成本只增加约一成。

摘要

Collecting large-scale, diverse, and high-quality post-training datasets for autoregressive language models is challenging. Consequently, Supervised Fine-Tuning (SFT) is typically applied to datasets significantly smaller than those used for pre-training, which leads to overfitting and the catastrophic forgetting of the tails of the data distribution. To prevent this collapse, we introduce Ambient SFT, a principled framework for leveraging imperfect predictions to augment the fine-tuning set. In our method, a “frozen” version of the pre-trained model serves as a noisy teacher that the fine-tuned model must match at a coarse level. In particular, we require that the high-temperature versions of the conditional distributions produced by the post-training model should match the high-temperature versions of the pre-training ones. The temperature hyperparameter controls the level of trust in the pre-trained model, with the limit $T\to\infty$ reverting back to regular SFT and $T=1$ to knowledge distillation. We demonstrate that our approach outperforms standard SFT across domain adaptation, instruction-tuning, and reasoning.

术语表

Ambient SFT
Ambient SFT
Supervised Fine-Tuning (SFT)
监督微调
autoregressive language model (ARLM)
自回归语言模型
Large Language Model (LLM)
大型语言模型
pre-training
预训练
post-training
后训练
instruction-tuning
指令微调
domain adaptation
领域适配
alignment
对齐
reasoning
推理
catastrophic forgetting
灾难性遗忘
alignment tax
对齐税
overfitting
过拟合
data efficiency
数据效率
noisy teacher
噪声教师模型
weak teacher
弱教师模型
knowledge distillation
知识蒸馏
Temperature Matching
温度匹配
sampling temperature
采样温度
conditional distribution
条件分布
high-temperature conditional distribution
高温条件分布
target distribution
目标分布
population negative log-likelihood
总体负对数似然
cross-entropy
交叉熵
logit
logit
perplexity
困惑度
pre-training corpus
预训练语料库
empirical distribution
经验分布