arXiv:2611.23229 · 中英对照阅读
Ambient LLM:通过温度匹配实现数据高效的监督微调
Ambient LLMs: Data-Efficient Supervised Fine-tuning via Temperature Matching
中文速览
小规模的后训练数据容易让语言模型过度贴合新数据,忘掉预训练时学到的广泛知识,研究者想找一种不必大量收集新数据的解决办法。他们提出 Ambient SFT:微调时,除了学习新数据里的正确答案,还让新模型在高温(也就是输出更随机、更平滑)下保留与冻结的预训练模型相似的预测,并用温度参数调节对旧模型的信任程度。实验显示,这种方法在领域适配、指令微调和推理任务上都优于普通微调,例如领域适配的困惑度最多降低 26.7%,推理评测中赢下六项里的五项。它的意义在于,模型能更充分地利用有限的后训练数据,在适应新任务的同时减少遗忘,而且训练成本只增加约一成。
摘要
Collecting large-scale, diverse, and high-quality post-training datasets for autoregressive language models is challenging. Consequently, Supervised Fine-Tuning (SFT) is typically applied to datasets significantly smaller than those used for pre-training, which leads to overfitting and the catastrophic forgetting of the tails of the data distribution. To prevent this collapse, we introduce Ambient SFT, a principled framework for leveraging imperfect predictions to augment the fine-tuning set. In our method, a “frozen” version of the pre-trained model serves as a noisy teacher that the fine-tuned model must match at a coarse level. In particular, we require that the high-temperature versions of the conditional distributions produced by the post-training model should match the high-temperature versions of the pre-training ones. The temperature hyperparameter controls the level of trust in the pre-trained model, with the limit $T\to\infty$ reverting back to regular SFT and $T=1$ to knowledge distillation. We demonstrate that our approach outperforms standard SFT across domain adaptation, instruction-tuning, and reasoning.
术语表
- Ambient SFT
- Ambient SFT
- Supervised Fine-Tuning (SFT)
- 监督微调
- autoregressive language model (ARLM)
- 自回归语言模型
- Large Language Model (LLM)
- 大型语言模型
- pre-training
- 预训练
- post-training
- 后训练
- instruction-tuning
- 指令微调
- domain adaptation
- 领域适配
- alignment
- 对齐
- reasoning
- 推理
- catastrophic forgetting
- 灾难性遗忘
- alignment tax
- 对齐税
- overfitting
- 过拟合
- data efficiency
- 数据效率
- noisy teacher
- 噪声教师模型
- weak teacher
- 弱教师模型
- knowledge distillation
- 知识蒸馏
- Temperature Matching
- 温度匹配
- sampling temperature
- 采样温度
- conditional distribution
- 条件分布
- high-temperature conditional distribution
- 高温条件分布
- target distribution
- 目标分布
- population negative log-likelihood
- 总体负对数似然
- cross-entropy
- 交叉熵
- logit
- logit
- perplexity
- 困惑度
- pre-training corpus
- 预训练语料库
- empirical distribution
- 经验分布