A Few More Examples May Be Worth Billions of Parameters
Yuval Kirstain♠ Patrick Lewis†‡ Sebastian Riedel†‡ Omer Levy♠‡ ♠ Tel-Aviv University † University College London ‡ Facebook AI Research
Abstract
We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks. Our exploration reveals that while scaling parameters consistently yields performance improvements, the contribution of additional examples highly depends on the task’s format. Specifically, in open question answering tasks, enlarging the training set does not improve performance. In contrast, classification, extractive question answering, and multiple choice tasks benefit so much from additional examples that collecting a few hundred examples is often “worth” billions of parameters. We hypothesize that unlike open question answering, which involves recalling specific information, solving strategies for tasks with a more restricted output space transfer across examples, and can therefore be learned with small amounts of labeled data.111Our code is publicly available: https://github.com/yuvalkirstain/lm-evaluation-harness.
中文速览
大规模语言模型（large language model, LLM）的能力到底来自"更多参数"还是"更多标注数据"，这个问题因任务类型不同而有截然不同的答案。研究者用T5的四种规格（77M到3B参数）在12个任务上系统训练了7680个模型，覆盖分类、抽取式问答、多项选择和开放式问答四类格式，标注样本量从32到2048不等。结果发现：开放式问答（open QA）的性能几乎只随参数量增长，增加数百甚至数千条训练样本几乎没有帮助；而分类、抽取式问答和多项选择任务则能从更多数据中大幅获益，仅凭512条标注样本微调的T5-L（8亿参数）就能超过GPT-3（1750亿参数）。研究者的解释是：开放式问答要求模型凭记忆召回特定知识，必须依赖参数量；而输出空间受限的任务存在可迁移的"解题策略"（如排除法），少量样本就能教会模型，因此数据与参数可以互相"替代"。这一发现对实际应用中如何权衡算力投入与数据标注成本具有直接指导意义。
原文 arXiv:2110.04374;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.04374v1