Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models
Yuhui Zhang † Michihiro Yasunaga∗ Zhengping Zhou∗ Jeff Z. HaoChen∗ James Zou Percy Liang Serena Yeung Department of Computer Science Stanford University Equal contributions. †Correspondence to: Yuhui Zhang
Abstract
Language models have been shown to exhibit positive scaling, where performance improves as models are scaled up in terms of size, compute, or data. In this work, we introduce NeQA, a dataset consisting of questions with negation in which language models do not exhibit straightforward positive scaling. We show that this task can exhibit inverse scaling, U-shaped scaling, or positive scaling, and the three scaling trends shift in this order as we use more powerful prompting methods or model families. We hypothesize that solving NeQA depends on two subtasks: question answering (task 1) and negation understanding (task 2). We find that task 1 has linear scaling, while task 2 has sigmoid-shaped scaling with an emergent transition point, and composing these two scaling trends yields the final scaling trend of NeQA. Our work reveals and provides a way to analyze the complex scaling trends of language models.
中文速览
大语言模型（LLM）在大多数任务上随着规模增大表现越来越好，但含有否定词的问答却是个例外——本文为此构建了 NeQA 数据集，专门收录带否定词的多选题，用来探究模型规模与性能之间的复杂关系。实验发现，在不同的提示方式或模型系列下，NeQA 上的扩展规律（scaling trend）会依次呈现「反向扩展（inverse scaling）→ U 形扩展→正向扩展」三种形态，提示越强、模型系列越先进，曲线就越向正向扩展偏移。为解释这一现象，作者将任务拆解为两个子任务：回答原始（无否定）问题的能力随规模线性提升，而理解否定的能力则呈 S 形（sigmoid）涌现，两者叠加便产生了最终观察到的各种扩展曲线形态。这项工作不仅揭示了语言模型处理否定语义的深层瓶颈，也提供了一套通过子任务分解来分析复杂扩展规律的新方法，对理解和改进大模型具有重要参考价值。
原文 arXiv:2305.17311;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.17311v1