Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
Mosh Levy1 Alon Jacoby Yoav Goldberg1,2 1Bar-Ilan University 2Allen Institute for AI {moshe0110, These authors contributed equally to this work.
Abstract
This paper explores the impact of extending input lengths on the capabilities of Large Language Models (LLMs). Despite LLMs advancements in recent times, their performance consistency across different input lengths is not well understood. We investigate this aspect by introducing a novel QA reasoning framework, specifically designed to assess the impact of input length. We isolate the effect of input length using multiple versions of the same sample, each being extended with padding of different lengths, types and locations. Our findings show a notable degradation in LLMs’ reasoning performance at much shorter input lengths than their technical maximum. We show that the degradation trend appears in every version of our dataset, although at different intensities. Additionally, our study reveals that the traditional metric of next word prediction correlates negatively with performance of LLMs’ on our reasoning dataset. We analyse our results and identify failure modes that can serve as useful guides for future research, potentially informing strategies to address the limitations observed in LLMs.
中文速览
大型语言模型（LLM）宣称支持越来越长的输入，但加长输入后推理能力究竟如何变化，此前缺乏严格控制的研究。为此，作者构建了一个名为 FLenQA 的问答数据集，核心设计是：对同一道需要联合推理两条关键信息的判断题，通过在原始内容前后或中间插入与任务无关的"填充文本"来人为拉长输入，从而把"输入长度"这一变量单独剥离出来加以考察。实验结果显示，模型的推理准确率在输入仅约 3000 个 token 时便已大幅下滑（平均从 0.92 降至 0.68），远未达到其官方标称的最大输入上限；而且无论填充文本的类型、来源还是关键信息在文中的位置如何变化，这一下降趋势始终存在。更值得注意的是，模型在长文本上的"下一词预测"困惑度与其长上下文推理准确率之间呈负相关，说明用语言模型常规指标来评估长文本推理能力并不可靠。这项工作揭示了当前 LLM 在处理长输入时存在系统性缺陷，并为后续改进提供了明确的失败模式参考。
原文 arXiv:2402.14848;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2402.14848v2