arXiv:2602.03587 · 中英对照阅读
CL-bench:上下文学习基准测试
CL-bench: A Benchmark for Context Learning
中文速览
现实任务常要求模型从陌生而复杂的材料中学会新知识,再把它用到具体问题上,但现有模型和评测往往只看它们能否调用已有知识、查找信息或读懂长文本。为此,研究者推出 CL-bench,收集专家设计的 500 组情境和近 1,900 项任务,考查模型能否学会材料中的专业知识、规则、操作流程和数据规律并据此完成任务。测试十种先进模型后,平均只有 17.2% 的任务被解决,表现最好的 GPT-5.1 也仅达到 23.7%,难题尤其容易让模型忽略材料内容或推理出错。这说明模型从上下文中真正学习并灵活应用新知识的能力仍很薄弱,而补上这一短板,是让 AI 更可靠地处理真实复杂工作的关键。
摘要
当前语言模型(LMs)擅长运用预训练中获得的知识对提示进行推理。然而,现实世界中的任务要复杂得多,也更依赖上下文:模型必须从特定任务的上下文中学习,并利用超出预训练所学范围的新知识来推理和解决任务。我们将这种能力称为上下文学习,这是一项至关重要的能力,人类天生具备,却在很大程度上被忽视。为此,我们推出了 CL-bench,这是一个面向现实世界的基准,由 500 个复杂上下文、1,899 项任务和 31,607 条验证标准组成,均由经验丰富的领域专家精心设计。每项任务的设计都确保解决任务所需的新内容包含在相应的上下文中。解决 CL-bench 中的任务,要求模型从上下文中学习,从新的领域专属知识、规则系统和复杂程序,到从经验数据中推导出的规律,内容涵盖广泛,而这些都未包含在预训练中。这远远超出了主要测试检索或阅读理解能力的长上下文任务,也超出了模型通过指令和示范学习简单任务模式的上下文学习任务。我们对十个前沿语言模型的评估发现,模型平均只能解决 17.2% 的任务。即使是表现最佳的模型 GPT-5.1,也仅能解决 23.7% 的任务,这表明语言模型尚未实现有效的上下文学习,而这构成了应对现实世界复杂上下文依赖型任务的关键瓶颈。CL-bench 是构建具备这一基础能力的语言模型的一步,将使模型更加智能,并推动其在现实世界场景中的部署。
Current language models (LMs) excel at reasoning over prompts using pre-trained knowledge. However, real-world tasks are far more complex and context-dependent: models must learn from task-specific context and leverage new knowledge beyond what is learned during pre-training to reason and resolve tasks. We term this capability context learning, a crucial ability that humans naturally possess but has been largely overlooked. To this end, we introduce CL-bench, a real-world benchmark consisting of 500 complex contexts, 1,899 tasks, and 31,607 verification rubrics, all crafted by experienced domain experts. Each task is designed such that the new content required to resolve it is contained within the corresponding context. Resolving tasks in CL-bench requires models to learn from the context, ranging from new domain-specific knowledge, rule systems, and complex procedures to laws derived from empirical data, all of which are absent from pre-training. This goes far beyond long-context tasks that primarily test retrieval or reading comprehension, and in-context learning tasks, where models learn simple task patterns via instructions and demonstrations. Our evaluations of ten frontier LMs find that models solve only 17.2% of tasks on average. Even the best-performing model, GPT-5.1, solves only 23.7%, revealing that LMs have yet to achieve effective context learning, which poses a critical bottleneck for tackling real-world, complex context-dependent tasks. CL-bench represents a step towards building LMs with this fundamental capability, making them more intelligent and advancing their deployment in real-world scenarios.
术语表
- language models (LMs)
- 语言模型
- context learning
- 上下文学习
- CL-bench
- CL-bench
- pre-training
- 预训练
- context-dependent tasks
- 上下文依赖型任务
- long-context tasks
- 长上下文任务
- in-context learning (ICL)
- 上下文学习(ICL)
- prompt engineering
- 提示工程
- context engineering
- 上下文工程
- verification rubrics
- 验证标准
- domain-specific knowledge
- 领域专属知识
- rule systems
- 规则系统
- procedural task execution
- 程序性任务执行
- empirical discovery
- 经验发现
- simulation
- 模拟
- domain knowledge reasoning
- 领域知识推理
- rule system application
- 规则系统应用
- long-context reasoning
- 长上下文推理
- instruction-following
- 指令遵循
- contamination-free
- 无数据污染
- Retrieval-Augmented Generation
- 检索增强生成
- RAG
- RAG
- memory systems
- 记忆系统
- agentic RAG pipelines
- 智能体式 RAG 流水线
- frontier LMs
- 前沿语言模型
- sandbox environments
- 沙盒环境
- task solve rate
- 任务解决率
- testbed
- 测试平台