DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph
Zhehao Zhang Dartmouth College、Jiaao Chen Georgia Institute of Technology Diyi Yang Stanford University
Abstract
The current paradigm of evaluating Large Language Models (LLMs) through static benchmarks comes with significant limitations, such as vulnerability to data contamination and a lack of adaptability to the evolving capabilities of LLMs. Therefore, evaluation methods that can adapt and generate evaluation data with controlled complexity are urgently needed. In this work, we introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks. We further use a code-augmented LLM to ensure the label correctness of newly generated data. We apply our DARG framework to diverse reasoning tasks in four domains with 15 state-of-the-art LLMs. Experimental results show that almost all LLMs experience a performance decrease with increased complexity and certain LLMs exhibit significant drops. Addit
中文速览
静态基准测试已经无法真实反映大型语言模型（LLM）的实际能力——数据污染和固定难度让评分结果越来越不可信。为此，研究者提出了DARG（基于自适应推理图演化的动态评测框架）：先从现有基准题目中提取"推理图"（reasoning graph），用节点表示推理单元、用边表示运算关系，再通过规则化的图扰动（增加深度、宽度或数值复杂度）自动生成难度可控的新测试题，最后借助代码增强的LLM验证新题的标签正确性。把DARG应用于数学、社会、空间和符号四类推理任务、覆盖15款主流模型后，结果显示：所有模型的得分都随题目复杂度上升而下降，部分模型下滑尤为显著；在社会和空间推理任务中，复杂度越高、模型暴露出的偏见也越明显，即便是GPT-4 Turbo这类顶尖模型也不例外。这项工作揭示了静态基准的根本局限，为动态、自适应地评估LLM真实能力提供了一套可复现、可扩展的新方法。
原文 arXiv:2406.17271;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2406.17271v1