AgentSims: An Open-Source Sandbox for Large Language Model Evaluation
Jiaju Lin1,2, Haoran Zhao1,3 , Aochi Zhang1, Yiting Wu1,4, Huqiuyue Ping1,5, Qin Chen6 1PTA Studio 2 Pennsylvania State University, 3 Beihang University, 4 Sun Yat-sen University, 5Zhejiang University, 6East China Normal University ∗ Corresponding author.
Abstract
With ChatGPT-like large language models (LLM) prevailing in the community, how to evaluate the ability of LLMs is an open question. Existing evaluation methods suffer from following shortcomings: (1) constrained evaluation abilities, (2) vulnerable benchmarks, (3) unobjective metrics. We suggest that task-based evaluation, where LLM agents complete tasks in a simulated environment, is a one-for-all solution to solve above problems. We present AgentSims, an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in. Researchers can build their evaluation tasks by adding agents and buildings on an interactive GUI or deploy and test new support mechanisms, i.e. memory, planning and tool-use systems, by a few lines of codes. Our demo is available at https://agentsims.com .
中文速览
大型语言模型（LLM）性能越来越强，但如何客观、全面地评估它们的能力却始终是个难题——现有基准测试要么格式单一、考察面窄，要么测试集容易泄露进训练数据被"刷榜"，要么依赖GPT-4这类模型打分而缺乏客观性。为此，研究者提出用"任务式评估"来替代传统问答测评：让LLM驱动的智能体（agent）在一个模拟社会环境中完成预设任务，以任务通过率作为客观指标。围绕这一思路，他们开发了AgentSims——一个可视化、交互式的沙盒平台，既支持非技术背景的研究者通过图形界面拖拽创建场景和角色，也允许开发者用少量代码灵活替换记忆、规划、工具调用等核心模块。实验结果表明，该平台能有效支撑多样化的评估任务设计，且任务通过率这一指标不依赖任何"黑盒"评分过程，更公平透明。这项工作的意义在于为整个NLP社区提供了一套标准化、可扩展的LLM评估基础设施，有望吸引行为经济学、社会心理学等不同领域的研究者共同参与，推动LLM评估走向更系统、更可信的新范式。
原文 arXiv:2308.04026;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.04026v1