Creator: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Cheng Qian1, Chi Han2, Yi R. Fung2, Yujia Qin1, Zhiyuan Liu1 , Heng Ji2∗ 1Tsinghua University, 2University of Illinois at Urbana-Champaign qianc20 @ mails.tsinghua.edu.cn Corresponding author.
Abstract
Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particularly when both planning and execution are involved. To overcome these limitations, we propose Creator, a novel framework that enables LLMs to create their own tools using documentation and code realization. Creator disentangles abstract tool creation and concrete decision execution, resulting in improved performance. We evaluate Creator on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems and diverse tabular contents. Remarkably, Creator outperforms existing chain-of-thought, program-of-thought, and tool-using baselines. Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs’ tool creation ability. Further research demonstrates that leveraging LLMs as tool creators facilitates knowledge transfer, and LLMs exhibit varying levels of tool creation abilities, enabling them to adapt to diverse situations. The tool creation ability revolutionizes the LLM’s problem-solving paradigm, driv
中文速览
大型语言模型（LLM）在借助外部工具解决问题时，往往受限于可用工具的数量不足、逐题推理容易出错、以及缺乏有效的错误处理机制。为此，研究者提出了 CREATOR 框架，让 LLM 不再只是工具的"使用者"，而是能够根据问题自行"创造"工具——先通过抽象推理生成带文档说明的通用函数代码，再决策如何调用这些工具求解具体问题，执行后若遇到报错还能自动根据错误追踪信息进行修正。在涵盖高难度数学竞赛题的 MATH 数据集和需要理解表格内容的 TabMWP 数据集上，基于 CREATOR 的 ChatGPT 分别达到 59.7% 和 94.7% 的准确率，显著超越了思维链（chain-of-thought）、程序化推理（program-of-thought）以及现有工具调用基线。这项工作的重要性在于：它将 LLM 的问题求解范式从"找工具用"升级为"按需造工具"，不仅提升了准确性和鲁棒性，还使所创工具可在相似问题间复用传递知识，朝着更强通用智能迈出了实质性的一步。
原文 arXiv:2305.14318;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.14318v3