TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage
Jingqing Ruan†‡ Yihong Chen†‡ Bin Zhang†‡ Zhiwei Xu†‡ Tianpeng Bao† Guoqing Du† Shiwei Shi† Hangyu Mao†∗ Ziyue Li + Xingyu Zeng Rui Zhao University of Chinese Academy of Sciences, China SenseTime Research, HKUST, Hong Kong
Abstract
With recent advancements in natural language processing, Large Language Models (LLMs) have emerged as powerful tools for various real-world applications. Despite their powers, the intrinsic generative abilities of LLMs may prove insufficient for handling complex tasks, which necessitate a combination of task planning and the usage of external tools. In this paper, we first propose a structured framework tailored for LLM-based AI Agents and then discuss the crucial capabilities necessary for tackling intricate problems. Within this framework, we design two distinct types of agents (i.e., one-step agent and sequential agent) to execute the inference process. Subsequently, we instantiate the framework using various LLMs and evaluate their Task Planning and Tool Usage (TPTU) abilities on typical tasks. By highlighting key findings and challenges, our goal is to provide a helpful resource for researchers and practitioners to leverage the power of LLMs in their AI applications. Our study emphasizes the substantial potential of these models while also identifying areas that need more investigation and improvement. The code and resources will be available on GitHub.
中文速览
大语言模型（LLM）在实际应用中常面临两大瓶颈：无法可靠处理逻辑推理问题，以及难以将复杂任务拆解并借助外部工具完成。为此，研究者提出了一套面向LLM智能体（AI Agent）的结构化框架，并在其中设计了"一步智能体"（一次性规划所有子任务）和"序列智能体"（逐步推进子任务）两种执行模式，系统评估了任务规划与工具调用（TPTU）能力。实验以SQL查询和Python代码生成作为核心工具，对ChatGPT、GPT-4等多个主流模型进行了测试，发现这些模型在格式遵循、任务理解、工具选择均衡性和结果归纳方面仍存在明显短板。该研究为希望将LLM嵌入复杂应用系统的研究者和开发者提供了清晰的能力评估基准，也点明了当前智能体落地的关键改进方向。
原文 arXiv:2308.03427;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.03427v4