Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
Sijia Chen1,2,3,∗ Yibo Wang1,2,3, Yi-Feng Wu3 Qing-Guo Chen3 Zhao Xu3 Weihua Luo3 Kaifu Zhang3 Lijun Zhang1,4,2, 1National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China 2School of Artificial Intelligence, Nanjing University, Nanjing, China 3Alibaba International Digital Commerce 4Pazhou Laboratory (Huangpu), Guangzhou, China {chensj, wangyb, {yixin.wyf, qingguo.cqg, changgong.xz, weihua.luowh, Equal contribution. Work done during the internship at Alibaba International Digital Commerce.Corresponding author.
Abstract
Tool-augmented large language models (LLMs) leverage tools, often in the form of APIs, to improve their reasoning capabilities on complex tasks. This enables them to act as intelligent agents interacting with the real world. The recently introduced ToolLLaMA model by Qin et al. (2023) utilizes the depth-first search-based decision tree (DFSDT) mechanism for multi-step reasoning with $16000+$ real-world APIs, effectively enhancing the performance of tool-augmented LLMs compared to traditional chain reasoning mechanisms. However, their approach only employs successful paths from decision trees (also called inference trees) for supervised fine-tuning (SFT), missing out on the potential learning opportunities from failed paths. Inspired by this, we propose an inference trajectory optimization framework based on preference learning to address this limitation. We first introduce a novel method for constructing step-wise preference data from tree-like expert trajectories, which leverages the previously ignored failed explorations in the decision trees. In the subsequent training phase, we first fine-tune the LLM with successful tool-usage expert trajectories and then apply direct preferen
中文速览
大型语言模型（LLM）在借助外部工具（API）处理复杂任务时，现有方法（如ToolLLaMA）只从决策树中成功的推理路径学习，白白浪费了失败路径里蕴含的宝贵经验。为此，研究者提出了一套基于偏好学习（preference learning）的推理轨迹优化框架：先从树状专家轨迹中同时提取成功与失败的分支，构造逐步骤的偏好数据集ToolPreference，再用监督微调打好底子，最后用直接偏好优化（DPO）让模型从"踩过的坑"里学会规避错误决策，得到ToolPrefer-LLaMA（TP-LLaMA）模型。实验结果显示，TP-LLaMA在几乎所有测试场景下均大幅超越基线，平均通过率提升至少12%，推理所需步骤数从32步降至约23步，对未见过的API也表现出更强的泛化能力。这项工作表明，让模型向失败案例学习是提升工具调用能力的有效途径，也为未来基于树状轨迹的偏好数据构建提供了新思路。
原文 arXiv:2406.07115;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2406.07115v2