FireAct: Toward Language Agent Fine-tuning
Baian Chen System2 Research、Chang Shu∗ University of Cambridge、Ehsan Shareghi Monash University \ANDNigel Collier University of Cambridge、Karthik Narasimhan PLI, Princeton University、Shunyu Yao PLI, Princeton University Equal contribution. Code, data, and models are available at https://fireact-agent.github.io.
Abstract
Recent efforts have augmented language models (LMs) with external tools or environments, leading to the development of language agents that can reason and act. However, most of these agents rely on few-shot prompting techniques with off-the-shelf LMs. In this paper, we investigate and argue for the overlooked direction of fine-tuning LMs to obtain language agents. Using a setup of question answering (QA) with a Google search API, we explore a variety of base LMs, prompting methods, fine-tuning data, and QA tasks, and find language agents are consistently improved after fine-tuning their backbone LMs. For example, fine-tuning Llama2-7B with 500 agent trajectories generated by GPT-4 leads to a 77% HotpotQA performance increase. Furthermore, we propose FireAct, a novel approach to fine-tuning LMs with trajectories from multiple tasks and prompting methods, and show having more diverse fine-tuning data can further improve agents. Along with other findings regarding scaling effects, robustness, generalization, efficiency and cost, our work establishes comprehensive benefits of fine-tuning LMs for agents, and provides an initial set of experimental designs, insights, as well as open ques
中文速览
用few-shot提示（few-shot prompting）驱动现成大模型来构建语言智能体（language agent）已是主流做法，但这种方式对小模型效果差、对GPT-4这类大模型又成本高昂。研究者系统探索了另一条路：直接对语言模型进行微调（fine-tuning），让它学会"边推理边调用工具"的智能体行为，并提出了FireAct方法——用GPT-4生成来自多任务、多提示策略的ReAct轨迹数据来混合训练较小的模型。实验结果表明，微调带来的提升极为显著，例如仅用500条轨迹微调Llama-2-7B，在多跳问答基准HotpotQA上的精确匹配得分就提升了77%，推理速度也加快约4倍，鲁棒性和泛化能力同步改善。这项工作证明，微调小型开源模型完全可以媲美甚至超越提示大型商业模型，为低成本、高性能语言智能体的落地提供了切实可行的方向。
原文 arXiv:2310.05915;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.05915v1