GeckOpt: LLM System Efficiency via Intent-Based Tool Selection
Michael Fore Microsoft CorporationRestonVAUSA , Simranjit Singh Microsoft CorporationSilicon Valley CampusCAUSA and Dimitrios Stamoulis Microsoft CorporationRedmondWAUSA
Abstract
In this preliminary study, we investigate a GPT-driven intent-based reasoning approach to streamline tool selection for large language models (LLMs) aimed at system efficiency. By identifying the intent behind user prompts at runtime, we narrow down the API toolset required for task execution, reducing token consumption by up to 24.6%. Early results on a real-world, massively parallel Copilot platform with over 100 GPT-4-Turbo nodes show cost reductions and potential towards improving LLM-based system efficiency.
中文速览
大型语言模型在调用工具完成任务时，往往每步只调用一个API，导致请求次数多、token消耗大、云端成本高。针对这一问题，研究者提出了GeckOpt方法：先用一次额外的LLM调用识别用户提示背后的"意图"，据此将庞大的API工具库缩减为一个更小的候选子集，再让模型在这个精简池里做后续规划，从而鼓励模型在单步内同时调用多个工具。在拥有百余个GPT-4-Turbo节点的大规模地理空间Copilot平台上测试，GeckOpt最高可将token消耗降低24.6%，同时任务成功率下降不超过1%。这项工作的意义在于，即使是近四分之一的token节省，乘以海量用户会话后也将带来可观的云成本缩减，并为未来在本地部署LLM、提升硬件利用率提供了新思路。
原文 arXiv:2404.15804;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2404.15804v1