ToolQA: A Dataset for LLM Question Answering with External Tools
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, Chao Zhang College of Computing, Georgia Institute of Technology, Atlanta GA {yczhuang, yueyu, kuanwang, haotian.sun, These authors contributed equally to this work.
Abstract
Large Language Models (LLMs) have demonstrated impressive performance in various NLP tasks, but they still suffer from challenges such as hallucination and weak numerical reasoning. To overcome these challenges, external tools can be used to enhance LLMs’ question-answering abilities. However, current evaluation methods do not distinguish between questions that can be answered using LLMs’ internal knowledge and those that require external information through tool use. To address this issue, we introduce a new dataset called ToolQA, which is designed to faithfully evaluate LLMs’ ability to use external tools for question answering. Our development of ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions. Importantly, we strive to minimize the overlap between our benchmark data and LLMs’ pre-training data, enabling a more precise evaluation of LLMs’ tool-use reasoning abilities. We conducted an in-depth diagnosis of existing tool-use LLMs to highlight their strengths, weaknesses, and potential improvements. Our findings set a new benchmark for evaluating LLMs an
中文速览
大型语言模型（LLMs）虽然能力出众，却存在"幻觉"（hallucination）和数值推理薄弱等缺陷，借助外部工具可以弥补这些不足，但现有评测数据集无法区分模型究竟是在调用工具解题还是仅仅在"背答案"，导致对工具使用能力的评估失真。为此，研究团队构建了 ToolQA 这一新基准，覆盖 8 个领域，配套 13 种专用工具，所有问题都只能通过工具查询外部参考语料才能作答，且参考语料有意避开了 LLM 预训练数据，从根本上杜绝了模型靠记忆"蒙对"的可能性。数据集的生成采用了"人工引导模板 + 程序化自动标注"的流水线，大幅降低了人工标注成本。实验结果显示，仅依赖内部知识的 ChatGPT 在简单题和难题上的正确率分别只有约 5% 和 2%，最好的工具增强型 LLM 在简单题上也只达到 43%、难题上仅有 8%，说明这一基准极具挑战性，也为工具增强型 LLM 的研究指出了明确的改进方向。
原文 arXiv:2306.13304;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.13304v1