Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments
Sitao Cheng1, Ziyuan Zhuang1\samethanks, Yong Xu2, Fangkai Yang2, Chaoyun Zhang2, Xiaoting Qin2, Xiang Huang1, Ling Chen2, Qingwei Lin2, Dongmei Zhang2, Saravan Rajmohan2, Qi Zhang2 1State Key Laboratory for Novel Software Technology, Nanjing University, China 2Microsoft {stcheng, {yox, This work is done during the internship at Microsoft.
Abstract
Large Language Models (LLMs) have shown potential in reasoning over structured environments, e.g., knowledge graphs and tables. Such tasks typically require multi-hop reasoning, i.e., match natural language utterance with instances in the environment. Previous works adopt LLMs to incrementally build a reasoning path, where LLMs either invoke tools or pick up items by step-by-step interacting with the environment. We propose Reasoning-Path-Editing (Readi), a novel framework where LLMs can efficiently and faithfully reason over structured environments. In Readi, LLMs initially generate a reasoning path given a query, and edit the path only when necessary. We instantiate the path on structured environments and provide feedback to edit the path if anything goes wrong. Experimental results on three KGQA and two TableQA datasets show the effectiveness of Readi, significantly surpassing previous LLM-based methods (by 9.1% Hit@1 on WebQSP, 12.4% on MQA-3H and 9.5% on WTQ), comparable with state-of-the-art fine-tuned methods (67% on CWQ and 74.7% on WebQSP) and substantially boosting the vanilla LLMs (by 14.9% on CWQ). Our code will be available on https://aka.ms/readi.
中文速览
大型语言模型（LLM）在知识图谱问答（KGQA）和表格问答（TableQA）等结构化环境推理任务中潜力巨大，但现有方法要么逐步与环境交互导致调用次数过多、效率低下，要么靠微调记忆环境但无法保证推理路径真正对齐结构化数据。本文提出 Readi（Reasoning-Path-Editing），让 LLM 先一次性生成完整的推理路径，再将该路径映射到知识图谱或表格上执行实例化；只有当实例化在某处卡住时，才收集包含卡点位置、已完成的中间实例、相关关系等信息的"推理日志"，并将其反馈给 LLM 进行针对性修改，从而实现"按需调用"。在三个 KGQA 和两个 TableQA 数据集上的实验表明，Readi 平均仅需 1.55 次编辑调用，却在多个基准上大幅超越其他基于 LLM 的方法（如在 WebQSP 上提升 9.1%、WTQ 上提升 9.5%），并接近甚至超过需要大量标注数据的微调方法，兼顾了推理的高效性与对结构化环境的忠实性。
原文 arXiv:2403.08593;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2403.08593v2