Pearl: Prompting Large Language Models to Plan and Execute Actions Over Long Documents
Simeng Sun1 Yang Liu2 Shuohang Wang2 Chenguang Zhu2 Mohit Iyyer1 University of Massachusetts Amherst1 Microsoft Research2 {simengsun, Work partially done during an internship at Microsoft.
Abstract
Strategies such as chain-of-thought prompting improve the performance of large language models (LLMs) on complex reasoning tasks by decomposing input examples into intermediate steps. However, it remains unclear how to apply such methods to reason over long input documents, in which both the decomposition and the output of each intermediate step are non-trivial to obtain. In this work, we propose Pearl, a prompting framework to improve reasoning over long documents, which consists of three stages: action mining, plan formulation, and plan execution. More specifically, given a question about a long document, Pearl decomposes the question into a sequence of actions (e.g., SUMMARIZE, FIND_EVENT, FIND_RELATION) and then executes them over the document to obtain the answer. Each stage of Pearl is implemented via zero-shot or few-shot prompting of LLMs (in our work, GPT-4) with minimal human input. We evaluate Pearl on a challenging subset of the QuALITY dataset, which contains questions that require complex reasoning over long narrative texts. Pearl outperforms zero-shot and chain-of-thought prompting on this dataset, and ablation experiments show that each stage of Pearl is critical to
中文速览
面对"如何让大语言模型（LLM）对长篇文档进行复杂推理"这一难题，研究者提出了一个名为 Pearl（Planning and Executable Actions for Reasoning over Long documents）的提示框架，将回答问题的过程分解为三个阶段：先从训练数据中自动挖掘适用于长文档推理的动作集合（如 SUMMARIZE、FIND_RELATION），再让模型根据具体问题生成由这些动作组成的可执行计划，最后逐步执行计划并结合原始文档得出答案。在需要全局理解长篇叙事文本的 QuALITY 数据集挑战子集上，Pearl 的表现明显优于直接提问和思维链（Chain-of-Thought）提示，消融实验也证实三个阶段缺一不可。这项工作的意义在于，它提供了一条无需大量人工标注、仅靠 LLM 自身生成动作和计划就能处理长文档复杂推理的可行路径，为该方向的后续研究奠定了基础。
原文 arXiv:2305.14564;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.14564v1