Compositional Semantic Parsing with Large Language Models
Andrew Drozdov1,2,* Nathanael Schärli1,* Ekin Akyürek1,3 Nathan Scales1 Xinying Song1 Xinyun Chen1 Olivier Bousquet1 Denny Zhou1 1Google Research 2UMass Amherst CICS 3MIT CSAIL *Equal contribution
Abstract
Humans can reason compositionally when presented with new tasks. Previous research shows that appropriate prompting techniques enable large language models (LLMs) to solve artificial compositional generalization tasks such as SCAN. In this work, we identify additional challenges in more realistic semantic parsing tasks with larger vocabulary and refine these prompting techniques to address them. Our best method is based on least-to-most prompting: it decomposes the problem using prompting-based syntactic parsing, then uses this decomposition to select appropriate exemplars and to sequentially generate the semantic parse. This method allows us to set a new state of the art for CFQ while requiring only 1% of the training data used by traditional approaches. Due to the general nature of our approach, we expect similar efforts will lead to new results in other tasks and domains, especially for knowledge-intensive applications.
中文速览
组合泛化（compositional generalization）是衡量模型能否像人类一样把已知成分重新组合、理解新句子的关键能力，但现有神经网络在CFQ这类真实语义解析任务上表现很差。研究者在"由少到多提示"（least-to-most prompting）的基础上提出"动态由少到多提示"（dynamic least-to-most prompting），核心思路是：先用大语言模型做句法分析把输入句子分解成树状子问题，再依据这棵树从小规模候选样本池中动态挑选最相关的示例，最后按照子问题顺序逐步生成语义解析结果。在CFQ基准上，该方法以仅约1%的训练数据作为候选示例，就将平均准确率提升至95%，相比此前最优方法降低了约45%的错误率，刷新了最先进水平；在COGS上也达到99.2%的泛化准确率，与专门定制的模型持平。这项工作表明，通过巧妙设计提示策略而无需修改模型结构或大量标注数据，大语言模型完全可以在复杂的真实语义解析任务中实现强大的组合泛化，为知识密集型自然语言处理应用提供了一条极具潜力的新路径。
原文 arXiv:2209.15003;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2209.15003v2