RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL
Jiexing Qi1, Jingyao Tang1, Ziwei He1, Xiangpeng Wan2, Yu Cheng3 Chenghu Zhou4, Xinbing Wang1, Quanshi Zhang1, Zhouhan Lin1 1Shanghai Jiao Tong University, Shanghai, China 2NetMind.AI and ProtagoLabs, Virginia, USA 3Microsoft Research, Redmond, Washington, USA 4IGSNRR, Chinese Academy of Sciences, Beijing, China {qi_jiexing, monstar, ziwei.he, zqs1022, Zhouhan Lin is the corresponding author.
Abstract
Relational structures such as schema linking and schema encoding have been validated as a key component to qualitatively translating natural language into SQL queries. However, introducing these structural relations comes with prices: they often result in a specialized model structure, which largely prohibits using large pretrained models in text-to-SQL. To address this problem, we propose RASAT: a Transformer seq2seq architecture augmented with relation-aware self-attention that could leverage a variety of relational structures while inheriting the pretrained parameters from the T5 model effectively. Our model can incorporate almost all types of existing relations in the literature, and in addition, we propose introducing co-reference relations for the multi-turn scenario. Experimental results on three widely used text-to-SQL datasets, covering both single-turn and multi-turn scenarios, have shown that RASAT could achieve state-of-the-art results across all three benchmarks (75.5% EX on Spider, 52.6% IEX on SParC, and 37.4% IEX on CoSQL). 111Our implementation is available at https://github.com/LUMIA-group/rasat.
中文速览
自然语言转SQL（text-to-SQL）领域的研究表明，把数据库结构关系（如模式链接 schema linking、模式编码 schema encoding）引入模型能显著提升查询质量，但这类方法通常需要定制化的模型结构，导致无法直接继承 T5 等大型预训练模型的参数红利。为此，研究者提出了 RASAT，其核心思路是在 T5 的编码器自注意力层中注入关系感知机制（relation-aware self-attention），将模式编码、模式链接、句法依存、数据库内容匹配以及多轮对话中新引入的共指关系（co-reference）统一编码为关系三元组，同时保留 T5 原有预训练参数，新增参数量不足原模型的 0.01%。在三个主流基准上的实验结果显示，RASAT 在单轮场景 Spider 上达到 75.5% 执行准确率，在多轮场景 SParC 和 CoSQL 上分别以 52.6% 和 37.4% 的交互执行准确率大幅刷新此前最优水平（绝对提升分别约 31 和 29 个百分点），证明了在保持预训练模型结构基本不变的前提下引入关系结构是一条兼顾效果与通用性的有效路径。
原文 arXiv:2205.06983;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.06983v2