Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering
Jiacheng Liu♡ Skyler Hallinan♡ Ximing Lu♡♠ Pengfei He♡ Sean Welleck♡♠ Hannaneh Hajishirzi♡♠ Yejin Choi♡♠ ♡Paul G. Allen School of Computer Science、Engineering, University of Washington ♠Allen Institute for Artificial Intelligence
Abstract
Knowledge underpins reasoning. Recent research demonstrates that when relevant knowledge is provided as additional context to commonsense question answering (QA), it can substantially enhance the performance even on top of state-of-the-art. The fundamental challenge is where and how to find such knowledge that is high quality and on point with respect to the question; knowledge retrieved from knowledge bases are incomplete and knowledge generated from language models are inconsistent.
中文速览
为了让语言模型在常识问答中表现更好，研究者通常需要为每道题提供相关背景知识，但从知识库检索往往覆盖不全，从GPT-3生成质量又不稳定且代价高昂。这项工作提出了Rainier（强化知识内省器），一个规模远小于GPT-3的生成模型，先通过模仿GPT-3输出的知识打好基础，再用强化学习（proximal policy optimization, PPO）以"生成的知识能否帮助问答模型答对题目"作为奖励信号，自主学习生成更有针对性的常识知识。在9个常识问答基准上的实验表明，Rainier带来的提升幅度超过了参数量是它16倍的GPT-3，即便在训练时从未见过的4个数据集上也保持稳定增益，且生成的知识在领域、关系类型和语法属性上具有丰富多样性。这项工作首次证明，一个中等规模的模型无需对知识本身做直接标注监督，也能产出质量超越GPT-3的常识知识，为以"模型生成知识"替代单纯堆砌模型参数提供了新的可行路径。
原文 arXiv:2210.03078;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.03078v2