Unsupervised Commonsense Question Answering with Self-Talk
Vered Shwartz1,2, Peter West1,2, Ronan Le Bras1, Chandra Bhagavatula1, and Yejin Choi1,2 1Allen Institute for Artificial Intelligence 2Paul G. Allen School of Computer Science、Engineering, University of Washington
Abstract
Natural language understanding involves reading between the lines with implicit background knowledge. Current systems either rely on pre-trained language models as the sole implicit source of world knowledge, or resort to external knowledge bases (KBs) to incorporate additional relevant knowledge. We propose an unsupervised framework based on self-talk as a novel alternative to multiple-choice commonsense tasks. Inspired by inquiry-based discovery learning Bruner (1961), our approach inquires language models with a number of information seeking questions such as “what is the definition of …” to discover additional background knowledge. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zero-shot language model baselines on four out of six commonsense benchmarks, and competes with models that obtain knowledge from external KBs. While our approach improves performance on several benchmarks, the self-talk induced knowledge even when leading to correct answers is not always seen as helpful by human judges, raising interesting questions about the inner-workings of pre-trained language models for commonsense reasoning.
中文速览
大语言模型在做常识推理时，往往只能靠自身"记住"的知识来答题，一旦遇到报告偏差、否定表达或多步推理就容易出错，而引入外部知识库又需要额外的标注和对齐工作。为此，研究者提出了一种无监督的"自我对话"（self-talk）框架：让语言模型先针对题目上下文自动生成若干澄清性问题（如"这件事的目的是什么"），再让模型回答这些问题，把生成的答案作为补充背景信息拼入原始输入，最终用另一个语言模型为选项打分并选出最优答案。在六个常识推理基准上的实验表明，该方法在其中四个任务上显著超越纯零样本语言模型基线，并可与依赖外部知识库的模型持平甚至更优。不过，人工评估发现，即便自我对话生成的信息帮助模型答对了题目，人类评判者也常常认为这些信息没有实际帮助甚至是错误的，这揭示了预训练语言模型在常识推理中"答对但理由可疑"的深层问题，呼吁未来研究更鲁棒、更可解释的知识融合方式。
原文 arXiv:2004.05483;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.05483v2