DREAM: Improving Situational QA by First Elaborating the Situation
Yuling Gu , Bhavana Dalvi Mishra, Peter Clark Allen Institute for AI, Seattle, WA First author was a student at University of Washington at the time of submission of this paper. This is work done during an internship at Allen Institute for AI.
Abstract
When people answer questions about a specific situation, e.g., “I cheated on my mid-term exam last week. Was that wrong?”, cognitive science suggests that they form a mental picture of that situation before answering. While we do not know how language models (LMs) answer such questions, we conjecture that they may answer more accurately if they are also provided with additional details about the question situation, elaborating the “scene”. To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then provide those elaborations as additional context to a question-answering (QA) model. We find that DREAM is able to create better scene elaborations (more accurate, useful, and consistent) than a representative state-of-the-art, zero-shot model (Macaw). We also find that using the scene elaborations as additional context improves the answer accuracy of a downstream QA system, including beyond that obtainable by simply further fine-tuning the QA system on DREAM’s training data. These results suggest that adding focused elaborations about a situation can improve a system’s reasoning about it, and may serve a
中文速览
语言模型在回答"我上周考试作弊，这样做错了吗？"这类具体情境问题时，往往因为缺乏对场景的充分理解而给出错误答案。受认知科学中"心理模型"概念的启发，研究者提出：如果在提问前先为模型补充关于该场景的细节——包括人物动机、情绪、社会规范和行为后果——能否帮助模型答得更准？为此，他们训练了一个名为 DREAM 的新模型，利用现有常识知识库自动生成这类"场景细化"信息，再将其作为额外上下文输入到下游问答系统。实验表明，DREAM 生成的场景细化比零样本基线模型 Macaw 更准确、更有用、更一致，并且将其加入上下文后，下游问答准确率得到了显著提升，效果甚至超过直接用 DREAM 的训练数据对问答模型进行微调。这项工作的意义在于，它提供了一种无需改动问答模型本身、就能向其注入新情境知识的模块化方法，且对不同规模的模型均有效，为提升机器对社会情境的推理能力开辟了新思路。
原文 arXiv:2112.08656;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.08656v3