Studying Strategically: Learning to Mask for Closed-book QA
Qinyuan Ye1† Belinda Z. Li2‡ Sinong Wang3 Benjamin Bolte3 Hao Ma3 Wen-tau Yih3 Xiang Ren1 Madian Khabsa3 1University of Southern California 2MIT CSAIL 3Facebook AI
Abstract
Closed-book question-answering (QA) is a challenging task that requires a model to directly answer questions without access to external knowledge. It has been shown that directly fine-tuning pre-trained language models with (question, answer) examples yields surprisingly competitive performance, which is further improved upon through adding an intermediate pre-training stage between general pre-training and fine-tuning. Prior work used a heuristic during this intermediate stage, whereby named entities and dates are masked, and the model is trained to recover these tokens. In this paper, we aim to learn the optimal masking strategy for the intermediate pre-training stage. We first train our masking policy to extract spans that are likely to be tested, using supervision from the downstream task itself, then deploy the learned policy during intermediate pre-training. Thus, our policy packs task-relevant knowledge into the parameters of a language model. Our approach is particularly effective on TriviaQA, outperforming strong heuristics when used to pre-train BART.
中文速览
让语言模型在不查阅任何外部资料的情况下直接回答问题（即"闭卷问答"），关键在于如何在预训练阶段把更多有用的知识"压进"模型参数。以往的做法是用启发式规则把文本中的命名实体和日期遮掉再让模型还原，本文转而用一个轻量级的双向LSTM策略网络，从问答数据集的（上下文、答案）对中学习"哪些文字片段最可能被拿来出题"，再把这个学到的遮掩策略用于中间预训练阶段，让语言模型集中记住与任务相关的知识。实验表明，在TriviaQA上，使用学习到的策略对BART进行中间预训练，比强启发式基线（显著命名实体遮掩）的精确匹配分数高出约1个百分点，甚至让参数更少的BART-base超越了更大的BART-large，相当于用更聪明的训练方式替代了盲目堆参数。这项工作说明，与其问"能往模型里塞多少知识"，不如先想清楚"该往里塞什么知识"，为高效利用预训练资源提供了新思路。
原文 arXiv:2012.15856;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.15856v2