How Optimal is Greedy Decoding for Extractive Question Answering?
\nameOr Castel1 \nameOri Ram1 \nameAvia Efrat1 \nameOmer Levy1,2 \addr1Blavatnik School of Computer Science, Tel Aviv University 2Meta AI Research
Abstract
Fine-tuned language models use greedy decoding to answer reading comprehension questions with relative success. However, this approach does not ensure that the answer is a span in the given passage, nor does it guarantee that it is the most probable one. Does greedy decoding actually perform worse than an algorithm that does adhere to these properties? To study the performance and optimality of greedy decoding, we present exact-extract, a decoding algorithm that efficiently finds the most probable answer span in the passage. We compare the performance of T5 with both decoding algorithms on zero-shot and few-shot extractive question answering. When no training examples are available, exact-extract significantly outperforms greedy decoding. However, greedy decoding quickly converges towards the performance of exact-extract with the introduction of a few training examples, becoming more extractive and increasingly likelier to generate the most probable span as the training set grows. We also show that self-supervised training can bias the model towards extractive behavior, increasing performance in the zero-shot setting without resorting to annotated examples. Overall, our results sug
中文速览
抽取式问答（Extractive QA）要求模型从给定段落中找出一段文字作为答案，但主流生成模型（如T5）惯用的贪婪解码（greedy decoding）既不保证答案来自原文，也不保证是概率最高的那个片段。为此，研究者设计了一种名为"exact-extract"的解码算法，利用动态规划高效枚举段落中所有可能的答案跨度，并选出模型赋予概率最高的那一个，从而在理论上实现最优的抽取式解码。实验在涵盖8个数据集、从零样本到1024个训练样本的少样本基准上对比两种算法，发现零样本时贪婪解码平均落后exact-extract约11.3个F1分，而仅用16个标注样本就能将差距缩小到2.8分，用1024个样本后差距几乎消失（仅0.3分）。这说明预训练语言模型适应抽取式问答的能力极强，少量示例就足以让贪婪解码"自发"学会从原文中提取答案，接近理论最优策略，这对实际部署中解码方式的选择具有重要指导意义。
原文 arXiv:2108.05857;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2108.05857v2