Learning with Instance Bundles for Reading Comprehension
Dheeru Dua♣♣\clubsuit, Pradeep Dasigi♠♠\spadesuit, Sameer Singh♣♣\clubsuit, and Matt Gardner♠♠\spadesuit ♣♣\clubsuitUniversity of California, Irvine, USA ♠♠\spadesuitAllen Institute for Artificial Intelligence
Abstract
When training most modern reading comprehension models, all the questions associated with a context are treated as being independent from each other. However, closely related questions and their corresponding answers are not independent, and leveraging these relationships could provide a strong supervision signal to a model. Drawing on ideas from contrastive estimation, we introduce several new supervision techniques that compare question-answer scores across multiple related instances. Specifically, we normalize these scores across various neighborhoods of closely contrasting questions and/or answers, adding another cross entropy loss term that is used in addition to traditional maximum likelihood estimation. Our techniques require bundles of related question-answer pairs, which we can either mine from within existing data or create using various automated heuristics. We empirically demonstrate the effectiveness of training with instance bundles on two datasets—HotpotQA and ROPES—showing up to 11% absolute gains in accuracy.
中文速览
阅读理解模型在训练时通常把每道题当作独立样本处理，却忽视了同一篇文章下那些措辞相近、答案截然不同的问题之间的内在关联。这篇论文借鉴对比估计（contrastive estimation）的思路，提出用"实例束（instance bundle）"——即一组高度相似但关键词不同的问题-答案对——来同时训练模型，通过在这些相关样本构成的"邻域"上额外做一次交叉熵归一化，迫使模型真正学会区分细微差异，而不是对相反问题给出相同答案。作者设计了多种自动挖掘或生成实例束的启发式方法，并在HotpotQA和ROPES两个数据集上验证了效果，最高取得了11个百分点的绝对精度提升。这一结果表明，在数据收集阶段主动构建成束的对比样本、并配合对比估计训练，能为阅读理解模型提供远比传统最大似然更强的监督信号。
原文 arXiv:2104.08735;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.08735v1