Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
Yichong Xu1, Xiaodong Liu2, Yelong Shen2, Jingjing Liu2 and Jianfeng Gao2 1 Carnegie Mellon University 2 Microsoft Research {xiaodl, yeshen, Most of this work was performed when the author was interning at Microsoft.
Abstract
We propose a multi-task learning framework to learn a joint Machine Reading Comprehension (MRC) model that can be applied to a wide range of MRC tasks in different domains. Inspired by recent ideas of data selection in machine translation, we develop a novel sample re-weighting scheme to assign sample-specific weights to the loss. Empirical study shows that our approach can be applied to many existing MRC models. Combined with contextual representations from pre-trained language models (such as ELMo), we achieve new state-of-the-art results on a set of MRC benchmark datasets. We release our code at https://github.com/xycforgithub/MultiTask-MRC.
中文速览
机器阅读理解（Machine Reading Comprehension, MRC）模型通常在单一数据集上训练，泛化能力有限，而简单地堆叠多个任务数据效果也不理想。研究者提出了一套多任务学习框架，核心创新是一种样本重加权机制——借鉴机器翻译中的数据选择思路，用语言模型为辅助任务中每条样本打分，衡量其与目标任务的相关性，从而在训练时自动分配权重，让模型更专注于"有用"的外部数据。在SQuAD、NewsQA、Who-Did-What等四个跨领域基准上的实验表明，该方法在单任务基线上取得了显著提升，结合ELMo等预训练上下文表示后更在NewsQA上超越人类表现，达到新的最优水平。这项工作的意义在于：它是首个系统探索MRC多任务学习的框架，训练灵活、无需额外大型预训练模型，且能与现有嵌入方法叠加使用，为构建通用阅读理解模型提供了一条实用路径。
原文 arXiv:1809.06963;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.06963v3