HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization
Xingxing Zhang, Furu Wei Ming Zhou Microsoft Research Asia, Beijing, China
Abstract
Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder with these inaccurate labels is challenging. Inspired by the recent work on pre-training transformer sentence encoders Devlin et al. (2018), we propose Hibert (as shorthand for HIerachical Bidirectional Encoder Representations from Transformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained Hibert to our summarization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets.
中文速览
抽取式摘要模型依赖分层编码器来理解文档，但训练标签只能靠规则方法启发式生成，准确性有限，导致复杂模型很快过拟合、训练数据利用不充分。受BERT预训练思路的启发，作者提出了HIBERT——一种基于层级双向Transformer的文档编码器，并设计了一套无监督预训练方法：随机遮盖文档中15%的句子，让模型利用上下文句子来预测这些被遮盖的句子，从而在大规模无标注数据上学习深层的文档表示。将预训练好的HIBERT用于抽取式摘要任务后，在CNN/DailyMail数据集上ROUGE分数比随机初始化的对照模型提升1.25，在纽约时报数据集上提升2.0，并在两个数据集上均达到当时最优水平。这项工作表明，将预训练扩展到文档级别的层级编码器，是缓解摘要任务标签噪声、充分挖掘无标注文本价值的有效途径。
原文 arXiv:1905.06566;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.06566v1