Primera: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Wen Xiao Iz Beltagy Giuseppe Carenini Arman Cohan University of British Columbia, Vancouver, Canada Allen Institute for AI, Seattle, WA, USA Paul G. Allen School of Computer Science、Engineering, University of Washington Work mainly done during an internship at AI2.
Abstract
We introduce Primera, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. Primera uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, Primera outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins.111The code and pre-trained models can be found at https://github.com/allenai/PRIMER
中文速览
多文档摘要（multi-document summarization）长期依赖针对特定数据集定制的复杂架构，且需要大量标注数据，Primera 正是为打破这一局面而生。研究者提出了一种名为"实体金字塔"（Entity Pyramid）的预训练掩码策略——借鉴摘要评估中的 Pyramid 方法，优先选取在多篇文档中高频出现的命名实体所在句子进行掩码，迫使模型学会跨文档聚合关键信息；同时，将所有输入文档拼接后用支持长序列的 Longformer 编码器-解码器处理，并在文档边界处添加全局注意力的分隔符，结构简洁却有效。在新闻、维基百科、科学文献共 6 个多文档摘要数据集上的实验表明，Primera 在零样本、少样本（少至 10 条样例即可领先高达 5 个 ROUGE-1 分）和全监督三种设置下均大幅超越现有最优模型。这项工作的意义在于，它提供了一个通用的预训练多文档摘要基座模型，大幅降低了下游任务对标注数据和定制化架构的依赖，具有很强的实用价值。
原文 arXiv:2110.08499;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.08499v2