Abstract

We introduce Primera, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. Primera uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, Primera outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins.111The code and pre-trained models can be found at https://github.com/allenai/PRIMER