GLGE: A New General Language Generation Evaluation Benchmark
Dayiheng Liu♠♠{}^{\spadesuit}{\,} Yu Yan♡†{}^{{\dagger}}{\,}^{\heartsuit} Yeyun Gong♣†{}^{{\dagger}}{\,}^{\clubsuit} Weizhen Qi♣ Hang Zhang♠ Jian Jiao♡ Weizhu Chen‡ Jie Fu♢ Linjun Shou\vardiamondsuit\vardiamondsuit{}^{\vardiamondsuit} Ming Gong\vardiamondsuit\vardiamondsuit{}^{\vardiamondsuit} Pengcheng Wang♡ Jiusheng Chen♡ Daxin Jiang\vardiamondsuit\vardiamondsuit{}^{\vardiamondsuit} Jiancheng Lv♠ Ruofei Zhang♡ Winnie Wu♡ Ming Zhou♣ Nan Duan♣ ♠College of Computer Science, Sichuan University ♣Microsoft Research Asia ♢ Mila ‡Microsoft Azure AI \vardiamondsuit\vardiamondsuit{}^{\vardiamondsuit}Microsoft Search Technology Center Asia ♡Microsoft †Equal contribution Work is done during internship at Microsoft Research Asia.
Abstract
Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP). These benchmarks mostly focus on a range of Natural Language Understanding (NLU) tasks, without considering the Natural Language Generation (NLG) models. In this paper, we present the General Language Generation Evaluation (GLGE), a new multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. For each task, we continue to design three subtasks in terms of task difficulty (GLGE-Easy, GLGE-Medium, and GLGE-Hard). This introduces 24 subtasks to comprehensively compare model performance. To encourage research on pretraining and transfer learning on NLG models, we make GLGE publicly available and build a leaderboard with strong baselines including MASS, BART, and ProphetNet111The source code and dataset are publicly available at https://github.com/microsoft/glge..
中文速览
NLG（自然语言生成）领域长期缺乏一个像GLUE那样统一、全面的多任务评测基准，导致各模型在不同数据集、不同指标上单打独斗、难以横向比较。为此，研究者推出了GLGE（General Language Generation Evaluation）基准，涵盖文本摘要、问题生成、生成式问答和个性化对话共8个英文生成任务，并通过"低资源"和"低词频"两种策略将每个任务拆分为简单、中等、困难三档，共构成24个子任务。在此基准上，作者以MASS、BART、ProphetNet等主流预训练模型和非预训练基线（LSTM Seq2Seq、Transformer）进行系统对比实验，结果表明预训练模型整体领先优势明显，但在Hard难度级别上仍存在较大提升空间。GLGE的公开发布及配套排行榜，为NLG预训练与迁移学习的标准化评测提供了重要基础设施，有助于推动该领域更系统、更可比的研究进展。
原文 arXiv:2011.11928;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.11928v3