Answers Unite! Unsupervised Metrics for Reinforced Summarization Models
Thomas Scialom⋆‡, Sylvain Lamprier‡, Benjamin Piwowarski⋄‡, Jacopo Staiano⋆ ⋄ CNRS, France ‡ Sorbonne Université, CNRS, LIP6, F-75005 Paris, France ⋆ reciTAL, Paris, France
Abstract
Abstractive summarization approaches based on Reinforcement Learning (RL) have recently been proposed to overcome classical likelihood maximization. RL enables to consider complex, possibly non-differentiable, metrics that globally assess the quality and relevance of the generated outputs. ROUGE, the most used summarization metric, is known to suffer from bias towards lexical similarity as well as from suboptimal accounting for fluency and readability of the generated abstracts. We thus explore and propose alternative evaluation measures: the reported human-evaluation analysis shows that the proposed metrics, based on Question Answering, favorably compares to ROUGE – with the additional property of not requiring reference summaries. Training a RL-based model on these metrics leads to improvements (both in terms of human or automated metrics) over current approaches that use ROUGE as a reward.
中文速览
摘要文摘（abstractive summarization）系统长期依赖ROUGE这一指标来衡量生成质量，但ROUGE只关注词语重叠，既偏袒字面相似度，又难以反映摘要的流畅性和可读性。本文提出用基于问答（Question Answering，QA）的新指标来替代ROUGE，其核心思路是：一篇好摘要应该能帮读者回答出原文中最关键的问题；尤其值得注意的是，无监督版本的QA指标不需要人工参考摘要，直接从原始文章中生成问题与答案。人工评估对比实验表明，这些QA指标在可读性和相关性上与人类判断的相关性显著优于ROUGE；进一步将这些指标作为强化学习（Reinforcement Learning，RL）的奖励信号来训练摘要模型，无论在自动指标还是人工评估上均超越了以ROUGE为奖励的现有方法。这项工作的意义在于，它为摘要系统的训练与评估提供了一条更贴近人类阅读体验、且无需人工标注参考摘要的新路径。
原文 arXiv:1909.01610;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.01610v1