Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Bhuwan Dhingra† Manaal Faruqui‡ Ankur Parikh‡ Ming-Wei Chang‡ Dipanjan Das‡ William W. Cohen†‡ † Carnegie Mellon University ‡ Google Research Work done during an internship at Google.
Abstract
Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio (Lebret et al., 2016), often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed in Wiseman et al. (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using the data from the WebNLG challenge.111Code and Data: http://www.cs.cmu.edu/~bdhingra/pages/parent.html
中文速览
自动构建的"表格转文本"数据集（如WikiBio）里，参考文本常常包含表格中根本没有的信息，导致BLEU、ROUGE等只靠参考文本打分的指标与人类判断严重脱节。为此，作者提出了新指标PARENT（Precision And Recall of Entailed N-grams from the Table），其核心思路是：计算精确率时把参考文本和原始表格的信息"取并集"，奖励那些虽未出现在参考文本但确实来自表格的正确内容；计算召回率时把两者"取交集"，忽略参考文本中表格无法支撑的多余信息，从而同时对生成文本和原始表格进行对齐评估。在对WikiBio上16个模型的1100条输出开展大规模人工评估后，PARENT与人类判断的相关性显著优于BLEU和ROUGE，与基于信息抽取的评估方法相当，但使用更简便，且在参考文本质量更高的WebNLG数据集上同样有效。这项工作为表格转文本生成任务提供了一个更可靠、更易用的自动评估工具，对数据集存在"参考文本偏移"问题的所有文本生成场景都具有参考价值。
原文 arXiv:1906.01081;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.01081v1