Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons
Margaret Li Facebook AI Research、Jason Weston Facebook AI Research、Stephen Roller Facebook AI Research
Abstract
While dialogue remains an important end-goal of natural language research, the difficulty of evaluation is an oft-quoted reason why it remains troublesome to make real progress towards its solution. Evaluation difficulties are actually two-fold: not only do automatic metrics not correlate well with human judgments, but also human judgments themselves are in fact difficult to measure. The two most used human judgment tests, single-turn pairwise evaluation and multi-turn Likert scores, both have serious flaws as we discuss in this work.
中文速览
对话系统的评测一直是个老大难问题：自动指标（如BLEU）与人类判断相关性差，而现有的两种主流人工评测方式——单轮对比评测和多轮Likert打分——也各有硬伤，前者忽略多轮对话的连贯性，后者因评测者偏差大、锚定效应强，往往无法得出统计显著的结论。为此，本文提出了ACUTE-Eval这一新评测方案：让标注者同时看两段完整的多轮对话，只关注其中一方的发言，并就"哪位说话者更吸引人/更像真人"等精心措辞的问题做出二选一的判断；研究者还系统优化了问题措辞以最大化标注者一致性，并验证了用模型"自说自话"生成的对话日志替代人机对话日志的可行性，大幅降低了测试成本。在PersonaChat和Wizard of Wikipedia两个任务上的实验表明，该方法能检测出多轮Likert评测无法发现的模型间细微但显著的差异，且自说自话评测结果与真实人机对话评测结论高度一致。这项工作为对话研究社区提供了一套更快、更便宜、更可复现的标准化人工评测流程，有望成为领域内的事实标准。
原文 arXiv:1909.03087;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.03087v1