Importance of Search and Evaluation Strategies in Neural Dialogue Modeling
Ilia Kulikov New York University、Alexander H. Miller Facebook AI Research、Kyunghyun Cho New York University Facebook AI Research CIFAR Azrieli Global Scholar、Jason Weston Facebook AI Research New York University
Abstract
We investigate the impact of search strategies in neural dialogue modeling. We first compare two standard search algorithms, greedy and beam search, as well as our newly proposed iterative beam search which produces a more diverse set of candidate responses. We evaluate these strategies in realistic full conversations with humans and propose a model-based Bayesian calibration to address annotator bias. These conversations are analyzed using two automatic metrics: log-probabilities assigned by the model and utterance diversity. Our experiments reveal that better search algorithms lead to higher rated conversations. However, finding the optimal selection mechanism to choose from a more diverse set of candidates is still an open question.
中文速览
对话系统生成回复时，搜索算法的选择往往被忽视，大多数研究只关注模型结构或训练方式。这项工作专门研究搜索策略对神经对话模型（neural dialogue model）生成质量的影响，比较了贪心搜索（greedy search）、束搜索（beam search）以及作者新提出的迭代束搜索（iterative beam search）——后者通过多次运行束搜索并排除已探索空间，生成更多样化的候选回复。实验采用真实多轮人机对话进行人工评测，并引入基于贝叶斯校准（Bayesian calibration）的方法消除标注者个人偏差的干扰，同时结合对数概率和回复多样性两类自动指标综合评估。结果显示，更好的搜索算法确实能让模型生成更符合自身偏好、也更多样的回复，但自动指标排名最好的策略并不总是人工评测中最受欢迎的，这表明搜索策略的改进和对话质量的可靠评估都是值得持续关注的开放性问题。
原文 arXiv:1811.00907;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.00907v3