Modeling Mutual Information and Promoting Diversity in Neural Machine Translation
Jiwei Li and Dan Jurafsky Computer Science Department Stanford University, Stanford, CA, 94305, USA
Abstract
Sequence-to-sequence neural translation models learn semantic and syntactic relations between sentence pairs by optimizing the likelihood of the target given the source, i.e., $p(y|x)$ , an objective that ignores other potentially useful sources of information. We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets. We implement the model with a simple re-ranking method, and also introduce a decoding algorithm that increases diversity in the N-best list produced by the first pass. Applied to the WMT German/English and French/English tasks, the proposed models offers a consistent performance boost on both standard LSTM and attention-based neural MT architectures.
中文速览
神经机器翻译模型通常只优化"给定源句生成目标句"的单向概率 p(y|x)，忽略了反向的目标到源依赖关系。这篇论文借鉴最大互信息（Maximum Mutual Information, MMI）思想，将翻译目标函数改造为同时考虑 p(y|x) 和 p(x|y) 的双向联合建模，具体做法是分别训练正向和反向两个序列到序列模型，先用正向模型生成 N-best 候选译文，再用反向模型对候选列表重新排序；为解决标准束搜索产生的候选译文高度雷同、重排序效果有限的问题，论文还专门设计了一种促进多样性的解码算法，通过对同一父节点扩展出的候选施加排名惩罚来拉开候选之间的差异。在 WMT 德英和法英翻译任务上，无论是标准 LSTM 还是带注意力机制（attention）的架构，该方法都带来了稳定的 BLEU 分数提升，表明将双向依赖信息引入神经机器翻译是一种简单有效且具有普适性的改进手段。
原文 arXiv:1601.00372;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1601.00372v2