Towards Decoding as Continuous Optimisation in Neural Machine Translation
Cong Duy Vu Hoang University of Melbourne Melbourne, VIC, Australia \ANDGholamreza Haffari Monash University Clayton, VIC, Australia、Trevor Cohn University of Melbourne Melbourne, VIC, Australia
Abstract
We propose a novel decoding approach for neural machine translation (NMT) based on continuous optimisation. We convert decoding – basically a discrete optimization problem – into a continuous optimization problem. The resulting constrained continuous optimisation problem is then tackled using gradient-based methods. Our powerful decoding framework enables decoding intractable models such as the intersection of left-to-right and right-to-left (bidirectional) as well as source-to-target and target-to-source (bilingual) NMT models. Our empirical results show that our decoding framework is effective, and leads to substantial improvements in translations generated from the intersected models where the typical greedy or beam search is not feasible. We also compare our framework against reranking, and analyse its advantages and disadvantages.
中文速览
神经机器翻译（NMT）的解码过程本质上是一个离散优化问题，传统的贪婪搜索和束搜索在面对需要同时考虑多个模型的复杂场景时力不从心——比如把从左到右和从右到左两个方向的翻译模型"交叉"起来联合解码，这对顺序搜索方法来说根本无从下手。这项工作提出把离散解码问题松弛（relax）成连续优化问题：用词汇表上的概率分布替代离散的词选择，再借助随机梯度下降（SGD）和指数梯度（EG）等梯度方法在概率单纯形上迭代求解，最后把结果"取整"回具体词序列。基于这一框架，研究者实现了双向模型（左到右与右到左）以及双语模型（源到目标与目标到源）的联合解码，在中英和德英翻译任务上取得了显著的BLEU提升，而这些改进是传统搜索算法无法实现的。这项工作为NMT解码开辟了一条新路径，使那些原本因搜索空间不兼容而"无法解码"的强力组合模型变得可用，对未来探索更复杂的全局约束解码具有重要参考价值。
原文 arXiv:1701.02854;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1701.02854v4