On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translation
Tamer Alkhouli, Gabriel Bretschner, and Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany
Abstract
This work investigates the alignment problem in state-of-the-art multi-head attention models based on the transformer architecture. We demonstrate that alignment extraction in transformer models can be improved by augmenting an additional alignment head to the multi-head source-to-target attention component. This is used to compute sharper attention weights. We describe how to use the alignment head to achieve competitive performance. To study the effect of adding the alignment head, we simulate a dictionary-guided translation task, where the user wants to guide translation using pre-defined dictionary entries. Using the proposed approach, we achieve up to $3.8$ % BLEU improvement when using the dictionary, in comparison to $2.4$ % BLEU in the baseline case. We also propose alignment pruning to speed up decoding in alignment-based neural machine translation (ANMT), which speeds up translation by a factor of $1.8$ without loss in translation performance. We carry out experiments on the shared WMT 2016 English $\to$ Romanian news task and the BOLT Chinese $\to$ English discussion forum task.
中文速览
在神经机器翻译领域，如何从以Transformer为代表的多头注意力模型中提取准确的词对齐信息，一直是个棘手的难题，而这种对齐能力在"按词典强制翻译特定词汇"等实际应用场景中至关重要。本文提出在Transformer的多头注意力模块中额外插入一个专用"对齐头"（alignment head），将预先计算好的硬对齐作为一个独热分布直接输入模型，并配合两阶段训练策略（先训练基线再加入对齐头微调），使模型既保留原有翻译质量，又具备精确的对齐感知能力。实验还引入了对齐剪枝技术加速解码，在不损失翻译质量的前提下将解码速度提升了1.8倍；在词典引导翻译任务上，所提方法相比无对齐基线将BLEU提升从2.4%进一步扩大到3.8%。这项工作证明了显式对齐与强大注意力机制可以兼得，为需要可控、可干预翻译输出的实际应用场景提供了一条切实可行的技术路径。
原文 arXiv:1809.03985;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.03985v1