Calibration of Encoder Decoder Models for Neural Machine Translation
Aviral Kumar University of California Berkeley、Sunita Sarawagi Indian Institute of Technology Bombay Work done when the author was a student at IIT Bombay
Abstract
We study the calibration of several state of the art neural machine translation (NMT) systems built on attention-based encoder-decoder models. For structured outputs like in NMT, calibration is important not just for reliable confidence with predictions, but also for proper functioning of beam-search inference. We show that most modern NMT models are surprisingly miscalibrated even when conditioned on the true previous tokens. Our investigation leads to two main reasons — severe miscalibration of EOS (end of sequence marker) and suppression of attention uncertainty. We design recalibration methods based on these signals and demonstrate improved accuracy, better sequence-level calibration, and more intuitive results from beam-search.
中文速览
神经机器翻译（NMT）系统在生成翻译时会给每个词分配一个概率，理想状态下这个概率应该真实反映预测的正确率——这就是"校准（calibration）"。研究发现，当前主流的基于注意力机制的NMT模型（包括GNMT、Transformer等）普遍存在严重的校准问题，即便在训练条件与测试条件完全一致的情况下也不例外，根本原因在于序列结束标记（EOS）的概率被严重高估，以及注意力机制的不确定性被压制。针对这两点，作者设计了一套基于输入覆盖率、注意力不确定性和词概率的参数化重校准方法，无需重新训练模型，只需对已有模型做后处理即可。经过重校准后，模型的BLEU分数最多提升0.4个点，序列级别的校准质量也明显改善，同时解决了"扩大束搜索（beam search）宽度反而导致翻译质量下降"这一长期令人困惑的现象，揭示出校准对结构化预测任务的推理算法同样至关重要。
原文 arXiv:1903.00802;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1903.00802v1