Classical Structured Prediction Losses for Sequence to Sequence Learning
Sergey Edunov , Myle Ott††, Michael Auli, David Grangier, Marc’Aurelio Ranzato Facebook AI Research Menlo Park, CA Equal contribution. New York, NY
Abstract
There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam. In this paper, we survey a range of classical objective functions that have been widely used to train linear models for structured prediction and apply them to neural sequence to sequence models. Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup. We also report new state of the art results on both IWSLT’14 German-English translation as well as Gigaword abstractive summarization. On the large WMT’14 English-French task, sequence-level training achieves 41.5 BLEU which is on par with the state of the art.111An implementation of the losses is available as part of fairseq at: https://github.com/pytorch/fairseq/tree/classic_seqlevel
中文速览
神经序列到序列模型（neural sequence-to-sequence）通常只在词级别做最大似然训练，但测试时输出的是整个句子，二者存在不一致，因此研究者开始探索序列级别的训练方法。这篇论文把经典线性结构预测（structured prediction）中广泛使用的目标函数——包括序列负对数似然、期望风险最小化（expected risk minimization）、最大间隔损失（max-margin）等——系统地移植到神经序列到序列模型上，并与近年流行的强化学习和束搜索优化方法进行全面比较。实验结果表明，这些"老"损失函数表现出人意料地好，在同等设置下略微超过束搜索优化方法，并在IWSLT'14德英翻译和Gigaword摘要任务上刷新了当时最优结果，在大规模WMT'14英法翻译上也达到41.5 BLEU的顶尖水平。这项工作提醒研究者，不必急着追新方法，经过充分验证的经典结构化预测目标函数在神经网络时代依然极具竞争力，值得作为强基线。
原文 arXiv:1711.04956;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.04956v5