Imagination improves Multimodal Translation
Desmond Elliott⋆⋄ Ákos Kádár† ⋆ILLC, University of Amsterdam ⋄School of Informatics, University of Edinburgh †TiCC, Tilburg University
Abstract
We decompose multimodal translation into two sub-tasks: learning to translate and learning visually grounded representations. In a multitask learning framework, translations are learned in an attention-based encoder-decoder, and grounded representations are learned through image representation prediction. Our approach improves translation performance compared to the state of the art on the Multi30K dataset. Furthermore, it is equally effective if we train the image prediction task on the external MS COCO dataset, and we find improvements if we train the translation model on the external News Commentary parallel text.
中文速览
多模态机器翻译（multimodal machine translation）旨在借助图像等视觉信息来提升文本翻译质量，但以往方法大多把图像直接输入解码器，效果提升有限。本文提出"Imagination"模型，将多模态翻译拆解为两个子任务：用基于注意力机制的编解码器学习翻译，同时用图像预测辅助任务让编码器学习具有视觉接地（visually grounded）的语言表示——模型在训练时"想象"句子对应的图像特征，推理时无需图像输入。这一多任务框架的关键优势在于两个子任务可分别利用外部资源训练：图像描述数据集（MS COCO）可用于图像预测任务，大规模平行语料（News Commentary）可用于翻译任务，在Multi30K英德翻译基准上最终达到59.3 Meteor的当时最优水平。该工作表明，即使翻译时看不到图像，模型也能通过辅助视觉学习目标将视觉语义融入语言表示，从而切实提升翻译质量。
原文 arXiv:1705.04350;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1705.04350v2