A Visual Attention Grounding Neural Model for Multimodal Machine Translation
Mingyang Zhou Runxiang Cheng Yong Jae Lee Zhou Yu Department of Computer Science University of California, Davis {minzhou, samcheng, yongjaelee,
Abstract
We introduce a novel multimodal machine translation model that utilizes parallel visual and textual information. Our model jointly optimizes the learning of a shared visual-language embedding and a translator. The model leverages a visual attention grounding mechanism that links the visual semantics with the corresponding textual semantics. Our approach achieves competitive state-of-the-art results on the Multi30K and the Ambiguous COCO datasets. We also collected a new multilingual multimodal product description dataset to simulate a real-world international online shopping scenario. On this dataset, our visual attention grounding model outperforms other methods by a large margin.
中文速览
多模态机器翻译（multimodal machine translation）希望借助图片来帮助模型消除文字歧义、提升翻译质量，但如何让视觉信息真正"嵌入"翻译过程一直是难题。研究者提出了VAG-NMT模型，核心思路是用一个视觉注意力机制（visual attention grounding）计算出每个源语言词与图片的语义相关度，再把这一加权向量投影到图文共享语义空间，并用它来初始化翻译解码器，使图像语义对翻译的影响落到最关键的词上；整个模型通过多任务学习同时优化翻译损失和图文排序损失。在公认基准数据集Multi30K和Ambiguous COCO上，VAG-NMT达到了与当时最优方法相当的水平；研究者还自建了来自IKEA和UNIQLO电商网站的多语言商品描述数据集，在该数据集上模型明显优于其他方法。这项工作的价值在于提供了一种更有效地将视觉语义融入翻译的机制，同时引入了更贴近真实商业场景的评测数据，推动了多模态翻译从图片描述任务走向实际应用。
原文 arXiv:1808.08266;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1808.08266v2