MaXM: Towards Multilingual Visual Question Answering
Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish V. Thapliyal Idan Szpektor, Julien Amelot, Xi Chen, Radu Soricut Google Research https://github.com/google-research-datasets/maxm
Abstract
Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require a considerable amount of resources. In this paper, we propose scalable solutions to multilingual visual question answering (mVQA), on both data and modeling fronts. We first propose a translation-based framework to mVQA data generation that requires much less human annotation efforts than the conventional approach of directly collection questions and answers. Then, we apply our framework to the multilingual captions in the Crossmodal-3600 dataset and develop an efficient annotation protocol to create $\mathrm{MaXM}$ , a test-only VQA benchmark in 7 diverse languages. Finally, we develop a simple, lightweight, and effective approach as well as benchmark state-of-the-art English and multilingual VQA models. We hope that our benchmark encourages further research on mVQA.
中文速览
多语言视觉问答（mVQA）长期被英语主导，根本原因是缺乏其他语言的高质量标注数据。为此，研究者提出了一套基于翻译的自动化数据生成框架 TransVQ²A：先把多语言图像描述翻译成英语，用已有的英语问答生成工具产出问答对，再将其翻译回目标语言，最后通过回译验证过滤噪声，从而大幅降低人工标注成本。在此基础上，他们对 Crossmodal-3600 数据集中7种类型多样的语言（英、法、印地、希伯来、罗马尼亚、泰、中）进行标注，构建了名为 MaXM 的多语言 VQA 测试基准，并提出了一个基于 ViT 和 mT5 的轻量级统一模型 Simple MPT 作为基线。实验表明，当前最优模型在该基准上仍有很大提升空间，这一数据集和框架为推动多语言视觉问答研究提供了重要的基础资源。
原文 arXiv:2209.05401;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2209.05401v3