Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model
Yixuan Qiao1 Hao Chen1 Jun Wang1,2 Shanshan Zhao2 Yihao Chen2 Xianbin Ye3 Ziliang Li4 Xianbiao Qi2 Peng Gao1 Guotong Xie1,5,6 1 SFE Deeplearning Platform Ping An Health Technology Beijing China. 2 Peking University Beijing China. 3 Visual Computing Group Ping An Property、Casualty Insurance Company Shenzhen China. 4 Jinan University Guangzhou China. 5 Central University of Finance and Economics Beijing China. 6 Ping An Health Cloud Company Limited Shenzhen China. 7 Ping An International Smart City Technology Co Ltd Shenzhen China. {qiaoyixuan528, chenhao305, zhaoshanshan233, wangjun916, chenyihao291, {gaopeng712,
Abstract
TextVQA requires models to read and reason about text in images to answer questions about them. Specifically, models need to incorporate a new modality of text present in the images and reason over it to answer TextVQA questions. In this challenge, we use generative model T5 for TextVQA task. Based on pre-trained checkpoint T5-3B from HuggingFace repository, two other pre-training tasks including masked language modeling(MLM) and relative position prediction(RPP) are designed to better align object feature and scene text. In the stage of pre-training, encoder is dedicate to handle the fusion among multiple modalities: question text, object text labels, scene text labels, object visual features, scene visual features. After that decoder generates the text sequence step-by-step, cross entropy loss is required by default. We use a large-scale scene text dataset in pre-training and then fine-tune the T5-3B with the TextVQA dataset only.
中文速览
看图回答问题（TextVQA）要求模型识别图中的文字并结合视觉内容进行推理，难点在于如何让模型同时理解图像、目标区域文字和场景文字三种模态。研究者以 T5-3B 为基础模型，构建了约 900 万张场景文字图像用于预训练，并设计了掩码语言建模（MLM）和相对位置预测（RPP）两个辅助任务，同时引入对抗训练与 KL 散度正则化，让多模态特征在嵌入空间中对齐得更加紧密。微调阶段在 TextVQA 数据集上逐步解码答案，最后还用模糊字符串匹配对输出做后处理纠错。该方案在 TextVQA 2021 竞赛中取得了有竞争力的成绩，验证了大规模场景文字预训练、强目标检测特征以及生成式语言模型相结合这一路线的有效性，为后续多模态问答研究提供了实用参考。
原文 arXiv:2106.15332;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.15332v2