PreSTU: Pre-Training for Scene-Text Understanding
Jihyung Kil Soravit Changpinyo2 Xi Chen2 Hexiang Hu2 Sebastian Goodman2 Wei-Lun Chao1 Radu Soricut2 1The Ohio State University 2Google Research Work done at Google Research.
Abstract
The ability to recognize and reason about text embedded in visual inputs is often lacking in vision-and-language (V&L) models, perhaps because V&L pre-training methods have often failed to include such an ability in their training objective. In this paper, we propose PreSTU, a novel pre-training recipe dedicated to scene-text understanding (STU). PreSTU introduces OCR-aware pre-training objectives that encourage the model to recognize text from an image and connect it to the rest of the image content. We implement PreSTU using a simple transformer-based encoder-decoder architecture, combined with large-scale image-text datasets with scene text obtained from an off-the-shelf OCR system. We empirically demonstrate the effectiveness of this pre-training approach on eight visual question answering and four image captioning benchmarks.
中文速览
现有的视觉-语言模型普遍不擅长理解图片中出现的文字（如路牌、票据、截图里的文本），根本原因在于预训练阶段从未专门设计过让模型"看懂图中文字"的学习目标。为此，研究者提出了 PreSTU 这套专为场景文字理解（Scene-Text Understanding）设计的预训练方案：核心思路是在一个基于 ViT 图像编码器与 mT5 语言编解码器的端到端框架里，引入 OCR 感知目标——让模型根据图像像素和部分 OCR 文字去预测剩余文字（SplitOCR），同时辅以视觉问答和图像描述的任务特定目标，迫使模型同时学会"识别图中文字"和"把文字与图像内容联系起来"这两项核心能力。在 8 个视觉问答和 4 个图像描述基准上的实验显示，PreSTU 相比不含此类预训练的强基线在 TextVQA 上提升超过 10 个百分点、在 TextCaps 上提升 42 个 CIDEr 分，甚至在下游任务不提供 OCR 输入的情况下仍保持明显优势。这项工作表明，只需借助通用图文数据和现成 OCR 工具，就能以低成本、高通用性的方式大幅增强视觉-语言模型对图中文字的理解能力。
原文 arXiv:2209.05534;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2209.05534v3