KAT: A Knowledge Augmented Transformer for Vision-and-Language
Liangke Gui§‡ Borui Wang†‡ Qiuyuan Huang‡ Alex Hauptmann§ Yonatan Bisk§‡ Jianfeng Gao‡ §Carnegie Mellon University †Yale University ‡Microsoft Research {liangkeg, alex, {qihua,
Abstract
The primary focus of recent work with large-scale transformers has been on optimizing the amount of information packed into the model’s parameters. In this work, we ask a complementary question: Can multimodal transformers leverage explicit knowledge in their reasoning? Existing, primarily unimodal, methods have explored approaches under the paradigm of knowledge retrieval followed by answer prediction, but leave open questions about the quality and relevance of the retrieved knowledge used, and how the reasoning processes over implicit and explicit knowledge should be integrated. To address these challenges, we propose a - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result (+6% absolute) on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. Additionally, explicit knowledge integration improves interpretability of model predictions in our analysis. Code and pre-trained models are released at https://github.com/guilk/KAT.
中文速览
大规模视觉问答系统面对需要外部知识才能回答的问题时常常力不从心——仅靠模型参数里"记住"的隐性知识远远不够，而直接用关键词检索回来的显性知识又往往太泛、太嘈杂。为此，研究者提出了知识增强Transformer（Knowledge Augmented Transformer，KAT），同时整合两类知识：显性知识方面，用基于对比学习的CLIP模型把图像局部区域与Wikidata实体精准对齐，检索出与视觉内容高度相关的知识条目；隐性知识方面，设计专门的提示词驱动冻结的GPT-3生成候选答案及其背后的支撑证据。两类知识被统一送入编码器-解码器架构，让模型在生成答案时对两者同步推理，而非简单拼接。在开放域多模态问答基准OK-VQA上，KAT将最优结果提升了6个百分点，同时检索到的显性知识还让模型的预测过程更加可解释，为构建真正能"查阅外部知识"的多模态智能系统提供了一条清晰的路径。
原文 arXiv:2112.08614;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.08614v2