Answer-Me: Multi-Task Learning for Generalization to Many Question-Answering Tasks
AJ Piergiovanni Wei Li Weicheng Kuo Mohammad Saffar Fred Bertsch and Anelia Angelova
Abstract
We present Answer-Me, a task-aware multi-task framework which unifies a variety of question answering tasks, such as, visual question answering, visual entailment, visual reasoning. In contrast to previous works using contrastive or generative captioning training, we propose a novel and simple recipe to pre-train a vision-language joint model, which is multi-task as well. The pre-training uses only noisy image captioning data, and is formulated to use the entire architecture end-to-end with both a strong language encoder and decoder. Our results show state-of-the-art performance, zero-shot generalization, robustness to forgetting, and competitive single-task results across a variety of question answering tasks. Our multi-task mixture training learns from tasks of various question intents and thus generalizes better, including on zero-shot vision-language tasks. We conduct experiments in the challenging multi-task and open-vocabulary settings and across a variety of datasets and tasks, such as VQA2.0, SNLI-VE, NLVR2, GQA. We observe that the proposed approach is able to generalize to unseen tasks and that more diverse mixtures lead to higher accuracy in both known and novel tasks.
中文速览
视觉问答领域长期存在一个痛点：现有模型要么预训练时忽略文本编码器、无法理解问题意图，要么针对单任务微调后会严重遗忘其他任务，且普遍依赖繁琐的目标检测框作为图像特征。Answer-Me 提出了一套简洁的多任务框架，用一种新颖的预训练方法——在噪声图像描述数据上同时训练图像编码器、文本编码器、跨模态融合模块和文本解码器全部组件——让模型真正理解问题意图，再通过将视觉问答、视觉蕴含、视觉推理等多种任务混合训练，使模型能够以自由文本形式回答各类问题。实验表明，Answer-Me 在 VQA2.0、SNLI-VE、NLVR2、GQA 等多个基准上达到最先进水平，还能零样本泛化到训练中未见过的新任务，并显著缓解了灾难性遗忘问题，且任务混合越多样，新旧任务的准确率越高。这项工作的价值在于，它用极简的架构（无需检测框）和统一的开放词表生成范式，证明了多任务混合训练本身就是抵御遗忘、提升泛化的有效"配方"，为构建通用视觉语言助手提供了新思路。
原文 arXiv:2205.00949;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.00949v2