Otter: A Multi-Modal Model with In-Context Instruction Tuning
Bo Li* Yuanhan Zhang* Liangyu Chen* Jinghao Wang* Jingkang Yang Ziwei Liu🖂 S-Lab, Nanyang Technological University, Singapore {libo0013, yuanhan002, lchen025, c190209, jingkang001, https://github.com/Luodian/Otter
Abstract
Large language models (LLMs) have demonstrated significant universal capabilities as few/zero-shot learners in various tasks due to their pre-training on vast amounts of text data, as exemplified by GPT-3, which boosted to InstrctGPT and ChatGPT, effectively following natural language instructions to accomplish real-world tasks. In this paper, we propose to introduce instruction tuning into multi-modal models, motivated by the Flamingo model’s upstream interleaved format pretraining dataset. We adopt a similar approach to construct our MultI-Modal In-Context Instruction Tuning (MIMIC-IT) dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind’s Flamingo), trained on MIMIC-IT and showcasing improved instruction-following ability and in-context learning. We also optimize OpenFlamingo’s implementation for researchers, democratizing the required training resources from 1 $\times$ A100 GPU to 4 $\times$ RTX-3090 GPUs, and integrate both OpenFlamingo and Otter into Huggingface Transformers for more researchers to incorporate the models into their customized training and inference pipelines.
中文速览
多模态大语言模型虽然具备强大的视觉-语言理解能力，却普遍缺乏对自然语言指令的精准遵循能力，尤其难以在推理时借助少量示例快速适应新任务。为此，研究者构建了一个名为MIMIC-IT的多模态上下文指令微调（Multi-Modal In-Context Instruction Tuning）数据集，将视觉问答、图像描述等多种任务整理成"图像-指令-回答"三元组，并为每条样本配备语义相关的上下文示例，从而同时提升指令遵循能力与上下文学习能力。在此基础上，他们训练了Otter模型——以开源的OpenFlamingo为骨架，冻结视觉和语言编码器，仅微调交叉注意力层等约13亿参数，使模型既能准确理解用户指令，又能通过少样本示例举一反三。实验定性分析表明，Otter在指令遵循和上下文学习两方面均明显优于OpenFlamingo基线，同时研究者还将训练所需显卡从单张A100降至4张RTX-3090，并将模型接入Hugging Face生态，大幅降低了学术界复现与二次开发的门槛，对推动多模态基础模型的民主化具有重要意义。
原文 arXiv:2305.03726;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.03726v2