InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining
Boxin Wang Wei Ping Lawrence McAfee Peng Xu Bo Li Mohammad Shoeybi Bryan Catanzaro
Abstract
Pretraining auto-regressive large language models (LLMs) with retrieval demonstrates better perplexity and factual accuracy by leveraging external databases. However, the size of existing pretrained retrieval-augmented LLM is still limited (e.g., Retro has 7.5B parameters), which limits the effectiveness of instruction tuning and zero-shot generalization. In this work, we introduce Retro 48B, the largest LLM pretrained with retrieval. Specifically, we continue to pretrain a 43B GPT model on additional 100 billion tokens using the Retro augmentation method by retrieving from 1.2 trillion tokens. Notably, the obtained foundation model, Retro 48B, largely outperforms the counterpart GPT 43B trained on 1.2T tokens in terms of perplexity with only 2.58% additional GPU hours, demonstrating the significant scaling potential of the method. After instruction tuning on Retro, InstructRetro demonstrates significant improvement over the instruction tuned GPT on a wide range of zero-shot tasks. Specifically, the average improvement of InstructRetro is 7% over its GPT counterpart across 8 short-form QA and reading comprehension tasks, 10% over GPT across 4 challenging long-form QA tasks, and 16%
中文速览
检索增强预训练（Retrieval-Augmented Pretraining）能让大语言模型更好地利用外部知识库，但此前最大的此类模型只有不到100亿参数，严重制约了指令微调和零样本泛化的效果。为此，研究团队以一个已在1.1万亿词元上预训练好的430亿参数GPT模型为起点，继续用检索增强方法在额外1000亿词元上训练，最终得到480亿参数的Retro 48B——迄今最大的检索增强预训练语言模型，而额外计算开销仅占总GPU时数的2.58%。在此基础上进行指令微调得到的InstructRetro，在短文本问答、长文本问答和摘要等多类零样本任务上分别比同等规模的GPT基线平均提升7%、10%和16%；更出人意料的是，直接去掉检索编码器、只保留解码器骨干，性能几乎不降，说明带检索的持续预训练本身就能让解码器变得更强。这一发现为"先用检索增强预训练、再做指令微调"提供了一条高效且实用的路线，对构建更强大的开放域问答和知识密集型应用具有重要意义。
原文 arXiv:2310.07713;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.07713v3