FastSeq: Make Sequence Generation Faster
Yu Yan1 , Fei Hu1 *, Jiusheng Chen1 * , Nikhil Bhendawade1 *, Ting Ye1, Yeyun Gong2, Nan Duan2, Desheng Cui1, Bingyu Chi1 and Ruofei Zhang1 1Microsoft, 2 Microsoft Research Asia {yyua,fhu,jiuchen,nibhenda,tiy, yegong,nanduan,decu,bingychi,bzhang}yyua,fhu,jiuchen,nibhenda,tiy, yegong,nanduan,decu,bingychi,bzhang\left\{\minipage{341.43306pt}\centering{yyua,fhu,jiuchen,nibhenda,tiy,}\\ Equal contribution Corresponding author
Abstract
Transformer-based models have made tremendous impacts in natural language generation. However the inference speed is a bottleneck due to large model size and intensive computing involved in auto-regressive decoding process. We develop FastSeq framework to accelerate sequence generation without accuracy loss. The proposed optimization techniques include an attention cache optimization, an efficient algorithm for detecting repeated n-grams, and an asynchronous generation pipeline with parallel I/O. These optimizations are general enough to be applicable to Transformer-based models (e.g., T5, GPT2, and UniLM). Our benchmark results on a set of widely used and diverse models demonstrate 4-9x inference speed gain. Additionally, FastSeq is easy to use with a simple one-line code change. The source code is available at https://github.com/microsoft/fastseq.
中文速览
自回归文本生成（auto-regressive generation）模型虽然效果出色，但推理速度极慢，主要卡在注意力缓存维护、重复n-gram检测和后处理这三个非计算环节上，合计占用了超过80%的生成时间。FastSeq框架针对这三个瓶颈分别提出了对应优化：通过去除beam搜索中冗余的键值缓存副本来压缩内存与计算量、用GPU并行内核替代逐步扫描的n-gram重复检测、以及用异步流水线让后处理与下一批次生成同时进行。在T5、GPT2、BART、UniLM等多种主流模型上的基准测试显示，推理速度提升了4到9倍，且不损失任何模型精度。这项工作的价值在于它无需改动模型结构或重新训练，用户只需一行代码就能接入，极大降低了将大模型部署到实际应用中的成本门槛。
原文 arXiv:2106.04718;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.04718v2