Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
Zangwei Zheng1, Xiaozhe Ren2, Fuzhao Xue1, Yang Luo1, Xin Jiang2, Yang You1 1Department of Computer Science, National University of Singapore 2Noah’s Ark Lab, Huawei. {zangwei, f-xue, yangluo, {renxiaozhe, https://github.com/zhengzangw/Sequence-Scheduling
Abstract
Large language models (LLMs) have revolutionized the field of AI, demonstrating unprecedented capacity across various tasks. However, the inference process for LLMs comes with significant computational costs. In this paper, we propose an efficient LLM inference pipeline that harnesses the power of LLMs. Our approach begins by tapping into the potential of LLMs to accurately perceive and predict the response length with minimal overhead. By leveraging this information, we introduce an efficient sequence scheduling technique that gathers queries with similar response lengths into micro-batches. We evaluate our approach on real-world instruction datasets using the LLaMA-based model, and our results demonstrate an impressive 86% improvement in inference throughput compared to the vanilla batch inference without compromising effectiveness. Notably, our method is orthogonal to other inference acceleration techniques, making it a valuable addition to many existing toolkits (e.g. FlashAttention, Quantization) for LLM inference.
中文速览
大型语言模型（LLM）在批量推理时，由于不同请求的回复长度差异悬殊，短序列必须等待长序列生成完毕，导致大量算力被浪费在无效的填充计算上。这篇论文发现LLM本身就具备预判自己回复长度的能力，并通过指令微调（instruction tuning）进一步强化了这一能力，使较小的模型（如LLaMA-7B）也能以极低的额外开销准确预测输出长度。基于此，作者设计了一套"序列调度"（sequence scheduling）推理流水线，将预测回复长度相近的请求归为同一个微批次（micro-batch）来并行处理，同时引入失败重计算和动态批大小等辅助策略。在真实指令数据集上以Vicuna模型进行测试，该方法相比原始批量推理将吞吐量提升了86%，且不损失生成质量，还可与FlashAttention、量化等现有加速技术叠加使用，具有很强的实用价值。
原文 arXiv:2305.13144;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.13144v2