Giraffe: Adventures in Expanding Context Lengths in LLMs
Arka Pal111Correspondence to Abacus.AI Deep Karkhanis Abacus.AI Manley Roberts Abacus.AI Samuel Dooley Abacus.AI Arvind Sundararajan Abacus.AI Siddartha Naidu Abacus.AI
Abstract
Modern large language models (LLMs) that rely on attention mechanisms are typically trained with fixed context lengths which enforce upper limits on the length of input sequences that they can handle at evaluation time. To use these models on sequences longer than the train-time context length, one might employ techniques from the growing family of context length extrapolation methods — most of which focus on modifying the system of positional encodings used in the attention mechanism to indicate where tokens or activations are located in the input sequence. We conduct a wide survey of existing methods of context length extrapolation on a base LLaMA or LLaMA 2 model, and introduce some of our own design as well — in particular, a new truncation strategy for modifying the basis for the position encoding.
中文速览
大型语言模型（LLM）在训练时通常设有固定的上下文长度上限，一旦推理时输入序列超过这个长度，模型性能就会严重下滑。研究团队对基于 LLaMA/LLaMA 2 的多种"上下文长度外推（context length extrapolation）"方法进行了系统性横向比较，并提出了一种新的截断位置编码基底（truncated basis）方案。为了更精准地衡量模型在长上下文下的真实表现，他们还设计了三个新评测任务——FreeFormQA、AlteredNumericQA 和 LongChat-Lines，并发现传统的困惑度（perplexity）指标过于粗糙，无法有效区分各方法的优劣。实验结果表明，线性缩放插值（linear scaling/positional interpolation）是目前最优的外推方法，且在微调尺度因子的基础上，推理时使用最多 2 倍更大的缩放因子还能进一步提升可用上下文长度；基于上述发现，团队公开发布了三个支持 4k、16k 和 32k 上下文的 13B 参数模型（统称 Giraffe），以及配套数据集和代码，为该领域后续研究提供了重要的开源基础。
原文 arXiv:2308.10882;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.10882v1