GAPE：门控自适应位置编码如何提升 LLM 长上下文泛化能力

View PDF HTML (experimental)

Abstract:Rotary Positional Encoding (RoPE) is widely used in modern large language models. However, when sequences are extended beyond the range seen during training, rotary phases can enter out-of-distribution regimes, leading to spurious long-range alignments, diffuse attention, and degraded retrieval. Existing remedies only partially address these failures, as they often trade local positional resolution for long-context stability. We propose GAPE (Gated Adaptive Positional Encoding), a drop-in augmentation for positional encodings that introduces a content-aware bias directly into the attention logits while preserving the rotary geometry. GAPE decouples distance-based suppression from token importance through a query-dependent gate that contracts irrelevant context and a key-dependent gate that preserves salient distant tokens. We show that weakly protected distant context is exponentially attenuated as a function of the query gate, while selected keys can remain accessible through landmark protection. We further show that GAPE can be implemented within standard scaled dot-product attention. Empirically, GAPE improves long-context robustness across controlled retrieval and language-modeling experiments, extrapolating up to 8x the training length. We further retrofit GAPE into a pretrained 7B model, maintaining performance on standard benchmarks and improving performance at the longest evaluated context. These results support adaptive context suppression as a complement to positional representation for long-context generalization.

Subjects:	Machine Learning (cs.LG)
Cite as:	arXiv:2605.10414 [cs.LG]
	(or arXiv:2605.10414v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.10414 arXiv-issued DOI via DataCite

Submission history

From: Christopher Irwin [view email]
[v1] Mon, 11 May 2026 11:52:06 UTC (8,785 KB)
[v2] Thu, 8 Oct 2026 13:40:19 UTC (8,313 KB)