arXiv：cs.CL· Guancheng Du, Luotian Huang, Shaowen Wang, Si Li, Kaifeng Lyu·· 6 小时前AI 评分44
WavePrune：RoPE 只需一个周期？
WavePrune: One period is often enough for RoPE
AI 导读
WavePrune 提出将 RoPE 每个通道限制在首个旋转周期内，以消除位置混叠带来的注意力干扰，在五个测试模型中的四个上无需额外调优即提升 HELMET 分数（如 Qwen3-8B 从 35.7 升至 40.0）。
正文
Abstract:Rotary Position Embedding (RoPE) encodes token positions by rotating each two-dimensional channel of the query and key vectors at a channel-specific frequency, making the attention logits invariant to a common shift of positions. However, this rotation is periodic, and it leads to position aliasing where relative positions separated by a full rotation period become hard to tell apart. To address this, we propose WavePrune, which restricts each channel to its first rotation period. We show that it removes the distractions in attention maps created by position aliasing and improves overall long-context performance. Specifically, WavePrune raises the HELMET score on four of five models we test without any extra tuning (e.g., 35.7 -> 40.0 on Qwen3-8B). When pretraining models from scratch, WavePrune also achieves lower validation loss at extrapolated lengths than pretraining without it. Because WavePrune restricts each channel to a sliding window, it induces a fine-grained sparsity that our hardware-aligned CUDA kernels exploit for 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. Together, these results show that RoPE's periodic structure, widely regarded as essential, is largely redundant beyond the first rotation period.
| Subjects: | Computation and Language (cs.CL); Sound (cs.SD) |
| Cite as: | arXiv:2610.06963 [cs.CL] |
| (or arXiv:2610.06963v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.06963 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guancheng Du [view email]
[v1]
Sat, 3 Oct 2026 17:01:53 UTC (26,116 KB)
来源：arXiv：cs.CL · arxiv.org