arXiv：cs.LG· Shuxuan Li, Jialiang Kang, Yanghao Zhou, Ziye Zhu, Xiao Yang·· 2 小时前
FOLD：面向长上下文投机解码的免位置远记忆方法
FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding
AI 导读
FOLD 是一种免训练的投机解码方法，在保留精确局部位置的同时，将远端条目视为共享位置的远记忆，并通过锐化注意力缓解记忆增长带来的注意力稀释。在六组 target-drafter 组合和最高 256K token 的多种基准上，FOLD 大多取得最高加速比，最佳情况较自回归解码加速从 2.08× 提升至 4.22×。
正文
Abstract:Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs, where KV-cache access costs can be amortized over multiple accepted tokens. Yet most drafters are trained on short sequences and degrade on long inputs, while long-context training is costly. Existing training-free approaches restrict the context window, rescale positions, or augment recent context with retrieved distant entries. Drafters, however, read target hidden states that already encode contextual and positional information, and their lightweight architecture makes full-history attention affordable. We propose FOLD, a training-free method that preserves exact local positions and treats all distant entries as far memory at a shared position. To counter attention dilution as the memory grows, we sharpen attention within it to emphasize relevant entries while separately controlling its total weight relative to the local window. Across six target-drafter pairs and diverse benchmarks of up to 256K tokens, FOLD predominantly delivers the highest speedup, with the best case more than doubling the speedup over autoregressive decoding, from 2.08$\times$ to 4.22$\times$.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20392 [cs.CL] |
| (or arXiv:2610.20392v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20392 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuxuan Li [view email]
[v1]
Wed, 14 Oct 2026 15:55:10 UTC (703 KB)
来源：arXiv：cs.LG · arxiv.org