arXiv：cs.AI· Zekun Shi, Alastair James Arthur Price·· 6 小时前
FreeSolo：零样本长度外推的线性时间稀疏注意力
FreeSolo: Linear-time Sparse Attention with Zero-Shot Length Extrapolation
AI 导读
FreeSolo 是一种原生稀疏注意力机制，结合短程滑动窗口分支与长程分支，后者选取关键单元直接进行 token 注意力计算。其线性时间变体 FS-L 通过短程 CPT 学习在查询相关选择前剪枝前缀，FS-B 则以 O(N log² N) 算法开销搜索平衡树。FreeSolo 在无需长上下文微调的情况下，零样本长度外推表现优于基线。
正文
Abstract:Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with $O(N\log^2 N)$ algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.17191 [cs.LG] |
| (or arXiv:2610.17191v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17191 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zekun Shi [view email]
[v1]
Mon, 12 Oct 2026 17:55:45 UTC (233 KB)
来源：arXiv：cs.AI · arxiv.org