HuggingFace Daily Papers·· 11 小时前
Continuum Attention：用连续记忆曲线积分替代离散键值查找的注意力算子
Continuum Attention
AI 导读
研究者提出 Continuum Attention（CA），一种因果 Transformer 注意力算子，用对连续分段线性记忆曲线的积分替代离散 key/value 查找，并通过闭式指数矩得到精确归一化注意力。
正文
Abstract:We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20586 [cs.LG] |
| (or arXiv:2610.20586v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20586 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Afshin Khadangi [view email]
[v1]
Wed, 14 Oct 2026 17:57:40 UTC (694 KB)
来源：HuggingFace Daily Papers · arxiv.org