arXiv：cs.LG· Ruijie Li, Shengnan Ding, Weimin Zhang, Derick Tang, Zhanpeng Zeng, Qinsong Zeng, Ming Chen, Jiaxi Hu, Yuxuan Liang·· 3 小时前AI 评分36
Gated Slot Attention-2：线性注意力中的双侧关联记忆校正
Gated Slot Attention-2: Two-Sided Associative Memory Correction in Linear Attention
AI 导读
研究者提出 Gated Slot Attention-2（GSA2），通过共享 latent slots 将用于 key 侧校正的 Gated Oja Rule-2 与用于 value 侧校正的 Gated Delta Rule-2 结合，实现线性注意力的双侧关联记忆校正。
正文
Abstract:Linear attention models have emerged as efficient alternatives to standard attention, but effectively managing their fixed-size recurrent memory remains challenging. To improve memory, recent work has explored two distinct directions: delta-rule variants for precise correction of values associated with keys, and slot-based architectures such as Gated Slot Attention for modeling key and value memories in two stages. We observe that these directions are complementary--the delta rule provides effective memory correction, while the two-stage structure provides a natural way to operate on both sides of an association. Building on this insight, we introduce a new Gated Oja Rule for key-side correction and extend it with decoupled erase and write control to obtain Gated Oja Rule-2. We then introduce Gated Slot Attention-2 (GSA2), which combines Gated Oja Rule-2 for key-side correction with Gated Delta Rule-2 for value-side correction through shared latent slots. We further derive a hardware-efficient chunkwise algorithm for parallel training. Experiments demonstrate that GSA2 consistently improves over strong linear-attention baselines across benchmarks while retaining linear-time sequence modeling and constant-memory recurrent decoding.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02816 [cs.LG] |
| (or arXiv:2610.02816v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02816 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruijie Li [view email]
[v1]
Fri, 2 Oct 2026 05:04:07 UTC (281 KB)
来源：arXiv：cs.LG · arxiv.org