arXiv：cs.AI· Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber·· 4 小时前
NoteBank 如何解锁线性注意力的长上下文能力
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
AI 导读
研究者提出稀疏情节记忆 NoteBank，无需训练、即插即用，通过回放原生前向计算读取证据，无需外部嵌入模型或独立检索器。在 100 万 token 下，它将 Gated DeltaNet 在六项 RULER 检索任务上的平均准确率从 1.7% 提升至 70.3%，且 4K 到 1M token 无明显检索退化。
正文
Authors:Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Piękos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, Jürgen Schmidhuber
Abstract:Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: \emph{recurrent computation need not bear the full burden of long-term storage}. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This \emph{training-free, plug-and-play} interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds $\leqslant$0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.
| Comments: | See our project in this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.15715 [cs.AI] |
| (or arXiv:2610.15715v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15715 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nanbo Li PhD [view email]
[v1]
Sun, 11 Oct 2026 15:58:04 UTC (1,878 KB)
来源：arXiv：cs.AI · arxiv.org