arXiv：cs.AI· Zhetong Zhang, Gayan K Kulatilleke·· 6 小时前
SuPER-Mamba：面向 Mamba 长上下文路由的 Surprisal 优先化框架
SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mamba
AI 导读
SuPER-Mamba 是一个免训练框架，利用模型自身的 Surprisal 信号为预训练 Mamba 模型做长程状态路由，通过 Local Pooling 聚合相邻响应并优先更新 Global-Channel，无需查询相关打分或额外训练。
正文
Abstract:State space models provide efficient linear-complexity processing for long sequences, but their performance frequently degrades as context length grows. We propose SuPER-Mamba, a training-free framework that uses Surprisal as a model-intrinsic signal for long-range state routing in pretrained Mamba models. SuPER-Mamba computes position-wise Surprisal from next-token predictions, aggregates neighboring responses through Local Pooling, and uses the resulting scores to prioritize Global-Channel updates without query-dependent scoring or additional training. We evaluate SuPER-Mamba on three state-space models across the LongBench-E benchmark. The results show that SuPER-Mamba improves the average score from 8.06 to 15.69, corresponding to a 94.7% relative gain over the Base Models and a 13.1% improvement over the state-of-the-art prior baseline. Ablation and length-wise analyses further support the effectiveness of the proposed routing framework.
| Comments: | 9 pages, 9 tables, source code |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16636 [cs.CL] |
| (or arXiv:2610.16636v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16636 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Gayan Kulatilleke [view email]
[v1]
Mon, 12 Oct 2026 11:57:50 UTC (193 KB)
来源：arXiv：cs.AI · arxiv.org