arXiv：cs.CL· Jorge L. Ruiz Williams·· 4 小时前AI 评分34
Attention-Mass Condensation 用于稀疏解码：保留注意力质量不等于稳定贪心决策
Attention-Mass Condensation for Sparse Decoding
AI 导读
一项针对稀疏解码的研究在 Qwen2-0.5B 上发现，60 次配对实验中没有一个运行能在 128 token 内与稠密解码保持完全一致，说明保留的注意力质量并不能保证稳定的贪心决策。
正文
Abstract:Attention-mass concentration creates an opportunity for sparse decoding, but retained mass alone does not guarantee a stable greedy decision: retrieval error, omitted value directions, and recursive decoding all matter. We formalize this distinction with an exact omitted-mass identity and a sufficient downstream margin condition, then characterize a query-dependent mean-pooled block selector. On Qwen2-0.5B, a paired fresh-selection sweep covers supports of 97--769 positions, contexts of 2K--16K, and five prefixes per context. The primary exact-match result is that none of 60 runs remains identical to dense decoding through 128 tokens. Distributional quality is distinct: for supports of at least 193, seven of nine context-support conditions have median teacher-forced continuation perplexity changes within 5\% of dense, but prompt-level ranges include severe 16K outliers. All seven runs with teacher-forced match below 70\% have perplexity increases above 100\%; these observations come from two prefixes and suggest a warning regime, not a general threshold. The measured perplexity is teacher-forced on the dense model's own continuation, not the sparse model's free-running output. Separate retrieval and attention-mass probes illustrate why captured mass alone is not a retrieval or decision guarantee. Isolated operator timings do not establish matched-quality acceleration or end-to-end serving speed.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2602.06317 [cs.LG] |
| (or arXiv:2602.06317v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.06317 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.5281/zenodo.18383733
DOI(s) linking to related resources |
Submission history
From: Jorge Ruiz Williams [view email]
[v1]
Fri, 6 Feb 2026 02:32:42 UTC (12 KB)
[v2]
Mon, 9 Feb 2026 21:18:55 UTC (15 KB)
[v3]
Wed, 7 Oct 2026 15:18:06 UTC (14 KB)
来源：arXiv：cs.CL · arxiv.org