arXiv：cs.CL· Valentino Maiorca, Walter Nelson, Francesco Locatello·· 4 小时前AI 评分46
语言模型边际注意力空间中的涌现结构
Emergent Structure in the Marginal Attention Space of Language Models
AI 导读
研究者将 post-softmax 注意力权重按 query 位置边缘化，构建 token-head 联合的"边际注意力空间"，在 60+ 个 LLM 上评估发现：按 token 归约得到跨模型稳健保留的文本内在信号，按 head 归约则形成跨文档保留的模型私有签名。
正文
Abstract:While representation similarity across independently trained language models is well-documented, how internal mechanics such as attention behave across models remains far less characterized. Inspired by this gap, we examine the structure of post-softmax attention weights by marginalizing over query positions, mapping them into a joint token-head "marginal attention space". Evaluating across 60+ diverse LLMs, we find that different properties emerge when reducing this space along its token and head axes. When reduced token-wise, marginal attention yields a text-intrinsic signal robustly conserved across models. To explain this property, we empirically connect marginal attention to the input-output Jacobian of the network, and prove theoretically that under a smoothness assumption, models with similar next-token distributions are guaranteed to have similar input-output Jacobian statistics. When reduced head-wise, it forms a model-private signature conserved across documents. Practically, this provides a natural way to estimate a per-head budget for key-value (KV) cache eviction, effectively decoupling model-specific budget allocation from text-intrinsic token scoring. On standard eviction benchmarks, a per-head budget precomputed offline on pretraining text, combined with a training-free token score, shows competitive performance with methods that recompute the budget on every document or train it per target. Code available at this https URL
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.03109 [cs.CL] |
| (or arXiv:2610.03109v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.03109 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Valentino Maiorca [view email]
[v1]
Fri, 2 Oct 2026 10:27:59 UTC (858 KB)
来源：arXiv：cs.CL · arxiv.org