arXiv：cs.LG· Tomohiro Hayase, Ryo Karakida·· 4 小时前AI 评分32
多头自注意力机制的 Gaussian 等价性
Gaussian Equivalence for Multi-Head Self-Attention
AI 导读
研究者利用随机矩阵理论证明了多头自注意力的 Gaussian 等价性：将 softmax 注意力替换为缩放分数加 Gaussian 噪声，可保持中心化输出的极限谱定律不变。该等价性还覆盖了依赖 key 的 value 和输出投影，得到的谱定律能分离头分配与投影宽度的影响，并区分跨头共享权重的谱保持性与头内 key-value 依赖性。
正文
Abstract:A theoretical understanding of multi-head self-attention is fundamental to the study of modern neural networks. Using random matrix theory, we establish Gaussian equivalence for multi-head self-attention: replacing softmax attention with rescaled scores plus Gaussian noise preserves the limiting spectral law of the centered output. This equivalence also covers value and output projections that depend on the keys. The resulting laws separate the effects of head allocation and projection widths, and distinguish spectrum-preserving across-head sharing from within-head key--value dependence.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG); Probability (math.PR) |
| MSC classes: | 60B20 (Primary), 68T07, 46L54 (Secondary) |
| ACM classes: | I.2.6; G.3 |
| Cite as: | arXiv:2610.10033 [stat.ML] |
| (or arXiv:2610.10033v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10033 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tomohiro Hayase [view email]
[v1]
Wed, 7 Oct 2026 13:18:08 UTC (214 KB)
来源：arXiv：cs.LG · arxiv.org