arXiv：cs.AI· Jerry Yao-Chieh Hu, Yu-An Chen, Maojiang Su, Han Liu·· 6 小时前
Transformer 注意力的统计力学：用现代 Hopfield 模型刻画相变与推理可靠性
On the Statistical Mechanics of Attention
AI 导读
研究通过现代 Hopfield 模型重新审视 Transformer 注意力的统计力学，提出自由能形式化框架来刻画稳健注意力推理的参数区间与临界阈值，并识别出注意力在玻璃态与稳健推理相之间的一阶相变。该框架进一步扩展到具有随机 query 和 key 的注意力，为刻画 Transformer 注意力机制的相结构与推理可靠性提供严格的统计力学基础。
正文
Abstract:We revisit the statistical mechanics of Transformer attention via modern Hopfield models. We present a free-energy formalism to characterize parameter regimes and critical thresholds for robust attention inference. The resulting free-energy landscape incorporates randomness in query states and attention weights. Attention then emerges as the equation of motion of the corresponding statistical-mechanical system (i.e., modern Hopfield model) through the standard free-energy minimization principle. This allows us to identify first-order phase transitions in attention between glassy and robust-inference phases. We further extend to attentions with stochastic queries and keys. Together, our results provide a rigorous statistical-mechanics framework for characterizing phase structure and inference reliability across a family of Transformer attention mechanisms.
| Subjects: | Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Statistical Mechanics (cond-mat.stat-mech); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.17347 [cs.LG] |
| (or arXiv:2610.17347v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17347 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jerry Yao-Chieh Hu [view email]
[v1]
Mon, 12 Oct 2026 19:55:12 UTC (355 KB)
来源：arXiv：cs.AI · arxiv.org