arXiv：cs.LG· Zhuolin Fu·· 13 小时前
把 Transformer 看作概率推断：Neural expectation-maximization 与 meta-models
Neural expectation-maximization and meta-models
AI 导读
研究者将 Transformer 解释为 max-product LBP 与 EM 的混合概率推断近似：self-attention 实现输入条件下动态结构 MRF 上 max-product LBP 的可微近似，注意力权重编码学到的结构亲和度。
正文
Abstract:We interpret Transformer networks as learned approximations to a hybrid probabilistic-inference procedure that combines max-product Loopy Belief Propagation (LBP) with Expectation--Maximization (EM). In this view, a Transformer performs inference in partially observed Markov Random Fields (MRFs) whose structure and parameters are not given a priori but are inferred from data. We show that self-attention implements a differentiable approximation to max-product LBP on an input-conditioned, dynamically structured MRF, with attention weights encoding learned structural affinities. We further establish an EM correspondence at the level of depth: each layer updates latent representations in an E-step-like forward pass, while the transition $\theta^{(l)} \to \theta^{(l+1)}$ across layers acts as an implicit M-step. Building on this interpretation, we introduce \emph{meta-models}, a design paradigm in which inputs generate instance-specific parameters, unifying mechanisms such as self-attention, mixture-of-experts, and gated units within a single framework. Empirical results support the proposed perspective.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12741 [cs.LG] |
| (or arXiv:2610.12741v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12741 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhuolin Fu [view email]
[v1]
Thu, 8 Oct 2026 21:37:24 UTC (4,155 KB)
来源：arXiv：cs.LG · arxiv.org