arXiv：cs.LG（机器学习，全量分类）· Jian Xu·· 18 小时前AI 评分35
研究发现 Large Language Bayes 不具备重参数化不变性
Large Language Bayes Is Not Reparameterisation-Invariant
AI 导读
研究证明 Large Language Bayes（LLB）的证据下界权重依赖模型写法，不具备重参数化不变性。在 eight schools 上，中心化与非中心化程序测度仅差 5.7×10⁻¹⁴，权重却相差 6.1×，重要性加权仅降至 2.2×，完整协方差高斯推断仍留 1.9× 差异，64 维下达 8.9×。
正文
Abstract:Large Language Bayes (LLB) answers an informal modelling question by sampling candidate probabilistic programs from a language model, running approximate inference on each, and averaging them with weights proportional to an exponentiated evidence bound. We show that this weighting depends on how a model is written. The log marginal likelihood is invariant to reparameterisation; the evidence bound is not. On eight schools the centered and non-centered programs are the same measure to $5.7\times10^{-14}$, yet their weights differ by $6.1\times$; importance weighting reduces this only to $2.2\times$, and reproducing the inference LLB actually runs, a full-covariance Gaussian matched to the posterior moments, still leaves $1.9\times$ on eight schools and $8.9\times$ in $64$ dimensions. Across likelihood families, dimensions and funnel severities the discrepancy reaches $31.9\times$ and reverses sign, so no single writing is uniformly preferable. It inverts Bayes factors against eight natural competitors, and the induced error in the model posterior, and in any downstream target, is controlled by the spread $\Delta$ of the bound shortfalls through a known sharp Hilbert-distance bound. Across $360$ programs from six language models the parameterisation written ranges from $0\%$ to $100\%$ centered and is stable within a model. Detecting equivalent programs statistically can falsely merge genuinely different models at practical sample budgets; verifying reparameterisations we generate ourselves cannot, and closes the window.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.00265 [cs.LG] |
| (or arXiv:2610.00265v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00265 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xu Jian [view email]
[v1]
Thu, 24 Sep 2026 09:06:19 UTC (62 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org