arXiv：cs.LG· Ethan Harvey, Geoff Pleiss, Michael C. Hughes·· 13 小时前
研究揭示 ELBO 超参学习的 SNR 偏差：协方差假设决定过拟合还是欠拟合
Occam's Razor is Only as Sharp as Your ELBO
AI 导读
一项关于贝叶斯线性回归的研究发现，在变分推断中，基于 ELBO 的超参数学习会因高斯近似后验的协方差矩阵假设而高估或低估信噪比（SNR）。其中对角或低秩加对角协方差矩阵会低估 SNR 并导致训练集均方误差偏高（欠拟合），低秩协方差矩阵则高估 SNR 并导致误差偏低（过拟合）。研究者提示，希望扩展到大规模模型的贝叶斯实践者需警惕可计算性所需的协方差假设对模型选择的影响。
正文
Abstract:The marginal likelihood, also known as the evidence, is regarded as a mathematical embodiment of Occam's razor, enabling hyperparameter selection that avoids overfitting. The evidence lower bound (ELBO) objective from variational inference has also been used for similar purposes. In this paper, we define a signal-to-noise ratio (SNR) for Bayesian linear regression in terms of key hyperparameters controlling signal and noise variance. We then study how ELBO-based hyperparameter learning can under or overestimate SNR depending on the assumed covariance matrix structure in a Gaussian approximate posterior. We prove that assuming a diagonal or low-rank plus diagonal covariance matrix underestimates the SNR, while a low-rank covariance matrix overestimates the SNR. Underestimating the SNR leads to higher than optimal mean squared error on the training set (underfitting), while overestimating SNR leads to lower error (overfitting). Bayesian practitioners hoping to scale to large models should be cautious about how covariance assumptions needed for tractability may impact model selection.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.25984 [stat.ML] |
| (or arXiv:2604.25984v2 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2604.25984 arXiv-issued DOI via DataCite |
Submission history
From: Ethan Harvey [view email]
[v1]
Tue, 28 Apr 2026 16:44:26 UTC (836 KB)
[v2]
Fri, 9 Oct 2026 12:42:27 UTC (3,057 KB)
来源：arXiv：cs.LG · arxiv.org