arXiv：cs.LG· Manuel H\"ulskamp, Julius Kobialka, Emanuel Sommer, Thomas Nagler, David R\"ugamer·· 13 小时前
PFN 如何摊销贝叶斯推理？一项关于先验数据拟合网络近似质量的研究
How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?
AI 导读
研究从经验风险最小化视角将 PFN 的部署误差拆解为先验误差、上下文误差和摊销误差，并在真实先验下对比参考 PPD 进行测量。结果显示 PFN 整体能很好地还原目标 PPD，但残差摊销误差在预训练分布下的稀有场景中会持续存在，即便扩大模型规模也难以消除，表现为独立预训练的 PFN 之间出现分歧。
正文
Abstract:Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in \textit{rare} settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.13764 [stat.ML] |
| (or arXiv:2610.13764v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13764 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Manuel Hülskamp [view email]
[v1]
Fri, 9 Oct 2026 15:49:24 UTC (237 KB)
来源：arXiv：cs.LG · arxiv.org