arXiv：cs.LG· Marius P Linhard, Maurizio Filippone·· 4 小时前AI 评分35
稀疏化随机性而非容量：通过先验尺度的深度权重分解实现部分随机性
Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales
AI 导读
研究提出用深度权重分解学习贝叶斯神经网络中哪些参数应随机，先验尺度低于阈值的参数转为确定性并在推理中优化，使正则化稀疏化随机性而非容量。在双峰目标上，学到的拆分在各预算下接近无约束参考且对阈值不敏感，随机掩码方案最多差两个数量级；UCI 基准上与全随机网络持平，约一半参数保持确定性。
正文
Abstract:Bayesian neural networks need not be fully stochastic to be universal conditional density approximators, but it remains open which parameters should be stochastic. We learn this split by applying deep weight factorization to the prior scales, which are the standard deviations of the parameter priors, while fitting the functional prior to a Gaussian process with a maximum mean discrepancy objective. A parameter whose prior scale falls below a cutoff becomes deterministic and is optimized during inference, so the regularizer sparsifies stochasticity rather than capacity. We give a certificate for universal conditional density approximation that is checkable in linear time, together with a minimal repair when it fails. We further show that the common hybrid scheme of sampling some parameters and optimizing the others is stochastic approximation for a type-II maximum a posteriori objective, and that coupled step sizes can leave a tracking error that does not vanish as the step size shrinks. On a bimodal target, the learned split stays close to an unconstrained reference across all budgets and is insensitive to the cutoff, while random masks that distribute the same prior scales across layers are worse by up to two orders of magnitude. On UCI benchmarks, our method performs on par with a fully stochastic network while keeping about half of its parameters deterministic.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.09886 [stat.ML] |
| (or arXiv:2610.09886v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09886 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Marius Peter Linhard [view email]
[v1]
Wed, 7 Oct 2026 11:45:24 UTC (953 KB)
来源：arXiv：cs.LG · arxiv.org