arXiv：cs.LG· Donghan He, Luhuan Wu·· 4 小时前AI 评分33
通过有限阶松弛将路径梯度的适用扩展到离散随机变量
Extending Pathwise Gradients to Discrete Random Variables via Finite-Order Relaxation
AI 导读
研究者提出一种有限阶精确路径梯度估计框架，可对 Poisson 等常见离散随机变量构造无偏估计量。该估计量在给定阶数多项式的所有无偏解中取最小范数解，保持硬前向采样、无需温度调节，且只需数行代码实现，在唯一性和权重方差最小化上优于其他可行解；同时给出超出指定函数类时的非渐近偏差界。实验中低阶方法在线性、非线性及层次潜变量模型上匹敌或超越调优基线，并在所有运行时基准中快于竞品。
正文
Abstract:Pathwise gradients are preferred for continuous random variables because they are unbiased, low variance, and work with a single sample. For discrete variables, however, the pathwise identity cannot generally be exact for every differentiable function. We propose a general framework to construct finite-order exact pathwise gradient estimators for a range of common discrete variables such as Poisson. The estimator is the least-norm solution among all solutions that are unbiased for polynomials of degree at most. The resulting estimators preserve the hard forward sample, require no temperature tuning, and can be implemented in a few lines of codes. Against other admissible solutions, our estimator is unique and minimizes weight variance; in contrast, prior works use categorical variables or augmented representations to approximate non-categorical variables that induces excess variance and computations. To understand approximation bias for functions beyond the prescribed class, we also derive a non-asymptotic bias bound. In experiments our low order methods match or improve tuned baselines across linear, nonlinear and hierarchical latent-variable models, while out-speeding competitors in every runtime benchmark.
| Subjects: | Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.07786 [cs.LG] |
| (or arXiv:2610.07786v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07786 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Donghan He [view email]
[v1]
Tue, 6 Oct 2026 05:30:48 UTC (1,232 KB)
来源：arXiv：cs.LG · arxiv.org