arXiv：cs.LG· Adonis Jamal, Samy Mekkaoui, Yadh Hafsi, Huy\^en Pham·· 4 小时前
Transport REINFORCE：面向无模型策略梯度平均场控制的随机传输映射
Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control
AI 导读
研究者提出 Transport REINFORCE，一种基于传输映射的无模型策略梯度方法，用于估计平均场控制中标准 REINFORCE 遗漏的种群分布贡献。该方法在有限状态空间直接扰动概率单纯形上的种群分布，在连续状态空间则将分布投影到高斯混合流形上再做随机化，并证明了扰动目标与梯度的一致性及样本梯度估计的偏差和均方误差界。在多个 MFC 基准上的数值实验显示其优于标准 REINFORCE。
正文
Abstract:We develop a model-free policy gradient method for discrete-time mean-field control (MFC). In MFC, the policy affects the objective both through the controlled dynamics and through the population distribution. Standard REINFORCE estimators capture the first effect but not the second. We introduce Transport REINFORCE, a transport map-based approach that perturbs a suitable transformation of the population distribution to estimate this missing mean-field contribution. The method applies to both finite and continuous state spaces. In finite state spaces, we perturb the population distribution directly on the probability simplex through a convex combination of the current population weights and random weights. In continuous state spaces, we project the population distribution onto the manifold of Gaussian mixtures, and then randomize it via a transport map that ensures the perturbed law remains within this manifold. We prove consistency of the perturbed objective and gradient as the perturbation vanishes, and derive bias and mean-square error bounds for the resulting sample-based gradient estimator. Numerical experiments on several MFC benchmarks show that Transport REINFORCE improves over standard REINFORCE.
| Subjects: | Optimization and Control (math.OC); Machine Learning (cs.LG) |
| MSC classes: | 93E20 (Primary) 49N80, 93E35, 68T05, 60J05, 60J10, 60K35 (Secondary) |
| Cite as: | arXiv:2610.11619 [math.OC] |
| (or arXiv:2610.11619v1 [math.OC] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11619 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Adonis Jamal [view email]
[v1]
Thu, 8 Oct 2026 09:58:46 UTC (981 KB)
来源：arXiv：cs.LG · arxiv.org