arXiv：cs.LG· A. Ch. Madhusudanarao, Rahul Singh·· 4 小时前AI 评分34
非线性双时间尺度随机逼近中的稳态偏差与外推
Stationary Bias and Extrapolation in Nonlinear Two-Timescale Stochastic Approximation
AI 导读
该论文研究了由外生有限状态马尔可夫链驱动的非线性双时间尺度递推中的稳态均值误差。推导出一阶偏差展开式，在慢步长远小于快步长时误差界保持均匀，并揭示出 $\varepsilon^2/\eta$ 的混合贡献项。研究表明，沿幂律步长路径的偏差指数不必为整数，因此 Richardson–Romberg 外推需要与路径匹配的权重，并通过可精确求解的非线性马尔可夫例子验证了系数。
正文
Abstract:Constant-step stochastic approximation generally has a nonzero stationary mean error that persists under time averaging. This paper studies that error for nonlinear two-timescale recursions driven by an exogenous finite-state Markov chain. Under stated smoothness assumptions and conditions on the stationary distribution, we derive a first-order bias expansion whose error bound remains uniform as the slow step size becomes much smaller than the fast step size. Fast-manifold coordinates keep the associated covariance equation regular in this limit. For fast step $\eta$ and slow step $\varepsilon$, the expansion reveals a mixed contribution $\varepsilon^2/\eta$ alongside terms linear in each step size. This dependence matters for bias reduction: along power-law step-size paths, the bias exponents need not be integers, so Richardson--Romberg extrapolation requires weights matched to the path. An exactly solvable nonlinear Markov example verifies the coefficients. We verify localization for temporal-difference learning and compare finite-run extrapolation at equal update budgets. For finite runs, we bound the initialization error of tail averages on both timescales under an additional coupling assumption. In the special case of additive independent noise, signed third-moment cancellation yields a sharper remainder.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.10246 [cs.LG] |
| (or arXiv:2610.10246v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10246 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Avvaru Ch Madhusudanarao [view email]
[v1]
Wed, 7 Oct 2026 15:28:13 UTC (156 KB)
来源：arXiv：cs.LG · arxiv.org