arXiv：cs.LG· Joseph Keslin, Wei Biao Wu·· 16 小时前
恒定学习率 SGD 的强高斯近似：偏差与轨迹推断
Strong Gaussian Approximation of Constant Learning Rate SGD: Bias and Trajectory Inference
AI 导读
该论文为恒定学习率 SGD 构建了 KMT 型强高斯近似，在重尾噪声下令迭代次数增长与学习率收缩同步进行，并给出 SGD 误差与其极限之差的概率上界。当学习率按 γ_n=c/√n+o(n^{-1/2}) 缩放时，偏差控制与初始化效应之间存在权衡，作者用 Burn-in 方案与 Richardson-Romberg 外推消除这两项贡献。
正文
Abstract:This paper develops strong Gaussian approximations (KMT type) for constant learning rate stochastic gradient descent (SGD). We work with heavy tailed noise and assume that the number of iterates grows and the learning rate shrinks simultaneously. Our work constructs simple Gaussian processes on the same space and bounds, in probability, the maximum difference between the SGD errors and their respective limits. While there exist central limit theorems for constant rate SGD centered at the optimum, these only provide individual iterate inference. By contrast, our quenched KMT approximations allow for finite sample, simultaneous, whole-trajectory-level inference and imply stochastic process convergence (Functional Central Limit Theorems (FCLTs)). For accumulated errors centered at the true optimum, we first obtain a KMT-type approximation bound when the learning rate scales as $\gamma_n=\frac{c}{\sqrt n}+o(n^{-1/2})$ for a constant $c \in \mathbb{R}^+$ and $n$ iterations. The result reveals a trade off between controlling the bias of SGD and the initialization effect, with no learning rate scaling generally making both contributions vanish in the accumulated error limit. To eliminate these contributions, we consider a Burn in scheme and Richardson-Romberg extrapolation and prove respective KMT and FCLT theorems. We also prove a finite-horizon KMT-type approximation and FCLT for the rescaled iterate trajectory to a stationary Ornstein--Uhlenbeck (OU) process. We consider applications in risk control and simultaneous inference and perform simulations to illustrate our results.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.14567 [stat.ML] |
| (or arXiv:2610.14567v2 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14567 arXiv-issued DOI via DataCite |
Submission history
From: Joseph Keslin [view email]
[v1]
Sat, 10 Oct 2026 04:38:25 UTC (622 KB)
[v2]
Wed, 14 Oct 2026 12:15:23 UTC (622 KB)
来源：arXiv：cs.LG · arxiv.org