arXiv：cs.LG· Junwei Su, Mengfan Liu, Yanyong Zhang, Chuan Wu·· 4 小时前AI 评分32
PPO-Clip 闭环非渐近收敛分析：带学习型 Critic 与裁剪的理论研究
A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
AI 导读
一项研究对带裁剪的 PPO-Clip 进行了闭环 actor–critic 非渐近分析，同时刻画策略平稳性与学习型 critic 的跟踪精度，并显式给出对算法参数的依赖关系。该分析涵盖 actor–critic 耦合、非光滑概率比裁剪、有限批量复用与可预测早停，并在两层时间尺度调度下给出 O(T^{-2/5}) 的平稳性与 critic 跟踪界，为 PPO 调参提供理论指导。
正文
Abstract:Despite its widespread use, Proximal Policy Optimization with clipping (PPO-Clip) remains difficult to tune, and the interactions among critic learning, clipping, and rollout reuse remain incompletely understood. We develop a \emph{non-asymptotic} analysis of PPO-Clip as a \emph{closed-loop actor--critic} system. It captures actor--critic coupling, nonsmooth probability-ratio clipping, finite-batch reuse, and predictable early stopping under explicit coverage and critic regularity assumptions, using raw GAE and Monte Carlo critic targets. Our synchronous and asynchronous guarantees jointly characterize policy stationarity and the tracking accuracy of the learned critic, with explicit dependence on algorithmic parameters. A sufficient coupling condition gives optimization, critic tracking, clipping, and finite-batch errors a common amplification bound. The asynchronous result also requires a delay-dependent critic stepsize restriction; violating these conditions does not establish divergence. For finite layered MDPs with tabular critics, a uniform bound on the actual clipped-gradient class replaces complete-trajectory counting. A verified growing-horizon family has polynomial sample complexity, and a two-time-scale schedule gives $O(T^{-2/5})$ stationarity and critic-tracking bounds with explicit fresh-rollout accounting. These results together advance our understanding about PPO and provide theoretical guidance in tuning.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.10273 [cs.LG] |
| (or arXiv:2610.10273v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10273 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junwei Su [view email]
[v1]
Wed, 7 Oct 2026 15:40:47 UTC (181 KB)
来源：arXiv：cs.LG · arxiv.org