arXiv：cs.LG· Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou·· 13 小时前
GPPO：从散度视角统一 LLM 强化学习算法，性能超越 GRPO 达 50%
Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
AI 导读
研究者提出 Generalized Proximal Policy Optimization（GPPO），从 f-散度视角统一理解和设计 LLM 强化学习算法，PPO 及 DAPO、DPPO、RIPO 等变体均为其特例。在八项竞赛级数学与编程基准上，最佳 GPPO 实例在数学和编程任务上分别超越标准 GRPO 达 50% 和 14%，达到 SOTA 性能。
正文
Abstract:Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that cannot adapt its resistance to policy shifts, leading to premature clipping and degraded exploration. Subsequent variants alleviate these issues through fragmented and heuristic modifications. We propose Generalized Proximal Policy Optimization (GPPO), a principled and unified framework for understanding and designing LLM RL algorithms through the lens of divergence. We establish a generalized $f$-divergence trust-region theory with a monotonic performance improvement guarantee, and show that different $f$-divergences induce distinct geometries and policy update dynamics. We further introduce novel estimator topologies that provide more faithful divergence estimates than the conventional single-sample Monte Carlo estimation. Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases, establishing a principled two-dimensional design space of {Divergence $\times$ Estimator}. Extensive experiments on eight competition-level math and coding benchmarks show that both dimensions are critical. The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding, delivering SOTA performance.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.13015 [cs.LG] |
| (or arXiv:2610.13015v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13015 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhicheng Cai [view email]
[v1]
Fri, 9 Oct 2026 05:17:26 UTC (3,400 KB)
来源：arXiv：cs.LG · arxiv.org