arXiv：cs.AI· Zhengqing Yan, Fan Guo, Xinyang Liu, Yi Zhang, Yao Liu, Qi Liu, Jihao Huang, Kang Song·· 16 小时前
GDEPO：面向样本高效强化学习的群组双动态与等权优势策略优化
GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization for Sample-Efficient Reinforcement Learning
AI 导读
针对GRPO在自动定理证明中存在的优势估计与二值验证反馈冲突、静态采样丢弃无效批次导致数据浪费等问题，作者提出GDEPO，包含动态补采样、等权优势与动态补迭代三项机制。在MinF2F-test、MathOlympiadBench和PutnamBench上的实验及消融研究验证了其有效性。
正文
Abstract:Automated Theorem Proving (ATP) is a fundamental AI challenge, requiring machine-verifiable proofs in formal languages like Lean to evaluate reasoning. Reinforcement Learning (RL), especially Group Relative Policy Optimization (GRPO), is a mainstream approach. However, GRPO faces two critical issues in ATP: its relative advantage estimation may conflict with binary verifier feedback under composite rewards, and its static sampling strategy discards batches without valid proofs, causing data waste. Repeated training on mastered problems also wastes computation. To address this, we propose Group Dual-dynamic and Equal-right Advantage Policy Optimization (GDEPO), incorporating three core mechanisms: 1) dynamic additional sampling, resampling ineffective batches until a valid proof is found; 2) equal-right advantage, decoupling the advantage function's sign (correctness) from its magnitude (auxiliary rewards) to ensure stable policy updates while increasing effective computation; and 3) dynamic additional iterations, applying extra gradient steps to initially failed but eventually successful samples to accelerate learning. Experiments on MinF2F-test, MathOlympiadBench, and PutnamBench confirm GDEPO's effectiveness, with ablation studies validating its synergistic components. The method enhances data utilization and optimization efficiency, offering a novel training paradigm for ATP.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.06795 [cs.AI] |
| (or arXiv:2601.06795v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2601.06795 arXiv-issued DOI via DataCite |
Submission history
From: Zhengqing Yan [view email]
[v1]
Sun, 11 Jan 2026 07:34:41 UTC (2,146 KB)
[v2]
Wed, 21 Jan 2026 02:58:47 UTC (2,146 KB)
[v3]
Thu, 22 Jan 2026 14:58:18 UTC (2,146 KB)
[v4]
Mon, 12 Oct 2026 13:47:23 UTC (2,103 KB)
来源：arXiv：cs.AI · arxiv.org