arXiv：cs.AI· Mohammad Rezaei, Jens Lehmann, Sahar Vahdati·· 20 小时前
PROGRS：用结果引导的过程奖励提升 LLM 数学推理强化学习
Process Rewards for Outcome-Guided Reasoning Steps in LLM Reinforcement Learning
AI 导读
PROGRS 框架将过程奖励模型（PRM）的分数作为结果分组内的相对偏好而非绝对目标，通过结果条件化中心化让错误轨迹的 PRM 分数在每组内均值为零，并与 Group Relative Policy Optimization（GRPO）结合。
正文
Abstract:Mathematical reasoning in large language models has improved substantially with reinforcement learning using verifiable rewards, where final answers can be checked automatically and converted into reliable training signals. Most such pipelines optimize outcome correctness only, which yields sparse feedback for long, multi-step solutions and offers limited guidance on intermediate reasoning errors. Recent work therefore introduces process reward models (PRMs) to score intermediate steps and provide denser supervision. In practice, PRM scores are often imperfectly aligned with final correctness and can reward locally fluent reasoning that still ends in an incorrect answer. When optimized as absolute rewards, such signals can amplify fluent failure modes and induce reward hacking.
We propose PROGRS, a framework that leverages PRMs while keeping outcome correctness dominant. PROGRS treats process rewards as relative preferences within outcome groups rather than absolute targets. We introduce outcome-conditioned centering, which shifts PRM scores of incorrect trajectories to have zero mean within each prompt group. It removes systematic bias while preserving informative rankings. PROGRS combines a frozen quantile-regression PRM with a multi-scale coherence evaluator. We integrate the resulting centered process bonus into Group Relative Policy Optimization (GRPO) without auxiliary objectives or additional trainable components. Across MATH-500, AMC, AIME, MinervaMath, and OlympiadBench, PROGRS consistently improves Pass@1 over outcome-only baselines and achieves stronger performance with fewer rollouts. These results show that outcome-conditioned centering enables safe and effective use of process rewards for mathematical reasoning.
| Comments: | 8 pages, 4 figures, 4 tables, accepted in IJCNN 2026 conference |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.02341 [cs.LG] |
| (or arXiv:2604.02341v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.02341 arXiv-issued DOI via DataCite |
Submission history
From: Mohammad Rezaei Ravari [view email]
[v1]
Sun, 8 Feb 2026 06:38:20 UTC (2,802 KB)
[v2]
Tue, 13 Oct 2026 08:25:36 UTC (196 KB)
来源：arXiv：cs.AI · arxiv.org