arXiv：cs.AI· Huining Yuan, Zelai Xu, Huaijie Wang, Xiangmin Yi, Jiaxuan Gao, Xiao-Ping Zhang, Yu Wang, Chao Yu, Yi Wu·· 19 小时前
面向智能体推理的可验证过程奖励框架 VPR
Verifiable Process Rewards for Agentic Reasoning
AI 导读
研究者提出可验证过程奖励（VPR）训练框架，将任务求解器改造为动作级过程验证器，为智能体推理的中间动作提供密集奖励，实现细粒度信用分配。VPR 在 Sokoban、Sudoku、Minesweeper 三类环境中实例化，配合 oracle 引导的状态分组 rollout 与局部归一化优化，训练出的 Qwen3-4B 在三个环境中均优于结果奖励 GRPO 和 VinePPO。
正文
Abstract:Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet existing approaches predominantly rely on sparse outcome-level feedback, posing a substantial credit-assignment challenge in long-horizon agentic tasks. In this work, we study structured agentic reasoning problems that admit symbolic or algorithmic task-solving oracles. We propose Verifiable Process Rewards (VPR), a training framework that repurposes such task oracles as action-level process verifiers, providing dense rewards for intermediate actions and enabling fine-grained credit assignment. We instantiate VPR in three distinct settings: a search-based oracle for Sokoban, a constraint-based oracle for Sudoku, and a posterior-based oracle for Minesweeper. To optimize policies with these rewards, we introduce oracle-guided state-group rollout and locally normalized optimization. Empirically, VPR-trained Qwen3-4B outperforms outcome-reward GRPO and VinePPO across all three training environments. When jointly trained on math data and all three game environments, VPR achieves the highest average general-reasoning performance and strongest agentic results among the compared training configurations. To further assess its applicability to more general agentic tool-use settings, we apply VPR-style action-level supervision to $\tau^2$-bench using an expert LLM as a reference policy and observe improved performance over outcome-reward GRPO on held-out tasks. Together, these results position VPR as a promising approach for leveraging reliable task oracles to improve agentic reasoning, while highlighting its dependence on the availability and reliability of task oracles and the challenge of extending intermediate verification to less structured, open-ended environments.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.10325 [cs.AI] |
| (or arXiv:2605.10325v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.10325 arXiv-issued DOI via DataCite |
Submission history
From: Huining Yuan [view email]
[v1]
Mon, 11 May 2026 10:30:53 UTC (2,152 KB)
[v2]
Wed, 27 May 2026 13:12:06 UTC (2,152 KB)
[v3]
Mon, 12 Oct 2026 08:50:38 UTC (2,506 KB)
来源：arXiv：cs.AI · arxiv.org