arXiv：cs.AI· Haoze Li, Peng Zhou, Peiyuan Wu, Weijie Zhao, Yinan Xin·· 17 小时前
VH-GPO：面向 LLM 智能体强化学习的验证式交接方法
VH-GPO: Verified Handoff for LLM Agent Reinforcement Learning
AI 导读
VH-GPO 通过让冻结教师模型在环境验证下修复学生失败轨迹，并在最早可独立确认的交接点交还控制权，使成功经验重新由学生生成。在 Qwen2.5-1.5B 上，ALFWorld 成功率从 GRPO 的 59.38% 提升至 94.79%，WebShop 从 71.62% 提升至 77.34%。
正文
Abstract:Group-relative reinforcement learning methods such as GRPO train large language model (LLM) agents by comparing rollouts for the same task. When every trajectory in a group fails with the same reward, relative advantages collapse toward zero, leaving no useful learning signal. A frozen teacher can recover from a student-visited state, but letting it complete the task makes the recovered suffix entirely teacher-generated. We ask: after repairing a student failure, when should control return to the student? We propose outcome-verified teacher-student handoff, instantiated in Verified Handoff for Group Policy Optimization (VH-GPO). The teacher provides enough support to reach a state from which the student can succeed, then returns control so that successful experience is again student-generated. Given an environment-verified teacher recovery, VH-GPO selects the earliest independently confirmed handoff point found within a bounded search budget. With Qwen2.5-1.5B, VH-GPO increases success rates over GRPO from 59.38% to 94.79% on ALFWorld and from 71.62% to 77.34% on WebShop. Improvements also hold with the 7B student, for which additional teacher and student output tokens amount to 1.77% and 2.61% of regular student rollout output tokens on ALFWorld and WebShop, respectively. Ablation studies demonstrate the effectiveness of the key components. These results support using verified teacher assistance to repair failures while reinforcing the student's ability to complete tasks independently.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14000 [cs.LG] |
| (or arXiv:2610.14000v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14000 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haoze Li [view email]
[v1]
Sun, 27 Sep 2026 05:25:37 UTC (437 KB)
来源：arXiv：cs.AI · arxiv.org