arXiv：cs.LG· Minchan Kim, Sungho Bae, Youngjin Kim, Hyunwoo Park·· 2 小时前
GRPO 长程 LLM 智能体课程学习研究：起始位置与推进规则如何影响任务完成
Where to Start, How to Move: Analyzing Curricula under GRPO for Long-Horizon LLM Agents
AI 导读
针对 GRPO 在长程任务中全失败组缺乏奖励信号的问题，Guide Assisted GRPO 通过在学习者继续前回放成功参考前缀来提供学习信号。研究在 BFCL 上测试三种模型规模，发现在固定回合级阈值规则下，从接近完成处开始更有效；基于结果信息的工具调用选择在已训练和未训练任务类别上均优于 GRPO，引导式起点也在 AppWorld 两个评测划分上提升了成功率。
正文
Abstract:Group Relative Policy Optimization (GRPO) provides no reward-relative learning signal when all rollouts receive identical rewards. In long-horizon tasks that require multiple interactions, rewards are given only upon successful task completion, making all-fail groups common early in training and limiting opportunities to practice later steps. Guide Assisted GRPO replays a successful reference prefix before the learner continues. We study how initial positions, handoff units, and movement rules affect unassisted task completion. Across three model sizes on BFCL, starting near completion is more effective under a fixed turn-level threshold rule. Finer handoffs alone do not consistently help, but outcome-informed tool-call selection outperforms GRPO on trained and untrained task categories. Guided starts also improve AppWorld success on both evaluation splits. These findings highlight where learning starts and how starting positions move as key curriculum choices.
| Comments: | 7 pages, 4 figures, 4 tables. Minchan Kim and Sungho Bae contributed equally |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19349 [cs.LG] |
| (or arXiv:2610.19349v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19349 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Minchan Kim [view email]
[v1]
Wed, 14 Oct 2026 00:42:53 UTC (205 KB)
来源：arXiv：cs.LG · arxiv.org