arXiv：cs.AI· Ruolin Chen, Yang Li, Jihang Wang, Zhuoya Zhao, Yuchen Liu, Jiaming Xu, Qian Zhang·· 19 小时前
PTGPO：前缀树分组策略优化
PTGPO: Prefix Tree Group Policy Optimization
AI 导读
研究者提出 PTGPO（Prefix Tree Group Policy Optimization），一种无需 critic 的方法，在 anchor-state 分组内对轨迹后缀构建前缀树，用每个树边子节点与父节点经验平均回报之差来分配信用。
正文
Abstract:Training language agents on long-horizon interactive tasks is challenging because sparse terminal rewards provide weak credit signals for intermediate actions. Existing group-based methods either assign a shared trajectory-level advantage or estimate step-level advantages from state groups, which may conflate the current action with subsequent decisions. We propose Prefix Tree Group Policy Optimization (PTGPO), a critic-free method that constructs prefix trees over trajectory suffixes within anchor-state groups. PTGPO assigns each tree edge a credit given by the difference between its child and parent empirical mean returns, capturing the conditional return change revealed by each additional action. These prefix-tree credits provide fine-grained learning signals without requiring a critic or additional rollouts. Experiments on interactive benchmarks ALFWorld, WebShop and Sokoban show that PTGPO consistently improves over RL baselines, demonstrating its effectiveness.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14737 [cs.LG] |
| (or arXiv:2610.14737v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14737 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jihang Wang [view email]
[v1]
Sat, 10 Oct 2026 08:57:33 UTC (2,155 KB)
来源：arXiv：cs.AI · arxiv.org