arXiv：cs.AI· Jie Yang, Mianqiu Huang, Yiwei Zhao, Bo Wang, Yang Liu, Yuxin Wang·· 16 小时前
PEARL：面向多轮智能体的过程经验感知强化学习
PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents
AI 导读
PEARL 是一种无需额外 critic 的强化学习方法，通过定位同一历史下两条不同结果续写的首个分歧点，构建各分支局部决策的特征表示并在近期失败与成功记忆中聚类，识别反复出现的局部模式，从而只选择性增强低回报决策的负优势。
正文
Abstract:Terminal rewards make multi-turn agents trainable without step-level annotations, but group-relative advantages provide little guidance about which intermediate decisions to adjust. Failed trajectories also contain reasonable actions, while unfavorable decision patterns can recur as training proceeds. We introduce PEARL (Process-Experience Aware Reinforcement Learning), which improves local credit assignment without an additional critic. Given two continuations from the same history with different outcomes, PEARL locates their first divergence and builds a feature representation of each branch's local decision. It clusters these decision features in recent failure and success memories to identify recurring local patterns. Using this pattern memory, PEARL selectively strengthens negative advantages at lower-reward decisions, while leaving positive advantages and the base policy optimizer unchanged. On AppWorld, PEARL improves over ARPO on Normal and Challenge tasks at both 4B and 8B scales. On ALFWorld, it achieves 97.9% mean success at 4B, 2.7 points above the strongest baseline, while remaining competitive at 8B. Ablations of shaping location, success-bank filtering, and memory window support the value of selective, recurring-pattern-based updates.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14636 [cs.AI] |
| (or arXiv:2610.14636v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14636 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jie Yang [view email]
[v1]
Sat, 10 Oct 2026 06:35:58 UTC (272 KB)
来源：arXiv：cs.AI · arxiv.org