arXiv：cs.LG· Huiwen Han, Ziyao Zeng, Zihao Zhu, Sisong Bei, Hao Wang, Hongyu Xu, Yan Han, Degui Zhi, Zhiwen Fan·· 4 小时前
Contextual Credit Policy Optimization（CCPO）：为 LLM 智能体提供上下文信用分配
Contextual Credit Policy Optimization
AI 导读
针对基于组的强化学习中观察匹配同等对待、成功轨迹内冗余动作被掩盖的问题，研究者提出 Contextual Credit Policy Optimization（CCPO），用冻结 LLM 表示计算上下文相似度对同伴基线加权，并结合历史信用与未来信用评估每步进展，无需辅助价值 critic。
正文
Abstract:Group-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long-horizon tasks. Recent methods derive step-level credit by comparing observation-matched experiences across trajectories. However, identical observations can occur in different contexts, and matched peers are not equally informative for a given decision. Moreover, favorable terminal outcomes can obscure redundant actions or detours within successful trajectories. To address these two challenges, we propose Contextual Credit Policy Optimization (CCPO). For the first, CCPO weights observation-matched peers from other trajectories by contextual similarity, computed from frozen LLM representations and accumulated trajectory statistics, allowing peers from similar contexts to contribute more to a step's baseline. For the second, CCPO pairs historical credit, which compares the realized return against the weighted peer baseline, with future credit, which measures the change in contextual outcome value over a short realized continuation. A step is thus credited for the progress it makes, not only for its trajectory's eventual outcome. Both credits are computed in hindsight from rollouts already collected for training, enabling CCPO to address these challenges without an auxiliary value critic. Across two Qwen2.5-Instruct backbones (1.5B and 7B), CCPO achieves up to 97.4% success on ALFWorld and 79.4% on WebShop and outperforms GRPO by up to 21.6 and 18.5 points. Ablations show that contextual weighting outperforms uniform weighting, and that historical and future credit serve as complementary learning signals.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19103 [cs.LG] |
| (or arXiv:2610.19103v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19103 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Huiwen Han [view email]
[v1]
Tue, 13 Oct 2026 19:39:52 UTC (618 KB)
来源：arXiv：cs.LG · arxiv.org