arXiv：cs.AI· Jiayi Wu, Ruobing Xie, Zeqian Huang, Lei Jiang, Can Xu, Kangyang Luo, Bochen Lin, Ming Gao, Xiang Li·· 19 小时前
CalibAdv：校准 GRPO 中负优势，提升搜索智能体训练稳定性
Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents
AI 导读
针对 GRPO 在多跳搜索中因优势分配粗糙、正负优势失衡导致的训练不稳定问题，研究者提出 CalibAdv 优势校准方法。它利用中间步骤正确性在细粒度上缩减过大的负优势，并重新平衡正负优势。在三个模型和七个 benchmark 上，CalibAdv 同时提升了性能与训练稳定性，代码已开源，论文被 EMNLP 2026 Findings 接收。
正文
Abstract:Search agents achieve strong question-answering performance through multi-turn interactions with search engines, with Group Relative Policy Optimization (GRPO) being a widely used training algorithm. However, GRPO-style algorithms still face several challenges in multi-hop search settings. First, correct intermediate steps are often penalized when the final answer is wrong. Second, training is highly unstable, often causing degradation of natural language ability or even catastrophic training collapse. Our analysis attributes these issues to coarse-grained advantage assignment and an imbalance between positive and negative advantages. To address these problems, we propose CalibAdv, an advantage calibration method specifically designed for search agents that enables more accurate and more stable modeling of penalties and rewards. Specifically, CalibAdv leverages the correctness of intermediate steps to downscale excessive negative advantages at a fine-grained level. It then further rebalances positive and negative advantages to improve training stability. Importantly, CalibAdv adopts a lightweight design that calibrates advantages from standard rollout signals, making it simple and easy to deploy. Extensive experiments across three models and seven benchmarks demonstrate that CalibAdv improves both model performance and training stability. Our code is available at this https URL.
| Comments: | Accepted by EMNLP 2026 Findings |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.18235 [cs.CL] |
| (or arXiv:2604.18235v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.18235 arXiv-issued DOI via DataCite |
Submission history
From: Jiayi Wu [view email]
[v1]
Mon, 20 Apr 2026 13:21:19 UTC (1,503 KB)
[v2]
Wed, 27 May 2026 15:09:35 UTC (1,529 KB)
[v3]
Mon, 12 Oct 2026 12:48:46 UTC (1,533 KB)
来源：arXiv：cs.AI · arxiv.org