arXiv：cs.LG· Du Cheng, Junyu Lu, Jingfan Zhang, Yuzhuo Zhang, Haoxuan Jia, Shichao Weng·· 11 小时前
CBPO：用共享回归目标统一离线示范与在线/离策略 rollout 的学习
Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target
AI 导读
研究者提出 Cross-Behavior Policy Optimization（CBPO），通过固定参考目标让离线示范与策略生成的 rollout 共用同一回归损失与回放缓冲区，无需协调 SFT 与 RL 目标，也不需要行为策略概率、重要性加权或学习 critic。
正文
Abstract:Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses. We propose \textbf{Cross-Behavior Policy Optimization} (CBPO), which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer. As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them. Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading \texttt{avg@32} and \texttt{pass@32} performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged \texttt{avg@32} and \texttt{pass@32} over SFT DAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12830 [cs.LG] |
| (or arXiv:2610.12830v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12830 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Du Cheng [view email]
[v1]
Fri, 9 Oct 2026 00:29:53 UTC (607 KB)
来源：arXiv：cs.LG · arxiv.org