arXiv：cs.AI· Xianwei Chen, Shimin Zhang, Jibin Wu·· 22 小时前
f-OPD：用新鲜度感知控制稳定长时程在线策略蒸馏
$\boldsymbol{f}$-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Control
AI 导读
针对 LLM 在线策略蒸馏（OPD）中异步执行与理想 on-policy 目标之间的偏差，研究者将其分解为 rollout drift 与 supervision drift，并提出样本级新鲜度分数，据此构建 f-OPD 框架，自适应调节过期样本影响并约束异步训练下的策略漂移。
正文
Abstract:Scaling on-policy distillation (OPD) for large language models (LLMs) confronts a fundamental tension: asynchronous execution is necessary for system efficiency, but structurally deviates from the ideal on-policy objective. To address this challenge, we theoretically decompose the objective discrepancy into rollout drift and supervision drift, capturing staleness in student rollout and teacher context, respectively. Building on this, we introduce a sample-level freshness score that quantifies the reliability of a buffered sample with respect to the on-policy objective. Guided by this signal, we further propose f-OPD, a novel framework that adaptively regulates stale-sample influence and constrains policy drift accumulated under asynchronous training. Across reasoning, tool-use, and coding-agent tasks of increasing interaction horizon, f-OPD consistently achieves task performance comparable to synchronous optimization while largely retaining the throughput advantages of asynchronous execution. Our results establish the first recipe for achieving a performance-efficiency trade-off in OPD, paving the way for long-horizon agentic post-training at scale.
| Comments: | This work has been accepted at NeurIPS 2026. This is the final camera-ready version |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.17862 [cs.LG] |
| (or arXiv:2605.17862v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.17862 arXiv-issued DOI via DataCite |
Submission history
From: Xianwei Chen [view email]
[v1]
Mon, 18 May 2026 05:14:18 UTC (1,069 KB)
[v2]
Sun, 11 Oct 2026 15:23:55 UTC (959 KB)
来源：arXiv：cs.AI · arxiv.org