arXiv：cs.AI· Yunjie Chen, Zihao Chen, Xiaoxin Chen, Fan Wang·· 22 小时前
REGEN：用回放回收与离线 RL 实现专家到通才的蒸馏
REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning
AI 导读
REGEN 提出用回放回收做专家到通才蒸馏的离线 RL 方法，直接复用教师模型专项 RL 训练产生的回放记忆来训练通才模型，完全解耦采样与反向传播以大幅降低训练成本。在数学推理、代码生成和指令跟随任务上，REGEN 以显著更低的成本达到与 MOPD 相当的准确率，并有望将在线 RL 从一次性学习阶段转变为数据合成过程。代码已开源。
正文
Abstract:Large-scale online reinforcement learning (RL) is the predominant means of eliciting advanced abilities including long-term reasoning and agentic tool use in large language models (LLMs). However, continuing to scale it across vast task domains of interest remains challenging in both computational infrastructure and cost, especially when considering RL as merely a one-off learning stage. Recently, a widely used technique for distilling knowledge across various domains and training stages, multi-teacher on-policy distillation (MOPD), helps to decouple the RL stage, saving costs, while maintaining generality across vast domains. Nonetheless, similar to online RL, MOPD requires coupled inference and backward passes, which continues to limit its scalability and computational efficiency. To address these challenges, we propose REGEN: Replay-recycling for Expert-to-Generalist Distillation with Offline RL. Instead of distilling from multiple teacher models, REGEN trains a generalist by simply recycling the replay memory -- the free by-product of the teachers' specialized RL training -- and employing offline RL algorithms. REGEN completely decouples the rollout sampling from the backward training process and thus greatly reduces the training cost. Across mathematical reasoning, code generation, and instruction following, REGEN matches the accuracy of MOPD at substantially lower cost. It potentially turns online RL into a data synthesis process instead of a one-off learning stage, and can be extended to large-scale post-training without requiring heavy computational load. Code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.19450 [cs.LG] |
| (or arXiv:2607.19450v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.19450 arXiv-issued DOI via DataCite |
Submission history
From: Yunjie Chen [view email]
[v1]
Tue, 21 Jul 2026 13:36:23 UTC (1,425 KB)
[v2]
Wed, 29 Jul 2026 12:12:34 UTC (1,425 KB)
[v3]
Sat, 12 Sep 2026 15:22:15 UTC (430 KB)
[v4]
Mon, 12 Oct 2026 14:09:54 UTC (430 KB)
来源：arXiv：cs.AI · arxiv.org