arXiv：cs.LG· Qinglin Su, Licheng Pan, Zhen Fang, Jiakang Wang, Qingnan Ren, Yunsheng Lu, Yan Gao, Zhixuan Chu·· 4 小时前
RM-OPD：用同策略蒸馏训练生成式奖励模型
RM-OPD: On-Policy Distillation for Generative Reward Models
AI 导读
RM-OPD 是一个两阶段框架，先将单域 GRPO 微调得到的专家教师模型，通过路由式同策略蒸馏、领域贡献重加权和可选的评判标准上下文蒸馏整合进一个学生模型。在多个 RM 基准和两个基座模型上，RM-OPD 持续优于原生 Mix-RL 及其他强基线，并出现"教师超越"效应。该工作旨在解决 Mix-RL 的奖励稀疏与跨域干扰问题，构建通用生成式奖励模型。
正文
Abstract:Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.
| Comments: | 34 pages, 4 figures; includes references and appendices |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.20174 [cs.LG] |
| (or arXiv:2610.20174v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20174 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qinglin Su [view email]
[v1]
Wed, 14 Oct 2026 13:20:55 UTC (717 KB)
来源：arXiv：cs.LG · arxiv.org