arXiv：cs.AI· Hao Li, Zhengkun Zhang, Gangqiang Hu, Zhen Zhang, Yude Gao, Dai Dai, Jing Liu·· 13 小时前
RLVR²：基于可验证评分标准的排序强化学习
RLVR$^{2}$: Reinforcement Learning with Verifiable Rubric-based Ranking
AI 导读
针对多维度评分标准下 RLVR 的聚合难题，研究者提出 RLVR²，将各评分标准的分数转为组内序数结果，从中恢复潜在效用并合并为单一训练信号，避免了对异质评分尺度的校准。在 3 种模型规模和 16 个基准上，RLVR² 均优于代表性评分标准基线，在多数基准上取得最佳整体表现。分析显示它能控制与推理效率和回复格式相关的系统性影响，同时保持质量目标。
正文
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) is expanding from tasks with well-defined correctness signals, such as mathematics and code, toward multifaceted quality requirements specified by multi-dimensional rubrics. Since policy optimization consumes one scalar per rollout, rubric-based pipelines must map multiple criterion scores into a scalar reward. This aggregation is often treated as score scaling, but it implicitly determines how quality dimensions trade off during training. The prevailing practice, normalizing each criterion and taking a linear combination, assumes that cardinal score differences are comparable across criteria and that gains on one criterion compensate for failures on another; both assumptions are unreliable when criteria are semantically heterogeneous. We propose Reinforcement Learning with Verifiable Rubric-based Ranking (RLVR$^2$), a verifiable ranking paradigm for rubric-based RLVR. For each criterion, RLVR$^2$ converts rubric scores into criterion-specific within-group ordinal outcomes, recovers a latent utility from the resulting comparison matrix, and merges these utilities into one training signal. By retaining only within-group ordering and discarding raw score magnitudes, RLVR$^2$ avoids calibrating heterogeneous rubric scales. It further supports objective-preserving attribute adjustment: auxiliary attributes that correlate with observed rankings but are not training objectives can enter the estimation without expanding the rubric or rewarding them directly. Across three model scales and 16 benchmarks, RLVR$^2$ consistently outperforms representative rubric-based baselines, achieving the best overall performance on most benchmarks at every scale. Analysis shows it controls systematic effects tied to reasoning efficiency and response formatting while preserving the quality objective.
| Comments: | Preprint |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.23457 [cs.LG] |
| (or arXiv:2609.23457v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.23457 arXiv-issued DOI via DataCite |
Submission history
From: Hao Li [view email]
[v1]
Sun, 20 Sep 2026 08:34:28 UTC (2,132 KB)
[v2]
Fri, 9 Oct 2026 10:31:18 UTC (5,400 KB)
来源：arXiv：cs.AI · arxiv.org