CorrGRPO：面向多奖励学习的相关性归一化 GRPO
CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning
研究者提出 CorrGRPO，将 GRPO 多奖励场景下的成对协方差归一化为 Pearson 相关系数，在保持中心化总奖励不变的同时，避免大尺度奖励主导归一化、压制小尺度奖励信号。在代码生成、工具调用和智能体安全三类任务上，用 0.5B 至 8B 参数模型与 GRPO 及其他变体对比，三个领域均取得提升。代码已开源。
Published on Sep 29
Authors:
,
,
,
,
Abstract
Group Relative Policy Optimization (GRPO) is widely used to train reasoning language models, where it computes advantages by centering and normalizing rewards across rollouts of the same prompt. For multiple rewards, GRPO sums the reward components and normalizes the total reward by its within-group standard deviation. The corresponding variance equals the sum of all pairwise reward covariances. For a fixed centered reward, larger aggregate covariance produces smaller advantages, and vice versa, allowing update magnitudes to adapt to reward dependence. However, correlated rewards with large scales can dominate this normalization and suppress signals from smaller-scale rewards. We propose Correlation-Normalized GRPO (CorrGRPO), which normalizes pairwise covariances into Pearson correlation coefficients. CorrGRPO keeps the centered total reward unchanged while balancing the influence of differently scaled rewards on the correlation-based normalization. This allows advantage magnitudes to adapt to reward correlations without the normalization being dominated by large-scale reward components. We compare CorrGRPO with GRPO and other variants on code generation, tool calling, and agent security, using models ranging from 0.5B to 8B parameters. These tasks all involve multiple rewards that can improve together or present tradeoffs. Results show improvements across three domains, including code generation, tool calling, and agent security. Our code is available at https://github.com/HKUST-KnowComp/CorrGRPO.
View arXiv page View PDF Project page GitHub 0 Add to collection
Get this paper in your agent:
hf papers read 2609.36820
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.36820 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.36820 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.36820 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
来源：HuggingFace Daily Papers（社区热门论文） · huggingface.co