arXiv：cs.LG· Haosen Wang·· 7 小时前
谁在决定 Judge Hacking 的速度？评委能力与策略-评委差距之争
Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap
AI 导读
研究在 GRPO 下用冻结的无参考 LLM judge 训练 0.5B 和 1.5B Qwen2.5 策略（GSM8K），发现 judge hacking 的深度随评委能力变化：平台期与 judge AUC 相关（12 个单元格 Spearman -0.69），同族中更大的评委被欺骗得更浅。
正文
Abstract:Under GRPO against a frozen, reference-free LLM judge, RL exploits the judge's false positives. Does the policy-judge capability gap or the judge itself set how fast and how far? We train 0.5B and 1.5B Qwen2.5 policies on GSM8K against four judges at least as large, and the 1.5B policy also against one smaller judge. In a post-hoc analysis, judge acceptance rose, but no cell of 12 gained more than 0.04 in seed-mean held-out greedy accuracy. A gold-reward control with the same recipe raised the 1.5B policy's accuracy by +0.054 (for the 0.5B policy, gold reward gave no detectable gain either). The depth of judge hacking tracks the judge: the plateau correlates with judge AUC (Spearman -0.69 over 12 cells), and within a family the larger judge is fooled less deeply. For the pace, two pre-specified pairs contradict the gap account: the cell with the larger gap was fooled 3.5x sooner at the pre-specified threshold. With 12 cells, however, we cannot formally separate judge AUC from the gap for either endpoint. A blind test of step-0 predictors on three held-out judges was inconclusive. At 7-8B, judges from three families plateaued between 0.37 and 0.85.
| Comments: | 20 pages, 7 figures. v2: shorter abstract, revised AI-use disclosure, layout fixes. Code, per-step logs and the read answer samples: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.15778 [cs.LG] |
| (or arXiv:2610.15778v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15778 arXiv-issued DOI via DataCite |
Submission history
From: Haosen Wang [view email]
[v1]
Sun, 11 Oct 2026 17:05:39 UTC (252 KB)
[v2]
Wed, 14 Oct 2026 09:42:42 UTC (251 KB)
来源：arXiv：cs.LG · arxiv.org