HuggingFace Daily Papers·· 1 天前
诊断并修正可验证指令遵循中的奖励黑客问题
Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following
AI 导读
可验证指令遵循基准只考察输出约束满足度，前沿模型可凭缺乏实质回答内容获得满分，将其用作训练奖励会诱发奖励黑客。CALIF 用现有冻结参考策略在仅给定请求下评估回答似然，为验证器奖励加入校准后的似然奖励，无需额外评判器或奖励模型。在 1.7B 至 8B 的推理与非推理模型上，CALIF 提升了 IFBench 的约束满足度，并在三个基于评分标准的基准上于 45 项模型-指标对比中胜出 40 项。
正文
Abstract:Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.
| Comments: | 15 pages |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19525 [cs.LG] |
| (or arXiv:2610.19525v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19525 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sangyeon Yoon [view email]
[v1]
Wed, 14 Oct 2026 03:42:59 UTC (1,629 KB)
来源：HuggingFace Daily Papers · arxiv.org