arXiv：cs.LG· Mohamad Amin Mohamadi, Tianhao Wang, Zhiyuan Li·· 7 小时前AI 评分45
用强化犹豫训练可信语言模型：让「我不知道」成为一等训练目标
Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation
AI 导读
针对前沿模型在 GSM8K、MedQA、GPQA 上几乎从不弃答、即便面临严重惩罚警告也照答不误的问题，研究者提出 Reinforced Hesitation（RH），将 RLVR 的二元奖励改为三元奖励（+1 正确、0 弃答、-λ 错误）。
正文
Abstract:Modern language models fail a fundamental requirement of trustworthy intelligence: knowing when not to answer. Despite achieving impressive accuracy on benchmarks, these models produce confident hallucinations, even when wrong answers carry catastrophic consequences. Our evaluations on GSM8K, MedQA and GPQA show frontier models almost never abstain despite explicit warnings of severe penalties, suggesting that prompts cannot override training that rewards any answer over no answer. As a remedy, we propose Reinforced Hesitation (RH): a modification to Reinforcement Learning from Verifiable Rewards (RLVR) to use ternary rewards (+1 correct, 0 abstention, -$\lambda$ error) instead of binary. Controlled experiments on logic puzzles reveal that varying $\lambda$ produces distinct models along a Pareto frontier, where each training penalty yields the optimal model for its corresponding risk regime: low penalties produce aggressive answerers, high penalties conservative abstainers. The same frontier holds on MATH Levels 4--5 and on medical QA, where it transfers to an unseen dataset. We then introduce two inference strategies that exploit trained abstention as a coordination signal: cascading routes queries through models with decreasing risk tolerance, while self-cascading re-queries the same model on abstention. Both outperform majority voting with lower computational cost. These results establish abstention as a first-class training objective that transforms ``I don't know'' from failure into a coordination signal, enabling models to earn trust through calibrated honesty about their limits.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2511.11500 [cs.LG] |
| (or arXiv:2511.11500v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2511.11500 arXiv-issued DOI via DataCite |
Submission history
From: Mohamad Amin Mohamadi [view email]
[v1]
Fri, 14 Nov 2025 17:20:45 UTC (1,047 KB)
[v2]
Fri, 21 Nov 2025 19:15:16 UTC (1,060 KB)
[v3]
Tue, 6 Oct 2026 05:40:49 UTC (726 KB)
来源：arXiv：cs.LG · arxiv.org