arXiv：cs.LG（机器学习，全量分类）· Tong Che, Rui Wu·· 1 天前AI 评分54
arXiv 论文：激励信号可翻转语言模型冻结策略的安全选择
Constitutional Value Potentials: reading and steering internal priority margins in language models
AI 导读
Tong Che 与 Rui Wu 在 arXiv 发表论文，在合成职场环境 MoneyWorld 中用 RL 训练三个家族的五个指令微调模型，发现冻结策略的安全选择随显示的收益信号翻转：信号指向不安全选项时不安全选择率为 100%，隐藏或指向安全选项时为 0%，$1 的数值优势下达 98.6%。
正文
Abstract:Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard naming which action pays best. We measure what that omission hides. In MoneyWorld, a synthetic workplace environment, we train five instruction-tuned models from three families with RL on non-safety tasks in which a visible payoff signal identifies a rewarded shortcut that sacrifices task quality. We then freeze each policy, present held-out safety conflicts, and change only the displayed signal. Each menu contains one compliant action and three violations. We report three findings, with rates for Qwen2.5-14B-Instruct. (i) Payoff signals control frozen safety choices: unsafe choice is 100% when the signal names an unsafe option and 0% when it is hidden or names the safe one. Hidden- and random-signal training controls stay at or below 0.3%, and the switch reproduces on all five bases. Numerical payouts reproduce it under sampled-action rewards, reaching 98.6% unsafe choice at a $1 advantage. (ii) Payoff identification and unsafe choice separate under a training-menu intervention: training on task-completing actions at the same payouts retains 99.8% identification while reducing unsafe choice to 7.9% at matched update budgets. Payoff-reading competence alone does not explain transfer. (iii) The switch does not reproduce in executed retail customer-service tasks using the same frozen adapters. In MoneyWorld, omitting incentive information conceals unsafe choices that appear when the same policy sees which action pays best.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.15420 [cs.LG] |
| (or arXiv:2606.15420v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.15420 arXiv-issued DOI via DataCite |
Submission history
From: Tong Che [view email]
[v1]
Sat, 13 Jun 2026 18:14:23 UTC (108 KB)
[v2]
Thu, 1 Oct 2026 02:30:22 UTC (405 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org