arXiv：cs.LG（机器学习，全量分类）· Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko·· 5 小时前AI 评分51
arXiv 论文提出基于探针训练的对齐方法，在不损失可监控性的前提下提升安全性与诚实性
Alignment via Training Against Probes Without Losing Monitorability
AI 导读
arXiv 论文（arXiv:2609.38645）提出 probe-guided fine-tuning，用检测模型激活中不良属性的探针作为直接训练信号，在无害性与诚实性两个对齐目标上训练。
正文
Abstract:Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without internalizing the intended behavior, for example by faking compliance during training. Such superficial compliance could be harder when the objective is defined on model internals rather than outputs. Therefore, we study probe-guided fine-tuning, using probes that detect undesired properties in model activations as a direct training signal. We evaluate linear and non-linear probes with different numbers of probes per layer across two alignment objectives: harmlessness and honesty. We find that training against probes that do not update during training is an easily exploitable objective, while continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. Probe-guided fine-tuning achieves better safety-utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and abliteration attacks. Moreover, the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by our method. Training against probes thus offers a way to shape what models represent rather than only what they output, which may become increasingly important as models get better at making their outputs look aligned.
| Comments: | 38 pages, 22 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2609.38645 [cs.LG] |
| (or arXiv:2609.38645v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38645 arXiv-issued DOI via DataCite |
Submission history
From: Lena Libon [view email]
[v1]
Tue, 29 Sep 2026 23:03:54 UTC (831 KB)
[v2]
Thu, 1 Oct 2026 13:38:27 UTC (831 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org