arXiv：cs.AI· Sixun Dong, Yebowen Hu, Ming Yin, Yiran Chen, Fei Liu, Chen Chen·· 23 小时前
FlipAct：测量语言模型何时采取行动
Sensitivity or Criterion? Measuring and Moving When Language Models Act
AI 导读
研究者提出 FlipAct，一种针对物理状态下工具使用的诊断方法，用标注工具属性上的规则计算答案与分级证据分数，将 Yes/No 边界固定为零。在 44 个开源模型上，敏感度随同一模型族内的规模上升，其排名与四个外部 when-to-act 基准中的三个一致。
正文
Abstract:Before an agent acts, it must decide whether action is warranted. A single accuracy cannot distinguish failure to recognize a warranted action from reluctance to act. Signal-detection theory separates sensitivity from criterion, but criteria estimated from authored binary labels depend on each benchmark's example mix. We introduce FlipAct, a diagnostic of tool use in physical states. A rule over annotated tool properties computes each answer and a graded evidence score, fixing the Yes/No boundary at zero across models. Questions form 2-by-2 quads that hold required knowledge fixed while the state flips. The slope of the fitted Yes-rate curve measures sensitivity; its crossing at one half measures criterion. Across 44 open models, sensitivity rises with size within a family, and its ranking agrees with three of four external when-to-act benchmarks. Models respond to evidence sign rather than magnitude, accepting actions that leave the state no better about as often as helpful ones. When Yes means act, most models are conservative across rephrasings; in some, this partly reflects a preference for No. Criterion is not determined by size. Activation steering shifts criterion with weights fixed; distillation from a model's own steered copy puts the shift into its weights without a significant sensitivity change. The distilled shift transfers to six external benchmarks, including two without Yes/No answers, while criterion-free AUC is preserved on the five benchmarks where it is available. Accuracy improves where models start conservative and declines or remains within error where they start permissive. Project page: this https URL
| Comments: | Project page: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.15415 [cs.CL] |
| (or arXiv:2610.15415v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15415 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sixun Dong [view email]
[v1]
Sun, 11 Oct 2026 07:36:38 UTC (244 KB)
来源：arXiv：cs.AI · arxiv.org