arXiv：cs.AI· Ramesh Arvind Naagarajan, Stefan Streif·· 16 小时前
结构基线对智能体失败检测与归因的影响
Structural Baselines Matter for Agent Failure Detection and Attribution
AI 导读
一项 arXiv 研究指出，LLM 智能体失败检测与归因任务的报告性能多来自无需模型的规律：失败运行更长，被标为决定性步骤的通常是最后一步，一条"总选最后一步"的规则在此处正确率达 67% 至 96%，在回放可确认的失败上则降至接近随机。
正文
Abstract:An LLM agent using tools can fail silently. It stops before the task is done, raises no error, and answers fluently. Two kinds of monitor address this, one deciding whether a run failed and the other naming the step where failure became unavoidable. Much of the reported performance on both tasks comes from regularities that need no model. Failing runs are longer, and the step labelled decisive is usually the last. A rule that always names the last step is correct $67\%$ to $96\%$ of the time here, without reading the run at all. That ceiling reflects how the labels were written, not how agents break. On failures whose decisive step replay can confirm, it falls to near chance. Reporting gain over that rule therefore changes what a score means. Under that correction, methods tested here gain nothing on give-up failures, while several published systems keep a clear advantage on other splits. For detection, the agent's internal states beat transcript length and text similarity on the agent benchmark, and hold up where the text baseline does not. Precision stays low when failures are rare, so such a monitor should rank runs for review, not replace it.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16661 [cs.AI] |
| (or arXiv:2610.16661v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16661 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ramesh Arvind Naagarajan Mr [view email]
[v1]
Mon, 12 Oct 2026 12:13:48 UTC (144 KB)
来源：arXiv：cs.AI · arxiv.org