arXiv：cs.AI· Tianyang Xu, Jing Gao·· 22 小时前
审计失败归因：标签可预测性与评判行为
Auditing Failure Attribution: Label Predictability and Judge Behavior
AI 导读
针对失败归因基准，论文提出四项测试以区分 LLM 评判者是真正核查出错步骤还是依赖表面线索。在三个基准上，简单预测器即可匹配或超越 GPT-4.1；在 Who&When Pro 的两个系统中，仅读单步推理的分类器能找出 95-100% 的出错步骤。在 Aegis 上，指责所有智能体在 micro-F1 上追平 GPT-4.1，但 exact-set accuracy 落后 19 分。
正文
Abstract:Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure. A high score should mean the judge checked the faulty step against the run (a cross-step check). However, spurious signals (surface cues) that predict the annotation can produce a high score. In this paper, we propose four tests to separate checking from cue use. Input restriction asks whether partial input predicts the annotation. Cue following checks whether moved cues change answers. Residual accuracy scores judges where cue-only predictors fail. Aggregation asks whether the failure mix or metric changes rankings. Real observations motivate these tests: on three benchmarks, simple predictors match or beat GPT-4.1. In two Who&When Pro systems, a classifier reading one step's reasoning finds 95-100% of faulty steps, motivating input restriction. In one of them, error injection leaves no reasoning after the faulty step; judges fine-tuned on its labels beat GPT-4.1 by 41 points. Adding task-related reasoning after that step moves this cue: fine-tuned judges lose 31-99 points, but zero-shot judges lose at most 9, motivating cue following. On Aegis, blaming every agent matches GPT-4.1 on micro-F1 but trails by 19 points in exact-set accuracy, motivating aggregation. We recommend reporting these tests when developing failure-attribution benchmarks.
| Comments: | 26 pages |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.17747 [cs.CL] |
| (or arXiv:2610.17747v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17747 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianyang Xu [view email]
[v1]
Tue, 13 Oct 2026 05:26:33 UTC (142 KB)
来源：arXiv：cs.AI · arxiv.org