arXiv：cs.AI· Xinming Tu (Minta), Tianze Wang (Minta), Yingzhou (Minta), Lu, Kexin Huang, Yuanhao Qu, Sara Mostafavi·· 4 小时前AI 评分54
BenchGuard：用 LLM 自动审计 LLM Agent 基准
Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
AI 导读
研究者提出 BenchGuard，首个面向执行式 Agent 基准的跨工件联合审计框架，用前沿 LLM 系统性核查基准本身的缺陷。在 ScienceAgentBench 上发现 12 个作者确认的问题，包括导致任务不可解的致命错误；在 BIXBench Verified-50 子集上与专家发现的问题匹配率达 83.3%，并捕获此前人工审查完全遗漏的缺陷。
正文
Abstract:As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all---they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this vision through BenchGuard, the first framework explicitly designed for joint cross-artifact auditing of execution-based agent benchmarks. BenchGuard cross-verifies all benchmark artifacts via structured LLM protocols, optionally incorporating agent solutions or execution traces as additional diagnostic evidence. Deployed on two prominent scientific benchmarks, BenchGuard identified 12 author-confirmed issues in ScienceAgentBench---including fatal errors rendering tasks unsolvable---and exactly matched 83.3% of expert-identified issues on the BIXBench Verified-50 subset, catching defects that prior human review missed entirely. A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated benchmark auditing a practical and valuable complement to human review. A preliminary native-format audit of ProgramBench further demonstrates cross-format applicability. These findings point toward AI-assisted benchmark development, where frontier models serve not only as subjects of evaluation but as active participants in validating the evaluation infrastructure itself.
| Comments: | Camera-ready version for COLM 2026. 24 pages |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2604.24955 [cs.CL] |
| (or arXiv:2604.24955v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.24955 arXiv-issued DOI via DataCite |
Submission history
From: Xinming Tu [view email]
[v1]
Mon, 27 Apr 2026 19:51:25 UTC (1,562 KB)
[v2]
Fri, 2 Oct 2026 03:37:56 UTC (1,573 KB)
来源：arXiv：cs.AI · arxiv.org