arXiv：cs.AI· Junjie Chen·· 16 小时前
LLM 智能体基准评测可靠性实证研究：Oracle 缺陷与配置依赖测量
When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
AI 导读
一项对 LLM 智能体基准评测框架的审计发现，奖励函数缺陷会在评分前改变环境状态，经回放修正后 4,320 个 episode 的数据库通过标签从 4,194 降至 2,883，成功率由 94.3% 降至 65.5%。
正文
Abstract:Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such software for reliability defects that score tables cannot reveal.
We introduce an oracle-integrity auditing approach with five state-integrity invariants and three known-answer controls. We evaluate it by injecting seven oracle defect classes across two task domains and a synthetic domain. All injected defect classes are detected, and the repaired oracle has no false positives. Applied to an agent benchmark, the audit reveals a reward-function defect that changes the environment state before post-run scoring. Across 4,320 episodes, replay-based correction changes database-pass labels from 4,194 to 2,883 and reduces measured success from 94.3% to 65.5%.
We also study configuration-dependent measurement effects. A component ablation over 3,240 episodes shows that the agent's policy documents, tool descriptions, and tool error strings influence observed trajectory reliability, with tool descriptions showing the largest exploratory effect (-9.2 pp) and error strings the smallest (-5.3 pp). A second task domain reverses the sign of the language effect, so the result is bounded to the tested configuration.
Finally, we report a portability boundary: the auditing approach cannot validate a second independently implemented benchmark because the required reference terminal state cannot be reconstructed from its recorded API-call log.
The paper releases a replication package containing the auditing suite, fault-injection catalogue, frozen records, deviation logs, and offline analysis entry points without network access, API keys, or model calls.
| Comments: | Submitted to Empirical Software Engineering |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16067 [cs.SE] |
| (or arXiv:2610.16067v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16067 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junjie Chen [view email]
[v1]
Mon, 12 Oct 2026 02:11:43 UTC (122 KB)
来源：arXiv：cs.AI · arxiv.org