BTS-AgentBench：从只读遥测日志到智能体基准的可复现确定性流水线

View PDF HTML (experimental)

Abstract:Operational facilities accumulate extensive local telemetry, but agent-ready interactive evaluations with source-verifiable answers are costly to construct from such archives. Site-specific measurement histories also motivate a local conversion workflow that can operate where raw operational data must remain under organizational control. We introduce BTS-AgentBench, a deterministic compiler that transforms read-only telemetry and equipment metadata into tool-backed, multi-turn agent tasks, phase-level targets, evidence and executable evaluation contracts. The BTS release includes 532 episodes across nine families; a second-corpus port produces 204 XAI4HEAT episodes. Frozen-input replay reproduces released artifacts byte-for-byte, and independent raw-source checks cover all 89 BTS and 41 XAI4HEAT test episodes. Replayable provenance links 1,170 trial identities to recorded execution and scorer conditions, while 1,125 completed traces retain model responses and tool results. An obligation-level comparison of 216 scored failures found that linked records exposed additional recorded failed obligations in 202 cases (93.5\%), including tool arguments, stream identity, state carryover and missing interaction steps. The pipeline thereby supports source-grounded task generation, exact reconstruction and auditable diagnosis of agent executions. Code, released benchmarks, and reproduction instructions are available at this https URL

Subjects:	Computation and Language (cs.CL); Software Engineering (cs.SE)
Cite as:	arXiv:2608.27334 [cs.CL]
	(or arXiv:2608.27334v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2608.27334 arXiv-issued DOI via DataCite

Submission history

From: Jeong-Yoon Kim [view email]
[v1] Thu, 27 Aug 2026 16:35:52 UTC (902 KB)
[v2] Wed, 14 Oct 2026 10:35:50 UTC (47 KB)