arXiv：cs.LG· Pranay Kothari·· 3 小时前AI 评分51
ArrivalBench：Agent 生成的数据管道单次验证正确、随时间推移出错
ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time
AI 导读
作者提出 ArrivalBench 基准，不再用固定快照单次运行给 Agent 生成的数据管道打分，而是在延迟、重复、乱序和重试等可重放的对抗性投递调度下重新执行管道，并要求最终状态等于对完整日志的批量重算。
正文
Abstract:Benchmarks for agent-generated data work grade a pipeline by running it once against a fixed snapshot. ArrivalBench instead re-executes the pipeline an agent leaves behind under adversarial but replayable delivery schedules (late, duplicated, out-of-order and retried records) and requires its final state to equal a batch recomputation of the complete log. Because the oracle recomputes rather than classifies, a wrong table and a crash are distinct verdicts: a crash is visible to monitoring a team already runs, and a wrong table is not. On 40 tasks we built, our reimplementation of single-execution grading certifies 86-100% of the pipelines eleven models produce; re-executing the same artifacts finds 7.0-79.2% of the certified ones silently wrong. The gap is not produced by the repair loop: within the same model and task, pipelines repaired against the snapshot test fail replay about as often as those that passed it first time. In every model, idempotency hazards fail more often than ordering hazards. Separating a wrong answer from a crash also changes how interventions read: a hazard warning cuts one model's silent failure from 48.2% to 10.5% while raising its crash rate from 9.0% to 37.0%, so all-in failure moves only from 51.0% to 44.0%. All eleven arms were independently re-run, and rates moved by at most 5.9 points.
| Comments: | Accepted as a poster at the NeurIPS 2026 Workshop "Who Verifies the Agents?". 20 pages |
| Subjects: | Machine Learning (cs.LG); Databases (cs.DB) |
| Cite as: | arXiv:2610.02363 [cs.LG] |
| (or arXiv:2610.02363v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02363 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pranay Kothari [view email]
[v1]
Thu, 1 Oct 2026 18:39:06 UTC (140 KB)
来源：arXiv：cs.LG · arxiv.org