arXiv：cs.LG· Jiawei Li·· 3 小时前AI 评分50
Fast Models, Slow Evidence：对 LLM Agent harness 的 System-1 决策模型进行配对与自审计评测
Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
AI 导读
论文对开源 Laya 与托管 Jev 两个 System-1 决策模型在 11 个 agent 决策点上做配对评测，基于 18 个公开来源构建 7,283 个基础用例和 6,640 个鲁棒性变体。
正文
Abstract:Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input carries an injection. System-1 decision models answer such questions in a single forward pass with class probabilities, promising large cost and latency savings over LLM calls. We present a paired evaluation of an open-weight (Laya) and a hosted (Jev) System-1 model on 11 agent decision points built from 18 public sources: 7,283 base cases plus 6,640 robustness variants, with byte-identical inputs, paired tests, and cross-hardware and cross-day reproducibility checks. Jev is significantly more accurate on 9 of 11 decision points (+10.8 to +46.0 pp). Neither model beats chance on zero-shot model routing, and they tie on RAG relevance gating. Laya changes 30% of its answers when the option order is reversed and degrades sharply with many or similar candidates (31% at 50 nearest-neighbour tools, vs. 98% for Jev on items with a unique correct tool). We also audit our own pipeline. Three analysis errors and one design confound distorted headline deployment claims: an omitted pre-screen cost (reported 23.9% saving, actual 4.3%), gate accuracy reported as end-to-end quality (58% vs. 98%), in-sample thresholds (5% target, up to 17% held-out misses), and a "channel effect" on injection false positives that vanishes with channel-native content. Two other suspected confounds did not change the conclusions. All cases, raw outputs and analysis code are available at this https URL.
| Comments: | 11 pages, 7 figures. Code and data: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02267 [cs.AI] |
| (or arXiv:2610.02267v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02267 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiawei Li [view email]
[v1]
Thu, 1 Oct 2026 05:57:14 UTC (296 KB)
来源：arXiv：cs.LG · arxiv.org