arXiv：cs.AI· Zhengxi Li, Junyi Zhou, Qiyuan Zhang, Chuhan Wu, Fuyuan Lyu, Le Tian, Xiao Zhou, Chen Ma, Yufei Wang·· 16 小时前
SkillForgeBench：以 Agent 技能规范为基础的智能体可靠性评测基准
SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications
AI 导读
研究者提出 SkillForgeBench，用 agent skills 作为统一执行蓝图，构建覆盖五大专业领域的 842 个沙箱任务，用于评测智能体严格满足流程复杂要求的能力。
正文
Abstract:Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability. Conducting such an evaluation demands agentic tasks that reconcile broad domain coverage with deep procedural complexity. Existing paradigms face a fundamental trade-off: expert-curated benchmarks achieve procedural depth but suffer from domain diversity bottlenecks, whereas crowdsourced benchmarks span diverse scenarios but lack execution rigor. To resolve this tension, we present SkillForgeBench, which leverages agent skills as unifying execution blueprints across three co-designed stages: (1) Resource Preparation uses domain-balanced skills to guide artifact retrieval, establishing broad workflow coverage; (2) Skill-driven Task Synthesis employs skills as procedural anchors to partition artifact roles, synthesize naturalistic task instructions, and package sandboxed scenarios for deep procedural execution; and (3) Holistic Evaluation Design translates skill-embedded requirements into fine-grained completion and process rubrics to enable strict requirement-level scoring. Comprising 842 sandboxed tasks across five major professional domains, SkillForgeBench reveals a steep capability hierarchy among eight mainstream agents, led by Opus 4.8 ($40.9\%$ strict completion). Requiring all constrains satisfaction uncovers severe capability bottlenecks across all model tiers that remain masked under individual rubric pass rates ($70.0\%$ to $85.7\%$ vs. $12.6\%$ to $40.9\%$). Crucially, under the external guidance from skills, top-tier agents obtain larger gains than low-tier agents (i.e., $\Delta 19.2\%$ for Opus 4.8 vs. $\Delta 5.8\%$ for Haiku 4.5). Finally, agents satisfy process rubrics far less frequently than completion rubrics (under $3\%$ for Haiku 4.5), proving that completion-only checks overestimate real-world agent reliability.
| Comments: | 38 pages, 9 figures, 7 tables, including appendices. The first two authors contributed equally to this work |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.14918 [cs.AI] |
| (or arXiv:2610.14918v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14918 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junyi Zhou [view email]
[v1]
Sat, 10 Oct 2026 13:40:38 UTC (1,277 KB)
来源：arXiv：cs.AI · arxiv.org