arXiv：cs.CL· Md Tahmid Rahman Laskar, Gundeep Singh, Xue-Yong Fu, Karol Chang, Kevin Sanders, Shi Zong, Tania Habib, Julien Bouvier Tremblay, Shayna Gardiner, Harsh Saini, Matthias Lee, Elena Khasanova, Quinten McNamara, Shashi Bhushan TN·· 16 小时前
评估 LLM 作为真实客服技能中的智能体工作流执行器
Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills
AI 导读
研究用从真实客服对话中挖掘的重复工作流构建基准，经结构化技能描述、工具映射和 MCP 兼容环境模拟生成多轮工具调用对话。对 17 个模型（1.7B-14B 开源及闭源）做零样本评测，GPT-5.4 大幅领先，QWEN3.5-9B 是最强开源模型，在每位评审下均超过多个闭源基线。10 组评审两两一致的 Kappa 均值仅 0.524，说明评审可靠性需与模型能力同等重视。
正文
Authors:Md Tahmid Rahman Laskar, Gundeep Singh, Xue-Yong Fu, Karol Chang, Kevin Sanders, Shi Zong, Tania Habib, Julien Bouvier Tremblay, Shayna Gardiner, Harsh Saini, Matthias Lee, Elena Khasanova, Quinten McNamara, Shashi Bhushan TN
Abstract:Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this problem through the lens of Skill LLMs: specialized executors that operate within a selected skill context and must complete a narrow, stateful workflow under policy and tool constraints. We present a benchmark construction pipeline that mines recurrent workflows from real-world transcripts, converts them into structured skill descriptions, maps them to tool requirements, simulates environment state with Model Context Protocol (MCP)-compatible mocks, and generates multi-turn tool-using dialogues. Using this benchmark, we conduct a large-scale zero-shot evaluation of 17 language models spanning closed model families and deployment-oriented open-weight models from 1.7B to 14B parameters, each judged independently by five strong LLM judges, plus a deterministic rule-based evaluator that checks tool coverage, hallucinated tool use, and unsupported claims. We find a consistent ranking in which GPT-5.4 leads by a wide margin and QWEN3.5-9B is the strongest open-weight model (below 20B), outperforming several closed baselines under every judge. We also find that judge choice is not a negligible methodological detail: pairwise judge agreement on the pass/fail verdict averages Kappa=0.524 across 10 judge pairs. These results suggest that comprehensive zero-shot evaluation of skill execution requires judge reliability as carefully as model capability.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.14346 [cs.CL] |
| (or arXiv:2610.14346v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14346 arXiv-issued DOI via DataCite |
Submission history
From: Md Tahmid Rahman Laskar [view email]
[v1]
Fri, 9 Oct 2026 21:45:51 UTC (202 KB)
[v2]
Wed, 14 Oct 2026 16:00:21 UTC (203 KB)
来源：arXiv：cs.CL · arxiv.org