arXiv：cs.AI· Huaiming Li, Can Huang, Chufan Wu, Danyu Liu, Xiaomin Yuan, Qinze Li, Yui Lo, Wenhui Bai, Qianlong Wang, Zengle Ge, Xiyu Yan, Yuheng Cheng, Jinhui Ren, Guokai Chen, Jiaqun Liu, Haoran Li, Qianhui Liu, Annan Li, Jianmin Wu, Dawei Yin, Dou Shen·· 16 小时前
FDE-Bench：评估智能体能否完成需求不明确真实业务任务的端到端交付
FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
AI 导读
研究者推出 FDE-Bench，首个基于真实客户项目构建的 benchmark，用于评估智能体在需求不明确业务任务中的端到端交付能力，包含 49 个组合优化与机器学习案例。在 18 种智能体配置中，最高平均交付得分仅 0.5510（FDE 参考为 1.0），最高交付成功率（DSR）仅 6.12%，表明可靠的前线交付工程师级端到端交付对现有智能体仍不可及。
正文
Authors:Huaiming Li, Can Huang, Chufan Wu, Danyu Liu, Xiaomin Yuan, Qinze Li, Yui Lo, Wenhui Bai, Qianlong Wang, Zengle Ge, Xiyu Yan, Yuheng Cheng, Jinhui Ren, Guokai Chen, Jiaqun Liu, Haoran Li, Qianhui Liu, Annan Li, Jianmin Wu, Dawei Yin, Dou Shen
Abstract:Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks. We introduce FDE-Bench, the first benchmark built from real customer projects to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization. Its 49 cases span combinatorial optimization and machine learning. Clarification targets are distilled from discussions between frontline practitioners and FDEs. Agents elicit missing requirements from a simulated customer, and case-specific evaluators assess final artifacts against acceptance criteria using customer-accepted FDE solutions as quality references. Across 18 agent configurations, the highest mean delivery score is only 0.5510, compared with an FDE reference of 1.0. The highest delivery success rate (DSR), defined as the fraction of planned runs reaching this reference, is only 6.12%. These results show that reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents on underspecified real-world business tasks. The benchmark project page is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2610.14539 [cs.AI] |
| (or arXiv:2610.14539v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14539 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Can Huang [view email]
[v1]
Sat, 10 Oct 2026 03:53:42 UTC (983 KB)
来源：arXiv：cs.AI · arxiv.org