arXiv：cs.AI· Xue Li, Hongqian Xia, Daniel Gomm, A\'ecio Santos, Cornelius Wolff, Madelon Hulsebos·· 4 小时前
智能体数据科学中的沉默失败：Agentic DS 评估不能只看准确率
The Silent Failures in Agentic Data Science
AI 导读
研究对 220 个数据科学任务中的常见智能体数据科学（agentic DS）框架与模型做大规模失败模式分析，发现多数智能体方案至少存在一种失败模式，与 harness 复杂度或模型能力无关。研究还发现数据泄漏失败的发生率与预测性能呈正相关，并指出 DS 智能体相比 Kaggle 竞赛获胜者普遍缺乏解题创造力和领域知识。作者认为失败模式分析应成为 agentic DS 评估的必要组成部分。
正文
Abstract:Large language models have significantly lowered the barrier to writing code. Naturally, this has also led to the development of data science agents that promise automating the design and execution of data science from task formulation to model development. Existing evaluations of agentic data science (agentic DS) assess downstream predictive performance and code correctness, providing preliminary insight in agentic DS capabilities. We argue that these evaluations overlook the fundamental principles of data science: methodological data science- and domain knowledge. As such, only assessing downstream performance masks methodological failures such as data leakage and inappropriate validation, limiting the development of robust agentic DS systems. In this paper, we evaluate agentic DS beyond accuracy from three different angles: 1) a large-scale analysis of failure modes in common agentic DS frameworks and models across 220 data science tasks, 2) a reusable and extensible test suite with curated data science tasks that test for six specific failures in a controlled manner, and 3) a qualitative assessment of agentic DS pipelines against expert solutions on Kaggle competitions. We find that the majority of agentic solutions contain at least one failure mode despite the complexity of agentic harness or model capacity. Moreover, we identify a positive correlation between the occurrence of data leakage failures and predictive performance, confirming the necessity to evaluate agents beyond accuracy. Our qualitative analysis shows that DS agents generally lack problem solving creativity and domain knowledge, compared to competition-winning data scientists. We believe that principled failure mode analysis, as presented here, should be integral part of agentic DS evaluations, and that the insights we surface will be key in developing trustworthy agentic DS pipelines.
| Comments: | preprint |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14115 [cs.AI] |
| (or arXiv:2610.14115v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14115 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xue Li [view email]
[v1]
Fri, 9 Oct 2026 08:05:09 UTC (1,187 KB)
来源：arXiv：cs.AI · arxiv.org