arXiv：cs.AI· Yiwei Chen, Amila Weerasinghe, Siddharth Biswal, Utkrisht Rajkumar, Olcay Boz, Taylor Berg-Kirkpatrick, Rose Yu·· 16 小时前
ACED-Bench：评估 LLM Agent 如何获取并依据因果证据行动
ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence
AI 导读
研究者推出 ACED-Bench，一个神经符号基准，用于评估 LLM Agent 能否主动设计实验、解读有限证据并做出因果决策。其主榜单 ACED-CORE 含 180 个任务，另有 150 题的 ACED-QUAL 资格检查。对七款当代 LLM 和两款早期模型的测试显示，Agent 常在错误干预、丢失证据或证据充分后仍不停止等环节失败。
正文
Abstract:Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system rather than merely answer a prompt. A useful evaluation must let agents choose experiments, interpret finite evidence, and make causal decisions, while keeping the environment coherent across repeated observations and interventions and preserving objective ground truth. We introduce ACED-BENCH (Active Causal Evidence and Decision Benchmark), a neuro-symbolic benchmark built around this requirement. The key idea is to separate the readable scientific surface from the formal causal world: LLMs generate natural-language scenarios and task text, while hidden probabilistic graphical models define the causal graph, data-generating process, intervention semantics, validation checks, and oracle answers. This yields scientific worlds that are natural for agents to investigate but exact enough for scalable, programmatic evaluation. The main leaderboard ACED-CORE contains 180 tasks: 60 set-valued causal ancestry/descendancy questions and 120 constrained causal-decision tasks. We also report ACED-QUAL, a 150-question yes/no qualification check using the same hidden-PGM interface. We evaluate seven contemporary LLMs and two earlier-generation models across three active agent designs. ACED-QUAL checks whether agents can use the interface to draw conclusions from evidence, while ACED-CORE tests whether they can decide what evidence to collect and when to commit to an answer. Evidence-ledger analysis localizes failures within the investigation: agents test the wrong intervention, lose relevant evidence, or continue after sufficient evidence has appeared. These results suggest that reliable AI co-scientists need explicit support for experiment design, evidence retention, answer commitment, and stopping.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14381 [cs.AI] |
| (or arXiv:2610.14381v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14381 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yiwei Chen [view email]
[v1]
Fri, 9 Oct 2026 23:01:57 UTC (424 KB)
来源：arXiv：cs.AI · arxiv.org