arXiv：cs.AI· Chengyang Shi, Xianglin Ji, Jintao Huang, Jicheng Wang, Yifeng He, Jiachen Liu·· 5 小时前AI 评分53
OEB 基准：从智能体执行记录评估其认知研究过程
Open-Endedness Bench: Measuring Epistemic Process from Agent Records
AI 导读
arXiv 论文（arXiv:2610.02588）提出 OEB（Open-Endedness Bench），一种不依赖参考答案或结果分数、只读智能体执行记录的评估方法：将记录编译为认知事件图，按证据、实验、修正和不作弊四个维度打分，并刻画六项主观人格特质。
正文
Abstract:Agents are increasingly given open-ended research tasks: discovering an empirical law from self-designed experiments, improving a heuristic whose optimum nobody knows, or beating a standing record. Their execution logs record every step of this research, yet the runs are still judged by their outcome score. That score alone does not establish whether an agent's claims follow from executed experiments, and a reference answer may be unavailable. We evaluate the agent's epistemic process: how it forms hypotheses, tests them, and revises them in response to evidence. We introduce OEB (Open-Endedness Bench), a benchmark-agnostic methodology that reads only the agent's execution record and never a reference answer or an outcome score. OEB compiles the record into a unified epistemic event graph whose edges connect the propositions the agent states to the executed actions that test them; each node carries an exact excerpt that code verifies against the record. One principle governs scoring: prose can state a proposition, but only evidence returned by an executed action can support or refute it, so OEB checks what the agent writes against what it actually ran. From the graph, OEB scores four competence axes (evidence, experiment, revision, and no reward hacking), mostly as the share of opportunities for sound research that the agent took, and profiles six subjective persona traits that describe the agent's research habits. We score 119 existing runs over 12 tasks from three benchmarks: LLM post-training, chip design, and a training-speed record. Against logged results, only 16-29% of the improvements agents claim are real. On 9 of 10 tasks, the best run tries more new ideas in its second half than the worst run. The persona readings follow the model: for every trait, the model that ran explains more of its variance across runs than the task (a median of 43% against 7%).
| Comments: | 18 pages, 7 figures. Code: this https URL . Data: this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.02588 [cs.AI] |
| (or arXiv:2610.02588v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02588 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chengyang Shi [view email]
[v1]
Thu, 1 Oct 2026 23:34:37 UTC (239 KB)
来源：arXiv：cs.AI · arxiv.org