arXiv：cs.AI· Idriss Malek, Omar Choukrani, Daniil Orel, Anh Duy Le Dinh, Zhuohan Xie, Zangir Iklassov, Martin Tak\'a\v{c}, Salem Lahlou·· 16 小时前
LLM-BabyBench：语言模型能在可模拟的世界中规划吗？
LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?
AI 导读
研究者提出 LLM-BabyBench，将程序化生成的 BabyAI 网格世界改造为完全可观测的纯文本环境，把规划能力从感知、检索等因素中单独分离出来评估。其 PPD 套件包含 Predict、Plan、Decompose 三项任务，对七个前沿与开源模型零样本测试后发现，所有模型的模拟能力都远超规划能力，最强模型已接近饱和，且所需解的长度决定规划失败。
正文
Abstract:Interactive benchmarks for LLMs usually report a single metric, even though succeeding in them requires models to show several different skills. While those benchmarks are useful for measuring the overall capability of models to perform the downstream task, it is harder to identify which skill was lacking and caused failure solely from the aggregate score. For example, in benchmarks where models are supposed to navigate a simulated world to perform a task, failures can come from perception, an ambiguous instruction, retrieval, missing commonsense about what actions do, unknown dynamics, or planning. \benchmark recasts the procedurally generated BabyAI gridworld as a fully observable, purely textual environment designed to evaluate planning separately from those factors. The nature of the environment and the structure of the prompt control for the other factors cited above. To evaluate the models, we define the PPD suite: \textbf{Predict} asks for the state that follows an action sequence, \textbf{Plan} asks for an action sequence that reaches a goal, and \textbf{Decompose} asks for a subgoal sequence that achieves a mission. Across seven frontier and open models, queried zero-shot, we find simulation far ahead of planning for every model, close to saturated for the strongest. The length of the required solution governs planning failure, and each of the four strongest models has a horizon beyond which single-attempt success collapses. The responses of claude-opus-4-6 and claude-sonnet-4-6, which show their working step by step, reveal that they plan by proposing L-shaped corridors, which is consistent with their successful plans staying near-optimal while success falls. At the subgoal level, decomposition precision likewise falls to zero on long missions, while comprehension of those missions persists.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2505.12135 [cs.AI] |
| (or arXiv:2505.12135v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2505.12135 arXiv-issued DOI via DataCite |
Submission history
From: Idriss Malek [view email]
[v1]
Sat, 17 May 2025 20:23:17 UTC (240 KB)
[v2]
Fri, 11 Sep 2026 14:58:03 UTC (327 KB)
[v3]
Sun, 11 Oct 2026 16:59:10 UTC (326 KB)
来源：arXiv：cs.AI · arxiv.org