arXiv：cs.LG· Ehsan Futuhi, Nathan R. Sturtevant·· 13 小时前
Transformer 规划新架构 COC：链式计算与结构化上下文窗口
Planning with Transformers: Chain of Computation and Structured Context Windows
AI 导读
研究提出 Chain of Computation（COC）架构，将 Transformer 语言模型置于迭代循环中，并以 Structured Context Window（SCW）作为外部记忆，使每步规划只访问所需上下文、保持近似恒定的输入规模。
正文
Abstract:Large Language Models (LLMs) have had a remarkable impact across many areas of machine learning. However, recent studies have shown that they struggle to reliably solve planning problems, particularly when compared with classical planners. Even with recent advances in reasoning-oriented models, the extent to which LLMs can genuinely perform planning remains an open question. At the same time, theoretical results have shown that transformers, the core architecture underlying modern LLMs, are Turing-complete. In this work, we investigate this apparent gap between the theoretical computational power of LLMs and their empirical planning performance. We propose Chain of Computation (COC), a computational architecture that places a transformer-based LM inside an iterative loop and leverages its strength as a pattern-matching system. COC uses a Structured Context Window (SCW) as external memory, allowing the model to access only the portion of the accumulated context required at each planning step and thereby maintain an approximately constant input size. Within this architecture, the LM learns the planning policy and world model and performs the arithmetic operations required during planning. We show that relatively small LMs trained from scratch can learn planning procedures from a small number of training instances and generalize to unseen problems, achieving success rates above 99\% on BlocksWorld and the Pancake puzzle. We further show that COC can be formulated as a pushdown automaton (PDA). This formulation can solve Tower of Hanoi (TOH) instances with up to 20 disks and enables the planner to execute the depth-first search (DFS) algorithm. Theoretically, we characterize how the accumulation of irrelevant context increases planning failure probability under the standard full-context setup and how the SCW mitigates this effect.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.17710 [cs.LG] |
| (or arXiv:2607.17710v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.17710 arXiv-issued DOI via DataCite |
Submission history
From: Ehsan Futuhi [view email]
[v1]
Mon, 20 Jul 2026 09:06:15 UTC (5,258 KB)
[v2]
Fri, 9 Oct 2026 00:43:22 UTC (5,274 KB)
来源：arXiv：cs.LG · arxiv.org