arXiv：cs.LG· Lorenzo Venturoli, Nedko Savov, Xi Wang, Marc Pollefeys, Luc Van Gool·· 4 小时前
StateCraft：用抽象状态推理增强视频世界模型
StateCraft: Augmenting Video World Models with Abstract-State Reasoning
AI 导读
StateCraft 通过将可持久更新的抽象状态与视觉状态解耦，用语言模型搭配视频扩散模型演化动态世界，在生成质量、记忆和可控性基准上超过 SOTA 基线。该架构在长轨迹中保持高自一致性与生成质量，且抽象状态可被直接编辑；概念验证的 3D 实验显示，其对未观测房间光照变化的动作同样适用。
正文
Abstract:Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities' changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves surpasses the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Project page: this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19795 [cs.CV] |
| (or arXiv:2610.19795v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19795 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lorenzo Venturoli [view email]
[v1]
Wed, 14 Oct 2026 08:39:41 UTC (17,489 KB)
来源：arXiv：cs.LG · arxiv.org