ALPS：面向世界模型的视觉状态编码器

View PDF HTML (experimental)

Abstract:World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.

Comments:	35 pages (15 main + 20 supplementary), 13 figures, 26 tables. Project page: this https URL Code: this https URL
Subjects:	Computer Vision and Pattern Recognition (cs.CV)
Cite as:	arXiv:2610.30120 [cs.CV]
	(or arXiv:2610.30120v1 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2610.30120 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yanqing Liu [view email]
[v1] Wed, 21 Oct 2026 17:57:11 UTC (21,022 KB)