arXiv：cs.AI· Yang Zhang, Jiangyuan Zhao, Chenyou Fan, Jiayu Hu, Xiu Yuan, Chenjia Bai, Xiu Li·· 6 小时前
V-JEPA Policy：在预测式视觉潜空间上构建世界-动作模型
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
AI 导读
V-JEPA Policy 提出一种世界-动作模型框架，在冻结的 V-JEPA 2.1 编码器潜空间上，从头联合训练指令条件未来潜变量预测器与流匹配动作专家。该模型总参数 0.9B、可训练参数 0.6B，在 LIBERO、LIBERO-Plus 和 RoboCasa-GR1 上达到与代表性 WAM 及视觉-语言-动作基线相当的性能。
正文
Abstract:World-action models (WAMs) couple future visual-state prediction with action generation. By adapting video generators or image-editing models pretrained at scale, a prominent line of recent WAMs inherits both predictive knowledge and the models in which it was learned. We ask whether a predictive visual latent space induced by large-scale predictive pretraining can instead provide a sufficient foundation for effective WAM learning without inheriting a complete pretrained visual generative model. To answer this question, we introduce V-JEPA Policy, a simple framework that builds a WAM on the latent space of a frozen V-JEPA 2.1 encoder. An instruction-conditioned future-latent predictor and a flow-matching action expert are jointly learned from scratch in a single downstream stage, with the predictor's future-informed context key--value states conditioning action generation. With 0.9B total parameters, of which 0.6B are trainable, V-JEPA Policy achieves competitive performance with representative WAM and vision-language-action baselines across LIBERO, LIBERO-Plus, and RoboCasa-GR1. Comparing visual foundations under the same downstream framework and training budget identifies V-JEPA latents as more effective than the discriminative, reconstructive, and video-understanding-oriented alternatives, particularly under distribution shifts. Beyond task-specific learning, pretraining the predictor on DROID video--instruction pairs without action labels and adapting it into a WAM yields substantial gains in downstream control and out-of-distribution generalization. Together, these findings establish predictive visual latents as a foundation for effective WAM learning from task-specific demonstrations and for transferring future-modeling knowledge acquired from broader in-the-wild videos. Project page: this https URL
| Comments: | 19 pages, 5 figures, 11 tables |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2609.37250 [cs.CV] |
| (or arXiv:2609.37250v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.37250 arXiv-issued DOI via DataCite |
Submission history
From: Yang Zhang [view email]
[v1]
Tue, 29 Sep 2026 11:03:02 UTC (1,627 KB)
[v2]
Sat, 10 Oct 2026 16:52:09 UTC (1,627 KB)
来源：arXiv：cs.AI · arxiv.org