arXiv：cs.LG· Boyang Sun, Jiajie Li, Yung-Hsu Yang, Chenyangguang Zhang, Tim Engelbracht, Sunghwan Hong, Cesar Cadena, Marc Pollefeys, Hermann Blum·· 2 小时前
LIME：从第一视角视频学习意图感知的相机运动
LIME: Learning Intent-aware Camera Motion from Egocentric Video
AI 导读
研究者提出 LIME，一个视觉语言相机运动生成器，可根据当前 RGB 观测和自然语言意图预测下一步观测的相对目标相机位姿。该方法通过 hindsight labeling 从第一视角视频中挖掘多意图相机运动监督，将每个相对 SE(3) 目标位姿与可能的意图及观测增益描述配对，并结合自回归观测增益输出与连续 flow-matching 位姿头。
正文
Abstract:Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent. While vision-language navigation translates instructions to base motion and vision-language-action policies map instructions to manipulation actions, vision-language conditioned camera motion remains comparatively underexplored as a first-class action. We formulate vision-language conditioned camera motion generation: given a current RGB observation and a free-form natural-language intent, predict a relative target camera pose for the next observation. The challenge is that useful viewpoint changes depend on latent perceptual intent, ranging from coarse spatial moves to fine-grained inspection or occlusion revealing. To model this structure, we mine multi-intent camera-motion supervision from egocentric video via hindsight labeling, pairing each relative SE(3) target pose with plausible intents and observation-gain descriptions inferred from the reached view. We propose LIME, a vision-language camera-motion generator that combines an autoregressive observation-gain output with a continuous flow-matching pose head. This design lets the model jointly predict what the next view should reveal while representing multi-hypothesis target views. Across viewpoint-prediction experiments and downstream robotic tasks, we show that LIME learns intent-conditioned camera motion from passive egocentric video, turning ordinary human recordings into supervision for a reusable active-perception primitive that supports general-purpose viewpoint selection, manipulation, embodied question answering, and mid-distance navigation.
| Subjects: | Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.02417 [cs.RO] |
| (or arXiv:2607.02417v2 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02417 arXiv-issued DOI via DataCite |
Submission history
From: Boyang Sun [view email]
[v1]
Thu, 2 Jul 2026 16:48:43 UTC (18,400 KB)
[v2]
Tue, 13 Oct 2026 22:58:41 UTC (18,542 KB)
来源：arXiv：cs.LG · arxiv.org