arXiv：cs.AI· Zihan Gao, Yuhong Zhang, Mengzhe Wang, Kaisheng Liu, Shengpeng Li, Siyao Xiao, Runqing Cheng, Xiao Lin, Zhifang Liu, Haoqian Wang·· 17 小时前
DexCast：从人类视频中学习真实世界灵巧手控制
DexCast: Learning Real-World Dexterous Control from Human Videos
AI 导读
DEXCAST 是一个无需遥操作、直接从单目 RGB-D 人类视频中学习灵巧机械手控制的框架，依托联合手-物交互（HOI）重建实现模仿学习。其提出的 DexHand Motion Perception Module（DMPM）以紧凑隐表征捕捉关节间协调与时序依赖，为扩散策略提供条件。大量真实世界实验表明，DEXCAST 在多种灵巧操作任务上表现优异。
正文
Abstract:Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14691 [cs.RO] |
| (or arXiv:2610.14691v2 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14691 arXiv-issued DOI via DataCite |
Submission history
From: Zihan Gao [view email]
[v1]
Sat, 10 Oct 2026 08:08:41 UTC (11,534 KB)
[v2]
Wed, 14 Oct 2026 07:14:55 UTC (11,534 KB)
来源：arXiv：cs.AI · arxiv.org