HuggingFace Daily Papers·· 2 天前
EgoXperience：从人类第一视角视频学习世界锚定的自我中心表征
EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience
AI 导读
EgoXperience 提出一套从数据采集、自动标注到统一建模与评测的完整范式，用于从第一视角视频联合重建相机、场景、手与物体。
正文 · 原文
Authors:Xinhao Ji, Hui En Pang, Hao Li, Kairui Hu, Jiner Wang, Mingze Sun, Ziang Cao, Yalun Dai, Thang Luu, Qin Ma, Hongyi Chen, Yukang Cao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu
Abstract:Reconstructing human activity from egocentric video requires more than recovering individual 3D components: the observer's motion, surrounding geometry, and hand-object interactions must fit together in a consistent world. Achieving this coherence requires both joint reconstruction and aligned supervision, yet existing methods and datasets offer limited support for either. We present EgoXperience, a full-stack paradigm that addresses these needs through data collection, automatic annotation, unified modeling, and benchmarking. Central to EgoXperience is an automatic annotation engine, EgoX-Engine, that couples geometrically verified object tracking with SLAM-based world alignment, yielding consistent supervision for the camera, scene, hands, and objects. This yields EgoX-Data-6M, a 50-hour dataset of task-driven manipulation and locomotion within and across indoor and outdoor environments, with second-level task captions. We carefully select one hour across these activity categories as EgoX-Bench to evaluate joint reconstruction. To translate this supervision into reconstruction, we introduce EgoX, trained with EgoX-Data-6M and complementary annotations from additional public datasets. Given monocular egocentric video and first-frame object masks, our model achieves joint metric reconstruction of the camera, scene, hands, and objects in a shared world coordinate frame with a single forward pass. Extensive experiments demonstrate state-of-the-art performance across scene estimation and world-space hand-object pose recovery. Against the strongest evaluated baselines, EgoX reduces camera, hand, and depth errors by 68.9%, 65.6%, and 42.9%, and improves object pose scores by 28.0%.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.26625 [cs.CV] |
| (or arXiv:2610.26625v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26625 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xinhao Ji [view email]
[v1]
Mon, 19 Oct 2026 15:56:05 UTC (37,412 KB)
来源：HuggingFace Daily Papers · arxiv.org