arXiv：cs.AI· Yunxiang Luo, Constantino \'Alvarez Casado, Xiaoting Wu, Nhi Nguyen, Le Nguyen, Manuel Lage Ca\~nellas, Miguel Bordallo L\'opez·· 22 小时前
用 JEPA 表示学习多模态传感器融合，实现呼吸活动与姿态识别
Learning Multimodal Sensor Fusion with JEPA Representations for Breathing Activity and Pose Recognition
AI 导读
研究将冻结的 V-JEPA2 特征迁移到 RGB 视频、深度视频与雷达时间序列三类异构传感器，用带模态查询、固定传感器槽位和显式可用性标记的共享融合模型，在 7 个非空传感器子集上识别呼吸活动与姿态。
正文
Abstract:Predictive video representations do not by themselves specify how a multimodal system should handle missing sensors. We transfer frozen V-JEPA2 features to a set of heterogeneous sensors comprised by RGB video, depth video, and radar time series. A shared fusion model uses modality queries, fixed sensor slots, and explicit availability to classify breathing activity and pose from all seven non-empty sensor subsets. The contribution is a controlled evaluation of subset supervision, pooling capacity, and representation transfer on 39 OMuSense-23 subjects with five subject-group folds and three seeds. With matched downstream parameter counts, the final model reaches recording-level activity macro-F1 of $75.23%\pm0.60$ averaged across seven subsets. Supervising all subsets in each batch improves the matched query-pooling control by 1.923 percentage points (conditional 95% subject-bootstrap interval [0.647, 3.209]), while query pooling shows no clear advantage over matched mean pooling under balanced supervision. Radar probes support transfer beyond pooled wavelet pixels, 64-token inference approximately preserves mean activity performance, and the tested Consensus-JEPA loss adds no measured benefit against its exact control.
| Comments: | 6 pages, 2 figures, 3 equations, 4 tables, conference paper |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15891 [cs.CV] |
| (or arXiv:2610.15891v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15891 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Constantino Álvarez Casado [view email]
[v1]
Sun, 11 Oct 2026 20:13:12 UTC (313 KB)
来源：arXiv：cs.AI · arxiv.org