HuggingFace Daily Papers·· 1 天前
Joints, Camera, Action! 新模型联合生成任意骨架绑定的 3D 动作与视频
Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs
AI 导读
研究提出一种模型，可为任意骨架绑定（rig）联合生成视频与 3D 动作，并用相机将该动作配准到场景中。它将支持任意骨架的 3D 运动模型与预训练视频模型通过少量 cross-attention 层连接，用视频、机器人和编辑数据构建近 200K 训练对。评估显示其从文本驱动多样骨架的动作比视频优先流程更忠实，并能与场景互动、实现视频与骨架动作互转及关键帧控制。
正文 · 原文
Abstract:Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Graphics (cs.GR); Machine Learning (cs.LG); Robotics (cs.RO) |
| Cite as: | arXiv:2610.28632 [cs.CV] |
| (or arXiv:2610.28632v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.28632 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Noam Rotstein [view email]
[v1]
Tue, 20 Oct 2026 17:59:56 UTC (6,339 KB)
来源：HuggingFace Daily Papers · arxiv.org