arXiv：cs.AI· Hongyu Wu, Xu Wu, Tianhao Wu, Jiawei Yu, Phuc Nguyen, Jian Liu, Yi Wu·· 6 小时前
SeRV：面向美国手语生成的语义对齐残差向量量化
SeRV: Semantic-Aligned Residual Vector Quantization for American Sign Language Generation
AI 导读
研究者提出 SeRV（语义对齐残差向量量化），一种面向美国手语（ASL）生成的语义对齐 RVQ tokenizer，通过全局运动-文本对齐与 token 级文本条件监督学习语义化残差 token 空间，并由 Hierarchical GPT 以粗到细方式预测残差运动 token，生成结构连贯、语义对齐的 3D ASL 运动。
正文
Abstract:American Sign Language (ASL) generation remains challenging due to limited paired text-ASL motion data and the difficulty of learning motion representations both precise for reconstruction and predictable from linguistic input. Existing methods rely on motion tokenizers optimized for reconstruction, without explicit semantic supervision from paired text. As a result, the learned tokens remain limited in supporting semantically consistent and fine-grained ASL motion generation. To address this limitation, we propose SeRV (Semantic-Aligned Residual Vector Quantization), a semantic-aligned RVQ tokenizer for ASL generation. SeRV learns a semantically structured residual token space by combining global-level motion-text alignment with token-level text-conditioned supervision. Building on this tokenizer, a Hierarchical GPT predicts residual motion tokens in a coarse-to-fine manner, generating structurally coherent and semantically aligned 3D ASL motion. We further construct a large-scale reconstructed 3D ASL motion-text benchmark by recovering paired 3D motion from OpenASL videos. Experiments on How2Sign and OpenASL show that SeRV outperforms six state-of-the-art baselines, with user studies assessing reconstruction quality and semantic fidelity. Samples of our reconstructed 3D ASL motion are available on our project page: this https URL.
| Comments: | 18 pages; revised manuscript with updated experiments and additional analyses |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.05742 [cs.CV] |
| (or arXiv:2609.05742v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.05742 arXiv-issued DOI via DataCite |
Submission history
From: Hongyu Wu [view email]
[v1]
Fri, 4 Sep 2026 21:41:17 UTC (43,450 KB)
[v2]
Tue, 13 Oct 2026 05:39:27 UTC (5,886 KB)
来源：arXiv：cs.AI · arxiv.org