Youtu-Parsing-Omni：一个编码器、一套 Schema 解析所有模态
Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities
腾讯优图提出全模态解析器 Youtu-Parsing-Omni，用单一编码器将文档、图像、音频和视频转为带时空定位的结构化记录。单个 5B 参数 checkpoint 在 OmniDocBench v1.6 取得 96.96 Overall 的 SOTA，在 OmniParsingBench 以 75.08 Avg. 领先所有开源权重模型，优势集中在音频与视频。代码与模型权重已公开。
Authors:Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, Jian Hu, Haihan Gao, Heting Gao, Xin Li, Yinsong Liu, Deqian Jiang, Di Yin, Haoyu Cao, Xing Sun, Bing Liu, Yanqiu Qu, Yunfei Wu, Kun Yin, Zhixiang Wei, Yongxiang Hua, Qianyu Li, Xiaoxing Hu, Peiyuan Zhang, Lijiang Li, Dong Xu, Zhenchuan Yang, Yiliang Jiang
Abstract:We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.39301 [cs.CV] |
| (or arXiv:2610.39301v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39301 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhongpeng Cai [view email]
[v1]
Thu, 29 Oct 2026 03:49:25 UTC (33,417 KB)
来源：HuggingFace Daily Papers · arxiv.org