arXiv：cs.CL· Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao·· 3 小时前
MetaEncoder：以自然语言接口探索双塔编码器在多模态 System One 决策中的极限
MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
AI 导读
MetaEncoder 将预训练的 Muse-Glimmer 30B 解码器微调为指令遵循的决策编码器，采用双塔架构和单向对比学习实现请求与候选项对齐，支持自然语言接口与图像、视频辅助输入。该模型覆盖小规模闭集（<256）和百万级开放集候选空间，在 11 个基准套件、190 项任务上评估，在部分多模态决策、理解与检索任务上超越 SOTA 多模态编码器，但在推理密集型任务上仍存在局限。
正文
Authors:Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao
Abstract:System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11316 [cs.CL] |
| (or arXiv:2610.11316v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11316 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jianpeng Cheng J [view email]
[v1]
Thu, 8 Oct 2026 06:22:48 UTC (324 KB)
来源：arXiv：cs.CL · arxiv.org