HuggingFace Daily Papers·· 2 天前
OpenVLHarness：基于多模态记忆的真实世界视觉智能体工具调用
OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory
AI 导读
OpenVLHarness 是一个围绕能力层与持久多模态记忆组织起来的多模态推理框架，让模型通过异构工具获取图像、结构化数据和网络引用并在推理中复用。在 22 个基准、8 个开源与闭源骨干模型上，平均分比无工具基线提升 9.7-12.6 分，其中 Qwen3-VL-8B 提升 11.2 分、Kimi K3 提升 9.7 分、GPT-6 Astra 提升 9.9 分。
正文 · 原文
Authors:Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang
Abstract:Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM) |
| Cite as: | arXiv:2610.26860 [cs.CL] |
| (or arXiv:2610.26860v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26860 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yu Zhou [view email]
[v1]
Mon, 19 Oct 2026 17:59:58 UTC (43,516 KB)
来源：HuggingFace Daily Papers · arxiv.org