arXiv：cs.AI· Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong·· 22 小时前
PixSearch：面向知识型大型多模态模型的区域锚定检索
PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models
AI 导读
PixSearch 是一个端到端的分割式大型多模态模型（LMM），将区域级视觉锚定与检索增强推理统一起来，在自回归生成中自行决定何时检索、如何在文本、整图和区域级检索间路由查询。在 CRAG-MM 上，它相比整图检索取得 18.3% 的相对准确率提升，在自指图像上增益尤为明显。相比缓存式 GroundedSAM-LLaVA 流水线，其推理延迟降低 36.4%。
正文
Authors:Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong
Abstract:Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on modular pipelines that separate visual grounding, query construction, and answer generation. This separation can propagate errors from intermediate detections, masks, or captions into retrieval while adding latency and limiting coordination with ongoing reasoning. These limitations are especially consequential for egocentric images, where questions often concern small, off-center, or long-tail entities that global retrieval can overlook. We propose PixSearch, an end-to-end segmenting Large Multimodal Model (LMM) that unifies region-level visual grounding with retrieval-augmented reasoning. During autoregressive generation, PixSearch learns when to retrieve, how to route queries across text, whole-image, and region-level retrieval, and how to crop the target entity as a visual query using a jointly trained mask decoder. A two-stage supervised fine-tuning regimen with search-interleaved supervision teaches retrieval triggering, query selection, and evidence-grounded answer generation while preserving segmentation ability. On egocentric and entity-centric VQA benchmarks, PixSearch achieves an 18.3% relative accuracy gain on CRAG-MM over whole-image retrieval, with particularly strong gains on egocentric images, while remaining competitive on broader VQA and text-only QA tasks. It also reduces inference latency by 36.4% compared with a cached GroundedSAM-LLaVA pipeline.
| Comments: | NeurIPS 2026 VLM4RWD (Oral) |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2601.19060 [cs.CV] |
| (or arXiv:2601.19060v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2601.19060 arXiv-issued DOI via DataCite |
Submission history
From: Jeonghwan Kim [view email]
[v1]
Tue, 27 Jan 2026 00:46:08 UTC (40,539 KB)
[v2]
Tue, 13 Oct 2026 04:03:22 UTC (40,560 KB)
来源：arXiv：cs.AI · arxiv.org