arXiv：cs.AI· Jeonghwan Kim, Sofia Stoica, Jiwan Chung, Ansel Blume, Hyeonjeong Ha, Zhenhailong Wang, Xin Luna Dong, Heng Ji·· 6 小时前
Mixture of Layers：面向视觉推理的动态层路由
Mixture of Layers: Dynamic Layer Routing for Visual Reasoning
AI 导读
针对多数 MLLM 只用视觉编码器最后或倒数第二层表示、视觉抽象与查询无关的问题，研究者提出 Mixture of Layers（MoL），在视觉 patch 级别按指令动态路由并 top-k 稀疏聚合中间层隐状态。
正文
Abstract:Pre-trained vision encoders contain layer-wise visual representations that differ in spatial granularity, semantic abstraction, and sensitivity to local details. However, most Multimodal Large Language Models (MLLMs) rely on only the final or penultimate vision encoder representations or fixed aggregation rules, making visual abstraction largely query-agnostic and limiting access to fine-grained cues such as small objects, spatial details, text, and subtle visual attributes. In this work, we propose Mixture of Layers (MoL), an instruction-conditioned layer routing approach at the visual patch level that dynamically aggregates query-relevant latent representations from intermediate vision encoder layers. Given a text query, MoL predicts routing probabilities over vision encoder layers and performs a top-k sparse aggregation over selected hidden states at either the image level, patch level, or through a hybrid routing mechanism. In doing so, MoL enables query-adaptive access to layer-specific visual features for fine-grained visual reasoning. Our experiments across 7 fine-grained visual reasoning tasks demonstrate substantial performance improvements, especially across fine-grained visual grounding and understanding tasks such as +18.9% improvement on V* in overall accuracy, +4.5% on HRBench4K, and +16.3% on CharXiv compared to the baseline MLLMs, without resorting to multi-resolution inputs, simple interleaving of multiple vision encoders, or increasing the number of patch tokens. We study vision encoders' receptive field scales across different layers and their sampling behaviors to provide an in-depth analysis of why layer-wise sampling is helpful, demonstrating that conditional visual representations are a key step towards better visual perception and reasoning in MLLMs. Our project page is available at this https URL.
| Comments: | NeurIPS 2026 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.09440 [cs.CV] |
| (or arXiv:2610.09440v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09440 arXiv-issued DOI via DataCite |
Submission history
From: Jeonghwan Kim [view email]
[v1]
Wed, 7 Oct 2026 04:50:48 UTC (10,653 KB)
[v2]
Sat, 10 Oct 2026 21:23:10 UTC (10,653 KB)
来源：arXiv：cs.AI · arxiv.org