arXiv：cs.LG· Miko{\l}aj Pi\'orczy\'nski, Wojciech Masarczyk, Damian S\'ojka, Sebastian Cygert, Bart{\l}omiej Twardowski·· 4 小时前
基于表征的测试时自适应为何无法拯救多模态 LLM
On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs
AI 导读
研究发现，基于表征的测试时自适应（TTA）无法恢复多模态大语言模型（MLLM）在常见损坏下的准确率，即便离线使用目标数据、违反标准 TTA 假设，样本盲的特征对齐也只能带来边际提升。作者将失败归因于 MLLM 视觉 token 序列的偏移近乎正交且遍布整个嵌入空间，而 CLIP 类模型在单一池化分类嵌入中享有共享的域偏移方向。该结果界定了单编码器位置、样本盲、位置共享校正的能力上限。
正文
Abstract:Vision-language models suffer significant accuracy drops under common corruptions. While test-time adaptation (TTA) successfully mitigates these shifts in CLIP-like contrastive models, its application to generative multimodal large language models (MLLMs) remains underexplored and uniquely challenging due to their sequential token generation. In this work, we demonstrate that representation-based TTA fails to recover performance in MLLMs. Our experiments reveal that even when violating standard TTA assumptions by leveraging target data offline, sample-blind feature alignment recovers only marginal accuracy. We attribute this failure to the nature of corruption shifts: unlike CLIP-like architectures, which benefit from a shared domain-shift direction within a single pooled classification embedding, MLLM visual token sequence shifts are nearly orthogonal and span the full embedding this http URL findings bound what any sample-blind, position-shared correction at a single encoder site can achieve, establishing that representation-space interventions alone might be insufficient to ensure corruption robustness in MLLMs.
| Comments: | Accepted at the Grounded and Faithful Vision-Language Models for Real-World Deployment (VLM4RWD) workshop at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.19892 [cs.LG] |
| (or arXiv:2610.19892v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19892 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mikołaj Piórczyński [view email]
[v1]
Wed, 14 Oct 2026 09:50:59 UTC (1,525 KB)
来源：arXiv：cs.LG · arxiv.org