arXiv：cs.AI· Chen Zhao, Xingping Dong, Jiachun Shi, Liang Peng, Chong Wang, Zhen Lei, Ran He, Bo Du·· 4 小时前
ResOT：用局部化分布对齐修复大型视觉语言模型的目标幻觉
From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment
AI 导读
针对大型视觉语言模型（LVLM）的目标幻觉问题，研究者提出免训练方法 ResOT，在推理阶段通过局部化分布对齐修复隐藏表示。该方法将主要幻觉方向投影出忠实子空间，形成低维残差子空间，并用高斯最优传输（OT）对齐幻觉分布与忠实分布，自适应控制每个 token 状态向 OT 目标的移动幅度。在三个代表性 LVLM 上，ResOT 显著降低目标幻觉，同时提升图像描述质量与多模态基准性能，代码将开源。
正文
Abstract:Object hallucination remains a major obstacle for large vision-language models (LVLMs) to generate reliable content. An intuitive mitigation strategy is to suppress hallucination-related components in hidden representations. However, these components may also contain useful information, and suppressing them can weaken the model's multimodal capabilities. In this paper, we propose ResOT, a training-free method that repairs representations at inference time through localized distribution alignment. Specifically, ResOT projects dominant hallucinated directions away from the faithful subspace, forming a low-dimensional residual subspace for intervention. Within this subspace, ResOT uses Gaussian optimal transport (OT) to align the hallucinated distribution with the faithful one. The resulting map defines repair targets with minimal changes to the original representations. At inference, ResOT adaptively controls how far each token state moves toward its OT target. Experiments on three representative LVLMs show that ResOT substantially reduces object hallucination while improving image caption quality and multimodal performance across multiple benchmarks. Code will be released.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11826 [cs.CV] |
| (or arXiv:2610.11826v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11826 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chen Zhao [view email]
[v1]
Thu, 8 Oct 2026 12:25:50 UTC (1,198 KB)
来源：arXiv：cs.AI · arxiv.org