arXiv：cs.AI· Zhengyang Luo, Xiaoyu Qiu, Zhiheng Tan, Yuchen Liang, Yingbin Liang, Liyue Shen·· 4 小时前
统一 VLM 如何理解条件引导：从注意力探针到 HeadTuned-CFG 引导调优
Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation
AI 导读
研究通过逐注意力头探针发现，统一 VLM 在 text-to-image 与 image-to-text 两个方向上，都存在一部分注意力头对无分类器引导（CFG）条件变化更敏感，另一部分则无关，且这一模式跨样本稳定。基于此提出免训练推理干预方法 HeadTuned-CFG，在注意力头级别选择并调整条件引导，在两种生成方向上均取得一致收益，T2I 生成上相对强基线提升显著。
正文
Abstract:Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how the backbone processes the condition in either direction remains poorly understood. We study this question from the perspective of conditional guidance: we probe every attention head to show the activation difference between two branches of classifier-free guidance (CFG) with different conditional guidance. On two unified VLMs of autoregressive and masked-diffusion paradigms, the probing reveals a subset of heads that are more sensitive to the change of condition while others are agnostic to it. This characteristic pattern holds in both T2I and I2T generation and is largely stable across data samples, demonstrating the internal mechanism of unified VLMs. Motivated by this, we propose a new inference-time intervention approach for unified multimodal generation, named \textit{HeadTuned-CFG}, which uses the probed condition-change awareness to select and tune the conditional guidance at the attention-head level. Experiments show consistent benefits in both multimodal generation directions of unified VLMs without additional training, and in particular substantial improvements over strong baselines in T2I generation. These results further support the key finding of our probing analysis: the observed pattern of functional specialization is an intrinsic property of unified VLMs.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15227 [cs.AI] |
| (or arXiv:2610.15227v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15227 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhengyang Luo [view email]
[v1]
Sat, 10 Oct 2026 23:40:27 UTC (1,432 KB)
来源：arXiv：cs.AI · arxiv.org