arXiv：cs.LG· Binchi Zhang, Apurva Narayan, Atrisha Sarkar·· 5 小时前AI 评分51
arXiv 论文提出定向语义替换攻击，小扰动即可重写 VLM 的视觉感知
It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $\epsilon \leq 4/255$
AI 导读
arXiv 论文（arXiv:2609.38298）提出针对视觉语言模型（VLM）的定向语义替换攻击，在白盒设定下对齐源图像与目标图像在受害模型 post-merger token 空间中的各流表示，在 ε≤4/255 的小扰动下即可成功。
正文
Abstract:Vision Language Models (VLMs) are widely deployed in safety-critical scenarios, and understanding to which extent they can be controlled by adversarial perturbation is a prerequisite for evaluating their trustworthiness. Existing representation-alignment attacks, which make a VLM perceive a target image, achieve limited success at $\varepsilon \leq 4/255$. Therefore, VLMs seems robust to perturbations in this range. We show that this robustness does not hold, as targeted semantic substitution succeeds within the same range. Specifically, we align each stream of the source image with its counterpart in the target image in the victim VLM's post-merger token space, operating under a white-box threat model. We evaluate under a strict success criterion, requiring the model to simultaneously name the target, confirm its presence, and deny the source. In images, target semantics appear at $\varepsilon = 2/255$ and complete replacement reaches 38% at $\varepsilon = 4/255$. On video, complete replacement reaches 35.9% at $\varepsilon = 1/255$. We also observe a phenomenon of semantic fusion, where Large Language Model (LLM) rationalizes contradictory visual signals into a coherent narrative.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.38298 [cs.CV] |
| (or arXiv:2609.38298v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38298 arXiv-issued DOI via DataCite |
Submission history
From: Binchi Zhang [view email]
[v1]
Tue, 29 Sep 2026 17:52:39 UTC (4,292 KB)
[v2]
Fri, 2 Oct 2026 03:10:21 UTC (4,292 KB)
来源：arXiv：cs.LG · arxiv.org