arXiv：cs.CL· Yisheng Li, Dongming Han, Hongkun Pan, Bingcheng Mao, Zijian Wang, Yunlong Mao, Wei Tong, Sheng Zhong·· 11 小时前
MisVisAgentBench：评估误导性图表如何影响 Web Agent 执行的配对基准
When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks
AI 导读
研究者发布 MisVisAgentBench，首个评估误导性可视化如何影响 Web Agent 执行的配对可执行基准，包含 280 个仅图表证据不同的干净-误导配对任务实例。在 13 个视觉语言 Web Agent 上，将干净图表替换为误导性图表后，平均任务成功率下降超过 34 个百分点，且失败主要沿目标误导分支发生。该基准已被 EMNLP 2026 接收，论文共 19 页。
正文
Abstract:Web agents increasingly browse, interpret, and act on web content on behalf of users. In many realistic workflows, however, the visual evidence they consume is not neutral: charts and dashboards can be misleading, shaping judgments without issuing any explicit malicious instruction. Unlike chart question answering, Web Agents treat charts as intermediate evidence whose misinterpretation can propagate into concrete downstream actions --- a reliability risk that remains underexplored. We introduce MisVisAgentBench, the first paired executable benchmark for evaluating how misleading visualizations affect Web Agent execution. It comprises 280 task instances in matched clean--misleading pairs that differ only in chart evidence. We further provide a diagnostic framework that isolates misleading-induced failures from baseline task difficulty (Zhu et al., 2025) and captures execution-level effects such as extra browsing steps and decision hesitation. Across thirteen vision-language Web Agents, replacing clean charts with misleading counterparts reduces the average task success rate by more than 34 percentage points. Moreover, these failures predominantly follow the targeted misleading action branch, suggesting that deceptive charts systematically redirect rather than randomly disrupt agent decisions. These results reveal that visualization robustness is a critical yet understudied dimension of Web Agent safety and reliability.
| Comments: | Accepted to EMNLP 2026. 19 pages |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.12823 [cs.CL] |
| (or arXiv:2610.12823v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12823 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yisheng Li [view email]
[v1]
Fri, 9 Oct 2026 00:03:13 UTC (2,475 KB)
来源：arXiv：cs.CL · arxiv.org