arXiv：cs.AI· Songze Li, Xiaoke Guo, Tianqi Liu, Biao Yi, Zhaoyan Gong, Zhiqiang Liu, Huajun Chen, Wen Zhang·· 4 小时前AI 评分43
多模态 GUI 推理缺了什么？UI-in-the-Loop 范式与 UI Comprehension-Bench
What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning
AI 导读
研究者提出 UI-in-the-Loop（UILoop）GUI 推理范式，将任务重构为 Screen-UI elements-Action 循环，让 MLLM 显式学习关键 UI 元素的定位、语义功能与实际用法，实现精准元素发现与可解释推理。
正文
Abstract:Existing Graphical User Interface (GUI) reasoning tasks remain challenging, particularly in UI understanding. Current methods typically rely on direct screen-based decision-making, which lacks interpretability and overlooks a comprehensive understanding of UI elements, ultimately leading to task failure. To enhance the understanding and interaction with UIs, we propose an innovative GUI reasoning paradigm called UI-in-the-Loop (UILoop). Our approach treats the GUI reasoning task as a cyclic Screen-UI elements-Action process. By enabling Multimodal Large Language Models (MLLMs) to explicitly learn the localization, semantic functions, and practical usage of key UI elements, UILoop achieves precise element discovery and performs interpretable reasoning. Furthermore, we introduce a more challenging UI Comprehension task centered on UI elements with three evaluation metrics. Correspondingly, we contribute a benchmark of 26K samples (UI Comprehension-Bench) to comprehensively evaluate existing methods' mastery of UI elements. Extensive experiments demonstrate that UILoop achieves state-of-the-art UI understanding performance while yielding superior results in GUI reasoning tasks.
| Comments: | Accepted by ACL 2026 Findings |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.06995 [cs.AI] |
| (or arXiv:2604.06995v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.06995 arXiv-issued DOI via DataCite |
Submission history
From: Songze Li [view email]
[v1]
Wed, 8 Apr 2026 12:12:09 UTC (4,422 KB)
[v2]
Sat, 30 May 2026 06:36:32 UTC (4,424 KB)
[v3]
Fri, 2 Oct 2026 06:30:10 UTC (4,427 KB)
来源：arXiv：cs.AI · arxiv.org