arXiv：cs.LG（机器学习，全量分类）· Geyi Yang, Zikun Qu, Xiang Li, Zhiyong Wang, Min Zhang, Shipei Zeng, Zhongxiang Dai·· 14 小时前AI 评分44
GUI-HARVEST：通过证据驱动 harness 进化实现自我改进的 GUI 智能体
GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution
AI 导读
GUI-HARVEST 是一个自动 harness 优化器，可在冻结主干模型的情况下让 GUI 智能体自我改进。在 OSWorld-Verified 上，Qwen3-VL-32B-Instruct 全套件提升 12.33 分；冻结 harness 迁移让 GPT-5 在 WindowsAgentArena 50 步下提升 13.87 个百分点。
正文
Abstract:The executable harness surrounding a GUI model determines how observations are assembled, actions are executed, and verification, recovery, and termination are controlled. Compared with harness optimization for non-GUI agents, automatically optimizing this harness poses three coupled challenges: reconciling model intent with observed visual effects, diagnosing failures under variable execution outcomes, and identifying recurrent failure patterns across tasks and translating them into reusable runtime changes. We introduce GUI-HARVEST, an automatic harness optimizer that enables self-improving GUI agents with frozen backbone models. First, to ground diagnosis in observed action effects, it aligns model outputs and executed actions with before-and-after screenshots, tying findings to specific interface transitions. Second, to account for execution variability, it treats repeated runs of the same task as a joint evidence unit, using within-task comparisons to locate outcome-relevant behavioral differences. Third, it consolidates verified findings across tasks into recurring failure patterns, maps them to bounded source-code edits with predictions recorded before evaluation, and checks the predicted behavioral effects alongside task performance through repeated execution. Experiments on OSWorld-Verified show consistent held-out gains across six general-purpose open, GUI-specialized open, and proprietary backbone models; Qwen3-VL-32B-Instruct gains 12.33 points on the full suite. Frozen-harness transfer improves GPT-5 by 13.87 percentage points on WindowsAgentArena at 50 steps without further optimization. With the same backbone and initial harness, GUI-HARVEST outperforms Self-Harness and Meta-Harness, suggesting that GUI-specific diagnosis and validation help harness improvements generalize to unseen tasks. The code is available at this https URL.
| Comments: | Preprint |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.00948 [cs.LG] |
| (or arXiv:2610.00948v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00948 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhongxiang Dai [view email]
[v1]
Thu, 1 Oct 2026 02:31:20 UTC (1,706 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org