arXiv：cs.AI· Zirui Shang, Xin Shu, Yang Liu, Zhi Gao, Xinxiao Wu, Lifeng Fan·· 6 小时前
Selective Knowledge Control：面向应用流的持续 GUI Agent 学习
Selective Knowledge Control for Continual GUI Agent Learning over Application Streams
AI 导读
针对 GUI 智能体在应用流上的持续学习，研究者提出激活条件选择性知识控制方法，通过神经元级梯度操纵实现选择性知识保留。该方法用紧凑历史知识状态保护高激活 MLP 神经元，新应用到来时按前向激活实时执行梯度手术：未激活神经元梯度截断防干扰，激活神经元梯度正交投影兼顾稳定与适应。多应用序列基准评测显示，该方法有效缓解对旧应用的灾难性遗忘，同时保持对新应用的稳健适应。
正文
Abstract:Continual learning is a crucial capability for Graphical User Interface (GUI) agents to adapt to evolving applications while retaining knowledge acquired from previous applications. Such application streams pose a challenging knowledge modeling problem: new applications often share underlying knowledge with past ones, yet also introduce specific knowledge that must not interfere with historical knowledge. In this paper, we propose activation-conditioned selective knowledge control, a lightweight method that achieves selective knowledge retention via neuron-level gradient manipulation. Our method maintains a compact historical knowledge state to protect highly activated MLP neurons that preserve previous knowledge. When a new application arrives, it performs real-time gradient surgery conditioned on forward activation. Concretely, the protected neurons are categorized into two types: unactivated neurons holding specific knowledge, whose gradients are truncated to prevent interference; and activated neurons holding shared knowledge, whose gradients are orthogonally projected to preserve stability while enabling adaptation. After each application stage, newly identified critical neurons are merged into the historical state for future learning. Empirical evaluations on multi-app sequential benchmark demonstrate that our method effectively mitigates catastrophic forgetting on prior applications while sustaining robust adaptation to new ones.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.06530 [cs.CV] |
| (or arXiv:2609.06530v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2609.06530 arXiv-issued DOI via DataCite |
Submission history
From: Zirui Shang [view email]
[v1]
Sun, 6 Sep 2026 10:50:55 UTC (3,671 KB)
来源：arXiv：cs.AI · arxiv.org