arXiv：cs.AI· Bofan Xie, Chaokun Wang, Guanchen Ge, Songyao Wang, Fang Niu, Hang Zhang, Wei Wang·· 16 小时前
Bemula：LLM 智能体记忆遗忘基准测试
Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents
AI 导读
研究者提出 Bemula，首个专门评估 LLM 智能体记忆遗忘能力的基准，包含 50 个多轮对话、6 类共 135 条遗忘请求和 1742 道测试题。在 3 个主干 LLM 上评测 6 种主流记忆系统及 Full Context LLM，结果均为负面：没有任何系统端到端通过超过 1.5% 的遗忘请求，多数系统一条都未通过。
正文
Abstract:Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18052 [cs.AI] |
| (or arXiv:2610.18052v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18052 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bofan Xie [view email]
[v1]
Tue, 13 Oct 2026 09:41:12 UTC (1,321 KB)
来源：arXiv：cs.AI · arxiv.org