arXiv：cs.AI· Junwei Liao, Haoting Shi, Ruiwen Zhou, Jiaqian Wang, Shengtao Zhang, Wei Zhang, Ying Wen, Zhiyu Li, Feiyu Xiong, Bo Tang, Weinan Zhang, Muning Wen·· 23 小时前
MemQ：将 Q-Learning 融入基于溯源 DAG 的自演化记忆智能体
MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs
AI 导读
MemQ 将 TD(λ) 资格迹应用于记忆 Q 值，通过记录记忆创建时检索来源的溯源 DAG 反向传播信用，信用权重随 DAG 深度 d 按 (γλ)^d 衰减。
正文
Authors:Junwei Liao, Haoting Shi, Ruiwen Zhou, Jiaqian Wang, Shengtao Zhang, Wei Zhang, Ying Wen, Zhiyu Li, Feiyu Xiong, Bo Tang, Weinan Zhang, Muning Wen
Abstract:Episodic memory allows LLM agents to accumulate and retrieve experience, but current methods treat each memory independently, i.e., evaluating retrieval quality in isolation without accounting for the dependency chains through which memories enable the creation of future memories. We introduce MemQ, which applies TD($\lambda$) eligibility traces to memory Q-values, propagating credit backward through a provenance DAG that records which memories were retrieved when each new memory was created. Credit weight decays as $(\gamma\lambda)^d$ with DAG depth $d$, replacing temporal distance with structural proximity. We formalize this setting as an Exogenous-Context MDP, separating externally supplied tasks from the evolving memory store, and represent retrieval-set values by a first-order mean of per-memory values plus an explicit approximation residual. Across five interaction, code-generation, and agentic benchmarks, MemQ improves average held-out, final runtime, and cumulative success rates over the baseline with the highest average for each metric by 2.53, 4.23, and 2.13 percentage points, respectively. Ablations on LiveCodeBench, BFCL, and DSBench support the benefit of bootstrapping and multi-depth credit propagation. Analyses of Q-values and runtime learning show that learned memory values are associated with subsequent success and that MemQ retains more of its historical task coverage. We also explore the limits of MemQ's benefits using multiple-choice reasoning tasks, where limited interaction offers fewer opportunities to reuse procedural experience. Code is available at this https URL.
| Comments: | 32 pages, 13 figures, including appendices |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.08374 [cs.AI] |
| (or arXiv:2605.08374v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.08374 arXiv-issued DOI via DataCite |
Submission history
From: Junwei Liao [view email]
[v1]
Fri, 8 May 2026 18:30:24 UTC (9,355 KB)
[v2]
Tue, 12 May 2026 12:18:41 UTC (9,355 KB)
[v3]
Thu, 14 May 2026 08:13:01 UTC (9,355 KB)
[v4]
Mon, 12 Oct 2026 11:08:11 UTC (9,600 KB)
来源：arXiv：cs.AI · arxiv.org