arXiv：cs.LG· Jiangze Yan, Yi Shen, Wenjing Zhang, Jieyun Huang, Ping Chen, Zhaoxiang Liu, Ning Wang, Kai Wang, Shiguo Lian·· 10 小时前
HiMPO：面向长时程智能体的后见信息记忆策略优化，降低信用纠缠
HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents
AI 导读
HiMPO 是一种后见信息记忆策略优化框架，为轨迹级强化学习补充记忆专属信用：每次更新时在固定其余写入前上下文的前提下，比较目标答案在旧记忆与新记忆下的 log-likelihood，并用基于组相对后见相关性的软门控加权局部效用优势。
正文
Abstract:Long-horizon agents compress interaction histories into memory, but final outcomes entangle memory-writing credit with tool failures and reasoning errors. We propose HiMPO, a Hindsight-Informed Memory Policy Optimization framework that supplements trajectory-level reinforcement learning with memory-specific credit. For each update, HiMPO compares the target answer's log-likelihood under the previous and updated memories while holding the remaining pre-write context fixed. A soft gate based on group-relative hindsight relevance then weights the local utility advantage. Trajectory-level credit applies to all generated tokens; memory tokens additionally receive the auxiliary advantage. Across judge-based open-domain tasks and objective compressive-memory QA, HiMPO improves task performance while preserving compressed-context efficiency. On Local Wiki Search, it improves average F1 and exact match over MemPO by 2.6 and 2.2 percentage points, respectively. Controlled interventions and live memory-reversion studies provide evidence of reduced blame leakage and alignment between memory credit and behavioral effects. With the policy fixed, the credit estimator remains robust to the tested forms of target noise.
| Comments: | Preprint. 2 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.16285 [cs.CL] |
| (or arXiv:2606.16285v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.16285 arXiv-issued DOI via DataCite |
Submission history
From: Jiangze Yan [view email]
[v1]
Mon, 15 Jun 2026 06:49:18 UTC (2,897 KB)
[v2]
Tue, 1 Sep 2026 03:33:12 UTC (489 KB)
[v3]
Wed, 14 Oct 2026 01:37:07 UTC (493 KB)
来源：arXiv：cs.LG · arxiv.org