arXiv：cs.AI· Yutong Chen, Zhichao Xu, Rui Liu, Kunpeng Liu·· 23 小时前
LEAF：用上下文记忆与自蒸馏提升 LLM 推理探索效率
Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning
AI 导读
针对 RLVR 中独立采样探索效率低的问题，研究者提出 LEAF，在每次 rollout 前向上下文注入动态的、问题专属的记忆，内容包含最近一次回答、其奖励及其他环境反馈，并在每次验证器调用后更新。
正文
Abstract:Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR). Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback. We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout. The memory contains the most recent response, its reward, and other environment feedback. It is updated after every verifier call. We decouple policy updates from rollouts and update the policy through self-distillation. On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline. On unseen test sets, it improves accuracy over the strongest baseline by 1.59 and 1.83 percentage points on the two models, while using less compute than any RL baseline. These results highlight the potential of LEAF in improving both exploration and learning efficiency. Code is available at: this https URL
| Comments: | preprint |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.17734 [cs.LG] |
| (or arXiv:2610.17734v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17734 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yutong Chen [view email]
[v1]
Tue, 13 Oct 2026 05:13:10 UTC (281 KB)
来源：arXiv：cs.AI · arxiv.org